A 3D-video-based computerized analysis of social and sexual interactions in rats

A dual-mode, image-enhanced, miniaturized microscopy system for incubator-compatible monitoring of live cells

Imaging live cells under stable culture conditions is essential to investigate cell physiological activities and proliferation. To achieve this goal, typically, a specialized incubation chamber that creates desired culture conditions needs to be incorporated into a microscopy system to perform cell monitoring. However, such imaging systems are generally large and costly, hampering their wide applications. Recent advances in the field of miniaturized microscopy systems have enabled incubator cell monitoring, providing a hospitable environment for live cells. Although these systems are more cost-effective, they are usually limited in imaging modalities and spatial temporal resolution. Here, we present a dual-mode, image-enhanced, miniaturized microscopy system (termed MiniCube) for direct monitoring of live cells inside incubators. MiniCube enables both bright field imaging and fluorescence imaging with single-cell spatial resolution and sub-second temporal resolution. Moreover, this system can also perform cell monitoring inside the incubator with tunable time scales ranging from a few seconds to days. Meanwhile, automatic cell segmentation and image enhancement are realized by the proposed data analysis pipeline of this system, and the signal-to-noise ratio (SNR) of acquired data is significantly improved using a deep learning based image denoising algorithm. Image data can be acquired with 5 times lower light exposure while maintaining comparable SNR. The versatility of this miniaturized microscopy system lends itself to various applications in biology studies, providing a practical platform and method for studying live cell dynamics within the incubator.

Versatile multiple object tracking in sparse 2D/3D videos via deformable image registration

Tracking body parts in behaving animals, extracting fluorescence signals from cells embedded in deforming tissue, and analyzing cell migration patterns during development all require tracking objects with partially correlated motion. As dataset sizes increase, manual tracking of objects becomes prohibitively inefficient and slow, necessitating automated and semi-automated computational tools. Unfortunately, existing methods for multiple object tracking (MOT) are either developed for specific datasets and hence do not generalize well to other datasets, or require large amounts of training data that are not readily available. This is further exacerbated when tracking fluorescent sources in moving and deforming tissues, where the lack of unique features and sparsely populated images create a challenging environment, especially for modern deep learning techniques. By leveraging technology recently developed for spatial transformer networks, we propose ZephIR, an image registration framework for semi-supervised MOT in 2D and 3D videos. ZephIR can generalize to a wide range of biological systems by incorporating adjustable parameters that encode spatial (sparsity, texture, rigidity) and temporal priors of a given data class. We demonstrate the accuracy and versatility of our approach in a variety of applications, including tracking the body parts of a behaving mouse and neurons in the brain of a freely moving C. elegans. We provide an open-source package along with a web-based graphical user interface that allows users to provide small numbers of annotations to interactively improve tracking results.

Development of a 3D tracking system for multiple marmosets under free-moving conditions

Assessment of social interactions and behavioral changes in nonhuman primates is useful for understanding brain function changes during life events and pathogenesis of neurological diseases. The common marmoset (Callithrix jacchus), which lives in a nuclear family like humans, is a useful model, but longitudinal automated behavioral observation of multiple animals has not been achieved. Here, we developed a Full Monitoring and Animal Identification (FulMAI) system for longitudinal detection of three-dimensional (3D) trajectories of each individual in multiple marmosets under free-moving conditions by combining video tracking, Light Detection and Ranging, and deep learning. Using this system, identification of each animal was more than 97% accurate. Location preferences and inter-individual distance could be calculated, and deep learning could detect grooming behavior. The FulMAI system allows us to analyze the natural behavior of individuals in a family over their lifetime and understand how behavior changes due to life events together with other data.

Markerless Mouse Tracking for Social Experiments

Automated behavior quantification in socially interacting animals requires accurate tracking. While many methods have been very successful and highly generalizable to different settings, issues of mistaken identities and lost information on key anatomical features are common, although they can be alleviated by increased human effort in training or post-processing. We propose a markerless video-based tool to simultaneously track two interacting mice of the same appearance in controlled settings for quantifying behaviors such as different types of sniffing, touching, and locomotion to improve tracking accuracy under these settings without increased human effort. It incorporates conventional handcrafted tracking and deep-learning-based techniques. The tool is trained on a small number of manually annotated images from a basic experimental setup and outputs body masks and coordinates of the snout and tail-base for each mouse. The method was tested on several commonly used experimental conditions including bedding in the cage and fiberoptic or headstage implants on the mice. Results obtained without any human corrections after the automated analysis showed a near elimination of identities switches and a ∼15% improvement in tracking accuracy over pure deep-learning-based pose estimation tracking approaches. Our approach can be optionally ensembled with such techniques for further improvement. Finally, we demonstrated an application of this approach in studies of social behavior of mice by quantifying and comparing interactions between pairs of mice in which some lack olfaction. Together, these results suggest that our approach could be valuable for studying group behaviors in rodents, such as social interactions.

An automated, low-latency environment for studying the neural basis of behavior in freely moving rats

BACKGROUND

Behavior consists of the interaction between an organism and its environment, and is controlled by the brain. Brain activity varies at sub-second time scales, but behavioral measures are usually coarse (often consisting of only binary trial outcomes).

RESULTS

To overcome this mismatch, we developed the Rat Interactive Foraging Facility (RIFF): a programmable interactive arena for freely moving rats with multiple feeding areas, multiple sound sources, high-resolution behavioral tracking, and simultaneous electrophysiological recordings. The paper provides detailed information about the construction of the RIFF and the software used to control it. To illustrate the flexibility of the RIFF, we describe two complex tasks implemented in the RIFF, a foraging task and a sound localization task. Rats quickly learned to obtain rewards in both tasks. Neurons in the auditory cortex as well as neurons in the auditory field in the posterior insula had sound-driven activity during behavior. Remarkably, neurons in both structures also showed sensitivity to non-auditory parameters such as location in the arena and head-to-body angle.

CONCLUSIONS

The RIFF provides insights into the cognitive capabilities and learning mechanisms of rats and opens the way to a better understanding of how brains control behavior. The ability to do so depends crucially on the combination of wireless electrophysiology and detailed behavioral documentation available in the RIFF.

Beyond the three-chamber test: toward a multimodal and objective assessment of social behavior in rodents

MAIN: In recent years, substantial advances in social neuroscience have been realized, including the generation of numerous rodent models of autism spectrum disorder. Still, it can be argued that those methods currently being used to analyze animal social behavior create a bottleneck that significantly slows down progress in this field. Indeed, the bulk of research still relies on a small number of simple behavioral paradigms, the results of which are assessed without considering behavioral dynamics. Moreover, only few variables are examined in each paradigm, thus overlooking a significant portion of the complexity that characterizes social interaction between two conspecifics, subsequently hindering our understanding of the neural mechanisms governing different aspects of social behavior. We further demonstrate these constraints by discussing the most commonly used paradigm for assessing rodent social behavior, the three-chamber test. We also point to the fact that although emotions greatly influence human social behavior, we lack reliable means for assessing the emotional state of animals during social tasks. As such, we also discuss current evidence supporting the existence of pro-social emotions and emotional cognition in animal models. We further suggest that adequate social behavior analysis requires a novel multimodal approach that employs automated and simultaneous measurements of multiple behavioral and physiological variables at high temporal resolution in socially interacting animals. We accordingly describe several computerized systems and computational tools for acquiring and analyzing such measurements. Finally, we address several behavioral and physiological variables that can be used to assess socio-emotional states in animal models and thus elucidate intricacies of social behavior so as to attain deeper insight into the brain mechanisms that mediate such behaviors. CONCLUSIONS: In summary, we suggest that combining automated multimodal measurements with machine-learning algorithms will help define socio-emotional states and determine their dynamics during various types of social tasks, thus enabling a more thorough understanding of the complexity of social behavior.

Leaving flatland: Advances in 3D behavioral measurement

Animals move in three dimensions (3D). Thus, 3D measurement is necessary to report the true kinematics of animal movement. Existing 3D measurement techniques draw on specialized hardware, such as motion capture or depth cameras, as well as deep multi-view and monocular computer vision. Continued advances at the intersection of deep learning and computer vision will facilitate 3D tracking across more anatomical features, with less training data, in additional species, and within more natural, occlusive environments. 3D behavioral measurement enables unique applications in phenotyping, investigating the neural basis of behavior, and designing artificial agents capable of imitating animal behavior.

Measuring Locomotor Activity and Behavioral Aspects of Rodents Living in the Home-Cage

Automatization and technological advances have led to a larger number of methods and systems to monitor and measure locomotor activity and more specific behavior of a wide variety of animal species in various environmental conditions in laboratory settings. In rodents, the majority of these systems require the animals to be temporarily taken away from their home-cage into separate observation cage environments which requires manual handling and consequently evokes distress for the animal and may alter behavioral responses. An automated high-throughput approach can overcome this problem. Therefore, this review describes existing automated methods and technologies which enable the measurement of locomotor activity and behavioral aspects of rodents in their most meaningful and stress-free laboratory environment: the home-cage. In line with the Directive 2010/63/EU and the 3R principles (replacement, reduction, refinement), this review furthermore assesses their suitability and potential for group-housed conditions as a refinement strategy, highlighting their current technological and practical limitations. It covers electrical capacitance technology and radio-frequency identification (RFID), which focus mainly on voluntary locomotor activity in both single and multiple rodents, respectively. Infrared beams and force plates expand the detection beyond locomotor activity toward basic behavioral traits but discover their full potential in individually housed rodents only. Despite the great premises of these approaches in terms of behavioral pattern recognition, more sophisticated methods, such as (RFID-assisted) video tracking technology need to be applied to enable the automated analysis of advanced behavioral aspects of individual animals in social housing conditions.

Automatic mapping of multiplexed social receptive fields by deep learning and GPU-accelerated 3D videography

Social interactions powerfully impact the brain and the body, but high-resolution descriptions of these important physical interactions and their neural correlates are lacking. Currently, most studies rely on labor-intensive methods such as manual annotation. Scalable and objective tracking methods are required to understand the neural circuits underlying social behavior. Here we describe a hardware/software system and analysis pipeline that combines 3D videography, deep learning, physical modeling, and GPU-accelerated robust optimization, with automatic analysis of neuronal receptive fields recorded in interacting mice. Our system ("3DDD Social Mouse Tracker") is capable of fully automatic multi-animal tracking with minimal errors (including in complete darkness) during complex, spontaneous social encounters, together with simultaneous electrophysiological recordings. We capture posture dynamics of multiple unmarked mice with high spatiotemporal precision (~2 mm, 60 frames/s). A statistical model that relates 3D behavior and neural activity reveals multiplexed 'social receptive fields' of neurons in barrel cortex. Our approach could be broadly useful for neurobehavioral studies of multiple animals interacting in complex low-light environments.

MouseVenue3D: A Markerless Three-Dimension Behavioral Tracking System for Matching Two-Photon Brain Imaging in Free-Moving Mice

Understanding the connection between brain and behavior in animals requires precise monitoring of their behaviors in three-dimensional (3-D) space. However, there is no available three-dimensional behavior capture system that focuses on rodents. Here, we present MouseVenue3D, an automated and low-cost system for the efficient capture of 3-D skeleton trajectories in markerless rodents. We improved the most time-consuming step in 3-D behavior capturing by developing an automatic calibration module. Then, we validated this process in behavior recognition tasks, and showed that 3-D behavioral data achieved higher accuracy than 2-D data. Subsequently, MouseVenue3D was combined with fast high-resolution miniature two-photon microscopy for synchronous neural recording and behavioral tracking in the freely-moving mouse. Finally, we successfully decoded spontaneous neuronal activity from the 3-D behavior of mice. Our findings reveal that subtle, spontaneous behavior modules are strongly correlated with spontaneous neuronal activity patterns.

Chemogenetic activation of nigrostriatal dopamine neurons in freely moving common marmosets

To interrogate particular neuronal pathways in nonhuman primates under natural and stress-free conditions, we applied designer receptors exclusively activated by designer drugs (DREADDs) technology to common marmosets. We injected adeno-associated virus vectors expressing the excitatory DREADD hM3Dq into the unilateral substantia nigra (SN) in four marmosets. Using multi-tracer positron emission tomography imaging, we detected DREADD expression in vivo, which was confirmed in nigrostriatal dopamine neurons by immunohistochemistry, as well as by assessed activation of the SN following agonist administration. The marmosets rotated in a contralateral direction relative to the activated side 30-90 min after consuming food containing the highly potent DREADD agonist deschloroclozapine (DCZ) but not on the following days without DCZ. These results indicate that non-invasive and reversible DREADD manipulation will extend the utility of marmosets as a primate model for linking neuronal activity and natural behavior in various contexts.

Multi-View Mouse Social Behaviour Recognition With Deep Graphic Model

Home-cage social behaviour analysis of mice is an invaluable tool to assess therapeutic efficacy of neurodegenerative diseases. Despite tremendous efforts made within the research community, single-camera video recordings are mainly used for such analysis. Because of the potential to create rich descriptions for mouse social behaviors, the use of multi-view video recordings for rodent observations is increasingly receiving much attention. However, identifying social behaviours from various views is still challenging due to the lack of correspondence across data sources. To address this problem, we here propose a novel multi-view latent-attention and dynamic discriminative model that jointly learns view-specific and view-shared sub-structures, where the former captures unique dynamics of each view whilst the latter encodes the interaction between the views. Furthermore, a novel multi-view latent-attention variational autoencoder model is introduced in learning the acquired features, enabling us to learn discriminative features in each view. Experimental results on the standard CRMI13 and our multi-view Parkinson's Disease Mouse Behaviour (PDMB) datasets demonstrate that our proposed model outperforms the other state of the arts technologies, has lower computational cost than the other graphical models and effectively deals with the imbalanced data problem.

Body language signals for rodent social communication

Integration of social cues to initiate adaptive emotional and behavioral responses is a fundamental aspect of animal and human behavior. In humans, social communication includes prominent nonverbal components, such as social touch, gestures and facial expressions. Comparative studies investigating the neural basis of social communication in rodents has historically been centered on olfactory signals and vocalizations, with relatively less focus on non-verbal social cues. Here, we outline two exciting research directions: First, we will review recent observations pointing to a role of social facial expressions in rodents. Second, we will review observations that point to a role of 'non-canonical' rodent body language: body posture signals beyond stereotyped displays in aggressive and sexual behavior. In both sections, we will outline how social neuroscience can build on recent advances in machine learning, robotics and micro-engineering to push these research directions forward towards a holistic systems neurobiology of rodent body language.

OptiFlex: Multi-Frame Animal Pose Estimation Combining Deep Learning With Optical Flow

Animal pose estimation tools based on deep learning have greatly improved animal behaviour quantification. These tools perform pose estimation on individual video frames, but do not account for variability of animal body shape in their prediction and evaluation. Here, we introduce a novel multi-frame animal pose estimation framework, referred to as OptiFlex. This framework integrates a flexible base model (i.e., FlexibleBaseline), which accounts for variability in animal body shape, with an OpticalFlow model that incorporates temporal context from nearby video frames. Pose estimation can be optimised using multi-view information to leverage all four dimensions (3D space and time). We evaluate FlexibleBaseline using datasets of four different lab animal species (mouse, fruit fly, zebrafish, and monkey) and introduce an intuitive evaluation metric-adjusted percentage of correct key points (aPCK). Our analyses show that OptiFlex provides prediction accuracy that outperforms current deep learning based tools, highlighting its potential for studying a wide range of behaviours across different animal species.

A hierarchical 3D-motion learning framework for animal spontaneous behavior mapping

Animal behavior usually has a hierarchical structure and dynamics. Therefore, to understand how the neural system coordinates with behaviors, neuroscientists need a quantitative description of the hierarchical dynamics of different behaviors. However, the recent end-to-end machine-learning-based methods for behavior analysis mostly focus on recognizing behavioral identities on a static timescale or based on limited observations. These approaches usually lose rich dynamic information on cross-scale behaviors. Here, inspired by the natural structure of animal behaviors, we address this challenge by proposing a parallel and multi-layered framework to learn the hierarchical dynamics and generate an objective metric to map the behavior into the feature space. In addition, we characterize the animal 3D kinematics with our low-cost and efficient multi-view 3D animal motion-capture system. Finally, we demonstrate that this framework can monitor spontaneous behavior and automatically identify the behavioral phenotypes of the transgenic animal disease model. The extensive experiment results suggest that our framework has a wide range of applications, including animal disease model phenotyping and the relationships modeling between the neural circuits and behavior.

Canonical versus non-canonical transsynaptic signaling of neuroligin 3 tunes development of sociality in mice

Neuroligin 3 (NLGN3) and neurexins (NRXNs) constitute a canonical transsynaptic cell-adhesion pair, which has been implicated in autism. In autism spectrum disorder (ASD) development of sociality can be impaired. However, the molecular mechanism underlying NLGN3-mediated social development is unclear. Here, we identify non-canonical interactions between NLGN3 and protein tyrosine phosphatase δ (PTPδ) splice variants, competing with NRXN binding. NLGN3-PTPδ complex structure revealed a splicing-dependent interaction mode and competition mechanism between PTPδ and NRXNs. Mice carrying a NLGN3 mutation that selectively impairs NLGN3-NRXN interaction show increased sociability, whereas mice where the NLGN3-PTPδ interaction is impaired exhibit impaired social behavior and enhanced motor learning, with imbalance in excitatory/inhibitory synaptic protein expressions, as reported in the Nlgn3 R451C autism model. At neuronal level, the autism-related Nlgn3 R451C mutation causes selective impairment in the non-canonical pathway. Our findings suggest that canonical and non-canonical NLGN3 pathways compete and regulate the development of sociality.

Improved 3D tracking and automated classification of rodents' behavioral activity using depth-sensing cameras

Analysis of rodents' behavior/activity is of fundamental importance in many research fields. However, many behavioral experiments still rely on manual scoring, with obvious problems in reproducibility. Despite important advances in video-analysis systems and computational ethology, automated behavior quantification is still a challenge. The need for large training datasets, background stability requirements, and reduction to two-dimensional analysis (impairing full posture characterization), limit their use. Here we present a novel integrated solution for behavioral analysis of individual rats, combining video segmentation, tracking of body parts, and automated classification of behaviors, using machine learning and computer vision methods. Low-cost depth cameras (RGB-D) are used to enable three-dimensional tracking and classification in dark conditions and absence of color contrast. Our solution automatically tracks five anatomical landmarks in dynamic environments and recognizes seven distinct behaviors, within the accuracy range of human annotations. The developed free software was validated in experiments where behavioral differences between Wistar Kyoto and Wistar rats were automatically quantified. The results reveal the capability for effective automated phenotyping. An extended annotated RGB-D dataset is also made publicly available. The proposed solution is an easy-to-use tool, with low-cost setup and powerful 3D segmentation methods (in static/dynamic environments). The ability to work in dark conditions means that natural animal behavior is not affected by recording lights. Furthermore, automated classification is possible with only ~30 minutes of annotated videos. By creating conditions for high-throughput analysis and reproducible quantitative measurements of animal behavior experiments, we believe this contribution can greatly improve behavioral analysis research.

Assessment of nest building and social interaction behavior in mice exposed to acute social defeat stress using a three-dimensional depth camera

Nest building is an instinctive behavior toward protection from predators, body temperature regulation, and courtship. Previously, we discovered that acute and chronic social defeat stress suppresses the onset of nest-building behavior in male mice (C57BL/6J). Here, we analyzed nest building and other behavioral deficits induced by acute social defeat stress (ASDS). We utilized a customized cage and specifically developed observational programs for nest building, social avoidance, and other behaviors using an infrared depth camera to acquire three-dimensional (3D) data of animal behavior (Negura system). We determined the volume of nesting materials from these 3D depth images. Mice exposed to ASDS showed increased spontaneous activities, decreased rearing, and delayed nest building; however, nest-building activity was gradually recovered during the dark period of the 24 hr observation interval. At the endpoint following 24 hr, the ASDS and control groups showed no differences in nest volumes. Furthermore, we observed the time courses of both nest building and social avoidance behaviors and their relationship using the Negura system. Our data demonstrated a weak positive correlation between nest-building delay and social avoidance in ASDS mice. The Negura system can observe various behaviors that reflect the effects of social defeat stress.

EthoLoop: automated closed-loop neuroethology in naturalistic environments

Accurate tracking and analysis of animal behavior is crucial for modern systems neuroscience. However, following freely moving animals in naturalistic, three-dimensional (3D) or nocturnal environments remains a major challenge. Here, we present EthoLoop, a framework for studying the neuroethology of freely roaming animals. Combining real-time optical tracking and behavioral analysis with remote-controlled stimulus-reward boxes, this system allows direct interactions with animals in their habitat. EthoLoop continuously provides close-up views of the tracked individuals and thus allows high-resolution behavioral analysis using deep-learning methods. The behaviors detected on the fly can be automatically reinforced either by classical conditioning or by optogenetic stimulation via wirelessly controlled portable devices. Finally, by combining 3D tracking with wireless neurophysiology we demonstrate the existence of place-cell-like activity in the hippocampus of freely moving primates. Taken together, we show that the EthoLoop framework enables interactive, well-controlled and reproducible neuroethological studies in large-field naturalistic settings.

Automated Measures of Force and Motion Can Improve Our Understanding of Infants’ Motor Persistence

Every day, young learners are confronted with challenges. The degree to which they persist in overcoming those challenges, and the different ways they persist, provides critical insights into the various cognitive, motoric, and affective processes that drive behavior. Here, we present a systematic overview of the methodologies that have been traditionally used to study persistence, and offer suggestions for new approaches to the study of persistence that will make strides in moving the field forward. We argue that automated measures of force and motion, which have long been used in the study of infants’ motoric behavior, can provide a means to unravel the psychological processes that guide infants’ trying behavior. To illustrate this, we present a case study that highlights the novel lessons to be learned by the use of automated measures of force and motion regarding infants’ persistence, along with an analysis of the benefits and drawbacks of this approach, as well as detailed instructions for application. In sum, we conclude that these measures, when used in conjunction with more traditional approaches, will provide creative new insights into the nature and development of early persistence.

Data-driven analyses of motor impairments in animal models of neurological disorders

Behavior provides important insights into neuronal processes. For example, analysis of reaching movements can give a reliable indication of the degree of impairment in neurological disorders such as stroke, Parkinson disease, or Huntington disease. The analysis of such movement abnormalities is notoriously difficult and requires a trained evaluator. Here, we show that a deep neural network is able to score behavioral impairments with expert accuracy in rodent models of stroke. The same network was also trained to successfully score movements in a variety of other behavioral tasks. The neural network also uncovered novel movement alterations related to stroke, which had higher predictive power of stroke volume than the movement components defined by human experts. Moreover, when the regression network was trained only on categorical information (control = 0; stroke = 1), it generated predictions with intermediate values between 0 and 1 that matched the human expert scores of stroke severity. The network thus offers a new data-driven approach to automatically derive ratings of motor impairments. Altogether, this network can provide a reliable neurological assessment and can assist the design of behavioral indices to diagnose and monitor neurological disorders.

Rodent Activity Detector (RAD), an Open Source Device for Measuring Activity in Rodent Home Cages

Physical activity is a critical behavioral variable in many research studies and is, therefore, important to quantify. However, existing methods for measuring physical activity have limitations which include high expense, specialized caging or equipment, and high computational overhead. To address these limitations, we present an open-source, cost-effective, device for measuring rodent activity. Our device is battery powered and designed to be placed in vivarium home cages to enable high-throughput, long-term operation with minimal investigator intervention. The primary aim of this study was to assess the feasibility of using passive infrared (PIR) sensors and microcontroller-based dataloggers in a rodent home cages to collect physical activity records. To this end, we developed an open-source PIR based data-logging device called the rodent activity detector (RAD). We publish the design files and code so others can readily build the RAD in their own labs. To demonstrate its utility, we used the RAD to collect physical activity data from 40 individually housed mice for up to 10 weeks. This dataset demonstrates the ability of the RAD to (1) operate in a high-throughput installation, (2) detect high-fat diet (HFD)-induced changes in physical activity, and (3) quantify circadian rhythms in individual animals. We further validated the data output of the RAD with simultaneous video tracking of mice in multiple caging configurations, to determine the features of physical activity that it detects. The RAD is easy to build, economical, and fits in vivarium caging. The scalability of such devices will enable high-throughput studies of physical activity in research studies.

Using DeepLabCut for 3D markerless pose estimation across species and behaviors

Noninvasive behavioral tracking of animals during experiments is critical to many scientific pursuits. Extracting the poses of animals without using markers is often essential to measuring behavioral effects in biomechanics, genetics, ethology, and neuroscience. However, extracting detailed poses without markers in dynamically changing backgrounds has been challenging. We recently introduced an open-source toolbox called DeepLabCut that builds on a state-of-the-art human pose-estimation algorithm to allow a user to train a deep neural network with limited training data to precisely track user-defined features that match human labeling accuracy. Here, we provide an updated toolbox, developed as a Python package, that includes new features such as graphical user interfaces (GUIs), performance improvements, and active-learning-based network refinement. We provide a step-by-step procedure for using DeepLabCut that guides the user in creating a tailored, reusable analysis pipeline with a graphical processing unit (GPU) in 1-12 h (depending on frame size). Additionally, we provide Docker environments and Jupyter Notebooks that can be run on cloud resources such as Google Colaboratory.

Circuit Investigations With Open-Source Miniaturized Microscopes: Past, Present and Future

The ability to simultaneously image the spatiotemporal activity signatures from many neurons during unrestrained vertebrate behaviors has become possible through the development of miniaturized fluorescence microscopes, or miniscopes, sufficiently light to be carried by small animals such as bats, birds and rodents. Miniscopes have permitted the study of circuits underlying song vocalization, action sequencing, head-direction tuning, spatial memory encoding and sleep to name a few. The foundation for these microscopes has been laid over the last two decades through academic research with some of this work resulting in commercialization. More recently, open-source initiatives have led to an even broader adoption of miniscopes in the neuroscience community. Open-source designs allow for rapid modification and extension of their function, which has resulted in a new generation of miniscopes that now permit wire-free or wireless recording, concurrent electrophysiology and imaging, two-color fluorescence detection, simultaneous optical actuation and read-out as well as wide-field and volumetric light-field imaging. These novel miniscopes will further expand the toolset of those seeking affordable methods to probe neural circuit function during naturalistic behaviors. Here, we will discuss the early development, present use and future potential of miniscopes.

DeepLabCut: markerless pose estimation of user-defined body parts with deep learning

Quantifying behavior is crucial for many applications in neuroscience. Videography provides easy methods for the observation and recording of animal behavior in diverse settings, yet extracting particular aspects of a behavior for further analysis can be highly time consuming. In motor control studies, humans or other animals are often marked with reflective markers to assist with computer-based tracking, but markers are intrusive, and the number and location of the markers must be determined a priori. Here we present an efficient method for markerless pose estimation based on transfer learning with deep neural networks that achieves excellent results with minimal training data. We demonstrate the versatility of this framework by tracking various body parts in multiple species across a broad collection of behaviors. Remarkably, even when only a small number of frames are labeled (~200), the algorithm achieves excellent tracking performance on test frames that is comparable to human accuracy.

Costa R. D-Track-A semi-automatic 3D video-tracking technique to analyse movements and routines of aquatic animals with application to captive dolphins

Scoring and tracking animal movements manually is a time consuming and subjective process, susceptible to errors due to fatigue. Automated and semi-automated video-based tracking methods have been developed to overcome the errors and biases of manual analyses. In this manuscript we present D-Track, an open-source semi-automatic tracking system able to quantify the 3D trajectories of dolphins, non-invasively, in the water. This software produces a three-dimensional reconstruction of the pool and tracks the animal at different depths, using standard cameras. D-Track allows the determination of spatial preferences of the animals, their speed and its variations, and the identification of behavioural routines. We tested the system with two captive dolphins during different periods of the day. Both animals spent around 85% of the time at the surface of the Deep Area of their pool (5-meters depth). Both dolphins showed a stable average speed throughout 31 sessions, with slow speeds predominant (maximum 1.7 ms^-1). Circular swimming was highly variable, with significant differences in the size and duration of the “circles”, between animals, within-animals and across sessions. The D-Track system is a novel tool to study the behaviour of aquatic animals, and it represents a convenient and inexpensive solution for laboratories and marine parks to monitor the preferences and routines of their animals.

An automated behavior analysis system for freely moving rodents using depth image

A rodent behavior analysis system is presented, capable of automated tracking, pose estimation, and recognition of nine behaviors in freely moving animals. The system tracks three key points on the rodent body (nose, center of body, and base of tail) to estimate its pose and head rotation angle in real time. A support vector machine (SVM)-based model, including label optimization steps, is trained to classify on a frame-by-frame basis: resting, walking, bending, grooming, sniffing, rearing supported, rearing unsupported, micro-movements, and "other" behaviors. Compared to conventional red-green-blue (RGB) camera-based methods, the proposed system operates on 3D depth images provided by the Kinect infrared (IR) camera, enabling stable performance regardless of lighting conditions and animal color contrast with the background. This is particularly beneficial for monitoring nocturnal animals' behavior. 3D features are designed to be extracted directly from the depth stream and combined with contour-based 2D features to further improve recognition accuracies. The system is validated on three freely behaving rats for 168 min in total. The behavior recognition model achieved a cross-validation accuracy of 86.8% on the rat used for training and accuracies of 82.1 and 83% on the other two "testing" rats. The automated head angle estimation aided by behavior recognition resulted in 0.76 correlation with human expert annotation. Graphical abstract Top view of a rat freely behaving in a standard homecage, captured by Kinect-v2 sensors. The depth image is used for constructing a 3D topography of the animal for pose estimation, behavior recognition, and head angle calculation. Results of the processed data are displayed on the user interface in various forms.

Integrated Change Detection and Classification in Urban Areas Based on Airborne Laser Scanning Point Clouds

This paper suggests a new approach for change detection (CD) in 3D point clouds. It combines classification and CD in one step using machine learning. The point cloud data of both epochs are merged for computing features of four types: features describing the point distribution, a feature relating to relative terrain elevation, features specific for the multi-target capability of laser scanning, and features combining the point clouds of both epochs to identify the change. All these features are merged in the points and then training samples are acquired to create the model for supervised classification, which is then applied to the whole study area. The final results reach an overall accuracy of over 90% for both epochs of eight classes: lost tree, new tree, lost building, new building, changed ground, unchanged building, unchanged tree, and unchanged ground.

A Markerless 3D Computerized Motion Capture System Incorporating a Skeleton Model for Monkeys

In this study, we propose a novel markerless motion capture system (MCS) for monkeys, in which 3D surface images of monkeys were reconstructed by integrating data from four depth cameras, and a skeleton model of the monkey was fitted onto 3D images of monkeys in each frame of the video. To validate the MCS, first, estimated 3D positions of body parts were compared between the 3D MCS-assisted estimation and manual estimation based on visual inspection when a monkey performed a shuttling behavior in which it had to avoid obstacles in various positions. The mean estimation error of the positions of body parts (3-14 cm) and of head rotation (35-43°) between the 3D MCS-assisted and manual estimation were comparable to the errors between two different experimenters performing manual estimation. Furthermore, the MCS could identify specific monkey actions, and there was no false positive nor false negative detection of actions compared with those in manual estimation. Second, to check the reproducibility of MCS-assisted estimation, the same analyses of the above experiments were repeated by a different user. The estimation errors of positions of most body parts between the two experimenters were significantly smaller in the MCS-assisted estimation than in the manual estimation. Third, effects of methamphetamine (MAP) administration on the spontaneous behaviors of four monkeys were analyzed using the MCS. MAP significantly increased head movements, tended to decrease locomotion speed, and had no significant effect on total path length. The results were comparable to previous human clinical data. Furthermore, estimated data following MAP injection (total path length, walking speed, and speed of head rotation) correlated significantly between the two experimenters in the MCS-assisted estimation (r = 0.863 to 0.999). The results suggest that the presented MCS in monkeys is useful in investigating neural mechanisms underlying various psychiatric disorders and developing pharmacological interventions.

Amygdalar Auditory Neurons Contribute to Self-Other Distinction during Ultrasonic Social Vocalization in Rats

Although, clinical studies reported hyperactivation of the auditory system and amygdala in patients with auditory hallucinations (hearing others' but not one's own voice, independent of any external stimulus), neural mechanisms of self/other attribution is not well understood. We recorded neuronal responses in the dorsal amygdala including the lateral amygdaloid nucleus to ultrasonic vocalization (USVs) emitted by subjects and conspecifics during free social interaction in 16 adult male rats. The animals emitting the USVs were identified by EMG recordings. One-quarter of the amygdalar neurons (15/60) responded to 50 kHz calls by the subject and/or conspecifics. Among the responsive neurons, most neurons (Type-Other neurons; 73%, 11/15) responded only to calls by conspecifics but not subjects. Two Type-Self neurons (13%, 2/15) responded to calls by the subject but not those by conspecifics, although their response selectivity to subjects vs. conspecifics was lower than that of Type-Other neurons. The remaining two neurons (13%) responded to calls by both the subject and conspecifics. Furthermore, population coding of the amygdalar neurons represented distinction of subject vs. conspecific calls. The present results provide the first neurophysiological evidence that the amygdala discriminately represents affective social calls by subject and conspecifics. These findings suggest that the amygdala is an important brain region for self/other attribution. Furthermore, pathological activation of the amygdala, where Type-Other neurons predominate, could induce external misattribution of percepts of vocalization.

Quick, Accurate, Smart: 3D Computer Vision Technology Helps Assessing Confined Animals' Behaviour

Mankind directly controls the environment and lifestyles of several domestic species for purposes ranging from production and research to conservation and companionship. These environments and lifestyles may not offer these animals the best quality of life. Behaviour is a direct reflection of how the animal is coping with its environment. Behavioural indicators are thus among the preferred parameters to assess welfare. However, behavioural recording (usually from video) can be very time consuming and the accuracy and reliability of the output rely on the experience and background of the observers. The outburst of new video technology and computer image processing gives the basis for promising solutions. In this pilot study, we present a new prototype software able to automatically infer the behaviour of dogs housed in kennels from 3D visual data and through structured machine learning frameworks. Depth information acquired through 3D features, body part detection and training are the key elements that allow the machine to recognise postures, trajectories inside the kennel and patterns of movement that can be later labelled at convenience. The main innovation of the software is its ability to automatically cluster frequently observed temporal patterns of movement without any pre-set ethogram. Conversely, when common patterns are defined through training, a deviation from normal behaviour in time or between individuals could be assessed. The software accuracy in correctly detecting the dogs’ behaviour was checked through a validation process. An automatic behaviour recognition system, independent from human subjectivity, could add scientific knowledge on animals’ quality of life in confinement as well as saving time and resources. This 3D framework was designed to be invariant to the dog’s shape and size and could be extended to farm, laboratory and zoo quadrupeds in artificial housing. The computer vision technique applied to this software is innovative in non-human animal behaviour science. Further improvements and validation are needed, and future applications and limitations are discussed.

Murine Depression Model and its Potential Applications for Discovering Foods and Farm Products with Antidepressant-Like Effects

Advanced societies face increased health problems related to various stresses. Chronic psychological stress is a major risk factor for psychiatric disorders such as depression. Although therapeutic agents reduce several symptoms of depression, most have side effects in a broad range of the population. Furthermore, some victims of depression do not show significant improvement with any drugs, so alternative approaches are needed. Good dietary habits may potentially reduce depressive symptoms, but there is little scientific evidence thus far. Murine depression models are useful to test nutritional approaches in vivo. Our model mice subjected to a subchronic mild social defeat stress (sCSDS) paradigm show several alterations in physiological parameters and social behavior. These stress-induced symptoms in sCSDS mice can be used as cues to identify antidepressant-like natural resources including foods and farm products. We previously discovered that sCSDS mice show more vulnerability to social stress by changing dietary condition. In addition, we developed a more objective system for analyzing mouse behavior using a 3D depth-sensing camera to understand relationships between diet and behavior. The combination of sCSDS mice with 3D behavioral analysis is a powerful method for screening ingredients in foods and farm products for antidepressant-like effects.

Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning

A lack of automated, quantitative, and accurate assessment of social behaviors in mammalian animal models has limited progress toward understanding mechanisms underlying social interactions and their disorders such as autism. Here we present a new integrated hardware and software system that combines video tracking, depth sensing, and machine learning for automatic detection and quantification of social behaviors involving close and dynamic interactions between two mice of different coat colors in their home cage. We designed a hardware setup that integrates traditional video cameras with a depth camera, developed computer vision tools to extract the body "pose" of individual animals in a social context, and used a supervised learning algorithm to classify several well-described social behaviors. We validated the robustness of the automated classifiers in various experimental settings and used them to examine how genetic background, such as that of Black and Tan Brachyury (BTBR) mice (a previously reported autism model), influences social behavior. Our integrated approach allows for rapid, automated measurement of social behaviors across diverse experimental designs and also affords the ability to develop new, objective behavioral metrics.

Strain differences in temporal changes of nesting behaviors in C57BL/6N, DBA/2N, and their F1 hybrid mice assessed by a three-dimensional monitoring system

Nest building is one of the innate behaviors that are widely observed throughout the animal kingdom. Previous studies have reported specific brain regions and genetic loci associated with nest building in mice. These studies mainly evaluated the nest structure, without observing the nesting process. In this study, we evaluated the effects of strain and learning on the nesting process of mice using a 3D depth camera. To determine the quality of the nest structure, a conventional scoring method, Deacon scores 1-5, was applied to the recorded depth images. The final score of the nest, latency to start nesting behavior, and latencies to reach Deacon scores 3-5, were determined using three genetically different mouse strains-C57BL/6NCrl (B6), DBA/2NCrlCrlj (DBA), and B6D2F1/Crl (B6D2F1). The final score of the DBA nest was significantly lower than that of the B6D2F1 nest, and DBA mice showed significantly longer latency to start nest building than the other two strains in the first trial. By observing the time course of nest building, we confirmed that DBA mice took significantly longer to build their nests than B6 and B6D2F1 mice. Although we did not find any significant differences between DBA and B6 mice in the final assessment of the nest based on the Deacon method, overnight monitoring of the nesting behavior using a 3D depth camera could elucidate the clear differences in the amount of time spent nesting between DBA and B6 mice. In addition, the learning effect was more evident in DBA mice than it was in B6 in terms of latencies to reach Deacon score 3-5 in five repeated trials. DBA mice showed a gradual decrease in latency to build, whereas nesting behaviors of B6 mice were relatively consistent throughout the five trials. Therefore, our 3D depth image method gives higher resolution and structural information regarding the nesting process in mice. Future genetic analyses using the 3D assessment system will provide novel insights into the complex genetic basis for nesting and other behaviors in animals.

Assessing nest-building behavior of mice using a 3D depth camera

We developed a novel method to evaluate the nest-building behavior of mice using an inexpensive depth camera. The depth camera clearly captured nest-building behavior. Using three-dimensional information from the depth camera, we obtained objective features for assessing nest-building behavior, including "volume," "radius," and "mean height". The "volume" represents the change in volume of the nesting material, a pressed cotton square that a mouse shreds and untangles in order to build its nest. During the nest-building process, the total volume of cotton fragments is increased. The "radius" refers to the radius of the circle enclosing the fragments of cotton. It describes the extent of nesting material dispersion. The "radius" averaged approximately 60mm when a nest was built. The "mean height" represents the change in the mean height of objects. If the nest walls were high, the "mean height" was also high. These features provided us with useful information for assessment of nest-building behavior, similar to conventional methods for the assessment of nest building. However, using the novel method, we found that JF1 mice built nests with higher walls than B6 mice, and B6 mice built nests faster than JF1 mice. Thus, our novel method can evaluate the differences in nest-building behavior that cannot be detected or quantified by conventional methods. In future studies, we will evaluate nest-building behaviors of genetically modified, as well as several inbred, strains of mice, with several nesting materials.

Ethological concepts enhance the translational value of animal models

The translational value of animal models is an issue of ongoing discussion. We argue that 'Refinement' of animal experiments is needed and this can be achieved by exploiting an ethological approach when setting up and conducting experiments. Ethology aims to assess the functional meaning of behavioral changes, due to experimental manipulation or treatment, in animal models. Although the use of ethological concepts is particularly important for studies involving the measurement of animal behavior (as is the case for most studies on neuro-psychiatric conditions), it will also substantially benefit other disciplines, such as those investigating the immune system or inflammatory response. Using an ethological approach also involves using more optimal testing conditions are employed that have a biological relevance to the animal. Moreover, using a more biological relevant analysis of the data will help to clarify the functional meaning of the modeled readout (e.g. whether it is psychopathological or adaptive in nature). We advocate for instance that more behavioral studies should use animals in group-housed conditions, including the recording of their ultrasonic vocalizations, because (1) social behavior is an essential feature of animal models for human 'social' psychopathologies, such as autism and schizophrenia, and (2) social conditions are indispensable conditions for appropriate behavioral studies in social species, such as the rat. Only when taking these elements into account, the validity of animal experiments and, thus, the translation value of animal models can be enhanced.

3D video analysis of the novel object recognition test in rats

The novel object recognition (NOR) test has been widely used to test memory function. We developed a 3D computerized video analysis system that estimates nose contact with an object in Long Evans rats to analyze object exploration during NOR tests. The results indicate that the 3D system reproducibly and accurately scores the NOR test. Furthermore, the 3D system captures a 3D trajectory of the nose during object exploration, enabling detailed analyses of spatiotemporal patterns of object exploration. The 3D trajectory analysis revealed a specific pattern of object exploration in the sample phase of the NOR test: normal rats first explored the lower parts of objects and then gradually explored the upper parts. A systematic injection of MK-801 suppressed changes in these exploration patterns. The results, along with those of previous studies, suggest that the changes in the exploration patterns reflect neophobia to a novel object and/or changes from spatial learning to object learning. These results demonstrate that the 3D tracking system is useful not only for detailed scoring of animal behaviors but also for investigation of characteristic spatiotemporal patterns of object exploration. The system has the potential to facilitate future investigation of neural mechanisms underlying object exploration that result from dynamic and complex brain activity.

Automated image-based tracking and its application in ecology

The behavior of individuals determines the strength and outcome of ecological interactions, which drive population, community, and ecosystem organization. Bio-logging, such as telemetry and animal-borne imaging, provides essential individual viewpoints, tracks, and life histories, but requires capture of individuals and is often impractical to scale. Recent developments in automated image-based tracking offers opportunities to remotely quantify and understand individual behavior at scales and resolutions not previously possible, providing an essential supplement to other tracking methodologies in ecology. Automated image-based tracking should continue to advance the field of ecology by enabling better understanding of the linkages between individual and higher-level ecological processes, via high-throughput quantitative analysis of complex ecological patterns and processes across scales, including analysis of environmental drivers.

A real-time 3D video tracking system for monitoring primate groups

To date, assessing the solitary and social behaviors of laboratory primates' colonies relies on time-consuming manual scoring methods. Here, we describe a real-time multi-camera 3D tracking system developed to measure the behavior of socially-housed primates. Their positions are identified using non-invasive color markers such as plastic collars, thus allowing to also track colored objects and to measure their usage. Compared to traditional manual ethological scoring, we show that this system can reliably evaluate solitary behaviors (foraging, solitary resting, toy usage, locomotion) as well as spatial proximity with peers, which is considered as a good proxy of their social motivation. Compared to existing video-based commercial systems currently available to measure animal activity, this system offers many possibilities (real-time data, large volume coverage, multiple animal tracking) at a lower hardware cost. Quantitative behavioral data of animal groups can now be obtained automatically over very long periods of time, thus opening new perspectives in particular for studying the neuroethology of social behavior in primates.