A Markerless 3D Computerized Motion Capture System Incorporating a Skeleton Model for Monkeys

Versatile multiple object tracking in sparse 2D/3D videos via deformable image registration

Tracking body parts in behaving animals, extracting fluorescence signals from cells embedded in deforming tissue, and analyzing cell migration patterns during development all require tracking objects with partially correlated motion. As dataset sizes increase, manual tracking of objects becomes prohibitively inefficient and slow, necessitating automated and semi-automated computational tools. Unfortunately, existing methods for multiple object tracking (MOT) are either developed for specific datasets and hence do not generalize well to other datasets, or require large amounts of training data that are not readily available. This is further exacerbated when tracking fluorescent sources in moving and deforming tissues, where the lack of unique features and sparsely populated images create a challenging environment, especially for modern deep learning techniques. By leveraging technology recently developed for spatial transformer networks, we propose ZephIR, an image registration framework for semi-supervised MOT in 2D and 3D videos. ZephIR can generalize to a wide range of biological systems by incorporating adjustable parameters that encode spatial (sparsity, texture, rigidity) and temporal priors of a given data class. We demonstrate the accuracy and versatility of our approach in a variety of applications, including tracking the body parts of a behaving mouse and neurons in the brain of a freely moving C. elegans. We provide an open-source package along with a web-based graphical user interface that allows users to provide small numbers of annotations to interactively improve tracking results.

A Layered, Hybrid Machine Learning Analytic Workflow for Mouse Risk Assessment Behavior

Accurate and efficient quantification of animal behavior facilitates the understanding of the brain. An emerging approach within machine learning (ML) field is to combine multiple ML-based algorithms to quantify animal behavior. These so-called hybrid models have emerged because of limitations associated with supervised [e.g., random forest (RF)] and unsupervised [e.g., hidden Markov model (HMM)] ML models. For example, RF models lack temporal information across video frames, and HMM latent states are often difficult to interpret. We sought to develop a hybrid model, and did so in the context of a study of mouse risk assessment behavior. We used DeepLabCut to estimate the positions of mouse body parts. Positional features were calculated using DeepLabCut outputs and were used to train RF and HMM models with equal number of states, separately. The per-frame predictions from RF and HMM models were then passed to a second HMM model layer ("reHMM"). The outputs of the reHMM layer showed improved interpretability over the initial HMM output. Finally, we combined predictions from RF and HMM models with selected positional features to train a third HMM model ("reHMM+"). This reHMM+ layered hybrid model unveiled distinctive temporal and human-interpretable behavioral patterns. We applied this workflow to investigate risk assessment to trimethylthiazoline and snake feces odor, finding unique behavioral patterns to each that were separable from attractive and neutral stimuli. We conclude that this layered, hybrid ML workflow represents a balanced approach for improving the depth and reliability of ML classifiers in chemosensory and other behavioral contexts.

Non-human primate models and systems for gait and neurophysiological analysis

Brain-computer interfaces (BCIs) have garnered extensive interest and become a groundbreaking technology to restore movement, tactile sense, and communication in patients. Prior to their use in human subjects, clinical BCIs require rigorous validation and verification (V&V). Non-human primates (NHPs) are often considered the ultimate and widely used animal model for neuroscience studies, including BCIs V&V, due to their proximity to humans. This literature review summarizes 94 NHP gait analysis studies until 1 June, 2022, including seven BCI-oriented studies. Due to technological limitations, most of these studies used wired neural recordings to access electrophysiological data. However, wireless neural recording systems for NHPs enabled neuroscience research in humans, and many on NHP locomotion, while posing numerous technical challenges, such as signal quality, data throughout, working distance, size, and power constraint, that have yet to be overcome. Besides neurological data, motion capture (MoCap) systems are usually required in BCI and gait studies to capture locomotion kinematics. However, current studies have exclusively relied on image processing-based MoCap systems, which have insufficient accuracy (error: ≥4° and 9 mm). While the role of the motor cortex during locomotion is still unclear and worth further exploration, future BCI and gait studies require simultaneous, high-speed, accurate neurophysiological, and movement measures. Therefore, the infrared MoCap system which has high accuracy and speed, together with a high spatiotemporal resolution neural recording system, may expand the scope and improve the quality of the motor and neurophysiological analysis in NHPs.

Leaving flatland: Advances in 3D behavioral measurement

Animals move in three dimensions (3D). Thus, 3D measurement is necessary to report the true kinematics of animal movement. Existing 3D measurement techniques draw on specialized hardware, such as motion capture or depth cameras, as well as deep multi-view and monocular computer vision. Continued advances at the intersection of deep learning and computer vision will facilitate 3D tracking across more anatomical features, with less training data, in additional species, and within more natural, occlusive environments. 3D behavioral measurement enables unique applications in phenotyping, investigating the neural basis of behavior, and designing artificial agents capable of imitating animal behavior.

Automated Video-Based Analysis Framework for Behavior Monitoring of Individual Animals in Zoos Using Deep Learning—A Study on Polar Bears

Simple Summary

Every institution that keeps animals under human care must ensure animal welfare. To analyze the state of an animal, various measurements can be performed, such as blood analysis or fur condition scoring. They also need to be observed as often as possible to gain further insight into their behavior. Such observations are performed manually in most cases, which makes them very labor- and time-intensive and prevent them from being performed on a continual basis. We present a camera-based framework that provides automated observation of animals. The system detects individual animals and analyzes their locations, walking paths, and activity. We test the framework on the two polar bears of the Nuremberg Zoo.

Abstract

The monitoring of animals under human care is a crucial tool for biologists and zookeepers to keep track of the animals’ physical and psychological health. Additionally, it enables the analysis of observed behavioral changes and helps to unravel underlying reasons. Enhancing our understanding of animals ensures and improves ex situ animal welfare as well as in situ conservation. However, traditional observation methods are time- and labor-intensive, as they require experts to observe the animals on-site during long and repeated sessions and manually score their behavior. Therefore, the development of automated observation systems would greatly benefit researchers and practitioners in this domain. We propose an automated framework for basic behavior monitoring of individual animals under human care. Raw video data are processed to continuously determine the position of the individuals within the enclosure. The trajectories describing their travel patterns are presented, along with fundamental analysis, through a graphical user interface (GUI). We evaluate the performance of the framework on captive polar bears (Ursus maritimus). We show that the framework can localize and identify individual polar bears with an F1 score of 86.4%. The localization accuracy of the framework is 19.9±7.6 cm, outperforming current manual observation methods. Furthermore, we provide a bounding-box-labeled dataset of the two polar bears housed in Nuremberg Zoo.

Automatic mapping of multiplexed social receptive fields by deep learning and GPU-accelerated 3D videography

Social interactions powerfully impact the brain and the body, but high-resolution descriptions of these important physical interactions and their neural correlates are lacking. Currently, most studies rely on labor-intensive methods such as manual annotation. Scalable and objective tracking methods are required to understand the neural circuits underlying social behavior. Here we describe a hardware/software system and analysis pipeline that combines 3D videography, deep learning, physical modeling, and GPU-accelerated robust optimization, with automatic analysis of neuronal receptive fields recorded in interacting mice. Our system ("3DDD Social Mouse Tracker") is capable of fully automatic multi-animal tracking with minimal errors (including in complete darkness) during complex, spontaneous social encounters, together with simultaneous electrophysiological recordings. We capture posture dynamics of multiple unmarked mice with high spatiotemporal precision (~2 mm, 60 frames/s). A statistical model that relates 3D behavior and neural activity reveals multiplexed 'social receptive fields' of neurons in barrel cortex. Our approach could be broadly useful for neurobehavioral studies of multiple animals interacting in complex low-light environments.

A Novel, Automated, and Real-Time Method for the Analysis of Non-Human Primate Behavioral Patterns Using a Depth Image Sensor

By virtue of their upright locomotion, similar to that of humans, motion analysis of non-human primates has been widely used in order to better understand musculoskeletal biomechanics and neuroscience problems. Given the difficulty of conducting a marker-based infrared optical tracking system for the behavior analysis of primates, a 2-dimensional (D) video analysis has been applied. Distinct from a conventional marker-based optical tracking system, a depth image sensor system provides 3-D information on movement without any skin markers. The specific aim of this study was to develop a novel algorithm to analyze the behavioral patterns of non-human primates in a home cage using a depth image sensor. The behavioral patterns of nine monkeys in their home cage, including sitting, standing, and pacing, were captured using a depth image sensor. Thereafter, these were analyzed by observers’ manual assessment and the newly written automated program. We confirmed that the measurement results from the observers’ manual assessments and the automated program with depth image analysis were statistically identical.

Markerless Dog Pose Recognition in the Wild Using ResNet Deep Learning Model

The analysis and perception of behavior has usually been a crucial task for researchers. The goal of this paper is to address the problem of recognition of animal poses, which has numerous applications in zoology, ecology, biology, and entertainment. We propose a methodology to recognize dog poses. The methodology includes the extraction of frames for labeling from videos and deep convolutional neural network (CNN) training for pose recognition. We employ a semi-supervised deep learning model of reinforcement. During training, we used a combination of restricted labeled data and a large amount of unlabeled data. Sequential CNN is also used for feature localization and to find the canine’s motions and posture for spatio-temporal analysis. To detect the canine’s features, we employ image frames to locate the annotations and estimate the dog posture. As a result of this process, we avoid starting from scratch with the feature model and reduce the need for a large dataset. We present the results of experiments on a dataset of more than 5000 images of dogs in different poses. We demonstrated the effectiveness of the proposed methodology for images of canine animals in various poses and behavior. The methodology implemented as a mobile app that can be used for animal tracking.

MouseVenue3D: A Markerless Three-Dimension Behavioral Tracking System for Matching Two-Photon Brain Imaging in Free-Moving Mice

Understanding the connection between brain and behavior in animals requires precise monitoring of their behaviors in three-dimensional (3-D) space. However, there is no available three-dimensional behavior capture system that focuses on rodents. Here, we present MouseVenue3D, an automated and low-cost system for the efficient capture of 3-D skeleton trajectories in markerless rodents. We improved the most time-consuming step in 3-D behavior capturing by developing an automatic calibration module. Then, we validated this process in behavior recognition tasks, and showed that 3-D behavioral data achieved higher accuracy than 2-D data. Subsequently, MouseVenue3D was combined with fast high-resolution miniature two-photon microscopy for synchronous neural recording and behavioral tracking in the freely-moving mouse. Finally, we successfully decoded spontaneous neuronal activity from the 3-D behavior of mice. Our findings reveal that subtle, spontaneous behavior modules are strongly correlated with spontaneous neuronal activity patterns.

Chemogenetic activation of nigrostriatal dopamine neurons in freely moving common marmosets

To interrogate particular neuronal pathways in nonhuman primates under natural and stress-free conditions, we applied designer receptors exclusively activated by designer drugs (DREADDs) technology to common marmosets. We injected adeno-associated virus vectors expressing the excitatory DREADD hM3Dq into the unilateral substantia nigra (SN) in four marmosets. Using multi-tracer positron emission tomography imaging, we detected DREADD expression in vivo, which was confirmed in nigrostriatal dopamine neurons by immunohistochemistry, as well as by assessed activation of the SN following agonist administration. The marmosets rotated in a contralateral direction relative to the activated side 30-90 min after consuming food containing the highly potent DREADD agonist deschloroclozapine (DCZ) but not on the following days without DCZ. These results indicate that non-invasive and reversible DREADD manipulation will extend the utility of marmosets as a primate model for linking neuronal activity and natural behavior in various contexts.

A markerless platform for ambulatory systems neuroscience

Motor systems neuroscience seeks to understand how the brain controls movement. To minimize confounding variables, large-animal studies typically constrain body movement from areas not under observation, ensuring consistent, repeatable behaviors. Such studies have fueled decades of research, but they may be artificially limiting the richness of neural data observed, preventing generalization to more natural movements and settings. Neuroscience studies of unconstrained movement would capture a greater range of behavior and a more complete view of neuronal activity, but instrumenting an experimental rig suitable for large animals presents substantial engineering challenges. Here, we present a markerless, full-body motion tracking and synchronized wireless neural electrophysiology platform for large, ambulatory animals. Composed of four depth (RGB-D) cameras that provide a 360° view of a 4.5-square-meters enclosed area, this system is designed to record a diverse range of neuroethologically relevant behaviors. This platform also allows for the simultaneous acquisition of hundreds of wireless neural recording channels in multiple brain regions. As behavioral and neuronal data are generated at rates below 200 megabytes per second, a single desktop can facilitate hours of continuous recording. This setup is designed for systems neuroscience and neuroengineering research, where synchronized kinematic behavior and neural data are the foundation for investigation. By enabling the study of previously unexplored movement tasks, this system can generate insights into the functioning of the mammalian motor system and provide a platform to develop brain-machine interfaces for unconstrained applications.

MacaquePose: A Novel "In the Wild" Macaque Monkey Pose Dataset for Markerless Motion Capture

Video-based markerless motion capture permits quantification of an animal's pose and motion, with a high spatiotemporal resolution in a naturalistic context, and is a powerful tool for analyzing the relationship between the animal's behaviors and its brain functions. Macaque monkeys are excellent non-human primate models, especially for studying neuroscience. Due to the lack of a dataset allowing training of a deep neural network for the macaque's markerless motion capture in the naturalistic context, it has been challenging to apply this technology for macaques-based studies. In this study, we created MacaquePose, a novel open dataset with manually labeled body part positions (keypoints) for macaques in naturalistic scenes, consisting of >13,000 images. We also validated the application of the dataset by training and evaluating an artificial neural network with the dataset. The results indicated that the keypoint estimation performance of the trained network was close to that of a human-level. The dataset will be instrumental to train/test the neural networks for markerless motion capture of the macaques and developments of the algorithms for the networks, contributing establishment of an innovative platform for behavior analysis for non-human primates for neuroscience and medicine, as well as other fields using macaques as a model organism.

Computer Vision Applied to Detect Lethargy through Animal Motion Monitoring: A Trial on African Swine Fever in Wild Boar

Early detection of infectious diseases is the most cost-effective strategy in disease surveillance for reducing the risk of outbreaks. Latest deep learning and computer vision improvements are powerful tools that potentially open up a new field of research in epidemiology and disease control. These techniques were used here to develop an algorithm aimed to track and compute animal motion in real time. This algorithm was used in experimental trials in order to assess African swine fever (ASF) infection course in Eurasian wild boar. Overall, the outcomes showed negative correlation between motion reduction and fever caused by ASF infection. In addition, infected animals computed significant lower movements compared to uninfected animals. The obtained results suggest that a motion monitoring system based on artificial vision may be used in indoors to trigger suspicions of fever. It would help farmers and animal health services to detect early clinical signs compatible with infectious diseases. This technology shows a promising non-intrusive, economic and real time solution in the livestock industry with especial interest in ASF, considering the current concern in the world pig industry.

Assessment of nest building and social interaction behavior in mice exposed to acute social defeat stress using a three-dimensional depth camera

Nest building is an instinctive behavior toward protection from predators, body temperature regulation, and courtship. Previously, we discovered that acute and chronic social defeat stress suppresses the onset of nest-building behavior in male mice (C57BL/6J). Here, we analyzed nest building and other behavioral deficits induced by acute social defeat stress (ASDS). We utilized a customized cage and specifically developed observational programs for nest building, social avoidance, and other behaviors using an infrared depth camera to acquire three-dimensional (3D) data of animal behavior (Negura system). We determined the volume of nesting materials from these 3D depth images. Mice exposed to ASDS showed increased spontaneous activities, decreased rearing, and delayed nest building; however, nest-building activity was gradually recovered during the dark period of the 24 hr observation interval. At the endpoint following 24 hr, the ASDS and control groups showed no differences in nest volumes. Furthermore, we observed the time courses of both nest building and social avoidance behaviors and their relationship using the Negura system. Our data demonstrated a weak positive correlation between nest-building delay and social avoidance in ASDS mice. The Negura system can observe various behaviors that reflect the effects of social defeat stress.

Non-invasive electroencephalographical (EEG) recording system in awake monkeys

Background

Human clinical studies reported that several electroencephalographical (EEG) parameters can be used as biomarkers of psychiatric disorders. EEGs recorded from non-human primates (monkeys) is useful for understanding of human pathologies of psychiatric disorders and development of new therapeutic agents.

New methods

In this study, we expand a previous non-invasive head holding system with face masks for awake monkeys to be applied to scalp EEG recording. The new design of a head holding system allows to attach scalp EEG electrodes on the positions comparable to human electrode placement and to present auditory stimuli.

Results

With this system, we could record auditory evoked potentials (AEPs) in auditory sensory gating and oddball paradigms, which are often used as biomarkers of psychiatric disorders in animal models and human patients. The recorded AEPs were comparable to previous human clinical data.

Comparison with existing methods

Compared with previous non-invasive head holding systems, top, side (cheek and ears), and rear of the head can be open for attachment of EEG electrodes and auditory stimulation in the present system.

Conclusions

The results suggest that the present system is useful in EEG recording from awake monkeys. Furthermore, this system can be applied to eye-tracking and chronic intra-cerebral recording experiments.

Wireless recording from unrestrained monkeys reveals motor goal encoding beyond immediate reach in frontoparietal cortex

System neuroscience of motor cognition regarding the space beyond immediate reach mandates free, yet experimentally controlled movements. We present an experimental environment (Reach Cage) and a versatile visuo-haptic interaction system (MaCaQuE) for investigating goal-directed whole-body movements of unrestrained monkeys. Two rhesus monkeys conducted instructed walk-and-reach movements towards targets flexibly positioned in the cage. We tracked 3D multi-joint arm and head movements using markerless motion capture. Movements show small trial-to-trial variability despite being unrestrained. We wirelessly recorded 192 broad-band neural signals from three cortical sensorimotor areas simultaneously. Single unit activity is selective for different reach and walk-and-reach movements. Walk-and-reach targets could be decoded from premotor and parietal but not motor cortical activity during movement planning. The Reach Cage allows systems-level sensorimotor neuroscience studies with full-body movements in a configurable 3D spatial setting with unrestrained monkeys. We conclude that the primate frontoparietal network encodes reach goals beyond immediate reach during movement planning.

Automated Measures of Force and Motion Can Improve Our Understanding of Infants’ Motor Persistence

Every day, young learners are confronted with challenges. The degree to which they persist in overcoming those challenges, and the different ways they persist, provides critical insights into the various cognitive, motoric, and affective processes that drive behavior. Here, we present a systematic overview of the methodologies that have been traditionally used to study persistence, and offer suggestions for new approaches to the study of persistence that will make strides in moving the field forward. We argue that automated measures of force and motion, which have long been used in the study of infants’ motoric behavior, can provide a means to unravel the psychological processes that guide infants’ trying behavior. To illustrate this, we present a case study that highlights the novel lessons to be learned by the use of automated measures of force and motion regarding infants’ persistence, along with an analysis of the benefits and drawbacks of this approach, as well as detailed instructions for application. In sum, we conclude that these measures, when used in conjunction with more traditional approaches, will provide creative new insights into the nature and development of early persistence.

Real-Time Analysis of Animal Feeding Behavior With a Low-Calculation-Power CPU

Our goal was to develop an automated system to determine whether animals have learned and changed their behavior in real-time using a low calculation-power central processing unit (CPU). The bottleneck of real-time analysis is the speed of image recognition. For fast image recognition, 99.5% of the image was excluded from image recognition by distinguishing between the subject and the background. We achieved this by applying a binarization and connected-component labeling technique. This task is important for developing a fully automated learning apparatus. The use of such an automated system can improve the efficiency and accuracy of biological studies. The pond snail Lymnaea stagnails can be classically conditioned to avoid food that naturally elicits feeding behavior, and to consolidate this aversion into long-term memory. Determining memory status in the snail requires real-time analysis of the number of bites the snail makes in response to food presentation. The main algorithm for counting bites comprises two parts: extracting the mouth images from the recorded video and measuring the bite rate corresponding to the memory status. Reinforcement-supervised learning and image recognition were used to extract the mouth images. A change in the size of the mouth area was used as the cue for counting the number of bites. The accuracy of the final judgment of whether or not the snail had learned was the same as that determined by human observation. This method to improve the processing speed of image recognition has the potential for broad application beyond biological fields.

Circuit Investigations With Open-Source Miniaturized Microscopes: Past, Present and Future

The ability to simultaneously image the spatiotemporal activity signatures from many neurons during unrestrained vertebrate behaviors has become possible through the development of miniaturized fluorescence microscopes, or miniscopes, sufficiently light to be carried by small animals such as bats, birds and rodents. Miniscopes have permitted the study of circuits underlying song vocalization, action sequencing, head-direction tuning, spatial memory encoding and sleep to name a few. The foundation for these microscopes has been laid over the last two decades through academic research with some of this work resulting in commercialization. More recently, open-source initiatives have led to an even broader adoption of miniscopes in the neuroscience community. Open-source designs allow for rapid modification and extension of their function, which has resulted in a new generation of miniscopes that now permit wire-free or wireless recording, concurrent electrophysiology and imaging, two-color fluorescence detection, simultaneous optical actuation and read-out as well as wide-field and volumetric light-field imaging. These novel miniscopes will further expand the toolset of those seeking affordable methods to probe neural circuit function during naturalistic behaviors. Here, we will discuss the early development, present use and future potential of miniscopes.

DeepLabCut: markerless pose estimation of user-defined body parts with deep learning

Quantifying behavior is crucial for many applications in neuroscience. Videography provides easy methods for the observation and recording of animal behavior in diverse settings, yet extracting particular aspects of a behavior for further analysis can be highly time consuming. In motor control studies, humans or other animals are often marked with reflective markers to assist with computer-based tracking, but markers are intrusive, and the number and location of the markers must be determined a priori. Here we present an efficient method for markerless pose estimation based on transfer learning with deep neural networks that achieves excellent results with minimal training data. We demonstrate the versatility of this framework by tracking various body parts in multiple species across a broad collection of behaviors. Remarkably, even when only a small number of frames are labeled (~200), the algorithm achieves excellent tracking performance on test frames that is comparable to human accuracy.