Real-Time Selective Markerless Tracking of Forepaws of Head Fixed Mice Using Deep Neural Networks

Non-consummatory behavior signals predict aversion-resistant alcohol drinking in head-fixed mice

A key facet of alcohol use disorder is continuing to drink alcohol despite negative consequences (so called "aversion-resistant drinking"). In this study, we sought to assess the degree to which head-fixed mice exhibit aversion-resistant drinking and to leverage behavioral analysis techniques available in head-fixture to relate non-consummatory behaviors to aversion-resistant drinking. We assessed aversion-resistant drinking in head-fixed female and male C57BL/6 J mice. We adulterated 20% (v/v) alcohol with varying concentrations of the bitter tastant quinine to measure the degree to which mice would continue to drink despite this aversive stimulus. We recorded high-resolution video of the mice during head-fixed drinking, tracked body parts with machine vision tools, and analyzed body movements in relation to consumption. Female and male head-fixed mice exhibited heterogenous levels of aversion-resistant drinking. Additionally, non-consummatory behaviors, such as paw movement and snout movement, were related to the intensity of aversion-resistant drinking. These studies demonstrate that head-fixed mice exhibit aversion-resistant drinking and that non-consummatory behaviors can be used to assess perceived aversiveness in this paradigm. Furthermore, these studies lay the groundwork for future experiments that will utilize advanced electrophysiological techniques to record from large populations of neurons during aversion-resistant drinking to understand the neurocomputational processes that drive this clinically relevant behavior. This article is part of the Special Issue on "PFC circuit function in psychiatric disease and relevant models".

Neurons in the primary visual cortex of freely moving rats encode both sensory and non-sensory task variables

Neurons in primary visual cortex (area V1) are strongly driven by both sensory stimuli and non-sensory events. However, although the representation of sensory stimuli has been well characterized, much less is known about the representation of non-sensory events. Here, we characterize the specificity and organization of non-sensory representations in rat V1 during a freely moving visual decision task. We find that single neurons encode diverse combinations of task features simultaneously and across task epochs. Despite heterogeneity at the level of single neuron response patterns, both visual and nonvisual task variables could be reliably decoded from small neural populations (5 to 40 units) throughout a trial. Interestingly, in animals trained to make an auditory decision following passive observation of a visual stimulus, some but not all task features could also be decoded from V1 activity. Our results support the view that even in V1-the earliest stage of the cortical hierarchy-bottom-up sensory information may be combined with top-down non-sensory information in a task-dependent manner.

Non-Consummatory Behavior Signals Predict Aversion-Resistant Alcohol Drinking in Head-Fixed Mice

A key facet of alcohol use disorder is continuing to drink alcohol despite negative consequences (so called "aversion-resistant drinking"). In this study, we sought to assess the degree to which head-fixed mice exhibit aversion-resistant drinking and to leverage behavioral analysis techniques available in head-fixture to relate non-consummatory behaviors to aversion-resistant drinking. We assessed aversion-resistant drinking in head-fixed female and male C57BL/6J mice. We adulterated 20% (v/v) alcohol with varying concentrations of the bitter tastant quinine to measure the degree to which mice would continue to drink despite this aversive stimulus. We recorded high-resolution video of the mice during head-fixed drinking, tracked body parts with machine vision tools, and analyzed body movements in relation to consumption. Female and male head-fixed mice exhibited heterogenous levels of aversion-resistant drinking. Additionally, non-consummatory behaviors, such as paw movement and snout movement, were related to the intensity of aversion-resistant drinking. These studies demonstrate that head-fixed mice exhibit aversion-resistant drinking and that non-consummatory behaviors can be used to assess perceived aversiveness in this paradigm. Furthermore, these studies lay the groundwork for future experiments that will utilize advanced electrophysiological techniques to record from large populations of neurons during aversion-resistant drinking to understand the neurocomputational processes that drive this clinically relevant behavior.

REVEALS: An Open Source Multi Camera GUI For Rodent Behavior Acquisition

Understanding the rich behavioral data generated by mice is essential for deciphering the function of the healthy and diseased brain. However, the current landscape lacks effective, affordable, and accessible methods for acquiring such data, especially when employing multiple cameras simultaneously. We have developed REVEALS (Rodent BEhaVior Multi-camErA Laboratory AcquiSition), a graphical user interface (GUI) written in python for acquiring rodent behavioral data via commonly used USB3 cameras. REVEALS allows for user-friendly control of recording from one or multiple cameras simultaneously while streamlining the data acquisition process, enabling researchers to collect and analyze large datasets efficiently. We release this software package as a stand-alone, open-source framework for researchers to use and modify according to their needs. We describe the details of the GUI implementation, including the camera control software and the video recording functionality. We validate results demonstrating the GUI's stability, reliability, and accuracy for capturing and analyzing rodent behavior using DeepLabCut pose estimation in both an object and social interaction assay. REVEALS can also be incorporated into other custom pipelines to analyze complex behavior, such as MoSeq. In summary, REVEALS provides an interface for collecting behavioral data from one or multiple perspectives that, combined with deep learning algorithms, will allow the scientific community to discover and characterize complex behavioral phenotypes to understand brain function better.

Multiscale imaging informs translational mouse modeling of neurological disease

Multiscale neurophysiology reveals that simple motor actions are associated with changes in neuronal firing in virtually every brain region studied. Accordingly, the assessment of focal pathology such as stroke or progressive neurodegenerative diseases must also extend widely across brain areas. To derive mechanistic information through imaging, multiple resolution scales and multimodal factors must be included, such as the structure and function of specific neurons and glial cells and the dynamics of specific neurotransmitters. Emerging multiscale methods in preclinical animal studies that span micro- to macroscale examinations fill this gap, allowing a circuit-based understanding of pathophysiological mechanisms. Combined with high-performance computation and open-source data repositories, these emerging multiscale and large field-of-view techniques include live functional ultrasound, multi- and single-photon wide-scale light microscopy, video-based miniscopes, and tissue-penetrating fiber photometry, as well as variants of post-mortem expansion microscopy. We present these technologies and outline use cases and data pipelines to uncover new knowledge within animal models of stroke, Alzheimer's disease, and movement disorders.

Automatic extraction of upper-limb kinematic activity using deep learning-based markerless tracking during deep brain stimulation implantation for Parkinson's disease: A proof of concept study

Optimal placement of deep brain stimulation (DBS) therapy for treating movement disorders routinely relies on intraoperative motor testing for target determination. However, in current practice, motor testing relies on subjective interpretation and correlation of motor and neural information. Recent advances in computer vision could improve assessment accuracy. We describe our application of deep learning-based computer vision to conduct markerless tracking for measuring motor behaviors of patients undergoing DBS surgery for the treatment of Parkinson's disease. Video recordings were acquired during intraoperative kinematic testing (N = 5 patients), as part of standard of care for accurate implantation of the DBS electrode. Kinematic data were extracted from videos post-hoc using the Python-based computer vision suite DeepLabCut. Both manual and automated (80.00% accuracy) approaches were used to extract kinematic episodes from threshold derived kinematic fluctuations. Active motor epochs were compressed by modeling upper limb deflections with a parabolic fit. A semi-supervised classification model, support vector machine (SVM), trained on the parameters defined by the parabolic fit reliably predicted movement type. Across all cases, tracking was well calibrated (i.e., reprojection pixel errors 0.016-0.041; accuracies >95%). SVM predicted classification demonstrated high accuracy (85.70%) including for two common upper limb movements, arm chain pulls (92.30%) and hand clenches (76.20%), with accuracy validated using a leave-one-out process for each patient. These results demonstrate successful capture and categorization of motor behaviors critical for assessing the optimal brain target for DBS surgery. Conventional motor testing procedures have proven informative and contributory to targeting but have largely remained subjective and inaccessible to non-Western and rural DBS centers with limited resources. This approach could automate the process and improve accuracy for neuro-motor mapping, to improve surgical targeting, optimize DBS therapy, provide accessible avenues for neuro-motor mapping and DBS implantation, and advance our understanding of the function of different brain areas.

Using DeepLabCut as a Real-Time and Markerless Tool for Cardiac Physiology Assessment in Zebrafish

Simple Summary

With the advancement of existing technology, artificial intelligence is widely applied in various fields of research, including cardiovascular studies. In this study, we explored the feasibility of conducting a markerless cardiac physiology assessment in zebrafish embryos by using DeepLabCut (DLC), a deep learning tool for motion analysis. Several cardiac parameters, such as heart rate, diastolic–systolic volumes (EDV/ESV), stroke volume, cardiac output, shortening fraction, and ejection fraction were obtained by the DLC-trained model and then compared to the previous published methods, Time Series Analysis and Kymograph. This new method has several advantages, having full automation, precise detection, and real-time labelling. This network was also trained to analyze zebrafish with cardiovascular defects (pericardial edema) induced by chemical treatments with ethanol and ponatinib. It was revealed that the heart rate, EDV/ESV, stroke volume, and cardiac output from both the ethanol and ponatinib groups displayed significant reductions compared with the control. Hopefully, this trained DLC network can contribute to a better understanding and investigation of the existing cardiovascular system and abnormalities.

Abstract

DeepLabCut (DLC) is a deep learning-based tool initially invented for markerless pose estimation in mammals. In this study, we explored the possibility of adopting this tool for conducting markerless cardiac physiology assessment in an important aquatic toxicology model of zebrafish (Danio rerio). Initially, high-definition videography was applied to capture heartbeat information at a frame rate of 30 frames per second (fps). Next, 20 videos from different individuals were used to perform convolutional neural network training by labeling the heart chamber (ventricle) with eight landmarks. Using Residual Network (ResNet) 152, a neural network with 152 convolutional neural network layers with 500,000 iterations, we successfully obtained a trained model that can track the heart chamber in a real-time manner. Later, we validated DLC performance with the previously published ImageJ Time Series Analysis (TSA) and Kymograph (KYM) methods. We also evaluated DLC performance by challenging experimental animals with ethanol and ponatinib to induce cardiac abnormality and heartbeat irregularity. The results showed that DLC is more accurate than the TSA method in several parameters tested. The DLC-trained model also detected the ventricle of zebrafish embryos even in the occurrence of heart abnormalities, such as pericardial edema. We believe that this tool is beneficial for research studies, especially for cardiac physiology assessment in zebrafish embryos.

Deep-learning based identification, tracking, pose estimation, and behavior classification of interacting primates and mice in complex environments

The quantification of behaviors of interest from video data is commonly used to study brain function, the effects of pharmacological interventions, and genetic alterations. Existing approaches lack the capability to analyze the behavior of groups of animals in complex environments. We present a novel deep learning architecture for classifying individual and social animal behavior, even in complex environments directly from raw video frames, while requiring no intervention after initial human supervision. Our behavioral classifier is embedded in a pipeline (SIPEC) that performs segmentation, identification, pose-estimation, and classification of complex behavior, outperforming the state of the art. SIPEC successfully recognizes multiple behaviors of freely moving individual mice as well as socially interacting non-human primates in 3D, using data only from simple mono-vision cameras in home-cage setups.

Wide-angle, monocular head tracking using passive markers

BACKGROUND

Camera images can encode large amounts of visual information of an animal and its environment, enabling high fidelity 3D reconstruction of the animal and its environment using computer vision methods. Most systems, both markerless (e.g. deep learning based) and marker-based, require multiple cameras to track features across multiple points of view to enable such 3D reconstruction. However, such systems can be expensive and are challenging to set up in small animal research apparatuses.

NEW METHODS

We present an open-source, marker-based system for tracking the head of a rodent for behavioral research that requires only a single camera with a potentially wide field of view. The system features a lightweight visual target and computer vision algorithms that together enable high-accuracy tracking of the six-degree-of-freedom position and orientation of the animal's head. The system, which only requires a single camera positioned above the behavioral arena, robustly reconstructs the pose over a wide range of head angles (360° in yaw, and approximately ± 120° in roll and pitch).

RESULTS

Experiments with live animals demonstrate that the system can reliably identify rat head position and orientation. Evaluations using a commercial optical tracker device show that the system achieves accuracy that rivals commercial multi-camera systems.

COMPARISON WITH EXISTING METHODS

Our solution significantly improves upon existing monocular marker-based tracking methods, both in accuracy and in allowable range of motion.

CONCLUSIONS

The proposed system enables the study of complex behaviors by providing robust, fine-scale measurements of rodent head motions in a wide range of orientations.

A New Look at Infant Problem-Solving: Using DeepLabCut to Investigate Exploratory Problem-Solving Approaches

When confronted with novel problems, problem-solvers must decide whether to copy a modeled solution or to explore their own unique solutions. While past work has established that infants can learn to solve problems both through their own exploration and through imitation, little work has explored the factors that influence which of these approaches infants select to solve a given problem. Moreover, past work has treated imitation and exploration as qualitatively distinct, although these two possibilities may exist along a continuum. Here, we apply a program novel to developmental psychology (DeepLabCut) to archival data (Lucca et al., 2020) to investigate the influence of the effort and success of an adult's modeled solution, and infants' firsthand experience with failure, on infants' imitative versus exploratory problem-solving approaches. Our results reveal that tendencies toward exploration are relatively immune to the information from the adult model, but that exploration generally increased in response to firsthand experience with failure. In addition, we found that increases in maximum force and decreases in trying time were associated with greater exploration, and that exploration subsequently predicted problem-solving success on a new iteration of the task. Thus, our results demonstrate that infants increase exploration in response to failure and that exploration may operate in a larger motivational framework with force, trying time, and expectations of task success.

Quantifying the kinematic features of dexterous finger movements in nonhuman primates with markerless tracking

Research using nonhuman primate models for human disease frequently requires behavioral observational techniques to quantify functional outcomes. The ability to assess reaching and grasping patterns is of particular interest in clinical conditions that affect the motor system (e.g., spinal cord injury, SCI). Here we explored the use of DeepLabCut, an open-source deep learning toolset, in combination with a standard behavioral task (Brinkman Board) to quantify nonhuman primate performance in precision grasping. We examined one male rhesus macaque (Macaca mulatta) in the task which involved retrieving rewards from variously-oriented shallow wells. Simultaneous recordings were made using GoPro Hero7 Black cameras (resolution 1920 x 1080 at 120 fps) from two different angles (from the side and top of the hand motion). The task/device design necessitates use of the right hand to complete the task. Two neural networks (corresponding to the top and side view cameras) were trained using 400 manually annotated images, tracking 19 unique landmarks each. Based on previous reports, this produced sufficient tracking (Side: trained pixel error of 2.15, test pixel error of 11.25; Top: trained pixel error of 2.06, test pixel error of 30.31) so that landmarks could be tracked on the remaining frames. Landmarks included in the tracking were the spatial location of the knuckles and the fingernails of each digit, and three different behavioral measures were quantified for assessment of hand movement (finger separation, middle digit extension and preshaping distance). Together, our preliminary results suggest that this markerless approach is a possible method to examine specific kinematic features of dexterous function.Clinical Relevance- The methodology presented below allows for the markerless tracking of kinematic features of dexterous finger movement by non-human primates. This method could allow for direct comparisons between human patients and non-human primate models of clinical conditions (e.g., spinal cord injury). This would provide objective quantitative metrics and crucial information for assessing movement impairments across populations and the potential translation of treatments, interventions and their outcomes.

Natural and Artificial Intelligence: A brief introduction to the interplay between AI and neuroscience research

Neuroscience and artificial intelligence (AI) share a long history of collaboration. Advances in neuroscience, alongside huge leaps in computer processing power over the last few decades, have given rise to a new generation of in silico neural networks inspired by the architecture of the brain. These AI systems are now capable of many of the advanced perceptual and cognitive abilities of biological systems, including object recognition and decision making. Moreover, AI is now increasingly being employed as a tool for neuroscience research and is transforming our understanding of brain functions. In particular, deep learning has been used to model how convolutional layers and recurrent connections in the brain's cerebral cortex control important functions, including visual processing, memory, and motor control. Excitingly, the use of neuroscience-inspired AI also holds great promise for understanding how changes in brain networks result in psychopathologies, and could even be utilized in treatment regimes. Here we discuss recent advancements in four areas in which the relationship between neuroscience and AI has led to major advancements in the field; (1) AI models of working memory, (2) AI visual processing, (3) AI analysis of big neuroscience datasets, and (4) computational psychiatry.

Closed-loop automated reaching apparatus (CLARA) for interrogating complex motor behaviors

Objective.Closed-loop neuromodulation technology is a rapidly expanding category of therapeutics for a broad range of indications. Development of these innovative neurological devices requires high-throughput systems for closed-loop stimulation of model organisms, while monitoring physiological signals and complex, naturalistic behaviors. To address this need, we developed CLARA, a closed-loop automated reaching apparatus.Approach.Using breakthroughs in computer vision, CLARA integrates fully-automated, markerless kinematic tracking of multiple features to classify animal behavior and precisely deliver neural stimulation based on behavioral outcomes. CLARA is compatible with advanced neurophysiological tools, enabling the testing of neurostimulation devices and identification of novel neurological biomarkers.Results.The CLARA system tracks unconstrained skilled reach behavior in 3D at 150 Hz without physical markers. The system fully automates trial initiation and pellet delivery and is capable of accurately delivering stimulation in response to trial outcome with short latency. Kinematic data from the CLARA system provided novel insights into the dynamics of reach consistency over the course of learning, suggesting that learning selectively improves reach failures but does not alter the kinematics of successful reaches. Additionally, using the closed-loop capabilities of CLARA, we demonstrate that vagus nerve stimulation (VNS) improves skilled reach performance and increases reach trajectory consistency in healthy animals.Significance.The CLARA system is the first mouse behavior apparatus that uses markerless pose tracking to provide real-time closed-loop stimulation in response to the outcome of an unconstrained motor task. Additionally, we demonstrate that the CLARA system was essential for our investigating the role of closed-loop VNS stimulation on motor performance in healthy animals. This approach has high translational relevance for developing neurostimulation technology based on complex human behavior.

Automation of training and testing motor and related tasks in pre-clinical behavioural and rehabilitative neuroscience

Testing and training animals in motor and related tasks is a cornerstone of pre-clinical behavioural and rehabilitative neuroscience. Yet manually testing and training animals in these tasks is time consuming and analyses are often subjective. Consequently, there have been many recent advances in automating both the administration and analyses of animal behavioural training and testing. This review is an in-depth appraisal of the history of, and recent developments in, the automation of animal behavioural assays used in neuroscience. We describe the use of common locomotor and non-locomotor tasks used for motor training and testing before and after nervous system injury. This includes a discussion of how these tasks help us to understand the underlying mechanisms of neurological repair and the utility of some tasks for the delivery of rehabilitative training to enhance recovery. We propose two general approaches to automation: automating the physical administration of behavioural tasks (i.e., devices used to facilitate task training, rehabilitative training, and motor testing) and leveraging the use of machine learning in behaviour analysis to generate large volumes of unbiased and comprehensive data. The advantages and disadvantages of automating various motor tasks as well as the limitations of machine learning analyses are examined. In closing, we provide a critical appraisal of the current state of automation in animal behavioural neuroscience and a prospective on some of the advances in machine learning we believe will dramatically enhance the usefulness of these approaches for behavioural neuroscientists.

Real-Time Closed-Loop Feedback in Behavioral Time Scales Using DeepLabCut

Computer vision approaches have made significant inroads into offline tracking of behavior and estimating animal poses. In particular, because of their versatility, deep-learning approaches have been gaining attention in behavioral tracking without any markers. Here, we developed an approach using DeepLabCut for real-time estimation of movement. We trained a deep-neural network (DNN) offline with high-speed video data of a mouse whisking, then transferred the trained network to work with the same mouse, whisking in real-time. With this approach, we tracked the tips of three whiskers in an arc and converted positions into a TTL output within behavioral time scales, i.e., 10.5 ms. With this approach, it is possible to trigger output based on movement of individual whiskers, or on the distance between adjacent whiskers. Flexible closed-loop systems like the one we have deployed here can complement optogenetic approaches and can be used to directly manipulate the relationship between movement and neural activity.

Systems Neuroscience of Natural Behaviors in Rodents

Animals evolved in complex environments, producing a wide range of behaviors, including navigation, foraging, prey capture, and conspecific interactions, which vary over timescales ranging from milliseconds to days. Historically, these behaviors have been the focus of study for ecology and ethology, while systems neuroscience has largely focused on short timescale behaviors that can be repeated thousands of times and occur in highly artificial environments. Thanks to recent advances in machine learning, miniaturization, and computation, it is newly possible to study freely moving animals in more natural conditions while applying systems techniques: performing temporally specific perturbations, modeling behavioral strategies, and recording from large numbers of neurons while animals are freely moving. The authors of this review are a group of scientists with deep appreciation for the common aims of systems neuroscience, ecology, and ethology. We believe it is an extremely exciting time to be a neuroscientist, as we have an opportunity to grow as a field, to embrace interdisciplinary, open, collaborative research to provide new insights and allow researchers to link knowledge across disciplines, species, and scales. Here we discuss the origins of ethology, ecology, and systems neuroscience in the context of our own work and highlight how combining approaches across these fields has provided fresh insights into our research. We hope this review facilitates some of these interactions and alliances and helps us all do even better science, together.

DeepLabStream enables closed-loop behavioral experiments using deep learning-based markerless, real-time posture detection

In general, animal behavior can be described as the neuronal-driven sequence of reoccurring postures through time. Most of the available current technologies focus on offline pose estimation with high spatiotemporal resolution. However, to correlate behavior with neuronal activity it is often necessary to detect and react online to behavioral expressions. Here we present DeepLabStream, a versatile closed-loop tool providing real-time pose estimation to deliver posture dependent stimulations. DeepLabStream has a temporal resolution in the millisecond range, can utilize different input, as well as output devices and can be tailored to multiple experimental designs. We employ DeepLabStream to semi-autonomously run a second-order olfactory conditioning task with freely moving mice and optogenetically label neuronal ensembles active during specific head directions.

Real-time, low-latency closed-loop feedback using markerless posture tracking

The ability to control a behavioral task or stimulate neural activity based on animal behavior in real-time is an important tool for experimental neuroscientists. Ideally, such tools are noninvasive, low-latency, and provide interfaces to trigger external hardware based on posture. Recent advances in pose estimation with deep learning allows researchers to train deep neural networks to accurately quantify a wide variety of animal behaviors. Here, we provide a new DeepLabCut-Live! package that achieves low-latency real-time pose estimation (within 15 ms, >100 FPS), with an additional forward-prediction module that achieves zero-latency feedback, and a dynamic-cropping mode that allows for higher inference speeds. We also provide three options for using this tool with ease: (1) a stand-alone GUI (called DLC-Live! GUI), and integration into (2) Bonsai, and (3) AutoPilot. Lastly, we benchmarked performance on a wide range of systems so that experimentalists can easily decide what hardware is required for their needs.