3D pose estimation enables virtual head fixation in freely moving rats

Deciphering social traits and pathophysiological conditions from natural behaviors in common marmosets

Nonhuman primates (NHPs) are indispensable animal models by virtue of the continuity of behavioral repertoires across primates, including humans. However, behavioral assessment at the laboratory level has so far been limited. Employing the application of three-dimensional (3D) pose estimation and the optimal integration of subsequent analytic methodologies, we demonstrate that our artificial intelligence (AI)-based approach has successfully deciphered the ethological, cognitive, and pathological traits of common marmosets from their natural behaviors. By applying multiple deep neural networks trained with large-scale datasets, we established an evaluation system that could reconstruct and estimate the 3D poses of the marmosets, a small NHP that is suitable for analyzing complex natural behaviors in laboratory setups. We further developed downstream analytic methodologies to quantify a variety of behavioral parameters beyond motion kinematics. We revealed the distinct parental roles of male and female marmosets through automated detections of food-sharing behaviors using a spatial-temporal filter on 3D poses. Employing a recurrent neural network to analyze 3D pose time series data during social interactions, we additionally discovered that marmosets adjusted their behaviors based on others' internal state, which is not directly observable but can be inferred from the sequence of others' actions. Moreover, a fully unsupervised approach enabled us to detect progressively appearing symptomatic behaviors over a year in a Parkinson's disease model. The high-throughput and versatile nature of an AI-driven approach to analyze natural behaviors will open a new avenue for neuroscience research dealing with big-data analyses of social and pathophysiological behaviors in NHPs.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

Development of a 3D tracking system for multiple marmosets under free-moving conditions

Assessment of social interactions and behavioral changes in nonhuman primates is useful for understanding brain function changes during life events and pathogenesis of neurological diseases. The common marmoset (Callithrix jacchus), which lives in a nuclear family like humans, is a useful model, but longitudinal automated behavioral observation of multiple animals has not been achieved. Here, we developed a Full Monitoring and Animal Identification (FulMAI) system for longitudinal detection of three-dimensional (3D) trajectories of each individual in multiple marmosets under free-moving conditions by combining video tracking, Light Detection and Ranging, and deep learning. Using this system, identification of each animal was more than 97% accurate. Location preferences and inter-individual distance could be calculated, and deep learning could detect grooming behavior. The FulMAI system allows us to analyze the natural behavior of individuals in a family over their lifetime and understand how behavior changes due to life events together with other data.

SaLSa: A Combinatory Approach of Semi-Automatic Labeling and Long Short-Term Memory to Classify Behavioral Syllables

Accurately and quantitatively describing mouse behavior is an important area. Although advances in machine learning have made it possible to track their behaviors accurately, reliable classification of behavioral sequences or syllables remains a challenge. In this study, we present a novel machine learning approach, called SaLSa (a combination of semi-automatic labeling and long short-term memory-based classification), to classify behavioral syllables of mice exploring an open field. This approach consists of two major steps. First, after tracking multiple body parts, spatial and temporal features of their egocentric coordinates are extracted. A fully automated unsupervised process identifies candidates for behavioral syllables, followed by manual labeling of behavioral syllables using a graphical user interface (GUI). Second, a long short-term memory (LSTM) classifier is trained with the labeled data. We found that the classification performance was marked over 97%. It provides a performance equivalent to a state-of-the-art model while classifying some of the syllables. We applied this approach to examine how hyperactivity in a mouse model of Alzheimer's disease develops with age. When the proportion of each behavioral syllable was compared between genotypes and sexes, we found that the characteristic hyperlocomotion of female Alzheimer's disease mice emerges between four and eight months. In contrast, age-related reduction in rearing is common regardless of genotype and sex. Overall, SaLSa enables detailed characterization of mouse behavior.

A unified open-source platform for multimodal neural recording and perturbation during naturalistic behavior

Behavioral neuroscience faces two conflicting demands: long-duration recordings from large neural populations and unimpeded animal behavior. To meet this challenge, we developed ONIX, an open-source data acquisition system with high data throughput (2GB/sec) and low closed-loop latencies (<1ms) that uses a novel 0.3 mm thin tether to minimize behavioral impact. Head position and rotation are tracked in 3D and used to drive active commutation without torque measurements. ONIX can acquire from combinations of passive electrodes, Neuropixels probes, head-mounted microscopes, cameras, 3D-trackers, and other data sources. We used ONIX to perform uninterrupted, long (~7 hours) neural recordings in mice as they traversed complex 3-dimensional terrain. ONIX allowed exploration with similar mobility as non-implanted animals, in contrast to conventional tethered systems which restricted movement. By combining long recordings with full mobility, our technology will enable new progress on questions that require high-quality neural recordings during ethologically grounded behaviors.

Evaluation of mouse behavioral responses to nutritive versus nonnutritive sugar using a deep learning-based 3D real-time pose estimation system

Animals are able to detect the nutritional content of sugar independently of taste. When given a choice between nutritive sugar and nonnutritive sugar, animals develop a preference for nutritive sugar over nonnutritive sugar during a period of food deprivation (Buchanan et al., 2022; Dus et al., 2011; 2015; Tan et al., 2020; Tellez et al., 2016). To quantify behavioral features during an episode of licking nutritive versus nonnutritive sugar, we implemented a multi-vision, deep learning-based 3D pose estimation system, termed the AI Vision Analysis for Three-dimensional Action in Real-Time (AVATAR)(Kim et al., 2022). Using this method, we found that mice exhibit significantly different approach behavioral responses toward nutritive sugar versus nonnutritive sugar even before licking a sugar solution. Notably, the behavioral sequences during the approach toward nutritive versus nonnutritive sugar became significantly different over time. These results suggest that the nutritional value of sugar not only promotes its consumption but also elicits distinct repertoires of feeding behavior in deprived mice.

Manipulation-specific cortical activity as mice handle food

Food handling offers unique yet largely unexplored opportunities to investigate how cortical activity relates to forelimb movements in a natural, ethologically essential, and kinematically rich form of manual dexterity. To determine these relationships, we recorded high-speed (1,000 fps) video and multi-channel electrophysiological cortical spiking activity while mice handled food. The high temporal resolution of the video allowed us to decompose active manipulation ("oromanual") events into characteristic submovements, enabling event-aligned analysis of cortical activity. Activity in forelimb M1 was strongly modulated during food handling, generally higher during oromanual events and lower during holding intervals. Optogenetic silencing and stimulation of forelimb M1 neurons partially affected food-handling movements, exerting suppressive and activating effects, respectively. We also extended the analysis to forelimb S1 and lateral M1, finding broadly similar oromanual-related activity across all three areas. However, each area's activity displayed a distinct timing and phasic/tonic temporal profile, which was further analyzed by non-negative matrix factorization and demonstrated to be attributable to area-specific composition of activity classes. Current or future forelimb position could be accurately predicted from activity in all three regions, indicating that the cortical activity in these areas contains high information content about forelimb movements during food handling. These results thus establish that cortical activity during food handling is manipulation specific, distributed, and broadly similar across multiple sensorimotor areas while also exhibiting area- and submovement-specific relationships with the fast kinematic hallmarks of this natural form of complex free-object-handling manual dexterity.

A flexible Python-based touchscreen chamber for operant conditioning reveals improved visual perception of cardinal orientations in mice

Natural scenes are composed of a wide range of edge angles and spatial frequencies, with a strong overrepresentation of vertical and horizontal edges. Correspondingly, many mammalian species are much better at discriminating these cardinal orientations compared to obliques. A potential reason for this increased performance could be an increased number of neurons in the visual cortex that are tuned to cardinal orientations, which is likely to be an adaptation to the natural scene statistics. Such biased angular tuning has recently been shown in the mouse primary visual cortex. However, it is still unknown if mice also show a perceptual dominance of cardinal orientations. Here, we describe the design of a novel custom-built touchscreen chamber that allows testing natural scene perception and orientation discrimination performance by applying different task designs. Using this chamber, we applied an iterative convergence towards orientation discrimination thresholds for cardinal or oblique orientations in different cohorts of mice. Surprisingly, the expert discrimination performance was similar for both groups but showed large inter-individual differences in performance and training time. To study the discrimination of cardinal and oblique stimuli in the same mice, we, therefore, applied, a different training regime where mice learned to discriminate cardinal and oblique gratings in parallel. Parallel training revealed a higher task performance for cardinal orientations in an early phase of the training. The performance for both orientations became similar after prolonged training, suggesting that learning permits equally high perceptual tuning towards oblique stimuli. In summary, our custom-built touchscreen chamber offers a flexible tool to test natural visual perception in rodents and revealed a training-induced increase in the perception of oblique gratings. The touchscreen chamber is entirely open-source, easy to build, and freely available to the scientific community to conduct visual or multimodal behavioral studies. It is also based on the FAIR principles for data management and sharing and could therefore serve as a catalyst for testing the perception of complex and natural visual stimuli across behavioral labs.