Automated pose estimation in primates

Ethological computational psychiatry: Challenges and opportunities

Studying the intricacies of individual subjects' moods and cognitive processing over extended periods of time presents a formidable challenge in medicine. While much of systems neuroscience appropriately focuses on the link between neural circuit functions and well-constrained behaviors over short timescales (e.g., trials, hours), many mental health conditions involve complex interactions of mood and cognition that are non-stationary across behavioral contexts and evolve over extended timescales. Here, we discuss opportunities, challenges, and possible future directions in computational psychiatry to quantify non-stationary continuously monitored behaviors. We suggest that this exploratory effort may contribute to a more precision-based approach to treating mental disorders and facilitate a more robust reverse translation across animal species. We conclude with ethical considerations for any field that aims to bridge artificial intelligence and patient monitoring.

Control over a mixture of policies determines change of mind topology during continuous choice

Behavior is naturally organized into categorically distinct states with corresponding patterns of neural activity; how does the brain control those states? We propose that states are regulated by specific neural processes that implement meta-control that can blend simpler control processes. To test this hypothesis, we recorded from neurons in the dorsal anterior cingulate cortex (dACC) and dorsal premotor cortex (PMd) while macaques performed a continuous pursuit task with two moving prey that followed evasive strategies. We used a novel control theoretic approach to infer subjects' moment-to-moment latent control variables, which in turn dictated their blend of distinct identifiable control processes. We identified low-dimensional subspaces in neuronal responses that reflected the current strategy, the value of the pursued target, and the relative value of the two targets. The top two principal components of activity tracked changes of mind in abstract and change-type-specific formats, respectively. These results indicate that control of behavioral state reflects the interaction of brain processes found in dorsal prefrontal regions that implement a mixture over low-level control policies.

Learning of object-in-context sequences in freely-moving macaques

Flexible learning is a hallmark of primate cognition, which arises through interactions with changing environments. Studies of the neural basis for this flexibility are typically limited by laboratory settings that use minimal environmental cues and restrict interactions with the environment, including active sensing and exploration. To address this, we constructed a 3-D enclosure containing touchscreens on its walls, for studying cognition in freely moving macaques. To test flexible learning, two monkeys completed trials consisting of a regular sequence of object selections across four touchscreens. On each screen, the monkeys had to select by touching the sole correct object item ('target') among a set of four items, irrespective of their positions on the screen. Each item was the target on exactly one screen of the sequence, making correct performance conditioned on the spatiotemporal sequence rule across screens. Both monkeys successfully learned multiple 4-item sets (N=14 and 22 sets), totaling over 50 and 80 unique, conditional item-context memoranda, with no indication of capacity limits. The enclosure allowed freedom of movements leading up to and following the touchscreen interactions. To determine whether movement economy changed with learning, we reconstructed 3D position and movement dynamics using markerless tracking software and gyroscopic inertial measurements. Whereas general body positions remained consistent across repeated sequences, fine head movements varied as monkeys learned, within and across sequence sets, demonstrating learning set or "learning to learn". These results demonstrate monkeys' rapid, capacious, and flexible learning within an integrated, multisensory 3-D space. Furthermore, this approach enables the measurement of continuous behavior while ensuring precise experimental control and behavioral repetition of sequences over time. Overall, this approach harmonizes the design features that are needed for electrophysiological studies with tasks that showcase fully situated, flexible cognition.

OpenApePose, a database of annotated ape photographs for pose estimation

Because of their close relationship with humans, non-human apes (chimpanzees, bonobos, gorillas, orangutans, and gibbons, including siamangs) are of great scientific interest. The goal of understanding their complex behavior would be greatly advanced by the ability to perform video-based pose tracking. Tracking, however, requires high-quality annotated datasets of ape photographs. Here we present OpenApePose, a new public dataset of 71,868 photographs, annotated with 16 body landmarks of six ape species in naturalistic contexts. We show that a standard deep net (HRNet-W48) trained on ape photos can reliably track out-of-sample ape photos better than networks trained on monkeys (specifically, the OpenMonkeyPose dataset) and on humans (COCO) can. This trained network can track apes almost as well as the other networks can track their respective taxa, and models trained without one of the six ape species can track the held-out species better than the monkey and human models can. Ultimately, the results of our analyses highlight the importance of large, specialized databases for animal tracking systems and confirm the utility of our new ape database.

Hierarchical action encoding in prefrontal cortex of freely moving macaques

Our natural behavioral repertoires include coordinated actions of characteristic types. To better understand how neural activity relates to the expression of actions and action switches, we studied macaques performing a freely moving foraging task in an open environment. We developed a novel analysis pipeline that can identify meaningful units of behavior, corresponding to recognizable actions such as sitting, walking, jumping, and climbing. On the basis of transition probabilities between these actions, we found that behavior is organized in a modular and hierarchical fashion. We found that, after regressing out many potential confounders, actions are associated with specific patterns of firing in each of six prefrontal brain regions and that, overall, encoding of action category is progressively stronger in more dorsal and more caudal prefrontal regions. Together, these results establish a link between selection of units of primate behavior on one hand and neuronal activity in prefrontal regions on the other.

Hierarchical organization of rhesus macaque behavior

Primatologists, psychologists and neuroscientists have long hypothesized that primate behavior is highly structured. However, delineating that structure has been impossible due to the difficulties of precision behavioral tracking. Here we analyzed a dataset consisting of continuous measures of the 3D position of two male rhesus macaques (Macaca mulatta) performing three different tasks in a large unrestrained environment over several hours. Using an unsupervised embedding approach on the tracked joints, we identified commonly repeated pose patterns, which we call postures. We found that macaques' behavior is characterized by 49 distinct postures, lasting an average of 0.6 seconds. We found evidence that behavior is hierarchically organized, in that transitions between poses tend to occur within larger modules, which correspond to identifiable actions; these actions are further organized hierarchically. Our behavioral decomposition allows us to identify universal (cross-individual and cross-task) and unique (specific to each individual and task) principles of behavior. These results demonstrate the hierarchical nature of primate behavior, provide a method for the automated ethogramming of primate behavior, and provide important constraints on neural models of pose generation.

Promoting Good Nonhuman Primate Welfare outside Regular Working Hours

Promoting good primate welfare outside of daylight hours is an important task. The responsibility to provide a complex environment and environmental enrichment is an essential element of primate wellbeing programs that should be approached from a 24-h perspective and planned according to the species and individual needs, including giving animals the ability to interact with and control their environment during hours when animal care staff are not present. One must be aware, however, that their needs may differ at night-time from their care needs during the day when staff are present. Assessing welfare and providing enrichment during times when staff are not on hand can be achieved through the use of a variety of technologies, such as night-view cameras, animal-centred technologies, and data loggers. This paper will address the relevant topics concerning the care and welfare of primates during off-hours, and the use of related technologies to facilitate and assess wellbeing at these times.

Open-source tools for behavioral video analysis: Setup, methods, and best practices

Recently developed methods for video analysis, especially models for pose estimation and behavior classification, are transforming behavioral quantification to be more precise, scalable, and reproducible in fields such as neuroscience and ethology. These tools overcome long-standing limitations of manual scoring of video frames and traditional 'center of mass' tracking algorithms to enable video analysis at scale. The expansion of open-source tools for video acquisition and analysis has led to new experimental approaches to understand behavior. Here, we review currently available open-source tools for video analysis and discuss how to set up these methods for labs new to video recording. We also discuss best practices for developing and using video analysis methods, including community-wide standards and critical needs for the open sharing of datasets and code, more widespread comparisons of video analysis methods, and better documentation for these methods especially for new users. We encourage broader adoption and continued development of these tools, which have tremendous potential for accelerating scientific progress in understanding the brain and behavior.

The Benefits and Challenges of Conducting Primate Research in Different Settings

Internationally, primate research takes place in laboratories, zoos, sanctuaries, and the wild. All of these settings present unique advantages and challenges in terms of methodology, translatability, animal welfare, and ethics. In this novel commentary, we explore the scientific and ethical benefits and drawbacks of conducting non-invasive psychological research with primates in each setting. We also suggest ways to overcome some of the barriers. We argue that while there may be greater experimental control in laboratory-based research, settings that more closely mirror primates' natural habitats are generally better suited to meet their specialized needs. More naturalistic research settings, including field studies, may also circumvent some ethical concerns associated with research in captivity, and yield more ecologically valid data.

The Promise of Behavioral Tracking Systems for Advancing Primate Animal Welfare

Recent years have witnessed major advances in the ability of computerized systems to track the positions of animals as they move through large and unconstrained environments. These systems have so far been a great boon in the fields of primatology, psychology, neuroscience, and biomedicine. Here, we discuss the promise of these technologies for animal welfare. Their potential benefits include identifying and reducing pain, suffering, and distress in captive populations, improving laboratory animal welfare within the context of the three Rs of animal research (reduction, refinement, and replacement), and applying our understanding of animal behavior to increase the "natural" behaviors in captive and wild populations facing human impact challenges. We note that these benefits are often incidental to the designed purpose of these tracking systems, a reflection of the fact that animal welfare is not inimical to research progress, but instead, that the aligned interests between basic research and welfare hold great promise for improvements to animal well-being.

Primatology in zoos: Studying behavior, cognition, and welfare

Primatological research in zoos is increasing globally. Such research allows scientists to study primate biology, behavior, and cognition while helping to advance the welfare of captive primates. Moreover, zoos welcome millions of visitors annually, which creates unique opportunities for public engagement with this research. Reflecting the importance of zoos in the field of primatology, the articles in this special issue showcase the range of primatological research currently being conducted in zoos around the world. With this special issue, I have chosen to focus on research examining primate behavior and cognition, addressing both basic and applied questions. The articles included in this special issue also highlight the array of technologies and methods being used to study zoo-housed primates. While zoos house a variety of primate species, potentially enabling the testing of under-studied species or widescale comparative research, great apes are disproportionately represented in current zoo-based research. Thus, while an interest in conducting research with primates in zoos continues to grow, there are still opportunities to increase the breadth and diversity of this study.

Nonhuman primate abnormal behavior: Etiology, assessment, and treatment

Across captive settings, nonhuman primates may develop an array of abnormal behaviors including stereotypic and self-injurious behavior. Abnormal behavior can indicate a state of poor welfare, since it is often associated with a suboptimal environment. However, this may not always be the case as some behaviors can develop independently of any psychological distress, be triggered in environments known to promote welfare, and be part of an animal's coping mechanism. Furthermore, not all animals develop abnormal behavior, which has led researchers to assess risk factors that differentiate individuals in the display of these behaviors. Intrinsic risk factors that have been identified include the animal's species and genetics, age, sex, temperament, and clinical condition, while environmental risk factors include variables such as the animal's rearing, housing condition, husbandry procedures, and research experiences. To identify specific triggers and at-risk animals, the expression of abnormal behavior in captive nonhuman primates should be routinely addressed in a consistent manner by appropriately trained staff. Which behaviors to assess, what assessment methods to use, which primates to monitor, and the aims of data collection should all be identified before proceeding to an intervention and/or treatment. This article provides guidance for this process, by presenting an overview of known triggers and risk factors that should be considered, steps to design a comprehensive evaluation plan, and strategies that might be used for prevention or treatment. It also outlines the tools and processes for assessing and evaluating behavior in an appendix. This process will lead to a better understanding of abnormal behavior in captive primate colonies and ultimately to improved welfare.

Deep-learning based identification, tracking, pose estimation, and behavior classification of interacting primates and mice in complex environments

The quantification of behaviors of interest from video data is commonly used to study brain function, the effects of pharmacological interventions, and genetic alterations. Existing approaches lack the capability to analyze the behavior of groups of animals in complex environments. We present a novel deep learning architecture for classifying individual and social animal behavior, even in complex environments directly from raw video frames, while requiring no intervention after initial human supervision. Our behavioral classifier is embedded in a pipeline (SIPEC) that performs segmentation, identification, pose-estimation, and classification of complex behavior, outperforming the state of the art. SIPEC successfully recognizes multiple behaviors of freely moving individual mice as well as socially interacting non-human primates in 3D, using data only from simple mono-vision cameras in home-cage setups.