1
|
Monosov IE, Zimmermann J, Frank MJ, Mathis MW, Baker JT. Ethological computational psychiatry: Challenges and opportunities. Curr Opin Neurobiol 2024; 86:102881. [PMID: 38696972 PMCID: PMC11162904 DOI: 10.1016/j.conb.2024.102881] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/26/2023] [Revised: 04/02/2024] [Accepted: 04/03/2024] [Indexed: 05/04/2024]
Abstract
Studying the intricacies of individual subjects' moods and cognitive processing over extended periods of time presents a formidable challenge in medicine. While much of systems neuroscience appropriately focuses on the link between neural circuit functions and well-constrained behaviors over short timescales (e.g., trials, hours), many mental health conditions involve complex interactions of mood and cognition that are non-stationary across behavioral contexts and evolve over extended timescales. Here, we discuss opportunities, challenges, and possible future directions in computational psychiatry to quantify non-stationary continuously monitored behaviors. We suggest that this exploratory effort may contribute to a more precision-based approach to treating mental disorders and facilitate a more robust reverse translation across animal species. We conclude with ethical considerations for any field that aims to bridge artificial intelligence and patient monitoring.
Collapse
Affiliation(s)
- Ilya E. Monosov
- Departments of Neuroscience, Biomedical Engineering, Electrical Engineering, and Neurosurgery, Washington University School of Medicine, St. Louis, MO, USA
| | - Jan Zimmermann
- Department of Neuroscience, University of Minnesota, Minneapolis, MN, USA
| | - Michael J. Frank
- Carney Center for Computational Brain Science, Brown University, Providence, RI, USA
| | | | | |
Collapse
|
2
|
Fine JM, Yoo SBM, Hayden BY. Control over a mixture of policies determines change of mind topology during continuous choice. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.04.18.590154. [PMID: 38712284 PMCID: PMC11071291 DOI: 10.1101/2024.04.18.590154] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/08/2024]
Abstract
Behavior is naturally organized into categorically distinct states with corresponding patterns of neural activity; how does the brain control those states? We propose that states are regulated by specific neural processes that implement meta-control that can blend simpler control processes. To test this hypothesis, we recorded from neurons in the dorsal anterior cingulate cortex (dACC) and dorsal premotor cortex (PMd) while macaques performed a continuous pursuit task with two moving prey that followed evasive strategies. We used a novel control theoretic approach to infer subjects' moment-to-moment latent control variables, which in turn dictated their blend of distinct identifiable control processes. We identified low-dimensional subspaces in neuronal responses that reflected the current strategy, the value of the pursued target, and the relative value of the two targets. The top two principal components of activity tracked changes of mind in abstract and change-type-specific formats, respectively. These results indicate that control of behavioral state reflects the interaction of brain processes found in dorsal prefrontal regions that implement a mixture over low-level control policies.
Collapse
|
3
|
Abbaspoor S, Rahman K, Zinke W, Hoffman KL. Learning of object-in-context sequences in freely-moving macaques. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.12.11.571113. [PMID: 38168449 PMCID: PMC10760043 DOI: 10.1101/2023.12.11.571113] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/05/2024]
Abstract
Flexible learning is a hallmark of primate cognition, which arises through interactions with changing environments. Studies of the neural basis for this flexibility are typically limited by laboratory settings that use minimal environmental cues and restrict interactions with the environment, including active sensing and exploration. To address this, we constructed a 3-D enclosure containing touchscreens on its walls, for studying cognition in freely moving macaques. To test flexible learning, two monkeys completed trials consisting of a regular sequence of object selections across four touchscreens. On each screen, the monkeys had to select by touching the sole correct object item ('target') among a set of four items, irrespective of their positions on the screen. Each item was the target on exactly one screen of the sequence, making correct performance conditioned on the spatiotemporal sequence rule across screens. Both monkeys successfully learned multiple 4-item sets (N=14 and 22 sets), totaling over 50 and 80 unique, conditional item-context memoranda, with no indication of capacity limits. The enclosure allowed freedom of movements leading up to and following the touchscreen interactions. To determine whether movement economy changed with learning, we reconstructed 3D position and movement dynamics using markerless tracking software and gyroscopic inertial measurements. Whereas general body positions remained consistent across repeated sequences, fine head movements varied as monkeys learned, within and across sequence sets, demonstrating learning set or "learning to learn". These results demonstrate monkeys' rapid, capacious, and flexible learning within an integrated, multisensory 3-D space. Furthermore, this approach enables the measurement of continuous behavior while ensuring precise experimental control and behavioral repetition of sequences over time. Overall, this approach harmonizes the design features that are needed for electrophysiological studies with tasks that showcase fully situated, flexible cognition.
Collapse
Affiliation(s)
- S Abbaspoor
- Department of Psychological Sciences, Vanderbilt University, Nashville, United States
- Vanderbilt Vision Research Center, Vanderbilt University, Nashville, United States
| | - K Rahman
- Department of Psychological Sciences, Vanderbilt University, Nashville, United States
- Vanderbilt Vision Research Center, Vanderbilt University, Nashville, United States
| | - W Zinke
- Department of Psychological Sciences, Vanderbilt University, Nashville, United States
- Vanderbilt Vision Research Center, Vanderbilt University, Nashville, United States
| | - K L Hoffman
- Department of Psychological Sciences, Vanderbilt University, Nashville, United States
- Vanderbilt Vision Research Center, Vanderbilt University, Nashville, United States
- Vanderbilt Brain Institute, Vanderbilt University, Nashville, United States
- Department of Biomedical Engineering, Vanderbilt University, Nashville, United States
| |
Collapse
|
4
|
Desai N, Bala P, Richardson R, Raper J, Zimmermann J, Hayden B. OpenApePose, a database of annotated ape photographs for pose estimation. eLife 2023; 12:RP86873. [PMID: 38078902 PMCID: PMC10712952 DOI: 10.7554/elife.86873] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/18/2023] Open
Abstract
Because of their close relationship with humans, non-human apes (chimpanzees, bonobos, gorillas, orangutans, and gibbons, including siamangs) are of great scientific interest. The goal of understanding their complex behavior would be greatly advanced by the ability to perform video-based pose tracking. Tracking, however, requires high-quality annotated datasets of ape photographs. Here we present OpenApePose, a new public dataset of 71,868 photographs, annotated with 16 body landmarks of six ape species in naturalistic contexts. We show that a standard deep net (HRNet-W48) trained on ape photos can reliably track out-of-sample ape photos better than networks trained on monkeys (specifically, the OpenMonkeyPose dataset) and on humans (COCO) can. This trained network can track apes almost as well as the other networks can track their respective taxa, and models trained without one of the six ape species can track the held-out species better than the monkey and human models can. Ultimately, the results of our analyses highlight the importance of large, specialized databases for animal tracking systems and confirm the utility of our new ape database.
Collapse
Affiliation(s)
- Nisarg Desai
- Department of Neuroscience and Center for Magnetic Resonance Research, University of MinnesotaMinneapolisUnited States
| | - Praneet Bala
- Department of Computer Science, University of MinnesotaMinneapolisUnited States
| | - Rebecca Richardson
- Emory National Primate Research Center, Emory UniversityAtlantaUnited States
| | - Jessica Raper
- Emory National Primate Research Center, Emory UniversityAtlantaUnited States
| | - Jan Zimmermann
- Department of Neuroscience and Center for Magnetic Resonance Research, University of MinnesotaMinneapolisUnited States
| | - Benjamin Hayden
- Department of Neuroscience and Center for Magnetic Resonance Research, University of MinnesotaMinneapolisUnited States
| |
Collapse
|
5
|
Voloh B, Maisson DJN, Cervera RL, Conover I, Zambre M, Hayden B, Zimmermann J. Hierarchical action encoding in prefrontal cortex of freely moving macaques. Cell Rep 2023; 42:113091. [PMID: 37656619 PMCID: PMC10591875 DOI: 10.1016/j.celrep.2023.113091] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2022] [Revised: 06/23/2023] [Accepted: 08/18/2023] [Indexed: 09/03/2023] Open
Abstract
Our natural behavioral repertoires include coordinated actions of characteristic types. To better understand how neural activity relates to the expression of actions and action switches, we studied macaques performing a freely moving foraging task in an open environment. We developed a novel analysis pipeline that can identify meaningful units of behavior, corresponding to recognizable actions such as sitting, walking, jumping, and climbing. On the basis of transition probabilities between these actions, we found that behavior is organized in a modular and hierarchical fashion. We found that, after regressing out many potential confounders, actions are associated with specific patterns of firing in each of six prefrontal brain regions and that, overall, encoding of action category is progressively stronger in more dorsal and more caudal prefrontal regions. Together, these results establish a link between selection of units of primate behavior on one hand and neuronal activity in prefrontal regions on the other.
Collapse
Affiliation(s)
- Benjamin Voloh
- Department of Neuroscience, University of Minnesota, Minneapolis, MN 55455, USA
| | - David J-N Maisson
- Department of Neuroscience, University of Minnesota, Minneapolis, MN 55455, USA
| | | | - Indirah Conover
- Department of Neuroscience, University of Minnesota, Minneapolis, MN 55455, USA
| | - Mrunal Zambre
- Department of Neuroscience, University of Minnesota, Minneapolis, MN 55455, USA
| | - Benjamin Hayden
- Department of Neuroscience, University of Minnesota, Minneapolis, MN 55455, USA; Department of Neurosurgery, Baylor College of Medicine, Houston, TX 77030, USA
| | - Jan Zimmermann
- Department of Neuroscience, University of Minnesota, Minneapolis, MN 55455, USA; Center for Magnetic Resonance Research, University of Minnesota, Minneapolis, MN 55455, USA.
| |
Collapse
|
6
|
Voloh B, Eisenreich BR, Maisson DJN, Ebitz RB, Park HS, Hayden BY, Zimmermann J. Hierarchical organization of rhesus macaque behavior. OXFORD OPEN NEUROSCIENCE 2023; 2:kvad006. [PMID: 37577290 PMCID: PMC10421634 DOI: 10.1093/oons/kvad006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Revised: 06/11/2023] [Accepted: 06/12/2023] [Indexed: 08/15/2023]
Abstract
Primatologists, psychologists and neuroscientists have long hypothesized that primate behavior is highly structured. However, delineating that structure has been impossible due to the difficulties of precision behavioral tracking. Here we analyzed a dataset consisting of continuous measures of the 3D position of two male rhesus macaques (Macaca mulatta) performing three different tasks in a large unrestrained environment over several hours. Using an unsupervised embedding approach on the tracked joints, we identified commonly repeated pose patterns, which we call postures. We found that macaques' behavior is characterized by 49 distinct postures, lasting an average of 0.6 seconds. We found evidence that behavior is hierarchically organized, in that transitions between poses tend to occur within larger modules, which correspond to identifiable actions; these actions are further organized hierarchically. Our behavioral decomposition allows us to identify universal (cross-individual and cross-task) and unique (specific to each individual and task) principles of behavior. These results demonstrate the hierarchical nature of primate behavior, provide a method for the automated ethogramming of primate behavior, and provide important constraints on neural models of pose generation.
Collapse
Affiliation(s)
- Benjamin Voloh
- Department of Neuroscience, Center for Magnetic Resonance Research, Center for Neuroengineering, 1 Baylor Plaza, Houston, TX 77030
| | - Benjamin R Eisenreich
- Department of Neuroscience, Center for Magnetic Resonance Research, Center for Neuroengineering, 1 Baylor Plaza, Houston, TX 77030
| | - David J-N Maisson
- Department of Neuroscience, Center for Magnetic Resonance Research, Center for Neuroengineering, 1 Baylor Plaza, Houston, TX 77030
| | - R Becket Ebitz
- Department of Neuroscience, Center for Magnetic Resonance Research, Center for Neuroengineering, 1 Baylor Plaza, Houston, TX 77030
| | - Hyun Soo Park
- Department of Computer Science and Engineering, University of Minnesota, 40 Church St, Minneapolis, MN 55455, USA
| | - Benjamin Y Hayden
- Department of Neuroscience, Center for Magnetic Resonance Research, Center for Neuroengineering, 1 Baylor Plaza, Houston, TX 77030
| | - Jan Zimmermann
- Department of Neuroscience, Center for Magnetic Resonance Research, Center for Neuroengineering, 1 Baylor Plaza, Houston, TX 77030
| |
Collapse
|
7
|
Brando S, Vitale A, Bacon M. Promoting Good Nonhuman Primate Welfare outside Regular Working Hours. Animals (Basel) 2023; 13:ani13081423. [PMID: 37106985 PMCID: PMC10135122 DOI: 10.3390/ani13081423] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2023] [Revised: 04/14/2023] [Accepted: 04/16/2023] [Indexed: 04/29/2023] Open
Abstract
Promoting good primate welfare outside of daylight hours is an important task. The responsibility to provide a complex environment and environmental enrichment is an essential element of primate wellbeing programs that should be approached from a 24-h perspective and planned according to the species and individual needs, including giving animals the ability to interact with and control their environment during hours when animal care staff are not present. One must be aware, however, that their needs may differ at night-time from their care needs during the day when staff are present. Assessing welfare and providing enrichment during times when staff are not on hand can be achieved through the use of a variety of technologies, such as night-view cameras, animal-centred technologies, and data loggers. This paper will address the relevant topics concerning the care and welfare of primates during off-hours, and the use of related technologies to facilitate and assess wellbeing at these times.
Collapse
Affiliation(s)
- Sabrina Brando
- AnimalConcepts, Teulada, P.O. Box 378, 03725 Alicante, Spain
| | - Augusto Vitale
- Center for Behavioural Sciences and Mental Health, Istituto Superiore di Sanità, 00161 Rome, Italy
| | - Madison Bacon
- Department of Animal Science, University of Minnesota-Twin Cities, Saint Paul, MN 55455, USA
| |
Collapse
|
8
|
Luxem K, Sun JJ, Bradley SP, Krishnan K, Yttri E, Zimmermann J, Pereira TD, Laubach M. Open-source tools for behavioral video analysis: Setup, methods, and best practices. eLife 2023; 12:79305. [PMID: 36951911 PMCID: PMC10036114 DOI: 10.7554/elife.79305] [Citation(s) in RCA: 16] [Impact Index Per Article: 16.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2022] [Accepted: 03/03/2023] [Indexed: 03/24/2023] Open
Abstract
Recently developed methods for video analysis, especially models for pose estimation and behavior classification, are transforming behavioral quantification to be more precise, scalable, and reproducible in fields such as neuroscience and ethology. These tools overcome long-standing limitations of manual scoring of video frames and traditional 'center of mass' tracking algorithms to enable video analysis at scale. The expansion of open-source tools for video acquisition and analysis has led to new experimental approaches to understand behavior. Here, we review currently available open-source tools for video analysis and discuss how to set up these methods for labs new to video recording. We also discuss best practices for developing and using video analysis methods, including community-wide standards and critical needs for the open sharing of datasets and code, more widespread comparisons of video analysis methods, and better documentation for these methods especially for new users. We encourage broader adoption and continued development of these tools, which have tremendous potential for accelerating scientific progress in understanding the brain and behavior.
Collapse
Affiliation(s)
- Kevin Luxem
- Cellular Neuroscience, Leibniz Institute for Neurobiology, Magdeburg, Germany
| | - Jennifer J Sun
- Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, United States
| | - Sean P Bradley
- Rodent Behavioral Core, National Institute of Mental Health, National Institutes of Health, Bethesda, United States
| | - Keerthi Krishnan
- Department of Biochemistry and Cellular & Molecular Biology, University of Tennessee, Knoxville, United States
| | - Eric Yttri
- Department of Biological Sciences, Carnegie Mellon University, Pittsburgh, United States
| | - Jan Zimmermann
- Department of Neuroscience, University of Minnesota, Minneapolis, United States
| | - Talmo D Pereira
- The Salk Institute of Biological Studies, La Jolla, United States
| | - Mark Laubach
- Department of Neuroscience, American University, Washington D.C., United States
| |
Collapse
|
9
|
The Benefits and Challenges of Conducting Primate Research in Different Settings. Animals (Basel) 2022; 13:ani13010133. [PMID: 36611742 PMCID: PMC9817835 DOI: 10.3390/ani13010133] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Revised: 12/07/2022] [Accepted: 12/24/2022] [Indexed: 12/30/2022] Open
Abstract
Internationally, primate research takes place in laboratories, zoos, sanctuaries, and the wild. All of these settings present unique advantages and challenges in terms of methodology, translatability, animal welfare, and ethics. In this novel commentary, we explore the scientific and ethical benefits and drawbacks of conducting non-invasive psychological research with primates in each setting. We also suggest ways to overcome some of the barriers. We argue that while there may be greater experimental control in laboratory-based research, settings that more closely mirror primates' natural habitats are generally better suited to meet their specialized needs. More naturalistic research settings, including field studies, may also circumvent some ethical concerns associated with research in captivity, and yield more ecologically valid data.
Collapse
|
10
|
Knaebe B, Weiss CC, Zimmermann J, Hayden BY. The Promise of Behavioral Tracking Systems for Advancing Primate Animal Welfare. Animals (Basel) 2022; 12:1648. [PMID: 35804547 PMCID: PMC9265027 DOI: 10.3390/ani12131648] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/20/2022] [Revised: 06/17/2022] [Accepted: 06/21/2022] [Indexed: 11/16/2022] Open
Abstract
Recent years have witnessed major advances in the ability of computerized systems to track the positions of animals as they move through large and unconstrained environments. These systems have so far been a great boon in the fields of primatology, psychology, neuroscience, and biomedicine. Here, we discuss the promise of these technologies for animal welfare. Their potential benefits include identifying and reducing pain, suffering, and distress in captive populations, improving laboratory animal welfare within the context of the three Rs of animal research (reduction, refinement, and replacement), and applying our understanding of animal behavior to increase the "natural" behaviors in captive and wild populations facing human impact challenges. We note that these benefits are often incidental to the designed purpose of these tracking systems, a reflection of the fact that animal welfare is not inimical to research progress, but instead, that the aligned interests between basic research and welfare hold great promise for improvements to animal well-being.
Collapse
Affiliation(s)
- Brenna Knaebe
- Department of Neuroscience and Center for Magnetic Resonance Research, University of Minnesota, Minneapolis, MN 55455, USA; (C.C.W.); (J.Z.); (B.Y.H.)
| | | | | | | |
Collapse
|
11
|
Hopper LM. Primatology in zoos: Studying behavior, cognition, and welfare. Am J Primatol 2022; 84:e23385. [PMID: 35451513 DOI: 10.1002/ajp.23385] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2022] [Accepted: 04/02/2022] [Indexed: 12/27/2022]
Abstract
Primatological research in zoos is increasing globally. Such research allows scientists to study primate biology, behavior, and cognition while helping to advance the welfare of captive primates. Moreover, zoos welcome millions of visitors annually, which creates unique opportunities for public engagement with this research. Reflecting the importance of zoos in the field of primatology, the articles in this special issue showcase the range of primatological research currently being conducted in zoos around the world. With this special issue, I have chosen to focus on research examining primate behavior and cognition, addressing both basic and applied questions. The articles included in this special issue also highlight the array of technologies and methods being used to study zoo-housed primates. While zoos house a variety of primate species, potentially enabling the testing of under-studied species or widescale comparative research, great apes are disproportionately represented in current zoo-based research. Thus, while an interest in conducting research with primates in zoos continues to grow, there are still opportunities to increase the breadth and diversity of this study.
Collapse
Affiliation(s)
- Lydia M Hopper
- Department of Molecular and Comparative Pathobiology, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA
| |
Collapse
|
12
|
Lutz CK, Coleman K, Hopper LM, Novak MA, Perlman JE, Pomerantz O. Nonhuman primate abnormal behavior: Etiology, assessment, and treatment. Am J Primatol 2022; 84:e23380. [PMID: 35383995 DOI: 10.1002/ajp.23380] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2022] [Revised: 03/07/2022] [Accepted: 03/12/2022] [Indexed: 12/29/2022]
Abstract
Across captive settings, nonhuman primates may develop an array of abnormal behaviors including stereotypic and self-injurious behavior. Abnormal behavior can indicate a state of poor welfare, since it is often associated with a suboptimal environment. However, this may not always be the case as some behaviors can develop independently of any psychological distress, be triggered in environments known to promote welfare, and be part of an animal's coping mechanism. Furthermore, not all animals develop abnormal behavior, which has led researchers to assess risk factors that differentiate individuals in the display of these behaviors. Intrinsic risk factors that have been identified include the animal's species and genetics, age, sex, temperament, and clinical condition, while environmental risk factors include variables such as the animal's rearing, housing condition, husbandry procedures, and research experiences. To identify specific triggers and at-risk animals, the expression of abnormal behavior in captive nonhuman primates should be routinely addressed in a consistent manner by appropriately trained staff. Which behaviors to assess, what assessment methods to use, which primates to monitor, and the aims of data collection should all be identified before proceeding to an intervention and/or treatment. This article provides guidance for this process, by presenting an overview of known triggers and risk factors that should be considered, steps to design a comprehensive evaluation plan, and strategies that might be used for prevention or treatment. It also outlines the tools and processes for assessing and evaluating behavior in an appendix. This process will lead to a better understanding of abnormal behavior in captive primate colonies and ultimately to improved welfare.
Collapse
Affiliation(s)
- Corrine K Lutz
- Institute for Laboratory Animal Research, The National Academies of Sciences, Engineering, and Medicine, Washington, District of Columbia, USA
| | - Kristine Coleman
- Division of Comparative Medicine, Oregon National Primate Research Center, Oregon Health and Science University, Beaverton, Oregon, USA
| | - Lydia M Hopper
- Department of Molecular and Comparative Pathobiology, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA
| | - Melinda A Novak
- Department of Psychological and Brain Sciences, University of Massachusetts, Amherst, Massachusetts, USA
| | - Jaine E Perlman
- Division of Animal Resources, Yerkes National Primate Research Center, Emory University, Atlanta, Georgia, USA
| | - Ori Pomerantz
- Population and Behavioral Health Services, California National Primate Research Center, University of California, Davis, California, USA
| |
Collapse
|
13
|
Marks M, Qiuhan J, Sturman O, von Ziegler L, Kollmorgen S, von der Behrens W, Mante V, Bohacek J, Yanik MF. Deep-learning based identification, tracking, pose estimation, and behavior classification of interacting primates and mice in complex environments. NAT MACH INTELL 2022; 4:331-340. [PMID: 35465076 PMCID: PMC7612650 DOI: 10.1038/s42256-022-00477-5] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2020] [Accepted: 03/13/2022] [Indexed: 11/24/2022]
Abstract
The quantification of behaviors of interest from video data is commonly used to study brain function, the effects of pharmacological interventions, and genetic alterations. Existing approaches lack the capability to analyze the behavior of groups of animals in complex environments. We present a novel deep learning architecture for classifying individual and social animal behavior, even in complex environments directly from raw video frames, while requiring no intervention after initial human supervision. Our behavioral classifier is embedded in a pipeline (SIPEC) that performs segmentation, identification, pose-estimation, and classification of complex behavior, outperforming the state of the art. SIPEC successfully recognizes multiple behaviors of freely moving individual mice as well as socially interacting non-human primates in 3D, using data only from simple mono-vision cameras in home-cage setups.
Collapse
Affiliation(s)
- Markus Marks
- Institute of Neuroinformatics ETH Zürich and University of Zürich, Switzerland
- Neuroscience Center Zurich, ETH Zürich and University of Zürich, Switzerland
| | - Jin Qiuhan
- Laboratory for Neuro- & Psychophysiology, Department of Neurosciences, KU Leuven, Belgium
| | - Oliver Sturman
- Laboratory of Molecular and Behavioral Neuroscience, Institute for Neuroscience, Department of Health Sciences and Technology, ETH Zurich, Switzerland
- Neuroscience Center Zurich, ETH Zürich and University of Zürich, Switzerland
| | - Lukas von Ziegler
- Laboratory of Molecular and Behavioral Neuroscience, Institute for Neuroscience, Department of Health Sciences and Technology, ETH Zurich, Switzerland
- Neuroscience Center Zurich, ETH Zürich and University of Zürich, Switzerland
| | - Sepp Kollmorgen
- Institute of Neuroinformatics ETH Zürich and University of Zürich, Switzerland
- Neuroscience Center Zurich, ETH Zürich and University of Zürich, Switzerland
| | - Wolfger von der Behrens
- Institute of Neuroinformatics ETH Zürich and University of Zürich, Switzerland
- Neuroscience Center Zurich, ETH Zürich and University of Zürich, Switzerland
| | - Valerio Mante
- Institute of Neuroinformatics ETH Zürich and University of Zürich, Switzerland
- Neuroscience Center Zurich, ETH Zürich and University of Zürich, Switzerland
| | - Johannes Bohacek
- Laboratory of Molecular and Behavioral Neuroscience, Institute for Neuroscience, Department of Health Sciences and Technology, ETH Zurich, Switzerland
- Neuroscience Center Zurich, ETH Zürich and University of Zürich, Switzerland
| | - Mehmet Fatih Yanik
- Institute of Neuroinformatics ETH Zürich and University of Zürich, Switzerland
- Neuroscience Center Zurich, ETH Zürich and University of Zürich, Switzerland
| |
Collapse
|