1
|
Weinreb C, Pearl JE, Lin S, Osman MAM, Zhang L, Annapragada S, Conlin E, Hoffmann R, Makowska S, Gillis WF, Jay M, Ye S, Mathis A, Mathis MW, Pereira T, Linderman SW, Datta SR. Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics. Nat Methods 2024; 21:1329-1339. [PMID: 38997595 PMCID: PMC11245396 DOI: 10.1038/s41592-024-02318-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Accepted: 05/22/2024] [Indexed: 07/14/2024]
Abstract
Keypoint tracking algorithms can flexibly quantify animal movement from videos obtained in a wide variety of settings. However, it remains unclear how to parse continuous keypoint data into discrete actions. This challenge is particularly acute because keypoint data are susceptible to high-frequency jitter that clustering algorithms can mistake for transitions between actions. Here we present keypoint-MoSeq, a machine learning-based platform for identifying behavioral modules ('syllables') from keypoint data without human supervision. Keypoint-MoSeq uses a generative model to distinguish keypoint noise from behavior, enabling it to identify syllables whose boundaries correspond to natural sub-second discontinuities in pose dynamics. Keypoint-MoSeq outperforms commonly used alternative clustering methods at identifying these transitions, at capturing correlations between neural activity and behavior and at classifying either solitary or social behaviors in accordance with human annotations. Keypoint-MoSeq also works in multiple species and generalizes beyond the syllable timescale, identifying fast sniff-aligned movements in mice and a spectrum of oscillatory behaviors in fruit flies. Keypoint-MoSeq, therefore, renders accessible the modular structure of behavior through standard video recordings.
Collapse
Affiliation(s)
- Caleb Weinreb
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Jonah E Pearl
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Sherry Lin
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | | | - Libby Zhang
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
| | | | - Eli Conlin
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Red Hoffmann
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Sofia Makowska
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | | | - Maya Jay
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Shaokai Ye
- Brain Mind and Neuro-X Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Alexander Mathis
- Brain Mind and Neuro-X Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Mackenzie W Mathis
- Brain Mind and Neuro-X Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Talmo Pereira
- Salk Institute for Biological Studies, La Jolla, CA, USA
| | - Scott W Linderman
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA.
- Department of Statistics, Stanford University, Stanford, CA, USA.
| | | |
Collapse
|
2
|
Lin S, Gillis WF, Weinreb C, Zeine A, Jones SC, Robinson EM, Markowitz J, Datta SR. Characterizing the structure of mouse behavior using Motion Sequencing. Nat Protoc 2024:10.1038/s41596-024-01015-w. [PMID: 38926589 DOI: 10.1038/s41596-024-01015-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2022] [Accepted: 04/12/2024] [Indexed: 06/28/2024]
Abstract
Spontaneous mouse behavior is composed from repeatedly used modules of movement (e.g., rearing, running or grooming) that are flexibly placed into sequences whose content evolves over time. By identifying behavioral modules and the order in which they are expressed, researchers can gain insight into the effect of drugs, genes, context, sensory stimuli and neural activity on natural behavior. Here we present a protocol for performing Motion Sequencing (MoSeq), an ethologically inspired method that uses three-dimensional machine vision and unsupervised machine learning to decompose spontaneous mouse behavior into a series of elemental modules called 'syllables'. This protocol is based upon a MoSeq pipeline that includes modules for depth video acquisition, data preprocessing and modeling, as well as a standardized set of visualization tools. Users are provided with instructions and code for building a MoSeq imaging rig and acquiring three-dimensional video of spontaneous mouse behavior for submission to the modeling framework; the outputs of this protocol include syllable labels for each frame of the video data as well as summary plots describing how often each syllable was used and how syllables transitioned from one to the other. In addition, we provide instructions for analyzing and visualizing the outputs of keypoint-MoSeq, a recently developed variant of MoSeq that can identify behavioral motifs from keypoints identified from standard (rather than depth) video. This protocol and the accompanying pipeline significantly lower the bar for users without extensive computational ethology experience to adopt this unsupervised, data-driven approach to characterize mouse behavior.
Collapse
Affiliation(s)
- Sherry Lin
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | | | - Caleb Weinreb
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Ayman Zeine
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Samuel C Jones
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Emma M Robinson
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Jeffrey Markowitz
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
- Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA, USA
| | | |
Collapse
|
3
|
Keleş MF, Sapci AOB, Brody C, Palmer I, Le C, Taştan Ö, Keleş S, Wu MN. FlyVISTA, an Integrated Machine Learning Platform for Deep Phenotyping of Sleep in Drosophila. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.10.30.564733. [PMID: 37961473 PMCID: PMC10635029 DOI: 10.1101/2023.10.30.564733] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/15/2023]
Abstract
Animal behavior depends on internal state. While subtle movements can signify significant changes in internal state, computational methods for analyzing these "microbehaviors" are lacking. Here, we present FlyVISTA, a machine-learning platform to characterize microbehaviors in freely-moving flies, which we use to perform deep phenotyping of sleep. This platform comprises a high-resolution closed-loop video imaging system, coupled with a deep-learning network to annotate 35 body parts, and a computational pipeline to extract behaviors from high-dimensional data. FlyVISTA reveals the distinct spatiotemporal dynamics of sleep-associated microbehaviors in flies. We further show that stimulation of dorsal fan-shaped body neurons induces micromovements, not sleep, whereas activating R5 ring neurons triggers rhythmic proboscis extension followed by persistent sleep. Importantly, we identify a novel microbehavior ("haltere switch") exclusively seen during quiescence that indicates a deeper sleep stage. These findings enable the rigorous analysis of sleep in Drosophila and set the stage for computational analyses of microbehaviors.
Collapse
Affiliation(s)
- Mehmet F Keleş
- Department of Neurology, Johns Hopkins University, Baltimore, MD 21205, USA
| | - Ali Osman Berk Sapci
- Department of Computer Science, Sabanci University, Tuzla, Istanbul, 34956, Turkey
| | - Casey Brody
- Department of Neurology, Johns Hopkins University, Baltimore, MD 21205, USA
| | - Isabelle Palmer
- Department of Neurology, Johns Hopkins University, Baltimore, MD 21205, USA
| | - Christin Le
- Department of Neurology, Johns Hopkins University, Baltimore, MD 21205, USA
| | - Öznur Taştan
- Department of Computer Science, Sabanci University, Tuzla, Istanbul, 34956, Turkey
| | - Sündüz Keleş
- Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, WI 53706, USA
| | - Mark N Wu
- Department of Neurology, Johns Hopkins University, Baltimore, MD 21205, USA
- Department of Neuroscience, Johns Hopkins University, Baltimore, MD 21287, USA
| |
Collapse
|
4
|
Tillmann JF, Hsu AI, Schwarz MK, Yttri EA. A-SOiD, an active-learning platform for expert-guided, data-efficient discovery of behavior. Nat Methods 2024; 21:703-711. [PMID: 38383746 DOI: 10.1038/s41592-024-02200-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2022] [Accepted: 01/29/2024] [Indexed: 02/23/2024]
Abstract
To identify and extract naturalistic behavior, two methods have become popular: supervised and unsupervised. Each approach carries its own strengths and weaknesses (for example, user bias, training cost, complexity and action discovery), which the user must consider in their decision. Here, an active-learning platform, A-SOiD, blends these strengths, and in doing so, overcomes several of their inherent drawbacks. A-SOiD iteratively learns user-defined groups with a fraction of the usual training data, while attaining expansive classification through directed unsupervised classification. In socially interacting mice, A-SOiD outperformed standard methods despite requiring 85% less training data. Additionally, it isolated ethologically distinct mouse interactions via unsupervised classification. We observed similar performance and efficiency using nonhuman primate and human three-dimensional pose data. In both cases, the transparency in A-SOiD's cluster definitions revealed the defining features of the supervised classification through a game-theoretic approach. To facilitate use, A-SOiD comes as an intuitive, open-source interface for efficient segmentation of user-defined behaviors and discovered sub-actions.
Collapse
Affiliation(s)
- Jens F Tillmann
- Institute of Experimental Epileptology and Cognition Research, University of Bonn, Bonn, Germany
| | - Alexander I Hsu
- Department of Biological Sciences, Carnegie Mellon University, Pittsburgh, PA, USA
| | - Martin K Schwarz
- Institute of Experimental Epileptology and Cognition Research, University of Bonn, Bonn, Germany.
| | - Eric A Yttri
- Department of Biological Sciences, Carnegie Mellon University, Pittsburgh, PA, USA.
- Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, USA.
| |
Collapse
|
5
|
Weinreb C, Pearl J, Lin S, Osman MAM, Zhang L, Annapragada S, Conlin E, Hoffman R, Makowska S, Gillis WF, Jay M, Ye S, Mathis A, Mathis MW, Pereira T, Linderman SW, Datta SR. Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.03.16.532307. [PMID: 36993589 PMCID: PMC10055085 DOI: 10.1101/2023.03.16.532307] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/25/2023]
Abstract
Keypoint tracking algorithms have revolutionized the analysis of animal behavior, enabling investigators to flexibly quantify behavioral dynamics from conventional video recordings obtained in a wide variety of settings. However, it remains unclear how to parse continuous keypoint data into the modules out of which behavior is organized. This challenge is particularly acute because keypoint data is susceptible to high frequency jitter that clustering algorithms can mistake for transitions between behavioral modules. Here we present keypoint-MoSeq, a machine learning-based platform for identifying behavioral modules ("syllables") from keypoint data without human supervision. Keypoint-MoSeq uses a generative model to distinguish keypoint noise from behavior, enabling it to effectively identify syllables whose boundaries correspond to natural sub-second discontinuities inherent to mouse behavior. Keypoint-MoSeq outperforms commonly used alternative clustering methods at identifying these transitions, at capturing correlations between neural activity and behavior, and at classifying either solitary or social behaviors in accordance with human annotations. Keypoint-MoSeq therefore renders behavioral syllables and grammar accessible to the many researchers who use standard video to capture animal behavior.
Collapse
Affiliation(s)
- Caleb Weinreb
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Jonah Pearl
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Sherry Lin
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | | | - Libby Zhang
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
| | | | - Eli Conlin
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Red Hoffman
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Sofia Makowska
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | | | - Maya Jay
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Shaokai Ye
- Brain Mind and Neuro-X Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Alexander Mathis
- Brain Mind and Neuro-X Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Mackenzie Weygandt Mathis
- Brain Mind and Neuro-X Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Talmo Pereira
- Salk Institute for Biological Studies, La Jolla, USA
| | - Scott W. Linderman
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Department of Statistics, Stanford University, Stanford, CA, USA
| | | |
Collapse
|
6
|
Lapp HE, Salazar MG, Champagne FA. Automated maternal behavior during early life in rodents (AMBER) pipeline. Sci Rep 2023; 13:18277. [PMID: 37880307 PMCID: PMC10600172 DOI: 10.1038/s41598-023-45495-4] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2023] [Accepted: 10/20/2023] [Indexed: 10/27/2023] Open
Abstract
Mother-infant interactions during the early postnatal period are critical for infant survival and the scaffolding of infant development. Rodent models are used extensively to understand how these early social experiences influence neurobiology across the lifespan. However, methods for measuring postnatal dam-pup interactions typically involve time-consuming manual scoring, vary widely between research groups, and produce low density data that limits downstream analytical applications. To address these methodological issues, we developed the Automated Maternal Behavior during Early life in Rodents (AMBER) pipeline for quantifying home-cage maternal and mother-pup interactions using open-source machine learning tools. DeepLabCut was used to track key points on rat dams (32 points) and individual pups (9 points per pup) in postnatal day 1-10 video recordings. Pose estimation models reached key point test errors of approximately 4.1-10 mm (14.39 pixels) and 3.44-7.87 mm (11.81 pixels) depending on depth of animal in the frame averaged across all key points for dam and pups respectively. Pose estimation data and human-annotated behavior labels from 38 videos were used with Simple Behavioral Analysis (SimBA) to generate behavior classifiers for dam active nursing, passive nursing, nest attendance, licking and grooming, self-directed grooming, eating, and drinking using random forest algorithms. All classifiers had excellent performance on test frames, with F1 scores above 0.886. Performance on hold-out videos remained high for nest attendance (F1 = 0.990), active nursing (F1 = 0.828), and licking and grooming (F1 = 0.766) but was lower for eating, drinking, and self-directed grooming (F1 = 0.534-0.554). A set of 242 videos was used with AMBER and produced behavior measures in the expected range from postnatal 1-10 home-cage videos. This pipeline is a major advancement in assessing home-cage dam-pup interactions in a way that reduces experimenter burden while increasing reproducibility, reliability, and detail of data for use in developmental studies without the need for special housing systems or proprietary software.
Collapse
Affiliation(s)
- Hannah E Lapp
- Department of Psychology, University of Texas at Austin, 108 E. Dean Keaton St, Austin, TX, 78712, USA.
| | - Melissa G Salazar
- Department of Psychology, University of Texas at Austin, 108 E. Dean Keaton St, Austin, TX, 78712, USA
| | - Frances A Champagne
- Department of Psychology, University of Texas at Austin, 108 E. Dean Keaton St, Austin, TX, 78712, USA
| |
Collapse
|
7
|
Ehlman SM, Scherer U, Bierbach D, Francisco FA, Laskowski KL, Krause J, Wolf M. Leveraging big data to uncover the eco-evolutionary factors shaping behavioural development. Proc Biol Sci 2023; 290:20222115. [PMID: 36722081 PMCID: PMC9890127 DOI: 10.1098/rspb.2022.2115] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/02/2023] Open
Abstract
Mapping the eco-evolutionary factors shaping the development of animals' behavioural phenotypes remains a great challenge. Recent advances in 'big behavioural data' research-the high-resolution tracking of individuals and the harnessing of that data with powerful analytical tools-have vastly improved our ability to measure and model developing behavioural phenotypes. Applied to the study of behavioural ontogeny, the unfolding of whole behavioural repertoires can be mapped in unprecedented detail with relative ease. This overcomes long-standing experimental bottlenecks and heralds a surge of studies that more finely define and explore behavioural-experiential trajectories across development. In this review, we first provide a brief guide to state-of-the-art approaches that allow the collection and analysis of high-resolution behavioural data across development. We then outline how such approaches can be used to address key issues regarding the ecological and evolutionary factors shaping behavioural development: developmental feedbacks between behaviour and underlying states, early life effects and behavioural transitions, and information integration across development.
Collapse
Affiliation(s)
- Sean M. Ehlman
- SCIoI Excellence Cluster, 10587 Berlin, Germany,Faculty of Life Sciences, Humboldt University, 10117 Berlin, Germany,Department of Fish Biology, Fisheries, and Aquaculture, Leibniz Institute of Freshwater Ecology and Inland Fisheries, 12587 Berlin, Germany
| | - Ulrike Scherer
- SCIoI Excellence Cluster, 10587 Berlin, Germany,Faculty of Life Sciences, Humboldt University, 10117 Berlin, Germany,Department of Fish Biology, Fisheries, and Aquaculture, Leibniz Institute of Freshwater Ecology and Inland Fisheries, 12587 Berlin, Germany
| | - David Bierbach
- SCIoI Excellence Cluster, 10587 Berlin, Germany,Faculty of Life Sciences, Humboldt University, 10117 Berlin, Germany,Department of Fish Biology, Fisheries, and Aquaculture, Leibniz Institute of Freshwater Ecology and Inland Fisheries, 12587 Berlin, Germany
| | - Fritz A. Francisco
- SCIoI Excellence Cluster, 10587 Berlin, Germany,Faculty of Life Sciences, Humboldt University, 10117 Berlin, Germany
| | - Kate L. Laskowski
- Department of Evolution and Ecology, University of California – Davis, Davis, CA 95616, USA
| | - Jens Krause
- SCIoI Excellence Cluster, 10587 Berlin, Germany,Faculty of Life Sciences, Humboldt University, 10117 Berlin, Germany,Department of Fish Biology, Fisheries, and Aquaculture, Leibniz Institute of Freshwater Ecology and Inland Fisheries, 12587 Berlin, Germany
| | - Max Wolf
- SCIoI Excellence Cluster, 10587 Berlin, Germany,Department of Fish Biology, Fisheries, and Aquaculture, Leibniz Institute of Freshwater Ecology and Inland Fisheries, 12587 Berlin, Germany
| |
Collapse
|
8
|
Geng Y, Yates C, Peterson RT. Social behavioral profiling by unsupervised deep learning reveals a stimulative effect of dopamine D3 agonists on zebrafish sociality. CELL REPORTS METHODS 2023; 3:100381. [PMID: 36814839 PMCID: PMC9939379 DOI: 10.1016/j.crmeth.2022.100381] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/23/2022] [Revised: 11/15/2022] [Accepted: 12/12/2022] [Indexed: 01/07/2023]
Abstract
It has been a major challenge to systematically evaluate and compare how pharmacological perturbations influence social behavioral outcomes. Although some pharmacological agents are known to alter social behavior, precise description and quantification of such effects have proven difficult. We developed a scalable social behavioral assay for zebrafish named ZeChat based on unsupervised deep learning to characterize sociality at high resolution. High-dimensional and dynamic social behavioral phenotypes are automatically classified using this method. By screening a neuroactive compound library, we found that different classes of chemicals evoke distinct patterns of social behavioral fingerprints. By examining these patterns, we discovered that dopamine D3 agonists possess a social stimulative effect on zebrafish. The D3 agonists pramipexole, piribedil, and 7-hydroxy-DPAT-HBr rescued social deficits in a valproic-acid-induced zebrafish autism model. The ZeChat platform provides a promising approach for dissecting the pharmacology of social behavior and discovering novel social-modulatory compounds.
Collapse
Affiliation(s)
- Yijie Geng
- Department of Pharmacology and Toxicology, College of Pharmacy, University of Utah, Salt Lake City, UT 84112, USA
| | - Christopher Yates
- Department of Pharmacology and Toxicology, College of Pharmacy, University of Utah, Salt Lake City, UT 84112, USA
| | - Randall T. Peterson
- Department of Pharmacology and Toxicology, College of Pharmacy, University of Utah, Salt Lake City, UT 84112, USA
| |
Collapse
|
9
|
Fernández-López P, Garriga J, Casas I, Yeste M, Bartumeus F. Predicting fertility from sperm motility landscapes. Commun Biol 2022; 5:1027. [PMID: 36171267 PMCID: PMC9519750 DOI: 10.1038/s42003-022-03954-0] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2022] [Accepted: 09/06/2022] [Indexed: 11/23/2022] Open
Abstract
Understanding the organisational principles of sperm motility has both evolutionary and applied impact. The emergence of computer aided systems in this field came with the promise of automated quantification and classification, potentially improving our understanding of the determinants of reproductive success. Yet, nowadays the relationship between sperm variability and fertility remains unclear. Here, we characterize pig sperm motility using t-SNE, an embedding method adequate to study behavioural variability. T-SNE reveals a hierarchical organization of sperm motility across ejaculates and individuals, enabling accurate fertility predictions by means of Bayesian logistic regression. Our results show that sperm motility features, like high-speed and straight-lined motion, correlate positively with fertility and are more relevant than other sources of variability. We propose the combined use of embedding methods with Bayesian inference frameworks in order to achieve a better understanding of the relationship between fertility and sperm motility in animals, including humans. Dimension reduction methods on porcine sperm motility landscapes reveal heterogeneity and hierarchy in sperm movement behavior and show high-speed and straight-lined motion as predictive features of fertility.
Collapse
Affiliation(s)
- Pol Fernández-López
- Theoretical and Computational Ecology Group, Centre d'Estudis Avançats de Blanes (CEAB-CSIC), Cala Sant Francesc, 14, 17300, Blanes, Spain
| | - Joan Garriga
- Theoretical and Computational Ecology Group, Centre d'Estudis Avançats de Blanes (CEAB-CSIC), Cala Sant Francesc, 14, 17300, Blanes, Spain
| | - Isabel Casas
- Biotechnology of Animal and Human Reproduction (TechnoSperm), Institute of Food and Agricultural Technology, University of Girona, 17003, Girona, Spain.,Unit of Cell Biology, Department of Biology, Faculty of Sciences, University of Girona, 17003, Girona, Spain
| | - Marc Yeste
- Biotechnology of Animal and Human Reproduction (TechnoSperm), Institute of Food and Agricultural Technology, University of Girona, 17003, Girona, Spain.,Unit of Cell Biology, Department of Biology, Faculty of Sciences, University of Girona, 17003, Girona, Spain.,Institució Catalana de Recerca i Estudis Avançats, ICREA, Passeig Lluís Companys, 23, 08010, Barcelona, Spain
| | - Frederic Bartumeus
- Institució Catalana de Recerca i Estudis Avançats, ICREA, Passeig Lluís Companys, 23, 08010, Barcelona, Spain. .,Centre de Recerca Ecològica i Aplicacions Forestals (CREAF), Cerdanyola del Vallès, 08193, Barcelona, Spain.
| |
Collapse
|
10
|
Flavell SW, Gogolla N, Lovett-Barron M, Zelikowsky M. The emergence and influence of internal states. Neuron 2022; 110:2545-2570. [PMID: 35643077 PMCID: PMC9391310 DOI: 10.1016/j.neuron.2022.04.030] [Citation(s) in RCA: 56] [Impact Index Per Article: 28.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2021] [Revised: 02/11/2022] [Accepted: 04/27/2022] [Indexed: 01/09/2023]
Abstract
Animal behavior is shaped by a variety of "internal states"-partially hidden variables that profoundly shape perception, cognition, and action. The neural basis of internal states, such as fear, arousal, hunger, motivation, aggression, and many others, is a prominent focus of research efforts across animal phyla. Internal states can be inferred from changes in behavior, physiology, and neural dynamics and are characterized by properties such as pleiotropy, persistence, scalability, generalizability, and valence. To date, it remains unclear how internal states and their properties are generated by nervous systems. Here, we review recent progress, which has been driven by advances in behavioral quantification, cellular manipulations, and neural population recordings. We synthesize research implicating defined subsets of state-inducing cell types, widespread changes in neural activity, and neuromodulation in the formation and updating of internal states. In addition to highlighting the significance of these findings, our review advocates for new approaches to clarify the underpinnings of internal brain states across the animal kingdom.
Collapse
Affiliation(s)
- Steven W Flavell
- Picower Institute for Learning and Memory, Department of Brain and Cognitive Science, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.
| | - Nadine Gogolla
- Emotion Research Department, Max Planck Institute of Psychiatry, 80804 Munich, Germany; Circuits for Emotion Research Group, Max Planck Institute of Neurobiology, 82152 Martinsried, Germany.
| | - Matthew Lovett-Barron
- Division of Biological Sciences-Neurobiology Section, University of California, San Diego, La Jolla, CA 92093, USA.
| | - Moriel Zelikowsky
- Department of Neurobiology, University of Utah, Salt Lake City, UT 84112, USA.
| |
Collapse
|
11
|
de Bivort B, Buchanan S, Skutt-Kakaria K, Gajda E, Ayroles J, O’Leary C, Reimers P, Akhund-Zade J, Senft R, Maloney R, Ho S, Werkhoven Z, Smith MAY. Precise Quantification of Behavioral Individuality From 80 Million Decisions Across 183,000 Flies. Front Behav Neurosci 2022; 16:836626. [PMID: 35692381 PMCID: PMC9178272 DOI: 10.3389/fnbeh.2022.836626] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2021] [Accepted: 03/22/2022] [Indexed: 01/18/2023] Open
Abstract
Individual animals behave differently from each other. This variability is a component of personality and arises even when genetics and environment are held constant. Discovering the biological mechanisms underlying behavioral variability depends on efficiently measuring individual behavioral bias, a requirement that is facilitated by automated, high-throughput experiments. We compiled a large data set of individual locomotor behavior measures, acquired from over 183,000 fruit flies walking in Y-shaped mazes. With this data set we first conducted a "computational ethology natural history" study to quantify the distribution of individual behavioral biases with unprecedented precision and examine correlations between behavioral measures with high power. We discovered a slight, but highly significant, left-bias in spontaneous locomotor decision-making. We then used the data to evaluate standing hypotheses about biological mechanisms affecting behavioral variability, specifically: the neuromodulator serotonin and its precursor transporter, heterogametic sex, and temperature. We found a variety of significant effects associated with each of these mechanisms that were behavior-dependent. This indicates that the relationship between biological mechanisms and behavioral variability may be highly context dependent. Going forward, automation of behavioral experiments will likely be essential in teasing out the complex causality of individuality.
Collapse
|
12
|
Rihani K, Sachse S. Shedding Light on Inter-Individual Variability of Olfactory Circuits in Drosophila. Front Behav Neurosci 2022; 16:835680. [PMID: 35548690 PMCID: PMC9084309 DOI: 10.3389/fnbeh.2022.835680] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2021] [Accepted: 03/29/2022] [Indexed: 12/25/2022] Open
Abstract
Inter-individual differences in behavioral responses, anatomy or functional properties of neuronal populations of animals having the same genotype were for a long time disregarded. The majority of behavioral studies were conducted at a group level, and usually the mean behavior of all individuals was considered. Similarly, in neurophysiological studies, data were pooled and normalized from several individuals. This approach is mostly suited to map and characterize stereotyped neuronal properties between individuals, but lacks the ability to depict inter-individual variability regarding neuronal wiring or physiological characteristics. Recent studies have shown that behavioral biases and preferences to olfactory stimuli can vary significantly among individuals of the same genotype. The origin and the benefit of these diverse “personalities” is still unclear and needs to be further investigated. A perspective taken into account the inter-individual differences is needed to explore the cellular mechanisms underlying this phenomenon. This review focuses on olfaction in the vinegar fly Drosophila melanogaster and summarizes previous and recent studies on odor-guided behavior and the underlying olfactory circuits in the light of inter-individual variability. We address the morphological and physiological variabilities present at each layer of the olfactory circuitry and attempt to link them to individual olfactory behavior. Additionally, we discuss the factors that might influence individuality with regard to olfactory perception.
Collapse
Affiliation(s)
- Karen Rihani
- Research Group Olfactory Coding, Max Planck Institute for Chemical Ecology, Jena, Germany
- Max Planck Center Next Generation Insect Chemical Ecology, Jena, Germany
| | - Silke Sachse
- Research Group Olfactory Coding, Max Planck Institute for Chemical Ecology, Jena, Germany
- Max Planck Center Next Generation Insect Chemical Ecology, Jena, Germany
- *Correspondence: Silke Sachse,
| |
Collapse
|
13
|
Klein CJMI, Budiman T, Homberg JR, Verma D, Keijer J, van Schothorst EM. Measuring Locomotor Activity and Behavioral Aspects of Rodents Living in the Home-Cage. Front Behav Neurosci 2022; 16:877323. [PMID: 35464142 PMCID: PMC9021872 DOI: 10.3389/fnbeh.2022.877323] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2022] [Accepted: 03/14/2022] [Indexed: 11/13/2022] Open
Abstract
Automatization and technological advances have led to a larger number of methods and systems to monitor and measure locomotor activity and more specific behavior of a wide variety of animal species in various environmental conditions in laboratory settings. In rodents, the majority of these systems require the animals to be temporarily taken away from their home-cage into separate observation cage environments which requires manual handling and consequently evokes distress for the animal and may alter behavioral responses. An automated high-throughput approach can overcome this problem. Therefore, this review describes existing automated methods and technologies which enable the measurement of locomotor activity and behavioral aspects of rodents in their most meaningful and stress-free laboratory environment: the home-cage. In line with the Directive 2010/63/EU and the 3R principles (replacement, reduction, refinement), this review furthermore assesses their suitability and potential for group-housed conditions as a refinement strategy, highlighting their current technological and practical limitations. It covers electrical capacitance technology and radio-frequency identification (RFID), which focus mainly on voluntary locomotor activity in both single and multiple rodents, respectively. Infrared beams and force plates expand the detection beyond locomotor activity toward basic behavioral traits but discover their full potential in individually housed rodents only. Despite the great premises of these approaches in terms of behavioral pattern recognition, more sophisticated methods, such as (RFID-assisted) video tracking technology need to be applied to enable the automated analysis of advanced behavioral aspects of individual animals in social housing conditions.
Collapse
Affiliation(s)
- Christian J. M. I. Klein
- Human and Animal Physiology, Wageningen University and Research, Wageningen, Netherlands
- TSE Systems GmbH, Berlin, Germany
| | | | - Judith R. Homberg
- Department of Cognitive Neuroscience, Donders Institute for Brain, Cognition and Behavior, Radboud University Medical Center, Nijmegen, Netherlands
| | | | - Jaap Keijer
- Human and Animal Physiology, Wageningen University and Research, Wageningen, Netherlands
| | | |
Collapse
|
14
|
Smith MAY, Honegger KS, Turner G, de Bivort B. Idiosyncratic learning performance in flies. Biol Lett 2022; 18:20210424. [PMID: 35104427 PMCID: PMC8807056 DOI: 10.1098/rsbl.2021.0424] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2021] [Accepted: 12/21/2021] [Indexed: 11/18/2022] Open
Abstract
Individuals vary in their innate behaviours, even when they have the same genome and have been reared in the same environment. The extent of individuality in plastic behaviours, like learning, is less well characterized. Also unknown is the extent to which intragenotypic differences in learning generalize: if an individual performs well in one assay, will it perform well in other assays? We investigated this using the fruit fly Drosophila melanogaster, an organism long-used to study the mechanistic basis of learning and memory. We found that isogenic flies, reared in identical laboratory conditions, and subject to classical conditioning that associated odorants with electric shock, exhibit clear individuality in their learning responses. Flies that performed well when an odour was paired with shock tended to perform well when the odour was paired with bitter taste or when other odours were paired with shock. Thus, individuality in learning performance appears to be prominent in isogenic animals reared identically, and individual differences in learning performance generalize across some aversive sensory modalities. Establishing these results in flies opens up the possibility of studying the genetic and neural circuit basis of individual differences in learning in a highly suitable model organism.
Collapse
Affiliation(s)
- Matthew A.-Y. Smith
- Department of Organismic and Evolutionary Biology and Center for Brain Science, Harvard University, Cambridge, MA 02138, USA
| | - Kyle S. Honegger
- Ann and Robert H. Lurie Children's Hospital of Chicago, Chicago, IL 60611, USA
| | - Glenn Turner
- Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA
| | - Benjamin de Bivort
- Department of Organismic and Evolutionary Biology and Center for Brain Science, Harvard University, Cambridge, MA 02138, USA
| |
Collapse
|
15
|
Werkhoven Z, Bravin A, Skutt-Kakaria K, Reimers P, Pallares LF, Ayroles J, de Bivort BL. The structure of behavioral variation within a genotype. eLife 2021; 10:64988. [PMID: 34664550 PMCID: PMC8526060 DOI: 10.7554/elife.64988] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2020] [Accepted: 09/14/2021] [Indexed: 12/13/2022] Open
Abstract
Individual animals vary in their behaviors. This is true even when they share the same genotype and were reared in the same environment. Clusters of covarying behaviors constitute behavioral syndromes, and an individual’s position along such axes of covariation is a representation of their personality. Despite these conceptual frameworks, the structure of behavioral covariation within a genotype is essentially uncharacterized and its mechanistic origins unknown. Passing hundreds of inbred Drosophila individuals through an experimental pipeline that captured hundreds of behavioral measures, we found sparse but significant correlations among small sets of behaviors. Thus, the space of behavioral variation has many independent dimensions. Manipulating the physiology of the brain, and specific neural populations, altered specific correlations. We also observed that variation in gene expression can predict an individual’s position on some behavioral axes. This work represents the first steps in understanding the biological mechanisms determining the structure of behavioral variation within a genotype.
Collapse
Affiliation(s)
- Zachary Werkhoven
- Center for Brain Science and Department of Organismic and Evolutionary Biology, Cambridge, United States
| | - Alyssa Bravin
- Center for Brain Science and Department of Organismic and Evolutionary Biology, Cambridge, United States
| | - Kyobi Skutt-Kakaria
- Center for Brain Science and Department of Organismic and Evolutionary Biology, Cambridge, United States.,Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, United States
| | - Pablo Reimers
- Center for Brain Science and Department of Organismic and Evolutionary Biology, Cambridge, United States.,Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, United States
| | - Luisa F Pallares
- Department of Ecology and Evolutionary Biology and Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, United States
| | - Julien Ayroles
- Center for Brain Science and Department of Organismic and Evolutionary Biology, Cambridge, United States.,Department of Ecology and Evolutionary Biology and Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, United States
| | - Benjamin L de Bivort
- Center for Brain Science and Department of Organismic and Evolutionary Biology, Cambridge, United States
| |
Collapse
|
16
|
Janisch J, Mitoyen C, Perinot E, Spezie G, Fusani L, Quigley C. Video Recording and Analysis of Avian Movements and Behavior: Insights from Courtship Case Studies. Integr Comp Biol 2021; 61:1378-1393. [PMID: 34037219 PMCID: PMC8516111 DOI: 10.1093/icb/icab095] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022] Open
Abstract
Video recordings are useful tools for advancing our understanding of animal movements and behavior. Over the past decades, a burgeoning area of behavioral research has put forward innovative methods to investigate animal movement using video analysis, which includes motion capture and machine learning algorithms. These tools are particularly valuable for the study of elaborate and complex motor behaviors, but can be challenging to use. We focus in particular on elaborate courtship displays, which commonly involve rapid and/or subtle motor patterns. Here, we review currently available tools and provide hands-on guidelines for implementing these techniques in the study of avian model species. First, we suggest a set of possible strategies and solutions for video acquisition based on different model systems, environmental conditions, and time or financial budget. We then outline the available options for video analysis and illustrate how different analytical tools can be chosen to draw inference about animal motor performance. Finally, a detailed case study describes how these guidelines have been implemented to study courtship behavior in golden-collared manakins (Manacus vitellinus).
Collapse
Affiliation(s)
- Judith Janisch
- Konrad Lorenz Institute of Ethology, University of Veterinary Medicine, 1160 Vienna, Austria
| | - Clementine Mitoyen
- Department of Cognitive Biology, University of Vienna, 1090 Vienna, Austria
| | - Elisa Perinot
- Konrad Lorenz Institute of Ethology, University of Veterinary Medicine, 1160 Vienna, Austria
| | - Giovanni Spezie
- Konrad Lorenz Institute of Ethology, University of Veterinary Medicine, 1160 Vienna, Austria
| | - Leonida Fusani
- Konrad Lorenz Institute of Ethology, University of Veterinary Medicine, 1160 Vienna, Austria
- Department of Cognitive Biology, University of Vienna, 1090 Vienna, Austria
| | - Cliodhna Quigley
- Konrad Lorenz Institute of Ethology, University of Veterinary Medicine, 1160 Vienna, Austria
- Department of Cognitive Biology, University of Vienna, 1090 Vienna, Austria
- Vienna Cognitive Science Hub, University of Vienna, 1090 Vienna, Austria
| |
Collapse
|
17
|
Hsu AI, Yttri EA. B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviors. Nat Commun 2021; 12:5188. [PMID: 34465784 PMCID: PMC8408193 DOI: 10.1038/s41467-021-25420-x] [Citation(s) in RCA: 89] [Impact Index Per Article: 29.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2020] [Accepted: 08/10/2021] [Indexed: 11/13/2022] Open
Abstract
Studying naturalistic animal behavior remains a difficult objective. Recent machine learning advances have enabled limb localization; however, extracting behaviors requires ascertaining the spatiotemporal patterns of these positions. To provide a link from poses to actions and their kinematics, we developed B-SOiD - an open-source, unsupervised algorithm that identifies behavior without user bias. By training a machine classifier on pose pattern statistics clustered using new methods, our approach achieves greatly improved processing speed and the ability to generalize across subjects or labs. Using a frameshift alignment paradigm, B-SOiD overcomes previous temporal resolution barriers. Using only a single, off-the-shelf camera, B-SOiD provides categories of sub-action for trained behaviors and kinematic measures of individual limb trajectories in any animal model. These behavioral and kinematic measures are difficult but critical to obtain, particularly in the study of rodent and other models of pain, OCD, and movement disorders.
Collapse
Affiliation(s)
- Alexander I Hsu
- Department of Biological Sciences, Carnegie Mellon University, Pittsburgh, PA, USA
| | - Eric A Yttri
- Department of Biological Sciences, Carnegie Mellon University, Pittsburgh, PA, USA.
- Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, USA.
| |
Collapse
|
18
|
Geuther BQ, Peer A, He H, Sabnis G, Philip VM, Kumar V. Action detection using a neural network elucidates the genetics of mouse grooming behavior. eLife 2021; 10:e63207. [PMID: 33729153 PMCID: PMC8043749 DOI: 10.7554/elife.63207] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2020] [Accepted: 03/05/2021] [Indexed: 12/25/2022] Open
Abstract
Automated detection of complex animal behaviors remains a challenging problem in neuroscience, particularly for behaviors that consist of disparate sequential motions. Grooming is a prototypical stereotyped behavior that is often used as an endophenotype in psychiatric genetics. Here, we used mouse grooming behavior as an example and developed a general purpose neural network architecture capable of dynamic action detection at human observer-level performance and operating across dozens of mouse strains with high visual diversity. We provide insights into the amount of human annotated training data that are needed to achieve such performance. We surveyed grooming behavior in the open field in 2457 mice across 62 strains, determined its heritable components, conducted GWAS to outline its genetic architecture, and performed PheWAS to link human psychiatric traits through shared underlying genetics. Our general machine learning solution that automatically classifies complex behaviors in large datasets will facilitate systematic studies of behavioral mechanisms.
Collapse
Affiliation(s)
| | - Asaf Peer
- The Jackson LaboratoryBar HarborUnited States
| | - Hao He
- The Jackson LaboratoryBar HarborUnited States
| | | | | | - Vivek Kumar
- The Jackson LaboratoryBar HarborUnited States
| |
Collapse
|
19
|
Marshall JD, Aldarondo DE, Dunn TW, Wang WL, Berman GJ, Ölveczky BP. Continuous Whole-Body 3D Kinematic Recordings across the Rodent Behavioral Repertoire. Neuron 2021; 109:420-437.e8. [PMID: 33340448 PMCID: PMC7864892 DOI: 10.1016/j.neuron.2020.11.016] [Citation(s) in RCA: 58] [Impact Index Per Article: 19.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2020] [Revised: 10/01/2020] [Accepted: 11/16/2020] [Indexed: 12/13/2022]
Abstract
In mammalian animal models, high-resolution kinematic tracking is restricted to brief sessions in constrained environments, limiting our ability to probe naturalistic behaviors and their neural underpinnings. To address this, we developed CAPTURE (Continuous Appendicular and Postural Tracking Using Retroreflector Embedding), a behavioral monitoring system that combines motion capture and deep learning to continuously track the 3D kinematics of a rat's head, trunk, and limbs for week-long timescales in freely behaving animals. CAPTURE realizes 10- to 100-fold gains in precision and robustness compared with existing convolutional network approaches to behavioral tracking. We demonstrate CAPTURE's ability to comprehensively profile the kinematics and sequential organization of natural rodent behavior, its variation across individuals, and its perturbation by drugs and disease, including identifying perseverative grooming states in a rat model of fragile X syndrome. CAPTURE significantly expands the range of behaviors and contexts that can be quantitatively investigated, opening the door to a new understanding of natural behavior and its neural basis.
Collapse
Affiliation(s)
- Jesse D Marshall
- Department of Organismic and Evolutionary Biology, Harvard University, Cambridge, MA 02138, USA.
| | - Diego E Aldarondo
- Program in Neuroscience, Harvard University, Cambridge, MA 02138, USA
| | - Timothy W Dunn
- Department of Statistical Science, Duke University, Durham, NC 27710, USA
| | - William L Wang
- Department of Organismic and Evolutionary Biology, Harvard University, Cambridge, MA 02138, USA
| | - Gordon J Berman
- Department of Biology, Emory University, Atlanta, GA 30322, USA
| | - Bence P Ölveczky
- Department of Organismic and Evolutionary Biology, Harvard University, Cambridge, MA 02138, USA.
| |
Collapse
|
20
|
Pereira TD, Shaevitz JW, Murthy M. Quantifying behavior to understand the brain. Nat Neurosci 2020; 23:1537-1549. [PMID: 33169033 PMCID: PMC7780298 DOI: 10.1038/s41593-020-00734-z] [Citation(s) in RCA: 110] [Impact Index Per Article: 27.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2020] [Accepted: 10/02/2020] [Indexed: 02/07/2023]
Abstract
Over the past years, numerous methods have emerged to automate the quantification of animal behavior at a resolution not previously imaginable. This has opened up a new field of computational ethology and will, in the near future, make it possible to quantify in near completeness what an animal is doing as it navigates its environment. The importance of improving the techniques with which we characterize behavior is reflected in the emerging recognition that understanding behavior is an essential (or even prerequisite) step to pursuing neuroscience questions. The use of these methods, however, is not limited to studying behavior in the wild or in strictly ethological settings. Modern tools for behavioral quantification can be applied to the full gamut of approaches that have historically been used to link brain to behavior, from psychophysics to cognitive tasks, augmenting those measurements with rich descriptions of how animals navigate those tasks. Here we review recent technical advances in quantifying behavior, particularly in methods for tracking animal motion and characterizing the structure of those dynamics. We discuss open challenges that remain for behavioral quantification and highlight promising future directions, with a strong emphasis on emerging approaches in deep learning, the core technology that has enabled the markedly rapid pace of progress of this field. We then discuss how quantitative descriptions of behavior can be leveraged to connect brain activity with animal movements, with the ultimate goal of resolving the relationship between neural circuits, cognitive processes and behavior.
Collapse
Affiliation(s)
- Talmo D Pereira
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | - Joshua W Shaevitz
- Department of Physics, Princeton University, Princeton, NJ, USA
- Lewis-Sigler Institute, Princeton University, Princeton, NJ, USA
| | - Mala Murthy
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
| |
Collapse
|
21
|
Gal A, Saragosti J, Kronauer DJC. anTraX, a software package for high-throughput video tracking of color-tagged insects. eLife 2020; 9:e58145. [PMID: 33211008 PMCID: PMC7676868 DOI: 10.7554/elife.58145] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2020] [Accepted: 10/29/2020] [Indexed: 12/14/2022] Open
Abstract
Recent years have seen a surge in methods to track and analyze animal behavior. Nevertheless, tracking individuals in closely interacting, group-living organisms remains a challenge. Here, we present anTraX, an algorithm and software package for high-throughput video tracking of color-tagged insects. anTraX combines neural network classification of animals with a novel approach for representing tracking data as a graph, enabling individual tracking even in cases where it is difficult to segment animals from one another, or where tags are obscured. The use of color tags, a well-established and robust method for marking individual insects in groups, relaxes requirements for image size and quality, and makes the software broadly applicable. anTraX is readily integrated into existing tools and methods for automated image analysis of behavior to further augment its output. anTraX can handle large-scale experiments with minimal human involvement, allowing researchers to simultaneously monitor many social groups over long time periods.
Collapse
Affiliation(s)
- Asaf Gal
- Laboratory of Social Evolution and Behavior, The Rockefeller UniversityNew YorkUnited States
| | - Jonathan Saragosti
- Laboratory of Social Evolution and Behavior, The Rockefeller UniversityNew YorkUnited States
| | - Daniel JC Kronauer
- Laboratory of Social Evolution and Behavior, The Rockefeller UniversityNew YorkUnited States
| |
Collapse
|
22
|
Seidenbecher SE, Sanders JI, von Philipsborn AC, Kvitsiani D. Reward foraging task and model-based analysis reveal how fruit flies learn value of available options. PLoS One 2020; 15:e0239616. [PMID: 33007023 PMCID: PMC7531776 DOI: 10.1371/journal.pone.0239616] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2020] [Accepted: 09/10/2020] [Indexed: 11/18/2022] Open
Abstract
Foraging animals have to evaluate, compare and select food patches in order to increase their fitness. Understanding what drives foraging decisions requires careful manipulation of the value of alternative options while monitoring animals choices. Value-based decision-making tasks in combination with formal learning models have provided both an experimental and theoretical framework to study foraging decisions in lab settings. While these approaches were successfully used in the past to understand what drives choices in mammals, very little work has been done on fruit flies. This is despite the fact that fruit flies have served as model organism for many complex behavioural paradigms. To fill this gap we developed a single-animal, trial-based decision making task, where freely walking flies experienced optogenetic sugar-receptor neuron stimulation. We controlled the value of available options by manipulating the probabilities of optogenetic stimulation. We show that flies integrate reward history of chosen options and forget value of unchosen options. We further discover that flies assign higher values to rewards experienced early in the behavioural session, consistent with formal reinforcement learning models. Finally, we also show that the probabilistic rewards affect walking trajectories of flies, suggesting that accumulated value is controlling the navigation vector of flies in a graded fashion. These findings establish the fruit fly as a model organism to explore the genetic and circuit basis of reward foraging decisions.
Collapse
Affiliation(s)
- Sophie E Seidenbecher
- Danish Research Institute of Translational Neuroscience - DANDRITE, Nordic-EMBL Partnership for Molecular Medicine, Aarhus, Denmark.,Department of Molecular Biology and Genetics, Aarhus University, Aarhus, Denmark
| | - Joshua I Sanders
- Danish Research Institute of Translational Neuroscience - DANDRITE, Nordic-EMBL Partnership for Molecular Medicine, Aarhus, Denmark.,Department of Molecular Biology and Genetics, Aarhus University, Aarhus, Denmark
| | - Anne C von Philipsborn
- Danish Research Institute of Translational Neuroscience - DANDRITE, Nordic-EMBL Partnership for Molecular Medicine, Aarhus, Denmark.,Department of Molecular Biology and Genetics, Aarhus University, Aarhus, Denmark
| | - Duda Kvitsiani
- Danish Research Institute of Translational Neuroscience - DANDRITE, Nordic-EMBL Partnership for Molecular Medicine, Aarhus, Denmark.,Department of Molecular Biology and Genetics, Aarhus University, Aarhus, Denmark
| |
Collapse
|
23
|
Honegger KS, Smith MAY, Churgin MA, Turner GC, de Bivort BL. Idiosyncratic neural coding and neuromodulation of olfactory individuality in Drosophila. Proc Natl Acad Sci U S A 2020; 117:23292-23297. [PMID: 31455738 PMCID: PMC7519279 DOI: 10.1073/pnas.1901623116] [Citation(s) in RCA: 40] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022] Open
Abstract
Innate behavioral biases and preferences can vary significantly among individuals of the same genotype. Though individuality is a fundamental property of behavior, it is not currently understood how individual differences in brain structure and physiology produce idiosyncratic behaviors. Here we present evidence for idiosyncrasy in olfactory behavior and neural responses in Drosophila We show that individual female Drosophila from a highly inbred laboratory strain exhibit idiosyncratic odor preferences that persist for days. We used in vivo calcium imaging of neural responses to compare projection neuron (second-order neurons that convey odor information from the sensory periphery to the central brain) responses to the same odors across animals. We found that, while odor responses appear grossly stereotyped, upon closer inspection, many individual differences are apparent across antennal lobe (AL) glomeruli (compact microcircuits corresponding to different odor channels). Moreover, we show that neuromodulation, environmental stress in the form of altered nutrition, and activity of certain AL local interneurons affect the magnitude of interfly behavioral variability. Taken together, this work demonstrates that individual Drosophila exhibit idiosyncratic olfactory preferences and idiosyncratic neural responses to odors, and that behavioral idiosyncrasies are subject to neuromodulation and regulation by neurons in the AL.
Collapse
Affiliation(s)
- Kyle S Honegger
- Department of Organismic and Evolutionary Biology and Center for Brain Science, Harvard University, Cambridge, MA 02138
- Computational Informatics and Visualization Laboratory, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL 60611
| | - Matthew A-Y Smith
- Department of Organismic and Evolutionary Biology and Center for Brain Science, Harvard University, Cambridge, MA 02138
| | - Matthew A Churgin
- Department of Organismic and Evolutionary Biology and Center for Brain Science, Harvard University, Cambridge, MA 02138
| | - Glenn C Turner
- Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147
| | - Benjamin L de Bivort
- Department of Organismic and Evolutionary Biology and Center for Brain Science, Harvard University, Cambridge, MA 02138;
| |
Collapse
|
24
|
Goodwin NL, Nilsson SRO, Golden SA. Rage Against the Machine: Advancing the study of aggression ethology via machine learning. Psychopharmacology (Berl) 2020; 237:2569-2588. [PMID: 32647898 PMCID: PMC7502501 DOI: 10.1007/s00213-020-05577-x] [Citation(s) in RCA: 20] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/07/2019] [Accepted: 06/01/2020] [Indexed: 12/24/2022]
Abstract
RATIONALE Aggression, comorbid with neuropsychiatric disorders, exhibits with diverse clinical presentations and places a significant burden on patients, caregivers, and society. This diversity is observed because aggression is a complex behavior that can be ethologically demarcated as either appetitive (rewarding) or reactive (defensive), each with its own behavioral characteristics, functionality, and neural basis that may transition from adaptive to maladaptive depending on genetic and environmental factors. There has been a recent surge in the development of preclinical animal models for studying appetitive aggression-related behaviors and identifying the neural mechanisms guiding their progression and expression. However, adoption of these procedures is often impeded by the arduous task of manually scoring complex social interactions. Manual observations are generally susceptible to observer drift, long analysis times, and poor inter-rater reliability, and are further incompatible with the sampling frequencies required of modern neuroscience methods. OBJECTIVES In this review, we discuss recent advances in the preclinical study of appetitive aggression in mice, paired with our perspective on the potential for machine learning techniques in producing automated, robust scoring of aggressive social behavior. We discuss critical considerations for implementing valid computer classifications within behavioral pharmacological studies. KEY RESULTS Open-source automated classification platforms can match or exceed the performance of human observers while removing the confounds of observer drift, bias, and inter-rater reliability. Furthermore, unsupervised approaches can identify previously uncharacterized aggression-related behavioral repertoires in model species. DISCUSSION AND CONCLUSIONS Advances in open-source computational approaches hold promise for overcoming current manual annotation caveats while also introducing and generalizing computational neuroethology to the greater behavioral neuroscience community. We propose that currently available open-source approaches are sufficient for overcoming the main limitations preventing wide adoption of machine learning within the context of preclinical aggression behavioral research.
Collapse
Affiliation(s)
- Nastacia L Goodwin
- Department of Biological Structure, University of Washington, Seattle, WA, USA
- Graduate Program in Neuroscience, University of Washington, Seattle, WA, USA
| | - Simon R O Nilsson
- Department of Biological Structure, University of Washington, Seattle, WA, USA
| | - Sam A Golden
- Department of Biological Structure, University of Washington, Seattle, WA, USA.
- Graduate Program in Neuroscience, University of Washington, Seattle, WA, USA.
- Center of Excellence in Neurobiology of Addiction, Pain, and Emotion (NAPE), University of Washington, Seattle, WA, USA.
| |
Collapse
|
25
|
Hierarchical Compression Reveals Sub-Second to Day-Long Structure in Larval Zebrafish Behavior. eNeuro 2020; 7:ENEURO.0408-19.2020. [PMID: 32241874 PMCID: PMC7405074 DOI: 10.1523/eneuro.0408-19.2020] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2019] [Revised: 01/22/2020] [Accepted: 02/26/2020] [Indexed: 12/11/2022] Open
Abstract
Animal behavior is dynamic, evolving over multiple timescales from milliseconds to days and even across a lifetime. To understand the mechanisms governing these dynamics, it is necessary to capture multi-timescale structure from behavioral data. Here, we develop computational tools and study the behavior of hundreds of larval zebrafish tracked continuously across multiple 24-h day/night cycles. We extracted millions of movements and pauses, termed bouts, and used unsupervised learning to reduce each larva’s behavior to an alternating sequence of active and inactive bout types, termed modules. Through hierarchical compression, we identified recurrent behavioral patterns, termed motifs. Module and motif usage varied across the day/night cycle, revealing structure at sub-second to day-long timescales. We further demonstrate that module and motif analysis can uncover novel pharmacological and genetic mutant phenotypes. Overall, our work reveals the organization of larval zebrafish behavior at multiple timescales and provides tools to identify structure from large-scale behavioral datasets.
Collapse
|
26
|
Francisco FA, Nührenberg P, Jordan A. High-resolution, non-invasive animal tracking and reconstruction of local environment in aquatic ecosystems. MOVEMENT ECOLOGY 2020; 8:27. [PMID: 32582448 PMCID: PMC7310323 DOI: 10.1186/s40462-020-00214-w] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/26/2020] [Accepted: 05/26/2020] [Indexed: 05/19/2023]
Abstract
BACKGROUND Acquiring high resolution quantitative behavioural data underwater often involves installation of costly infrastructure, or capture and manipulation of animals. Aquatic movement ecology can therefore be limited in taxonomic range and ecological coverage. METHODS Here we present a novel deep-learning based, multi-individual tracking approach, which incorporates Structure-from-Motion in order to determine the 3D location, body position and the visual environment of every recorded individual. The application is based on low-cost cameras and does not require the animals to be confined, manipulated, or handled in any way. RESULTS Using this approach, single individuals, small heterospecific groups and schools of fish were tracked in freshwater and marine environments of varying complexity. Positional tracking errors as low as 1.09 ± 0.47 cm (RSME) in underwater areas up to 500 m2 were recorded. CONCLUSIONS This cost-effective and open-source framework allows the analysis of animal behaviour in aquatic systems at an unprecedented resolution. Implementing this versatile approach, quantitative behavioural analysis can be employed in a wide range of natural contexts, vastly expanding our potential for examining non-model systems and species.
Collapse
Affiliation(s)
- Fritz A Francisco
- Centre for the Advanced Study of Collective Behaviour, University of Konstanz, Universitätsstraße 10, Konstanz, 78457 Germany
- Department of Collective Behavior, Max Planck Institute of Animal Behavior, Universitätsstraße 10, Konstanz, 78457 Germany
- Department of Biology, University of Konstanz, Universitätsstraße 10, Konstanz, 78457 Germany
| | - Paul Nührenberg
- Centre for the Advanced Study of Collective Behaviour, University of Konstanz, Universitätsstraße 10, Konstanz, 78457 Germany
- Department of Collective Behavior, Max Planck Institute of Animal Behavior, Universitätsstraße 10, Konstanz, 78457 Germany
- Department of Biology, University of Konstanz, Universitätsstraße 10, Konstanz, 78457 Germany
| | - Alex Jordan
- Centre for the Advanced Study of Collective Behaviour, University of Konstanz, Universitätsstraße 10, Konstanz, 78457 Germany
- Department of Collective Behavior, Max Planck Institute of Animal Behavior, Universitätsstraße 10, Konstanz, 78457 Germany
- Department of Biology, University of Konstanz, Universitätsstraße 10, Konstanz, 78457 Germany
| |
Collapse
|
27
|
Johnson RE, Linderman S, Panier T, Wee CL, Song E, Herrera KJ, Miller A, Engert F. Probabilistic Models of Larval Zebrafish Behavior Reveal Structure on Many Scales. Curr Biol 2020; 30:70-82.e4. [PMID: 31866367 PMCID: PMC6958995 DOI: 10.1016/j.cub.2019.11.026] [Citation(s) in RCA: 60] [Impact Index Per Article: 15.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2019] [Revised: 09/11/2019] [Accepted: 11/07/2019] [Indexed: 12/12/2022]
Abstract
Nervous systems have evolved to combine environmental information with internal state to select and generate adaptive behavioral sequences. To better understand these computations and their implementation in neural circuits, natural behavior must be carefully measured and quantified. Here, we collect high spatial resolution video of single zebrafish larvae swimming in a naturalistic environment and develop models of their action selection across exploration and hunting. Zebrafish larvae swim in punctuated bouts separated by longer periods of rest called interbout intervals. We take advantage of this structure by categorizing bouts into discrete types and representing their behavior as labeled sequences of bout types emitted over time. We then construct probabilistic models-specifically, marked renewal processes-to evaluate how bout types and interbout intervals are selected by the fish as a function of its internal hunger state, behavioral history, and the locations and properties of nearby prey. Finally, we evaluate the models by their predictive likelihood and their ability to generate realistic trajectories of virtual fish swimming through simulated environments. Our simulations capture multiple timescales of structure in larval zebrafish behavior and expose many ways in which hunger state influences their action selection to promote food seeking during hunger and safety during satiety.
Collapse
Affiliation(s)
- Robert Evan Johnson
- Department of Molecular and Cellular Biology, Harvard University, 16 Divinity Avenue, Cambridge, MA 02138, USA; Graduate Program in Neuroscience, Harvard University, 220 Longwood Avenue, Boston, MA 02115, USA.
| | - Scott Linderman
- Department of Statistics, Stanford University, 390 Serra Mall, Stanford, CA 94305, USA; Wu Tsai Neurosciences Institute, Stanford University, 318 Campus Drive, Stanford, CA 94305, USA
| | - Thomas Panier
- Department of Molecular and Cellular Biology, Harvard University, 16 Divinity Avenue, Cambridge, MA 02138, USA; Sorbonne Université, CNRS, Institut de Biologie Paris-Seine, Laboratoire Jean Perrin, 4 Place Jussieu, 75005 Paris, France
| | - Caroline Lei Wee
- Department of Molecular and Cellular Biology, Harvard University, 16 Divinity Avenue, Cambridge, MA 02138, USA; Graduate Program in Neuroscience, Harvard University, 220 Longwood Avenue, Boston, MA 02115, USA; Institute of Molecular and Cell Biology, A(∗)STAR, 61 Biopolis Drive, 138673 Singapore, Singapore
| | - Erin Song
- Department of Molecular and Cellular Biology, Harvard University, 16 Divinity Avenue, Cambridge, MA 02138, USA
| | - Kristian Joseph Herrera
- Department of Molecular and Cellular Biology, Harvard University, 16 Divinity Avenue, Cambridge, MA 02138, USA
| | - Andrew Miller
- Data Science Institute, Columbia University, 550 W 120th Street, New York City, NY 10027, USA
| | - Florian Engert
- Department of Molecular and Cellular Biology, Harvard University, 16 Divinity Avenue, Cambridge, MA 02138, USA
| |
Collapse
|
28
|
Werkhoven Z, Rohrsen C, Qin C, Brembs B, de Bivort B. MARGO (Massively Automated Real-time GUI for Object-tracking), a platform for high-throughput ethology. PLoS One 2019; 14:e0224243. [PMID: 31765421 PMCID: PMC6876843 DOI: 10.1371/journal.pone.0224243] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2019] [Accepted: 10/08/2019] [Indexed: 12/20/2022] Open
Abstract
Fast object tracking in real time allows convenient tracking of very large numbers of animals and closed-loop experiments that control stimuli for many animals in parallel. We developed MARGO, a MATLAB-based, real-time animal tracking suite for custom behavioral experiments. We demonstrated that MARGO can rapidly and accurately track large numbers of animals in parallel over very long timescales, typically when spatially separated such as in multiwell plates. We incorporated control of peripheral hardware, and implemented a flexible software architecture for defining new experimental routines. These features enable closed-loop delivery of stimuli to many individuals simultaneously. We highlight MARGO's ability to coordinate tracking and hardware control with two custom behavioral assays (measuring phototaxis and optomotor response) and one optogenetic operant conditioning assay. There are currently several open source animal trackers. MARGO's strengths are 1) fast and accurate tracking, 2) high throughput, 3) an accessible interface and data output and 4) real-time closed-loop hardware control for for sensory and optogenetic stimuli, all of which are optimized for large-scale experiments.
Collapse
Affiliation(s)
- Zach Werkhoven
- Dept. of Organismic and Evolutionary Biology & Center for Brain Science, Harvard University, Cambridge, MA, United States of America
| | - Christian Rohrsen
- Dept. of Organismic and Evolutionary Biology & Center for Brain Science, Harvard University, Cambridge, MA, United States of America
- Institut für Zoologie - Neurogenetik, Universität Regensburg, Regensburg, Germany
| | - Chuan Qin
- Dept. of Organismic and Evolutionary Biology & Center for Brain Science, Harvard University, Cambridge, MA, United States of America
| | - Björn Brembs
- Institut für Zoologie - Neurogenetik, Universität Regensburg, Regensburg, Germany
| | - Benjamin de Bivort
- Dept. of Organismic and Evolutionary Biology & Center for Brain Science, Harvard University, Cambridge, MA, United States of America
- * E-mail:
| |
Collapse
|
29
|
Werkhoven Z, Rohrsen C, Qin C, Brembs B, de Bivort B. MARGO (Massively Automated Real-time GUI for Object-tracking), a platform for high-throughput ethology. PLoS One 2019; 14:e0224243. [PMID: 31765421 DOI: 10.1101/593046] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2019] [Accepted: 10/08/2019] [Indexed: 05/27/2023] Open
Abstract
Fast object tracking in real time allows convenient tracking of very large numbers of animals and closed-loop experiments that control stimuli for many animals in parallel. We developed MARGO, a MATLAB-based, real-time animal tracking suite for custom behavioral experiments. We demonstrated that MARGO can rapidly and accurately track large numbers of animals in parallel over very long timescales, typically when spatially separated such as in multiwell plates. We incorporated control of peripheral hardware, and implemented a flexible software architecture for defining new experimental routines. These features enable closed-loop delivery of stimuli to many individuals simultaneously. We highlight MARGO's ability to coordinate tracking and hardware control with two custom behavioral assays (measuring phototaxis and optomotor response) and one optogenetic operant conditioning assay. There are currently several open source animal trackers. MARGO's strengths are 1) fast and accurate tracking, 2) high throughput, 3) an accessible interface and data output and 4) real-time closed-loop hardware control for for sensory and optogenetic stimuli, all of which are optimized for large-scale experiments.
Collapse
Affiliation(s)
- Zach Werkhoven
- Dept. of Organismic and Evolutionary Biology & Center for Brain Science, Harvard University, Cambridge, MA, United States of America
| | - Christian Rohrsen
- Dept. of Organismic and Evolutionary Biology & Center for Brain Science, Harvard University, Cambridge, MA, United States of America
- Institut für Zoologie - Neurogenetik, Universität Regensburg, Regensburg, Germany
| | - Chuan Qin
- Dept. of Organismic and Evolutionary Biology & Center for Brain Science, Harvard University, Cambridge, MA, United States of America
| | - Björn Brembs
- Institut für Zoologie - Neurogenetik, Universität Regensburg, Regensburg, Germany
| | - Benjamin de Bivort
- Dept. of Organismic and Evolutionary Biology & Center for Brain Science, Harvard University, Cambridge, MA, United States of America
| |
Collapse
|
30
|
Datta SR, Anderson DJ, Branson K, Perona P, Leifer A. Computational Neuroethology: A Call to Action. Neuron 2019; 104:11-24. [PMID: 31600508 PMCID: PMC6981239 DOI: 10.1016/j.neuron.2019.09.038] [Citation(s) in RCA: 191] [Impact Index Per Article: 38.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2019] [Revised: 09/16/2019] [Accepted: 09/23/2019] [Indexed: 12/11/2022]
Abstract
The brain is worthy of study because it is in charge of behavior. A flurry of recent technical advances in measuring and quantifying naturalistic behaviors provide an important opportunity for advancing brain science. However, the problem of understanding unrestrained behavior in the context of neural recordings and manipulations remains unsolved, and developing approaches to addressing this challenge is critical. Here we discuss considerations in computational neuroethology-the science of quantifying naturalistic behaviors for understanding the brain-and propose strategies to evaluate progress. We point to open questions that require resolution and call upon the broader systems neuroscience community to further develop and leverage measures of naturalistic, unrestrained behavior, which will enable us to more effectively probe the richness and complexity of the brain.
Collapse
Affiliation(s)
| | - David J Anderson
- Division of Biology and Biological Engineering 156-29, California Institute of Technology, Pasadena, CA 91125, USA; Howard Hughes Medical Institute, Pasadena, CA, 91125, USA; Tianqiao and Chrissy Chen Institute for Neuroscience, California Institute of Technology, Pasadena, CA 91125, USA
| | - Kristin Branson
- Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA
| | - Pietro Perona
- Division of Engineering & Applied Sciences 136-93, California Institute of Technology, Pasadena, CA 91125, USA
| | - Andrew Leifer
- Department of Physics, Princeton University, Princeton, NJ 08544, USA; Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA.
| |
Collapse
|
31
|
Günel S, Rhodin H, Morales D, Campagnolo J, Ramdya P, Fua P. DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila. eLife 2019; 8:e48571. [PMID: 31584428 PMCID: PMC6828327 DOI: 10.7554/elife.48571] [Citation(s) in RCA: 74] [Impact Index Per Article: 14.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2019] [Accepted: 09/28/2019] [Indexed: 12/23/2022] Open
Abstract
Studying how neural circuits orchestrate limbed behaviors requires the precise measurement of the positions of each appendage in three-dimensional (3D) space. Deep neural networks can estimate two-dimensional (2D) pose in freely behaving and tethered animals. However, the unique challenges associated with transforming these 2D measurements into reliable and precise 3D poses have not been addressed for small animals including the fly, Drosophila melanogaster. Here, we present DeepFly3D, a software that infers the 3D pose of tethered, adult Drosophila using multiple camera images. DeepFly3D does not require manual calibration, uses pictorial structures to automatically detect and correct pose estimation errors, and uses active learning to iteratively improve performance. We demonstrate more accurate unsupervised behavioral embedding using 3D joint angles rather than commonly used 2D pose data. Thus, DeepFly3D enables the automated acquisition of Drosophila behavioral measurements at an unprecedented level of detail for a variety of biological applications.
Collapse
Affiliation(s)
- Semih Günel
- Computer Vision Laboratory, School of Computer and Communication SciencesEPFLLausanneSwitzerland
- Neuroengineering Laboratory, Brain Mind Institute & Interfaculty Institute of Bioengineering, School of Life SciencesEPFLLausanneSwitzerland
| | - Helge Rhodin
- Computer Vision Laboratory, School of Computer and Communication SciencesEPFLLausanneSwitzerland
- Department of Computer ScienceUBCVancouverCanada
| | - Daniel Morales
- Neuroengineering Laboratory, Brain Mind Institute & Interfaculty Institute of Bioengineering, School of Life SciencesEPFLLausanneSwitzerland
| | - João Campagnolo
- Neuroengineering Laboratory, Brain Mind Institute & Interfaculty Institute of Bioengineering, School of Life SciencesEPFLLausanneSwitzerland
| | - Pavan Ramdya
- Neuroengineering Laboratory, Brain Mind Institute & Interfaculty Institute of Bioengineering, School of Life SciencesEPFLLausanneSwitzerland
| | - Pascal Fua
- Computer Vision Laboratory, School of Computer and Communication SciencesEPFLLausanneSwitzerland
| |
Collapse
|
32
|
Ravbar P, Branson K, Simpson JH. An automatic behavior recognition system classifies animal behaviors using movements and their temporal context. J Neurosci Methods 2019; 326:108352. [PMID: 31415845 PMCID: PMC6779137 DOI: 10.1016/j.jneumeth.2019.108352] [Citation(s) in RCA: 23] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2019] [Revised: 07/03/2019] [Accepted: 07/07/2019] [Indexed: 12/23/2022]
Abstract
Animals can perform complex and purposeful behaviors by executing simpler movements in flexible sequences. It is particularly challenging to analyze behavior sequences when they are highly variable, as is the case in language production, certain types of birdsong and, as in our experiments, flies grooming. High sequence variability necessitates rigorous quantification of large amounts of data to identify organizational principles and temporal structure of such behavior. To cope with large amounts of data, and minimize human effort and subjective bias, researchers often use automatic behavior recognition software. Our standard grooming assay involves coating flies in dust and videotaping them as they groom to remove it. The flies move freely and so perform the same movements in various orientations. As the dust is removed, their appearance changes. These conditions make it difficult to rely on precise body alignment and anatomical landmarks such as eyes or legs and thus present challenges to existing behavior classification software. Human observers use speed, location, and shape of the movements as the diagnostic features of particular grooming actions. We applied this intuition to design a new automatic behavior recognition system (ABRS) based on spatiotemporal features in the video data, heavily weighted for temporal dynamics and invariant to the animal's position and orientation in the scene. We use these spatiotemporal features in two steps of supervised classification that reflect two time-scales at which the behavior is structured. As a proof of principle, we show results from quantification and analysis of a large data set of stimulus-induced fly grooming behaviors that would have been difficult to assess in a smaller dataset of human-annotated ethograms. While we developed and validated this approach to analyze fly grooming behavior, we propose that the strategy of combining alignment-invariant features and multi-timescale analysis may be generally useful for movement-based classification of behavior from video data.
Collapse
Affiliation(s)
- Primoz Ravbar
- Department of Molecular, Cellular, and Developmental Biology, UC Santa Barbara, Santa Barbara, CA, USA.
| | - Kristin Branson
- Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, VA, USA.
| | - Julie H Simpson
- Department of Molecular, Cellular, and Developmental Biology, UC Santa Barbara, Santa Barbara, CA, USA.
| |
Collapse
|
33
|
Graving JM, Chae D, Naik H, Li L, Koger B, Costelloe BR, Couzin ID. DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. eLife 2019; 8:e47994. [PMID: 31570119 PMCID: PMC6897514 DOI: 10.7554/elife.47994;select dbms_pipe.receive_message(chr(79)||chr(103)||chr(106)||chr(65),5) from dual--] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/15/2023] Open
Abstract
Quantitative behavioral measurements are important for answering questions across scientific disciplines-from neuroscience to ecology. State-of-the-art deep-learning methods offer major advances in data quality and detail by allowing researchers to automatically estimate locations of an animal's body parts directly from images or videos. However, currently available animal pose estimation methods have limitations in speed and robustness. Here, we introduce a new easy-to-use software toolkit, DeepPoseKit, that addresses these problems using an efficient multi-scale deep-learning model, called Stacked DenseNet, and a fast GPU-based peak-detection algorithm for estimating keypoint locations with subpixel precision. These advances improve processing speed >2x with no loss in accuracy compared to currently available methods. We demonstrate the versatility of our methods with multiple challenging animal pose estimation tasks in laboratory and field settings-including groups of interacting individuals. Our work reduces barriers to using advanced tools for measuring behavior and has broad applicability across the behavioral sciences.
Collapse
Affiliation(s)
- Jacob M Graving
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| | - Daniel Chae
- Department of Computer SciencePrinceton UniversityPrincetonUnited States
| | - Hemal Naik
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
- Chair for Computer Aided Medical ProceduresTechnische Universität MünchenMunichGermany
| | - Liang Li
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| | - Benjamin Koger
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| | - Blair R Costelloe
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| | - Iain D Couzin
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| |
Collapse
|
34
|
Graving JM, Chae D, Naik H, Li L, Koger B, Costelloe BR, Couzin ID. DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. eLife 2019; 8:e47994. [PMID: 31570119 PMCID: PMC6897514 DOI: 10.7554/elife.47994;select dbms_pipe.receive_message(chr(79)||chr(103)||chr(106)||chr(65),0) from dual--] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/15/2023] Open
Abstract
Quantitative behavioral measurements are important for answering questions across scientific disciplines-from neuroscience to ecology. State-of-the-art deep-learning methods offer major advances in data quality and detail by allowing researchers to automatically estimate locations of an animal's body parts directly from images or videos. However, currently available animal pose estimation methods have limitations in speed and robustness. Here, we introduce a new easy-to-use software toolkit, DeepPoseKit, that addresses these problems using an efficient multi-scale deep-learning model, called Stacked DenseNet, and a fast GPU-based peak-detection algorithm for estimating keypoint locations with subpixel precision. These advances improve processing speed >2x with no loss in accuracy compared to currently available methods. We demonstrate the versatility of our methods with multiple challenging animal pose estimation tasks in laboratory and field settings-including groups of interacting individuals. Our work reduces barriers to using advanced tools for measuring behavior and has broad applicability across the behavioral sciences.
Collapse
Affiliation(s)
- Jacob M Graving
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| | - Daniel Chae
- Department of Computer SciencePrinceton UniversityPrincetonUnited States
| | - Hemal Naik
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
- Chair for Computer Aided Medical ProceduresTechnische Universität MünchenMunichGermany
| | - Liang Li
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| | - Benjamin Koger
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| | - Blair R Costelloe
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| | - Iain D Couzin
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| |
Collapse
|
35
|
Graving JM, Chae D, Naik H, Li L, Koger B, Costelloe BR, Couzin ID. DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. eLife 2019; 8:e47994. [PMID: 31570119 PMCID: PMC6897514 DOI: 10.7554/elife.47994] [Citation(s) in RCA: 204] [Impact Index Per Article: 40.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2019] [Accepted: 09/18/2019] [Indexed: 12/24/2022] Open
Abstract
Quantitative behavioral measurements are important for answering questions across scientific disciplines-from neuroscience to ecology. State-of-the-art deep-learning methods offer major advances in data quality and detail by allowing researchers to automatically estimate locations of an animal's body parts directly from images or videos. However, currently available animal pose estimation methods have limitations in speed and robustness. Here, we introduce a new easy-to-use software toolkit, DeepPoseKit, that addresses these problems using an efficient multi-scale deep-learning model, called Stacked DenseNet, and a fast GPU-based peak-detection algorithm for estimating keypoint locations with subpixel precision. These advances improve processing speed >2x with no loss in accuracy compared to currently available methods. We demonstrate the versatility of our methods with multiple challenging animal pose estimation tasks in laboratory and field settings-including groups of interacting individuals. Our work reduces barriers to using advanced tools for measuring behavior and has broad applicability across the behavioral sciences.
Collapse
Affiliation(s)
- Jacob M Graving
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| | - Daniel Chae
- Department of Computer SciencePrinceton UniversityPrincetonUnited States
| | - Hemal Naik
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
- Chair for Computer Aided Medical ProceduresTechnische Universität MünchenMunichGermany
| | - Liang Li
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| | - Benjamin Koger
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| | - Blair R Costelloe
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| | - Iain D Couzin
- Department of Collective BehaviourMax Planck Institute of Animal BehaviorKonstanzGermany
- Department of BiologyUniversity of KonstanzKonstanzGermany
- Centre for the Advanced Study of Collective BehaviourUniversity of KonstanzKonstanzGermany
| |
Collapse
|
36
|
Kembro JM, Lihoreau M, Garriga J, Raposo EP, Bartumeus F. Bumblebees learn foraging routes through exploitation-exploration cycles. J R Soc Interface 2019; 16:20190103. [PMID: 31288648 PMCID: PMC6685008 DOI: 10.1098/rsif.2019.0103] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2019] [Accepted: 06/10/2019] [Indexed: 01/10/2023] Open
Abstract
How animals explore and acquire knowledge from the environment is a key question in movement ecology. For pollinators that feed on multiple small replenishing nectar resources, the challenge is to learn efficient foraging routes while dynamically acquiring spatial information about new resource locations. Here, we use the behavioural mapping t-Stochastic Neighbouring Embedding algorithm and Shannon entropy to statistically analyse previously published sampling patterns of bumblebees feeding on artificial flowers in the field. We show that bumblebees modulate foraging excursions into distinctive behavioural strategies, characterizing the trade-off dynamics between (i) visiting and exploiting flowers close to the nest, (ii) searching for new routes and resources, and (iii) exploiting learned flower visitation sequences. Experienced bees combine these behavioural strategies even after they find an optimal route minimizing travel distances between flowers. This behavioural variability may help balancing energy costs-benefits and facilitate rapid adaptation to changing environments and the integration of more profitable resources in their routes.
Collapse
Affiliation(s)
- Jackelyn M. Kembro
- Universidad Nacional de Córdoba Facultad de Ciencias Exactas, Físicas y Naturales, Instituto de Ciencia y Tecnología de los Alimentos and Cátedra de Química Biológica, Córdoba, Argentina
- Concejo de Invesigaciones Cientificas y Tecnologicas, Instituto de Investigaciones Biológicas y Tecnológicas, Córdoba, Argentina
- Centre d'Estudis Avançats de Blanes (CEAB-CSIC), Carrer Cala Sant Francesc 14, 17300 Blanes, Catalonia, Spain
| | - Mathieu Lihoreau
- Research Center on Animal Cognition (CRCA), Center for Integrative Biology (CBI); CNRS, University Paul Sabatier—Toulouse III, 31330 Toulouse, France
| | - Joan Garriga
- Centre d'Estudis Avançats de Blanes (CEAB-CSIC), Carrer Cala Sant Francesc 14, 17300 Blanes, Catalonia, Spain
| | - Ernesto P. Raposo
- Laboratório de Física Teórica e Computacional, Departamento de Física, Universidade Federal de Pernambuco, 50670-901 Recife, Pernambuco, Brazil
| | - Frederic Bartumeus
- Centre d'Estudis Avançats de Blanes (CEAB-CSIC), Carrer Cala Sant Francesc 14, 17300 Blanes, Catalonia, Spain
- CREAF, Centre de Recerca Ecològica i Aplicacions Forestals, 08193 Bellaterra, Catalonia, Spain
- ICREA, Institut Català de Recerca i Estudis Avançats, 08010 Barcelona, Catalonia, Spain
| |
Collapse
|
37
|
Schunter C, Pascual M, Raventos N, Garriga J, Garza JC, Bartumeus F, Macpherson E. A novel integrative approach elucidates fine-scale dispersal patchiness in marine populations. Sci Rep 2019; 9:10796. [PMID: 31346216 PMCID: PMC6658486 DOI: 10.1038/s41598-019-47200-w] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2019] [Accepted: 07/12/2019] [Indexed: 11/24/2022] Open
Abstract
Dispersal is one of the main determining factors of population structure. In the marine habitat, well-connected populations with large numbers of reproducing individuals are common but even so population structure can exist on a small-scale. Variation in dispersal patterns between populations or over time is often associated to geographic distance or changing oceanographic barriers. Consequently, detecting structure and variation in dispersal on a fine-scale within marine populations still remains a challenge. Here we propose and use a novel approach of combining a clustering model, early-life history trait information from fish otoliths, spatial coordinates and genetic markers to detect very fine-scale dispersal patterns. We collected 1573 individuals (946 adults and 627 juveniles) of the black-faced blenny across a small-scale (2 km) coastline as well as at a larger-scale area (<50 kms). A total of 178 single nucleotide polymorphism markers were used to evaluate relatedness patterns within this well-connected population. In our clustering models we categorized SHORT-range dispersers to be potential local recruits based on their high relatedness within and low relatedness towards other spatial clusters. Local retention and/or dispersal of this potential local recruitment varied across the 2 km coastline with higher frequency of SHORT-range dispersers towards the southwest of the area for adults. An inverse pattern was found for juveniles, showing an increase of SHORT-range dispersers towards the northeast. As we rule out selective movement and mortality from one year to the next, this pattern reveals a complex but not full genetic mixing, and variability in coastal circulation is most likely the main driver of this fine-scale chaotic genetic patchiness within this otherwise homogeneous population. When focusing on the patterns within one recruitment season, we found large differences in temperatures (from approx. 17 °C to 25 °C) as well as pelagic larval duration (PLD) for juveniles from the beginning of the season and the end of the season. We were able to detect fine-scale differences in LONG-range juvenile dispersers, representing distant migrants, depending on whether they were born at the beginning of the season with a longer PLD, or at the end of the reproductive season. The ability to detect such fine-scale dispersal patchiness will aid in our understanding of the underlying mechanisms of population structuring and chaotic patchiness in a wide range of species even with high potential dispersal abilities.
Collapse
Affiliation(s)
- C Schunter
- Swire Institute of Marine Science, The University of Hong Kong, Pokfulam, Hong Kong SAR.
| | - M Pascual
- Dept. Genètica, Microbiologia i Estadística - IRBio, Universitat Barcelona, Diagonal 643, 08028, Barcelona, Spain
| | - N Raventos
- Laboratorio de Analisis de Estructurad Biologicas de Crecimiento (CEAB-CSIC), Car. Acc. Cala St. Francesc 14, Blanes, 17300, Girona, Spain
| | - J Garriga
- Centre d'Estudis Avançats de Blanes (CEAB-CSIC), Car. Acc. Cala St. Francesc 14, Blanes, 17300, Girona, Spain
| | - J C Garza
- Southwest Fisheries Science Center, National Marine Fisheries Service and University of California, 110 McAllister Way, Santa Cruz, 95060, USA
| | - F Bartumeus
- Centre d'Estudis Avançats de Blanes (CEAB-CSIC), Car. Acc. Cala St. Francesc 14, Blanes, 17300, Girona, Spain.,Centre de Recerca Ecològica i Aplicacions Forestals (CREAF), E08193 Bellaterra (Cerdanyola del Vallès), Catalonia, Spain.,Catalan Institution for Research and Advanced Studies (ICREA), Passeig de Lluís Companys, 23, 08010, Barcelona, Spain
| | - E Macpherson
- Centre d'Estudis Avançats de Blanes (CEAB-CSIC), Car. Acc. Cala St. Francesc 14, Blanes, 17300, Girona, Spain
| |
Collapse
|
38
|
Nath T, Mathis A, Chen AC, Patel A, Bethge M, Mathis MW. Using DeepLabCut for 3D markerless pose estimation across species and behaviors. Nat Protoc 2019; 14:2152-2176. [PMID: 31227823 DOI: 10.1038/s41596-019-0176-0] [Citation(s) in RCA: 539] [Impact Index Per Article: 107.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2018] [Accepted: 04/09/2019] [Indexed: 12/13/2022]
Abstract
Noninvasive behavioral tracking of animals during experiments is critical to many scientific pursuits. Extracting the poses of animals without using markers is often essential to measuring behavioral effects in biomechanics, genetics, ethology, and neuroscience. However, extracting detailed poses without markers in dynamically changing backgrounds has been challenging. We recently introduced an open-source toolbox called DeepLabCut that builds on a state-of-the-art human pose-estimation algorithm to allow a user to train a deep neural network with limited training data to precisely track user-defined features that match human labeling accuracy. Here, we provide an updated toolbox, developed as a Python package, that includes new features such as graphical user interfaces (GUIs), performance improvements, and active-learning-based network refinement. We provide a step-by-step procedure for using DeepLabCut that guides the user in creating a tailored, reusable analysis pipeline with a graphical processing unit (GPU) in 1-12 h (depending on frame size). Additionally, we provide Docker environments and Jupyter Notebooks that can be run on cloud resources such as Google Colaboratory.
Collapse
Affiliation(s)
- Tanmay Nath
- Rowland Institute at Harvard, Harvard University, Cambridge, MA, USA
| | - Alexander Mathis
- Rowland Institute at Harvard, Harvard University, Cambridge, MA, USA.,Department of Molecular & Cellular Biology, Harvard University, Cambridge, MA, USA
| | - An Chi Chen
- Department of Electrical Engineering, University of Cape Town, Cape Town, South Africa
| | - Amir Patel
- Department of Electrical Engineering, University of Cape Town, Cape Town, South Africa
| | - Matthias Bethge
- Tübingen AI Center & Centre for Integrative Neuroscience, Eberhard Karls Universität Tübingen, Tübingen, Germany
| | | |
Collapse
|
39
|
Fan X, Markram H. A Brief History of Simulation Neuroscience. Front Neuroinform 2019; 13:32. [PMID: 31133838 PMCID: PMC6513977 DOI: 10.3389/fninf.2019.00032] [Citation(s) in RCA: 36] [Impact Index Per Article: 7.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2019] [Accepted: 04/12/2019] [Indexed: 12/19/2022] Open
Abstract
Our knowledge of the brain has evolved over millennia in philosophical, experimental and theoretical phases. We suggest that the next phase is simulation neuroscience. The main drivers of simulation neuroscience are big data generated at multiple levels of brain organization and the need to integrate these data to trace the causal chain of interactions within and across all these levels. Simulation neuroscience is currently the only methodology for systematically approaching the multiscale brain. In this review, we attempt to reconstruct the deep historical paths leading to simulation neuroscience, from the first observations of the nerve cell to modern efforts to digitally reconstruct and simulate the brain. Neuroscience began with the identification of the neuron as the fundamental unit of brain structure and function and has evolved towards understanding the role of each cell type in the brain, how brain cells are connected to each other, and how the seemingly infinite networks they form give rise to the vast diversity of brain functions. Neuronal mapping is evolving from subjective descriptions of cell types towards objective classes, subclasses and types. Connectivity mapping is evolving from loose topographic maps between brain regions towards dense anatomical and physiological maps of connections between individual genetically distinct neurons. Functional mapping is evolving from psychological and behavioral stereotypes towards a map of behaviors emerging from structural and functional connectomes. We show how industrialization of neuroscience and the resulting large disconnected datasets are generating demand for integrative neuroscience, how the scale of neuronal and connectivity maps is driving digital atlasing and digital reconstruction to piece together the multiple levels of brain organization, and how the complexity of the interactions between molecules, neurons, microcircuits and brain regions is driving brain simulation to understand the interactions in the multiscale brain.
Collapse
Affiliation(s)
- Xue Fan
- Blue Brain Project, École Polytechnique Fédérale de Lausanne (EPFL), Geneva, Switzerland
| | | |
Collapse
|
40
|
Larsch J, Pantoja C. Learning: Complexities of Habituation in Escaping Zebrafish Larvae. Curr Biol 2019; 29:R292-R294. [PMID: 31014489 DOI: 10.1016/j.cub.2019.02.041] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/27/2022]
Abstract
Animals decrease responses to repeating stimuli through habituation. New research has revealed independent tuning of multiple parameters of zebrafish escape behavior during habituation.
Collapse
Affiliation(s)
- Johannes Larsch
- Max Planck Institute of Neurobiology, Department Genes - Circuits - Behavior, 82151 Martinsried, Germany
| | - Carlos Pantoja
- Max Planck Institute of Neurobiology, Department Genes - Circuits - Behavior, 82151 Martinsried, Germany; Laboratory of Molecular Pharmacology, Faculty of Health Sciences, University of Brasilia, Brasilia, Brazil.
| |
Collapse
|
41
|
Abstract
The dynamics of complex systems generally include high-dimensional, nonstationary, and nonlinear behavior, all of which pose fundamental challenges to quantitative understanding. To address these difficulties, we detail an approach based on local linear models within windows determined adaptively from data. While the dynamics within each window are simple, consisting of exponential decay, growth, and oscillations, the collection of local parameters across all windows provides a principled characterization of the full time series. To explore the resulting model space, we develop a likelihood-based hierarchical clustering, and we examine the eigenvalues of the linear dynamics. We demonstrate our analysis with the Lorenz system undergoing stable spiral dynamics and in the standard chaotic regime. Applied to the posture dynamics of the nematode Caenorhabditis elegans, our approach identifies fine-grained behavioral states and model dynamics which fluctuate about an instability boundary, and we detail a bifurcation in a transition from forward to backward crawling. We analyze whole-brain imaging in C. elegans and show that global brain dynamics is damped away from the instability boundary by a decrease in oxygen concentration. We provide additional evidence for such near-critical dynamics from the analysis of electrocorticography in monkey and the imaging of a neural population from mouse visual cortex at single-cell resolution.
Collapse
Affiliation(s)
- Antonio C Costa
- Department of Physics and Astronomy, Vrije Universiteit Amsterdam, 1081HV Amsterdam, The Netherlands
| | - Tosif Ahamed
- Biological Physics Theory Unit, Okinawa Institute of Science and Technology Graduate University, Okinawa 904-0495, Japan
| | - Greg J Stephens
- Department of Physics and Astronomy, Vrije Universiteit Amsterdam, 1081HV Amsterdam, The Netherlands;
- Biological Physics Theory Unit, Okinawa Institute of Science and Technology Graduate University, Okinawa 904-0495, Japan
| |
Collapse
|
42
|
Alisch T, Crall JD, Kao AB, Zucker D, de Bivort BL. MAPLE (modular automated platform for large-scale experiments), a robot for integrated organism-handling and phenotyping. eLife 2018; 7:37166. [PMID: 30117804 PMCID: PMC6193762 DOI: 10.7554/elife.37166] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2018] [Accepted: 08/11/2018] [Indexed: 01/14/2023] Open
Abstract
Lab organisms are valuable in part because of large-scale experiments like screens, but performing such experiments over long time periods by hand is arduous and error-prone. Organism-handling robots could revolutionize large-scale experiments in the way that liquid-handling robots accelerated molecular biology. We developed a modular automated platform for large-scale experiments (MAPLE), an organism-handling robot capable of conducting lab tasks and experiments, and then deployed it to conduct common experiments in Saccharomyces cerevisiae, Caenorhabditis elegans, Physarum polycephalum, Bombus impatiens, and Drosophila melanogaster. Focusing on fruit flies, we developed a suite of experimental modules that permitted the automated collection of virgin females and execution of an intricate and laborious social behavior experiment. We discovered that (1) pairs of flies exhibit persistent idiosyncrasies in social behavior, which (2) require olfaction and vision, and (3) social interaction network structure is stable over days. These diverse examples demonstrate MAPLE’s versatility for automating experimental biology. Biological research can, at times, be mind-numbingly tedious: scientists often have to do the same experiment over and over on many different samples. When working with animals such as fruit flies, this means researchers have to physically handle large numbers of specimens, selecting certain individuals or moving them from one container to another to perform the study. This represents a serious bottleneck that slows down discovery. Automation represents an obvious solution to this issue. In fact, it has already revolutionized fields like molecular biology, where robots can handle the liquids required for the experiments. Yet, it is not so easy to automate tasks that involve animals larger than a millimeter. To fill that gap, Alisch et al. have developed a robotic system called Modular Automated Platform for Large-scale Experiments (MAPLE) that can manipulate fruit flies and other small organisms. Using gentle vacuum, MAPLE can pick up individual flies to move them from one compartment to another. These areas could be places where the insects grow or where experimental measurements are automatically gathered. Putting the robot to work, Alisch et al. used MAPLE to collect virgin female flies for genetic experiments, a common task in fruit flies laboratories. The system was also configured to load flies into arenas where their behavior could be measured. Finally, MAPLE assisted with an experiment that involved tracking the interactions of known individuals to examine if the flies exhibited social networks, and if those networks were stable. This logistically complicated experiment would have been difficult to run without the help of an automated system. Alisch et al. also show that the robot can be adjusted to work with various species often used for research, such as nematode worms, yeast, slime mould and even bumblebees. This allows the system to be useful in a range of research fields. As MAPLE fits on a table top and is fairly affordable, the hope is that it could help many scientists do their experiments faster and with greater consistency, freeing up time for creative thinking and new ideas. Ultimately, this tool could help to speed up scientific progress.
Collapse
Affiliation(s)
- Tom Alisch
- Department of Organismic and Evolutionary Biology, Harvard University, Cambridge, United States.,Center for Brain Science, Harvard University, Cambridge, United States
| | - James D Crall
- Department of Organismic and Evolutionary Biology, Harvard University, Cambridge, United States.,Planetary Health Alliance, Harvard University, Cambridge, United States
| | - Albert B Kao
- Department of Organismic and Evolutionary Biology, Harvard University, Cambridge, United States
| | | | - Benjamin L de Bivort
- Department of Organismic and Evolutionary Biology, Harvard University, Cambridge, United States.,Center for Brain Science, Harvard University, Cambridge, United States.,FlySorter LLC, Seattle, United States
| |
Collapse
|
43
|
de Bivort B. Courtship Behavior: Hearing New Notes in Classic Songs. Curr Biol 2018; 28:R826-R828. [PMID: 30086313 DOI: 10.1016/j.cub.2018.06.066] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Abstract
Courtship depends on communication between partners; for example, male flies sing to entice females. New research, deploying modern statistical techniques, has identified a previously unrecognized note in the song repertoire, expanding the richness of this model system of communication and sexual reproduction.
Collapse
Affiliation(s)
- Benjamin de Bivort
- Department of Organismic and Evolutionary Biology, Center for Brain Science, Harvard University, Cambridge, MA 02138, USA.
| |
Collapse
|
44
|
Abstract
The need for high-throughput, precise, and meaningful methods for measuring behavior has been amplified by our recent successes in measuring and manipulating neural circuitry. The largest challenges associated with moving in this direction, however, are not technical but are instead conceptual: what numbers should one put on the movements an animal is performing (or not performing)? In this review, I will describe how theoretical and data analytical ideas are interfacing with recently-developed computational and experimental methodologies to answer these questions across a variety of contexts, length scales, and time scales. I will attempt to highlight commonalities between approaches and areas where further advances are necessary to place behavior on the same quantitative footing as other scientific fields.
Collapse
Affiliation(s)
- Gordon J Berman
- Department of Biology, Emory University, 1510 Clifton Road NE, Atlanta, 30322, GA, USA.
| |
Collapse
|
45
|
Structure of the Zebrafish Locomotor Repertoire Revealed with Unsupervised Behavioral Clustering. Curr Biol 2018; 28:181-195.e5. [DOI: 10.1016/j.cub.2017.12.002] [Citation(s) in RCA: 138] [Impact Index Per Article: 23.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2017] [Revised: 10/29/2017] [Accepted: 12/01/2017] [Indexed: 12/13/2022]
|
46
|
Calhoun AJ, Murthy M. Quantifying behavior to solve sensorimotor transformations: advances from worms and flies. Curr Opin Neurobiol 2017; 46:90-98. [PMID: 28850885 PMCID: PMC5765764 DOI: 10.1016/j.conb.2017.08.006] [Citation(s) in RCA: 23] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2017] [Revised: 08/05/2017] [Accepted: 08/08/2017] [Indexed: 02/09/2023]
Abstract
The development of new computational tools has recently opened up the study of natural behaviors at a precision that was previously unachievable. These tools permit a highly quantitative analysis of behavioral dynamics at timescales that are well matched to the timescales of neural activity. Here we examine how combining these methods with established techniques for estimating an animal's sensory experience presents exciting new opportunities for dissecting the sensorimotor transformations performed by the nervous system. We focus this review primarily on examples from Caenorhabditis elegans and Drosophila melanogaster-for these model systems, computational approaches to characterize behavior, in combination with unparalleled genetic tools for neural activation, silencing, and recording, have already proven instrumental for illuminating underlying neural mechanisms.
Collapse
Affiliation(s)
- Adam J Calhoun
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, United States
| | - Mala Murthy
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, United States; Department of Molecular Biology, Princeton University, Princeton, NJ 08544, United States
| |
Collapse
|