Automated audiovisual behavior recognition in wild primates

Application of Machine Learning for Automating Behavioral Tracking of Captive Bornean Orangutans (Pongo Pygmaeus)

This article applies object detection to CCTV video material to investigate the potential of using machine learning to automate behavior tracking. This study includes video tapings of two captive Bornean orangutans and their behavior. From a 2 min training video containing the selected behaviors, 334 images were extracted and labeled using Rectlabel. The labeled training material was used to construct an object detection model using Create ML. The use of object detection was shown to have potential for automating tracking, especially of locomotion, whilst filtering out false positives. Potential improvements regarding this tool are addressed, and future implementation should take these into consideration. These improvements include using adequately diverse training material and limiting iterations to avoid overfitting the model.

A toolkit for the dynamic study of air sacs in siamang and other elastic circular structures

Biological structures are defined by rigid elements, such as bones, and elastic elements, like muscles and membranes. Computer vision advances have enabled automatic tracking of moving animal skeletal poses. Such developments provide insights into complex time-varying dynamics of biological motion. Conversely, the elastic soft-tissues of organisms, like the nose of elephant seals, or the buccal sac of frogs, are poorly studied and no computer vision methods have been proposed. This leaves major gaps in different areas of biology. In primatology, most critically, the function of air sacs is widely debated; many open questions on the role of air sacs in the evolution of animal communication, including human speech, remain unanswered. To support the dynamic study of soft-tissue structures, we present a toolkit for the automated tracking of semi-circular elastic structures in biological video data. The toolkit contains unsupervised computer vision tools (using Hough transform) and supervised deep learning (by adapting DeepLabCut) methodology to track inflation of laryngeal air sacs or other biological spherical objects (e.g., gular cavities). Confirming the value of elastic kinematic analysis, we show that air sac inflation correlates with acoustic markers that likely inform about body size. Finally, we present a pre-processed audiovisual-kinematic dataset of 7+ hours of closeup audiovisual recordings of siamang (Symphalangus syndactylus) singing. This toolkit (https://github.com/WimPouw/AirSacTracker) aims to revitalize the study of non-skeletal morphological structures across multiple species.

GesturalOrigins: A bottom-up framework for establishing systematic gesture data across ape species

Current methodologies present significant hurdles to understanding patterns in the gestural communication of individuals, populations, and species. To address this issue, we present a bottom-up data collection framework for the study of gesture: GesturalOrigins. By "bottom-up", we mean that we minimise a priori structural choices, allowing researchers to define larger concepts (such as 'gesture types', 'response latencies', or 'gesture sequences') flexibly once coding is complete. Data can easily be re-organised to provide replication of, and comparison with, a wide range of datasets in published and planned analyses. We present packages, templates, and instructions for the complete data collection and coding process. We illustrate the flexibility that our methodological tool offers with worked examples of (great ape) gestural communication, demonstrating differences in the duration of action phases across distinct gesture action types and showing how species variation in the latency to respond to gestural requests may be revealed or masked by methodological choices. While GesturalOrigins is built from an ape-centred perspective, the basic framework can be adapted across a range of species and potentially to other communication systems. By making our gesture coding methods transparent and open access, we hope to enable a more direct comparison of findings across research groups, improve collaborations, and advance the field to tackle some of the long-standing questions in comparative gesture research.

OpenApePose, a database of annotated ape photographs for pose estimation

Because of their close relationship with humans, non-human apes (chimpanzees, bonobos, gorillas, orangutans, and gibbons, including siamangs) are of great scientific interest. The goal of understanding their complex behavior would be greatly advanced by the ability to perform video-based pose tracking. Tracking, however, requires high-quality annotated datasets of ape photographs. Here we present OpenApePose, a new public dataset of 71,868 photographs, annotated with 16 body landmarks of six ape species in naturalistic contexts. We show that a standard deep net (HRNet-W48) trained on ape photos can reliably track out-of-sample ape photos better than networks trained on monkeys (specifically, the OpenMonkeyPose dataset) and on humans (COCO) can. This trained network can track apes almost as well as the other networks can track their respective taxa, and models trained without one of the six ape species can track the held-out species better than the monkey and human models can. Ultimately, the results of our analyses highlight the importance of large, specialized databases for animal tracking systems and confirm the utility of our new ape database.

Elephants and algorithms: a review of the current and future role of AI in elephant monitoring

Artificial intelligence (AI) and machine learning (ML) present revolutionary opportunities to enhance our understanding of animal behaviour and conservation strategies. Using elephants, a crucial species in Africa and Asia's protected areas, as our focal point, we delve into the role of AI and ML in their conservation. Given the increasing amounts of data gathered from a variety of sensors like cameras, microphones, geophones, drones and satellites, the challenge lies in managing and interpreting this vast data. New AI and ML techniques offer solutions to streamline this process, helping us extract vital information that might otherwise be overlooked. This paper focuses on the different AI-driven monitoring methods and their potential for improving elephant conservation. Collaborative efforts between AI experts and ecological researchers are essential in leveraging these innovative technologies for enhanced wildlife conservation, setting a precedent for numerous other species.

Hierarchical action encoding in prefrontal cortex of freely moving macaques

Our natural behavioral repertoires include coordinated actions of characteristic types. To better understand how neural activity relates to the expression of actions and action switches, we studied macaques performing a freely moving foraging task in an open environment. We developed a novel analysis pipeline that can identify meaningful units of behavior, corresponding to recognizable actions such as sitting, walking, jumping, and climbing. On the basis of transition probabilities between these actions, we found that behavior is organized in a modular and hierarchical fashion. We found that, after regressing out many potential confounders, actions are associated with specific patterns of firing in each of six prefrontal brain regions and that, overall, encoding of action category is progressively stronger in more dorsal and more caudal prefrontal regions. Together, these results establish a link between selection of units of primate behavior on one hand and neuronal activity in prefrontal regions on the other.

Hierarchical organization of rhesus macaque behavior

Primatologists, psychologists and neuroscientists have long hypothesized that primate behavior is highly structured. However, delineating that structure has been impossible due to the difficulties of precision behavioral tracking. Here we analyzed a dataset consisting of continuous measures of the 3D position of two male rhesus macaques (Macaca mulatta) performing three different tasks in a large unrestrained environment over several hours. Using an unsupervised embedding approach on the tracked joints, we identified commonly repeated pose patterns, which we call postures. We found that macaques' behavior is characterized by 49 distinct postures, lasting an average of 0.6 seconds. We found evidence that behavior is hierarchically organized, in that transitions between poses tend to occur within larger modules, which correspond to identifiable actions; these actions are further organized hierarchically. Our behavioral decomposition allows us to identify universal (cross-individual and cross-task) and unique (specific to each individual and task) principles of behavior. These results demonstrate the hierarchical nature of primate behavior, provide a method for the automated ethogramming of primate behavior, and provide important constraints on neural models of pose generation.

Dynamic Curriculum Learning for Great Ape Detection in the Wild

We propose a novel end-to-end curriculum learning approach for sparsely labelled animal datasets leveraging large volumes of unlabelled data to improve supervised species detectors. We exemplify the method in detail on the task of finding great apes in camera trap footage taken in challenging real-world jungle environments. In contrast to previous semi-supervised methods, our approach adjusts learning parameters dynamically over time and gradually improves detection quality by steering training towards virtuous self-reinforcement. To achieve this, we propose integrating pseudo-labelling with curriculum learning policies and show how learning collapse can be avoided. We discuss theoretical arguments, ablations, and significant performance improvements against various state-of-the-art systems when evaluating on the Extended PanAfrican Dataset holding approx. 1.8M frames. We also demonstrate our method can outperform supervised baselines with significant margins on sparse label versions of other animal datasets such as Bees and Snapshot Serengeti. We note that performance advantages are strongest for smaller labelled ratios common in ecological applications. Finally, we show that our approach achieves competitive benchmarks for generic object detection in MS-COCO and PASCAL-VOC indicating wider applicability of the dynamic learning concepts introduced. We publish all relevant source code, network weights, and data access details for full reproducibility.

Integrating remote international experience and community engagement into course-based animal behavior research

Human-centered, active-learning approaches can help students develop core competencies in biology and other STEM fields, including the ability to conduct research, use quantitative reasoning, communicate across disciplinary boundaries, and connect science education to pressing social and environmental challenges. Promising approaches for incorporating active learning into biology courses include the use of course-based research, community engagement, and international experiences. Disruption to higher education due to the COVID-19 pandemic made each of these approaches more challenging or impossible to execute. Here, we describe a scalable course-based undergraduate research experience (CURE) for an animal behavior course that integrates research and community engagement in a remote international experience. Students in courses at two U.S. universities worked with community partners to analyze the behavior of African goats grazing near informal settlements in Western Cape, South Africa. Partners established a relationship with goat herders, and then created 2-min videos of individual goats that differed in criteria (goat sex and time of day) specified by students. Students worked in small groups to choose dependent variables, and then compared goat behavior across criteria using a factorial design. In postcourse surveys, students from both universities indicated overall enthusiasm for the experience. In general, students indicated that the laboratory provided them with "somewhat more" of a research-based experience compared with biology laboratories they had taken of similar length, and "somewhat more" to "much more" of a community-engagement and international experience. Educational benefits were complemented by the fact that international educational partners facing economic hardship due to the pandemic received payment for services. Future iterations of the CURE can focus on goat behavior differences across ecological conditions to help herders increase production in the face of continued environmental and social challenges. More generally, applying the structure of this CURE could facilitate mutually beneficial collaborations with residents of under-resourced areas around the world.

Chimpanzee play sequences are structured hierarchically as games

Social play is ubiquitous in the development of many animal species and involves players adapting actions flexibly to their own previous actions and partner responses. Play differs from other behavioural contexts for which fine-scale analyses of action sequences are available, such as tool use and communication, in that its form is not defined by its function, making it potentially more unpredictable. In humans, play is often organised in games, where players know context-appropriate actions but string them together unpredictably. Here, we use the sequential nature of play elements to explore whether play elements in chimpanzees are structured hierarchically and follow predictable game-like patterns. Based on 5,711 play elements from 143 bouts, we extracted individual-level play sequences of 11 Western chimpanzees (Pan troglodytes verus) of different ages from the Bossou community. We detected transition probabilities between play elements that exceeded expected levels and show that play elements form hierarchically clustered and interchangeable groups, indicative of at least six games that can be identified from transition networks, some with different roles for different players. We also show that increased information about preceding play elements improved predictability of subsequent elements, further indicating that play elements are not strung together randomly but that flexible action rules underlie their usage. Thus, chimpanzee play is hierarchically structured in short games which limit acceptable play elements and allow players to predict and adapt to partners' actions. This "grammar of action" approach to social interactions can be valuable in understanding cognitive and communicative abilities within and across species.

The Promise of Behavioral Tracking Systems for Advancing Primate Animal Welfare

Recent years have witnessed major advances in the ability of computerized systems to track the positions of animals as they move through large and unconstrained environments. These systems have so far been a great boon in the fields of primatology, psychology, neuroscience, and biomedicine. Here, we discuss the promise of these technologies for animal welfare. Their potential benefits include identifying and reducing pain, suffering, and distress in captive populations, improving laboratory animal welfare within the context of the three Rs of animal research (reduction, refinement, and replacement), and applying our understanding of animal behavior to increase the "natural" behaviors in captive and wild populations facing human impact challenges. We note that these benefits are often incidental to the designed purpose of these tracking systems, a reflection of the fact that animal welfare is not inimical to research progress, but instead, that the aligned interests between basic research and welfare hold great promise for improvements to animal well-being.

Evaluating Cognitive Enrichment for Zoo-Housed Gorillas Using Facial Recognition

The use of computer technology within zoos is becoming increasingly popular to help achieve high animal welfare standards. However, despite its various positive applications to wildlife in recent years, there has been little uptake of machine learning in zoo animal care. In this paper, we describe how a facial recognition system, developed using machine learning, was embedded within a cognitive enrichment device (a vertical, modular finger maze) for a troop of seven Western lowland gorillas (Gorilla gorilla gorilla) at Bristol Zoo Gardens, UK. We explored whether machine learning could automatically identify individual gorillas through facial recognition, and automate the collection of device-use data including the order, frequency and duration of use by the troop. Concurrent traditional video recording and behavioral coding by eye was undertaken for comparison. The facial recognition system was very effective at identifying individual gorillas (97% mean average precision) and could automate specific downstream tasks (for example, duration of engagement). However, its development was a heavy investment, requiring specialized hardware and interdisciplinary expertise. Therefore, we suggest a system like this is only appropriate for long-term projects. Additionally, researcher input was still required to visually identify which maze modules were being used by gorillas and how. This highlights the need for additional technology, such as infrared sensors, to fully automate cognitive enrichment evaluation. To end, we describe a future system that combines machine learning and sensor technology which could automate the collection of data in real-time for use by researchers and animal care staff.

Postural behavior recognition of captive nocturnal animals based on deep learning: a case study of Bengal slow loris

The precise identification of postural behavior plays a crucial role in evaluation of animal welfare and captive management. Deep learning technology has been widely used in automatic behavior recognition of wild and domestic fauna species. The Asian slow loris is a group of small, nocturnal primates with a distinctive locomotion mode, and a large number of individuals were confiscated into captive settings due to illegal trade, making the species an ideal as a model for postural behavior monitoring. Captive animals may suffer from being housed in an inappropriate environment and may display abnormal behavior patterns. Traditional data collection methods are time-consuming and laborious, impeding efforts to improve lorises' captive welfare and to develop effective reintroduction strategies. This study established the first human-labeled postural behavior dataset of slow lorises and used deep learning technology to recognize postural behavior based on object detection and semantic segmentation. The precision of the classification based on YOLOv5 reached 95.1%. The Dilated Residual Networks (DRN) feature extraction network showed the best performance in semantic segmentation, and the classification accuracy reached 95.2%. The results imply that computer automatic identification of postural behavior may offer advantages in assessing animal activity and can be applied to other nocturnal taxa.

To drum or not to drum: Selectivity in tree buttress drumming by chimpanzees (Pan troglodytes verus) in the Nimba Mountains, Guinea

Chimpanzees live in fission-fusion social organizations, which means that party size, composition, and spatial distribution are constantly in flux. Moreover, chimpanzees use a remarkably extensive repertoire of vocal and nonvocal forms of communication, thought to help convey information in such a socially and spatially dynamic setting. One proposed form of nonvocal communication in chimpanzees is buttress drumming, in which an individual hits a tree buttress with its hands and/or feet, thereby producing a low-frequency acoustic signal. It is often presumed that this behavior functions to communicate over long distances and is, therefore, goal-oriented. If so, we would expect chimpanzees to exhibit selectivity in the choice of trees and buttresses used in buttress drumming. Selectivity is a key attribute of many other goal-directed chimpanzee behaviors, such as nut-cracking and ant dipping. Here, we investigate whether chimpanzees at the Seringbara study site in the Nimba Mountains, Guinea, West Africa, show selectivity in their buttress drumming behavior. Our results indicate that Seringbara chimpanzees are more likely to use larger trees and select buttresses that are thinner and have a greater surface area. These findings imply that tree buttress drumming is not a random act, but rather goal-oriented and requires knowledge of suitable trees and buttresses. Our results also point to long-distance communication as a probable function of buttress drumming based on selectivity for buttress characteristics likely to impact sound propagation. This study provides a foundation for further assessing the cognitive underpinnings and functions of buttress drumming in wild chimpanzees.

Computational bioacoustics with deep learning: a review and roadmap

Animal vocalisations and natural soundscapes are fascinating objects of study, and contain valuable evidence about animal behaviours, populations and ecosystems. They are studied in bioacoustics and ecoacoustics, with signal processing and analysis an important component. Computational bioacoustics has accelerated in recent decades due to the growth of affordable digital sound recording devices, and to huge progress in informatics such as big data, signal processing and machine learning. Methods are inherited from the wider field of deep learning, including speech and image processing. However, the tasks, demands and data characteristics are often different from those addressed in speech or music analysis. There remain unsolved problems, and tasks for which evidence is surely present in many acoustic signals, but not yet realised. In this paper I perform a review of the state of the art in deep learning for computational bioacoustics, aiming to clarify key concepts and identify and analyse knowledge gaps. Based on this, I offer a subjective but principled roadmap for computational bioacoustics with deep learning: topics that the community should aim to address, in order to make the most of future developments in AI and informatics, and to use audio data in answering zoological and ecological questions.

Harmonisation of welfare indicators for macaques and marmosets used or bred for research

Background: Accurate assessment of the welfare of non-human primates (NHPs) used and bred for scientific purposes is essential for effective implementation of obligations to optimise their well-being, for validation of refinement techniques and novel welfare indicators, and for ensuring the highest quality data is obtained from these animals. Despite the importance of welfare assessment in NHP research, there is little consensus on what should be measured. Greater harmonisation of welfare indicators between facilities would enable greater collaboration and data sharing to address welfare-related questions in the management and use of NHPs. Methods: A Delphi consultation was used to survey attendees of the 2019 NC3Rs Primate Welfare Meeting (73 respondents) to build consensus on which welfare indicators for macaques and marmosets are reliable, valid, and practicable, and how these can be measured. Results: Self-harm behaviour, social enrichment, cage dimensions, body weight, a health monitoring programme, appetite, staff training, and positive reinforcement training were considered valid, reliable, and practicable indicators for macaques (≥70% consensus) within a hypothetical scenario context involving 500 animals. Indicators ranked important for assessing marmoset welfare were body weight, NHP induced and environmentally induced injuries, cage furniture, huddled posture, mortality, blood in excreta, and physical enrichment. Participants working with macaques in infectious disease and breeding identified a greater range of indicators as valid and reliable than did those working in neuroscience and toxicology, where animal-based indicators were considered the most important. The findings for macaques were compared with a previous Delphi consultation, and the expert-defined consensus from the two surveys used to develop a prototype protocol for assessing macaque welfare in research settings. Conclusions: Together the Delphi results and proto-protocol enable those working with research NHPs to more effectively assess the welfare of the animals in their care and to collaborate to advance refinement of NHP management and use.

Harmonisation of welfare indicators for macaques and marmosets used or bred for research

Background: Accurate assessment of the welfare of non-human primates (NHPs) used and bred for scientific purposes is essential for effective implementation of obligations to optimise their well-being, for validation of refinement techniques and novel welfare indicators, and for ensuring the highest quality data is obtained from these animals. Despite the importance of welfare assessment in NHP research, there is little consensus on what should be measured. Greater harmonisation of welfare indicators between facilities would enable greater collaboration and data sharing to address welfare-related questions in the management and use of NHPs. Methods: A Delphi consultation was used to survey attendees of the 2019 NC3Rs Primate Welfare Meeting (73 respondents) to build consensus on which welfare indicators for macaques and marmosets are reliable, valid, and practicable, and how these can be measured. Results: Self-harm behaviour, social enrichment, cage dimensions, body weight, a health monitoring programme, appetite, staff training, and positive reinforcement training were considered valid, reliable, and practicable indicators for macaques (≥70% consensus) within a hypothetical scenario context involving 500 animals. Indicators ranked important for assessing marmoset welfare were body weight, NHP induced and environmentally induced injuries, cage furniture, huddled posture, mortality, blood in excreta, and physical enrichment. Participants working with macaques in infectious disease and breeding identified a greater range of indicators as valid and reliable than did those working in neuroscience and toxicology, where animal-based indicators were considered the most important. The findings for macaques were compared with a previous Delphi consultation, and the expert-defined consensus from the two surveys used to develop a prototype protocol for assessing macaque welfare in research settings. Conclusions: Together the Delphi results and proto-protocol enable those working with research NHPs to more effectively assess the welfare of the animals in their care and to collaborate to advance refinement of NHP management and use.

Harmonisation of welfare indicators for macaques and marmosets used or bred for research

Background: Accurate assessment of the welfare of non-human primates (NHPs) used and bred for scientific purposes is essential for effective implementation of obligations to optimise their well-being, for validation of refinement techniques and novel welfare indicators, and for ensuring the highest quality data is obtained from these animals. Despite the importance of welfare assessment in NHP research, there is little consensus on what should be measured. Greater harmonisation of welfare indicators between facilities would enable greater collaboration and data sharing to address welfare-related questions in the management and use of NHPs. Methods: A Delphi consultation was used to survey attendees of the 2019 NC3Rs Primate Welfare Meeting (73 respondents) to build consensus on which welfare indicators for macaques and marmosets are reliable, valid, and practicable, and how these can be measured. Results: Self-harm behaviour, social enrichment, cage dimensions, body weight, a health monitoring programme, appetite, staff training, and positive reinforcement training were considered valid, reliable, and practicable indicators for macaques (≥70% consensus) within a hypothetical scenario context involving 500 animals. Indicators ranked important for assessing marmoset welfare were body weight, NHP induced and environmentally induced injuries, cage furniture, huddled posture, mortality, blood in excreta, and physical enrichment. Participants working with macaques in infectious disease and breeding identified a greater range of indicators as valid and reliable than did those working in neuroscience and toxicology, where animal-based indicators were considered the most important. The findings for macaques were compared with a previous Delphi consultation, and the expert-defined consensus from the two surveys used to develop a prototype protocol for assessing macaque welfare in research settings. Conclusions: Together the Delphi results and proto-protocol enable those working with research NHPs to more effectively assess the welfare of the animals in their care and to collaborate to advance refinement of NHP management and use.