Automated face recognition of rhesus macaques

The use of gaze to study cognition: limitations, solutions, and applications to animal welfare

The study of gaze responses, typically using looking time paradigms, has become a popular approach to improving our understanding of cognitive processes in non-verbal individuals. Our interpretation of data derived from these paradigms, however, is constrained by how we conceptually and methodologically approach these problems. In this perspective paper, we outline the application of gaze studies in comparative cognitive and behavioral research and highlight current limitations in the interpretation of commonly used paradigms. Further, we propose potential solutions, including improvements to current experimental approaches, as well as broad-scale benefits of technology and collaboration. Finally, we outline the potential benefits of studying gaze responses from an animal welfare perspective. We advocate the implementation of these proposals across the field of animal behavior and cognition to aid experimental validity, and further advance our knowledge on a variety of cognitive processes and welfare outcomes.

Group-based, autonomous, individualized training and testing of long-tailed macaques (Macaca fascicularis) in their home enclosure to a visuo-acoustic discrimination task

In recent years, the utility and efficiency of automated procedures for cognitive assessment in psychology and neuroscience have been demonstrated in non-human primates (NHP). This approach mimics conventional shaping principles of breaking down a final desired behavior into smaller components that can be trained in a staircase manner. When combined with home-cage-based approaches, this could lead to a reduction in human workload, enhancement in data quality, and improvement in animal welfare. However, to our knowledge, there are no reported attempts to develop automated training and testing protocols for long-tailed macaques (Macaca fascicularis), a ubiquitous NHP model in neuroscience and pharmaceutical research. In the current work, we present the results from 6 long-tailed macaques that were trained using an automated unsupervised training (AUT) protocol for introducing the animals to the basics of a two-alternative choice (2 AC) task where they had to discriminate a conspecific vocalization from a pure tone relying on images presented on a touchscreen to report their response. We found that animals (1) consistently engaged with the device across several months; (2) interacted in bouts of high engagement; (3) alternated peacefully to interact with the device; and (4) smoothly ascended from step to step in the visually guided section of the procedure, in line with previous results from other NHPs. However, we also found (5) that animals' performance remained at chance level as soon as the acoustically guided steps were reached; and (6) that the engagement level decreased significantly with decreasing performance during the transition from visual to acoustic-guided sections. We conclude that with an autonomous approach, it is possible to train long-tailed macaques in their social group using computer vision techniques and without dietary restriction to solve a visually guided discrimination task but not an acoustically guided task. We provide suggestions on what future attempts could take into consideration to instruct acoustically guided discrimination tasks successfully.

The Future of Artificial Intelligence in Monitoring Animal Identification, Health, and Behaviour

Simple Summary

Due to climate change and human interference, many species are now without habitats and on the brink of extinction. Zoos and other conservation spaces allow for non-human animal preservation and public education about endangered species and ecosystems. Monitoring the health and well-being of animals in care, while providing species-specific environments, is critical for zoo and conservation staff. In order to best provide such care, keepers and researchers need to gather as much information as possible about individual animals and species as a whole. This paper focuses on existing technology to monitor animals, providing a review on the history of technology, including recent technological advancements and current limitations. Subsequently, we provide a brief introduction to our proposed novel computer software: an artificial intelligence software capable of unobtrusively and non-invasively tracking individuals’ location, estimating position, and analyzing behaviour. This innovative technology is currently being trained with orangutans at the Toronto Zoo and will allow for mass data collection, permitting keepers and researchers to closely monitor individual animal welfare, learn about the variables impacting behaviour and provide additional enrichment or interventions accordingly.

Abstract

With many advancements, technologies are now capable of recording non-human animals’ location, heart rate, and movement, often using a device that is physically attached to the monitored animals. However, to our knowledge, there is currently no technology that is able to do this unobtrusively and non-invasively. Here, we review the history of technology for use with animals, recent technological advancements, current limitations, and a brief introduction to our proposed novel software. Canadian tech mogul EAIGLE Inc. has developed an artificial intelligence (AI) software solution capable of determining where people and assets are within public places or attractions for operational intelligence, security, and health and safety applications. The solution also monitors individual temperatures to reduce the potential spread of COVID-19. This technology has been adapted for use at the Toronto Zoo, initiated with a focus on Sumatran orangutans (Pongo abelii) given the close physical similarity between orangutans and humans as great ape species. This technology will be capable of mass data collection, individual identification, pose estimation, behaviour monitoring and tracking orangutans’ locations, in real time on a 24/7 basis, benefitting both zookeepers and researchers looking to review this information.

An Adaptive Embedding Network with Spatial Constraints for the Use of Few-Shot Learning in Endangered-Animal Detection

Image recording is now ubiquitous in the fields of endangered-animal conservation and GIS. However, endangered animals are rarely seen, and, thus, only a few samples of images of them are available. In particular, the study of endangered-animal detection has a vital spatial component. We propose an adaptive, few-shot learning approach to endangered-animal detection through data augmentation by applying constraints on the mixture of foreground and background images based on species distributions. First, the pre-trained, salient network U2-Net segments the foregrounds and backgrounds of images of endangered animals. Then, the pre-trained image completion network CR-Fill is used to repair the incomplete environment. Furthermore, our approach identifies a foreground–background mixture of different images to produce multiple new image examples, using the relation network to permit a more realistic mixture of foreground and background images. It does not require further supervision, and it is easy to embed into existing networks, which learn to compensate for the uncertainties and nonstationarities of few-shot learning. Our experimental results are in excellent agreement with theoretical predictions by different evaluation metrics, and they unveil the future potential of video surveillance to address endangered-animal detection in studies of their behavior and conservation.

Flexible auditory training, psychophysics, and enrichment of common marmosets with an automated, touchscreen-based system

Devising new and more efficient protocols to analyze the phenotypes of non-human primates, as well as their complex nervous systems, is rapidly becoming of paramount importance. This is because with genome-editing techniques, recently adopted to non-human primates, new animal models for fundamental and translational research have been established. One aspect in particular, namely cognitive hearing, has been difficult to assess compared to visual cognition. To address this, we devised autonomous, standardized, and unsupervised training and testing of auditory capabilities of common marmosets with a cage-based standalone, wireless system. All marmosets tested voluntarily operated the device on a daily basis and went from naïve to experienced at their own pace and with ease. Through a series of experiments, here we show, that animals autonomously learn to associate sounds with images; to flexibly discriminate sounds, and to detect sounds of varying loudness. The developed platform and training principles combine in-cage training of common marmosets for cognitive and psychoacoustic assessment with an enriched environment that does not rely on dietary restriction or social separation, in compliance with the 3Rs principle.

The authors present a cage-based stand-alone platform for autonomous, standardized, and unsupervised training and testing of visuo-auditory-cued behaviours of common marmosets. The experiments do not require dietary restriction or social separation.

Harmonisation of welfare indicators for macaques and marmosets used or bred for research

Background: Accurate assessment of the welfare of non-human primates (NHPs) used and bred for scientific purposes is essential for effective implementation of obligations to optimise their well-being, for validation of refinement techniques and novel welfare indicators, and for ensuring the highest quality data is obtained from these animals. Despite the importance of welfare assessment in NHP research, there is little consensus on what should be measured. Greater harmonisation of welfare indicators between facilities would enable greater collaboration and data sharing to address welfare-related questions in the management and use of NHPs. Methods: A Delphi consultation was used to survey attendees of the 2019 NC3Rs Primate Welfare Meeting (73 respondents) to build consensus on which welfare indicators for macaques and marmosets are reliable, valid, and practicable, and how these can be measured. Results: Self-harm behaviour, social enrichment, cage dimensions, body weight, a health monitoring programme, appetite, staff training, and positive reinforcement training were considered valid, reliable, and practicable indicators for macaques (≥70% consensus) within a hypothetical scenario context involving 500 animals. Indicators ranked important for assessing marmoset welfare were body weight, NHP induced and environmentally induced injuries, cage furniture, huddled posture, mortality, blood in excreta, and physical enrichment. Participants working with macaques in infectious disease and breeding identified a greater range of indicators as valid and reliable than did those working in neuroscience and toxicology, where animal-based indicators were considered the most important. The findings for macaques were compared with a previous Delphi consultation, and the expert-defined consensus from the two surveys used to develop a prototype protocol for assessing macaque welfare in research settings. Conclusions: Together the Delphi results and proto-protocol enable those working with research NHPs to more effectively assess the welfare of the animals in their care and to collaborate to advance refinement of NHP management and use.

Harmonisation of welfare indicators for macaques and marmosets used or bred for research

Background: Accurate assessment of the welfare of non-human primates (NHPs) used and bred for scientific purposes is essential for effective implementation of obligations to optimise their well-being, for validation of refinement techniques and novel welfare indicators, and for ensuring the highest quality data is obtained from these animals. Despite the importance of welfare assessment in NHP research, there is little consensus on what should be measured. Greater harmonisation of welfare indicators between facilities would enable greater collaboration and data sharing to address welfare-related questions in the management and use of NHPs. Methods: A Delphi consultation was used to survey attendees of the 2019 NC3Rs Primate Welfare Meeting (73 respondents) to build consensus on which welfare indicators for macaques and marmosets are reliable, valid, and practicable, and how these can be measured. Results: Self-harm behaviour, social enrichment, cage dimensions, body weight, a health monitoring programme, appetite, staff training, and positive reinforcement training were considered valid, reliable, and practicable indicators for macaques (≥70% consensus) within a hypothetical scenario context involving 500 animals. Indicators ranked important for assessing marmoset welfare were body weight, NHP induced and environmentally induced injuries, cage furniture, huddled posture, mortality, blood in excreta, and physical enrichment. Participants working with macaques in infectious disease and breeding identified a greater range of indicators as valid and reliable than did those working in neuroscience and toxicology, where animal-based indicators were considered the most important. The findings for macaques were compared with a previous Delphi consultation, and the expert-defined consensus from the two surveys used to develop a prototype protocol for assessing macaque welfare in research settings. Conclusions: Together the Delphi results and proto-protocol enable those working with research NHPs to more effectively assess the welfare of the animals in their care and to collaborate to advance refinement of NHP management and use.

Harmonisation of welfare indicators for macaques and marmosets used or bred for research

Background: Accurate assessment of the welfare of non-human primates (NHPs) used and bred for scientific purposes is essential for effective implementation of obligations to optimise their well-being, for validation of refinement techniques and novel welfare indicators, and for ensuring the highest quality data is obtained from these animals. Despite the importance of welfare assessment in NHP research, there is little consensus on what should be measured. Greater harmonisation of welfare indicators between facilities would enable greater collaboration and data sharing to address welfare-related questions in the management and use of NHPs. Methods: A Delphi consultation was used to survey attendees of the 2019 NC3Rs Primate Welfare Meeting (73 respondents) to build consensus on which welfare indicators for macaques and marmosets are reliable, valid, and practicable, and how these can be measured. Results: Self-harm behaviour, social enrichment, cage dimensions, body weight, a health monitoring programme, appetite, staff training, and positive reinforcement training were considered valid, reliable, and practicable indicators for macaques (≥70% consensus) within a hypothetical scenario context involving 500 animals. Indicators ranked important for assessing marmoset welfare were body weight, NHP induced and environmentally induced injuries, cage furniture, huddled posture, mortality, blood in excreta, and physical enrichment. Participants working with macaques in infectious disease and breeding identified a greater range of indicators as valid and reliable than did those working in neuroscience and toxicology, where animal-based indicators were considered the most important. The findings for macaques were compared with a previous Delphi consultation, and the expert-defined consensus from the two surveys used to develop a prototype protocol for assessing macaque welfare in research settings. Conclusions: Together the Delphi results and proto-protocol enable those working with research NHPs to more effectively assess the welfare of the animals in their care and to collaborate to advance refinement of NHP management and use.

A socio-ecological perspective on the gestural communication of great ape species, individuals, and social units

Over the last 30 years, most research on non-human primate gestural communication has been produced by psychologists, which has shaped the questions asked and the methods used. These researchers have drawn on concepts from philosophy, linguistics, anthropology, and ethology, but despite these broad influences the field has neglected to situate gestures into the socio-ecological context in which the diverse species, individuals, and social-units exist. In this review, we present current knowledge about great ape gestural communication in terms of repertoires, meanings, and development. We fold this into a conversation about variation in other types of ape social behaviour to identify areas for future research on variation in gestural communication. Given the large variation in socio-ecological factors across species and social-units (and the individuals within these groups), we may expect to find different preferences for specific gesture types; different needs for communicating specific meanings; and different rates of encountering specific contexts. New tools, such as machine-learning based automated movement tracking, may allow us to uncover potential variation in the speed and form of gesture actions or parts of gesture actions. New multi-group multi-generational datasets provide the opportunity to apply analyses, such as Bayesian modelling, which allows us to examine these rich behavioural landscapes. Together, by expanding our questions and our methods, researchers may finally be able to study great ape gestures from the perspective of the apes themselves and explore what this gestural communication system reveals about apes’ thinking and experience of their world.

Automatic Recognition of Macaque Facial Expressions for Detection of Affective States

Internal affective states produce external manifestations such as facial expressions. In humans, the Facial Action Coding System (FACS) is widely used to objectively quantify the elemental facial action units (AUs) that build complex facial expressions. A similar system has been developed for macaque monkeys-the Macaque FACS (MaqFACS); yet, unlike the human counterpart, which is already partially replaced by automatic algorithms, this system still requires labor-intensive coding. Here, we developed and implemented the first prototype for automatic MaqFACS coding. We applied the approach to the analysis of behavioral and neural data recorded from freely interacting macaque monkeys. The method achieved high performance in the recognition of six dominant AUs, generalizing between conspecific individuals (Macaca mulatta) and even between species (Macaca fascicularis). The study lays the foundation for fully automated detection of facial expressions in animals, which is crucial for investigating the neural substrates of social and affective states.

Modelling behaviors relevant to brain disorders in the nonhuman primate: Are we there yet?

Recent years have seen a profound resurgence of activity with nonhuman primates (NHPs) to model human brain disorders. From marmosets to macaques, the study of NHP species offers a unique window into the function of primate-specific neural circuits that are impossible to examine in other models. Examining how these circuits manifest into the complex behaviors of primates, such as advanced cognitive and social functions, has provided enormous insights to date into the mechanisms underlying symptoms of numerous neurological and neuropsychiatric illnesses. With the recent optimization of modern techniques to manipulate and measure neural activity in vivo, such as optogenetics and calcium imaging, NHP research is more well-equipped than ever to probe the neural mechanisms underlying pathological behavior. However, methods for behavioral experimentation and analysis in NHPs have noticeably failed to keep pace with these advances. As behavior ultimately lies at the junction between preclinical findings and its translation to clinical outcomes for brain disorders, approaches to improve the integrity, reproducibility, and translatability of behavioral experiments in NHPs requires critical evaluation. In this review, we provide a unifying account of existing brain disorder models using NHPs, and provide insights into the present and emerging contributions of behavioral studies to the field.

Assessing the reliability of an automated method for measuring dominance hierarchy in non-human primates

Among animal societies, dominance is an important social factor that influences inter-individual relationships. However, assessing dominance hierarchy can be a time-consuming activity which is potentially impeded by environmental factors, difficulties in the recognition of animals, or disturbance of animals during data collection. Here we took advantage of novel devices, machines for automated learning and testing (MALT), designed primarily to study non-human primate cognition, to additionally measure the dominance hierarchy of a semi-free-ranging primate group. When working on a MALT, an animal can be replaced by another, which could reflect an asymmetric dominance relationship. To assess the reliability of our method, we analysed a sample of the automated conflicts with video scoring and found that 74% of these replacements included genuine forms of social displacements. In 10% of the cases, we did not identify social interactions and in the remaining 16% we observed affiliative contacts between the monkeys. We analysed months of daily use of MALT by up to 26 semi-free-ranging Tonkean macaques (Macaca tonkeana) and found that dominance relationships inferred from these interactions strongly correlated with the ones derived from observations of spontaneous agonistic interactions collected during the same time period. An optional filtering procedure designed to exclude chance-driven displacements or affiliative contacts suggests that the presence of 26% of these interactions in data sets did not impair the reliability of this new method. We demonstrate that this method can be used to assess the dynamics of both individual social status, and group-wide hierarchical stability longitudinally with minimal research labour. Further, it facilitates a continuous assessment of dominance hierarchies in captive groups, even during unpredictable environmental or challenging social events, which underlines the usefulness of this method for group management purposes. Altogether, this study supports the use of MALT as a reliable tool to automatically and dynamically assess dominance hierarchy within captive groups of non-human primates, including juveniles, under conditions in which such technology can be used.

Beyond MRI: on the scientific value of combining non-human primate neuroimaging with metadata

Sharing and pooling large amounts of non-human primate neuroimaging data offer new exciting opportunities to understand the primate brain. The potential of big data in non-human primate neuroimaging could however be tremendously enhanced by combining such neuroimaging data with other types of information. Here we describe metadata that have been identified as particularly valuable by the non-human primate neuroimaging community, including behavioural, genetic, physiological and phylogenetic data.

Automated facial recognition for wildlife that lack unique markings: A deep learning approach for brown bears

Emerging technologies support a new era of applied wildlife research, generating data on scales from individuals to populations. Computer vision methods can process large datasets generated through image-based techniques by automating the detection and identification of species and individuals. With the exception of primates, however, there are no objective visual methods of individual identification for species that lack unique and consistent body markings. We apply deep learning approaches of facial recognition using object detection, landmark detection, a similarity comparison network, and an support vector machine-based classifier to identify individuals in a representative species, the brown bear Ursus arctos. Our open-source application, BearID, detects a bear's face in an image, rotates and extracts the face, creates an "embedding" for the face, and uses the embedding to classify the individual. We trained and tested the application using labeled images of 132 known individuals collected from British Columbia, Canada, and Alaska, USA. Based on 4,674 images, with an 80/20% split for training and testing, respectively, we achieved a facial detection (ability to find a face) average precision of 0.98 and an individual classification (ability to identify the individual) accuracy of 83.9%. BearID and its annotated source code provide a replicable methodology for applying deep learning methods of facial recognition applicable to many other species that lack distinguishing markings. Further analyses of performance should focus on the influence of certain parameters on recognition accuracy, such as age and body size. Combining BearID with camera trapping could facilitate fine-scale behavioral research such as individual spatiotemporal activity patterns, and a cost-effective method of population monitoring through mark-recapture studies, with implications for species and landscape conservation and management. Applications to practical conservation include identifying problem individuals in human-wildlife conflicts, and evaluating the intrapopulation variation in efficacy of conservation strategies, such as wildlife crossings.

Mymou: A low-cost, wireless touchscreen system for automated training of nonhuman primates

Training nonhuman primates (NHPs) to perform cognitive tasks is essential for many neuroscientific investigations, yet laboratory training is a time-consuming process with inherent limitations. Habituating NHPs to the laboratory staff and experimental equipment can take months before NHPs are ready to proceed to the primary tasks. Laboratory training also necessarily separates NHPs from their home-room social group and typically involves some form of restraint or limited mobility, and data collection is often limited to a few hours per day so that multiple NHPs can be trained on the same equipment. Consequently, it can often take a year to train NHPs on complex cognitive tasks. To overcome these issues, we developed a low-cost, open-source, wireless touchscreen training system that can be installed in the home-room environment. The automated device can run continuously all day, including over weekends, without experimenter intervention. The system utilizes real-time facial recognition to initiate subject-specific tasks and provide accurate data logging, without the need for implanted microchips or separation of the NHPs. The system allows NHPs to select their preferred reward on each trial and to work when and for as long as they desire, and it can analyze task performance in real time and adapt the task parameters in order to expedite training. We demonstrate that NHPs consistently use this system on a daily basis to quickly learn complex behavioral tasks. The system therefore addresses many of the welfare and experimental limitations of laboratory-based training of NHPs and provides a platform for wireless electrophysiological investigations in more naturalistic, freely moving environments.

Chimpanzee face recognition from videos in the wild using deep learning

Video recording is now ubiquitous in the study of animal behavior, but its analysis on a large scale is prohibited by the time and resources needed to manually process large volumes of data. We present a deep convolutional neural network (CNN) approach that provides a fully automated pipeline for face detection, tracking, and recognition of wild chimpanzees from long-term video records. In a 14-year dataset yielding 10 million face images from 23 individuals over 50 hours of footage, we obtained an overall accuracy of 92.5% for identity recognition and 96.2% for sex recognition. Using the identified faces, we generated co-occurrence matrices to trace changes in the social network structure of an aging population. The tools we developed enable easy processing and annotation of video datasets, including those from other species. Such automated analysis unveils the future potential of large-scale longitudinal video archives to address fundamental questions in behavior and conservation.

Using imaging photoplethysmography for heart rate estimation in non-human primates

For humans and for non-human primates heart rate is a reliable indicator of an individual's current physiological state, with applications ranging from health checks to experimental studies of cognitive and emotional state. In humans, changes in the optical properties of the skin tissue correlated with cardiac cycles (imaging photoplethysmogram, iPPG) allow non-contact estimation of heart rate by its proxy, pulse rate. Yet, there is no established simple and non-invasive technique for pulse rate measurements in awake and behaving animals. Using iPPG, we here demonstrate that pulse rate in rhesus monkeys can be accurately estimated from facial videos. We computed iPPGs from eight color facial videos of four awake head-stabilized rhesus monkeys. Pulse rate estimated from iPPGs was in good agreement with reference data from a contact pulse-oximeter: the error of pulse rate estimation was below 5% of the individual average pulse rate in 83% of the epochs; the error was below 10% for 98% of the epochs. We conclude that iPPG allows non-invasive and non-contact estimation of pulse rate in non-human primates, which is useful for physiological studies and can be used toward welfare-assessment of non-human primates in research.