Using RGB displays to portray color realistic imagery to animal eyes

Through Hawks’ Eyes: Synthetically Reconstructing the Visual Field of a Bird in Flight

Birds of prey rely on vision to execute flight manoeuvres that are key to their survival, such as intercepting fast-moving targets or navigating through clutter. A better understanding of the role played by vision during these manoeuvres is not only relevant within the field of animal behaviour, but could also have applications for autonomous drones. In this paper, we present a novel method that uses computer vision tools to analyse the role of active vision in bird flight, and demonstrate its use to answer behavioural questions. Combining motion capture data from Harris’ hawks with a hybrid 3D model of the environment, we render RGB images, semantic maps, depth information and optic flow outputs that characterise the visual experience of the bird in flight. In contrast with previous approaches, our method allows us to consider different camera models and alternative gaze strategies for the purposes of hypothesis testing, allows us to consider visual input over the complete visual field of the bird, and is not limited by the technical specifications and performance of a head-mounted camera light enough to attach to a bird’s head in flight. We present pilot data from three sample flights: a pursuit flight, in which a hawk intercepts a moving target, and two obstacle avoidance flights. With this approach, we provide a reproducible method that facilitates the collection of large volumes of data across many individuals, opening up new avenues for data-driven models of animal behaviour.

Exploiting colour space geometry for visual stimulus design across animals

Colour vision represents a vital aspect of perception that ultimately enables a wide variety of species to thrive in the natural world. However, unified methods for constructing chromatic visual stimuli in a laboratory setting are lacking. Here, we present stimulus design methods and an accompanying programming package to efficiently probe the colour space of any species in which the photoreceptor spectral sensitivities are known. Our hardware-agnostic approach incorporates photoreceptor models within the framework of the principle of univariance. This enables experimenters to identify the most effective way to combine multiple light sources to create desired distributions of light, and thus easily construct relevant stimuli for mapping the colour space of an organism. We include methodology to handle uncertainty of photoreceptor spectral sensitivity as well as to optimally reconstruct hyperspectral images given recent hardware advances. Our methods support broad applications in colour vision science and provide a framework for uniform stimulus designs across experimental systems. This article is part of the theme issue 'Understanding colour vision: molecular, physiological, neuronal and behavioural studies in arthropods'.

Adding colour-realistic video images to audio playbacks increases stimulus engagement but does not enhance vocal learning in zebra finches

Bird song and human speech are learned early in life and for both cases engagement with live social tutors generally leads to better learning outcomes than passive audio-only exposure. Real-world tutor–tutee relations are normally not uni- but multimodal and observations suggest that visual cues related to sound production might enhance vocal learning. We tested this hypothesis by pairing appropriate, colour-realistic, high frame-rate videos of a singing adult male zebra finch tutor with song playbacks and presenting these stimuli to juvenile zebra finches (Taeniopygia guttata). Juveniles exposed to song playbacks combined with video presentation of a singing bird approached the stimulus more often and spent more time close to it than juveniles exposed to audio playback only or audio playback combined with pixelated and time-reversed videos. However, higher engagement with the realistic audio–visual stimuli was not predictive of better song learning. Thus, although multimodality increased stimulus engagement and biologically relevant video content was more salient than colour and movement equivalent videos, the higher engagement with the realistic audio–visual stimuli did not lead to enhanced vocal learning. Whether the lack of three-dimensionality of a video tutor and/or the lack of meaningful social interaction make them less suitable for facilitating song learning than audio–visual exposure to a live tutor remains to be tested.

Decontextualized learning for interpretable hierarchical representations of visual patterns

Apart from discriminative modeling, the application of deep convolutional neural networks to basic research utilizing natural imaging data faces unique hurdles. Here, we present decontextualized hierarchical representation learning (DHRL), designed specifically to overcome these limitations. DHRL enables the broader use of small datasets, which are typical in most studies. It also captures spatial relationships between features, provides novel tools for investigating latent variables, and achieves state-of-the-art disentanglement scores on small datasets. DHRL is enabled by a novel preprocessing technique inspired by generative model chaining and an improved ladder network architecture and regularization scheme. More than an analytical tool, DHRL enables novel capabilities for virtual experiments performed directly on a latent representation, which may transform the way we perform investigations of natural image features, directly integrating analytical, empirical, and theoretical approaches.

Von Uexküll Revisited: Addressing Human Biases in the Study of Animal Perception

More than 100 years ago, the biologist Jakob von Uexküll suggested that, because sensory systems are diverse, animals likely inhabit different sensory worlds (umwelten) than we do. Since von Uexküll, work across sensory modalities has confirmed that animals sometimes perceive sensory information that humans cannot, and it is now well-established that one must account for this fact when studying an animal's behavior. We are less adept, however, at recognizing cases in which non-human animals may not detect or perceive stimuli the same way we do, which is our focus here. In particular, we discuss three ways in which our own perception can result in misinformed hypotheses about the function of various stimuli. In particular, we may (1) make untested assumptions about how sensory information is perceived, based on how we perceive or measure it, (2) attribute undue significance to stimuli that we perceive as complex or striking, and (3) assume that animals divide the sensory world in the same way that we as scientists do. We discuss each of these biases and provide examples of cases where animals cannot perceive or are not attending to stimuli in the same way that we do, and how this may lead us to mistaken assumptions. Because what an animal perceives affects its behavior, we argue that these biases are especially important for researchers in sensory ecology, cognition, and animal behavior and communication to consider. We suggest that studying animal umwelten requires integrative approaches that combine knowledge of sensory physiology with behavioral assays.

I spy with my little eye: a simple behavioral assay to test color sensitivity on digital displays

Passive and interactive virtual reality (VR) environments are becoming increasingly popular in the field of behavioral neuroscience. While the technique was originally developed for human observers, corresponding applications have been adopted for the research of visual-driven behavior and neural circuits in animals. RGB color reproduction using red, green and blue primary color pixels is generally calibrated for humans, questioning if the distinct parameters are also readily transferable to other species. In particular, a visual image in the RGB color space has a clearly defined contrast pattern for humans, but this may not necessarily be the case for other mammals or even non-mammalian species, thereby impairing any interpretation of color-related behavioral or neuronal results. Here, we present a simple method to estimate the sensitivity of animals to the three primary colors of digital display devices based on the performance of object motion-driven visuo-motor reflexes and demonstrate differences in the color sensitivity between Xenopus laevis and Ambystoma mexicanum (Axolotl).

This article has an associated First Person interview with the first author of the paper.

Summary: We present a simple method to estimate an animals’ sensitivity to component colors of RGB images based on visuo-motor reflexes and demonstrate its applicability in Xenopus laevis and Ambystoma mexicanum.

Using virtual reality to study visual performances of honeybees

Virtual reality (VR) offers an appealing experimental framework for studying visual performances of insects under highly controlled conditions. In the case of the honeybee Apis mellifera, this possibility may fill the gap between behavioural analyses in free-flight and cellular analyses in the laboratory. Using automated, computer-controlled systems, it is possible to generate virtual stimuli or even entire environments that can be modified to test hypotheses on bee visual behaviour. The bee itself can remain tethered in place, making it possible to record neural activity while the bees is performing behavioural tasks. Recent studies have examined visual navigation and attentional processes in VR on flying or walking tethered bees, but experimental paradigms for examining visual learning and memory are only just emerging. Behavioural performances of bees under current experimental conditions are often lower in VR than in natural environments, but further improvements on current experimental protocols seem possible. Here we discuss current developments and conclude that it is essential to tailor the specifications of the VR simulation to the visual processing of honeybees to improve the success of this research endeavour.

Three-dimensional computer graphic animations for studying social approach behaviour in medaka fish: Effects of systematic manipulation of morphological and motion cues

We studied social approach behaviour in medaka fish using three-dimensional computer graphic (3DCG) animations based on the morphological features and motion characteristics obtained from real fish. This is the first study which used 3DCG animations and examined the relative effects of morphological and motion cues on social approach behaviour in medaka. Various visual stimuli, e.g., lack of motion, lack of colour, alternation in shape, lack of locomotion, lack of body motion, and normal virtual fish in which all four features (colour, shape, locomotion, and body motion) were reconstructed, were created and presented to fish using a computer display. Medaka fish presented with normal virtual fish spent a long time in proximity to the display, whereas time spent near the display was decreased in other groups when compared with normal virtual medaka group. The results suggested that the naturalness of visual cues contributes to the induction of social approach behaviour. Differential effects between body motion and locomotion were also detected. 3DCG animations can be a useful tool to study the mechanisms of visual processing and social behaviour in medaka.

Technical and conceptual considerations for using animated stimuli in studies of animal behavior

Rapid technical advances in the field of computer animation (CA) and virtual reality (VR) have opened new avenues in animal behavior research. Animated stimuli are powerful tools as they offer standardization, repeatability, and complete control over the stimulus presented, thereby "reducing" and "replacing" the animals used, and "refining" the experimental design in line with the 3Rs. However, appropriate use of these technologies raises conceptual and technical questions. In this review, we offer guidelines for common technical and conceptual considerations related to the use of animated stimuli in animal behavior research. Following the steps required to create an animated stimulus, we discuss (I) the creation, (II) the presentation, and (III) the validation of CAs and VRs. Although our review is geared toward computer-graphically designed stimuli, considerations on presentation and validation also apply to video playbacks. CA and VR allow both new behavioral questions to be addressed and existing questions to be addressed in new ways, thus we expect a rich future for these methods in both ultimate and proximate studies of animal behavior.