Bumblebees display characteristics of active vision during robust obstacle avoidance flight

Prospective control of steering through multiple waypoints

Some locomotor tasks involve steering at high speeds through multiple waypoints within cluttered environments. Although in principle actors could treat each individual waypoint in isolation, skillful performance would seem to require them to adapt their trajectory to the most immediate waypoint in anticipation of subsequent waypoints. To date, there have been few studies of such behavior, and the evidence that does exist is inconclusive about whether steering is affected by multiple future waypoints. The present study was designed to address the need for a clearer understanding of how humans adapt their steering movements in anticipation of future goals. Subjects performed a simulated drone flying task in a forest-like virtual environment that was presented on a monitor while their eye movements were tracked. They were instructed to steer through a series of gates while the distance at which gates first became visible (i.e., lookahead distance) was manipulated between trials. When gates became visible at least 1-1/2 segments in advance, subjects successfully flew through a high percentage of gates, rarely collided with obstacles, and maintained a consistent speed. They also approached the most immediate gate in a way that depended on the angular position of the subsequent gate. However, when the lookahead distance was less than 1-1/2 segments, subjects followed longer paths and flew at slower, more variable speeds. The findings demonstrate that the control of steering through multiple waypoints does indeed depend on information from beyond the most immediate waypoint. Discussion focuses on the possible control strategies for steering through multiple waypoints.

Perception of motion salience shapes the emergence of collective motions

Despite the profound implications of self-organization in animal groups for collective behaviors, understanding the fundamental principles and applying them to swarm robotics remains incomplete. Here we propose a heuristic measure of perception of motion salience (MS) to quantify relative motion changes of neighbors from first-person view. Leveraging three large bird-flocking datasets, we explore how this perception of MS relates to the structure of leader-follower (LF) relations, and further perform an individual-level correlation analysis between past perception of MS and future change rate of velocity consensus. We observe prevalence of the positive correlations in real flocks, which demonstrates that individuals will accelerate the convergence of velocity with neighbors who have higher MS. This empirical finding motivates us to introduce the concept of adaptive MS-based (AMS) interaction in swarm model. Finally, we implement AMS in a swarm of ~102 miniature robots. Swarm experiments show the significant advantage of AMS in enhancing self-organization of the swarm for smooth evacuations from confined environments.

Looking across the gap: Understanding the evolution of eyes and vision among insects

The compound eyes of insects exhibit stunning variation in size, structure, and function, which has allowed these animals to use their vision to adapt to a huge range of different environments and lifestyles, and evolve complex behaviors. Much of our knowledge of eye development has been learned from Drosophila, while visual adaptations and behaviors are often more striking and better understood from studies of other insects. However, recent studies in Drosophila and other insects, including bees, beetles, and butterflies, have begun to address this gap by revealing the genetic and developmental bases of differences in eye morphology and key new aspects of compound eye structure and function. Furthermore, technical advances have facilitated the generation of high-resolution connectomic data from different insect species that enhances our understanding of visual information processing, and the impact of changes in these processes on the evolution of vision and behavior. Here, we review these recent breakthroughs and propose that future integrated research from the development to function of visual systems within and among insect species represents a great opportunity to understand the remarkable diversification of insect eyes and vision.

Analysis of collision avoidance in honeybee flight

Insects are excellent at flying in dense vegetation and navigating through other complex spatial environments. This study investigates the strategies used by honeybees (Apis mellifera) to avoid collisions with an obstacle encountered frontally during flight. Bees were trained to fly through a tunnel that contained a solitary vertically oriented cylindrical obstacle placed along the midline. Flight trajectories of bees were recorded for six conditions in which the diameter of the obstructing cylinder was systematically varied from 25 mm to 160 mm. Analysis of salient events during the bees' flight, such as the deceleration before the obstacle, and the initiation of the deviation in flight path to avoid collisions, revealed a strategy for obstacle avoidance that is based on the relative retinal expansion velocity generated by the obstacle when the bee is on a collision course. We find that a quantitative model, featuring a controller that extracts specific visual cues from the frontal visual field, provides an accurate characterization of the geometry and the dynamics of the manoeuvres adopted by honeybees to avoid collisions. This study paves the way for the design of unmanned aerial systems, by identifying the visual cues that are used by honeybees for performing robust obstacle avoidance flight.

Finding the gap: neuromorphic motion-vision in dense environments

Animals have evolved mechanisms to travel safely and efficiently within different habitats. On a journey in dense terrains animals avoid collisions and cross narrow passages while controlling an overall course. Multiple hypotheses target how animals solve challenges faced during such travel. Here we show that a single mechanism enables safe and efficient travel. We developed a robot inspired by insects. It has remarkable capabilities to travel in dense terrain, avoiding collisions, crossing gaps and selecting safe passages. These capabilities are accomplished by a neuromorphic network steering the robot toward regions of low apparent motion. Our system leverages knowledge about vision processing and obstacle avoidance in insects. Our results demonstrate how insects might safely travel through diverse habitats. We anticipate our system to be a working hypothesis to study insects' travels in dense terrains. Furthermore, it illustrates that we can design novel hardware systems by understanding the underlying mechanisms driving behaviour.

Flying, nectar-loaded honey bees conserve water and improve heat tolerance by reducing wingbeat frequency and metabolic heat production

Heat waves are becoming increasingly common due to climate change, making it crucial to identify and understand the capacities for insect pollinators, such as honey bees, to avoid overheating. We examined the effects of hot, dry air temperatures on the physiological and behavioral mechanisms that honey bees use to fly when carrying nectar loads, to assess how foraging is limited by overheating or desiccation. We found that flight muscle temperatures increased linearly with load mass at air temperatures of 20 or 30 °C, but, remarkably, there was no change with increasing nectar loads at an air temperature of 40 °C. Flying, nectar-loaded bees were able to avoid overheating at 40 °C by reducing their flight metabolic rates and increasing evaporative cooling. At high body temperatures, bees apparently increase flight efficiency by lowering their wingbeat frequency and increasing stroke amplitude to compensate, reducing the need for evaporative cooling. However, even with reductions in metabolic heat production, desiccation likely limits foraging at temperatures well below bees' critical thermal maxima in hot, dry conditions.

MBT3D: Deep learning based multi-object tracker for bumblebee 3D flight path estimation

This work presents the Multi-Bees-Tracker (MBT3D) algorithm, a Python framework implementing a deep association tracker for Tracking-By-Detection, to address the challenging task of tracking flight paths of bumblebees in a social group. While tracking algorithms for bumblebees exist, they often come with intensive restrictions, such as the need for sufficient lighting, high contrast between the animal and background, absence of occlusion, significant user input, etc. Tracking flight paths of bumblebees in a social group is challenging. They suddenly adjust movements and change their appearance during different wing beat states while exhibiting significant similarities in their individual appearance. The MBT3D tracker, developed in this research, is an adaptation of an existing ant tracking algorithm for bumblebee tracking. It incorporates an offline trained appearance descriptor along with a Kalman Filter for appearance and motion matching. Different detector architectures for upstream detections (You Only Look Once (YOLOv5), Faster Region Proposal Convolutional Neural Network (Faster R-CNN), and RetinaNet) are investigated in a comparative study to optimize performance. The detection models were trained on a dataset containing 11359 labeled bumblebee images. YOLOv5 reaches an Average Precision of AP = 53, 8%, Faster R-CNN achieves AP = 45, 3% and RetinaNet AP = 38, 4% on the bumblebee validation dataset, which consists of 1323 labeled bumblebee images. The tracker's appearance model is trained on 144 samples. The tracker (with Faster R-CNN detections) reaches a Multiple Object Tracking Accuracy MOTA = 93, 5% and a Multiple Object Tracking Precision MOTP = 75, 6% on a validation dataset containing 2000 images, competing with state-of-the-art computer vision methods. The framework allows reliable tracking of different bumblebees in the same video stream with rarely occurring identity switches (IDS). MBT3D has much lower IDS than other commonly used algorithms, with one of the lowest false positive rates, competing with state-of-the-art animal tracking algorithms. The developed framework reconstructs the 3-dimensional (3D) flight paths of the bumblebees by triangulation. It also handles and compares two alternative stereo camera pairs if desired.

Optic flow enrichment via Drosophila head and retina motions to support inflight position regulation

Developing a functional description of the neural control circuits and visual feedback paths underlying insect flight behaviors is an active research area. Feedback controllers incorporating engineering models of the insect visual system outputs have described some flight behaviors, yet they do not explain how insects are able to stabilize their body position relative to nearby targets such as neighbors or forage sources, especially in challenging environments in which optic flow is poor. The insect experimental community is simultaneously recording a growing library of in-flight head and eye motions that may be linked to increased perception. This study develops a quantitative model of the optic flow experienced by a flying insect or robot during head yawing rotations (distinct from lateral peering motions in previous work) with a single other target in view. This study then applies a model of insect visuomotor feedback to show via analysis and simulation of five species that these head motions sufficiently enrich the optic flow and that the output feedback can provide relative position regulation relative to the single target (asymptotic stability). In the simplifying case of pure rotation relative to the body, theoretical analysis provides a stronger stability guarantee. The results are shown to be robust to anatomical neck angle limits and body vibrations, persist with more detailed Drosophila lateral-directional flight dynamics simulations, and generalize to recent retinal motion studies. Together, these results suggest that the optic flow enrichment provided by head or pseudopupil rotation could be used in an insect's neural processing circuit to enable position regulation.

Through Hawks’ Eyes: Synthetically Reconstructing the Visual Field of a Bird in Flight

Birds of prey rely on vision to execute flight manoeuvres that are key to their survival, such as intercepting fast-moving targets or navigating through clutter. A better understanding of the role played by vision during these manoeuvres is not only relevant within the field of animal behaviour, but could also have applications for autonomous drones. In this paper, we present a novel method that uses computer vision tools to analyse the role of active vision in bird flight, and demonstrate its use to answer behavioural questions. Combining motion capture data from Harris’ hawks with a hybrid 3D model of the environment, we render RGB images, semantic maps, depth information and optic flow outputs that characterise the visual experience of the bird in flight. In contrast with previous approaches, our method allows us to consider different camera models and alternative gaze strategies for the purposes of hypothesis testing, allows us to consider visual input over the complete visual field of the bird, and is not limited by the technical specifications and performance of a head-mounted camera light enough to attach to a bird’s head in flight. We present pilot data from three sample flights: a pursuit flight, in which a hawk intercepts a moving target, and two obstacle avoidance flights. With this approach, we provide a reproducible method that facilitates the collection of large volumes of data across many individuals, opening up new avenues for data-driven models of animal behaviour.

Optic flow based spatial vision in insects

The optic flow, i.e., the displacement of retinal images of objects in the environment induced by self-motion, is an important source of spatial information, especially for fast-flying insects. Spatial information over a wide range of distances, from the animal's immediate surroundings over several hundred metres to kilometres, is necessary for mediating behaviours, such as landing manoeuvres, collision avoidance in spatially complex environments, learning environmental object constellations and path integration in spatial navigation. To facilitate the processing of spatial information, the complexity of the optic flow is often reduced by active vision strategies. These result in translations and rotations being largely separated by a saccadic flight and gaze mode. Only the translational components of the optic flow contain spatial information. In the first step of optic flow processing, an array of local motion detectors provides a retinotopic spatial proximity map of the environment. This local motion information is then processed in parallel neural pathways in a task-specific manner and used to control the different components of spatial behaviour. A particular challenge here is that the distance information extracted from the optic flow does not represent the distances unambiguously, but these are scaled by the animal's speed of locomotion. Possible ways of coping with this ambiguity are discussed.