Markerless tracking of an entire honey bee colony

Individual honey bee tracking in a beehive environment using deep learning and Kalman filter

The honey bee is the most essential pollinator and a key contributor to the natural ecosystem. There are numerous ways for thousands of bees in a hive to communicate with one another. Individual trajectories and social interactions are thus complex behavioral features that can provide valuable information for an ecological study. To study honey bee behavior, the key challenges that have resulted from unreliable studies include complexity (high density of similar objects, small objects, and occlusion), the variety of background scenes, the dynamism of individual bee movements, and the similarity between the bee body and the background in the beehive. This study investigated the tracking of individual bees in a beehive environment using a deep learning approach and a Kalman filter. Detection of multiple bees and individual object segmentation were performed using Mask R-CNN with a ResNet-101 backbone network. Subsequently, the Kalman filter was employed for tracking multiple bees by tracking the body of each bee across a sequence of image frames. Three metrics were used to assess the proposed framework: mean average precision (mAP) for multiple-object detection and segmentation tasks, CLEAR MOT for multiple object tracking tasks, and MOTS for multiple object tracking and segmentation tasks. For CLEAR MOT and MOTS metrics, accuracy (MOTA and MOTSA) and precision (MOTP and MOTSP) are considered. By employing videos from a custom-designed observation beehive, recorded at a frame rate of 30 frames per second (fps) and utilizing a continuous frame rate of 10 fps as input data, our system displayed impressive performance. It yielded satisfactory outcomes for tasks involving segmentation and tracking of multiple instances of bee behavior. For the multiple-object segmentation task based on Mask R-CNN, we achieved a 0.85 mAP. For the multiple-object-tracking task with the Kalman filter, we achieved 77.48% MOTA, 79.79% MOTSP, and 79.56% recall. For the overall system for multiple-object tracking and segmentation tasks, we achieved 77.00% MOTSA, 75.60% MOTSP, and 80.30% recall.

Rapid automated 3-D pose estimation of larval zebrafish using a physical model-trained neural network

Quantitative ethology requires an accurate estimation of an organism's postural dynamics in three dimensions plus time. Technological progress over the last decade has made animal pose estimation in challenging scenarios possible with unprecedented detail. Here, we present (i) a fast automated method to record and track the pose of individual larval zebrafish in a 3-D environment, applicable when accurate human labeling is not possible; (ii) a rich annotated dataset of 3-D larval poses for ethologists and the general zebrafish and machine learning community; and (iii) a technique to generate realistic, annotated larval images in different behavioral contexts. Using a three-camera system calibrated with refraction correction, we record diverse larval swims under free swimming conditions and in response to acoustic and optical stimuli. We then employ a convolutional neural network to estimate 3-D larval poses from video images. The network is trained against a set of synthetic larval images rendered using a 3-D physical model of larvae. This 3-D model samples from a distribution of realistic larval poses that we estimate a priori using a template-based pose estimation of a small number of swim bouts. Our network model, trained without any human annotation, performs larval pose estimation three orders of magnitude faster and with accuracy comparable to the template-based approach, capturing detailed kinematics of 3-D larval swims. It also applies accurately to other datasets collected under different imaging conditions and containing behavioral contexts not included in our training.

Schrödinger Dynamics and Berry Phase of Undulatory Locomotion

Spectral mode representations play an essential role in various areas of physics, from quantum mechanics to fluid turbulence, but they are not yet extensively used to characterize and describe the behavioral dynamics of living systems. Here, we show that mode-based linear models inferred from experimental live-imaging data can provide an accurate low-dimensional description of undulatory locomotion in worms, centipedes, robots, and snakes. By incorporating physical symmetries and known biological constraints into the dynamical model, we find that the shape dynamics are generically governed by Schrödinger equations in mode space. The eigenstates of the effective biophysical Hamiltonians and their adiabatic variations enable the efficient classification and differentiation of locomotion behaviors in natural, simulated, and robotic organisms using Grassmann distances and Berry phases. While our analysis focuses on a widely studied class of biophysical locomotion phenomena, the underlying approach generalizes to other physical or living systems that permit a mode representation subject to geometric shape constraints.

A robotic honeycomb for interaction with a honeybee colony

Robotic technologies have shown the capability to interact with living organisms and even to form integrated mixed societies composed of living and artificial agents. Biocompatible robots, incorporating sensing and actuation capable of generating and responding to relevant stimuli, can be a tool to study collective behaviors previously unattainable with traditional techniques. To investigate collective behaviors of the western honeybee (Apis mellifera), we designed a robotic system capable of observing and modulating the bee cluster using an array of thermal sensors and actuators. We initially integrated the system into a beehive populated with about 4000 bees for several months. The robotic system was able to observe the colony by continuously collecting spatiotemporal thermal profiles of the winter cluster. Furthermore, we found that our robotic device reliably modulated the superorganism's response to dynamic thermal stimulation, influencing its spatiotemporal reorganization. In addition, after identifying the thermal collapse of a colony, we used the robotic system in a "life-support" mode via its thermal actuators. Ultimately, we demonstrated a robotic device capable of autonomous closed-loop interaction with a cluster comprising thousands of individual bees. Such biohybrid societies open the door to investigation of collective behaviors that necessitate observing and interacting with the animals within a complete social context, as well as for potential applications in augmenting the survivability of these pollinators crucial to our ecosystems and our food supply.

Density of predating Asian hornets at hives disturbs the 3D flight performance of honey bees and decreases predation success

Automated 3D image-based tracking systems are new and promising devices to investigate the foraging behavior of flying animals with great accuracy and precision. 3D analyses can provide accurate assessments of flight performance in regard to speed, curvature, and hovering. However, there have been few applications of this technology in ecology, particularly for insects. We used this technology to analyze the behavioral interactions between the Western honey bee Apis mellifera and its invasive predator the Asian hornet, Vespa velutina nigrithorax. We investigated whether predation success could be affected by flight speed, flight curvature, and hovering of the Asian hornet and honey bees in front of one beehive. We recorded a total of 603,259 flight trajectories and 5175 predator-prey flight interactions leading to 126 successful predation events, representing 2.4% predation success. Flight speeds of hornets in front of hive entrances were much lower than that of their bee prey; in contrast to hovering capacity, while curvature range overlapped between the two species. There were large differences in speed, curvature, and hovering between the exit and entrance flights of honey bees. Interestingly, we found hornet density affected flight performance of both honey bees and hornets. Higher hornet density led to a decrease in the speed of honey bees leaving the hive, and an increase in the speed of honey bees entering the hive, together with more curved flight trajectories. These effects suggest some predator avoidance behavior by the bees. Higher honey bee flight curvature resulted in lower hornet predation success. Results showed an increase in predation success when hornet number increased up to 8 individuals, above which predation success decreased, likely due to competition among predators. Although based on a single colony, this study reveals interesting outcomes derived from the use of automated 3D tracking to derive accurate measures of individual behavior and behavioral interactions among flying species.

Behavioral variation across the days and lives of honey bees

In honey bee colonies, workers generally change tasks with age (from brood care, to nest work, to foraging). While these trends are well established, our understanding of how individuals distribute tasks during a day, and how individuals differ in their lifetime behavioral trajectories, is limited. Here, we use automated tracking to obtain long-term data on 4,100+ bees tracked continuously at 3 Hz, across an entire summer, and use behavioral metrics to compare behavior at different timescales. Considering single days, we describe how bees differ in space use, detection, and movement. Analyzing the behavior exhibited across their entire lives, we find consistent inter-individual differences in the movement characteristics of individuals. Bees also differ in how quickly they transition through behavioral space to ultimately become foragers, with fast-transitioning bees living the shortest lives. Our analysis framework provides a quantitative approach to describe individual behavioral variation within a colony from single days to entire lifetimes.

Long-term tracking and quantification of individual behavior in bumble bee colonies

Social insects are ecologically dominant and provide vital ecosystem services. It is critical to understand collective responses of social insects such as bees to ecological perturbations. However, studying behavior of individual insects across entire colonies and across timescales relevant for colony performance (i.e., days or weeks) remains a central challenge. Here, we describe an approach for long-term monitoring of individuals within multiple bumble bee (Bombus spp.) colonies that combines the complementary strengths of multiple existing methods. Specifically, we combine (a) automated monitoring, (b) fiducial tag tracking, and (c) pose estimation to quantify behavior across multiple colonies over a 48 h period. Finally, we demonstrate the benefits of this approach by quantifying an important but subtle behavior (antennal activity) in bumble bee colonies, and how this behavior is impacted by a common environmental stressor (a neonicotinoid pesticide).

Bayesian Multi-Targets Strategy to Track Apis mellifera Movements at Colony Level

Simple Summary

The number of honey bee, Apis mellifera L., colonies has reduced around the globe, and one potential cause is their unintended exposure to sublethal stressors such as agricultural pesticides. The quantification of such effects at colony level is a very complex task due to the innumerable collective activities done by the individual within colonies. Here, we present a Bayesian and computational approach capable of tracking the movements of bees within colonies, which allows the comparison of the collective activities of colonies that received bees previously exposed to uncontaminated diets or to diets containing sublethal concentrations of an agricultural pesticide (a commercial formulation containing the synthetic fungicides thiophanate-methyl and chlorothalonil). Our Bayesian tracking technique proved successful and superior to comparable algorithms, allowing the estimation of dynamical parameters such as entropy and kinetic energy. Our efforts demonstrated that fungicide-contaminated colonies behaved differently from uncontaminated colonies, as the former exhibited anticipated collective activities in peripheral hive areas and had reduced swarm entropy and kinetic energies. Such findings may facilitate the electronic monitoring of potential unintended effects in social pollinators, at colony level, mediated by environmental stressors (e.g., pesticides, electromagnetic fields, noise, and light intensities) alone or in combination.

Abstract

Interactive movements of bees facilitate the division and organization of collective tasks, notably when they need to face internal or external environmental challenges. Here, we present a Bayesian and computational approach to track the movement of several honey bee, Apis mellifera, workers at colony level. We applied algorithms that combined tracking and Kernel Density Estimation (KDE), allowing measurements of entropy and Probability Distribution Function (PDF) of the motion of tracked organisms. We placed approximately 200 recently emerged and labeled bees inside an experimental colony, which consists of a mated queen, approximately 1000 bees, and a naturally occurring beehive background. Before release, labeled bees were fed for one hour with uncontaminated diets or diets containing a commercial mixture of synthetic fungicides (thiophanate-methyl and chlorothalonil). The colonies were filmed (12 min) at the 1st hour, 5th and 10th days after the bees’ release. Our results revealed that the algorithm tracked the labeled bees with great accuracy. Pesticide-contaminated colonies showed anticipated collective activities in peripheral hive areas, far from the brood area, and exhibited reduced swarm entropy and energy values when compared to uncontaminated colonies. Collectively, our approach opens novel possibilities to quantify and predict potential alterations mediated by pollutants (e.g., pesticides) at the bee colony-level.

Automatic mapping of multiplexed social receptive fields by deep learning and GPU-accelerated 3D videography

Social interactions powerfully impact the brain and the body, but high-resolution descriptions of these important physical interactions and their neural correlates are lacking. Currently, most studies rely on labor-intensive methods such as manual annotation. Scalable and objective tracking methods are required to understand the neural circuits underlying social behavior. Here we describe a hardware/software system and analysis pipeline that combines 3D videography, deep learning, physical modeling, and GPU-accelerated robust optimization, with automatic analysis of neuronal receptive fields recorded in interacting mice. Our system ("3DDD Social Mouse Tracker") is capable of fully automatic multi-animal tracking with minimal errors (including in complete darkness) during complex, spontaneous social encounters, together with simultaneous electrophysiological recordings. We capture posture dynamics of multiple unmarked mice with high spatiotemporal precision (~2 mm, 60 frames/s). A statistical model that relates 3D behavior and neural activity reveals multiplexed 'social receptive fields' of neurons in barrel cortex. Our approach could be broadly useful for neurobehavioral studies of multiple animals interacting in complex low-light environments.

Beehive Air Sampling and Sensing Device Operation in Apicultural Applications-Methodological and Technical Aspects

The basis of effective beekeeping is the information about the state of the bee colony. A rich source of respective information is beehive air. This source may be explored by applying gas sensing. It allows for classifying bee colony states based on beehive air measurements. In this work, we discussed the essential aspects of beehive air sampling and sensing device operation in apicultural applications. They are the sampling method (diffusive vs. dynamic, temporal aspects), sampling system (sample probe, sampling point selection, sample conditioning unit and sample delivery system) and device operation mode ('exposure-cleaning' operation). It was demonstrated how factors associated with the beehive, bee colony and ambient environment define prerequisites for these elements of the measuring instrument. These requirements have to be respected in order to assure high accuracy of measurement and high-quality information. The presented results are primarily based on the field measurement study performed in summer 2020, in three apiaries, in various meteorological conditions. Two exemplars of a prototype gas sensing device were used. These sensor devices were constructed according to our original concept.