Context-dependent stimulus presentation to freely moving animals in 3D

Insect visuomotor delay adjustments in group flight support swarm cohesion

Flying insects routinely demonstrate coordinated flight in crowded assemblies despite strict communication and processing constraints. This study experimentally records multiple flying insects tracking a moving visual stimulus. System identification techniques are used to robustly identify the tracking dynamics, including a visuomotor delay. The population delay distributions are quantified for solo and group behaviors. An interconnected visual swarm model incorporating heterogeneous delays is developed, and bifurcation analysis and swarm simulation are applied to assess swarm stability under the delays. The experiment recorded 450 insect trajectories and quantified visual tracking delay variation. Solitary tasks showed a 30ms average delay and standard deviation of 50ms, while group behaviors show a 15ms average and 8ms standard deviation. Analysis and simulation indicate that the delay adjustments during group flight support swarm formation and center stability, and are robust to measurement noise. These results quantify the role of visuomotor delay heterogeneity in flying insects and their role in supporting swarm cohesion through implicit communication.

Experimental identification of individual insect visual tracking delays in free flight and their effects on visual swarm patterns

Insects are model systems for swarming robotic agents, yet engineered descriptions do not fully explain the mechanisms by which they provide onboard sensing and feedback to support such motions; in particular, the exact value and population distribution of visuomotor processing delays are not yet quantified, nor the effect of such delays on a visually-interconnected swarm. This study measures untethered insects performing a solo in-flight visual tracking task and applies system identification techniques to build an experimentally-consistent model of the visual tracking behaviors, and then integrates the measured experimental delay and its variation into a visually interconnected swarm model to develop theoretical and simulated solutions and stability limits. The experimental techniques include the development of a moving visual stimulus and real-time multi camera based tracking system called VISIONS (Visual Input System Identification from Outputs of Naturalistic Swarms) providing the capability to recognize and simultaneously track both a visual stimulus (input) and an insect at a frame rate of 60-120 Hz. A frequency domain analysis of honeybee tracking trajectories is conducted via fast Fourier and Chirp Z transforms, identifying a coherent linear region and its model structure. The model output is compared in time and frequency domain simulations. The experimentally measured delays are then related to probability density functions, and both the measured delays and their distribution are incorporated as inter-agent interaction delays in a second order swarming dynamics model. Linear stability and bifurcation analysis on the long range asymptotic behavior is used to identify delay distributions leading to a family of solutions with stable and unstable swarm center of mass (barycenter) locations. Numerical simulations are used to verify these results with both continuous and measured distributions. The results of this experiment quantify a model structure and temporal lag (transport delay) in the closed loop dynamics, and show that this delay varies across 50 individuals from 5-110ms, with an average delay of 22ms and a standard deviation of 40ms. When analyzed within the swarm model, the measured delays support a diversity of solutions and indicate an unstable barycenter.

Behavioural analysis of swarming mosquitoes reveals higher hearing sensitivity than previously measured with electrophysiology methods

Mosquitoes of many species mate in station-keeping swarms. Mating chases ensue as soon as a male detects the flight tones of a female with his auditory organs. Previous studies of hearing thresholds have mainly used electrophysiological methods that prevent the mosquito from flying naturally. The main aim of this study was to quantify behaviourally the sound-level threshold at which males can hear females. Free-flying male Anopheles coluzzii were released in a large arena (∼2 m high x 2 m x 1 m) with a conspicuous object on the ground that stimulates swarming behaviour. Males were exposed to a range of natural and synthetic played-back sounds of female flight. We monitored the responses of males and their distance to the speaker by recording changes in their wingbeat frequency and angular speed. We show that the mean male behavioural threshold of particle-velocity hearing lies between 13-20 dB SVL (95%-CI). A conservative estimate of 20 dB SVL (i.e.,<0.5 µm/s particle velocity) is already 12 to 26 dB lower than most of the published electrophysiological measurements from the Johnston's organ. In addition, we suggest that 1) the first harmonic of female flight-sound is sufficient for males to detect her presence, 2) males respond with a greater amplitude to single-female sounds than to the sound of a group of females and 3) the response of males to the playback of the flight sound of a live female is the same as that of a recorded sound of constant frequency and amplitude.

Three-Dimensional Tracking of Multiple Small Insects by a Single Camera

Many systems to monitor insect behavior have been developed recently. Yet most of these can only detect two-dimensional behavior for convenient analysis and exclude other activities, such as jumping or flying. Therefore, the development of a three-dimensional (3D) monitoring system is necessary to investigate the 3D behavior of insects. In such a system, multiple-camera setups are often used to accomplish this purpose. Here, a system with a single camera for tracking small insects in a 3D space is proposed, eliminating the synchronization problems that typically occur when multiple cameras are instead used. With this setup, two other images are obtained via mirrors fixed at other viewing angles. Using the proposed algorithms, the tracking accuracy of five individual drain flies, Clogmia albipunctata (Williston) (Diptera: Psychodidae), flitting about in a spherical arena (78 mm in diameter) is as high as 98.7%, whereas the accuracy of 10 individuals is 96.3%. With this proposed method, the 3D trajectory monitoring experiments of insects can be performed more efficiently.

Mosquito sound communication: are male swarms loud enough to attract females?

Given the unsurpassed sound sensitivity of mosquitoes among arthropods and the sound source power required for long-range hearing, we investigated the distance over which female mosquitoes detect species-specific cues in the sound of station-keeping mating swarms. A common misunderstanding, that mosquitoes cannot hear at long range because their hearing organs are 'particle-velocity' receptors, has clouded the fact that particle velocity is an intrinsic component of sound whatever the distance to the sound source. We exposed free-flying Anopheles coluzzii females to pre-recorded sounds of male An. coluzzii and An. gambiae s.s. swarms over a range of natural sound levels. Sound levels tested were related to equivalent distances between the female and the swarm for a given number of males, enabling us to infer distances over which females might hear large male swarms. We show that females do not respond to swarm sound up to 48 dB sound pressure level (SPL) and that louder SPLs are not ecologically relevant for a swarm. Considering that swarms are the only mosquito sound source that would be loud enough to be heard at long range, we conclude that inter-mosquito acoustic communication is restricted to close-range pair interactions. We also showed that the sensitivity to sound in free-flying males is much enhanced compared to that of tethered ones.

The Antarium: A Reconstructed Visual Reality Device for Ant Navigation Research

We constructed a large projection device (the Antarium) with 20,000 UV-Blue-Green LEDs that allows us to present tethered ants with views of their natural foraging environment. The ants walk on an air-cushioned trackball, their movements are registered and can be fed back to the visual panorama. Views are generated in a 3D model of the ants’ environment so that they experience the changing visual world in the same way as they do when foraging naturally. The Antarium is a biscribed pentakis dodecahedron with 55 facets of identical isosceles triangles. The length of the base of the triangles is 368 mm resulting in a device that is roughly 1 m in diameter. Each triangle contains 361 blue/green LEDs and nine UV LEDs. The 55 triangles of the Antarium have 19,855 Green and Blue pixels and 495 UV pixels, covering 360° azimuth and elevation from −50° below the horizon to +90° above the horizon. The angular resolution is 1.5° for Green and Blue LEDs and 6.7° for UV LEDs, offering 65,536 intensity levels at a flicker frequency of more than 9,000 Hz and a framerate of 190 fps. Also, the direction and degree of polarisation of the UV LEDs can be adjusted through polarisers mounted on the axles of rotary actuators. We build 3D models of the natural foraging environment of ants using purely camera-based methods. We reconstruct panoramic scenes at any point within these models, by projecting panoramic images onto six virtual cameras which capture a cube-map of images to be projected by the LEDs of the Antarium. The Antarium is a unique instrument to investigate visual navigation in ants. In an open loop, it allows us to provide ants with familiar and unfamiliar views, with completely featureless visual scenes, or with scenes that are altered in spatial or spectral composition. In closed-loop, we can study the behavior of ants that are virtually displaced within their natural foraging environment. In the future, the Antarium can also be used to investigate the dynamics of navigational guidance and the neurophysiological basis of ant navigation in natural visual environments.

Movement-Related Signals in Sensory Areas: Roles in Natural Behavior

Recent studies have demonstrated prominent and widespread movement-related signals in the brain of head-fixed mice, even in primary sensory areas. However, it is still unknown what role these signals play in sensory processing. Why are these sensory areas 'contaminated' by movement signals? During natural behavior, animals actively acquire sensory information as they move through the environment and use this information to guide ongoing actions. In this context, movement-related signals could allow sensory systems to predict self-induced sensory changes and extract additional information about the environment. In this review we summarize recent findings on the presence of movement-related signals in sensory areas and discuss how their study, in the context of natural freely moving behaviors, could advance models of sensory processing.

Gene Regulation and Species-Specific Evolution of Free Flight Odor Tracking in Drosophila

The flying ability of insects has coevolved with the development of organs necessary to take-off from the ground, generate, and modulate lift during flight in complex environments. Flight orientation to the appropriate food source and mating partner depends on the perception and integration of multiple chemical signals. We used a wind tunnel-based assay to investigate the natural and molecular evolution of free flight odor tracking in Drosophila. First, the comparison of female and male flies of several populations and species revealed substantial sex-, inter-, and intra-specific variations for distinct flight features. In these flies, we compared the molecular structure of desat1, a fast-evolving gene involved in multiple aspects of Drosophila pheromonal communication. We manipulated desat1 regulation and found that both neural and nonneural tissues affect distinct flight features. Together, our data suggest that desat1 is one of the genes involved in the evolution of free-flight odor tracking behaviors in Drosophila.

Free flight odor tracking in Drosophila: Effect of wing chemosensors, sex and pheromonal gene regulation

The evolution of powered flight in insects had major consequences for global biodiversity and involved the acquisition of adaptive processes allowing individuals to disperse to new ecological niches. Flies use both vision and olfactory input from their antennae to guide their flight; chemosensors on fly wings have been described, but their function remains mysterious. We studied Drosophila flight in a wind tunnel. By genetically manipulating wing chemosensors, we show that these structures play an essential role in flight performance with a sex-specific effect. Pheromonal systems are also involved in Drosophila flight guidance: transgenic expression of the pheromone production and detection gene, desat1, produced low, rapid flight that was absent in control flies. Our study suggests that the sex-specific modulation of free-flight odor tracking depends on gene expression in various fly tissues including wings and pheromonal-related tissues.

Mapping and cracking sensorimotor circuits in genetic model organisms

One central goal of systems neuroscience is to understand how neural circuits implement the computations that link sensory inputs to behavior. Work combining electrophysiological and imaging-based approaches to measure neural activity with pharmacological and electrophysiological manipulations has provided fundamental insights. More recently, genetic approaches have been used to monitor and manipulate neural activity, opening up new experimental opportunities and challenges. Here, we discuss issues associated with applying genetic approaches to circuit dissection in sensorimotor transformations, outlining important considerations for experimental design and considering how modeling can complement experimental approaches.

Embodied linearity of speed control in Drosophila melanogaster

Fruitflies regulate flight speed by adjusting their body angle. To understand how low-level posture control serves an overall linear visual speed control strategy, we visually induced free-flight acceleration responses in a wind tunnel and measured the body kinematics using high-speed videography. Subsequently, we reverse engineered the transfer function mapping body pitch angle onto flight speed. A linear model is able to reproduce the behavioural data with good accuracy. Our results show that linearity in speed control is realized already at the level of body posture-mediated speed control and is therefore embodied at the level of the complex aerodynamic mechanisms of body and wings. Together with previous results, this study reveals the existence of a linear hierarchical control strategy, which can provide relevant control principles for biomimetic implementations, such as autonomous flying micro air vehicles.

A low-cost, portable, micro-controlled device for multi-channel LED visual stimulation

Light emitting diodes (LEDs) are extensively used as light sources to investigate visual and visually related function and dysfunction. Here, we describe the design of a compact, low-cost, stand-alone LED-based system that enables the configuration, storage and presentation of elaborate visual stimulation paradigms. The core functionality of this system is provided by a microcontroller whose ultra-low power consumption makes it well suited for long lasting battery applications. The effective use of hardware resources is managed by multi-layered architecture software that provides an intuitive and user-friendly interface. In the configuration mode, different stimulation sequences can be created and memorized for ten channels, independently. LED-driving current output can be set either as continuous or pulse modulated, up to 500 Hz, by duty cycle adjustments. In run mode, multiple-channel stimulus sequences are automatically applied according to the pre-programmed protocol. Steady state visual evoked potentials were successfully recorded in five subjects with no visible electromagnetic interferences from the stimulator, demonstrating the efficacy of combining our prototyped equipment with electrophysiological techniques. Finally, we discuss a number of possible improvements for future development of our project.

Computer-assisted 3D kinematic analysis of all leg joints in walking insects

High-speed video can provide fine-scaled analysis of animal behavior. However, extracting behavioral data from video sequences is a time-consuming, tedious, subjective task. These issues are exacerbated where accurate behavioral descriptions require analysis of multiple points in three dimensions. We describe a new computer program written to assist a user in simultaneously extracting three-dimensional kinematics of multiple points on each of an insect's six legs. Digital video of a walking cockroach was collected in grayscale at 500 fps from two synchronized, calibrated cameras. We improved the legs' visibility by painting white dots on the joints, similar to techniques used for digitizing human motion. Compared to manual digitization of 26 points on the legs over a single, 8-second bout of walking (or 106,496 individual 3D points), our software achieved approximately 90% of the accuracy with 10% of the labor. Our experimental design reduced the complexity of the tracking problem by tethering the insect and allowing it to walk in place on a lightly oiled glass surface, but in principle, the algorithms implemented are extensible to free walking. Our software is free and open-source, written in the free language Python and including a graphical user interface for configuration and control. We encourage collaborative enhancements to make this tool both better and widely utilized.

Multi-camera real-time three-dimensional tracking of multiple flying animals

Automated tracking of animal movement allows analyses that would not otherwise be possible by providing great quantities of data. The additional capability of tracking in real time--with minimal latency--opens up the experimental possibility of manipulating sensory feedback, thus allowing detailed explorations of the neural basis for control of behaviour. Here, we describe a system capable of tracking the three-dimensional position and body orientation of animals such as flies and birds. The system operates with less than 40 ms latency and can track multiple animals simultaneously. To achieve these results, a multi-target tracking algorithm was developed based on the extended Kalman filter and the nearest neighbour standard filter data association algorithm. In one implementation, an 11-camera system is capable of tracking three flies simultaneously at 60 frames per second using a gigabit network of nine standard Intel Pentium 4 and Core 2 Duo computers. This manuscript presents the rationale and details of the algorithms employed and shows three implementations of the system. An experiment was performed using the tracking system to measure the effect of visual contrast on the flight speed of Drosophila melanogaster. At low contrasts, speed is more variable and faster on average than at high contrasts. Thus, the system is already a useful tool to study the neurobiology and behaviour of freely flying animals. If combined with other techniques, such as 'virtual reality'-type computer graphics or genetic manipulation, the tracking system would offer a powerful new way to investigate the biology of flying animals.

Behavioural system identification of visual flight speed control in Drosophila melanogaster

Behavioural control in many animals involves complex mechanisms with intricate sensory-motor feedback loops. Modelling allows functional aspects to be captured without relying on a description of the underlying complex, and often unknown, mechanisms. A wide range of engineering techniques are available for modelling, but their ability to describe time-continuous processes is rarely exploited to describe sensory-motor control mechanisms in biological systems. We performed a system identification of visual flight speed control in the fruitfly Drosophila, based on an extensive dataset of open-loop responses previously measured under free flight conditions. We identified a second-order under-damped control model with just six free parameters that well describes both the transient and steady-state characteristics of the open-loop data. We then used the identified control model to predict flight speed responses after a visual perturbation under closed-loop conditions and validated the model with behavioural measurements performed in free-flying flies under the same closed-loop conditions. Our system identification of the fruitfly's flight speed response uncovers the high-level control strategy of a fundamental flight control reflex without depending on assumptions about the underlying physiological mechanisms. The results are relevant for future investigations of the underlying neuromotor processing mechanisms, as well as for the design of biomimetic robots, such as micro-air vehicles.

Visual control of flight speed in Drosophila melanogaster

Flight control in insects depends on self-induced image motion (optic flow), which the visual system must process to generate appropriate corrective steering maneuvers. Classic experiments in tethered insects applied rigorous system identification techniques for the analysis of turning reactions in the presence of rotating pattern stimuli delivered in open-loop. However, the functional relevance of these measurements for visual free-flight control remains equivocal due to the largely unknown effects of the highly constrained experimental conditions. To perform a systems analysis of the visual flight speed response under free-flight conditions, we implemented a 'one-parameter open-loop' paradigm using 'TrackFly' in a wind tunnel equipped with real-time tracking and virtual reality display technology. Upwind flying flies were stimulated with sine gratings of varying temporal and spatial frequencies, and the resulting speed responses were measured from the resulting flight speed reactions. To control flight speed, the visual system of the fruit fly extracts linear pattern velocity robustly over a broad range of spatio-temporal frequencies. The speed signal is used for a proportional control of flight speed within locomotor limits. The extraction of pattern velocity over a broad spatio-temporal frequency range may require more sophisticated motion processing mechanisms than those identified in flies so far. In Drosophila, the neuromotor pathways underlying flight speed control may be suitably explored by applying advanced genetic techniques, for which our data can serve as a baseline. Finally, the high-level control principles identified in the fly can be meaningfully transferred into a robotic context, such as for the robust and efficient control of autonomous flying micro air vehicles.

Vision egg: an open-source library for realtime visual stimulus generation

Modern computer hardware makes it possible to produce visual stimuli in ways not previously possible. Arbitrary scenes, from traditional sinusoidal gratings to naturalistic 3D scenes can now be specified on a frame-by-frame basis in realtime. A programming library called the Vision Egg that aims to make it easy to take advantage of these innovations. The Vision Egg is a free, open-source library making use of OpenGL and written in the high-level language Python with extensions in C. Careful attention has been paid to the issues of luminance and temporal calibration, and several interfacing techniques to input devices such as mice, movement tracking systems, and digital triggers are discussed. Together, these make the Vision Egg suitable for many psychophysical, electrophysiological, and behavioral experiments. This software is available for free download at visionegg.org.

Software techniques for two- and three-dimensional kinematic measurements of biological and biomimetic systems

Researchers studying aspects of locomotion or movement in biological and biomimetic systems commonly use video or stereo video recordings to quantify the behaviour of the system in question, often with an emphasis on measures of position, velocity and acceleration. However, despite the apparent simplicity of video analysis, it can require substantial investment of time and effort, even when performed with adequate software tools. This paper reviews the underlying principles of video and stereo video analysis as well as its automation and is accompanied by fully functional and freely available software implementation.