Sensory fusion in the hoverfly righting reflex

We study how falling hoverflies use sensory cues to trigger appropriate roll righting behavior. Before being released in a free fall, flies were placed upside-down with their legs contacting the substrate. The prior leg proprioceptive information about their initial orientation sufficed for the flies to right themselves properly. However, flies also use visual and antennal cues to recover faster and disambiguate sensory conflicts. Surprisingly, in one of the experimental conditions tested, hoverflies flew upside-down while still actively flapping their wings. In all the other conditions, flies were able to right themselves using two roll dynamics: fast ([Formula: see text]50ms) and slow ([Formula: see text]110ms) in the presence of consistent and conflicting cues, respectively. These findings suggest that a nonlinear sensory integration of the three types of sensory cues occurred. A ring attractor model was developed and discussed to account for this cue integration process.

Accommodating unobservability to control flight attitude with optic flow

Attitude control is an essential flight capability. Whereas flying robots commonly rely on accelerometers¹ for estimating attitude, flying insects lack an unambiguous sense of gravity^2,3. Despite the established role of several sense organs in attitude stabilization^3-5, the dependence of flying insects on an internal gravity direction estimate remains unclear. Here we show how attitude can be extracted from optic flow when combined with a motion model that relates attitude to acceleration direction. Although there are conditions such as hover in which the attitude is unobservable, we prove that the ensuing control system is still stable, continuously moving into and out of these conditions. Flying robot experiments confirm that accommodating unobservability in this manner leads to stable, but slightly oscillatory, attitude control. Moreover, experiments with a bio-inspired flapping-wing robot show that residual, high-frequency attitude oscillations from flapping motion improve observability. The presented approach holds a promise for robotics, with accelerometer-less autopilots paving the road for insect-scale autonomous flying robots⁶. Finally, it forms a hypothesis on insect attitude estimation and control, with the potential to provide further insight into known biological phenomena^5,7,8 and to generate new predictions such as reduced head and body attitude variance at higher flight speeds⁹.

Recovery mechanisms in the dragonfly righting reflex

Insects have evolved sophisticated reflexes to right themselves in mid-air. Their recovery mechanisms involve complex interactions among the physical senses, muscles, body, and wings, and they must obey the laws of flight. We sought to understand the key mechanisms involved in dragonfly righting reflexes and to develop physics-based models for understanding the control strategies of flight maneuvers. Using kinematic analyses, physical modeling, and three-dimensional flight simulations, we found that a dragonfly uses left-right wing pitch asymmetry to roll its body 180 degrees to recover from falling upside down in ~200 milliseconds. Experiments of dragonflies with blocked vision further revealed that this rolling maneuver is initiated by their ocelli and compound eyes. These results suggest a pathway from the dragonfly's visual system to the muscles regulating wing pitch that underly the recovery. The methods developed here offer quantitative tools for inferring insects' internal actions from their acrobatics, and are applicable to a broad class of natural and robotic flying systems.

Autonomous Flying With Neuromorphic Sensing

Autonomous flight for large aircraft appears to be within our reach. However, launching autonomous systems for everyday missions still requires an immense interdisciplinary research effort supported by pointed policies and funding. We believe that concerted endeavors in the fields of neuroscience, mathematics, sensor physics, robotics, and computer science are needed to address remaining crucial scientific challenges. In this paper, we argue for a bio-inspired approach to solve autonomous flying challenges, outline the frontier of sensing, data processing, and flight control within a neuromorphic paradigm, and chart directions of research needed to achieve operational capabilities comparable to those we observe in nature. One central problem of neuromorphic computing is learning. In biological systems, learning is achieved by adaptive and relativistic information acquisition characterized by near-continuous information retrieval with variable rates and sparsity. This results in both energy and computational resource savings being an inspiration for autonomous systems. We consider pertinent features of insect, bat and bird flight behavior as examples to address various vital aspects of autonomous flight. Insects exhibit sophisticated flight dynamics with comparatively reduced complexity of the brain. They represent excellent objects for the study of navigation and flight control. Bats and birds enable more complex models of attention and point to the importance of active sensing for conducting more complex missions. The implementation of neuromorphic paradigms for autonomous flight will require fundamental changes in both traditional hardware and software. We provide recommendations for sensor hardware and processing algorithm development to enable energy efficient and computationally effective flight control.

How do hoverflies use their righting reflex?

When taking off from a sloping surface, flies have to reorient themselves dorsoventrally and stabilize their body by actively controlling their flapping wings. We have observed that righting is achieved solely by performing a rolling manoeuvre. How flies manage to do this has not yet been elucidated. It was observed here for the first time that hoverfly reorientation is entirely achieved within 6 wingbeats (48.8 ms) at angular roll velocities of up to 10×103 deg s-1 and that the onset of their head rotation consistently follows that of their body rotation after a time lag of 16 ms. The insects' body roll was found to be triggered by the asymmetric wing stroke amplitude, as expected. The righting process starts immediately with the first wingbeat and seems unlikely to depend on visual feedback. A dynamic model for the fly's righting reflex is presented, which accounts for the head/body movements and the time lag recorded in these experiments. This model consists of a closed-loop control of the body roll, combined with a feedforward control of the head/body angle. During the righting manoeuvre, a strong coupling seems to exist between the activation of the halteres (which measure the body's angular speed) and the gaze stabilization reflex. These findings again confirm the fundamental role played by the halteres in both body and head stabilization processes.

Role of the light source position in freely falling hoverflies' stabilization performances

The stabilization of plummeting hoverflies was filmed and analysed in terms of their wingbeat initiation times as well as the crash and stabilization rates. The flies experienced near-weightlessness for a period of time that depended on their ability to counteract the free fall by triggering their wingbeats. In this paradigm, hoverflies' flight stabilization strategies were investigated here for the first time under two different positions of the light source (overhead and bottom lighting). The crash rates were higher in bottom lighting conditions than with top lighting. In addition, adding a texture to the walls reduced the crash rates only in the overhead lighting condition. The position of the lighting also significantly affected both the stabilization rates and the time taken by the flies to stabilize, which decreased and increased under bottom lighting conditions, respectively, whereas textured walls increased the stabilization rates under both lighting conditions. These results support the idea that flies may mainly base their flight control strategy on visual cues and particularly that the light distribution in the visual field may provide reliable, efficient cues for estimating their orientation with respect to an allocentric reference frame. In addition, the finding that the hoverflies' optic flow-based motion detection ability is affected by the position of the light source in their visual field suggests the occurrence of interactions between movement perception and this visual vertical perception process.

Image statistics of the environment surrounding freely behaving hoverflies

Natural scenes are not as random as they might appear, but are constrained in both space and time. The 2-dimensional spatial constraints can be described by quantifying the image statistics of photographs. Human observers perceive images with naturalistic image statistics as more pleasant to view, and both fly and vertebrate peripheral and higher order visual neurons are tuned to naturalistic image statistics. However, for a given animal, what is natural differs depending on the behavior, and even if we have a broad understanding of image statistics, we know less about the scenes relevant for particular behaviors. To mitigate this, we here investigate the image statistics surrounding Episyrphus balteatus hoverflies, where the males hover in sun shafts created by surrounding trees, producing a rich and dense background texture and also intricate shadow patterns on the ground. We quantified the image statistics of photographs of the ground and the surrounding panorama, as the ventral and lateral visual field is particularly important for visual flight control, and found differences in spatial statistics in photos where the hoverflies were hovering compared to where they were flying. Our results can, in the future, be used to create more naturalistic stimuli for experimenter-controlled experiments in the laboratory.

Haltere removal alters responses to gravity in standing flies

Animals detect the force of gravity with multiple sensory organs, from subcutaneous receptors at body joints to specialized sensors like the vertebrate inner ear. The halteres of flies, specialized mechanoreceptive organs derived from hindwings, are known to detect body rotations during flight, and some groups of flies also oscillate their halteres while walking. The dynamics of halteres are such that they could act as gravity detectors for flies standing on substrates, but their utility during non-flight behaviors is not known. We observed the behaviors of intact and haltere-ablated flies during walking and during perturbations in which the acceleration due to gravity suddenly changed. We found that intact halteres are necessary for flies to maintain normal walking speeds on vertical surfaces and to respond to sudden changes in gravity. Our results suggest that halteres can serve multiple sensory purposes during different behaviors, expanding their role beyond their canonical use in flight.

Visual approach computation in feeding hoverflies

On warm sunny days, female hoverflies are often observed feeding from a wide range of wild and cultivated flowers. In doing so, hoverflies serve a vital role as alternative pollinators, and are suggested to be the most important pollinators after bees and bumblebees. Unless the flower hoverflies are feeding from is large, they do not readily share the space with other insects, but instead opt to leave if another insect approaches. We used high-speed videography followed by 3D reconstruction of flight trajectories to quantify how female Eristalis hoverflies respond to approaching bees, wasps and two different hoverfly species. We found that, in 94% of the interactions, the occupant female left the flower when approached by another insect. We found that compared with spontaneous take-offs, the occupant hoverfly's escape response was performed at ∼3 times higher speed (spontaneous take-off at 0.2±0.05 m s⁻¹ compared with 0.55±0.08 m s⁻¹ when approached by another Eristalis). The hoverflies tended to take off upward and forward, while taking the incomer's approach angle into account. Intriguingly, we found that, when approached by wasps, the occupant Eristalis took off at a higher speed and when the wasp was further away. This suggests that feeding hoverflies may be able to distinguish these predators, demanding impressive visual capabilities. Our results, including quantification of the visual information available before occupant take-off, provide important insight into how freely behaving hoverflies perform escape responses from competitors and predators (e.g. wasps) in the wild.

Highlighted Article: Reconstruction of the take-off and flight of feeding female hoverflies when approached by other insects, and quantification of visual parameters, reveals how freely behaving hoverflies perform escape responses from competitors and predators in the wild.

Modeling visual-based pitch, lift and speed control strategies in hoverflies

To avoid crashing onto the floor, a free falling fly needs to trigger its wingbeats quickly and control the orientation of its thrust accurately and swiftly to stabilize its pitch and hence its speed. Behavioural data have suggested that the vertical optic flow produced by the fall and crossing the visual field plays a key role in this anti-crash response. Free fall behavior analyses have also suggested that flying insect may not rely on graviception to stabilize their flight. Based on these two assumptions, we have developed a model which accounts for hoverflies´ position and pitch orientation recorded in 3D with a fast stereo camera during experimental free falls. Our dynamic model shows that optic flow-based control combined with closed-loop control of the pitch suffice to stabilize the flight properly. In addition, our model sheds a new light on the visual-based feedback control of fly´s pitch, lift and thrust. Since graviceptive cues are possibly not used by flying insects, the use of a vertical reference to control the pitch is discussed, based on the results obtained on a complete dynamic model of a virtual fly falling in a textured corridor. This model would provide a useful tool for understanding more clearly how insects may or not estimate their absolute attitude.

On the basis of vision-based feedback control of optic flow occurring during insects’ flight, we developed a dynamic model that accounts for the pitch orientation and speed in plummeting flies. We compared the hoverflies’ responses with our model and showed that an optic-flow based control strategy can be used to correct the initial pitch misorientation caused by the free fall situation. To complete the model, we combined the closed-loop control of the vertical optic flow with an additional feedback control loop based on the value of the absolute pitch orientation. The need for this measurement to stabilize the pitch orientation raises the question as whether this is also the case in dipterans. After ruling out the possibility that insects may use gravity acceleration cues to control their flight, for which no experimental evidence has been found so far, we discussed the three main sensory processes possibly involved in in their ability to control their attitude. Our model provides a useful tool for studying the various sensory processes possibly involved in dipterans’ flight stabilization abilities as well as the interactions between these processes.