Neuronal matched filters for optic flow processing in flying insects

Optic flow based spatial vision in insects

The optic flow, i.e., the displacement of retinal images of objects in the environment induced by self-motion, is an important source of spatial information, especially for fast-flying insects. Spatial information over a wide range of distances, from the animal's immediate surroundings over several hundred metres to kilometres, is necessary for mediating behaviours, such as landing manoeuvres, collision avoidance in spatially complex environments, learning environmental object constellations and path integration in spatial navigation. To facilitate the processing of spatial information, the complexity of the optic flow is often reduced by active vision strategies. These result in translations and rotations being largely separated by a saccadic flight and gaze mode. Only the translational components of the optic flow contain spatial information. In the first step of optic flow processing, an array of local motion detectors provides a retinotopic spatial proximity map of the environment. This local motion information is then processed in parallel neural pathways in a task-specific manner and used to control the different components of spatial behaviour. A particular challenge here is that the distance information extracted from the optic flow does not represent the distances unambiguously, but these are scaled by the animal's speed of locomotion. Possible ways of coping with this ambiguity are discussed.

The lobula plate is exclusive to insects

Just one superorder of insects is known to possess a neuronal network that mediates extremely rapid reactions in flight in response to changes in optic flow. Research on the identity and functional organization of this network has over the course of almost half a century focused exclusively on the order Diptera, a member of the approximately 300-million-year-old clade Holometabola defined by its mode of development. However, it has been broadly claimed that the pivotal neuropil containing the network, the lobula plate, originated in the Cambrian before the divergence of Hexapoda and Crustacea from a mandibulate ancestor. This essay defines the traits that designate the lobula plate and argues against a homologue in Crustacea. It proposes that the origin of the lobula plate is relatively recent and may relate to the origin of flight.

A biohybrid fly-robot interface system that performs active collision avoidance

We have designed a bio-hybrid fly-robot interface (FRI) to study sensorimotor control in insects. The FRI consists of a miniaturized recording platform mounted on a two-wheeled robot and is controlled by the neuronal spiking activity of an identified visual interneuron, the blowfly H1-cell. For a given turning radius of the robot, we found a proportional relationship between the spike rate of the H1-cell and the relative distance of the FRI from the patterned wall of an experimental arena. Under closed-loop conditions during oscillatory forward movements biased towards the wall, collision avoidance manoeuvres were triggered whenever the H1-cell spike rate exceeded a certain threshold value. We also investigated the FRI behaviour in corners of the arena. The ultimate goal is to enable autonomous and energy-efficient manoeuvrings of the FRI within arbitrary visual environments.

Abdominal-Waving Control of Tethered Bumblebees Based on Sarsa With Transformed Reward

Cyborg insects have attracted great attention as the flight performance they have is incomparable by micro aerial vehicles and play a critical role in supporting extensive applications. Approaches to construct cyborg insects consist of two major issues: 1) the stimulating paradigm and 2) the control policy. At present, most cyborg insects are constructed based on invasive methods, requiring the implantation of electrodes into neural or muscle systems, which would harm the insects. As the control policy is basically manual control, the shortcomings of which lie in the requirement of excessive amount of experiments and focused attention. This paper presents the design and implementation of a noninvasive and much safer cyborg insect system based on visual stimulation. The tethered paradigm is adopted here and we look at controlling the flight behavior of bumblebees, especially the abdominal-waving behavior, in the context of a model-free reinforcement learning problem. The problem is formulated as a finite and deterministic Markov decision process, where the agent is designed to change the abdominal-waving behavior from the initial state to the target state. Sarsa with transformed reward function which can speed up the learning process is employed to learn the optimal control policy. Learned policies are compared to the stochastic one by evaluating the results of ten bumblebees, demonstrating that abdominal-waving state can be modulated to approximate the target state quickly with small deviation.

Bi-directional Control of Walking Behavior by Horizontal Optic Flow Sensors

Moving animals experience constant sensory feedback, such as panoramic image shifts on the retina, termed optic flow. Underlying neuronal signals are thought to be important for exploratory behavior by signaling unintended course deviations and by providing spatial information about the environment [1, 2]. Particularly in insects, the encoding of self-motion-related optic flow is well understood [1-5]. However, a gap remains in understanding how the associated neuronal activity controls locomotor trajectories. In flies, visual projection neurons belonging to two groups encode panoramic horizontal motion: horizontal system (HS) cells respond with depolarization to front-to-back motion and hyperpolarization to the opposite direction [6, 7], and other neurons have the mirror-symmetrical response profile [6, 8, 9]. With primarily monocular sensitivity, the neurons' responses are ambiguous for different rotational and translational self-movement components. Such ambiguities can be greatly reduced by combining signals from both eyes [10-12] to determine turning and movement speed [13-16]. Here, we explore the underlying functional logic by optogenetic HS cell manipulation in tethered walking Drosophila. We show that de- and hyperpolarization evoke opposite turning behavior, indicating that both direction-selective signals are transmitted to descending pathways for course control. Further experiments reveal a negative effect of bilaterally symmetric de- and hyperpolarization on walking velocity. Our results are therefore consistent with a functional architecture in which the HS cells' membrane potential influences walking behavior bi-directionally via two decelerating pathways.

Small fruit flies sacrifice temporal acuity to maintain contrast sensitivity

Holometabolous insects, like fruit flies, grow primarily during larval development. Scarce larval feeding is common in nature and generates smaller adults. Despite the importance of vision to flies, eye size scales proportionately with body size, and smaller eyes confer poorer vision due to smaller optics. Variable larval feeding, therefore, causes within-species differences in visual processing, which have gone largely unnoticed due to ad libitum feeding in the lab that results in generally large adults. Do smaller eyes have smaller ommatidial lenses, reducing sensitivity, or broader inter-ommatidial angles, reducing acuity? And to what extent might neural processes adapt to these optical challenges with temporal and spatial summation? To understand this in the fruit fly, we generated a distribution of body lengths (1.67-2.34 mm; n = 24) and eye lengths (0.33-0.44 mm; n = 24), resembling the distribution of wild-caught flies, by removing larvae from food during their third instar. We find smaller eyes (0.19 vs.0.07 mm²) have substantially fewer (978 vs. 540, n = 45) and smaller ommatidia (222 vs. 121 μm²;n = 45) separated by slightly wider inter-ommatidial angles (4.5 vs.5.5°; n = 34). This corresponds to a greater loss in contrast sensitivity (<50%) than spatial acuity (<20%). Using a flight arena and psychophysics paradigm, we find that smaller flies lose little spatial acuity (0.126 vs. 0.118CPD; n = 45), and recover contrast sensitivity (2.22 for both; n = 65) by sacrificing temporal acuity (26.3 vs. 10.8Hz; n = 112) at the neural level. Therefore, smaller flies sacrifice contrast sensitivity to maintain spatial acuity optically, but recover contrast sensitivity, almost completely, by sacrificing temporal acuity neurally.

Evolution of Biological Image Stabilization

The use of vision to coordinate behavior requires an efficient control design that stabilizes the world on the retina or directs the gaze towards salient features in the surroundings. With a level gaze, visual processing tasks are simplified and behaviorally relevant features from the visual environment can be extracted. No matter how simple or sophisticated the eye design, mechanisms have evolved across phyla to stabilize gaze. In this review, we describe functional similarities in eyes and gaze stabilization reflexes, emphasizing their fundamental role in transforming sensory information into motor commands that support postural and locomotor control. We then focus on gaze stabilization design in flying insects and detail some of the underlying principles. Systems analysis reveals that gaze stabilization often involves several sensory modalities, including vision itself, and makes use of feedback as well as feedforward signals. Independent of phylogenetic distance, the physical interaction between an animal and its natural environment - its available senses and how it moves - appears to shape the adaptation of all aspects of gaze stabilization.

Vision for navigation: What can we learn from ants?

The visual systems of all animals are used to provide information that can guide behaviour. In some cases insects demonstrate particularly impressive visually-guided behaviour and then we might reasonably ask how the low-resolution vision and limited neural resources of insects are tuned to particular behavioural strategies. Such questions are of interest to both biologists and to engineers seeking to emulate insect-level performance with lightweight hardware. One behaviour that insects share with many animals is the use of learnt visual information for navigation. Desert ants, in particular, are expert visual navigators. Across their foraging life, ants can learn long idiosyncratic foraging routes. What's more, these routes are learnt quickly and the visual cues that define them can be implemented for guidance independently of other social or personal information. Here we review the style of visual navigation in solitary foraging ants and consider the physiological mechanisms that underpin it. Our perspective is to consider that robust navigation comes from the optimal interaction between behavioural strategy, visual mechanisms and neural hardware. We consider each of these in turn, highlighting the value of ant-like mechanisms in biomimetic endeavours.

3D Visual Response Properties of MSTd Emerge from an Efficient, Sparse Population Code

Neurons in the dorsal subregion of the medial superior temporal (MSTd) area of the macaque respond to large, complex patterns of retinal flow, implying a role in the analysis of self-motion. Some neurons are selective for the expanding radial motion that occurs as an observer moves through the environment ("heading"), and computational models can account for this finding. However, ample evidence suggests that MSTd neurons exhibit a continuum of visual response selectivity to large-field motion stimuli. Furthermore, the underlying computational principles by which these response properties are derived remain poorly understood. Here we describe a computational model of macaque MSTd based on the hypothesis that neurons in MSTd efficiently encode the continuum of large-field retinal flow patterns on the basis of inputs received from neurons in MT with receptive fields that resemble basis vectors recovered with non-negative matrix factorization. These assumptions are sufficient to quantitatively simulate neurophysiological response properties of MSTd cells, such as 3D translation and rotation selectivity, suggesting that these properties might simply be a byproduct of MSTd neurons performing dimensionality reduction on their inputs. At the population level, model MSTd accurately predicts eye velocity and heading using a sparse distributed code, consistent with the idea that biological MSTd might be well equipped to efficiently encode various self-motion variables. The present work aims to add some structure to the often contradictory findings about macaque MSTd, and offers a biologically plausible account of a wide range of visual response properties ranging from single-unit selectivity to population statistics.

SIGNIFICANCE STATEMENT

Using a dimensionality reduction technique known as non-negative matrix factorization, we found that a variety of medial superior temporal (MSTd) neural response properties could be derived from MT-like input features. The responses that emerge from this technique, such as 3D translation and rotation selectivity, spiral tuning, and heading selectivity, can account for a number of empirical results. These findings (1) provide a further step toward a scientific understanding of the often nonintuitive response properties of MSTd neurons; (2) suggest that response properties, such as complex motion tuning and heading selectivity, might simply be a byproduct of MSTd neurons performing dimensionality reduction on their inputs; and (3) imply that motion perception in the cortex is consistent with ideas from the efficient-coding and free-energy principles.

Insect-Inspired Self-Motion Estimation with Dense Flow Fields--An Adaptive Matched Filter Approach

The control of self-motion is a basic, but complex task for both technical and biological systems. Various algorithms have been proposed that allow the estimation of self-motion from the optic flow on the eyes. We show that two apparently very different approaches to solve this task, one technically and one biologically inspired, can be transformed into each other under certain conditions. One estimator of self-motion is based on a matched filter approach; it has been developed to describe the function of motion sensitive cells in the fly brain. The other estimator, the Koenderink and van Doorn (KvD) algorithm, was derived analytically with a technical background. If the distances to the objects in the environment can be assumed to be known, the two estimators are linear and equivalent, but are expressed in different mathematical forms. However, for most situations it is unrealistic to assume that the distances are known. Therefore, the depth structure of the environment needs to be determined in parallel to the self-motion parameters and leads to a non-linear problem. It is shown that the standard least mean square approach that is used by the KvD algorithm leads to a biased estimator. We derive a modification of this algorithm in order to remove the bias and demonstrate its improved performance by means of numerical simulations. For self-motion estimation it is beneficial to have a spherical visual field, similar to many flying insects. We show that in this case the representation of the depth structure of the environment derived from the optic flow can be simplified. Based on this result, we develop an adaptive matched filter approach for systems with a nearly spherical visual field. Then only eight parameters about the environment have to be memorized and updated during self-motion.

Influence of environmental information in natural scenes and the effects of motion adaptation on a fly motion-sensitive neuron during simulated flight

Gaining information about the spatial layout of natural scenes is a challenging task that flies need to solve, especially when moving at high velocities. A group of motion sensitive cells in the lobula plate of flies is supposed to represent information about self-motion as well as the environment. Relevant environmental features might be the nearness of structures, influencing retinal velocity during translational self-motion, and the brightness contrast. We recorded the responses of the H1 cell, an individually identifiable lobula plate tangential cell, during stimulation with image sequences, simulating translational motion through natural sceneries with a variety of differing depth structures. A correlation was found between the average nearness of environmental structures within large parts of the cell's receptive field and its response across a variety of scenes, but no correlation was found between the brightness contrast of the stimuli and the cell response. As a consequence of motion adaptation resulting from repeated translation through the environment, the time-dependent response modulations induced by the spatial structure of the environment were increased relatively to the background activity of the cell. These results support the hypothesis that some lobula plate tangential cells do not only serve as sensors of self-motion, but also as a part of a neural system that processes information about the spatial layout of natural scenes.

Motion as a source of environmental information: a fresh view on biological motion computation by insect brains

Despite their miniature brains insects, such as flies, bees and wasps, are able to navigate by highly erobatic flight maneuvers in cluttered environments. They rely on spatial information that is contained in the retinal motion patterns induced on the eyes while moving around ("optic flow") to accomplish their extraordinary performance. Thereby, they employ an active flight and gaze strategy that separates rapid saccade-like turns from translatory flight phases where the gaze direction is kept largely constant. This behavioral strategy facilitates the processing of environmental information, because information about the distance of the animal to objects in the environment is only contained in the optic flow generated by translatory motion. However, motion detectors as are widespread in biological systems do not represent veridically the velocity of the optic flow vectors, but also reflect textural information about the environment. This characteristic has often been regarded as a limitation of a biological motion detection mechanism. In contrast, we conclude from analyses challenging insect movement detectors with image flow as generated during translatory locomotion through cluttered natural environments that this mechanism represents the contours of nearby objects. Contrast borders are a main carrier of functionally relevant object information in artificial and natural sceneries. The motion detection system thus segregates in a computationally parsimonious way the environment into behaviorally relevant nearby objects and-in many behavioral contexts-less relevant distant structures. Hence, by making use of an active flight and gaze strategy, insects are capable of performing extraordinarily well even with a computationally simple motion detection mechanism.

Gaze characteristics of freely walking blowflies Calliphora vicina in a goal-directed task

In contrast to flying flies, walking flies experience relatively strong rotational gaze shifts, even during overall straight phases of locomotion. These gaze shifts are caused by the walking apparatus and modulated by the stride frequency. Accordingly, even during straight walking phases, the retinal image flow is composed of both translational and rotational optic flow, which might affect spatial vision, as well as fixation behavior. We addressed this issue for an orientation task where walking blowflies approached a black vertical bar. The visual stimulus was stationary, or either the bar or the background moved horizontally. The stride-coupled gaze shifts of flies walking toward the bar had similar amplitudes under all visual conditions tested. This finding indicates that these shifts are an inherent feature of walking, which are not even compensated during a visual goal fixation task. By contrast, approaching flies showed a frequent stop-and-go behavior that was affected by the stimulus conditions. As sustained image rotations may impair distance estimation during walking, we propose a hypothesis that explains how rotation-independent translatory image flow containing distance information can be determined. The algorithm proposed works without requiring differentiation at the behavioral level of the rotational and translational flow components. By contrast, disentangling both has been proposed to be necessary during flight. By comparing the retinal velocities of the edges of the goal, its rotational image motion component can be removed. Consequently, the expansion velocity of the goal and, thus, its proximity can be extracted, irrespective of distance-independent stride-coupled rotational image shifts.

Texture-defined objects influence responses of blowfly motion-sensitive neurons under natural dynamical conditions

The responses of visual interneurons of flies involved in the processing of motion information do not only depend on the velocity, but also on other stimulus parameters, such as the contrast and the spatial frequency content of the stimulus pattern. These dependencies have been known for long, but it is still an open question how they affect the neurons' performance in extracting information about the structure of the environment under the specific dynamical conditions of natural flight. Free-flight of blowflies is characterized by sequences of phases of translational movements lasting for just 30-100 ms interspersed with even shorter and extremely rapid saccade-like rotational shifts in flight and gaze direction. Previous studies already analyzed how nearby objects, leading to relative motion on the retina with respect to a more distant background, influenced the response of a class of fly motion sensitive visual interneurons, the horizontal system (HS) cells. In the present study, we focused on objects that differed from their background by discontinuities either in their brightness contrast or in their spatial frequency content. We found strong object-induced effects on the membrane potential even during the short intersaccadic intervals, if the background contrast was small and the object contrast sufficiently high. The object evoked similar response increments provided that it contained higher spatial frequencies than the background, but not under reversed conditions. This asymmetry in the response behavior is partly a consequence of the depolarization level induced by the background. Thus, our results suggest that, under the specific dynamical conditions of natural flight, i.e., on a very short timescale, the responses of HS cells represent object information depending on the polarity of the difference between object and background contrast and spatial frequency content.

Encoding of naturalistic optic flow by motion sensitive neurons of nucleus rotundus in the zebra finch (Taeniopygia guttata)

The retinal image changes that occur during locomotion, the optic flow, carry information about self-motion and the three-dimensional structure of the environment. Especially fast moving animals with only little binocular vision depend on these depth cues for maneuvering. They actively control their gaze to facilitate perception of depth based on cues in the optic flow. In the visual system of birds, nucleus rotundus neurons were originally found to respond to object motion but not to background motion. However, when background and object were both moving, responses increased the more the direction and velocity of object and background motion on the retina differed. These properties may play a role in representing depth cues in the optic flow. We therefore investigated, how neurons in nucleus rotundus respond to optic flow that contains depth cues. We presented simplified and naturalistic optic flow on a panoramic LED display while recording from single neurons in nucleus rotundus of anaesthetized zebra finches. Unlike most studies on motion vision in birds, our stimuli included depth information. We found extensive responses of motion selective neurons in nucleus rotundus to optic flow stimuli. Simplified stimuli revealed preferences for optic flow reflecting translational or rotational self-motion. Naturalistic optic flow stimuli elicited complex response modulations, but the presence of objects was signaled by only few neurons. The neurons that did respond to objects in the optic flow, however, show interesting properties.

Flying fruit flies correct for visual sideslip depending on relative speed of forward optic flow

As a fly flies through its environment, static objects produce moving images on its retina, and this optic flow is essential for steering and course corrections. Different types of rotation and translation produce unique flow fields, which fly brains are wired to identify. However, a feature of optic flow unique to translational motion is that adjacent images may move across the retina at different speeds, depending on their distance from the observer. Many insects take advantage of this depth cue, called motion parallax, to determine the distance to objects. We wanted to know if differential object speeds affect the corrective responses of fruit flies when they experience unplanned course deviations. We presented tethered flying flies with optic flow and measured their corrective responses to sideways perturbations of images with different relative forward speeds. We found that flying flies attend to the relative speed of dots during forward motion, and adjust their corrective responses to sideslip deviations depending on this cue. With no other distinguishing features (such as brightness or size), flies mounted a greater response to sideways deviations that were signaled by faster moving dots in the forward flow field, those that appeared radially closer by their speeds. This is consistent with the interpretation that fruit flies attend to seemingly nearer objects, and correct more strongly when they indicate a perturbation.

Spatial vision in insects is facilitated by shaping the dynamics of visual input through behavioral action

Insects such as flies or bees, with their miniature brains, are able to control highly aerobatic flight maneuvres and to solve spatial vision tasks, such as avoiding collisions with obstacles, landing on objects, or even localizing a previously learnt inconspicuous goal on the basis of environmental cues. With regard to solving such spatial tasks, these insects still outperform man-made autonomous flying systems. To accomplish their extraordinary performance, flies and bees have been shown by their characteristic behavioral actions to actively shape the dynamics of the image flow on their eyes ("optic flow"). The neural processing of information about the spatial layout of the environment is greatly facilitated by segregating the rotational from the translational optic flow component through a saccadic flight and gaze strategy. This active vision strategy thus enables the nervous system to solve apparently complex spatial vision tasks in a particularly efficient and parsimonious way. The key idea of this review is that biological agents, such as flies or bees, acquire at least part of their strength as autonomous systems through active interactions with their environment and not by simply processing passively gained information about the world. These agent-environment interactions lead to adaptive behavior in surroundings of a wide range of complexity. Animals with even tiny brains, such as insects, are capable of performing extraordinarily well in their behavioral contexts by making optimal use of the closed action-perception loop. Model simulations and robotic implementations show that the smart biological mechanisms of motion computation and visually-guided flight control might be helpful to find technical solutions, for example, when designing micro air vehicles carrying a miniaturized, low-weight on-board processor.

A model of ant navigation based on visual prediction

A model of visual navigation in ants is presented which is based on a simple network predicting the changes of a visual scene under translatory movements. The model contains two behavioral components: the acquisition of multiple snapshots in different orientations during a learning walk, and the selection of a movement direction by a scanning behavior where the ant searches through different headings. Both components fit with observations in experiments with desert ants. The model is in most aspects biologically plausible with respect to the equivalent neural networks, and it produces reliable homing behavior in a simulated environment with a complex random surface texture. The model is closely related to the algorithmic min-warping method for visual robot navigation which shows good homing performance in real-world environments.

Binocular interactions underlying the classic optomotor responses of flying flies

In response to imposed course deviations, the optomotor reactions of animals reduce motion blur and facilitate the maintenance of stable body posture. In flies, many anatomical and electrophysiological studies suggest that disparate motion cues stimulating the left and right eyes are not processed in isolation but rather are integrated in the brain to produce a cohesive panoramic percept. To investigate the strength of such inter-ocular interactions and their role in compensatory sensory–motor transformations, we utilize a virtual reality flight simulator to record wing and head optomotor reactions by tethered flying flies in response to imposed binocular rotation and monocular front-to-back and back-to-front motion. Within a narrow range of stimulus parameters that generates large contrast insensitive optomotor responses to binocular rotation, we find that responses to monocular front-to-back motion are larger than those to panoramic rotation, but are contrast sensitive. Conversely, responses to monocular back-to-front motion are slower than those to rotation and peak at the lowest tested contrast. Together our results suggest that optomotor responses to binocular rotation result from the influence of non-additive contralateral inhibitory as well as excitatory circuit interactions that serve to confer contrast insensitivity to flight behaviors influenced by rotatory optic flow.

Visual control of navigation in insects and its relevance for robotics

Flying insects display remarkable agility, despite their diminutive eyes and brains. This review describes our growing understanding of how these creatures use visual information to stabilize flight, avoid collisions with objects, regulate flight speed, detect and intercept other flying insects such as mates or prey, navigate to a distant food source, and orchestrate flawless landings. It also outlines the ways in which these insights are now being used to develop novel, biologically inspired strategies for the guidance of autonomous, airborne vehicles.

Neuronal processing of translational optic flow in the visual system of the shore crab Carcinus maenas

This paper describes a search for neurones sensitive to optic flow in the visual system of the shore crab Carcinus maenas using a procedure developed from that of Krapp and Hengstenberg. This involved determining local motion sensitivity and its directional selectivity at many points within the neurone's receptive field and plotting the results on a map. Our results showed that local preferred directions of motion are independent of velocity, stimulus shape and type of motion (circular or linear). Global response maps thus clearly represent real properties of the neurones' receptive fields. Using this method, we have discovered two families of interneurones sensitive to translational optic flow. The first family has its terminal arborisations in the lobula of the optic lobe, the second family in the medulla. The response maps of the lobula neurones (which appear to be monostratified lobular giant neurones) show a clear focus of expansion centred on or just above the horizon, but at significantly different azimuth angles. Response maps such as these, consisting of patterns of movement vectors radiating from a pole, would be expected of neurones responding to self-motion in a particular direction. They would be stimulated when the crab moves towards the pole of the neurone's receptive field. The response maps of the medulla neurones show a focus of contraction, approximately centred on the horizon, but at significantly different azimuth angles. Such neurones would be stimulated when the crab walked away from the pole of the neurone's receptive field. We hypothesise that both the lobula and the medulla interneurones are representatives of arrays of cells, each of which would be optimally activated by self-motion in a different direction. The lobula neurones would be stimulated by the approaching scene and the medulla neurones by the receding scene. Neurones tuned to translational optic flow provide information on the three-dimensional layout of the environment and are thought to play a role in the judgment of heading.

The many facets of adaptation in fly visual motion processing

Neuronal adaptation has been studied extensively in visual motion-sensitive neurons of the fly Calliphora vicina, a model system in which the computational principles of visual motion processing are amenable on a single-cell level. Evidenced by several recent papers, the original idea had to be dismissed that motion adaptation adjusts velocity coding to the current stimulus range by a simple parameter change in the motion detection scheme. In contrast, linear encoding of velocity modulations and total information rates might even go down in the course of adaptation. Thus it seems that rather than improving absolute velocity encoding motion adaptation might bring forward an efficient extraction of those features in the visual input signal that are most relevant for visually guided course control and obstacle avoidance.

Honeybees as a model for the study of visually guided flight, navigation, and biologically inspired robotics

Research over the past century has revealed the impressive capacities of the honeybee, Apis mellifera, in relation to visual perception, flight guidance, navigation, and learning and memory. These observations, coupled with the relative ease with which these creatures can be trained, and the relative simplicity of their nervous systems, have made honeybees an attractive model in which to pursue general principles of sensorimotor function in a variety of contexts, many of which pertain not just to honeybees, but several other animal species, including humans. This review begins by describing the principles of visual guidance that underlie perception of the world in three dimensions, obstacle avoidance, control of flight speed, and orchestrating smooth landings. We then consider how navigation over long distances is accomplished, with particular reference to how bees use information from the celestial compass to determine their flight bearing, and information from the movement of the environment in their eyes to gauge how far they have flown. Finally, we illustrate how some of the principles gleaned from these studies are now being used to design novel, biologically inspired algorithms for the guidance of unmanned aerial vehicles.

Frequency response of lift control in Drosophila

The flight control responses of the fruitfly represent a powerful model system to explore neuromotor control mechanisms, whose system level control properties can be suitably characterized with a frequency response analysis. We characterized the lift response dynamics of tethered flying Drosophila in presence of vertically oscillating visual patterns, whose oscillation frequency we varied between 0.1 and 13 Hz. We justified these measurements by showing that the amplitude gain and phase response is invariant to the pattern oscillation amplitude and spatial frequency within a broad dynamic range. We also showed that lift responses are largely linear and time invariant (LTI), a necessary condition for a meaningful analysis of frequency responses and a remarkable characteristic given its nonlinear constituents. The flies responded to increasing oscillation frequencies with a roughly linear decrease in response gain, which dropped to background noise levels at about 6 Hz. The phase lag decreased linearly, consistent with a constant reaction delay of 75 ms. Next, we estimated the free-flight response of the fly to generate a Bode diagram of the lift response. The limitation of lift control to frequencies below 6 Hz is explained with inertial body damping, which becomes dominant at higher frequencies. Our work provides the detailed background and techniques that allow optomotor lift responses of Drosophila to be measured with comparatively simple, affordable and commercially available techniques. The identification of an LTI, pattern velocity dependent, lift control strategy is relevant to the underlying motion computation mechanisms and serves a broader understanding of insects' flight control strategies. The relevance and potential pitfalls of applying system identification techniques in tethered preparations is discussed.

Control theoretic interpretation of directional motion preferences in optic flow processing interneurons

In this article, we formalize the processing of optic flow in identified fly lobula plate tangential cells and develop a control theoretic framework that suggests how the signals of these cells may be combined and used to achieve reflex-like navigation behavior. We show that this feedback gain synthesis task can be cast as a combined static state estimation and linear feedback control problem. Our framework allows us to analyze and determine the relationship between optic flow measurements and actuator commands, which greatly simplifies the implementation of biologically inspired control architectures on terrestrial and aerial robotic platforms.

Localized direction selective responses in the dendrites of visual interneurons of the fly

BACKGROUND

The various tasks of visual systems, including course control, collision avoidance and the detection of small objects, require at the neuronal level the dendritic integration and subsequent processing of many spatially distributed visual motion inputs. While much is known about the pooled output in these systems, as in the medial superior temporal cortex of monkeys or in the lobula plate of the insect visual system, the motion tuning of the elements that provide the input has yet received little attention. In order to visualize the motion tuning of these inputs we examined the dendritic activation patterns of neurons that are selective for the characteristic patterns of wide-field motion, the lobula-plate tangential cells (LPTCs) of the blowfly. These neurons are known to sample direction-selective motion information from large parts of the visual field and combine these signals into axonal and dendro-dendritic outputs.

RESULTS

Fluorescence imaging of intracellular calcium concentration allowed us to take a direct look at the local dendritic activity and the resulting local preferred directions in LPTC dendrites during activation by wide-field motion in different directions. These 'calcium response fields' resembled a retinotopic dendritic map of local preferred directions in the receptive field, the layout of which is a distinguishing feature of different LPTCs.

CONCLUSIONS

Our study reveals how neurons acquire selectivity for distinct visual motion patterns by dendritic integration of the local inputs with different preferred directions. With their spatial layout of directional responses, the dendrites of the LPTCs we investigated thus served as matched filters for wide-field motion patterns.

Behavioural state affects motion-sensitive neurones in the fly visual system

The strength of stimulus-induced responses at the neuronal and the behavioural level often depends on the internal state of an animal. Within pathways processing sensory information and eventually controlling behavioural responses, such gain changes can originate at several sites. Using motion-sensitive lobula plate tangential cells (LPTCs) of blowflies, we address whether and in which way information processing changes for two different states of motor activity. We distinguish between the two states on the basis of haltere movements. Halteres are the evolutionarily transformed hindwings of flies. They oscillate when the animals walk or fly. LPTCs mediate, amongst other behaviours, head optomotor responses. These are either of large or small amplitude depending on the state of motor activity. Here we find that LPTC responses also depend on the motor activity of flies. In particular, LPTC responses are enhanced when halteres oscillate. Nevertheless, the response changes of LPTCs do not account for the corresponding large gain changes of head movements. Moreover, haltere activity itself does not change the activity of LPTCs. Instead, we propose that a central signal associated with motor activity changes the gain of head optomotor responses and the response properties of LPTCs.

Nonlinear integration of visual and haltere inputs in fly neck motor neurons

Animals use information from multiple sensory organs to generate appropriate behavior. Exactly how these different sensory inputs are fused at the motor system is not well understood. Here we study how fly neck motor neurons integrate information from two well characterized sensory systems: visual information from the compound eye and gyroscopic information from the mechanosensory halteres. Extracellular recordings reveal that a subpopulation of neck motor neurons display "gating-like" behavior: they do not fire action potentials in response to visual stimuli alone but will do so if the halteres are coactivated. Intracellular recordings show that these motor neurons receive small, sustained subthreshold visual inputs in addition to larger inputs that are phase locked to haltere movements. Our results suggest that the nonlinear gating-like effect results from summation of these two inputs with the action potential threshold providing the nonlinearity. As a result of this summation, the sustained visual depolarization is transformed into a temporally structured train of action potentials synchronized to the haltere beating movements. This simple mechanism efficiently fuses two different sensory signals and may also explain the context-dependent effects of visual inputs on fly behavior.

Adaptation accentuates responses of fly motion-sensitive visual neurons to sudden stimulus changes

Adaptation in sensory and neuronal systems usually leads to reduced responses to persistent or frequently presented stimuli. In contrast to simple fatigue, adapted neurons often retain their ability to encode changes in stimulus intensity and to respond when novel stimuli appear. We investigated how the level of adaptation of a fly visual motion-sensitive neuron affects its responses to discontinuities in the stimulus, i.e. sudden brief changes in one of the stimulus parameters (velocity, contrast, grating orientation and spatial frequency). Although the neuron's overall response decreased gradually during ongoing motion stimulation, the response transients elicited by stimulus discontinuities were preserved or even enhanced with adaptation. Moreover, the enhanced sensitivity to velocity changes by adaptation was not restricted to a certain velocity range, but was present regardless of whether the neuron was adapted to a baseline velocity below or above its steady-state velocity optimum. Our results suggest that motion adaptation helps motion-sensitive neurons to preserve their sensitivity to novel stimuli even in the presence of strong tonic stimulation, for example during self-motion.

Variability of blowfly head optomotor responses

Behavioural responses of an animal are variable even when the animal experiences the same sensory input several times. This variability can arise from stochastic processes inherent to the nervous system. Also, the internal state of an animal may influence a particular behavioural response. In the present study, we analyse the variability of visually induced head pitch responses of tethered blowflies by high-speed cinematography. We found these optomotor responses to be highly variable in amplitude. Most of the variability can be attributed to two different internal states of the flies with high and low optomotor gain, respectively. Even within a given activity state, there is some variability of head optomotor responses. The amount of this variability differs for the two optomotor gain states. Moreover, these two activity states can be distinguished on a fine timescale and without visual stimulation, on the basis of the occurrence of peculiar head jitter movements. Head jitter goes along with high gain optomotor responses and haltere oscillations. Halteres are evolutionary transformed hindwings that oscillate when blowflies walk or fly. Their main function is to serve as equilibrium organs by detecting Coriolis forces and to mediate gaze stabilisation. However, their basic oscillating activity was also suggested to provide a gain-modulating signal. Our experiments demonstrate that halteres are not necessary for high gain head pitch to occur. Nevertheless, we find the halteres to be responsible for one component of head jitter movements. This component may be the inevitable consequence of their function as equilibrium and gaze-stabilising organs.

Sensory integration: neuronal adaptations for robust visual self-motion estimation

Recent studies on the fly visual system have revealed how the morphology of visual interneurons and their lateral electrical connectivity helps overcome a notorious problem in visuomotor control - the ambiguity of local sensor signals.

Visuomotor transformation in the fly gaze stabilization system

For sensory signals to control an animal's behavior, they must first be transformed into a format appropriate for use by its motor systems. This fundamental problem is faced by all animals, including humans. Beyond simple reflexes, little is known about how such sensorimotor transformations take place. Here we describe how the outputs of a well-characterized population of fly visual interneurons, lobula plate tangential cells (LPTCs), are used by the animal's gaze-stabilizing neck motor system. The LPTCs respond to visual input arising from both self-rotations and translations of the fly. The neck motor system however is involved in gaze stabilization and thus mainly controls compensatory head rotations. We investigated how the neck motor system is able to selectively extract rotation information from the mixed responses of the LPTCs. We recorded extracellularly from fly neck motor neurons (NMNs) and mapped the directional preferences across their extended visual receptive fields. Our results suggest that-like the tangential cells-NMNs are tuned to panoramic retinal image shifts, or optic flow fields, which occur when the fly rotates about particular body axes. In many cases, tangential cells and motor neurons appear to be tuned to similar axes of rotation, resulting in a correlation between the coordinate systems the two neural populations employ. However, in contrast to the primarily monocular receptive fields of the tangential cells, most NMNs are sensitive to visual motion presented to either eye. This results in the NMNs being more selective for rotation than the LPTCs. Thus, the neck motor system increases its rotation selectivity by a comparatively simple mechanism: the integration of binocular visual motion information.

Saccadic flight strategy facilitates collision avoidance: closed-loop performance of a cyberfly

Behavioural and electrophysiological experiments suggest that blowflies employ an active saccadic strategy of flight and gaze control to separate the rotational from the translational optic flow components. As a consequence, this allows motion sensitive neurons to encode during translatory intersaccadic phases of locomotion information about the spatial layout of the environment. So far, it has not been clear whether and how a motor controller could decode the responses of these neurons to prevent a blowfly from colliding with obstacles. Here we propose a simple model of the blowfly visual course control system, named cyberfly, and investigate its performance and limitations. The sensory input module of the cyberfly emulates a pair of output neurons subserving the two eyes of the blowfly visual motion pathway. We analyse two sensory-motor interfaces (SMI). An SMI coupling the differential signal of the sensory neurons proportionally to the yaw rotation fails to avoid obstacles. A more plausible SMI is based on a saccadic controller. Even with sideward drift after saccades as is characteristic of real blowflies, the cyberfly is able to successfully avoid collisions with obstacles. The relative distance information contained in the optic flow during translatory movements between saccades is provided to the SMI by the responses of the visual output neurons. An obvious limitation of this simple mechanism is its strong dependence on the textural properties of the environment.

The spatial, temporal and contrast properties of expansion and rotation flight optomotor responses inDrosophila

Fruit flies respond to panoramic retinal patterns of visual expansion with robust steering maneuvers directed away from the focus of expansion to avoid collisions and maintain an upwind flight posture. Panoramic rotation elicits comparatively weak syndirectional steering maneuvers, which also maintain visual stability. Full-field optic flow patterns like expansion and rotation are elicited by distinct flight maneuvers such as body translation during straight flight or body rotation during hovering, respectively. Recent analyses suggest that under some experimental conditions the rotation optomotor response reflects the linear sum of different expansion response components. Are expansion and rotation-mediated visual stabilization responses part of a single optomotor response subserved by a neural circuit that is differentially stimulated by the two flow fields, or rather do the two behavioral responses reflect two distinct control systems? Guided by the principle that the properties of neural circuits are revealed in the behaviors they mediate, we systematically varied the spatial, temporal and contrast properties of expansion and rotation stimuli, and quantified the time course and amplitude of optomotor responses during tethered flight. Our results support the conclusion that expansion and rotation optomotor responses are indeed two separate reflexes, which draw from the same system of elementary motion detectors, but are likely mediated by separate pre-motor circuits having different spatial integration properties, low-pass characteristics and contrast sensitivity.

Robust coding of flow-field parameters by axo-axonal gap junctions between fly visual interneurons

Complex flight maneuvers require a sophisticated system to exploit the optic flow resulting from moving images of the environment projected onto the retina. In the fly's visual course control center, the lobula plate, 10 so-called vertical system (VS) cells are thought to match, with their complex receptive fields, the optic flow resulting from rotation around different body axes. However, signals of single VS cells are unreliable indicators of such optic flow parameters in the context of their noisy, texture-dependent input from local motion measurements. Here we propose an alternative encoding scheme based on network simulations of biophysically realistic compartmental models of VS cells. The simulations incorporate recent data about the highly selective connectivity between VS cells consisting of an electrical axo-axonal coupling between adjacent cells and a reciprocal inhibition between the most distant cells. We find that this particular wiring performs a linear interpolation between the output signals of VS cells, leading to a robust representation of the axis of rotation even in the presence of textureless patches of the visual surround.

A motion-sensitive neurone responds to signals from the two visual systems of the blowfly, the compound eyes and ocelli

In the blowfly Calliphora vicina, lobula plate tangential cells (LPTCs) estimate self-motion by integrating local motion information from the compound eyes. Each LPTC is sensitive to a particular (preferred) rotation of the fly's head. The fly can also sense rotation using its three ocelli (simple eyes), by comparing the light intensities measured at each ocellus. We report that an individually identified tangential cell, V1, responds in an apparently rotation-specific manner to stimulation of the ocelli. This effect was seen with or without additional stimulation of the compound eye. We delivered stimuli to the ocelli which mimicked rotation of the fly's head close to that of the preferred axis of rotation of V1. Alternating between preferred and anti-preferred rotation elicited a strongly phasic response, the amplitude of which increased with the rate of change of light intensity at the ocelli. With combined stimulation of one compound eye and the ocelli, V1 displayed a robust response to ocellar stimuli over its entire response range. These findings provide the opportunity to study quantitatively the interactions of two different visual mechanisms which both encode the same variable--the animal's rotation in space.

A `bright zone' in male hoverfly (Eristalis tenax) eyes and associated faster motion detection and increased contrast sensitivity

Eyes of the hoverfly Eristalis tenax are sexually dimorphic such that males have a fronto-dorsal region of large facets. In contrast to other large flies in which large facets are associated with a decreased interommatidial angle to form a dorsal `acute zone' of increased spatial resolution, we show that a dorsal region of large facets in males appears to form a `bright zone' of increased light capture without substantially increased spatial resolution. Theoretically, more light allows for increased performance in tasks such as motion detection. To determine the effect of the bright zone on motion detection, local properties of wide field motion detecting neurons were investigated using localized sinusoidal gratings. The pattern of local preferred directions of one class of these cells, the HS cells, in Eristalis is similar to that reported for the blowfly Calliphora. The bright zone seems to contribute to local contrast sensitivity; high contrast sensitivity exists in portions of the receptive field served by large diameter facet lenses of males and is not observed in females. Finally, temporal frequency tuning is also significantly faster in this frontal portion of the world, particularly in males, where it overcompensates for the higher spatial-frequency tuning and shifts the predicted local velocity optimum to higher speeds. These results indicate that increased retinal illuminance due to the bright zone of males is used to enhance contrast sensitivity and speed motion detector responses. Additionally, local neural properties vary across the visual world in a way not expected if HS cells serve purely as matched filters to measure yaw-induced visual motion.

Local visual homing by matched-filter descent in image distances

In natural images, the distance measure between two images taken at different locations rises smoothly with increasing distance between the locations. This fact can be exploited for local visual homing where the task is to reach a goal location that is characterized by a snapshot image: descending in the image distance will lead the agent to the goal location. To compute an estimate of the spatial gradient in the distance measure, its value must be sampled at three noncollinear points. An animal or robot would have to insert exploratory movements into its home trajectory to collect these samples. Here we suggest a method based on the matched-filter concept that allows one to estimate the gradient without exploratory movements. Two matched filters--optical flow fields resulting from translatory movements in the horizontal plane--are used to predict two images in perpendicular directions from the current location. We investigate the relation to differential flow methods applied to the local homing problem and show that the matched-filter approach produces reliable homing behavior on image databases. Two alternative methods that only require a single matched filter are suggested. The matched-filter concept is also applied to derive a home-vector equation for a Fourier-based parameter method.

Representation of behaviourally relevant information by blowfly motion-sensitive visual interneurons requires precise compensatory head movements

Flying blowflies shift their gaze by saccadic turns of body and head, keeping their gaze basically fixed between saccades. For the head, this results in almost pure translational optic flow between saccades, enabling visual interneurons in the fly motion pathway to extract information about translation of the animal and thereby about the spatial layout of the environment. There are noticeable differences between head and body movements during flight. Head saccades are faster and shorter than body saccades, and the head orientation is more stable between saccades than the body orientation. Here, we analyse the functional importance of these differences by probing visual interneurons of the blowfly motion pathway with optic flow based on either head movements or body movements, as recorded accurately with a magnetic search coil technique. We find that the precise head-body coordination is essential for the visual system to separate the translational from the rotational optic flow. If the head were tightly coupled to the body, the resulting optic flow would not contain the behaviourally important information on translation. Since it is difficult to resolve head orientation in many experimental paradigms, even when employing state-of-the-art digital video techniques, we introduce a 'headifying algorithm', which transforms the time-dependent body orientation in free flight into an estimate of head orientation. We show that application of this algorithm leads to an estimated head orientation between saccades that is sufficiently stable to enable recovering information on translation. The algorithm may therefore be of practical use when head orientation is needed but cannot be measured.

Spatial integration of optic flow signals in fly motion-sensitive neurons

Neurons in the fly lobula plate integrate motion signals over large regions of visual space in a directionally selective manner. This study is concerned with the details of this integration process. We used a stimulus consisting of a 4 x 4 lattice of locally moving Gabor patches, in which each patch could take any direction independently. We also presented only one patch at a time or two patches at a time. Across all possible directions of motion, the firing rate response r1+2 to two simultaneously presented patches was well described by r1+2(d1, d2) = G x [r1(d1) + r2(d2)] + S, where r1 and r2 are responses to individual patches moving in directions d1 and d2, and G approximately 0.81, S approximately -23. However, this quasi-linear scaling expression failed to account for three main empirical observations: 1) the directional-tuning curve for one patch is broader in the presence of another patch moving in the neuron's preferred direction (PD); 2) the vertical compression of this curve is greater when the second patch moves in the antipreferred direction (AD) as opposed to PD; 3) the ability of the neuronal response to discriminate the direction of a patch is greater when the other patch is moving in the PD as opposed to AD, where this ability is assessed using both information theory and a standard discriminability index. To account for these departures from the simple scaling model, we used a normalization model very similar to one used for macaque area MT/V5. This model can qualitatively explain all three departures from the scaling equation described above, suggesting that a gain-control normalization network may be at work within the fly lobula plate.

Population Coding of Self-Motion: Applying Bayesian Analysis to a Population of Visual Interneurons in the Fly

Coding of sensory information often involves the activity of neuronal populations. We demonstrate how the accuracy of a population code depends on integration time, the size of the population, and noise correlation between the participating neurons. The population we study consists of 10 identified visual interneurons in the blowfly Calliphora vicina involved in optic flow processing. These neurons are assumed to encode the animal's head or body rotations around horizontal axes by means of graded potential changes. From electrophysiological experiments we obtain parameters for modeling the neurons' responses. From applying a Bayesian analysis on the modeled population response we draw three major conclusions. First, integration of neuronal activities over a time period of only 5 ms after response onset is sufficient to decode accurately the rotation axis. Second, noise correlation between neurons has only little impact on the population's performance. And third, although a population of only two neurons would be sufficient to encode any horizontal rotation axis, the population of 10 vertical system neurons is advantageous if the available integration time is short. For the fly, short integration times to decode neuronal responses are important when controlling rapid flight maneuvers.

Responses of blowfly motion-sensitive neurons to reconstructed optic flow along outdoor flight paths

The retinal image flow a blowfly experiences in its daily life on the wing is determined by both the structure of the environment and the animal's own movements. To understand the design of visual processing mechanisms, there is thus a need to analyse the performance of neurons under natural operating conditions. To this end, we recorded flight paths of flies outdoors and reconstructed what they had seen, by moving a panoramic camera along exactly the same paths. The reconstructed image sequences were later replayed on a fast, panoramic flight simulator to identified, motion sensitive neurons of the so-called horizontal system (HS) in the lobula plate of the blowfly, which are assumed to extract self-motion parameters from optic flow. We show that under real life conditions HS-cells not only encode information about self-rotation, but are also sensitive to translational optic flow and, thus, indirectly signal information about the depth structure of the environment. These properties do not require an elaboration of the known model of these neurons, because the natural optic flow sequences generate--at least qualitatively--the same depth-related response properties when used as input to a computational HS-cell model and to real neurons.

Visual motor computations in insects

With their relatively simple nervous systems and purpose-designed behaviors and reflexes, insects are an excellent organism in which to investigate how visual information is acquired and processed to guide locomotion and navigation. Flies maintain a straight course and monitor their motion through the environment by sensing the patterns of optic flow induced in the eyes. Bees negotiate narrow gaps by balancing the speeds of the images in their two eyes, and they control flight speed by holding constant the average image velocity as seen with their two eyes. Bees achieve a smooth landing on a horizontal surface by holding the image velocity of the surface constant during approach, thus ensuring that flight speed is automatically close to zero at touchdown. Foraging bees estimate the distance that they have traveled to reach a food source by integrating the optic flow experienced en route; this integration gives them a visually driven "odometer." Insects have also evolved sophisticated visuomotor mechanisms for pursuing prey or mates and possibly for concealing their own motion while shadowing objects of interest.

Spatial organization of visuomotor reflexes in Drosophila

In most animals, the visual system plays a central role in locomotor guidance. Here, we examined the functional organization of visuomotor reflexes in the fruit fly, Drosophila, using an electronic flight simulator. Flies exhibit powerful avoidance responses to visual expansion centered laterally. The amplitude of these expansion responses is three times larger than those generated by image rotation. Avoidance of a laterally positioned focus of expansion emerges from an inversion of the optomotor response when motion is restricted to the rear visual hemisphere. Furthermore, motion restricted to rear quarter-fields elicits turning responses that are independent of the direction of image motion about the animal's yaw axis. The spatial heterogeneity of visuomotor responses explains a seemingly peculiar behavior in which flies robustly fixate the contracting pole of a translating flow field.

Robustness of the tuning of fly visual interneurons to rotatory optic flow

The sophisticated receptive field organization of motion-sensitive tangential cells in the visual system of the blowfly Calliphora vicina matches the structure of particular optic flow fields. Hypotheses on the tuning of particular tangential cells to rotatory self-motion are based on local motion measurements. So far, tangential cells have never been tested with global optic flow stimuli. Therefore we measured the responses of an identifiable neuron, the V1 tangential cell, to wide-field motion stimuli mimicking optic flow fields similar to those the fly encounters during particular self-motions. The stimuli were generated by a "planetarium-projector," casting a pattern of moving light dots on a large spherical projection screen. We determined the tuning curves of the V1-cell to optic flow fields as induced by the animal during 1) rotation about horizontally aligned body axes, 2) upward/downward translation, and 3) a combination of both components. We found that the V1-cell does not respond as specifically to self-rotations, as had been concluded from its receptive field organization. The neuron responds strongly to upward translation and its tuning to rotations is much coarser than expected. The discrepancies between the responses to global optic flow and the predictions based on the receptive field organization are likely due to nonlinear integration properties of tangential neurons. Response parameters like orientation, shape, and width of the tuning curve are largely unaffected by changes in rotation velocity or a superposition of rotational and translational optic flow.