A look into the cockpit of the developing locust: looming detectors and predator avoidance

Recent advances in insect vision in a 3D world: looming stimuli and escape behaviour

Detecting looming motion directly towards the insect is vital to its survival. Looming detection in two insects, flies and locusts, is described and contrasted. Pathways using looming detectors to trigger action and their topographical layout in the brain is explored in relation to facilitating behavioural selection. Similar visual stimuli, such as looming motion, are processed by nearby glomeruli in the brain. Insect-inspired looming motion detectors are combined to detect and avoid collision in different scenarios by robots, vehicles and unmanned aerial vehicle (UAV)s.

Enhancing LGMD-based model for collision prediction via binocular structure

Introduction

Lobular giant motion detector (LGMD) neurons, renowned for their distinctive response to looming stimuli, inspire the development of visual neural network models for collision prediction. However, the existing LGMD-based models could not yet incorporate the invaluable feature of depth distance and still suffer from the following two primary drawbacks. Firstly, they struggle to effectively distinguish the three fundamental motion patterns of approaching, receding, and translating, in contrast to the natural abilities of LGMD neurons. Secondly, due to their reliance on a general determination process employing an activation function and fixed threshold for output, these models exhibit dramatic fluctuations in prediction effectiveness across different scenarios.

Methods

To address these issues, we propose a novel LGMD-based model with a binocular structure (Bi-LGMD). The depth distance of the moving object is extracted by calculating the binocular disparity facilitating a clear differentiation of the motion patterns, after obtaining the moving object's contour through the basic components of the LGMD network. In addition, we introduce a self-adaptive warning depth-distance, enhancing the model's robustness in various motion scenarios.

Results

The effectiveness of the proposed model is verified using computer-simulated and real-world videos.

Discussion

Furthermore, the experimental results demonstrate that the proposed model is robust to contrast and noise.

A look into feedback neural computation upon collision selectivity

Physiological studies have shown that a group of locust's lobula giant movement detectors (LGMDs) has a diversity of collision selectivity to approaching objects, relatively darker or brighter than their backgrounds in cluttered environments. Such diversity of collision selectivity can serve locusts to escape from attack by natural enemies, and migrate in swarm free of collision. For computational studies, endeavours have been made to realize the diverse selectivity which, however, is still one of the most challenging tasks especially in complex and dynamic real world scenarios. The existing models are mainly formulated as multi-layered neural networks with merely feed-forward information processing, and do not take into account the effect of re-entrant signals in feedback loop, which is an essential regulatory loop for motion perception, yet never been explored in looming perception. In this paper, we inaugurate feedback neural computation for constructing a new LGMD-based model, named F-LGMD to look into the efficacy upon implementing different collision selectivity. Accordingly, the proposed neural network model features both feed-forward processing and feedback loop. The feedback control propagates output signals of parallel ON/OFF channels back into their starting neurons, thus makes part of the feed-forward neural network, i.e. the ON/OFF channels and the feedback loop form an iterative cycle system. Moreover, the feedback control is instantaneous, which leads to the existence of a fixed point whereby the fixed point theorem is applied to rigorously derive valid range of feedback coefficients. To verify the effectiveness of the proposed method, we conduct systematic experiments covering synthetic and natural collision datasets, and also online robotic tests. The experimental results show that the F-LGMD, with a unified network, can fulfil the diverse collision selectivity revealed in physiology, which not only reduces considerably the handcrafted parameters compared to previous studies, but also offers a both efficient and robust scheme for collision perception through feedback neural computation.

Motion perception based on ON/OFF channels: A survey

Motion perception is an essential ability for animals and artificially intelligent systems interacting effectively, safely with surrounding objects and environments. Biological visual systems, that have naturally evolved over hundreds-million years, are quite efficient and robust for motion perception, whereas artificial vision systems are far from such capability. This paper argues that the gap can be significantly reduced by formulation of ON/OFF channels in motion perception models encoding luminance increment (ON) and decrement (OFF) responses within receptive field, separately. Such signal-bifurcating structure has been found in neural systems of many animal species articulating early motion is split and processed in segregated pathways. However, the corresponding biological substrates, and the necessity for artificial vision systems have never been elucidated together, leaving concerns on uniqueness and advantages of ON/OFF channels upon building dynamic vision systems to address real world challenges. This paper highlights the importance of ON/OFF channels in motion perception through surveying current progress covering both neuroscience and computationally modelling works with applications. Compared to related literature, this paper for the first time provides insights into implementation of different selectivity to directional motion of looming, translating, and small-sized target movement based on ON/OFF channels in keeping with soundness and robustness of biological principles. Existing challenges and future trends of such bio-plausible computational structure for visual perception in connection with hotspots of machine learning, advanced vision sensors like event-driven camera finally are discussed.

Approaching object acceleration differentially affects the predictions of neuronal collision avoidance models

The processing of visual information for collision avoidance has been investigated at the biophysical level in several model systems. In grasshoppers, the (so-called) [Formula: see text] model captures reasonably well the visual processing performed by an identified neuron called the lobular giant movement detector as it tracks approaching objects. Similar phenomenological models have been used to describe either the firing rate or the membrane potential of neurons responsible for visually guided collision avoidance in other animals. Specifically, in goldfish, the [Formula: see text] model has been proposed to describe the Mauthner cell, an identified neuron involved in startle escape responses. In the vinegar fly, a third model was developed for the giant fiber neuron, which triggers last resort escapes immediately before an impending collision. One key property of these models is their prediction that peak neuronal responses occur at a fixed delay after the simulated approaching object reaches a threshold angular size on the retina. This prediction is valid for simulated objects approaching at a constant speed. We tested whether it remains valid when approaching objects accelerate. After characterizing and comparing the models' responses to accelerating and constant speed stimuli, we find that the prediction holds true for the [Formula: see text] and the giant fiber model, but not for the [Formula: see text] model. These results suggest that acceleration in the approach trajectory of an object may help distinguish and further constrain the neuronal computations required for collision avoidance in grasshoppers, fish and vinegar flies.

A directionally selective collision-sensing visual neural network based on fractional-order differential operator

In this paper, we propose a directionally selective fractional-order lobular giant motion detector (LGMD) visual neural network. Unlike most collision-sensing network models based on LGMDs, our model can not only sense collision threats but also obtain the motion direction of the collision object. Firstly, this paper simulates the membrane potential response of neurons using the fractional-order differential operator to generate reliable collision response spikes. Then, a new correlation mechanism is proposed to obtain the motion direction of objects. Specifically, this paper performs correlation operation on the signals extracted from two pixels, utilizing the temporal delay of the signals to obtain their position relationship. In this way, the response characteristics of direction-selective neurons can be characterized. Finally, ON/OFF visual channels are introduced to encode increases and decreases in brightness, respectively, thereby modeling the bipolar response of special neurons. Extensive experimental results show that the proposed visual neural system conforms to the response characteristics of biological LGMD and direction-selective neurons, and that the performance of the system is stable and reliable.

Robustness of Bio-Inspired Visual Systems for Collision Prediction in Critical Robot Traffic

Collision prevention sets a major research and development obstacle for intelligent robots and vehicles. This paper investigates the robustness of two state-of-the-art neural network models inspired by the locust’s LGMD-1 and LGMD-2 visual pathways as fast and low-energy collision alert systems in critical scenarios. Although both the neural circuits have been studied and modelled intensively, their capability and robustness against real-time critical traffic scenarios where real-physical crashes will happen have never been systematically investigated due to difficulty and high price in replicating risky traffic with many crash occurrences. To close this gap, we apply a recently published robotic platform to test the LGMDs inspired visual systems in physical implementation of critical traffic scenarios at low cost and high flexibility. The proposed visual systems are applied as the only collision sensing modality in each micro-mobile robot to conduct avoidance by abrupt braking. The simulated traffic resembles on-road sections including the intersection and highway scenes wherein the roadmaps are rendered by coloured, artificial pheromones upon a wide LCD screen acting as the ground of an arena. The robots with light sensors at bottom can recognise the lanes and signals, tightly follow paths. The emphasis herein is laid on corroborating the robustness of LGMDs neural systems model in different dynamic robot scenes to timely alert potential crashes. This study well complements previous experimentation on such bio-inspired computations for collision prediction in more critical physical scenarios, and for the first time demonstrates the robustness of LGMDs inspired visual systems in critical traffic towards a reliable collision alert system under constrained computation power. This paper also exhibits a novel, tractable, and affordable robotic approach to evaluate online visual systems in dynamic scenes.

The complex synaptic pathways onto a looming-detector neuron revealed using serial block-face scanning electron microscopy

The ability of locusts to detect looming stimuli and avoid collisions or predators depends on a neuronal circuit in the locust's optic lobe. Although comprehensively studied for over three decades, there are still major questions about the computational steps of this circuit. We used fourth instar larvae of Locusta migratoria to describe the connection between the lobula giant movement detector 1 (LGMD1) neuron in the lobula complex and the upstream neuropil, the medulla. Serial block-face scanning electron microscopy (SBEM) was used to characterize the morphology of the connecting neurons termed trans-medullary afferent (TmA) neurons and their synaptic connectivity. This enabled us to trace neurons over several hundred micrometers between the medulla and the lobula complex while identifying their synapses. We traced two different TmA neurons, each from a different individual, from their synapses with the LGMD in the lobula complex up into the medulla and describe their synaptic relationships. There is not a simple downstream transmission of the signal from a lamina neuron onto these TmA neurons; there is also a feedback loop in place with TmA neurons making outputs as well as receiving inputs. More than one type of neuron shapes the signal of the TmA neurons in the medulla. We found both columnar and trans-columnar neurons connected with the traced TmA neurons in the medulla. These findings indicate that there are computational steps in the medulla that have not been included in models of the neuronal pathway for looming detection.

A Robust Collision Perception Visual Neural Network With Specific Selectivity to Darker Objects

Building an efficient and reliable collision perception visual system is a challenging problem for future robots and autonomous vehicles. The biological visual neural networks, which have evolved over millions of years in nature and are working perfectly in the real world, could be ideal models for designing artificial vision systems. In the locust's visual pathways, a lobula giant movement detector (LGMD), that is, the LGMD2, has been identified as a looming perception neuron that responds most strongly to darker approaching objects relative to their backgrounds; similar situations which many ground vehicles and robots are often faced with. However, little has been done on modeling the LGMD2 and investigating its potential in robotics and vehicles. In this article, we build an LGMD2 visual neural network which possesses the similar collision selectivity of an LGMD2 neuron in locust via the modeling of biased-ON and -OFF pathways splitting visual signals into parallel ON/OFF channels. With stronger inhibition (bias) in the ON pathway, this model responds selectively to darker looming objects. The proposed model has been tested systematically with a range of stimuli including real-world scenarios. It has also been implemented in a micro-mobile robot and tested with real-time experiments. The experimental results have verified the effectiveness and robustness of the proposed model for detecting darker looming objects against various dynamic and cluttered backgrounds.

Molecular characterization and distribution of the voltage-gated sodium channel, Para, in the brain of the grasshopper and vinegar fly

Voltage-gated sodium (NaV) channels, encoded by the gene para, play a critical role in the rapid processing and propagation of visual information related to collision avoidance behaviors. We investigated their localization by immunostaining the optic lobes and central brain of the grasshopper Schistocerca americana and the vinegar fly Drosophila melanogaster with an antibody that recognizes the channel peptide domain responsible for fast inactivation gating. NaV channels were detected at high density at all stages of development. In the optic lobe, they revealed stereotypically repeating fascicles consistent with the regular structure of the eye. In the central brain, major axonal tracts were strongly labeled, particularly in the grasshopper olfactory system. We used the NaV channel sequence of Drosophila to identify an ortholog in the transcriptome of Schistocerca. The grasshopper, vinegar fly, and human NaV channels exhibit a high degree of conservation at gating and ion selectivity domains. Comparison with three species evolutionarily close to Schistocerca identified splice variants of Para and their relation to those of Drosophila. The anatomical distribution of NaV channels molecularly analogous to those of humans in grasshoppers and vinegar flies provides a substrate for rapid signal propagation and visual processing in the context of visually-guided collision avoidance.

Towards Computational Models and Applications of Insect Visual Systems for Motion Perception: A Review

Motion perception is a critical capability determining a variety of aspects of insects' life, including avoiding predators, foraging, and so forth. A good number of motion detectors have been identified in the insects' visual pathways. Computational modeling of these motion detectors has not only been providing effective solutions to artificial intelligence, but also benefiting the understanding of complicated biological visual systems. These biological mechanisms through millions of years of evolutionary development will have formed solid modules for constructing dynamic vision systems for future intelligent machines. This article reviews the computational motion perception models originating from biological research on insects' visual systems in the literature. These motion perception models or neural networks consist of the looming-sensitive neuronal models of lobula giant movement detectors (LGMDs) in locusts, the translation-sensitive neural systems of direction-selective neurons (DSNs) in fruit flies, bees, and locusts, and the small-target motion detectors (STMDs) in dragonflies and hoverflies. We also review the applications of these models to robots and vehicles. Through these modeling studies, we summarize the methodologies that generate different direction and size selectivity in motion perception. Finally, we discuss multiple systems integration and hardware realization of these bio-inspired motion perception models.

Shaping the collision selectivity in a looming sensitive neuron model with parallel ON and OFF pathways and spike frequency adaptation

Shaping the collision selectivity in vision-based artificial collision-detecting systems is still an open challenge. This paper presents a novel neuron model of a locust looming detector, i.e. the lobula giant movement detector (LGMD1), in order to provide effective solutions to enhance the collision selectivity of looming objects over other visual challenges. We propose an approach to model the biologically plausible mechanisms of ON and OFF pathways and a biophysical mechanism of spike frequency adaptation (SFA) in the proposed LGMD1 visual neural network. The ON and OFF pathways can separate both dark and light looming features for parallel spatiotemporal computations. This works effectively on perceiving a potential collision from dark or light objects that approach; such a bio-plausible structure can also separate LGMD1's collision selectivity to its neighbouring looming detector - the LGMD2. The SFA mechanism can enhance the LGMD1's collision selectivity to approaching objects rather than receding and translating stimuli, which is a significant improvement compared with similar LGMD1 neuron models. The proposed framework has been tested using off-line tests of synthetic and real-world stimuli, as well as on-line bio-robotic tests. The enhanced collision selectivity of the proposed model has been validated in systematic experiments. The computational simplicity and robustness of this work have also been verified by the bio-robotic tests, which demonstrates potential in building neuromorphic sensors for collision detection in both a fast and reliable manner.

Two identified looming detectors in the locust: ubiquitous lateral connections among their inputs contribute to selective responses to looming objects

In locusts, two lobula giant movement detector neurons (LGMDs) act as looming object detectors. Their reproducible responses to looming and their ethological significance makes them models for single neuron computation. But there is no comprehensive picture of the neurons that connect directly to each LGMD. We used high-through-put serial block-face scanning-electron-microscopy to reconstruct the network of input-synapses onto the LGMDs over spatial scales ranging from single synapses and small circuits, up to dendritic branches and total excitatory input. Reconstructions reveal that many trans-medullary-afferents (TmAs) connect the eye with each LGMD, one TmA per facet per LGMD. But when a TmA synapses with an LGMD it also connects laterally with another TmA. These inter-TmA synapses are always reciprocal. Total excitatory input to the LGMD 1 and 2 comes from 131,000 and 186,000 synapses reaching densities of 3.1 and 2.6 synapses per μm² respectively. We explored the computational consequences of reciprocal synapses between each TmA and 6 others from neighbouring columns. Since any lateral interactions between LGMD inputs have always been inhibitory we may assume these reciprocal lateral connections are most likely inhibitory. Such reciprocal inhibitory synapses increased the LGMD’s selectivity for looming over passing objects, particularly at the beginning of object approach.

Optimizing the 3D-reconstruction technique for serial block-face scanning electron microscopy

BACKGROUND

Elucidating the anatomy of neuronal circuits and localizing the synaptic connections between neurons, can give us important insights in how the neuronal circuits work. We are using serial block-face scanning electron microscopy (SBEM) to investigate the anatomy of a collision detection circuit including the Lobula Giant Movement Detector (LGMD) neuron in the locust, Locusta migratoria. For this, thousands of serial electron micrographs are produced that allow us to trace the neuronal branching pattern.

NEW METHOD

The reconstruction of neurons was previously done manually by drawing cell outlines of each cell in each image separately. This approach was very time consuming and troublesome. To make the process more efficient a new interactive software was developed. It uses the contrast between the neuron under investigation and its surrounding for semi-automatic segmentation.

RESULTS

For segmentation the user sets starting regions manually and the algorithm automatically selects a volume within the neuron until the edges corresponding to the neuronal outline are reached. Internally the algorithm optimizes a 3D active contour segmentation model formulated as a cost function taking the SEM image edges into account. This reduced the reconstruction time, while staying close to the manual reference segmentation result.

COMPARISON WITH EXISTING METHODS

Our algorithm is easy to use for a fast segmentation process, unlike previous methods it does not require image training nor an extended computing capacity.

CONCLUSION

Our semi-automatic segmentation algorithm led to a dramatic reduction in processing time for the 3D-reconstruction of identified neurons.

Object approach computation by a giant neuron and its relation with the speed of escape in the crab Neohelice

Upon detection of an approaching object the crab Neohelice granulata continuously regulates the direction and speed of escape according to ongoing visual information. These visuomotor transformations are thought to be largely accounted for by a small number of motion-sensitive giant neurons projecting from the lobula (third optic neuropil) towards the supraesophageal ganglion. One of these elements, the monostratified lobula giant neurons of type 2 (MLG2), proved to be highly sensitive to looming stimuli (a 2D representation of an object approach). By performing in vivo intracellular recordings we assessed the response of the MLG2 neuron to a variety of looming stimuli representing objects of different sizes and velocities of approach. This allowed us: a) to identify some of the physiological mechanisms involved in the regulation of the MLG2 activity and to test a simplified biophysical model of its response to looming stimuli; b) to identify the stimulus optical parameters encoded by the MLG2, and to formulate a phenomenological model able to predict the temporal course of the neural firing responses to all looming stimuli; c) to incorporate the MLG2 encoded information of the stimulus (in terms of firing rate) into a mathematical model able to fit the speed of the escape run of the animal. The agreement between the model predictions and the actual escape speed measured on a treadmill for all tested stimuli strengthens our interpretation of the computations performed by the MLG2 and of the involvement of this neuron in the regulation of the animal's speed of run while escaping from objects approaching with constant speed.