A model for the detection of moving targets in visual clutter inspired by insect physiology

Temperature modulates the tuning properties of small target motion detector neurons in the dragonfly visual system

Dragonflies are poikilothermic animals with limited thermoregulation; therefore, their entire bodies, including the brain, experience a range of temperatures during their daily activities.1,2 These flying insects exhibit hunting prowess, pursuing prey or conspecifics whether in direct sunlight or under the cover of cloud.3,4 Likely to underlie these aerobatic feats are the small target motion detector (STMD) neurons.5 These visual neurons are sensitive to target contrast and tuned to the target's size and velocity, with some neurons exhibiting complex predictive and selective properties, well suited for prey interception and feeding amid swarms.3,4,6,7,8,9 Increased temperature can modulate the biochemical processes underlying neuronal processing, increasing sensitivity and quickening the responsiveness of insect photoreceptors and downstream optic flow neurons,10,11,12 while in other neuronal pathways, compensatory processes have been shown to account for temperature changes.13,14 We determined the ethological range of temperatures experienced by the dragonfly, Hemicordulia tau, in its natural environment. Across this behaviorally relevant range, we showed increased temperatures having a large 8.7-fold increase in the contrast sensitivity of STMD neurons. However, suppression of responses to larger targets was unaltered. STMD tuning for target velocities was changed remarkably, not only increasing the optimum but extending the fastest velocities encoded by an order of magnitude. These results caution against interpreting functionality underlying spike rates at constrained, experimental temperatures. Moreover, they raise intriguing new questions about how information is represented within the brain of these flying insects, given the relationship between visual stimulus parameters and neuronal activity varies so dramatically depending on current environmental conditions.

Localized and Long-Lasting Adaptation in Dragonfly Target-Detecting Neurons

Some visual neurons in the dragonfly (Hemicordulia tau) optic lobe respond to small, moving targets, likely underlying their fast pursuit of prey and conspecifics. In response to repetitive targets presented at short intervals, the spiking activity of these "small target motion detector" (STMD) neurons diminishes over time. Previous experiments limited this adaptation by including intertrial rest periods of varying durations. However, the characteristics of this effect have never been quantified. Here, using extracellular recording techniques lasting for several hours, we quantified both the spatial and temporal properties of STMD adaptation. We found that the time course of adaptation was variable across STMD units. In any one STMD, a repeated series led to more rapid adaptation, a minor accumulative effect more akin to habituation. Following an adapting stimulus, responses recovered quickly, though the rate of recovery decreased nonlinearly over time. We found that the region of adaptation is highly localized, with targets displaced by ∼2.5° eliciting a naive response. Higher frequencies of target stimulation converged to lower levels of sustained response activity. We determined that adaptation itself is a target-tuned property, not elicited by moving bars or luminance flicker. As STMD adaptation is a localized phenomenon, dependent on recent history, it is likely to play an important role in closed-loop behavior where a target is foveated in a localized region for extended periods of the pursuit duration.

Visually guided swarm motion coordination via insect-inspired small target motion reactions

Despite progress developing experimentally-consistent models of insect in-flight sensing and feedback for individual agents, a lack of systematic understanding of the multi-agent and group performance of the resulting bio-inspired sensing and feedback approaches remains a barrier to robotic swarm implementations. This study introduces the small-target motion reactive (STMR) swarming approach by designing a concise engineering model of the small target motion detector (STMD) neurons found in insect lobula complexes. The STMD neuron model identifies the bearing angle at which peak optic flow magnitude occurs, and this angle is used to design an output feedback switched control system. A theoretical stability analysis provides bi-agent stability and state boundedness in group contexts. The approach is simulated and implemented on ground vehicles for validation and behavioral studies. The results indicate despite having the lowest connectivity of contemporary approaches (each agent instantaneously regards only a single neighbor), STMR achieves collective group motion. STMR group level metric analysis also highlights continuously varying polarization and decreasing heading variance.

Moving object detection based on bioinspired background subtraction

Flying insects rely mainly upon visual motion to detect and track objects. There has been a lot of research on fly inspired algorithms for object detection, but few have been developed based on visual motion alone. One of the daunting difficulties is that the neural and circuit mechanisms underlying the foreground-background segmentation are still unclear. Our previous modeling study proposed that the lobula held parallel pathways with distinct directional selectivity, each of which could retinotopically discriminate figures moving in its own preferred direction based on relative motion cues. The previous model, however, did not address how the multiple parallel pathways gave the only detection output at their common downstream. Since the preferred directions of the pathways along either horizontal or vertical axis were opposite to each other, the background moving in the opposite direction to an object also activated the corresponding lobula pathway. Indiscriminate or ungated projection from all the pathways to their downstream would mix objects with the moving background, making the previous model fail with non-stationary background. Here, we extend the previous model by proposing that the background motion-dependent gating of individual lobula projections is the key to object detection. Large-field lobula plate tangential cells are hypothesized to perform the gating to realize bioinspired background subtraction. The model is shown to be capable of implementing a robust detection of moving objects in video sequences with either a moving camera that induces translational optic flow or a static camera. The model sheds light on the potential of the concise fly algorithm in real-world applications.

Dragonfly visual evolutionary neural network: A novel bionic optimizer with related LSGO and engineering design optimization

Biological visual systems intrinsically include multiple kinds of motion-sensitive neurons. Some of them have been successfully used to construct neural computational models for problem-specific engineering applications such as motion detection, object tracking, etc. Nevertheless, it remains unclear how these neurons' response mechanisms can be contributed to the topic of optimization. Hereby, the dragonfly's visual response mechanism is integrated with the inspiration of swarm evolution to develop a dragonfly visual evolutionary neural network for large-scale global optimization (LSGO) problems. Therein, a grayscale image input-based dragonfly visual neural network online outputs multiple global learning rates, and later, such learning rates guide a population evolution-like state update strategy to seek the global optimum. The comparative experiments show that the neural network is a competitive optimizer capable of effectively solving LSGO benchmark suites with 2000 dimensions per example and the design of an operational amplifier.

A Bio-Inspired Probabilistic Neural Network Model for Noise-Resistant Collision Perception

Bio-inspired models based on the lobula giant movement detector (LGMD) in the locust's visual brain have received extensive attention and application for collision perception in various scenarios. These models offer advantages such as low power consumption and high computational efficiency in visual processing. However, current LGMD-based computational models, typically organized as four-layered neural networks, often encounter challenges related to noisy signals, particularly in complex dynamic environments. Biological studies have unveiled the intrinsic stochastic nature of synaptic transmission, which can aid neural computation in mitigating noise. In alignment with these biological findings, this paper introduces a probabilistic LGMD (Prob-LGMD) model that incorporates a probability into the synaptic connections between multiple layers, thereby capturing the uncertainty in signal transmission, interaction, and integration among neurons. Comparative testing of the proposed Prob-LGMD model and two conventional LGMD models was conducted using a range of visual stimuli, including indoor structured scenes and complex outdoor scenes, all subject to artificial noise. Additionally, the model's performance was compared to standard engineering noise-filtering methods. The results clearly demonstrate that the proposed model outperforms all comparative methods, exhibiting a significant improvement in noise tolerance. This study showcases a straightforward yet effective approach to enhance collision perception in noisy environments.

Descending neurons of the hoverfly respond to pursuits of artificial targets

Many animals use motion vision information to control dynamic behaviors. Predatory animals, for example, show an exquisite ability to detect rapidly moving prey, followed by pursuit and capture. Such target detection is not only used by predators but is also important in conspecific interactions, such as for male hoverflies defending their territories against conspecific intruders. Visual target detection is believed to be subserved by specialized target-tuned neurons found in a range of species, including vertebrates and arthropods. However, how these target-tuned neurons respond to actual pursuit trajectories is currently not well understood. To redress this, we recorded extracellularly from target-selective descending neurons (TSDNs) in male Eristalis tenax hoverflies. We show that they have dorso-frontal receptive fields with a preferred direction up and away from the visual midline. We reconstructed visual flow fields as experienced during pursuits of artificial targets (black beads). We recorded TSDN responses to six reconstructed pursuits and found that each neuron responded consistently at remarkably specific time points but that these time points differed between neurons. We found that the observed spike probability was correlated with the spike probability predicted from each neuron's receptive field and size tuning. Interestingly, however, the overall response rate was low, with individual neurons responding to only a small part of each reconstructed pursuit. In contrast, the TSDN population responded to substantially larger proportions of the pursuits but with lower probability. This large variation between neurons could be useful if different neurons control different parts of the behavioral output.

Attention and Prediction-Guided Motion Detection for Low-Contrast Small Moving Targets

Small target motion detection within complex natural environments is an extremely challenging task for autonomous robots. Surprisingly, the visual systems of insects have evolved to be highly efficient in detecting mates and tracking prey, even though targets occupy as small as a few degrees of their visual fields. The excellent sensitivity to small target motion relies on a class of specialized neurons, called small target motion detectors (STMDs). However, existing STMD-based models are heavily dependent on visual contrast and perform poorly in complex natural environments, where small targets generally exhibit extremely low contrast against neighboring backgrounds. In this article, we develop an attention-and-prediction-guided visual system to overcome this limitation. The developed visual system comprises three main subsystems, namely: 1) an attention module; 2) an STMD-based neural network; and 3) a prediction module. The attention module searches for potential small targets in the predicted areas of the input image and enhances their contrast against a complex background. The STMD-based neural network receives the contrast-enhanced image and discriminates small moving targets from background false positives. The prediction module foresees future positions of the detected targets and generates a prediction map for the attention module. The three subsystems are connected in a recurrent architecture, allowing information to be processed sequentially to activate specific areas for small target detection. Extensive experiments on synthetic and real-world datasets demonstrate the effectiveness and superiority of the proposed visual system for detecting small, low-contrast moving targets against complex natural environments.

Motion perception based on ON/OFF channels: A survey

Motion perception is an essential ability for animals and artificially intelligent systems interacting effectively, safely with surrounding objects and environments. Biological visual systems, that have naturally evolved over hundreds-million years, are quite efficient and robust for motion perception, whereas artificial vision systems are far from such capability. This paper argues that the gap can be significantly reduced by formulation of ON/OFF channels in motion perception models encoding luminance increment (ON) and decrement (OFF) responses within receptive field, separately. Such signal-bifurcating structure has been found in neural systems of many animal species articulating early motion is split and processed in segregated pathways. However, the corresponding biological substrates, and the necessity for artificial vision systems have never been elucidated together, leaving concerns on uniqueness and advantages of ON/OFF channels upon building dynamic vision systems to address real world challenges. This paper highlights the importance of ON/OFF channels in motion perception through surveying current progress covering both neuroscience and computationally modelling works with applications. Compared to related literature, this paper for the first time provides insights into implementation of different selectivity to directional motion of looming, translating, and small-sized target movement based on ON/OFF channels in keeping with soundness and robustness of biological principles. Existing challenges and future trends of such bio-plausible computational structure for visual perception in connection with hotspots of machine learning, advanced vision sensors like event-driven camera finally are discussed.

Dual Receptive Fields Underlying Target and Wide-Field Motion Sensitivity in Looming-Sensitive Descending Neurons

Responding rapidly to visual stimuli is fundamental for many animals. For example, predatory birds and insects alike have amazing target detection abilities, with incredibly short neural and behavioral delays, enabling efficient prey capture. Similarly, looming objects need to be rapidly avoided to ensure immediate survival, as these could represent approaching predators. Male Eristalis tenax hoverflies are nonpredatory, highly territorial insects that perform high-speed pursuits of conspecifics and other territorial intruders. During the initial stages of the pursuit, the retinal projection of the target is very small, but this grows to a larger object before physical interaction. Supporting such behaviors, E. tenax and other insects have both target-tuned and loom-sensitive neurons in the optic lobes and the descending pathways. We here show that these visual stimuli are not necessarily encoded in parallel. Indeed, we describe a class of descending neurons that respond to small targets, to looming and to wide-field stimuli. We show that these descending neurons have two distinct receptive fields where the dorsal receptive field is sensitive to the motion of small targets and the ventral receptive field responds to larger objects or wide-field stimuli. Our data suggest that the two receptive fields have different presynaptic input, where the inputs are not linearly summed. This novel and unique arrangement could support different behaviors, including obstacle avoidance, flower landing, and target pursuit or capture.

Bioinspired figure-ground discrimination via visual motion smoothing

Flies detect and track moving targets among visual clutter, and this process mainly relies on visual motion. Visual motion is analyzed or computed with the pathway from the retina to T4/T5 cells. The computation of local directional motion was formulated as an elementary movement detector (EMD) model more than half a century ago. Solving target detection or figure-ground discrimination problems can be equivalent to extracting boundaries between a target and the background based on the motion discontinuities in the output of a retinotopic array of EMDs. Individual EMDs cannot measure true velocities, however, due to their sensitivity to pattern properties such as luminance contrast and spatial frequency content. It remains unclear how local directional motion signals are further integrated to enable figure-ground discrimination. Here, we present a computational model inspired by fly motion vision. Simulations suggest that the heavily fluctuating output of an EMD array is naturally surmounted by a lobula network, which is hypothesized to be downstream of the local motion detectors and have parallel pathways with distinct directional selectivity. The lobula network carries out a spatiotemporal smoothing operation for visual motion, especially across time, enabling the segmentation of moving figures from the background. The model qualitatively reproduces experimental observations in the visually evoked response characteristics of one type of lobula columnar (LC) cell. The model is further shown to be robust to natural scene variability. Our results suggest that the lobula is involved in local motion-based target detection.

A Time-Delay Feedback Neural Network for Discriminating Small, Fast-Moving Targets in Complex Dynamic Environments

Discriminating small moving objects within complex visual environments is a significant challenge for autonomous micro-robots that are generally limited in computational power. By exploiting their highly evolved visual systems, flying insects can effectively detect mates and track prey during rapid pursuits, even though the small targets equate to only a few pixels in their visual field. The high degree of sensitivity to small target movement is supported by a class of specialized neurons called small target motion detectors (STMDs). Existing STMD-based computational models normally comprise four sequentially arranged neural layers interconnected via feedforward loops to extract information on small target motion from raw visual inputs. However, feedback, another important regulatory circuit for motion perception, has not been investigated in the STMD pathway and its functional roles for small target motion detection are not clear. In this article, we propose an STMD-based neural network with feedback connection (feedback STMD), where the network output is temporally delayed, then fed back to the lower layers to mediate neural responses. We compare the properties of the model with and without the time-delay feedback loop and find that it shows a preference for high-velocity objects. Extensive experiments suggest that the feedback STMD achieves superior detection performance for fast-moving small targets, while significantly suppressing background false positive movements which display lower velocities. The proposed feedback model provides an effective solution in robotic visual systems for detecting fast-moving small targets that are always salient and potentially threatening.

Exploration of motion inhibition for the suppression of false positives in biologically inspired small target detection algorithms from a moving platform

Detecting small moving targets against a cluttered background in visual data is a challenging task. The main problems include spatio-temporal target contrast enhancement, background suppression and accurate target segmentation. When targets are at great distances from a non-stationary camera, the difficulty of these challenges increases. In such cases the moving camera can introduce large spatial changes between frames which may cause issues in temporal algorithms; furthermore targets can approach a single pixel, thereby affecting spatial methods. Previous literature has shown that biologically inspired methods, based on the vision systems of insects, are robust to such conditions. It has also been shown that the use of divisive optic-flow inhibition with these methods enhances the detectability of small targets. However, the location within the visual pathway the inhibition should be applied was ambiguous. In this paper, we investigated the tunings of some of the optic-flow filters and use of a nonlinear transform on the optic-flow signal to modify motion responses for the purpose of suppressing false positives and enhancing small target detection. Additionally, we looked at multiple locations within the biologically inspired vision (BIV) algorithm where inhibition could further enhance detection performance, and look at driving the nonlinear transform with a global motion estimate. To get a better understanding of how the BIV algorithm performs, we compared to other state-of-the-art target detection algorithms, and look at how their performance can be enhanced with the optic-flow inhibition. Our explicit use of the nonlinear inhibition allows for the incorporation of a wider dynamic range of inhibiting signals, along with spatio-temporal filter refinement, which further increases target-background discrimination in the presence of camera motion. Extensive experiments shows that our proposed approach achieves an improvement of 25% over linearly conditioned inhibition schemes and 2.33 times the detection performance of the BIV model without inhibition. Moreover, our approach achieves between 10 and 104 times better detection performance compared to any conventional state-of-the-art moving object detection algorithm applied to the same, highly cluttered and moving scenes. Applying the nonlinear inhibition to other algorithms showed that their performance can be increased by up to 22 times. These findings show that the application of optic-flow- based signal suppression should be applied to enhance target detection from moving platforms. Furthermore, they indicate where best to look for evidence of such signals within the insect brain.

Visual and motor signatures of locomotion dynamically shape a population code for feature detection in Drosophila

Natural vision is dynamic: as an animal moves, its visual input changes dramatically. How can the visual system reliably extract local features from an input dominated by self-generated signals? In Drosophila, diverse local visual features are represented by a group of projection neurons with distinct tuning properties. Here, we describe a connectome-based volumetric imaging strategy to measure visually evoked neural activity across this population. We show that local visual features are jointly represented across the population, and a shared gain factor improves trial-to-trial coding fidelity. A subset of these neurons, tuned to small objects, is modulated by two independent signals associated with self-movement, a motor-related signal, and a visual motion signal associated with rotation of the animal. These two inputs adjust the sensitivity of these feature detectors across the locomotor cycle, selectively reducing their gain during saccades and restoring it during intersaccadic intervals. This work reveals a strategy for reliable feature detection during locomotion.

Mathematical study of neural feedback roles in small target motion detection

Building an efficient and reliable small target motion detection visual system is challenging for artificial intelligence robotics because a small target only occupies few pixels and hardly displays visual features in images. Biological visual systems that have evolved over millions of years could be ideal templates for designing artificial visual systems. Insects benefit from a class of specialized neurons, called small target motion detectors (STMDs), which endow them with an excellent ability to detect small moving targets against a cluttered dynamic environment. Some bio-inspired models featured in feed-forward information processing architectures have been proposed to imitate the functions of the STMD neurons. However, feedback, a crucial mechanism for visual system regulation, has not been investigated deeply in the STMD-based neural circuits and its roles in small target motion detection remain unclear. In this paper, we propose a time-delay feedback STMD model for small target motion detection in complex backgrounds. The main contributions of this study are as follows. First, a feedback pathway is designed by transmitting information from output-layer neurons to lower-layer interneurons in the STMD pathway and the role of the feedback is analyzed from the view of mathematical analysis. Second, to estimate the feedback constant, the existence and uniqueness of solutions for nonlinear dynamical systems formed by feedback loop are analyzed via Schauder's fixed point theorem and contraction mapping theorem. Finally, an iterative algorithm is designed to solve the nonlinear problem and the performance of the proposed model is tested by experiments. Experimental results demonstrate that the feedback is able to weaken background false positives while maintaining a minor effect on small targets. It outperforms existing STMD-based models regarding the accuracy of fast-moving small target detection in visual clutter. The proposed feedback approach could inspire the relevant modeling of robust motion perception robotics visual systems.

A Looming Spatial Localization Neural Network Inspired by MLG1 Neurons in the Crab Neohelice

Similar to most visual animals, the crab Neohelice granulata relies predominantly on visual information to escape from predators, to track prey and for selecting mates. It, therefore, needs specialized neurons to process visual information and determine the spatial location of looming objects. In the crab Neohelice granulata, the Monostratified Lobula Giant type1 (MLG1) neurons have been found to manifest looming sensitivity with finely tuned capabilities of encoding spatial location information. MLG1s neuronal ensemble can not only perceive the location of a looming stimulus, but are also thought to be able to influence the direction of movement continuously, for example, escaping from a threatening, looming target in relation to its position. Such specific characteristics make the MLG1s unique compared to normal looming detection neurons in invertebrates which can not localize spatial looming. Modeling the MLG1s ensemble is not only critical for elucidating the mechanisms underlying the functionality of such neural circuits, but also important for developing new autonomous, efficient, directionally reactive collision avoidance systems for robots and vehicles. However, little computational modeling has been done for implementing looming spatial localization analogous to the specific functionality of MLG1s ensemble. To bridge this gap, we propose a model of MLG1s and their pre-synaptic visual neural network to detect the spatial location of looming objects. The model consists of 16 homogeneous sectors arranged in a circular field inspired by the natural arrangement of 16 MLG1s' receptive fields to encode and convey spatial information concerning looming objects with dynamic expanding edges in different locations of the visual field. Responses of the proposed model to systematic real-world visual stimuli match many of the biological characteristics of MLG1 neurons. The systematic experiments demonstrate that our proposed MLG1s model works effectively and robustly to perceive and localize looming information, which could be a promising candidate for intelligent machines interacting within dynamic environments free of collision. This study also sheds light upon a new type of neuromorphic visual sensor strategy that can extract looming objects with locational information in a quick and reliable manner.

Nonlinear, neuronal adaptation in insect vision models improves target discrimination within repetitively moving backgrounds

Neurons which respond selectively to small moving targets, even against a cluttered background, have been identified in several insect species. To investigate what underlies these robust and highly selective responses, researchers have probed the neuronal circuitry in target-detecting, visual pathways. Observations in flies reveal nonlinear adaptation over time, composed of a fast onset and gradual decay. This adaptive processing is seen in both of the independent, parallel pathways encoding either luminance increments (ON channel) or decrements (OFF channel). The functional significance of this adaptive phenomenon has not been determined from physiological studies, though the asymmetrical time course suggests a role in suppressing responses to repetitive stimuli. We tested this possibility by comparing an implementation of fast adaptation against alternatives, using a model of insect 'elementary small target motion detectors'. We conducted target-detecting simulations on various natural backgrounds, that were shifted via several movement profiles (and target velocities). Using performance metrics, we confirmed that the fast adaptation observed in neuronal systems enhances target detection against a repetitively moving background. Such background movement would be encountered via natural ego-motion as the insect travels through the world. These findings show that this form of nonlinear, fast-adaptation (suitably implementable via cellular biophysics) plays a role analogous to background subtraction techniques in conventional computer vision.

Facilitation of neural responses to targets moving against optic flow

Target detection in visual clutter is a difficult computational task that insects, with their poor spatial resolution compound eyes and small brains, do successfully and with extremely short behavioral delays. We here show that the responses of target selective descending neurons are attenuated by background motion in the same direction as target motion but facilitated by background motion in the opposite direction. This finding is important for understanding how target pursuit can occur in tandem with gaze stabilization. Indeed, the neural facilitation would come into effect if the hoverfly is subjected to background motion in one direction but the target it is pursuing moves in the opposite direction and could therefore be used to override gaze stabilizing corrective turns.

For the human observer, it can be difficult to follow the motion of small objects, especially when they move against background clutter. In contrast, insects efficiently do this, as evidenced by their ability to capture prey, pursue conspecifics, or defend territories, even in highly textured surrounds. We here recorded from target selective descending neurons (TSDNs), which likely subserve these impressive behaviors. To simulate the type of optic flow that would be generated by the pursuer’s own movements through the world, we used the motion of a perspective corrected sparse dot field. We show that hoverfly TSDN responses to target motion are suppressed when such optic flow moves syn-directional to the target. Indeed, neural responses are strongly suppressed when targets move over either translational sideslip or rotational yaw. More strikingly, we show that TSDNs are facilitated by optic flow moving counterdirectional to the target, if the target moves horizontally. Furthermore, we show that a small, frontal spatial window of optic flow is enough to fully facilitate or suppress TSDN responses to target motion. We argue that such TSDN response facilitation could be beneficial in modulating corrective turns during target pursuit.

Modeling Nonlinear Dendritic Processing of Facilitation in a Dragonfly Target-Tracking Neuron

Dragonflies are highly skilled and successful aerial predators that are even capable of selectively attending to one target within a swarm. Detection and tracking of prey is likely to be driven by small target motion detector (STMD) neurons identified from several insect groups. Prior work has shown that dragonfly STMD responses are facilitated by targets moving on a continuous path, enhancing the response gain at the present and predicted future location of targets. In this study, we combined detailed morphological data with computational modeling to test whether a combination of dendritic morphology and nonlinear properties of NMDA receptors could explain these observations. We developed a hybrid computational model of neurons within the dragonfly optic lobe, which integrates numerical and morphological components. The model was able to generate potent facilitation for targets moving on continuous trajectories, including a localized spotlight of maximal sensitivity close to the last seen target location, as also measured during in vivo recordings. The model did not, however, include a mechanism capable of producing a traveling or spreading wave of facilitation. Our data support a strong role for the high dendritic density seen in the dragonfly neuron in enhancing non-linear facilitation. An alternative model based on the morphology of an unrelated type of motion processing neuron from a dipteran fly required more than three times higher synaptic gain in order to elicit similar levels of facilitation, despite having only 20% fewer synapses. Our data support a potential role for NMDA receptors in target tracking and also demonstrate the feasibility of combining biologically plausible dendritic computations with more abstract computational models for basic processing as used in earlier studies.

Inhibitory Interactions and Columnar Inputs to an Object Motion Detector in Drosophila

The direction-selective T4/T5 cells innervate optic-flow processing projection neurons in the lobula plate of the fly that mediate the visual control of locomotion. In the lobula, visual projection neurons coordinate complex behavioral responses to visual features, however, the input circuitry and computations that bestow their feature-detecting properties are less clear. Here, we study a highly specialized small object motion detector, LC11, and demonstrate that its responses are suppressed by local background motion. We show that LC11 expresses GABA-A receptors that serve to sculpt responses to small objects but are not responsible for the rejection of background motion. Instead, LC11 is innervated by columnar T2 and T3 neurons that are themselves highly sensitive to small static or moving objects, insensitive to wide-field motion and, unlike T4/T5, respond to both ON and OFF luminance steps.

Detecting Small Size and Minimal Thermal Signature Targets in Infrared Imagery Using Biologically Inspired Vision

Thermal infrared imaging provides an effective sensing modality for detecting small moving objects at long range. Typical challenges that limit the efficiency and robustness of the detection performance include sensor noise, minimal target contrast and cluttered backgrounds. These issues become more challenging when the targets are of small physical size and present minimal thermal signatures. In this paper, we experimentally show that a four-stage biologically inspired vision (BIV) model of the flying insect visual system have an excellent ability to overcome these challenges simultaneously. The early two stages of the model suppress spatio-temporal clutter and enhance spatial target contrast while compressing the signal in a computationally manageable bandwidth. The later two stages provide target motion enhancement and sub-pixel motion detection capabilities. To show the superiority of the BIV target detector over existing traditional detection methods, we perform extensive experiments and performance comparisons using high bit-depth, real-world infrared image sequences of small size and minimal thermal signature targets at long ranges. Our results show that the BIV target detector significantly outperformed 10 conventional spatial-only and spatiotemporal methods for infrared small target detection. The BIV target detector resulted in over 25 dB improvement in the median signal-to-clutter-ratio over the raw input and achieved 43% better detection rate than the best performing existing method.

Bio-Inspired Video Enhancement for Small Moving Target Detection

Moving targets at a very large distance from a camera appear small and of low contrast. The low signal-to-noise-ratio and the presence of clutter in the background degrade the detection performance of conventional moving object detection techniques. To address these challenges, we propose temporal pre-processing of video frames using a biologically-inspired vision model. The bio-inspired model consists of multiple layers of processing analogous to the photoreceptor cells in the visual system of small insects. The adaptive filtering mechanism in the photoreceptor cells suppresses clutter and expands the possible range of input signal changes which improves the target background contrast. We perform experiments on real world video sequences of small moving targets captured with a high bit depth, high resolution and high frame-rate camera. Experimental results show that the biological vision based pre-processing leads to improved detection performance when used in conjunction with a variety of computer vision based moving object detection algorithms. The temporal bio-processing alone has improved the area under the receiver operating characteristic (AUROC) curve of the best performing algorithm by 75.4%. Our results suggest that the bio-inspired pre-processing has strong potential to become a key component of a practical small target detection system.

Modelling Drosophila motion vision pathways for decoding the direction of translating objects against cluttered moving backgrounds

Decoding the direction of translating objects in front of cluttered moving backgrounds, accurately and efficiently, is still a challenging problem. In nature, lightweight and low-powered flying insects apply motion vision to detect a moving target in highly variable environments during flight, which are excellent paradigms to learn motion perception strategies. This paper investigates the fruit fly Drosophila motion vision pathways and presents computational modelling based on cutting-edge physiological researches. The proposed visual system model features bio-plausible ON and OFF pathways, wide-field horizontal-sensitive (HS) and vertical-sensitive (VS) systems. The main contributions of this research are on two aspects: (1) the proposed model articulates the forming of both direction-selective and direction-opponent responses, revealed as principal features of motion perception neural circuits, in a feed-forward manner; (2) it also shows robust direction selectivity to translating objects in front of cluttered moving backgrounds, via the modelling of spatiotemporal dynamics including combination of motion pre-filtering mechanisms and ensembles of local correlators inside both the ON and OFF pathways, which works effectively to suppress irrelevant background motion or distractors, and to improve the dynamic response. Accordingly, the direction of translating objects is decoded as global responses of both the HS and VS systems with positive or negative output indicating preferred-direction or null-direction translation. The experiments have verified the effectiveness of the proposed neural system model, and demonstrated its responsive preference to faster-moving, higher-contrast and larger-size targets embedded in cluttered moving backgrounds.

Insect Neurobiology: How a Small Spot Stops a Fly

Animals often respond to small moving features very differently than they do to large moving fields. A new study finds that viewing small spots causes walking fruit flies to stop in their tracks, and identifies the cellular pathway that processes this signal.

Non-canonical Receptive Field Properties and Neuromodulation of Feature-Detecting Neurons in Flies

Several fundamental aspects of motion vision circuitry are prevalent across flies and mice. Both taxa segregate ON and OFF signals. For any given spatial pattern, motion detectors in both taxa are tuned to speed, selective for one of four cardinal directions, and modulated by catecholamine neurotransmitters. These similarities represent conserved, canonical properties of the functional circuits and computational algorithms for motion vision. Less is known about feature detectors, including how receptive field properties differ from the motion pathway or whether they are under neuromodulatory control to impart functional plasticity for the detection of salient objects from a moving background. Here, we investigated 19 types of putative feature selective lobula columnar (LC) neurons in the optic lobe of the fruit fly Drosophila melanogaster to characterize divergent properties of feature selection. We identified LC12 and LC15 as feature detectors. LC15 encodes moving bars, whereas LC12 is selective for the motion of discrete objects, mostly independent of size. Neither is selective for contrast polarity, speed, or direction, highlighting key differences in the underlying algorithms for feature detection and motion vision. We show that the onset of background motion suppresses object responses by LC12 and LC15. Surprisingly, the application of octopamine, which is released during flight, reverses the suppressive influence of background motion, rendering both LCs able to track moving objects superimposed against background motion. Our results provide a comparative framework for the function and modulation of feature detectors and new insights into the underlying neuronal mechanisms involved in visual feature detection.

Object-Displacement-Sensitive Visual Neurons Drive Freezing in Drosophila

Visual systems are often equipped with neurons that detect small moving objects, which may represent prey, predators, or conspecifics. Although the processing properties of those neurons have been studied in diverse organisms, links between the proposed algorithms and animal behaviors or circuit mechanisms remain elusive. Here, we have investigated behavioral function, computational algorithm, and neurochemical mechanisms of an object-selective neuron, LC11, in Drosophila. With genetic silencing and optogenetic activation, we show that LC11 is necessary for a visual object-induced stopping behavior in walking flies, a form of short-term freezing, and its activity can promote stopping. We propose a new quantitative model for small object selectivity based on the physiology and anatomy of LC11 and its inputs. The model accurately reproduces LC11 responses by pooling fast-adapting, tightly size-tuned inputs. Direct visualization of neurotransmitter inputs to LC11 confirmed the model conjectures about upstream processing. Our results demonstrate how adaptation can enhance selectivity for behaviorally relevant, dynamic visual features.

Crossing the Cleft: Communication Challenges Between Neuroscience and Artificial Intelligence

Historically, neuroscience principles have heavily influenced artificial intelligence (AI), for example the influence of the perceptron model, essentially a simple model of a biological neuron, on artificial neural networks. More recently, notable recent AI advances, for example the growing popularity of reinforcement learning, often appear more aligned with cognitive neuroscience or psychology, focusing on function at a relatively abstract level. At the same time, neuroscience stands poised to enter a new era of large-scale high-resolution data and appears more focused on underlying neural mechanisms or architectures that can, at times, seem rather removed from functional descriptions. While this might seem to foretell a new generation of AI approaches arising from a deeper exploration of neuroscience specifically for AI, the most direct path for achieving this is unclear. Here we discuss cultural differences between the two fields, including divergent priorities that should be considered when leveraging modern-day neuroscience for AI. For example, the two fields feed two very different applications that at times require potentially conflicting perspectives. We highlight small but significant cultural shifts that we feel would greatly facilitate increased synergy between the two fields.

A Directionally Selective Small Target Motion Detecting Visual Neural Network in Cluttered Backgrounds

Discriminating targets moving against a cluttered background is a huge challenge, let alone detecting a target as small as one or a few pixels and tracking it in flight. In the insect's visual system, a class of specific neurons, called small target motion detectors (STMDs), have been identified as showing exquisite selectivity for small target motion. Some of the STMDs have also demonstrated direction selectivity which means these STMDs respond strongly only to their preferred motion direction. Direction selectivity is an important property of these STMD neurons which could contribute to tracking small targets such as mates in flight. However, little has been done on systematically modeling these directionally selective STMD neurons. In this paper, we propose a directionally selective STMD-based neural network for small target detection in a cluttered background. In the proposed neural network, a new correlation mechanism is introduced for direction selectivity via correlating signals relayed from two pixels. Then, a lateral inhibition mechanism is implemented on the spatial field for size selectivity of the STMD neurons. Finally, a population vector algorithm is used to encode motion direction of small targets. Extensive experiments showed that the proposed neural network not only is in accord with current biological findings, i.e., showing directional preferences but also worked reliably in detecting the small targets against cluttered backgrounds.

A Robust Visual System for Small Target Motion Detection Against Cluttered Moving Backgrounds

Monitoring small objects against cluttered moving backgrounds is a huge challenge to future robotic vision systems. As a source of inspiration, insects are quite apt at searching for mates and tracking prey, which always appear as small dim speckles in the visual field. The exquisite sensitivity of insects for small target motion, as revealed recently, is coming from a class of specific neurons called small target motion detectors (STMDs). Although a few STMD-based models have been proposed, these existing models only use motion information for small target detection and cannot discriminate small targets from small-target-like background features (named fake features). To address this problem, this paper proposes a novel visual system model (STMD+) for small target motion detection, which is composed of four subsystems-ommatidia, motion pathway, contrast pathway, and mushroom body. Compared with the existing STMD-based models, the additional contrast pathway extracts directional contrast from luminance signals to eliminate false positive background motion. The directional contrast and the extracted motion information by the motion pathway are integrated into the mushroom body for small target discrimination. Extensive experiments showed the significant and consistent improvements of the proposed visual system model over the existing STMD-based models against fake features.

Neural Network Model for Detection of Edges Defined by Image Dynamics

Insects can detect the presence of discrete objects in their visual fields based on a range of differences in spatiotemporal characteristics between the images of object and background. This includes but is not limited to relative motion. Evidence suggests that edge detection is an integral part of this capability, and this study examines the ability of a bio-inspired processing model to detect the presence of boundaries between two regions of a one-dimensional visual field, based on general differences in image dynamics. The model consists of two parts. The first is an early vision module inspired by insect visual processing, which implements adaptive photoreception, ON and OFF channels with transient and sustained characteristics, and delayed and undelayed signal paths. This is replicated for a number of photoreceptors in a small linear array. It is followed by an artificial neural network trained to discriminate the presence vs. absence of an edge based on the array output signals. Input data are derived from natural imagery and feature both static and moving edges between regions with moving texture, flickering texture, and static patterns in all possible combinations. The model can discriminate the presence of edges, stationary or moving, at rates far higher than chance. The resources required (numbers of neurons and visual signals) are realistic relative to those available in the insect second optic ganglion, where the bulk of such processing would be likely to take place.

Differential Tuning to Visual Motion Allows Robust Encoding of Optic Flow in the Dragonfly

Visual cues provide an important means for aerial creatures to ascertain their self-motion through the environment. In many insects, including flies, moths, and bees, wide-field motion-sensitive neurons in the third optic ganglion are thought to underlie such motion encoding; however, these neurons can only respond robustly over limited speed ranges. The task is more complicated for some species of dragonflies that switch between extended periods of hovering flight and fast-moving pursuit of prey and conspecifics, requiring motion detection over a broad range of velocities. Since little is known about motion processing in these insects, we performed intracellular recordings from hawking, emerald dragonflies (Hemicordulia spp.) and identified a diverse group of motion-sensitive neurons that we named lobula tangential cells (LTCs). Following prolonged visual stimulation with drifting gratings, we observed significant differences in both temporal and spatial tuning of LTCs. Cluster analysis of these changes confirmed several groups of LTCs with distinctive spatiotemporal tuning. These differences were associated with variation in velocity tuning in response to translated, natural scenes. LTCs with differences in velocity tuning ranges and optima may underlie how a broad range of motion velocities are encoded. In the hawking dragonfly, changes in LTC tuning over time are therefore likely to support their extensive range of behaviors, from hovering to fast-speed pursuits.SIGNIFICANCE STATEMENT Understanding how animals navigate the world is an inherently difficult and interesting problem. Insects are useful models for understanding neuronal mechanisms underlying these activities, with neurons that encode wide-field motion previously identified in insects, such as flies, hawkmoths, and butterflies. Like some Dipteran flies, dragonflies exhibit complex aerobatic behaviors, such as hovering, patrolling, and aerial combat. However, dragonflies lack halteres that support such diverse behavior in flies. To understand how dragonflies might address this problem using only visual cues, we recorded from their wide-field motion-sensitive neurons. We found these differ strongly in the ways they respond to sustained motion, allowing them collectively to encode the very broad range of velocities experienced during diverse behavior.

Properties of predictive gain modulation in a dragonfly visual neuron

Dragonflies pursue and capture tiny prey and conspecifics with extremely high success rates. These moving targets represent a small visual signal on the retina and successful chases require accurate detection and amplification by downstream neuronal circuits. This amplification has been observed in a population of neurons called small target motion detectors (STMDs), through a mechanism we term predictive gain modulation. As targets drift through the neuron's receptive field, spike frequency builds slowly over time. This increased likelihood of spiking or gain is modulated across the receptive field, enhancing sensitivity just ahead of the target's path, with suppression of activity in the remaining surround. Whilst some properties of this mechanism have been described, it is not yet known which stimulus parameters modulate the amount of response gain. Previous work suggested that the strength of gain enhancement was predominantly determined by the duration of the target's prior path. Here, we show that predictive gain modulation is more than a slow build-up of responses over time. Rather, the strength of gain is dependent on the velocity of a prior stimulus combined with the current stimulus attributes (e.g. angular size). We also describe response variability as a major challenge of target-detecting neurons and propose that the role of predictive gain modulation is to drive neurons towards response saturation, thus minimising neuronal variability despite noisy visual input signals.

Towards Computational Models and Applications of Insect Visual Systems for Motion Perception: A Review

Motion perception is a critical capability determining a variety of aspects of insects' life, including avoiding predators, foraging, and so forth. A good number of motion detectors have been identified in the insects' visual pathways. Computational modeling of these motion detectors has not only been providing effective solutions to artificial intelligence, but also benefiting the understanding of complicated biological visual systems. These biological mechanisms through millions of years of evolutionary development will have formed solid modules for constructing dynamic vision systems for future intelligent machines. This article reviews the computational motion perception models originating from biological research on insects' visual systems in the literature. These motion perception models or neural networks consist of the looming-sensitive neuronal models of lobula giant movement detectors (LGMDs) in locusts, the translation-sensitive neural systems of direction-selective neurons (DSNs) in fruit flies, bees, and locusts, and the small-target motion detectors (STMDs) in dragonflies and hoverflies. We also review the applications of these models to robots and vehicles. Through these modeling studies, we summarize the methodologies that generate different direction and size selectivity in motion perception. Finally, we discuss multiple systems integration and hardware realization of these bio-inspired motion perception models.

Integration of Small- and Wide-Field Visual Features in Target-Selective Descending Neurons of both Predatory and Nonpredatory Dipterans

For many animals, target motion carries high ecological significance as this may be generated by a predator, prey, or potential mate. Indeed, animals whose survival depends on early target detection are often equipped with a sharply tuned visual system, yielding robust performance in challenging conditions. For example, many fast-flying insects use visual cues for identifying targets, such as prey (e.g., predatory dragonflies and robberflies) or conspecifics (e.g., nonpredatory hoverflies), and can often do so against self-generated background optic flow. Supporting these behaviors, the optic lobes of insects that pursue targets harbor neurons that respond robustly to the motion of small moving objects, even when displayed against syn-directional background clutter. However, in diptera, the encoding of target information by the descending neurons, which are more directly involved in generating the behavioral output, has received less attention. We characterized target-selective neurons by recording in the ventral nerve cord of male and female predatory Holcocephala fusca robberflies and of male nonpredatory Eristalis tenax hoverflies. We show that both species have dipteran target-selective descending neurons that only respond to target motion if the background is stationary or moving slowly, moves in the opposite direction, or has un-naturalistic spatial characteristics. The response to the target is suppressed when background and target move at similar velocities, which is strikingly different to the response of target neurons in the optic lobes. As the neurons we recorded from are premotor, our findings affect our interpretation of the neurophysiology underlying target-tracking behaviors.

SIGNIFICANCE STATEMENT Many animals use sensory cues to detect moving targets that may represent predators, prey, or conspecifics. For example, birds of prey show superb sensitivity to the motion of small prey, and intercept these at high speeds. In a similar manner, predatory insects visually track moving prey, often against cluttered backgrounds. Accompanying this behavior, the brains of insects that pursue targets contain neurons that respond exclusively to target motion. We here show that dipteran insects also have target-selective descending neurons in the part of their nervous system that corresponds to the vertebrate spinal cord. Surprisingly, and in contrast to the neurons in the brain, these premotor neurons are inhibited by background patterns moving in the same direction as the target.

Shaping the collision selectivity in a looming sensitive neuron model with parallel ON and OFF pathways and spike frequency adaptation

Shaping the collision selectivity in vision-based artificial collision-detecting systems is still an open challenge. This paper presents a novel neuron model of a locust looming detector, i.e. the lobula giant movement detector (LGMD1), in order to provide effective solutions to enhance the collision selectivity of looming objects over other visual challenges. We propose an approach to model the biologically plausible mechanisms of ON and OFF pathways and a biophysical mechanism of spike frequency adaptation (SFA) in the proposed LGMD1 visual neural network. The ON and OFF pathways can separate both dark and light looming features for parallel spatiotemporal computations. This works effectively on perceiving a potential collision from dark or light objects that approach; such a bio-plausible structure can also separate LGMD1's collision selectivity to its neighbouring looming detector - the LGMD2. The SFA mechanism can enhance the LGMD1's collision selectivity to approaching objects rather than receding and translating stimuli, which is a significant improvement compared with similar LGMD1 neuron models. The proposed framework has been tested using off-line tests of synthetic and real-world stimuli, as well as on-line bio-robotic tests. The enhanced collision selectivity of the proposed model has been validated in systematic experiments. The computational simplicity and robustness of this work have also been verified by the bio-robotic tests, which demonstrates potential in building neuromorphic sensors for collision detection in both a fast and reliable manner.

Linear Summation Underlies Direction Selectivity in Drosophila

While linear mechanisms lay the foundations of feature selectivity in many brain areas, direction selectivity in the elementary motion detector (EMD) of the fly has become a paradigm of nonlinear neuronal computation. We have bridged this divide by demonstrating that linear spatial summation can generate direction selectivity in the fruit fly Drosophila. Using linear systems analysis and two-photon imaging of a genetically encoded voltage indicator, we measure the emergence of direction-selective (DS) voltage signals in the Drosophila OFF pathway. Our study is a direct, quantitative investigation of the algorithm underlying directional signals, with the striking finding that linear spatial summation is sufficient for the emergence of direction selectivity. A linear stage of the fly EMD strongly resembles similar computations in vertebrate visual cortex, demands a reappraisal of the role of upstream nonlinearities, and implicates the voltage-to-calcium transformation in the refinement of feature selectivity in this system. VIDEO ABSTRACT.

Spatial Encoding of Translational Optic Flow in Planar Scenes by Elementary Motion Detector Arrays

Elementary Motion Detectors (EMD) are well-established models of visual motion estimation in insects. The response of EMDs are tuned to specific temporal and spatial frequencies of the input stimuli, which matches the behavioural response of insects to wide-field image rotation, called the optomotor response. However, other behaviours, such as speed and position control, cannot be fully accounted for by EMDs because these behaviours are largely unaffected by image properties and appear to be controlled by the ratio between the flight speed and the distance to an object, defined here as relative nearness. We present a method that resolves this inconsistency by extracting an unambiguous estimate of relative nearness from the output of an EMD array. Our method is suitable for estimation of relative nearness in planar scenes such as when flying above the ground or beside large flat objects. We demonstrate closed loop control of the lateral position and forward velocity of a simulated agent flying in a corridor. This finding may explain how insects can measure relative nearness and control their flight despite the frequency tuning of EMDs. Our method also provides engineers with a relative nearness estimation technique that benefits from the low computational cost of EMDs.

A predictive focus of gain modulation encodes target trajectories in insect vision

When a human catches a ball, they estimate future target location based on the current trajectory. How animals, small and large, encode such predictive processes at the single neuron level is unknown. Here we describe small target-selective neurons in predatory dragonflies that exhibit localized enhanced sensitivity for targets displaced to new locations just ahead of the prior path, with suppression elsewhere in the surround. This focused region of gain modulation is driven by predictive mechanisms, with the direction tuning shifting selectively to match the target’s prior path. It involves a large local increase in contrast gain which spreads forward after a delay (e.g. an occlusion) and can even transfer between brain hemispheres, predicting trajectories moved towards the visual midline from the other eye. The tractable nature of dragonflies for physiological experiments makes this a useful model for studying the neuronal mechanisms underlying the brain’s remarkable ability to anticipate moving stimuli.

DOI:http://dx.doi.org/10.7554/eLife.26478.001

Catching a ball requires a person to track the speed and direction of a small moving target often against a cluttered and varying background. Predatory insects, like dragonflies, face a similar challenge when they pursue their prey through the air. The task is made a little easier, however, by the fact that most moving targets tend to follow predictable trajectories. Indeed, animals are also better at tracking targets that follow smooth continuous trajectories, suggesting that brains have evolved to exploit the normal behavior of visual stimuli to reduce their workload

To find out how this process works, Wiederman, Fabian et al. studied the brains of dragonflies as they watched a black square intended to mimic prey. Brain cells called Small Target Motion Detectors (or STMD neurons for short) became more active in response to the target. But rather than simply following the target, the STMD neurons instead predicted its future location. In fact, individual neurons were more sensitive to movements occurring just ahead of the target’s current position, and less sensitive to movements occurring elsewhere.

If the target abruptly disappeared, the point in space where the neurons were most sensitive to movement continued to gradually move forward over time. Given that real-life targets typically disappear when they move behind other objects, this suggests that the brain is predicting where the target is most likely to reappear. The STMD neurons became more sensitive to movement by increasing their ability to detect differences in brightness between the target and its background. In some cases, the neurons increased their sensitivity more than five-fold.

Insects and mammals last shared a common ancestor more than 500 million years ago, and, in many respects, mammalian brains are substantially more complex than insect brains. Nevertheless, the results of Wiederman, Fabian et al. show that the insect brain can perform visual tasks that were previously associated only with mammals. Neuroscientists and engineers have used the insect brain for decades to study the circuits that support biological processes. In the coming years, insects such as the dragonfly may enable us to explore how visual regions of the brain predict future events. This knowledge could ultimately be applied to artificial vision systems, such as those in self-driving cars.

DOI:http://dx.doi.org/10.7554/eLife.26478.002

Performance of an insect-inspired target tracker in natural conditions

Robust and efficient target-tracking algorithms embedded on moving platforms, are a requirement for many computer vision and robotic applications. However, deployment of a real-time system is challenging, even with the computational power of modern hardware. As inspiration, we look to biological lightweight solutions-lightweight and low-powered flying insects. For example, dragonflies pursue prey and mates within cluttered, natural environments, deftly selecting their target amidst swarms. In our laboratory, we study the physiology and morphology of dragonfly 'small target motion detector' neurons likely to underlie this pursuit behaviour. Here we describe our insect-inspired tracking model derived from these data and compare its efficacy and efficiency with state-of-the-art engineering models. For model inputs, we use both publicly available video sequences, as well as our own task-specific dataset (small targets embedded within natural scenes). In the context of the tracking problem, we describe differences in object statistics within the video sequences. For the general dataset, our model often locks on to small components of larger objects, tracking these moving features. When input imagery includes small moving targets, for which our highly nonlinear filtering is matched, the robustness outperforms state-of-the-art trackers. In all scenarios, our insect-inspired tracker runs at least twice the speed of the comparison algorithms.