From skylight input to behavioural output: A computational model of the insect polarised light compass

Passive Polarized Vision for Autonomous Vehicles: A Review

This review article aims to address common research questions in passive polarized vision for robotics. What kind of polarization sensing can we embed into robots? Can we find our geolocation and true north heading by detecting light scattering from the sky as animals do? How should polarization images be related to the physical properties of reflecting surfaces in the context of scene understanding? This review article is divided into three main sections to address these questions, as well as to assist roboticists in identifying future directions in passive polarized vision for robotics. After an introduction, three key interconnected areas will be covered in the following sections: embedded polarization imaging; polarized vision for robotics navigation; and polarized vision for scene understanding. We will then discuss how polarized vision, a type of vision commonly used in the animal kingdom, should be implemented in robotics; this type of vision has not yet been exploited in robotics service. Passive polarized vision could be a supplemental perceptive modality of localization techniques to complement and reinforce more conventional ones.

Skylight Polarization Pattern Simulator Based on a Virtual-Real-Fusion Framework for Urban Bionic Polarization Navigation

In a data-driven context, bionic polarization navigation requires a mass of skylight polarization pattern data with diversity, complete ground truth, and scene information. However, acquiring such data in urban environments, where bionic polarization navigation is widely utilized, remains challenging. In this paper, we proposed a virtual-real-fusion framework of the skylight polarization pattern simulator and provided a data preparation method complementing the existing pure simulation or measurement method. The framework consists of a virtual part simulating the ground truth of skylight polarization pattern, a real part measuring scene information, and a fusion part fusing information of the first two parts according to the imaging projection relationship. To illustrate the framework, we constructed a simulator instance adapted to the urban environment and clear weather and verified it in 174 urban scenes. The results showed that the simulator can provide a mass of diverse urban skylight polarization pattern data with scene information and complete ground truth based on a few practical measurements. Moreover, we released a dataset based on the results and opened our code to facilitate researchers preparing and adapting their datasets to their research targets.

SkyPole-A method for locating the north celestial pole from skylight polarization patterns

True north can be determined on Earth by three means: magnetic compasses, stars, and via the global navigation satellite systems (GNSS), each of which has its own drawbacks. GNSS are sensitive to jamming and spoofing, magnetic compasses are vulnerable to magnetic interferences, and the stars can be used only at night with a clear sky. As an alternative to these methods, nature-inspired navigational cues are of particular interest. Celestial polarization, which is used by insects such as Cataglyphis ants, can provide useful directional cues. Migrating birds calibrate their magnetic compasses by observing the celestial rotation at night. By combining these cues, we have developed a bioinspired optical method for finding the celestial pole during the daytime. This method, which we have named SkyPole, is based on the rotation of the skylight polarization pattern. A polarimetric camera was used to measure the degree of skylight polarization rotating with the Sun. Image difference processes were then applied to the time-varying measurements in order to determine the north celestial pole's position and thus the observer's latitude and bearing with respect to the true north.

Biomimetic Polarized Light Navigation Sensor: A Review

A polarized light sensor is applied to the front-end detection of a biomimetic polarized light navigation system, which is an important part of analyzing the atmospheric polarization mode and realizing biomimetic polarized light navigation, having received extensive attention in recent years. In this paper, biomimetic polarized light navigation in nature, the mechanism of polarized light navigation, point source sensor, imaging sensor, and a sensor based on micro nano machining technology are compared and analyzed, which provides a basis for the optimal selection of different polarized light sensors. The comparison results show that the point source sensor can be divided into basic point source sensor with simple structure and a point source sensor applied to integrated navigation. The imaging sensor can be divided into a simple time-sharing imaging sensor, a real-time amplitude splitting sensor that can detect images of multi-directional polarization angles, a real-time aperture splitting sensor that uses a light field camera, and a real-time focal plane light splitting sensor with high integration. In recent years, with the development of micro and nano machining technology, polarized light sensors are developing towards miniaturization and integration. In view of this, this paper also summarizes the latest progress of polarized light sensors based on micro and nano machining technology. Finally, this paper summarizes the possible future prospects and current challenges of polarized light sensor design, providing a reference for the feasibility selection of different polarized light sensors.

A virtuous cycle between invertebrate and robotics research: perspective on a decade of Living Machines research

Many invertebrates are ideal model systems on which to base robot design principles due to their success in solving seemingly complex tasks across domains while possessing smaller nervous systems than vertebrates. Three areas are particularly relevant for robot designers: Research on flying and crawling invertebrates has inspired new materials and geometries from which robot bodies (their morphologies) can be constructed, enabling a new generation of softer, smaller, and lighter robots. Research on walking insects has informed the design of new systems for controlling robot bodies (their motion control) and adapting their motion to their environment without costly computational methods. And research combining wet and computational neuroscience with robotic validation methods has revealed the structure and function of core circuits in the insect brain responsible for the navigation and swarming capabilities (their mental faculties) displayed by foraging insects. The last decade has seen significant progress in the application of principles extracted from invertebrates, as well as the application of biomimetic robots to model and better understand how animals function. This Perspectives paper on the past 10 years of the Living Machines conference outlines some of the most exciting recent advances in each of these fields before outlining lessons gleaned and the outlook for the next decade of invertebrate robotic research.

Polarized light compass decoding

The brains of some insects can encode and decode polarization information and obtain heading angle information. Referring to the encoding ability of insects, exponential function encoding is designed to improve the stability of the polarized light compass artificial neural network. However, in the decoding process, only neurons with the largest activation degree are used for decoding (maximum value decoding), so the heading information contained in other neurons is not used. Therefore, average value decoding (AVD) and weighted AVD are proposed to use the heading information contained in multiple neurons to determine the heading. In addition, concerning the phenomenon of threshold activation of insect neurons, threshold value decoding (TVD) and weighted TVD are proposed, which can effectively eliminate the interference of neurons with low activation. Moreover, this paper proposes to improve the heading determination accuracy of the artificial neural network through pre-training. The simulation and experimental results show that the new, to the best of our knowledge, decoding methods and pre-training can effectively improve the heading determination accuracy of the artificial neural network.

Polarized light sun position determination artificial neural network

Our previous work has constructed a polarized light orientation determination (PLOD) artificial neural network. Although a PLOD network can determine the solar azimuth angle, it cannot determine the solar elevation angle. Therefore, this paper proposes an artificial neural network for polarized light solar position determination (PLSPD), which has two branches: the solar azimuth angle determination branch and the solar elevation angle determination branch. Since the solar elevation angle has no cyclic characteristics, and the angle range of the solar elevation angle is different from that of the solar azimuth angle, the solar elevation angle exponential function encoding is redesigned. In addition, compared with the PLOD, the PLSPD deletes a local full connection layer to simplify the network structure. The experimental results show that the PLSPD can determine not only the solar azimuth angle but also the solar elevation angle, and the solar azimuth angle determination accuracy of the PLSPD is higher than that of the PLOD.

Synaptic targets of photoreceptors specialized to detect color and skylight polarization in Drosophila

Color and polarization provide complementary information about the world and are detected by specialized photoreceptors. However, the downstream neural circuits that process these distinct modalities are incompletely understood in any animal. Using electron microscopy, we have systematically reconstructed the synaptic targets of the photoreceptors specialized to detect color and skylight polarization in Drosophila, and we have used light microscopy to confirm many of our findings. We identified known and novel downstream targets that are selective for different wavelengths or polarized light, and followed their projections to other areas in the optic lobes and the central brain. Our results revealed many synapses along the photoreceptor axons between brain regions, new pathways in the optic lobes, and spatially segregated projections to central brain regions. Strikingly, photoreceptors in the polarization-sensitive dorsal rim area target fewer cell types, and lack strong connections to the lobula, a neuropil involved in color processing. Our reconstruction identifies shared wiring and modality-specific specializations for color and polarization vision, and provides a comprehensive view of the first steps of the pathways processing color and polarized light inputs.

How the insect central complex could coordinate multimodal navigation

The central complex of the insect midbrain is thought to coordinate insect guidance strategies. Computational models can account for specific behaviours, but their applicability across sensory and task domains remains untested. Here, we assess the capacity of our previous model (Sun et al. 2020) of visual navigation to generalise to olfactory navigation and its coordination with other guidance in flies and ants. We show that fundamental to this capacity is the use of a biologically plausible neural copy-and-shift mechanism that ensures sensory information is presented in a format compatible with the insect steering circuit regardless of its source. Moreover, the same mechanism is shown to allow the transfer cues from unstable/egocentric to stable/geocentric frames of reference, providing a first account of the mechanism by which foraging insects robustly recover from environmental disturbances. We propose that these circuits can be flexibly repurposed by different insect navigators to address their unique ecological needs.

An optimized pulse coupled neural network image de-noising method for a field-programmable gate array based polarization camera

The quality of polarization images is easy to be affected by the noise in the image acquired by a polarization camera. Consequently, a de-noising method optimized with a Pulse Coupled Neural Network (PCNN) for polarization images is proposed for a Field-Programmable Gate Array (FPGA)-based polarization camera in this paper, in which the polarization image de-noising is implemented using an adaptive PCNN improved by Gray Wolf Optimization (GWO) and Bi-Dimensional Empirical Mode Decomposition (BEMD). Unlike other artificial neural networks, PCNN does not need to be trained, but the parameters of PCNN such as the exponential decay time constant, the synaptic junction strength factor, and the inherent voltage constant play a critical influence on its de-noising performance. GWO is able to start optimization by generating a set of random solutions as the first population and saves the optimized solutions of PCNN. In addition, BEMD can decompose a complicated image into different Bi-Dimensional Intrinsic Mode Functions with local stabilized characteristics according to the input source image, and the decomposition result is able to lower the complexity of heavy noise image analysis. Moreover, the circuit in the polarization camera is accomplished by FPGA so as to obtain the polarization image with higher quality synchronously. These two schemes are combined to attenuate different types of noises and improve the quality of the polarization image significantly. Compared with the state-of-the-art image de-noising algorithms, the noise in the polarization image is suppressed effectively by the proposed optimized image de-noising method according to the indices of peak signal-to-noise ratio, standard deviation, mutual information, structural similarity, and root mean square error.

Looking down: a model for visual route following in flying insects

Insect visual navigation is often assumed to depend on panoramic views of the horizon, and how these change as the animal moves. However, it is known that honey bees can visually navigate in flat, open meadows where visual information at the horizon is minimal, or would remain relatively constant across a wide range of positions. In this paper we hypothesise that these animals can navigate using view memories of the ground. We find that in natural scenes, low resolution views from an aerial perspective of ostensibly self-similar terrain (e.g. within a field of grass) provide surprisingly robust descriptors of precise spatial locations. We propose a new visual route following approach that makes use of transverse oscillations to centre a flight path along a sequence of learned views of the ground. We deploy this model on an autonomous quadcopter and demonstrate that it provides robust performance in the real world on journeys of up to 30 m. The success of our method is contingent on a robust view matching process which can evaluate the familiarity of a view with a degree of translational invariance. We show that a previously developed wavelet based bandpass orientated filter approach fits these requirements well, exhibiting double the catchment area of standard approaches. Using a realistic simulation package, we evaluate the robustness of our approach to variations in heading direction and aircraft height between inbound and outbound journeys. We also demonstrate that our approach can operate using a vision system with a biologically relevant visual acuity and viewing direction.

A visual pathway for skylight polarization processing in Drosophila

Many insects use patterns of polarized light in the sky to orient and navigate. Here, we functionally characterize neural circuitry in the fruit fly, Drosophila melanogaster, that conveys polarized light signals from the eye to the central complex, a brain region essential for the fly's sense of direction. Neurons tuned to the angle of polarization of ultraviolet light are found throughout the anterior visual pathway, connecting the optic lobes with the central complex via the anterior optic tubercle and bulb, in a homologous organization to the 'sky compass' pathways described in other insects. We detail how a consistent, map-like organization of neural tunings in the peripheral visual system is transformed into a reduced representation suited to flexible processing in the central brain. This study identifies computational motifs of the transformation, enabling mechanistic comparisons of multisensory integration and central processing for navigation in the brains of insects.

The Antarium: A Reconstructed Visual Reality Device for Ant Navigation Research

We constructed a large projection device (the Antarium) with 20,000 UV-Blue-Green LEDs that allows us to present tethered ants with views of their natural foraging environment. The ants walk on an air-cushioned trackball, their movements are registered and can be fed back to the visual panorama. Views are generated in a 3D model of the ants’ environment so that they experience the changing visual world in the same way as they do when foraging naturally. The Antarium is a biscribed pentakis dodecahedron with 55 facets of identical isosceles triangles. The length of the base of the triangles is 368 mm resulting in a device that is roughly 1 m in diameter. Each triangle contains 361 blue/green LEDs and nine UV LEDs. The 55 triangles of the Antarium have 19,855 Green and Blue pixels and 495 UV pixels, covering 360° azimuth and elevation from −50° below the horizon to +90° above the horizon. The angular resolution is 1.5° for Green and Blue LEDs and 6.7° for UV LEDs, offering 65,536 intensity levels at a flicker frequency of more than 9,000 Hz and a framerate of 190 fps. Also, the direction and degree of polarisation of the UV LEDs can be adjusted through polarisers mounted on the axles of rotary actuators. We build 3D models of the natural foraging environment of ants using purely camera-based methods. We reconstruct panoramic scenes at any point within these models, by projecting panoramic images onto six virtual cameras which capture a cube-map of images to be projected by the LEDs of the Antarium. The Antarium is a unique instrument to investigate visual navigation in ants. In an open loop, it allows us to provide ants with familiar and unfamiliar views, with completely featureless visual scenes, or with scenes that are altered in spatial or spectral composition. In closed-loop, we can study the behavior of ants that are virtually displaced within their natural foraging environment. In the future, the Antarium can also be used to investigate the dynamics of navigational guidance and the neurophysiological basis of ant navigation in natural visual environments.

Design and calibration model of a bioinspired attitude and heading reference system based on compound eye polarization compass

Insects such as honeybees are capable of fusing the information sensed by multiple sensory organs for attitude and heading determination. In this paper, inspired by the sensory fusion mechanism of insects' polarization compass and haltere, a bioinspired polarization-based attitude and heading reference system (PAHRS) is presented. The PAHRS consists of compound eye polarization compass and inertial measurement unit (IMU). By simulating multi-view structure of the dorsal rim area in insects' compound eyes, a non-coplanar 'polarization-opponent (POL)-type' architecture is adopted for the compound eye polarization compass. The polarization compass has multi-directional observation channels, which is capable of adaptively selecting the angle of polarization and obtaining the polarization vectors. Therefore, the environmental adaptability of the polarization compass can be enhanced. In addition, the integration strategy between the compound eye polarization compass and IMU is proposed. Moreover, the sources of system errors are analyzed to improve the heading angle accuracy, based on which a new calibration model is established to compensate the installation errors of the PAHRS. Finally, experiments are carried out under both clear sky and cloudy conditions. The test results show that the error root mean square of heading angle is 0.14° in clear sky, and 0.42° in partly cloudy conditions.

Matched-filter coding of sky polarization results in an internal sun compass in the brain of the desert locust

Many animals use celestial cues for spatial orientation. These include the sun and, in insects, the polarization pattern of the sky, which depends on the position of the sun. The central complex in the insect brain plays a key role in spatial orientation. In desert locusts, the angle of polarized light in the zenith above the animal and the direction of a simulated sun are represented in a compass-like fashion in the central complex, but how both compasses fit together for a unified representation of external space remained unclear. To address this question, we analyzed the sensitivity of intracellularly recorded central-complex neurons to the angle of polarized light presented from up to 33 positions in the animal's dorsal visual field and injected Neurobiotin tracer for cell identification. Neurons were polarization sensitive in large parts of the virtual sky that in some cells extended to the horizon in all directions. Neurons, moreover, were tuned to spatial patterns of polarization angles that matched the sky polarization pattern of particular sun positions. The horizontal components of these calculated solar positions were topographically encoded in the protocerebral bridge of the central complex covering 360° of space. This whole-sky polarization compass does not support the earlier reported polarization compass based on stimulation from a small spot above the animal but coincides well with the previously demonstrated direct sun compass based on unpolarized light stimulation. Therefore, direct sunlight and whole-sky polarization complement each other for robust head direction coding in the locust central complex.

A decentralised neural model explaining optimal integration of navigational strategies in insects

Insect navigation arises from the coordinated action of concurrent guidance systems but the neural mechanisms through which each functions, and are then coordinated, remains unknown. We propose that insects require distinct strategies to retrace familiar routes (route-following) and directly return from novel to familiar terrain (homing) using different aspects of frequency encoded views that are processed in different neural pathways. We also demonstrate how the Central Complex and Mushroom Bodies regions of the insect brain may work in tandem to coordinate the directional output of different guidance cues through a contextually switched ring-attractor inspired by neural recordings. The resultant unified model of insect navigation reproduces behavioural data from a series of cue conflict experiments in realistic animal environments and offers testable hypotheses of where and how insects process visual cues, utilise the different information that they provide and coordinate their outputs to achieve the adaptive behaviours observed in the wild.

Multimodal interactions in insect navigation

Animals travelling through the world receive input from multiple sensory modalities that could be important for the guidance of their journeys. Given the availability of a rich array of cues, from idiothetic information to input from sky compasses and visual information through to olfactory and other cues (e.g. gustatory, magnetic, anemotactic or thermal) it is no surprise to see multimodality in most aspects of navigation. In this review, we present the current knowledge of multimodal cue use during orientation and navigation in insects. Multimodal cue use is adapted to a species’ sensory ecology and shapes navigation behaviour both during the learning of environmental cues and when performing complex foraging journeys. The simultaneous use of multiple cues is beneficial because it provides redundant navigational information, and in general, multimodality increases robustness, accuracy and overall foraging success. We use examples from sensorimotor behaviours in mosquitoes and flies as well as from large scale navigation in ants, bees and insects that migrate seasonally over large distances, asking at each stage how multiple cues are combined behaviourally and what insects gain from using different modalities.