1
|
Loisy A, Heinonen RA. Deep reinforcement learning for the olfactory search POMDP: a quantitative benchmark. THE EUROPEAN PHYSICAL JOURNAL. E, SOFT MATTER 2023; 46:17. [PMID: 36939979 DOI: 10.1140/epje/s10189-023-00277-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Accepted: 03/05/2023] [Indexed: 06/18/2023]
Abstract
The olfactory search POMDP (partially observable Markov decision process) is a sequential decision-making problem designed to mimic the task faced by insects searching for a source of odor in turbulence, and its solutions have applications to sniffer robots. As exact solutions are out of reach, the challenge consists in finding the best possible approximate solutions while keeping the computational cost reasonable. We provide a quantitative benchmarking of a solver based on deep reinforcement learning against traditional POMDP approximate solvers. We show that deep reinforcement learning is a competitive alternative to standard methods, in particular to generate lightweight policies suitable for robots.
Collapse
Affiliation(s)
- Aurore Loisy
- Aix Marseille Univ, CNRS, Centrale Marseille, IRPHE, Marseille, France.
| | - Robin A Heinonen
- Department of Physics and INFN, University of Rome "Tor Vergata", Via della Ricerca Scientifica 1, 00133, Rome, Italy.
| |
Collapse
|
2
|
Fan S, Hao D, Sun X, Sultan YM, Li Z, Xia K. A Study of Modified Infotaxis Algorithms in 2D and 3D Turbulent Environments. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2020; 2020:4159241. [PMID: 32908473 PMCID: PMC7468623 DOI: 10.1155/2020/4159241] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/06/2019] [Revised: 03/16/2020] [Accepted: 07/06/2020] [Indexed: 11/17/2022]
Abstract
Emergency response to hazardous gases in the environment is an important research field in environmental monitoring. In recent years, with the rapid development of sensor technology and mobile device technology, more autonomous search algorithms for hazardous gas emission sources are proposed in uncertain environment, which can avoid emergency personnel from contacting hazardous gas in a short distance. Infotaxis is an autonomous search strategy without a concentration gradient, which uses scattered sensor data to track the location of the release source in turbulent environment. This paper optimizes the imbalance of exploitation and exploration in the reward function of Infotaxis algorithm and proposes a mobile strategy for the three-dimensional scene. In two-dimensional and three-dimensional scenes, the average steps of search tasks are used as the evaluation criteria to analyze the information trend algorithm combined with different reward functions and mobile strategies. The results show that the balance between the exploitation item and exploration item of the reward function proposed in this paper is better than that of the reward function in the Infotaxis algorithm, no matter in the two-dimensional scenes or in the three-dimensional scenes.
Collapse
Affiliation(s)
- Shurui Fan
- Tianjin Key Laboratory of Electronic Materials Devices, School of Electronic and Information Engineering, Hebei University of Technology, Tianjin 300401, China
| | - Dongxia Hao
- Tianjin Key Laboratory of Electronic Materials Devices, School of Electronic and Information Engineering, Hebei University of Technology, Tianjin 300401, China
| | - Xudong Sun
- Tianjin Key Laboratory of Electronic Materials Devices, School of Electronic and Information Engineering, Hebei University of Technology, Tianjin 300401, China
| | - Yusuf Mohamed Sultan
- Tianjin Key Laboratory of Electronic Materials Devices, School of Electronic and Information Engineering, Hebei University of Technology, Tianjin 300401, China
| | - Zirui Li
- Tianjin Key Laboratory of Electronic Materials Devices, School of Electronic and Information Engineering, Hebei University of Technology, Tianjin 300401, China
| | - Kewen Xia
- Tianjin Key Laboratory of Electronic Materials Devices, School of Electronic and Information Engineering, Hebei University of Technology, Tianjin 300401, China
| |
Collapse
|
3
|
Autonomous Searching for a Diffusive Source Based on Minimizing the Combination of Entropy and Potential Energy. SENSORS 2019; 19:s19112465. [PMID: 31146473 PMCID: PMC6603600 DOI: 10.3390/s19112465] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/16/2019] [Revised: 05/20/2019] [Accepted: 05/27/2019] [Indexed: 11/17/2022]
Abstract
The infotaxis scheme is a search strategy for a diffusive source, where the sensor platform is driven to reduce the uncertainty about the source through climbing the information gradient. The infotaxis scheme has been successfully applied in many source searching tasks and has demonstrated fast and stable searching capabilities. However, the infotaxis scheme focuses on gathering information to reduce the uncertainty down to zero, rather than chasing the most probable estimated source when a reliable estimation is obtained. This leads the sensor to spend more time exploring the space and yields a longer search path. In this paper, from the context of exploration-exploitation balance, a novel search scheme based on minimizing free energy that combines the entropy and the potential energy is proposed. The term entropy is implemented as the exploration to gather more information. The term potential energy, leveraging the distance to the estimated sources, is implemented as the exploitation to reinforce the chasing behavior with the receding of the uncertainty. It results in a faster effective search strategy by which the sensor determines its actions by minimizing the free energy rather than only the entropy in traditional infotaxis. Simulations of the source search task based on the computational plume verify the efficiency of the proposed strategy, achieving a shorter mean search time.
Collapse
|
4
|
Ristic B, Angley D, Moran B, Palmer JL. Autonomous Multi-Robot Search for a Hazardous Source in a Turbulent Environment. SENSORS 2017; 17:s17040918. [PMID: 28430120 PMCID: PMC5428082 DOI: 10.3390/s17040918] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/12/2017] [Revised: 04/11/2017] [Accepted: 04/18/2017] [Indexed: 11/16/2022]
Abstract
Finding the source of an accidental or deliberate release of a toxic substance into the atmosphere is of great importance for national security. The paper presents a search algorithm for turbulent environments which falls into the class of cognitive (infotaxi) algorithms. Bayesian estimation of the source parameter vector is carried out using the Rao-Blackwell dimension-reduction method, while the robots are controlled autonomously to move in a scalable formation. Estimation and control are carried out in a centralised replicated fusion architecture assuming all-to-all communication. The paper presents a comprehensive numerical analysis of the proposed algorithm, including the search-time and displacement statistics.
Collapse
Affiliation(s)
- Branko Ristic
- School of Engineering, RMIT University, Melbourne, VIC 3000, Australia.
| | - Daniel Angley
- School of Engineering, RMIT University, Melbourne, VIC 3000, Australia.
| | - Bill Moran
- School of Engineering, RMIT University, Melbourne, VIC 3000, Australia.
| | - Jennifer L Palmer
- Aerospace Division, Defence Science and Technology, Fishermans Bend, VIC 3207, Australia.
| |
Collapse
|
5
|
Zhang S, Martinez D, Masson JB. Multi-Robot Searching with Sparse Binary Cues and Limited Space Perception. Front Robot AI 2015. [DOI: 10.3389/frobt.2015.00012] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
|
6
|
Bau J, Cardé RT. Modeling Optimal Strategies for Finding a Resource-Linked, Windborne Odor Plume: Theories, Robotics, and Biomimetic Lessons from Flying Insects. Integr Comp Biol 2015; 55:461-77. [DOI: 10.1093/icb/icv036] [Citation(s) in RCA: 27] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/14/2022] Open
|
7
|
Voges N, Chaffiol A, Lucas P, Martinez D. Reactive searching and infotaxis in odor source localization. PLoS Comput Biol 2014; 10:e1003861. [PMID: 25330317 PMCID: PMC4211930 DOI: 10.1371/journal.pcbi.1003861] [Citation(s) in RCA: 49] [Impact Index Per Article: 4.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2014] [Accepted: 08/15/2014] [Indexed: 11/19/2022] Open
Abstract
Male moths aiming to locate pheromone-releasing females rely on stimulus-adapted search maneuvers complicated by a discontinuous distribution of pheromone patches. They alternate sequences of upwind surge when perceiving the pheromone and cross- or downwind casting when the odor is lost. We compare four search strategies: three reactive versus one cognitive. The former consist of pre-programmed movement sequences triggered by pheromone detections while the latter uses Bayesian inference to build spatial probability maps. Based on the analysis of triphasic responses of antennal lobe neurons (On, inhibition, Off), we propose three reactive strategies. One combines upwind surge (representing the On response to a pheromone detection) and spiral casting, only. The other two additionally include crosswind (zigzag) casting representing the Off phase. As cognitive strategy we use the infotaxis algorithm which was developed for searching in a turbulent medium. Detection events in the electroantennogram of a moth attached to a robot indirectly control this cyborg, depending on the strategy in use. The recorded trajectories are analyzed with regard to success rates, efficiency, and other features. In addition, we qualitatively compare our robotic trajectories to behavioral search paths. Reactive searching is more efficient (yielding shorter trajectories) for higher pheromone doses whereas cognitive searching works better for lower doses. With respect to our experimental conditions (2 m from starting position to pheromone source), reactive searching with crosswind zigzag yields the shortest trajectories (for comparable success rates). Assuming that the neuronal Off response represents a short-term memory, zigzagging is an efficient movement to relocate a recently lost pheromone plume. Accordingly, such reactive strategies offer an interesting alternative to complex cognitive searching.
Collapse
Affiliation(s)
- Nicole Voges
- CNRS, LORIA, UMR 7503, Vandoeuvre-les-Nancy, France
- * E-mail:
| | | | - Philippe Lucas
- INRA, UMR 1392, Institute of Ecology and Environmental Sciences of Paris, Versailles, France
| | | |
Collapse
|
8
|
Martinez D, Arhidi L, Demondion E, Masson JB, Lucas P. Using insect electroantennogram sensors on autonomous robots for olfactory searches. J Vis Exp 2014:e51704. [PMID: 25145980 PMCID: PMC4692349 DOI: 10.3791/51704] [Citation(s) in RCA: 21] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/03/2022] Open
Abstract
Robots designed to track chemical leaks in hazardous industrial facilities or explosive traces in landmine fields face the same problem as insects foraging for food or searching for mates: the olfactory search is constrained by the physics of turbulent transport. The concentration landscape of wind borne odors is discontinuous and consists of sporadically located patches. A pre-requisite to olfactory search is that intermittent odor patches are detected. Because of its high speed and sensitivity, the olfactory organ of insects provides a unique opportunity for detection. Insect antennae have been used in the past to detect not only sex pheromones but also chemicals that are relevant to humans, e.g., volatile compounds emanating from cancer cells or toxic and illicit substances. We describe here a protocol for using insect antennae on autonomous robots and present a proof of concept for tracking odor plumes to their source. The global response of olfactory neurons is recorded in situ in the form of electroantennograms (EAGs). Our experimental design, based on a whole insect preparation, allows stable recordings within a working day. In comparison, EAGs on excised antennae have a lifetime of 2 hr. A custom hardware/software interface was developed between the EAG electrodes and a robot. The measurement system resolves individual odor patches up to 10 Hz, which exceeds the time scale of artificial chemical sensors. The efficiency of EAG sensors for olfactory searches is further demonstrated in driving the robot toward a source of pheromone. By using identical olfactory stimuli and sensors as in real animals, our robotic platform provides a direct means for testing biological hypotheses about olfactory coding and search strategies. It may also prove beneficial for detecting other odorants of interests by combining EAGs from different insect species in a bioelectronic nose configuration or using nanostructured gas sensors that mimic insect antennae.
Collapse
Affiliation(s)
- Dominique Martinez
- UMR 7503, Laboratoire Lorrain de Recherche en Informatique et ses Applications (LORIA), Centre National de la Recherche Scientifique (CNRS);
| | - Lotfi Arhidi
- UMR 7503, Laboratoire Lorrain de Recherche en Informatique et ses Applications (LORIA), Centre National de la Recherche Scientifique (CNRS)
| | - Elodie Demondion
- UMR 1392 iEES-Paris, Institut d'Ecologie et des Sciences de l'Environnement de Paris
| | | | - Philippe Lucas
- UMR 1392 iEES-Paris, Institut d'Ecologie et des Sciences de l'Environnement de Paris
| |
Collapse
|
9
|
Rhodes PA, Anderson TO. Evolving a neural olfactorimotor system in virtual and real olfactory environments. FRONTIERS IN NEUROENGINEERING 2012; 5:22. [PMID: 23112772 PMCID: PMC3482690 DOI: 10.3389/fneng.2012.00022] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/05/2011] [Accepted: 09/04/2012] [Indexed: 11/25/2022]
Abstract
To provide a platform to enable the study of simulated olfactory circuitry in context, we have integrated a simulated neural olfactorimotor system with a virtual world which simulates both computational fluid dynamics as well as a robotic agent capable of exploring the simulated plumes. A number of the elements which we developed for this purpose have not, to our knowledge, been previously assembled into an integrated system, including: control of a simulated agent by a neural olfactorimotor system; continuous interaction between the simulated robot and the virtual plume; the inclusion of multiple distinct odorant plumes and background odor; the systematic use of artificial evolution driven by olfactorimotor performance (e.g., time to locate a plume source) to specify parameter values; the incorporation of the realities of an imperfect physical robot using a hybrid model where a physical robot encounters a simulated plume. We close by describing ongoing work toward engineering a high dimensional, reversible, low power electronic olfactory sensor which will allow olfactorimotor neural circuitry evolved in the virtual world to control an autonomous olfactory robot in the physical world. The platform described here is intended to better test theories of olfactory circuit function, as well as provide robust odor source localization in realistic environments.
Collapse
|
10
|
Hernandez Bennetts V, Lilienthal AJ, Neumann PP, Trincavelli M. Mobile robots for localizing gas emission sources on landfill sites: is bio-inspiration the way to go? FRONTIERS IN NEUROENGINEERING 2012; 4:20. [PMID: 22319493 PMCID: PMC3268183 DOI: 10.3389/fneng.2011.00020] [Citation(s) in RCA: 53] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/05/2011] [Accepted: 12/17/2011] [Indexed: 12/04/2022]
Abstract
Roboticists often take inspiration from animals for designing sensors, actuators, or algorithms that control the behavior of robots. Bio-inspiration is motivated with the uncanny ability of animals to solve complex tasks like recognizing and manipulating objects, walking on uneven terrains, or navigating to the source of an odor plume. In particular the task of tracking an odor plume up to its source has nearly exclusively been addressed using biologically inspired algorithms and robots have been developed, for example, to mimic the behavior of moths, dung beetles, or lobsters. In this paper we argue that biomimetic approaches to gas source localization are of limited use, primarily because animals differ fundamentally in their sensing and actuation capabilities from state-of-the-art gas-sensitive mobile robots. To support our claim, we compare actuation and chemical sensing available to mobile robots to the corresponding capabilities of moths. We further characterize airflow and chemosensor measurements obtained with three different robot platforms (two wheeled robots and one flying micro-drone) in four prototypical environments and show that the assumption of a constant and unidirectional airflow, which is the basis of many gas source localization approaches, is usually far from being valid. This analysis should help to identify how underlying principles, which govern the gas source tracking behavior of animals, can be usefully “translated” into gas source localization approaches that fully take into account the capabilities of mobile robots. We also describe the requirements for a reference application, monitoring of gas emissions at landfill sites with mobile robots, and discuss an engineered gas source localization approach based on statistics as an alternative to biologically inspired algorithms.
Collapse
Affiliation(s)
- Victor Hernandez Bennetts
- Center for Applied Autonomous Sensor Systems, School of Science and Technology, Örebro University Örebro, Sweden
| | | | | | | |
Collapse
|
11
|
Schmuker M, Yamagata N, Nawrot MP, Menzel R. Parallel representation of stimulus identity and intensity in a dual pathway model inspired by the olfactory system of the honeybee. FRONTIERS IN NEUROENGINEERING 2011; 4:17. [PMID: 22232601 PMCID: PMC3246696 DOI: 10.3389/fneng.2011.00017] [Citation(s) in RCA: 32] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 09/05/2011] [Accepted: 12/01/2011] [Indexed: 11/13/2022]
Abstract
The honeybee Apis mellifera has a remarkable ability to detect and locate food sources during foraging, and to associate odor cues with food rewards. In the honeybee's olfactory system, sensory input is first processed in the antennal lobe (AL) network. Uniglomerular projection neurons (PNs) convey the sensory code from the AL to higher brain regions via two parallel but anatomically distinct pathways, the lateral and the medial antenno-cerebral tract (l- and m-ACT). Neurons innervating either tract show characteristic differences in odor selectivity, concentration dependence, and representation of mixtures. It is still unknown how this differential stimulus representation is achieved within the AL network. In this contribution, we use a computational network model to demonstrate that the experimentally observed features of odor coding in PNs can be reproduced by varying lateral inhibition and gain control in an otherwise unchanged AL network. We show that odor coding in the l-ACT supports detection and accurate identification of weak odor traces at the expense of concentration sensitivity, while odor coding in the m-ACT provides the basis for the computation and following of concentration gradients but provides weaker discrimination power. Both coding strategies are mutually exclusive, which creates a tradeoff between detection accuracy and sensitivity. The development of two parallel systems may thus reflect an evolutionary solution to this problem that enables honeybees to achieve both tasks during bee foraging in their natural environment, and which could inspire the development of artificial chemosensory devices for odor-guided navigation in robots.
Collapse
Affiliation(s)
- Michael Schmuker
- Neuroinformatics and Theoretical Neuroscience, Institute of Biology, Freie Universität Berlin Berlin, Germany
| | | | | | | |
Collapse
|