Multi-Robot Path Planning Method Using Reinforcement Learning

A Local-and-Global Attention Reinforcement Learning Algorithm for Multiagent Cooperative Navigation

The cooperative navigation algorithm is the crucial technology for multirobot systems to accomplish autonomous collaborative operations, and it is still a challenge for researchers. In this work, we propose a new multiagent reinforcement learning algorithm called multiagent local-and-global attention actor-critic (MLGA2C) for multiagent cooperative navigation. Inspired by the attention mechanism, we design the local-and-global attention module to dynamically extract and encode critical environmental features. Meanwhile, based on the centralized training and decentralized execution (CTDE) paradigm, we extend a new actor-critic method to handle feature encoding and make navigation decisions. We also evaluate the proposed algorithm in two cooperative navigation scenarios: static target navigation and dynamic pedestrian target tracking. The multiple experimental results show that our algorithm performs well in cooperative navigation tasks with increasing agents.

Dynamic 3D Point-Cloud-Driven Autonomous Hierarchical Path Planning for Quadruped Robots

Aiming at effectively generating safe and reliable motion paths for quadruped robots, a hierarchical path planning approach driven by dynamic 3D point clouds is proposed in this article. The developed path planning model is essentially constituted of two layers: a global path planning layer, and a local path planning layer. At the global path planning layer, a new method is proposed for calculating the terrain potential field based on point cloud height segmentation. Variable step size is employed to improve the path smoothness. At the local path planning layer, a real-time prediction method for potential collision areas and a strategy for temporary target point selection are developed. Quadruped robot experiments were carried out in an outdoor complex environment. The experimental results verified that, for global path planning, the smoothness of the path is improved and the complexity of the passing ground is reduced. The effective step size is increased by a maximum of 13.4 times, and the number of iterations is decreased by up to 1/6, compared with the traditional fixed step size planning algorithm. For local path planning, the path length is shortened by 20%, and more efficient dynamic obstacle avoidance and more stable velocity planning are achieved by using the improved dynamic window approach (DWA).

Enhancing Stability and Performance in Mobile Robot Path Planning with PMR-Dueling DQN Algorithm

Path planning for mobile robots in complex circumstances is still a challenging issue. This work introduces an improved deep reinforcement learning strategy for robot navigation that combines dueling architecture, Prioritized Experience Replay, and shaped Rewards. In a grid world and two Gazebo simulation environments with static and dynamic obstacles, the Dueling Deep Q-Network with Modified Rewards and Prioritized Experience Replay (PMR-Dueling DQN) algorithm is compared against Q-learning, DQN, and DDQN in terms of path optimality, collision avoidance, and learning speed. To encourage the best routes, the shaped Reward function takes into account target direction, obstacle avoidance, and distance. Prioritized replay concentrates training on important events while a dueling architecture separates value and advantage learning. The results show that the PMR-Dueling DQN has greatly increased convergence speed, stability, and overall performance across conditions. In both grid world and Gazebo environments the PMR-Dueling DQN achieved higher cumulative rewards. The combination of deep reinforcement learning with reward design, network architecture, and experience replay enables the PMR-Dueling DQN to surpass traditional approaches for robot path planning in complex environments.

Research on Time Series-Based Pipeline Ground Penetrating Radar Calibration Angle Prediction Algorithm

The pipeline ground-penetrating radar stands as an indispensable detection device for ensuring underground space security. A wheeled pipeline robot is deployed to traverse the interior of urban underground drainage pipelines along their central axis. It is subject to influences such as resistance, speed, and human factors, leading to deviations in its posture. A guiding wheel is employed to rectify its roll angle and ensure the precise spatial positioning of defects both inside and outside the pipeline, as detected by the radar antenna. By analyzing its deflection factors and correction trajectories, the intelligent correction control of the pipeline ground-penetrating radar falls into the realm of nonlinear multi-constraint optimization. Consequently, a time-series-based correction angle prediction algorithm is proposed. The application of the long short-term memory (LSTM) deep learning model facilitates the prediction of correction angles and torque for the guiding wheel. This study compares the performance of LSTM with an autoregressive integrated moving average model under identical dataset conditions. The subsequent findings reveal a reduction of 4.11° and 8.25 N·m in mean absolute error, and a decrease of 10.66% and 7.27% in mean squared error for the predicted correction angles and torques, respectively. These outcomes are achieved utilizing the three-channel drainage pipeline ground-penetrating radar device with top antenna operating at 1.2 GHz and left/right antennas at 750 MHz. The LSTM prediction model intelligently corrects its posture. Experimental results demonstrate an average correction speed of 5 s and an average angular error of ±1°. It is verified that the model can correct its attitude in real-time with small errors, thereby enhancing the accuracy of ground-penetrating radar antennas in locating pipeline defects.

A multi-robot deep Q-learning framework for priority-based sanitization of railway stations

Sanitizing railway stations is a relevant issue, primarily due to the recent evolution of the Covid-19 pandemic. In this work, we propose a multi-robot approach to sanitize railway stations based on a distributed Deep Q-Learning technique. The proposed framework relies on anonymous data from existing WiFi networks to dynamically estimate crowded areas within the station and to develop a heatmap of prioritized areas to be sanitized. Such heatmap is then provided to a team of cleaning robots - each endowed with a robot-specific convolutional neural network - that learn how to effectively cooperate and sanitize the station's areas according to the associated priorities. The proposed approach is evaluated in a realistic simulation scenario provided by the Italian largest railways station: Roma Termini. In this setting, we consider different case studies to assess how the approach scales with the number of robots and how the trained system performs with a real dataset retrieved from a one-day data recording of the station's WiFi network.

Multi-Agent Deep Reinforcement Learning for Multi-Robot Applications: A Survey

Deep reinforcement learning has produced many success stories in recent years. Some example fields in which these successes have taken place include mathematics, games, health care, and robotics. In this paper, we are especially interested in multi-agent deep reinforcement learning, where multiple agents present in the environment not only learn from their own experiences but also from each other and its applications in multi-robot systems. In many real-world scenarios, one robot might not be enough to complete the given task on its own, and, therefore, we might need to deploy multiple robots who work together towards a common global objective of finishing the task. Although multi-agent deep reinforcement learning and its applications in multi-robot systems are of tremendous significance from theoretical and applied standpoints, the latest survey in this domain dates to 2004 albeit for traditional learning applications as deep reinforcement learning was not invented. We classify the reviewed papers in our survey primarily based on their multi-robot applications. Our survey also discusses a few challenges that the current research in this domain faces and provides a potential list of future applications involving multi-robot systems that can benefit from advances in multi-agent deep reinforcement learning.

Dynamic Analysis and Path Planning of a Turtle-Inspired Amphibious Spherical Robot

A dynamic path-planning algorithm based on a general constrained optimization problem (GCOP) model and a sequential quadratic programming (SQP) method with sensor input is proposed in this paper. In an unknown underwater space, the turtle-inspired amphibious spherical robot (ASR) can realise the path-planning control movement and achieve collision avoidance. Due to the special underwater environments, thrusters and diamond parallel legs (DPLs) are installed in the lower hemisphere to realise accurate motion control. A propulsion model for a novel water-jet thruster based on experimental analysis and a modified Denavit-Hartenberg (MDH) algorithm are developed for multiple degrees of freedom (MDOF) to realize high-precision and high-speed motion control. Simulations and experiments verify that the effectiveness of the GCOP and SQP algorithms can realize reasonable path planning and make it possible to improve the flexibility of underwater movement with a small estimation error.

A Framework and Algorithm for Human-Robot Collaboration Based on Multimodal Reinforcement Learning

Despite the emergence of various human-robot collaboration frameworks, most are not sufficiently flexible to adapt to users with different habits. In this article, a Multimodal Reinforcement Learning Human-Robot Collaboration (MRLC) framework is proposed. It integrates reinforcement learning into human-robot collaboration and continuously adapts to the user's habits in the process of collaboration with the user to achieve the effect of human-robot cointegration. With the user's multimodal features as states, the MRLC framework collects the user's speech through natural language processing and employs it to determine the reward of the actions made by the robot. Our experiments demonstrate that the MRLC framework can adapt to the user's habits after repeated learning and better understand the user's intention compared to traditional solutions.

Controlling Fleets of Autonomous Mobile Robots with Reinforcement Learning: A Brief Survey

Controlling a fleet of autonomous mobile robots (AMR) is a complex problem of optimization. Many approached have been conducted for solving this problem. They range from heuristics, which usually do not find an optimum, to mathematical models, which are limited due to their high computational effort. Machine Learning (ML) methods offer another potential trajectory for solving such complex problems. The focus of this brief survey is on Reinforcement Learning (RL) as a particular type of ML. Due to the reward-based optimization, RL offers a good basis for the control of fleets of AMR. In the context of this survey, different control approaches are investigated and the aspects of fleet control of AMR with respect to RL are evaluated. As a result, six fundamental key problems should be put on the current research agenda to enable a broader application in industry: (1) overcoming the “sim-to-real gap”, (2) increasing the robustness of algorithms, (3) improving data efficiency, (4) integrating different fields of application, (5) enabling heterogeneous fleets with different types of AMR and (6) handling of deadlocks.

Distributed Multi-Mobile Robot Path Planning and Obstacle Avoidance Based on ACO–DWA in Unknown Complex Terrain

Multi-robot systems are popularly distributed in logistics, transportation, and other fields. We propose a distributed multi-mobile robot obstacle-avoidance algorithm to coordinate the path planning and motion navigation among multiple robots and between robots and unknown territories. This algorithm fuses the ant colony optimization (ACO) and the dynamic window approach (DWA) to coordinate a multi-robot system through a priority strategy. Firstly, to ensure the optimality of robot motion in complex terrains, we proposed the dual-population heuristic functions and a sort ant pheromone update strategy to enhance the search capability of ACO, and the globally optimal path is achieved by a redundant point deletion strategy. Considering the robot’s path-tracking accuracy and local target unreachability problems, an adaptive navigation strategy is presented. Furthermore, we propose the obstacle density evaluation function to improve the robot’s decision-making difficulty in high-density obstacle environments and modify the evaluation function coefficients adaptively by combining environmental characteristics. Finally, the robots’ motion conflict is resolved by combining our obstacle avoidance and multi-robot priority strategies. The experimental results show that this algorithm can realize the cooperative obstacle avoidance of AGVs in unknown environments with high safety and global optimality, which can provide a technical reference for distributed multi-robot in practical applications.

Reinforcement Learning-Based Algorithm to Avoid Obstacles by the Anthropomorphic Robotic Arm

In this paper, the application of the policy gradient Reinforcement Learning-based (RL) method for obstacle avoidance is proposed. This method was successfully used to control the movements of a robot using trial-and-error interactions with its environment. In this paper, an approach based on a Deep Deterministic Policy Gradient (DDPG) algorithm combined with a Hindsight Experience Replay (HER) algorithm for avoiding obstacles has been investigated. In order to ensure that the robot avoids obstacles and reaches the desired position as quickly and as accurately as possible, a special approach to the training and architecture of two RL agents working simultaneously was proposed. The implementation of this RL-based approach was first implemented in a simulation environment, which was used to control the 6-axis robot simulation model. Then, the same algorithm was used to control a real 6-DOF (degrees of freedom) robot. The results obtained in the simulation were compared with results obtained in laboratory conditions.

The Intelligent Path Planning System of Agricultural Robot via Reinforcement Learning

Agricultural robots are one of the important means to promote agricultural modernization and improve agricultural efficiency. With the development of artificial intelligence technology and the maturity of Internet of Things (IoT) technology, people put forward higher requirements for the intelligence of robots. Agricultural robots must have intelligent control functions in agricultural scenarios and be able to autonomously decide paths to complete agricultural tasks. In response to this requirement, this paper proposes a Residual-like Soft Actor Critic (R-SAC) algorithm for agricultural scenarios to realize safe obstacle avoidance and intelligent path planning of robots. In addition, in order to alleviate the time-consuming problem of exploration process of reinforcement learning, this paper proposes an offline expert experience pre-training method, which improves the training efficiency of reinforcement learning. Moreover, this paper optimizes the reward mechanism of the algorithm by using multi-step TD-error, which solves the probable dilemma during training. Experiments verify that our proposed method has stable performance in both static and dynamic obstacle environments, and is superior to other reinforcement learning algorithms. It is a stable and efficient path planning method and has visible application potential in agricultural robots.

A Path-Planning Approach Based on Potential and Dynamic Q-Learning for Mobile Robots in Unknown Environment

The path-planning approach plays an important role in determining how long the mobile robots can travel. To solve the path-planning problem of mobile robots in an unknown environment, a potential and dynamic Q-learning (PDQL) approach is proposed, which combines Q-learning with the artificial potential field and dynamic reward function to generate a feasible path. The proposed algorithm has a significant improvement in computing time and convergence speed compared to its classical counterpart. Experiments undertaken on simulated maps confirm that the PDQL when used for the path-planning problem of mobile robots in an unknown environment outperforms the state-of-the-art algorithms with respect to two metrics: path length and turning angle. The simulation results show the effectiveness and practicality of the proposal for mobile robot path planning.

Neural Tracking Control of a Four-Wheeled Mobile Robot with Mecanum Wheels

This study designed an algorithm for the intelligent control of the motion of a mobile robot with mecanum wheels. After reviewing the model kinematics and dynamics of the robot, we conducted a synthesis of the neural control algorithm to determine network weight adaptation, according to Lyapunov stability theory. Using a MATLAB/Simulink computing environment, we developed a numerical simulation for the implementation of the robot’s motion path with parametric disturbances acting on the control object. To determine the quality of the implementation of the desired motion path, a numerical test of the robot’s motion, controlled with the use of a PD controller, was conducted. The proposed control algorithm was verified on a laboratory stand equipped with a dSpace DS1103 controller board and a Husarion Panther four-wheeled mobile robot with mecanum wheels. The conducted research confirmed the improved implementation of the desired motion path by a robot controlled with the use of an intelligent control system.

Counterfactual-Based Action Evaluation Algorithm in Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (MARL) algorithms have made great achievements in various scenarios, but there are still many problems in solving sequential social dilemmas (SSDs). In SSDs, the agent’s actions not only change the instantaneous state of the environment but also affect the latent state which will, in turn, affect all agents. However, most of the current reinforcement learning algorithms focus on analyzing the value of instantaneous environment state while ignoring the study of the latent state, which is very important for establishing cooperation. Therefore, we propose a novel counterfactual reasoning-based multi-agent reinforcement learning algorithm to evaluate the continuous contribution of agent actions on the latent state. We compute that using simulation reasoning and building an action evaluation network. Then through counterfactual reasoning, we can get a single agent’s influence on the environment. Using this continuous contribution as an intrinsic reward enables the agent to consider the collective, thereby promoting cooperation. We conduct experiments in the SSDs environment, and the results show that the collective reward is increased by at least 25% which demonstrates the excellent performance of our proposed algorithm compared to the state-of-the-art algorithms.

Decentralized Multi-Robot Collision Avoidance: A Systematic Review from 2015 to 2021

An exploration task can be performed by a team of mobile robots more efficiently than human counterparts. They can access and give live updates for hard-to-reach areas such as a disaster site or a sewer. However, they face some issues hindering them from optimal path planning due to the symmetrical shape of the environments. Multiple robots are expected to explore more areas in less time while solving robot localization and collision-avoidance issues. When deploying a multi-robot system, it is ensured that the hardware parts do not collide with each other or the surroundings, especially in symmetric environments. Two types of methods are used for collision avoidance: centralized and decentralized. The decentralized approach has mainly been used in recent times, as it is computationally less expensive. This article aims to conduct a systematic literature review of different collision-avoidance strategies and analyze the performance of innovative collision-avoidance techniques. Different methods such as Reinforcement Learning (RL), Model Predictive Control (MPC), Altruistic Coordination, and other approaches followed by selected studies are also discussed. A total of 17 studies are included in this review, extracted from seven databases. Two experimental designs are studied: empty/open space and confined indoor space. Our analysis observed that most of the studies focused on empty/open space scenarios and verified the proposed model only through simulation. ORCA is the primary method, against which all the state-of-the-art techniques are evaluated. This article provides a comparison between different methods used for multi-robot collision avoidance. It discusses if the methods used are focused on safety or path planning. It also sheds light on the limitations of the studies included and possible future directions.

An Experimental Safety Response Mechanism for an Autonomous Moving Robot in a Smart Manufacturing Environment Using Q-Learning Algorithm and Speech Recognition

The industrial manufacturing sector is undergoing a tremendous revolution moving from traditional production processes to intelligent techniques. Under this revolution, known as Industry 4.0 (I40), a robot is no longer static equipment but an active workforce to the factory production alongside human operators. Safety becomes crucial for humans and robots to ensure a smooth production run in such environments. The loss of operating moving robots in plant evacuation can be avoided with the adequate safety induction for them. Operators are subject to frequent safety inductions to react in emergencies, but very little is done for robots. Our research proposes an experimental safety response mechanism for a small manufacturing plant, through which an autonomous robot learns the obstacle-free trajectory to the closest safety exit in emergencies. We implement a reinforcement learning (RL) algorithm, Q-learning, to enable the path learning abilities of the robot. After obtaining the robot optimal path selection options with Q-learning, we code the outcome as a rule-based system for the safety response. We also program a speech recognition system for operators to react timeously, with a voice command, to an emergency that requires stopping all plant activities even when they are far away from the emergency stops (ESTOPs) button. An ESTOP or a voice command sent directly to the factory central controller can give the factory an emergency signal. We tested this functionality on real hardware from an S7-1200 Siemens programmable logic controller (PLC). We simulate a simple and small manufacturing environment overview to test our safety procedure. Our results show that the safety response mechanism successfully generates paths without obstacles to the closest safety exits from all the factory locations. Our research benefits any manufacturing SME intending to implement the initial and primary use of autonomous moving robots (AMR) in their factories. It also impacts manufacturing SMEs using legacy devices such as traditional PLCs by offering them intelligent strategies to incorporate current state-of-the-art technologies such as speech recognition to improve their performances. Our research empowers SMEs to adopt advanced and innovative technological concepts within their operations.

Indoor Emergency Path Planning Based on the Q-Learning Optimization Algorithm

The internal structure of buildings is becoming increasingly complex. Providing a scientific and reasonable evacuation route for trapped persons in a complex indoor environment is important for reducing casualties and property losses. In emergency and disaster relief environments, indoor path planning has great uncertainty and higher safety requirements. Q-learning is a value-based reinforcement learning algorithm that can complete path planning tasks through autonomous learning without establishing mathematical models and environmental maps. Therefore, we propose an indoor emergency path planning method based on the Q-learning optimization algorithm. First, a grid environment model is established. The discount rate of the exploration factor is used to optimize the Q-learning algorithm, and the exploration factor in the ε-greedy strategy is dynamically adjusted before selecting random actions to accelerate the convergence of the Q-learning algorithm in a large-scale grid environment. An indoor emergency path planning experiment based on the Q-learning optimization algorithm was carried out using simulated data and real indoor environment data. The proposed Q-learning optimization algorithm basically converges after 500 iterative learning rounds, which is nearly 2000 rounds higher than the convergence rate of the Q-learning algorithm. The SASRA algorithm has no obvious convergence trend in 5000 iterations of learning. The results show that the proposed Q-learning optimization algorithm is superior to the SARSA algorithm and the classic Q-learning algorithm in terms of solving time and convergence speed when planning the shortest path in a grid environment. The convergence speed of the proposed Q- learning optimization algorithm is approximately five times faster than that of the classic Q- learning algorithm. The proposed Q-learning optimization algorithm in the grid environment can successfully plan the shortest path to avoid obstacle areas in a short time.

Plugin Framework-Based Neuro-Symbolic Grounded Task Planning for Multi-Agent System

As the roles of robots continue to expand in general, there is an increasing demand for research on automated task planning for a multi-agent system that can independently execute tasks in a wide and dynamic environment. This study introduces a plugin framework in which multiple robots can be involved in task planning in a broad range of areas by combining symbolic and connectionist approaches. The symbolic approach for understanding and learning human knowledge is useful for task planning in a wide and static environment. The network-based connectionist approach has the advantage of being able to respond to an ever-changing dynamic environment. A planning domain definition language-based planning algorithm, which is a symbolic approach, and the cooperative-competitive reinforcement learning algorithm, which is a connectionist approach, were utilized in this study. The proposed architecture is verified through a simulation. It is also verified through an experiment using 10 unmanned surface vehicles that the given tasks were successfully executed in a wide and dynamic environment.

An Improved Dueling Deep Double-Q Network Based on Prioritized Experience Replay for Path Planning of Unmanned Surface Vehicles

Unmanned Surface Vehicle (USV) has a broad application prospect and autonomous path planning as its crucial technology has developed into a hot research direction in the field of USV research. This paper proposes an Improved Dueling Deep Double-Q Network Based on Prioritized Experience Replay (IPD3QN) to address the slow and unstable convergence of traditional Deep Q Network (DQN) algorithms in autonomous path planning of USV. Firstly, we use the deep double Q-Network to decouple the selection and calculation of the target Q value action to eliminate overestimation. The prioritized experience replay method is adopted to extract experience samples from the experience replay unit, increase the utilization rate of actual samples, and accelerate the training speed of the neural network. Then, the neural network is optimized by introducing a dueling network structure. Finally, the soft update method is used to improve the stability of the algorithm, and the dynamic ϵ-greedy method is used to find the optimal strategy. The experiments are first conducted in the Open AI Gym test platform to pre-validate the algorithm for two classical control problems: the Cart pole and Mountain Car problems. The impact of algorithm hyperparameters on the model performance is analyzed in detail. The algorithm is then validated in the Maze environment. The comparative analysis of simulation experiments shows that IPD3QN has a significant improvement in learning performance regarding convergence speed and convergence stability compared with DQN, D3QN, PD2QN, PDQN, PD3QN. Also, USV can plan the optimal path according to the actual navigation environment with the IPD3QN algorithm.

Methods of Condition Monitoring and Fault Detection for Electrical Machines

Nowadays, electrical machines and drive systems are playing an essential role in different applications. Eventually, various failures occur in long-term continuous operation. Due to the increased influence of such devices on industry, industrial branches, as well as ordinary human life, condition monitoring and timely fault diagnostics have gained a reasonable importance. In this review article, there are studied different diagnostic techniques that can be used for algorithms’ training and realization of predictive maintenance. Benefits and drawbacks of intelligent diagnostic techniques are highlighted. The most widespread faults of electrical machines are discussed as well as techniques for parameters’ monitoring are introduced.

Reinforcement-Learning-Based Route Generation for Heavy-Traffic Autonomous Mobile Robot Systems

Autonomous mobile robots (AMRs) are increasingly used in modern intralogistics systems as complexity and performance requirements become more stringent. One way to increase performance is to improve the operation and cooperation of multiple robots in their shared environment. The paper addresses these problems with a method for off-line route planning and on-line route execution. In the proposed approach, pre-computation of routes for frequent pick-up and drop-off locations limits the movements of AMRs to avoid conflict situations between them. The paper proposes a reinforcement learning approach where an agent builds the routes on a given layout while being rewarded according to different criteria based on the desired characteristics of the system. The results show that the proposed approach performs better in terms of throughput and reliability than the commonly used shortest-path-based approach for a large number of AMRs operating in the system. The use of the proposed approach is recommended when the need for high throughput requires the operation of a relatively large number of AMRs in relation to the size of the space in which the robots operate.

Research on Motion Planning Based on Flocking Control and Reinforcement Learning for Multi-Robot Systems

Robots have poor adaptive ability in terms of formation control and obstacle avoidance control in unknown complex environments. To address this problem, in this paper, we propose a new motion planning method based on flocking control and reinforcement learning. It uses flocking control to implement a multi-robot orderly motion. To avoid the trap of potential fields faced during flocking control, the flocking control is optimized, and the strategy of wall-following behavior control is designed. In this paper, reinforcement learning is adopted to implement the robotic behavioral decision and to enhance the analytical and predictive abilities of the robot during motion planning in an unknown environment. A visual simulation platform is developed in this paper, on which researchers can test algorithms for multi-robot motion control, such as obstacle avoidance control, formation control, path planning and reinforcement learning strategy. As shown by the simulation experiments, the motion planning method presented in this paper can enhance the abilities of multi-robot systems to self-learn and self-adapt under a fully unknown environment with complex obstacles.

Multi AGV Coordination Tolerant to Communication Failures

Most path planning algorithms used presently in multi-robot systems are based on offline planning. The Timed Enhanced A* (TEA*) algorithm gives the possibility of planning in real time, rather than planning in advance, by using a temporal estimation of the robot’s positions at any given time. In this article, the implementation of a control system for multi-robot applications that operate in environments where communication faults can occur and where entire sections of the environment may not have any connection to the communication network will be presented. This system uses the TEA* to plan multiple robot paths and a supervision system to control communications. The supervision system supervises the communication with the robots and checks whether the robot’s movements are synchronized. The implemented system allowed the creation and execution of paths for the robots that were both safe and kept the temporal efficiency of the TEA* algorithm. Using the Simtwo2020 simulation software, capable of simulating movement dynamics and the Lazarus development environment, it was possible to simulate the execution of several different missions by the implemented system and analyze their results.

Path Planning for Multi-Arm Manipulators Using Deep Reinforcement Learning: Soft Actor-Critic with Hindsight Experience Replay

Since path planning for multi-arm manipulators is a complicated high-dimensional problem, effective and fast path generation is not easy for the arbitrarily given start and goal locations of the end effector. Especially, when it comes to deep reinforcement learning-based path planning, high-dimensionality makes it difficult for existing reinforcement learning-based methods to have efficient exploration which is crucial for successful training. The recently proposed soft actor-critic (SAC) is well known to have good exploration ability due to the use of the entropy term in the objective function. Motivated by this, in this paper, a SAC-based path planning algorithm is proposed. The hindsight experience replay (HER) is also employed for sample efficiency and configuration space augmentation is used in order to deal with complicated configuration space of the multi-arms. To show the effectiveness of the proposed algorithm, both simulation and experiment results are given. By comparing with existing results, it is demonstrated that the proposed method outperforms the existing results.

An Overview of Reinforcement Learning Methods for Variable Speed Limit Control

Variable Speed Limit (VSL) control systems are widely studied as solutions for improving safety and throughput on urban motorways. Machine learning techniques, specifically Reinforcement Learning (RL) methods, are a promising alternative for setting up VSL since they can learn and react to different traffic situations without knowing the explicit model of the motorway dynamics. However, the efficiency of combined RL-VSL is highly related to the class of the used RL algorithm, and description of the managed motorway section in which the RL-VSL agent sets the appropriate speed limits. Currently, there is no existing overview of RL algorithm applications in the domain of VSL. Therefore, a comprehensive survey on the state of the art of RL-VSL is presented. Best practices are summarized, and new viewpoints and future research directions, including an overview of current open research questions are presented.

A Fuzzy Analytic Hierarchy Process and Cooperative Game Theory Combined Multiple Mobile Robot Navigation Algorithm

This study presents a multi-robot navigation strategy based on a multi-objective decision-making algorithm, the Fuzzy Analytic Hierarchy Process (FAHP). FAHP analytically selects an optimal position as a sub-goal among points on the sensing boundary of a mobile robot considering the following three objectives: the travel distance to the target, collision safety with obstacles, and the rotation of the robot to face the target. Alternative solutions are evaluated by quantifying the relative importance of the objectives. As the FAHP algorithm is insufficient for multi-robot navigation, cooperative game theory is added to improve it. The performance of the proposed multi-robot navigation algorithm is tested with up to 12 mobile robots in several simulation conditions, altering factors such as the number of operating robots and the warehouse layout.

Special Issue on Mobile Robots Navigation

In recent years, the presence of mobile robots in diverse scenarios has considerably increased, to solve a variety of tasks [...]

Motion Planning of Robot Manipulators for a Smoother Path Using a Twin Delayed Deep Deterministic Policy Gradient with Hindsight Experience Replay

In order to enhance performance of robot systems in the manufacturing industry, it is essential to develop motion and task planning algorithms. Especially, it is important for the motion plan to be generated automatically in order to deal with various working environments. Although PRM (Probabilistic Roadmap) provides feasible paths when the starting and goal positions of a robot manipulator are given, the path might not be smooth enough, which can lead to inefficient performance of the robot system. This paper proposes a motion planning algorithm for robot manipulators using a twin delayed deep deterministic policy gradient (TD3) which is a reinforcement learning algorithm tailored to MDP with continuous action. Besides, hindsight experience replay (HER) is employed in the TD3 to enhance sample efficiency. Since path planning for a robot manipulator is an MDP (Markov Decision Process) with sparse reward and HER can deal with such a problem, this paper proposes a motion planning algorithm using TD3 with HER. The proposed algorithm is applied to 2-DOF and 3-DOF manipulators and it is shown that the designed paths are smoother and shorter than those designed by PRM.