Multi-robot cooperative autonomous exploration via task allocation in terrestrial environments

Cooperative autonomous exploration is a challenging task for multi-robot systems, which can cover larger areas in a shorter time or path length. Using multiple mobile robots for cooperative exploration of unknown environments can be more efficient than a single robot, but there are also many difficulties in multi-robot cooperative autonomous exploration. The key to successful multi-robot cooperative autonomous exploration is effective coordination between the robots. This paper designs a multi-robot cooperative autonomous exploration strategy for exploration tasks. Additionally, considering the fact that mobile robots are inevitably subject to failure in harsh conditions, we propose a self-healing cooperative autonomous exploration method that can recover from robot failures.

On the role and opportunities in teamwork design for advanced multi-robot search systems

Intelligent robotic systems are becoming ever more present in our lives across a multitude of domains such as industry, transportation, agriculture, security, healthcare and even education. Such systems enable humans to focus on the interesting and sophisticated tasks while robots accomplish tasks that are either too tedious, routine or potentially dangerous for humans to do. Recent advances in perception technologies and accompanying hardware, mainly attributed to rapid advancements in the deep-learning ecosystem, enable the deployment of robotic systems equipped with onboard sensors as well as the computational power to perform autonomous reasoning and decision making online. While there has been significant progress in expanding the capabilities of single and multi-robot systems during the last decades across a multitude of domains and applications, there are still many promising areas for research that can advance the state of cooperative searching systems that employ multiple robots. In this article, several prospective avenues of research in teamwork cooperation with considerable potential for advancement of multi-robot search systems will be visited and discussed. In previous works we have shown that multi-agent search tasks can greatly benefit from intelligent cooperation between team members and can achieve performance close to the theoretical optimum. The techniques applied can be used in a variety of domains including planning against adversarial opponents, control of forest fires and coordinating search-and-rescue missions. The state-of-the-art on methods of multi-robot search across several selected domains of application is explained, highlighting the pros and cons of each method, providing an up-to-date view on the current state of the domains and their future challenges.

A Multirobot Person Search System for Finding Multiple Dynamic Users in Human-Centered Environments

Multirobot coordination for finding multiple users in an environment can be used in numerous robotic applications, including search and rescue, surveillance/monitoring, and activities of daily living assistance. Existing approaches have limited coordination between robots when generating team plans or do not consider user location probability within these plans. This results in long searches and robots potentially revisiting the same locations in succession. In this article, we present a novel multirobot person search system to generate search plans for multirobot teams to find multiple dynamic users before a deadline. Our approach is unique in that it simultaneously considers the search actions of all robots and user location probabilities when generating team plans, where user location probabilities are represented as conditional spatial-temporal probability density functions. We model this multirobot person search problem as a two-stage optimization problem to maximize the expected number of users found before the deadline. Stage 1 solves the action selection problem to determine a set of team actions, and the second stage solves the action allocation problem to distribute these actions amongst the robots. Namely, in stage 1, a novel conditional multiperiod multiknapsack problem is modeled as a min-flow graph solved sequentially by the Bellman-Ford shortest path algorithm. Stage 2 is a variant of the min-max multitraveling salesperson problem which models the environment topology as a search region network and search times selected by the previous stage. This stage is solved by a novel fuzzy clustering method. Numerous experiments comparing our proposed method to other existing approaches with varying environment sizes, search durations, and the number of users showed that our approach was able to find more target users before a defined deadline.

Robotic Vision for Human-Robot Interaction and Collaboration: A Survey and Systematic Review

Robotic vision for human-robot interaction and collaboration is a critical process for robots to collect and interpret detailed information related to human actions, goals, and preferences, enabling robots to provide more useful services to people. This survey and systematic review presents a comprehensive analysis on robotic vision in human-robot interaction and collaboration over the last 10 years. From a detailed search of 3850 articles, systematic extraction and evaluation was used to identify and explore 310 papers in depth. These papers described robots with some level of autonomy using robotic vision for locomotion, manipulation and/or visual communication to collaborate or interact with people. This paper provides an in-depth analysis of current trends, common domains, methods and procedures, technical processes, data sets and models, experimental testing, sample populations, performance metrics and future challenges. This manuscript found that robotic vision was often used in action and gesture recognition, robot movement in human spaces, object handover and collaborative actions, social communication and learning from demonstration. Few high-impact and novel techniques from the computer vision field had been translated into human-robot interaction and collaboration. Overall, notable advancements have been made on how to develop and deploy robots to assist people.

UAV Computing-Assisted Search and Rescue Mission Framework for Disaster and Harsh Environment Mitigation

Disasters are crisis circumstances that put human life in jeopardy. During disasters, public communication infrastructure is particularly damaged, obstructing Search And Rescue (SAR) efforts, and it takes significant time and effort to re-establish functioning communication infrastructure. SAR is a critical component of mitigating human and environmental risks in disasters and harsh environments. As a result, there is an urgent need to construct communication networks swiftly to help SAR efforts exchange emergency data. UAV technology has the potential to provide key solutions to mitigate such disaster situations. UAVs can be used to provide an adaptable and reliable emergency communication backbone and to resolve major issues in disasters for SAR operations. In this paper, we evaluate the network performance of UAV-assisted intelligent edge computing to expedite SAR missions and functionality, as this technology can be deployed within a short time and can help to rescue most people during a disaster. We have considered network parameters such as delay, throughput, and traffic sent and received, as well as path loss for the proposed network. It is also demonstrated that with the proposed parameter optimization, network performance improves significantly, eventually leading to far more efficient SAR missions in disasters and harsh environments.

A Review: Machine Learning for Combinatorial Optimization Problems in Energy Areas

Combinatorial optimization problems (COPs) are a class of NP-hard problems with great practical significance. Traditional approaches for COPs suffer from high computational time and reliance on expert knowledge, and machine learning (ML) methods, as powerful tools have been used to overcome these problems. In this review, the COPs in energy areas with a series of modern ML approaches, i.e., the interdisciplinary areas of COPs, ML and energy areas, are mainly investigated. Recent works on solving COPs using ML are sorted out firstly by methods which include supervised learning (SL), deep learning (DL), reinforcement learning (RL) and recently proposed game theoretic methods, and then problems where the timeline of the improvements for some fundamental COPs is the layout. Practical applications of ML methods in the energy areas, including the petroleum supply chain, steel-making, electric power system and wind power, are summarized for the first time, and challenges in this field are analyzed.

Mixed-initiative Variable Autonomy for Remotely Operated Mobile Robots

This article presents an Expert-guided Mixed-initiative Control Switcher (EMICS) for remotely operated mobile robots. The EMICS enables switching between different levels of autonomy during task execution initiated by either the human operator and/or the EMICS. The EMICS is evaluated in two disaster-response-inspired experiments, one with a simulated robot and test arena, and one with a real robot in a realistic environment. Analyses from the two experiments provide evidence that: (a) Human-Initiative (HI) systems outperform systems with single modes of operation, such as pure teleoperation, in navigation tasks; (b) in the context of the simulated robot experiment, Mixed-initiative (MI) systems provide improved performance in navigation tasks, improved operator performance in cognitive demanding secondary tasks, and improved operator workload compared to HI. Last, our experiment on a physical robot provides empirical evidence that identify two major challenges for MI control: (a) the design of context-aware MI control systems; and (b) the conflict for control between the robot’s MI control system and the operator. Insights regarding these challenges are discussed and ways to tackle them are proposed.

Multivehicle Flocking With Collision Avoidance via Distributed Model Predictive Control

Flocking control has been studied extensively along with the wide applications of multivehicle systems. In this article, the distributed flocking control strategy is studied for a network of autonomous vehicles with limited communication range. The main difference from the existing methods lies in that collision avoidance is considered a necessary condition while the vehicles are driven to follow a common desired trajectory under the proximity network. The sufficient conditions for system feasibility and stability are given by the proposed strategy. First, a centralized standard model predictive control (MPC) scheme is adopted to formulate the multivehicle flocking control problem by setting collision avoidance as an optimization constraint under the proximity network. Further, an equivalent distributed MPC (DMPC) is developed based on the consensus of local controllers under the existing framework of the alternating direction method of multiplier (ADMM). However, it may require infinite time to achieve consensus for all vehicles and, thus, the local controllers resulting in a limited number of ADMM iterations may not satisfy the given constraints. The constraints for each local controller are then modified so that the collision between vehicles is avoided all of the time. The feasibility and stability of the proposed method are analyzed under practical conditions. Simulation and experimental results show that the flocking of vehicles can track the common desired trajectory stably with no collisions by the proposed method.

Frequency Variability Feature for Life Signs Detection and Localization in Natural Disasters

The locations and breathing signal of people in disaster areas are significant information for search and rescue missions in prioritizing operations to save more lives. For detecting the living people who are lying on the ground and covered with dust, debris or ashes, a motion magnification-based method has recently been proposed. This current method estimates the locations and breathing signal of people from a drone video by assuming that only human breathing-related motions exist in the video. However, in natural disasters, background motions, such as swing trees and grass caused by wind, are mixed with human breathing, that distort this assumption, resulting in misleading or even no life signs locations. Therefore, the life signs in disaster areas are challenging to be detected due to the undesired background motions. Note that human breathing is a natural physiological phenomenon, and it is a periodic motion with a steady peak frequency; while background motion always involves complex space-time behaviors, their peak frequencies seem to be variable over time. Therefore, in this work we analyze and focus on the frequency properties of motions to model a frequency variability feature used for extracting only human breathing, while eliminating irrelevant background motions in the video, which would ease the challenge in detection and localization of life signs. The proposed method was validated with both drone and camera videos recorded in the wild. The average precision measures of our method for drone and camera videos were 0.94 and 0.92, which are higher than that of compared methods, demonstrating that our method is more robust and accurate to background motions. The implications and limitations regarding the frequency variability feature were discussed.

Detection and Localisation of Life Signs from the Air Using Image Registration and Spatio-Temporal Filtering

In search and rescue operations, it is crucial to rapidly identify those people who are alive from those who are not. If this information is known, emergency teams can prioritize their operations to save more lives. However, in some natural disasters the people may be lying on the ground covered with dust, debris, or ashes making them difficult to detect by video analysis that is tuned to human shapes. We present a novel method to estimate the locations of people from aerial video using image and signal processing designed to detect breathing movements. We have shown that this method can successfully detect clearly visible people and people who are fully occluded by debris. First, the aerial videos were stabilized using the key points of adjacent image frames. Next, the stabilized video was decomposed into tile videos and the temporal frequency bands of interest were motion magnified while the other frequencies were suppressed. Image differencing and temporal filtering were performed on each tile video to detect potential breathing signals. Finally, the detected frequencies were remapped to the image frame creating a life signs map that indicates possible human locations. The proposed method was validated with both aerial and ground recorded videos in a controlled environment. Based on the dataset, the results showed good reliability for aerial videos and no errors for ground recorded videos where the average precision measures for aerial videos and ground recorded videos were 0.913 and 1 respectively.

A Hybrid Strategy for Target Search Using Static and Mobile Sensors

Locating a mobile target, untrackable in real-time, is pertinent to numerous time-critical applications, such as wilderness search and rescue. This paper proposes a hybrid approach to this dynamic problem, where both static and mobile sensors are utilized for the goal of detecting a target. The approach is novel in that a team of robots utilized to deploy a static-sensor network also actively searches for the target via on-board sensors. Synergy is achieved through: 1) optimal deployment planning of static-sensor networks and 2) optimal routing and motion planning of the robots for the deployment of the network and target search. The static-sensor network is planned first to maximize the likelihood of target detection while ensuring (temporal and spatial) unbiasedness in target motion. Robot motions are, subsequently, planned in two stages: 1) route planning and 2) trajectory planning. In the first stage, given a static-sensor network configuration, robot routes are planned to maximize the amount of spare time available to the mobile agents/sensors, for target search in between (just-in-time) static-sensor deployments. In the second stage, given robot routes (i.e., optimal sequences of sensor delivery locations and times), the corresponding robot trajectories are planned to make effective use of any spare time the mobile agents may have to search for the target. The proposed search strategy was validated through extensive simulations, some of which are given in detail here. An analysis of the method's performance in terms of target-search success is also included.

A Sensor-Network-Supported Mobile-Agent-Search Strategy for Wilderness Rescue

Mobile target search is a problem pertinent to a variety of applications, including wilderness search and rescue. This paper proposes a hybrid approach for target search utilizing a team of mobile agents supported by a network of static sensors. The approach is novel in that the mobile agents deploy the sensors at optimized times and locations while they themselves travel along their respective optimized search trajectories. In the proposed approach, mobile-agent trajectories are first planned to maximize the likelihood of target detection. The deployment of the static-sensor network is subsequently planned. Namely, deployment locations and times are optimized while being constrained by the already planned mobile-agent trajectories. The latter optimization problem, as formulated and solved herein, aims to minimize an overall network-deployment error. This overall error comprises three main components, each quantifying a deviation from one of three main objectives the network aims to achieve: (i) maintaining directional unbiasedness in target-motion consideration, (ii) maintaining unbiasedness in temporal search-effort distribution, and, (iii) maximizing the likelihood of target detection. We solve this unique optimization problem using an iterative heuristic-based algorithm with random starts. The proposed hybrid search strategy was validated through the extensive simulations presented in this paper. Furthermore, its performance was evaluated with respect to an alternative hybrid search strategy, where it either outperformed or performed comparably depending on the search resources available.

Unscented Transformation-Based Multi-Robot Collaborative Self-Localization and Distributed Target Tracking

The problem of multi-robot collaborative self-localization and distributed target tracking in practical scenarios is studied in this work. The major challenge in solving the problem in a distributed fashion is properly dealing with inter-robot and robot–target correlations in order to realize consistent state estimates of the local robots and the target simultaneously. In this paper, an unscented transformation-based collaborative self-localization and target tracking algorithm is proposed. Inter-robot correlations are approximated in a distributed fashion, and robot–target correlations are safely discarded with a conservative covariance intersection method. Furthermore, the state update is realized in an asynchronous manner with different kinds of measurements while accounting for measurement and communication limitations. Finally, to deal with nonlinearity in the processes and measurement models, the unscented transformation approach is adopted. Unscented transformation is better able to characterize nonlinearity than the extended Kalman filter-based method and does not require computation of the Jacobian matrix. Simulations are extensively studied to show that the proposed method can realize stable state estimates of both local robots and targets, and results show that it outperforms the EKF-based method. Moreover, the effectiveness of the proposed method is verified on experimental quadrotor platforms carrying off-the-shelf onboard sensors.

Design and implementation of a millirobot for swarm studies –mROBerTO

The use of millirobots, particularly in swarm studies, would enable researchers to verify their proposed autonomous cooperative behavior algorithms under realistic conditions with a large number of agents. While multiple designs for such robots have been proposed, they, typically, require custom-made components, which make replication and manufacturing difficult, and, mostly, employ non-modular integral designs. Furthermore, these robots' proposed small sizes tend to limit sensory perception capabilities and operational time. Some have resolved few of the above issues through the use of extensions that, unfortunately, add to their size.

In contribution to the pertinent field, thus, a novel millirobot with an open-source design, addressing the above concerns, is presented in this paper. Our proposed millirobot has a modular design and uses easy to source, off-the-shelf components. Themilli-robot-Toronto (mROBerTO) also includes a variety of sensors and has a 16 × 16 mm2footprint.mROBerTO's wireless communication capabilities include ANT™, Bluetooth Smart, or both simultaneously. Data-processing is handled by an ARM processor with 256 KB of flash memory. Additionally, the sensing modules allow for extending or changing the robot's perception capabilities without adding to the robot's size. For example, the swarm-sensing module, designed to facilitate swarm studies, allows for measuring proximity and bearing to neighboring robots and performing local communications.

Extensive experiments, some of which are presented herein, have illustrated the capability ofmROBerTOunits for use in implementing a variety of commonly proposed swarm algorithms.

A survey of formation control and motion planning of multiple unmanned vehicles

The increasing deployment of multiple unmanned vehicles systems has generated large research interest in recent decades. This paper therefore provides a detailed survey to review a range of techniques related to the operation of multi-vehicle systems in different environmental domains, including land based, aerospace and marine with the specific focuses placed on formation control and cooperative motion planning. Differing from other related papers, this paper pays a special attention to the collision avoidance problem and specifically discusses and reviews those methods that adopt flexible formation shape to achieve collision avoidance for multi-vehicle systems. In the conclusions, some open research areas with suggested technologies have been proposed to facilitate the future research development.