A review of spatial reasoning and interaction for real-world robotics

Robotic world models-conceptualization, review, and engineering best practices

The term "world model" (WM) has surfaced several times in robotics, for instance, in the context of mobile manipulation, navigation and mapping, and deep reinforcement learning. Despite its frequent use, the term does not appear to have a concise definition that is consistently used across domains and research fields. In this review article, we bootstrap a terminology for WMs, describe important design dimensions found in robotic WMs, and use them to analyze the literature on WMs in robotics, which spans four decades. Throughout, we motivate the need for WMs by using principles from software engineering, including "Design for use," "Do not repeat yourself," and "Low coupling, high cohesion." Concrete design guidelines are proposed for the future development and implementation of WMs. Finally, we highlight similarities and differences between the use of the term "world model" in robotic mobile manipulation and deep reinforcement learning.

Fast Search Strategy for Robots in Dynamic Home Environment

In an unstructured home environment, environmental information is mostly disorganized. It is difficult for a service robot to obtain sufficient service information, which significantly hinders task execution. To solve this problem, a new object search strategy is proposed for improving the speed and accuracy of object search in a complex family environment. In this method, a family-environment knowledge graph is constructed using real environmental information and human knowledge, which plays a guiding role in task execution. The home environment is divided into three levels: functional rooms, static objects, and dynamic objects. The co-occurrence probabilities are obtained from open knowledge sources, including the probabilities between static and dynamic objects and between static objects and functional rooms. They are combined with ontology knowledge based on the home to form prior knowledge of a service robot. Inspired by the human search process, a distance function is introduced to calculate the distance between the robot and target objects for optimizing the search strategy. To improve the robustness of robotic services, we designed a probabilistic update model based on the service tasks and knowledge databases. Experimental results indicated that the proposed search strategy can significantly shorten the search time and increase the search accuracy compared with methods without prior knowledge and the distance function.

Everyday Activities

The ease with which humans usually perform everyday activities masks their inherit complexity. Tasks such as setting a table prior to a meal or preparing a hot beverage require the coordination of several cognitive abilities. At the same time, many everyday activities are simple enough to afford investigation in controlled lab settings. One main goal of this issue is to raise awareness of everyday activities as a topic and a field of study in its own right, which allows investigating (a) selected cognitive abilities with high ecological validity and (b) the interplay and integration of key cognitive abilities. To this end, this topic consists of eight papers that span different aspects of everyday activities, ranging from neuroscience through philosophical considerations and implications to lessons from robotics.

Dynamic Semantic World Models and Increased Situational Awareness for Highly Automated Inland Waterway Transport

Automated surface vessels must integrate many tasks and motions at the same time. Moreover, vessels as well as monitoring and control services need to react to physical disturbances, to dynamically allocate software resources available within a particular environment, and to communicate with various other actors in particular navigation and traffic situations. In this work, the responsibility for the situational awareness is given to a mediator that decides how: 1) to assess the impact of the actual physical environment on the quality and performance of the ongoing task executions; 2) to make sure these tasks satisfy the system requirements; and 3) to be robust against disturbances. This paper proposes a set of semantic world models within the context of inland waterway transport, and discusses policies and methodologies to compose, use, and connect these models. Model-conform entities and relations are composed dynamically, that is, corresponding to the opportunities and challenges offered by the actual situation. The semantic world models discussed in this work are divided into two main categories: 1) the semantic description of a vessel’s own properties and relationships, called the internal world model, or body model, and 2) the semantic description of its local environment, called the external world model, or map. A range of experiments illustrate the potential of using such models to decide the reactions of the application at runtime. Furthermore, three dynamic, context-dependent, ship domains are integrated in the map as two-dimensional geometric entities around a moving vessel to increase the situational awareness of automated vessels. Their geometric representations depend on the associated relations; for example, with: 1) the motion of the vessel, 2) the actual, desired, or hypothesised tasks, 3) perception sensor information, and 4) other geometries, e.g., features from the Inland Electronic Navigational Charts. The ability to unambiguously understand the environmental context, as well as the motion or position of surrounding entities, allows for resource-efficient and straightforward control decisions. The semantic world models facilitate knowledge sharing between actors, and significantly enhance explainability of the actors’ behaviour and control decisions.

Social and Robust Navigation for Indoor Robots Based on Object Semantic Grid and Topological Map

For the indoor navigation of service robots, human–robot interaction and adapting to the environment still need to be strengthened, including determining the navigation goal socially, improving the success rate of passing doors, and optimizing the path planning efficiency. This paper proposes an indoor navigation system based on object semantic grid and topological map, to optimize the above problems. First, natural language is used as a human–robot interaction form, from which the target room, object, and spatial relationship can be extracted by using speech recognition and word segmentation. Then, the robot selects the goal point from the target space by object affordance theory. To improve the navigation success rate and safety, we generate auxiliary navigation points on both sides of the door to correct the robot trajectory. Furthermore, based on the topological map and auxiliary navigation points, the global path is segmented into each topological area. The path planning algorithm is carried on respectively in every room, which significantly improves the navigation efficiency. This system has demonstrated to support autonomous navigation based on language interaction and significantly improve the safety, efficiency, and robustness of indoor robot navigation. Our system has been successfully tested in real domestic environments.

Object Semantic Grid Mapping with 2D LiDAR and RGB-D Camera for Domestic Robot Navigation

Occupied grid maps are sufficient for mobile robots to complete metric navigation tasks in domestic environments. However, they lack semantic information to endow the robots with the ability of social goal selection and human-friendly operation modes. In this paper, we propose an object semantic grid mapping system with 2D Light Detection and Ranging (LiDAR) and RGB-D sensors to solve this problem. At first, we use a laser-based Simultaneous Localization and Mapping (SLAM) to generate an occupied grid map and obtain a robot trajectory. Then, we employ object detection to get an object’s semantics of color images and use joint interpolation to refine camera poses. Based on object detection, depth images, and interpolated poses, we build a point cloud with object instances. To generate object-oriented minimum bounding rectangles, we propose a method for extracting the dominant directions of the room. Furthermore, we build object goal spaces to help the robots select navigation goals conveniently and socially. We have used the Robot@Home dataset to verify the system; the verification results show that our system is effective.

Improved and scalable online learning of spatial concepts and language models with mapping

We propose a novel online learning algorithm, called SpCoSLAM 2.0, for spatial concepts and lexical acquisition with high accuracy and scalability. Previously, we proposed SpCoSLAM as an online learning algorithm based on unsupervised Bayesian probabilistic model that integrates multimodal place categorization, lexical acquisition, and SLAM. However, our original algorithm had limited estimation accuracy owing to the influence of the early stages of learning, and increased computational complexity with added training data. Therefore, we introduce techniques such as fixed-lag rejuvenation to reduce the calculation time while maintaining an accuracy higher than that of the original algorithm. The results show that, in terms of estimation accuracy, the proposed algorithm exceeds the original algorithm and is comparable to batch learning. In addition, the calculation time of the proposed algorithm does not depend on the amount of training data and becomes constant for each step of the scalable algorithm. Our approach will contribute to the realization of long-term spatial language interactions between humans and robots.

Building semantic grid maps for domestic robot navigation

This article proposes a semantic grid mapping method for domestic robot navigation. Occupancy grid maps are sufficient for mobile robots to complete point-to-point navigation tasks in 2-D small-scale environments. However, when used in the real domestic scene, grid maps are lack of semantic information for end users to specify navigation tasks conveniently. Semantic grid maps, enhancing the occupancy grid map with the semantics of objects and rooms, endowing the robots with the capacity of robust navigation skills and human-friendly operation modes, are thus proposed to overcome this limitation. In our method, an object semantic grid map is built with low-cost sonar and binocular stereovision sensors by correctly fusing the occupancy grid map and object point clouds. Topological spaces of each object are defined to make robots autonomously select navigation destinations. Based on the domestic common sense of the relationship between rooms and objects, topological segmentation is used to get room semantics. Our method is evaluated in a real homelike environment, and the results show that the generated map is at a satisfactory precision and feasible for a domestic mobile robot to complete navigation tasks commanded in natural language with a high success rate.

A Robotic Context Query-Processing Framework Based on Spatio-Temporal Context Ontology

Service robots operating in indoor environments should recognize dynamic changes from sensors, such as RGB-depth (RGB-D) cameras, and recall the past context. Therefore, we propose a context query-processing framework, comprising spatio-temporal robotic context query language (ST-RCQL) and a spatio-temporal robotic context query-processing system (ST-RCQP), for service robots. We designed them based on spatio-temporal context ontology. ST-RCQL can query not only the current context knowledge, but also the past. In addition, ST-RCQL includes a variety of time operators and time constants; thus, queries can be written very efficiently. The ST-RCQP is a query-processing system equipped with a perception handler, working memory, and backward reasoner for real-time query-processing. Moreover, ST-RCQP accelerates query-processing speed by building a spatio-temporal index in the working memory, where percepts are stored. Through various qualitative and quantitative experiments, we demonstrate the high efficiency and performance of the proposed context query-processing framework.