Exploring a GPT-based large language model for variable autonomy in a VR-based human-robot teaming simulation

In a rapidly evolving digital landscape autonomous tools and robots are becoming commonplace. Recognizing the significance of this development, this paper explores the integration of Large Language Models (LLMs) like Generative pre-trained transformer (GPT) into human-robot teaming environments to facilitate variable autonomy through the means of verbal human-robot communication. In this paper, we introduce a novel simulation framework for such a GPT-powered multi-robot testbed environment, based on a Unity Virtual Reality (VR) setting. This system allows users to interact with simulated robot agents through natural language, each powered by individual GPT cores. By means of OpenAI's function calling, we bridge the gap between unstructured natural language input and structured robot actions. A user study with 12 participants explores the effectiveness of GPT-4 and, more importantly, user strategies when being given the opportunity to converse in natural language within a simulated multi-robot environment. Our findings suggest that users may have preconceived expectations on how to converse with robots and seldom try to explore the actual language and cognitive capabilities of their simulated robot collaborators. Still, those users who did explore were able to benefit from a much more natural flow of communication and human-like back-and-forth. We provide a set of lessons learned for future research and technical implementations of similar systems.

Path Planning Method of Mobile Robot Using Improved Deep Reinforcement Learning

A mobile robot path planning method based on improved deep reinforcement learning is proposed. First, in order to conform to the actual kinematics model of the robot, the continuous environmental state space and discrete action state space are designed. In addition, an improved deep Q-network (DQN) method is proposed, which takes the directly collected information as the training samples and combines the environmental state characteristics of the robot and the target point to be reached as the input of the network. DQN method takes the Q value at the current position as the output of the network model and uses $\epsilon$ -greedy strategy for action selection. Finally, the reward function combined with the artificial potential field method is designed to optimize the state-action space. The reward function solves the problem of sparse reward in the environmental state space and makes the action selection of the robot more accurate. Experiments show that compared with the classical DQN method, the average loss function value is reduced by 36.87% and the average reward value is increased by 12.96%, which can effectively improve the working efficiency of mobile robot.

Expression Recognition Using Improved AlexNet Network in Robot Intelligent Interactive System

Aiming at the insufficient feature extraction in the expression feature extraction stage of traditional convolutional neural network and the misclassification of mislabeled samples, an expression recognition and robot intelligent interaction method using deep learning is proposed. First, in image preprocessing, the dimension of the color image is reduced by image gray adjustment to reduce the amount of calculation, the shadow interference is eliminated by the average method, and the image is enhanced by histogram equalization. Second, multichannel convolution is used to replace the single convolution size in the second convolution layer in AlexNet, the Global Average Pooling layer is introduced to replace the fully connected layer, and Batch Normalization is introduced to improve the feature extraction ability of the model and avoid gradient explosion. Finally, the Focal Loss is improved by setting the probability threshold to avoid the impact of mislabeling samples on the classification performance of the model. The experimental results show that the recognition accuracy of the model on FER2013 data set is 98.36%. The effectiveness of the algorithm is verified on the intelligent interactive system of service robot based on expression recognition. Compared with other expression recognition methods, the proposed method can extract more expression features and recognize facial expression more accurately.

Low-Bandwidth and Compute-Bound RGB-D Planar Semantic SLAM

Visual simultaneous location and mapping (SLAM) using RGB-D cameras has been a necessary capability for intelligent mobile robots. However, when using point-cloud map representations as most RGB-D SLAM systems do, limitations in onboard compute resources, and especially communication bandwidth can significantly limit the quantity of data processed and shared. This article proposes techniques that help address these challenges by mapping point clouds to parametric models in order to reduce computation and bandwidth load on agents. This contribution is coupled with a convolutional neural network (CNN) that extracts semantic information. Semantics provide guidance in object modeling which can reduce the geometric complexity of the environment. Pairing a parametric model with a semantic label allows agents to share the knowledge of the world with much less complexity, opening a door for multi-agent systems to perform complex tasking, and human–robot cooperation. This article takes the first step towards a generalized parametric model by limiting the geometric primitives to a planar surface and providing semantic labels when appropriate. Two novel compression algorithms for depth data and a method to independently fit planes to RGB-D data are provided, so that plane data can be used for real-time odometry estimation and mapping. Additionally, we extend maps with semantic information predicted from sparse geometries (planes) by a CNN. In experiments, the advantages of our approach in terms of computational and bandwidth resources savings are demonstrated and compared with other state-of-the-art SLAM systems.