Reinforcement Learning Approaches in Social Robotics

Parental involvement in robot-mediated intervention: a systematic review

Introduction

Over the years, the conceptual approach to pediatric rehabilitation has reevaluated the parent's role in the therapeutic process, considering parental involvement as a necessary condition for the effectiveness of the intervention. In the field of pediatric intervention, the therapeutic use of robots represents a growing clinical interest, but the feasibility and applicability of these robotic interventions, including those involving parents, remain unclear. This systematic review aims to investigate parental involvement in robot-mediated interventions (RMI) for children and adolescents in the current literature. Our main goal is to analyze and summarize all existing studies to discuss and draw future research directions and implications for clinical practice.

Method

After collecting results from 1,106 studies, the studies selected were analyzed using thematic analysis. The literature review was conducted in accordance with the PRISMA guidelines by searching databases such as PubMed and Web of Science until 07 February 2023. Studies that met the following inclusion criteria were included: (1) the use of a robot as a therapeutic-rehabilitation tool and (2) the presence/involvement of parents/caregivers in child-robot therapeutic sessions.

Results

A total of 10 articles were included. The extracted data included study design, participant characteristics, type of robot used, outcome measures, aim, and type of intervention. The results reveal that parental involvement in RMI could be feasible and useful in improving intervention efficacy, particularly in improving the child's social-communicative abilities and the caregiver's educational skills.

Discussion

RMI intervention with parental participation could be a useful therapeutic strategy in pediatrics. However, to date, few studies have investigated this specific topic, and the reported results may enhance future research to understand its effectiveness in specific areas of use.

Systematic review registration

identifier: CRD42024553214.

Bridging Requirements, Planning, and Evaluation: A Review of Social Robot Navigation

Navigation lies at the core of social robotics, enabling robots to navigate and interact seamlessly in human environments. The primary focus of human-aware robot navigation is minimizing discomfort among surrounding humans. Our review explores user studies, examining factors that cause human discomfort, to perform the grounding of social robot navigation requirements and to form a taxonomy of elementary necessities that should be implemented by comprehensive algorithms. This survey also discusses human-aware navigation from an algorithmic perspective, reviewing the perception and motion planning methods integral to social navigation. Additionally, the review investigates different types of studies and tools facilitating the evaluation of social robot navigation approaches, namely datasets, simulators, and benchmarks. Our survey also identifies the main challenges of human-aware navigation, highlighting the essential future work perspectives. This work stands out from other review papers, as it not only investigates the variety of methods for implementing human awareness in robot control systems but also classifies the approaches according to the grounded requirements regarded in their objectives.

Research on deep reinforcement learning basketball robot shooting skills improvement based on end to end architecture and multi-modal perception

Introduction

In the realm of basketball, refining shooting skills and decision-making levels using intelligent agents has garnered significant interest. This study addresses the challenge by introducing an innovative framework that combines multi-modal perception and deep reinforcement learning. The goal is to create basketball robots capable of executing precise shots and informed choices by effectively integrating sensory inputs and learned strategies.

Methods

The proposed approach consists of three main components: multi-modal perception, deep reinforcement learning, and end-to-end architecture. Multi-modal perception leverages the multi-head attention mechanism (MATT) to merge visual, motion, and distance cues for a holistic perception of the basketball scenario. The deep reinforcement learning framework utilizes the Deep Q-Network (DQN) algorithm, enabling the robots to learn optimal shooting strategies over iterative interactions with the environment. The end-to-end architecture connects these components, allowing seamless integration of perception and decision-making processes.

Results

The experiments conducted demonstrate the effectiveness of the proposed approach. Basketball robots equipped with multi-modal perception and deep reinforcement learning exhibit improved shooting accuracy and enhanced decision-making abilities. The multi-head attention mechanism enhances the robots' perception of complex scenes, leading to more accurate shooting decisions. The application of the DQN algorithm results in gradual skill improvement and strategic optimization through interaction with the environment.

Discussion

The integration of multi-modal perception and deep reinforcement learning within an end-to-end architecture presents a promising avenue for advancing basketball robot training and performance. The ability to fuse diverse sensory inputs and learned strategies empowers robots to make informed decisions and execute accurate shots. The research not only contributes to the field of robotics but also has potential implications for human basketball training and coaching methodologies.

Data-driven decision-making for precision diagnosis of digestive diseases

Modern omics technologies can generate massive amounts of biomedical data, providing unprecedented opportunities for individualized precision medicine. However, traditional statistical methods cannot effectively process and utilize such big data. To meet this new challenge, machine learning algorithms have been developed and applied rapidly in recent years, which are capable of reducing dimensionality, extracting features, organizing data and forming automatable data-driven clinical decision systems. Data-driven clinical decision-making have promising applications in precision medicine and has been studied in digestive diseases, including early diagnosis and screening, molecular typing, staging and stratification of digestive malignancies, as well as precise diagnosis of Crohn's disease, auxiliary diagnosis of imaging and endoscopy, differential diagnosis of cystic lesions, etiology discrimination of acute abdominal pain, stratification of upper gastrointestinal bleeding (UGIB), and real-time diagnosis of esophageal motility function, showing good application prospects. Herein, we reviewed the recent progress of data-driven clinical decision making in precision diagnosis of digestive diseases and discussed the limitations of data-driven decision making after a brief introduction of methods for data-driven decision making.

Transformer Decoder-Based Enhanced Exploration Method to Alleviate Initial Exploration Problems in Reinforcement Learning

In reinforcement learning, the epsilon (ε)-greedy strategy is commonly employed as an exploration technique This method, however, leads to extensive initial exploration and prolonged learning periods. Existing approaches to mitigate this issue involve constraining the exploration range using expert data or utilizing pretrained models. Nevertheless, these methods do not effectively reduce the initial exploration range, as the exploration by the agent is limited to states adjacent to those included in the expert data. This paper proposes a method to reduce the initial exploration range in reinforcement learning through a pretrained transformer decoder on expert data. The proposed method involves pretraining a transformer decoder with massive expert data to guide the agent's actions during the early learning stages. After achieving a certain learning threshold, the actions are determined using the epsilon-greedy strategy. An experiment was conducted in the basketball game FreeStyle1 to compare the proposed method with the traditional Deep Q-Network (DQN) using the epsilon-greedy strategy. The results indicated that the proposed method yielded approximately 2.5 times the average reward and a 26% higher win rate, proving its enhanced performance in reducing exploration range and optimizing learning times. This innovative method presents a significant improvement over traditional exploration techniques in reinforcement learning.

Learning Rat-Like Behavioral Interaction Using a Small-Scale Robotic Rat

In this paper, we propose a novel method for emulating rat-like behavioral interactions in robots using reinforcement learning. Specifically, we develop a state decision method to optimize the interaction process among 6 known behavior types that have been identified in previous research on rat interactions. The novelty of our method lies in using the temporal difference (TD) algorithm to optimize the state decision process, which enables the robots to make informed decisions about their behavior choices. To assess the similarity between robot and rat behavior, we use Pearson correlation. We then use TD-λ to update the state value function and make state decisions based on probability. The robots execute these decisions using our dynamics-based controller. Our results demonstrate that our method can generate rat-like behaviors on both short- and long-term timescales, with interaction information entropy comparable to that between real rats. Overall, our approach shows promise for controlling robots in robot-rat interactions and highlights the potential of using reinforcement learning to develop more sophisticated robotic systems.

Use of machine learning in the field of prosthetics and orthotics: A systematic narrative review

Although machine learning is not yet being used in clinical practice within the fields of prosthetics and orthotics, several studies on the use of prosthetics and orthotics have been conducted. We intend to provide relevant knowledge by conducting a systematic review of prior studies on using machine learning in the fields of prosthetics and orthotics. We searched the Medical Literature Analysis and Retrieval System Online (MEDLINE), Cochrane, Embase, and Scopus databases and retrieved studies published until July 18, 2021. The study included the application of machine learning algorithms to upper-limb and lower-limb prostheses and orthoses. The criteria of the Quality in Prognosis Studies tool were used to assess the methodological quality of the studies. A total of 13 studies were included in this systematic review. In the realm of prostheses, machine learning has been used to identify prosthesis, select an appropriate prosthesis, train after wearing the prosthesis, detect falls, and manage the temperature in the socket. In the field of orthotics, machine learning was used to control real-time movement while wearing an orthosis and predict the need for an orthosis. The studies included in this systematic review are limited to the algorithm development stage. However, if the developed algorithms are actually applied to clinical practice, it is expected that it will be useful for medical staff and users to handle prosthesis and orthosis.

A Systematic Literature Review of Decision-Making and Control Systems for Autonomous and Social Robots

In the last years, considerable research has been carried out to develop robots that can improve our quality of life during tedious and challenging tasks. In these contexts, robots operating without human supervision open many possibilities to assist people in their daily activities. When autonomous robots collaborate with humans, social skills are necessary for adequate communication and cooperation. Considering these facts, endowing autonomous and social robots with decision-making and control models is critical for appropriately fulfiling their initial goals. This manuscript presents a systematic review of the evolution of decision-making systems and control architectures for autonomous and social robots in the last three decades. These architectures have been incorporating new methods based on biologically inspired models and Machine Learning to enhance these systems’ possibilities to developed societies. The review explores the most novel advances in each application area, comparing their most essential features. Additionally, we describe the current challenges of software architecture devoted to action selection, an analysis not provided in similar reviews of behavioural models for autonomous and social robots. Finally, we present the future directions that these systems can take in the future.

Dynamic sub-route-based self-adaptive beam search Q-learning algorithm for traveling salesman problem

In this paper, a dynamic sub-route-based self-adaptive beam search Q-learning (DSRABSQL) algorithm is proposed that provides a reinforcement learning (RL) framework combined with local search to solve the traveling salesman problem (TSP). DSRABSQL builds upon the Q-learning (QL) algorithm. Considering its problems of slow convergence and low accuracy, four strategies within the QL framework are designed first: the weighting function-based reward matrix, the power function-based initial Q-table, a self-adaptive ε-beam search strategy, and a new Q-value update formula. Then, a self-adaptive beam search Q-learning (ABSQL) algorithm is designed. To solve the problem that the sub-route is not fully optimized in the ABSQL algorithm, a dynamic sub-route optimization strategy is introduced outside the QL framework, and then the DSRABSQL algorithm is designed. Experiments are conducted to compare QL, ABSQL, DSRABSQL, our previously proposed variable neighborhood discrete whale optimization algorithm, and two advanced reinforcement learning algorithms. The experimental results show that DSRABSQL significantly outperforms the other algorithms. In addition, two groups of algorithms are designed based on the QL and DSRABSQL algorithms to test the effectiveness of the five strategies. From the experimental results, it can be found that the dynamic sub-route optimization strategy and self-adaptive ε-beam search strategy contribute the most for small-, medium-, and large-scale instances. At the same time, collaboration exists between the four strategies within the QL framework, which increases with the expansion of the instance scale.

Telepresence Social Robotics towards Co-Presence: A Review

Telepresence robots are becoming popular in social interactions involving health care, elderly assistance, guidance, or office meetings. There are two types of human psychological experiences to consider in robot-mediated interactions: (1) telepresence, in which a user develops a sense of being present near the remote interlocutor, and (2) co-presence, in which a user perceives the other person as being present locally with him or her. This work presents a literature review on developments supporting robotic social interactions, contributing to improving the sense of presence and co-presence via robot mediation. This survey aims to define social presence, co-presence, identify autonomous “user-adaptive systems” for social robots, and propose a taxonomy for “co-presence” mechanisms. It presents an overview of social robotics systems, applications areas, and technical methods and provides directions for telepresence and co-presence robot design given the actual and future challenges. Finally, we suggest evaluation guidelines for these systems, having as reference face-to-face interaction.

KFSENet: A Key Frame-Based Skeleton Feature Estimation and Action Recognition Network for Improved Robot Vision with Face and Emotion Recognition

In this paper, we propose an integrated approach to robot vision: a key frame-based skeleton feature estimation and action recognition network (KFSENet) that incorporates action recognition with face and emotion recognition to enable social robots to engage in more personal interactions. Instead of extracting the human skeleton features from the entire video, we propose a key frame-based approach for their extraction using pose estimation models. We select the key frames using the gradient of a proposed total motion metric that is computed using dense optical flow. We use the extracted human skeleton features from the selected key frames to train a deep neural network (i.e., the double-feature double-motion network (DDNet)) for action recognition. The proposed KFSENet utilizes a simpler model to learn and differentiate between the different action classes, is computationally simpler and yields better action recognition performance when compared with existing methods. The use of key frames allows the proposed method to eliminate unnecessary and redundant information, which improves its classification accuracy and decreases its computational cost. The proposed method is tested on both publicly available standard benchmark datasets and self-collected datasets. The performance of the proposed method is compared to existing state-of-the-art methods. Our results indicate that the proposed method yields better performance compared with existing methods. Moreover, our proposed framework integrates face and emotion recognition to enable social robots to engage in more personal interaction with humans.

Position control of a planar cable-driven parallel robot using reinforcement learning

This study proposes a method based on reinforcement learning (RL) for point-to-point and dynamic reference position tracking control of a planar cable-driven parallel robots, which is a multi-input multi-output system (MIMO). The method eliminates the use of a tension distribution algorithm in controlling the system’s dynamics and inherently optimizes the cable tensions based on the reward function during the learning process. The deep deterministic policy gradient algorithm is utilized for training the RL agents in point-to-point and dynamic reference tracking tasks. The performances of the two agents are tested on their specifically trained tasks. Moreover, we also implement the agent trained for point-to-point tasks on the dynamic reference tracking and vice versa. The performances of the RL agents are compared with a classical PD controller. The results show that RL can perform quite well without the requirement of designing different controllers for each task if the system’s dynamics is learned well.

Artificial Intelligence Methodologies for Data Management

This study analyses the main challenges, trends, technological approaches, and artificial intelligence methods developed by new researchers and professionals in the field of machine learning, with an emphasis on the most outstanding and relevant works to date. This literature review evaluates the main methodological contributions of artificial intelligence through machine learning. The methodology used to study the documents was content analysis; the basic terminology of the study corresponds to machine learning, artificial intelligence, and big data between the years 2017 and 2021. For this study, we selected 181 references, of which 120 are part of the literature review. The conceptual framework includes 12 categories, four groups, and eight subgroups. The study of data management using AI methodologies presents symmetry in the four machine learning groups: supervised learning, unsupervised learning, semi-supervised learning, and reinforced learning. Furthermore, the artificial intelligence methods with more symmetry in all groups are artificial neural networks, Support Vector Machines, K-means, and Bayesian Methods. Finally, five research avenues are presented to improve the prediction of machine learning.