A Systematic Review of Virtual Reality Interfaces for Controlling and Interacting with Robots

A review on cultivating effective learning: synthesizing educational theories and virtual reality for enhanced educational experiences

Immersive technology, especially virtual reality (VR), transforms education. It offers immersive and interactive learning experiences. This study presents a systematic review focusing on VR's integration with educational theories in higher education. The review evaluates the literature on VR applications combined with pedagogical frameworks. It aims to identify effective strategies for enhancing educational experiences through VR. The process involved analyzing studies about VR and educational theories, focusing on methodologies, outcomes, and effectiveness. Findings show that VR improves learning outcomes when aligned with theories such as constructivism, experiential learning, and collaborative learning. These integrations offer personalized, immersive, and interactive learning experiences. The study highlights the importance of incorporating educational principles into VR application development. It suggests a promising direction for future research and implementation in education. This approach aims to maximize VR's pedagogical value, enhancing learning outcomes across educational settings.

Clustering user preferences for personalized teleoperation control schemes via trajectory similarity analysis

Successful operation of a teleoperated robot depends on a well-designed control scheme to translate human motion into robot motion; however, a single control scheme may not be suitable for all users. On the other hand, individual personalization of control schemes may be infeasible for designers to produce. In this paper, we present a method by which users may be classified into groups with mutually compatible control scheme preferences. Users are asked to demonstrate freehand motions to control a simulated robot in a virtual reality environment. Hand pose data is captured and compared with other users using SLAM trajectory similarity analysis techniques. The resulting pairwise trajectory error metrics are used to cluster participants based on their control motions, without foreknowledge of the number or types of control scheme preferences that may exist. The clusters identified for two different robots shows that a small number of clusters form stably for each case, each with its own control scheme paradigm. Survey data from participants validates that the clusters identified through this method correspond to the participants' control scheme rationales, and also identify nuances in participant control scheme descriptions that may not be obvious to designers relying only on participant explanations of their preferences.

Research on Teleoperated Virtual Reality Human-Robot Five-Dimensional Collaboration System

In the realm of industrial robotics, there is a growing challenge in simplifying human-robot collaboration (HRC), particularly in complex settings. The demand for more intuitive teleoperation systems is on the rise. However, optimizing robot control interfaces and streamlining teleoperation remains a formidable task due to the need for operators to possess specialized knowledge and the limitations of traditional methods regarding operational space and time constraints. This study addresses these issues by introducing a virtual reality (VR) HRC system with five-dimensional capabilities. Key advantages of our approach include: (1) real-time observation of robot work, whereby operators can seamlessly monitor the robot's real-time work environment and motion during teleoperation; (2) leveraging VR device capabilities, whereby the strengths of VR devices are harnessed to simplify robot motion control, significantly reducing the learning time for operators; and (3) adaptability across platforms and environments: our system effortlessly adapts to various platforms and working conditions, ensuring versatility across different terminals and scenarios. This system represents a significant advancement in addressing the challenges of HRC, offering improved teleoperation, simplified control, and enhanced accessibility, particularly for operators with limited prior exposure to robot operation. It elevates the overall HRC experience in complex scenarios.

Overview of the role of robots in upper limb disabilities rehabilitation: a scoping review

BACKGROUND

Neuromotor rehabilitation and improvement of upper limb functions are necessary to improve the life quality of patients who have experienced injuries or have pathological outcomes. Modern approaches, such as robotic-assisted rehabilitation can help to improve rehabilitation processes and thus improve upper limb functions. Therefore, the aim of this study was to investigate the role of robots in upper limb disability improvement and rehabilitation.

METHODS

This scoping review was conducted by search in PubMed, Web of Science, Scopus, and IEEE (January 2012- February 2022). Articles related to upper limb rehabilitation robots were selected. The methodological quality of all the included studies will be appraised using the Mixed Methods Appraisal Tool (MMAT). We used an 18-field data extraction form to extract data from articles and extracted the information such as study year, country, type of study, purpose, illness or accident leading to disability, level of disability, assistive technologies, number of participants in the study, sex, age, rehabilitated part of the upper limb using a robot, duration and frequency of treatment, methods of performing rehabilitation exercises, type of evaluation, number of participants in the evaluation process, duration of intervention, study outcomes, and study conclusions. The selection of articles and data extraction was made by three authors based on inclusion and exclusion criteria. Disagreements were resolved through consultation with the fifth author. Inclusion criteria were articles involving upper limb rehabilitation robots, articles about upper limb disability caused by any illness or injury, and articles published in English. Also, articles involving other than upper limb rehabilitation robots, robots related to rehabilitation of diseases other than upper limb, systematic reviews, reviews, and meta-analyses, books, book chapters, letters to the editor, and conference papers were also excluded. Descriptive statistics methods (frequency and percentage) were used to analyses the data.

RESULTS

We finally included 55 relevant articles. Most of the studies were done in Italy (33.82%). Most robots were used to rehabilitate stroke patients (80%). About 60.52% of the studies used games and virtual reality rehabilitate the upper limb disabilities using robots. Among the 14 types of applied evaluation methods, "evaluation and measurement of upper limb function and dexterity" was the most applied evaluation method. "Improvement in musculoskeletal functions", "no adverse effect on patients", and "Safe and reliable treatment" were the most cited outcomes, respectively.

CONCLUSIONS

Our findings show that robots can improve musculoskeletal functions (musculoskeletal strength, sensation, perception, vibration, muscle coordination, less spasticity, flexibility, and range of motion) and empower people by providing a variety of rehabilitation capabilities.

Evaluating the Effects of Virtual Reality Environment Learning on Subsequent Robot Teleoperation in an Unfamiliar Building

Using a map in an unfamiliar environment requires identifying correspondences between elements of the map's allocentric representation and elements in egocentric views. Aligning the map with the environment can be challenging. Virtual reality (VR) allows learning about unfamiliar environments in a sequence of egocentric views that correspond closely to the perspectives and views that are experienced in the actual environment. We compared three methods to prepare for localization and navigation tasks performed by teleoperating a robot in an office building: studying a floor plan of the building and two forms of VR exploration. One group of participants studied a building plan, a second group explored a faithful VR reconstruction of the building from a normal-sized avatar's perspective, and a third group explored the VR from a giant-sized avatar's perspective. All methods contained marked checkpoints. The subsequent tasks were identical for all groups. The self-localization task required indication of the approximate location of the robot in the environment. The navigation task required navigation between checkpoints. Participants took less time to learn with the giant VR perspective and with the floorplan than with the normal VR perspective. Both VR learning methods significantly outperformed the floorplan in the orientation task. Navigation was performed quicker after learning in the giant perspective compared to the normal perspective and the building plan. We conclude that the normal perspective and especially the giant perspective in VR are viable options for preparing for teleoperation in unfamiliar environments when a virtual model of the environment is available.

Analysis of the Snake Robot Kinematics with Virtual Reality Visualisation

In this article, we present a novel approach to performing engineering simulation in an interactive environment. A synesthetic design approach is employed, which enables the user to gather information about the system's behaviour more holistically, at the same time as facilitating interaction with the simulated system. The system considered in this work is a snake robot moving on a flat surface. The dynamic simulation of the robot's movement is realised in dedicated engineering software, whereas this software exchanges information with the 3D visualisation software and a Virtual Reality (VR) headset. Several simulation scenarios have been presented, comparing the proposed method with standard ways for visualising the robot's motion, such as 2D plots and 3D animations on a computer screen. This illustrates how, in the engineering context, this more immersive experience, allowing the viewer to observe the simulation results and modify the simulation parameters within the VR environment, can facilitate the analysis and design of systems.

Mixed-Reality-Enhanced Human–Robot Interaction with an Imitation-Based Mapping Approach for Intuitive Teleoperation of a Robotic Arm-Hand System

This paper presents an integrated mapping of motion and visualization scheme based on a Mixed Reality (MR) subspace approach for the intuitive and immersive telemanipulation of robotic arm-hand systems. The effectiveness of different control-feedback methods for the teleoperation system is validated and compared. The robotic arm-hand system consists of a 6 Degrees-of-Freedom (DOF) industrial manipulator and a low-cost 2-finger gripper, which can be manipulated in a natural manner by novice users physically distant from the working site. By incorporating MR technology, the user is fully immersed in a virtual operating space augmented by real-time 3D visual feedback from the robot working site. Imitation-based velocity-centric motion mapping is implemented via the MR subspace to accurately track operator hand movements for robot motion control and enables spatial velocity-based control of the robot Tool Center Point (TCP). The user control space and robot working space are overlaid through the MR subspace, and the local user and a digital twin of the remote robot share the same environment in the MR subspace. The MR-based motion and visualization mapping scheme for telerobotics is compared to conventional 2D Baseline and MR tele-control paradigms over two tabletop object manipulation experiments. A user survey of 24 participants was conducted to demonstrate the effectiveness and performance enhancements enabled by the proposed system. The MR-subspace-integrated 3D mapping of motion and visualization scheme reduced the aggregate task completion time by 48% compared to the 2D Baseline module and 29%, compared to the MR SpaceMouse module. The perceived workload decreased by 32% and 22%, compared to the 2D Baseline and MR SpaceMouse approaches.

An Approach Based on VR to Design Industrial Human-Robot Collaborative Workstations

This paper presents an integrated approach for the design of human-robot collaborative workstations in industrial shop floors. In particular, the paper presents how to use virtual reality (VR) technologies to support designers in the creation of interactive workstation prototypes and in early validation of design outcomes. VR allows designers to consider and evaluate in advance the overall user experience, adopting a user-centered perspective. The proposed approach relies on two levels: the first allows designers to have an automatic generation and organization of the workstation physical layout in VR, starting from a conceptual description of its functionalities and required tools; the second aims at supporting designers during the design of Human-Machine Interfaces (HMIs) by interaction mapping, HMI prototyping and testing in VR. The proposed approach has been applied on two realistic industrial case studies related to the design of an intensive warehouse and a collaborative assembly workstation for automotive industry, respectively. The two case studies demonstrate how the approach is suited for early prototyping of complex environments and human-machine interactions by taking into account the user experience from the early phases of design.

Motion Polytopes in Virtual Reality for Shared Control in Remote Manipulation Applications

In remote applications that mandate human supervision, shared control can prove vital by establishing a harmonious balance between the high-level cognition of a user and the low-level autonomy of a robot. Though in practice, achieving this balance is a challenging endeavor that largely depends on whether the operator effectively interprets the underlying shared control. Inspired by recent works on using immersive technologies to expose the internal shared control, we develop a virtual reality system to visually guide human-in-the-loop manipulation. Our implementation of shared control teleoperation employs end effector manipulability polytopes, which are geometrical constructs that embed joint limit and environmental constraints. These constructs capture a holistic view of the constrained manipulator’s motion and can thus be visually represented as feedback for users on their operable space of movement. To assess the efficacy of our proposed approach, we consider a teleoperation task where users manipulate a screwdriver attached to a robotic arm’s end effector. A pilot study with prospective operators is first conducted to discern which graphical cues and virtual reality setup are most preferable. Feedback from this study informs the final design of our virtual reality system, which is subsequently evaluated in the actual screwdriver teleoperation experiment. Our experimental findings support the utility of using polytopes for shared control teleoperation, but hint at the need for longer-term studies to garner their full benefits as virtual guides.