Wearable Drone Controller: Machine Learning-Based Hand Gesture Recognition and Vibrotactile Feedback

We proposed a wearable drone controller with hand gesture recognition and vibrotactile feedback. The intended hand motions of the user are sensed by an inertial measurement unit (IMU) placed on the back of the hand, and the signals are analyzed and classified using machine learning models. The recognized hand gestures control the drone, and the obstacle information in the heading direction of the drone is fed back to the user by activating the vibration motor attached to the wrist. Simulation experiments for drone operation were performed, and the participants' subjective evaluations regarding the controller's convenience and effectiveness were investigated. Finally, experiments with a real drone were conducted and discussed to validate the proposed controller.

Examining the Influence of Using First-Person View Drones as Auxiliary Devices in Matte Painting Courses on College Students’ Continuous Learning Intention

In terms of the teaching process of matte painting, it is essential for students to develop a sound understanding of the relationship between virtual and physical environments. In this study, first-person view (FPV) drones are applied to matte painting courses to evaluate the effectiveness of the teaching, and to propose more effective design suggestions for FPV drones that are more suitable for teaching. This provides students with a better learning environment using a digital education system. The results of the study indicate that the flow experience, learning interest, and continuous learning intention of students who use FPV drones in matte painting are significantly greater than those of students who only utilize traditional teaching methods. Furthermore, the technology incentive model (TIM) was developed in this study after being verified by the structural equation model. The results demonstrate that the second-order construct ‘technology incentive’ comprising perceived interactivity, perceived vividness, and novel experience positively influence students’ learning interest and continuous learning intentions under the mediation of flow experience.

Visual Collaboration Leader-Follower UAV-Formation for Indoor Exploration

UAVs operating in a leader-follower formation demand the knowledge of the relative pose between the collaborating members. This necessitates the RF-communication of this information which increases the communication latency and can easily result in lost data packets. In this work, rather than relying on this autopilot data exchange, a visual scheme using passive markers is presented. Each formation-member carries passive markers in a RhOct configuration. These markers are visually detected and the relative pose of the members is on-board determined, thus eliminating the need for RF-communication. A reference path is then evaluated for each follower that tracks the leader and maintains a constant distance between the formation-members. Experimental studies show a mean position detection error (5 × 5 × 10cm) or less than 0.0031% of the available workspace [0.5 up to 5m, 50.43° × 38.75° Field of View (FoV)]. The efficiency of the suggested scheme against varying delays are examined in these studies, where it is shown that a delay up to 1.25s can be tolerated for the follower to track the leader as long as the latter one remains within its FoV.

In-Device Feedback in Immersive Head-Mounted Displays for Distance Perception During Teleoperation of Unmanned Ground Vehicles

In recent years, Virtual Reality (VR) Head-Mounted Displays (HMD) have been used to provide an immersive, first-person view in real-time for the remote-control of Unmanned Ground Vehicles (UGV). One critical issue is that it is challenging to perceive the distance of obstacles surrounding the vehicle from 2D views in the HMD, which deteriorates the control of UGV. Conventional distance indicators used in HMD take up screen space which leads clutter on the display and can further reduce situation awareness of the physical environment. To address the issue, in this paper we propose off-screen in-device feedback using vibro-tactile and/or light-visual cues to provide real-time distance information for the remote control of UGV. Results from a study show a significantly better performance with either feedback type, reduced workload and improved usability in a driving task that requires continuous perception of the distance between the UGV and its environmental objects or obstacles. Our findings show a solid case for in-device vibro-tactile and/or light-visual feedback to support remote operation of UGVs that highly relies on distance perception of objects.

Technologies for Multimodal Interaction in Extended Reality—A Scoping Review

When designing extended reality (XR) applications, it is important to consider multimodal interaction techniques, which employ several human senses simultaneously. Multimodal interaction can transform how people communicate remotely, practice for tasks, entertain themselves, process information visualizations, and make decisions based on the provided information. This scoping review summarized recent advances in multimodal interaction technologies for head-mounted display-based (HMD) XR systems. Our purpose was to provide a succinct, yet clear, insightful, and structured overview of emerging, underused multimodal technologies beyond standard video and audio for XR interaction, and to find research gaps. The review aimed to help XR practitioners to apply multimodal interaction techniques and interaction researchers to direct future efforts towards relevant issues on multimodal XR. We conclude with our perspective on promising research avenues for multimodal interaction technologies.

Multi-User Drone Flight Training in Mixed Reality

The development of services and applications involving drones is promoting the growth of the unmanned-aerial-vehicle industry. Moreover, the supply of low-cost compact drones has greatly contributed to the popularization of drone flying. However, flying first-person-view (FPV) drones requires considerable experience because the remote pilot views a video transmitted from a camera mounted on the drone. In this paper, we propose a remote training system for FPV drone flying in mixed reality. Thereby, beginners who are inexperienced in FPV drone flight control can practice under the guidance of remote experts.

UAV swarm autonomous control based on Internet of Things and artificial intelligence algorithms

At present, the UAV swarm positioning solution has the problems of poor positioning accuracy and instability. Therefore, it is necessary to design a sliding mode formation controller to realize formation. This study analyzes the self-service control strategy of the UAV swarm and establishes the behavior-based formation control strategy as the main research point of this article. This paper combines the Internet of Things and artificial intelligence algorithms to build an autonomous control model for the UAV swarm and designs the UAV formation control law from the disturbed and undisturbed conditions respectively. With reference to the basic architecture of the Internet of Things, this study imitates ZigBee’s self-organizing network to propose an adaptive networking scheme based on the Internet of Things by using the AP+STA working mode of the Internet of Things module in the node device. The results of the experiment show that the positioning accuracy of the UAV is high, which can meet the needs of cluster flight. Based on the Z-axis coordinates of the UAV, the accuracy of the laser distance measurement and the barometer value is significantly improved, the root mean square error is reduced, and the positioning result is significantly better than the data of direct traditional positioning.

An Aerial Mixed-Reality Environment for First-Person-View Drone Flying

A drone be able to fly without colliding to preserve the surroundings and its own safety. In addition, it must also incorporate numerous features of interest for drone users. In this paper, an aerial mixed-reality environment for first-person-view drone flying is proposed to provide an immersive experience and a safe environment for drone users by creating additional virtual obstacles when flying a drone in an open area. The proposed system is effective in perceiving the depth of obstacles, and enables bidirectional interaction between real and virtual worlds using a drone equipped with a stereo camera based on human binocular vision. In addition, it synchronizes the parameters of the real and virtual cameras to effectively and naturally create virtual objects in a real space. Based on user studies that included both general and expert users, we confirm that the proposed system successfully creates a mixed-reality environment using a flying drone by quickly recognizing real objects and stably combining them with virtual objects.