Deep Learning Framework for Controlling Work Sequence in Collaborative Human-Robot Assembly Processes

On the Evaluation of Diverse Vision Systems towards Detecting Human Pose in Collaborative Robot Applications

Tracking human operators working in the vicinity of collaborative robots can improve the design of safety architecture, ergonomics, and the execution of assembly tasks in a human-robot collaboration scenario. Three commercial spatial computation kits were used along with their Software Development Kits that provide various real-time functionalities to track human poses. The paper explored the possibility of combining the capabilities of different hardware systems and software frameworks that may lead to better performance and accuracy in detecting the human pose in collaborative robotic applications. This study assessed their performance in two different human poses at six depth levels, comparing the raw data and noise-reducing filtered data. In addition, a laser measurement device was employed as a ground truth indicator, together with the average Root Mean Square Error as an error metric. The obtained results were analysed and compared in terms of positional accuracy and repeatability, indicating the dependence of the sensors' performance on the tracking distance. A Kalman-based filter was applied to fuse the human skeleton data and then to reconstruct the operator's poses considering their performance in different distance zones. The results indicated that at a distance less than 3 m, Microsoft Azure Kinect demonstrated better tracking performance, followed by Intel RealSense D455 and Stereolabs ZED2, while at ranges higher than 3 m, ZED2 had superior tracking performance.

Elderly and visually impaired indoor activity monitoring based on Wi-Fi and Deep Hybrid convolutional neural network

A drop in physical activity and a deterioration in the capacity to undertake daily life activities are both connected with ageing and have negative effects on physical and mental health. An Elderly and Visually Impaired Human Activity Monitoring (EV-HAM) system that keeps tabs on a person's routine and steps in if a change in behaviour or a crisis might greatly help an elderly person or a visually impaired. These individuals may find greater freedom with the help of an EVHAM system. As the backbone of human-centric applications like actively supported living and in-home monitoring for the elderly and visually impaired, an EVHAM system is essential. Big data-driven product design is flourishing in this age of 5G and the IoT. Recent advancements in processing power and software architectures have also contributed to the emergence and development of artificial intelligence (AI). In this context, the digital twin has emerged as a state-of-the-art technology that bridges the gap between the real and virtual worlds by evaluating data from several sensors using artificial intelligence algorithms. Although promising findings have been reported by Wi-Fi-based human activity identification techniques so far, their effectiveness is vulnerable to environmental variations. Using the environment-independent fingerprints generated from the Wi-Fi channel state information (CSI), we introduce Wi-Sense. This human activity identification system employs a Deep Hybrid convolutional neural network (DHCNN). The proposed system begins by collecting the CSI with a regular Wi-Fi Network Interface Controller. Wi-Sense uses the CSI ratio technique to lessen the effect of noise and the phase offset. The t- Distributed Stochastic Neighbor Embedding (t-SNE) is used to eliminate unnecessary data further. The data dimension is decreased, and the negative effects on the environment are eliminated in this process. The resulting spectrogram of the processed data exposes the activity's micro-Doppler fingerprints as a function of both time and location. These spectrograms are put to use in the training of a DHCNN. Based on our findings, EVHAM can accurately identify these actions 99% of the time.

Regression-Based Camera Pose Estimation through Multi-Level Local Features and Global Features

Accurate and robust camera pose estimation is essential for high-level applications such as augmented reality and autonomous driving. Despite the development of global feature-based camera pose regression methods and local feature-based matching guided pose estimation methods, challenging conditions, such as illumination changes and viewpoint changes, as well as inaccurate keypoint localization, continue to affect the performance of camera pose estimation. In this paper, we propose a novel relative camera pose regression framework that uses global features with rotation consistency and local features with rotation invariance. First, we apply a multi-level deformable network to detect and describe local features, which can learn appearances and gradient information sensitive to rotation variants. Second, we process the detection and description processes using the results from pixel correspondences of the input image pairs. Finally, we propose a novel loss that combines relative regression loss and absolute regression loss, incorporating global features with geometric constraints to optimize the pose estimation model. Our extensive experiments report satisfactory accuracy on the 7Scenes dataset with an average mean translation error of 0.18 m and a rotation error of 7.44° using image pairs as input. Ablation studies were also conducted to verify the effectiveness of the proposed method in the tasks of pose estimation and image matching using the 7Scenes and HPatches datasets.

A Novel Simulated Annealing-Based Hyper-Heuristic Algorithm for Stochastic Parallel Disassembly Line Balancing in Smart Remanufacturing

Remanufacturing prolongs the life cycle and increases the residual value of various end-of-life (EoL) products. As an inevitable process in remanufacturing, disassembly plays an essential role in retrieving the high-value and useable components of EoL products. To disassemble massive quantities and multi-types of EoL products, disassembly lines are introduced to improve the cost-effectiveness and efficiency of the disassembly processes. In this context, disassembly line balancing problem (DLBP) becomes a critical challenge that determines the overall performance of disassembly lines. Currently, the DLBP is mostly studied in straight disassembly lines using single-objective optimization methods, which cannot represent the actual disassembly environment. Therefore, in this paper, we extend the mathematical model of the basic DLBP to stochastic parallel complete disassembly line balancing problem (DLBP-SP). A novel simulated annealing-based hyper-heuristic algorithm (HH) is proposed for multi-objective optimization of the DLBP-SP, considering the number of workstations, working load index, and profits. The feasibility, superiority, stability, and robustness of the proposed HH algorithm are validated through computational experiments, including a set of comparison experiments and a case study of gearboxes disassembly. To the best of our knowledge, this research is the first to introduce gearboxes as a case study in DLBP which enriches the research on disassembly of industrial equipment.