The spatial Laplacian and temporal energy pyramid representation for human action recognition using depth sequences

A Unified Multimodal De- and Re-Coupling Framework for RGB-D Motion Recognition

Motion recognition is a promising direction in computer vision, but the training of video classification models is much harder than images due to insufficient data and considerable parameters. To get around this, some works strive to explore multimodal cues from RGB-D data. Although improving motion recognition to some extent, these methods still face sub-optimal situations in the following aspects: (i) Data augmentation, i.e., the scale of the RGB-D datasets is still limited, and few efforts have been made to explore novel data augmentation strategies for videos; (ii) Optimization mechanism, i.e., the tightly space-time-entangled network structure brings more challenges to spatiotemporal information modeling; And (iii) cross-modal knowledge fusion, i.e., the high similarity between multimodal representations leads to insufficient late fusion. To alleviate these drawbacks, we propose to improve RGB-D-based motion recognition both from data and algorithm perspectives in this article. In more detail, firstly, we introduce a novel video data augmentation method dubbed ShuffleMix, which acts as a supplement to MixUp, to provide additional temporal regularization for motion recognition. Secondly, a Unified Multimodal De-coupling and multi-stage Re-coupling framework, termed UMDR, is proposed for video representation learning. Finally, a novel cross-modal Complement Feature Catcher (CFCer) is explored to mine potential commonalities features in multimodal information as the auxiliary fusion stream, to improve the late fusion results. The seamless combination of these novel designs forms a robust spatiotemporal representation and achieves better performance than state-of-the-art methods on four public motion datasets. Specifically, UMDR achieves unprecedented improvements of ↑ 4.5% on the Chalearn IsoGD dataset.

Dance Movement Recognition Based on Multimodal Environmental Monitoring Data

Fine motion recognition is a challenging topic in computer vision, and it has been a trendy research direction in recent years. This study combines motion recognition technology with dance movements and the problems such as the high complexity of dance movements and fully considers the human body's self-occlusion. The excellent motion recognition content in the dance field was studied and analyzed. A compelling feature extraction method was proposed for the dance video dataset, segmented video, and accumulated edge feature operation. By extracting directional gradient histogram features, a set of directional gradient histogram feature vectors is used to characterize the shape features of the dance video movements. A dance movement recognition method is adopted based on the fusion direction gradient histogram feature, optical flow direction histogram feature, and audio signature feature. Three components are combined for dance movement recognition by a multicore learning method. Experimental results show that the cumulative edge feature algorithm proposed in this study outperforms traditional models in the recognition results of HOG features extracted from images. After adding edge features, the description of the dance movement shape is more effective. The algorithm can guarantee a specific recognition rate of complex dance movements. The results also verify the effectiveness of the movement recognition algorithm in this study for dance movement recognition.

Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition

Gesture recognition has attracted considerable attention owing to its great potential in applications. Although the great progress has been made recently in multi-modal learning methods, existing methods still lack effective integration to fully explore synergies among spatio-temporal modalities effectively for gesture recognition. The problems are partially due to the fact that the existing manually designed network architectures have low efficiency in the joint learning of multi-modalities. In this paper, we propose the first neural architecture search (NAS)-based method for RGB-D gesture recognition. The proposed method includes two key components: 1) enhanced temporal representation via the proposed 3D Central Difference Convolution (3D-CDC) family, which is able to capture rich temporal context via aggregating temporal difference information; and 2) optimized backbones for multi-sampling-rate branches and lateral connections among varied modalities. The resultant multi-modal multi-rate network provides a new perspective to understand the relationship between RGB and depth modalities and their temporal dynamics. Comprehensive experiments are performed on three benchmark datasets (IsoGD, NvGesture, and EgoGesture), demonstrating the state-of-the-art performance in both single- and multi-modality settings. The code is available at https://github.com/ZitongYu/3DCDC-NAS.

Active Thermal Sensing for Bonding Structure Damage Detection of Hidden Frame Glass Curtain Wall

Adhesive bonded structure damage of hidden frame glass curtain wall will cause falling glass, which threatens the security of people and property. Therefore, the damage detection of the adhesive bonded structure of glass curtain wall has great significance. In this paper, active thermal sensing technology for bonding structure damage detection was introduced. Firstly, the thermal wave propagation of bonded structure was analyzed. Second, the simulated annealing algorithm and TracePro simulation were utilized to design the heat source. Thirdly, the platform of active thermal sensing was built, and experiments were conducted. Finally, image fusion enhancement of Laplacian pyramid was utilized to the enhancement process of thermal images. The simulation results showed that the irradiance of the cross-optimization was more uniform, and the uniformity was 17.50% higher than the original design value. The experiments results showed that defects of different sizes and depths can be distinguished. The gray differences of the damages on the depth of 0 mm and 4 mm were 0.025 and 0.045, respectively. The thermal wave testing can detect damage intuitively and rapidly, which is significant for the future research of unmanned detection of bonding structure damage of hidden frame glass curtain wall.