Robust Gait Recognition by Integrating Inertial and RGBD Sensors

Effective lane detection on complex roads with convolutional attention mechanism in autonomous vehicles

Autonomous Vehicles (AV's) have achieved more popularity in vehicular technology in recent years. For the development of secure and safe driving, these AV's help to reduce the uncertainties such as crashes, heavy traffic, pedestrian behaviours, random objects, lane detection, different types of roads and their surrounding environments. In AV's, Lane Detection is one of the most important aspects which helps in lane holding guidance and lane departure warning. From Literature, it is observed that existing deep learning models perform better on well maintained roads and in favourable weather conditions. However, performance in extreme weather conditions and curvy roads need focus. The proposed work focuses on presenting an accurate lane detection approach on poor roads, particularly those with curves, broken lanes, or no lane markings and extreme weather conditions. Lane Detection with Convolutional Attention Mechanism (LD-CAM) model is proposed to achieve this outcome. The proposed method comprises an encoder, an enhanced convolution block attention module (E-CBAM), and a decoder. The encoder unit extracts the input image features, while the E-CBAM focuses on quality of feature maps in input images extracted from the encoder, and the decoder provides output without loss of any information in the original image. The work is carried out using the distinct data from three datasets called Tusimple for different weather condition images, Curve Lanes for different curve lanes images and Cracks and Potholes for damaged road images. The proposed model trained using these datasets showcased an improved performance attaining an Accuracy of 97.90%, Precision of 98.92%, F1-Score of 97.90%, IoU of 98.50% and Dice Co-efficient as 98.80% on both structured and defective roads in extreme weather conditions.

Multi-Level Adversarial Spatio-Temporal Learning for Footstep Pressure Based FoG Detection

Freezing of gait (FoG) is one of the most common symptoms of Parkinson's disease, which is a neurodegenerative disorder of the central nervous system impacting millions of people around the world. To address the pressing need to improve the quality of treatment for FoG, devising a computer-aided detection and quantification tool for FoG has been increasingly important. As a non-invasive technique for collecting motion patterns, the footstep pressure sequences obtained from pressure sensitive gait mats provide a great opportunity for evaluating FoG in the clinic and potentially in the home environment. In this study, FoG detection is formulated as a sequential modelling task and a novel deep learning architecture, namely Adversarial Spatio-temporal Network (ASTN), is proposed to learn FoG patterns across multiple levels. ASTN introduces a novel adversarial training scheme with a multi-level subject discriminator to obtain subject-independent FoG representations, which helps to reduce the over-fitting risk due to the high inter-subject variance. As a result, robust FoG detection can be achieved for unseen subjects. The proposed scheme also sheds light on improving subject-level clinical studies from other scenarios as it can be integrated with many existing deep architectures. To the best of our knowledge, this is one of the first studies of footstep pressure-based FoG detection and the approach of utilizing ASTN is the first deep neural network architecture in pursuit of subject-independent representations. In our experiments on 393 trials collected from 21 subjects, the proposed ASTN achieved an AUC 0.85, clearly outperforming conventional learning methods.

Human Action Recognition From Various Data Modalities: A Review

Human Action Recognition (HAR) aims to understand human behavior and assign a label to each action. It has a wide range of applications, and therefore has been attracting increasing attention in the field of computer vision. Human actions can be represented using various data modalities, such as RGB, skeleton, depth, infrared, point cloud, event stream, audio, acceleration, radar, and WiFi signal, which encode different sources of useful yet distinct information and have various advantages depending on the application scenarios. Consequently, lots of existing works have attempted to investigate different types of approaches for HAR using various modalities. In this article, we present a comprehensive survey of recent progress in deep learning methods for HAR based on the type of input data modality. Specifically, we review the current mainstream deep learning methods for single data modalities and multiple data modalities, including the fusion-based and the co-learning-based frameworks. We also present comparative results on several benchmark datasets for HAR, together with insightful observations and inspiring future research directions.

Multimodal Gait Recognition for Neurodegenerative Diseases

In recent years, single modality-based gait recognition has been extensively explored in the analysis of medical images or other sensory data, and it is recognized that each of the established approaches has different strengths and weaknesses. As an important motor symptom, gait disturbance is usually used for diagnosis and evaluation of diseases; moreover, the use of multimodality analysis of the patient's walking pattern compensates for the one-sidedness of single modality gait recognition methods that only learn gait changes in a single measurement dimension. The fusion of multiple measurement resources has demonstrated promising performance in the identification of gait patterns associated with individual diseases. In this article, as a useful tool, we propose a novel hybrid model to learn the gait differences between three neurodegenerative diseases, between patients with different severity levels of Parkinson's disease, and between healthy individuals and patients, by fusing and aggregating data from multiple sensors. A spatial feature extractor (SFE) is applied to generating representative features of images or signals. In order to capture temporal information from the two modality data, a new correlative memory neural network (CorrMNN) architecture is designed for extracting temporal features. Afterward, we embed a multiswitch discriminator to associate the observations with individual state estimations. Compared with several state-of-the-art techniques, our proposed framework shows more accurate classification results.

An Overview of the Practical Use of the CCTV System in a Simple Assembly in a Flexible Manufacturing System

In this article, experiments are realized in the flexible manufacturing system ICIM 3000 (FESTO, Germany), and its assembly system, located at the Institute of production technologies, Faculty of Material Sciences and Technologies, Slovak University of Technology. The assembly system is the final product assembled, and this process consists of five components. Unwanted inaccuracies in the assembly process of the elements, such as the insertion of thermometers and hygrometers into the base plate, usually arise. Based on these inaccuracies, we realize some experiments by the camera system SBOC-Q-R3-WB. This deals with the method of image processing. The camera system parameters are set-up. At the end of this contribution, a base of evaluated results is suggested and some minor design changes are realized in the assembly station. The goal of these changes is the higher reliability of the assembly process.

A Unified Local–Global Feature Extraction Network for Human Gait Recognition Using Smartphone Sensors

Smartphone-based gait recognition has been considered a unique and promising technique for biometric-based identification. It is integrated with multiple sensors to collect inertial data while a person walks. However, captured data may be affected by several covariate factors due to variations of gait sequences such as holding loads, wearing types, shoe types, etc. Recent gait recognition approaches either work on global or local features, causing failure to handle these covariate-based features. To address these issues, a novel weighted multi-scale CNN (WMsCNN) architecture is designed to extract local to global features for boosting recognition accuracy. Specifically, a weight update sub-network (Ws) is proposed to increase or reduce the weights of features concerning their contribution to the final classification task. Thus, the sensitivity of these features toward the covariate factors decreases using the weight updated technique. Later, these features are fed to a fusion module used to produce global features for the overall classification. Extensive experiments have been conducted on four different benchmark datasets, and the demonstrated results of the proposed model are superior to other state-of-the-art deep learning approaches.

On Learning Disentangled Representations for Gait Recognition

Gait, the walking pattern of individuals, is one of the important biometrics modalities. Most of the existing gait recognition methods take silhouettes or articulated body models as gait features. These methods suffer from degraded recognition performance when handling confounding variables, such as clothing, carrying and viewing angle. To remedy this issue, we propose a novel AutoEncoder framework, GaitNet, to explicitly disentangle appearance, canonical and pose features from RGB imagery. The LSTM integrates pose features over time as a dynamic gait feature while canonical features are averaged as a static gait feature. Both of them are utilized as classification features. In addition, we collect a Frontal-View Gait (FVG) dataset to focus on gait recognition from frontal-view walking, which is a challenging problem since it contains minimal gait cues compared to other views. FVG also includes other important variations, e.g., walking speed, carrying, and clothing. With extensive experiments on CASIA-B, USF, and FVG datasets, our method demonstrates superior performance to the SOTA quantitatively, the ability of feature disentanglement qualitatively, and promising computational efficiency. We further compare our GaitNet with state-of-the-art face recognition to demonstrate the advantages of gait biometrics identification under certain scenarios, e.g., long-distance/lower resolutions, cross viewing angles. Source code is available at http://cvlab.cse.msu.edu/project-gaitnet.html.

Gait Recognition Using Optical Motion Capture: A Decision Fusion Based Method

Human identification based on motion capture data has received signification attentions for its wide applications in authentication and surveillance systems. The optical motion capture system (OMCS) can dynamically capture the high-precision three-dimensional locations of optical trackers that are implemented on a human body, but its potential in applications on gait recognition has not been studied in existing works. On the other hand, a typical OMCS can only support one player one time, which limits its capability and efficiency. In this paper, our goals are investigating the performance of OMCS-based gait recognition performance, and realizing gait recognition in OMCS such that it can support multiple players at the same time. We develop a gait recognition method based on decision fusion, and it includes the following four steps: feature extraction, unreliable feature calibration, classification of single motion frame, and decision fusion of multiple motion frame. We use kernel extreme learning machine (KELM) for single motion classification, and in particular we propose a reliability weighted sum (RWS) decision fusion method to combine the fuzzy decisions of the motion frames. We demonstrate the performance of the proposed method by using walking gait data collected from 76 participants, and results show that KELM significantly outperforms support vector machine (SVM) and random forest in the single motion frame classification task, and demonstrate that the proposed RWS decision fusion rule can achieve better fusion accuracy compared with conventional fusion rules. Our results also show that, with 10 motion trackers that are implemented on lower body locations, the proposed method can achieve 100% validation accuracy with less than 50 gait motion frames.

Cross-Subject and Cross-Modal Transfer for Generalized Abnormal Gait Pattern Recognition

For abnormal gait recognition, pattern-specific features indicating abnormalities are interleaved with the subject-specific differences representing biometric traits. Deep representations are, therefore, prone to overfitting, and the models derived cannot generalize well to new subjects. Furthermore, there is limited availability of abnormal gait data obtained from precise Motion Capture (Mocap) systems because of regulatory issues and slow adaptation of new technologies in health care. On the other hand, data captured from markerless vision sensors or wearable sensors can be obtained in home environments, but noises from such devices may prevent the effective extraction of relevant features. To address these challenges, we propose a cascade of deep architectures that can encode cross-modal and cross-subject transfer for abnormal gait recognition. Cross-modal transfer maps noisy data obtained from RGBD and wearable sensors to accurate 4-D representations of the lower limb and joints obtained from the Mocap system. Subsequently, cross-subject transfer allows disentangling subject-specific from abnormal pattern-specific gait features based on a multiencoder autoencoder architecture. To validate the proposed methodology, we obtained multimodal gait data based on a multicamera motion capture system along with synchronized recordings of electromyography (EMG) data and 4-D skeleton data extracted from a single RGBD camera. Classification accuracy was improved significantly in both Mocap and noisy modalities.

A Modified Chaotic Binary Particle Swarm Optimization Scheme and Its Application in Face-Iris Multimodal Biometric Identification

In order to improve the recognition rate of the biometric identification system, the features of each unimodal biometric are often combined in a certain way. However, there are some mutually exclusive redundant features in those combined features, which will degrade the identification performance. To solve this problem, this paper proposes a novel multimodal biometric identification system for face-iris recognition.It is based on binary particle swarm optimization. The face features are extracted by 2D Log-Gabor and Curvelet transform, while iris features are extracted by Curvelet transform. In order to reduce the complexity of the feature-level fusion, we propose a modified chaotic binary particle swarm optimization (MCBPSO) algorithm to select features. It uses kernel extreme learning machine (KELM) as a fitness function and chaotic binary sequences to initialize particle swarms. After the global optimal position (Gbest) is generated in each iteration, the position of Gbest is varied by using chaotic binary sequences, which is useful to realize chaotic local search and avoid falling into the local optimal position. The experiments are conducted on CASIA multimodal iris and face dataset from Chinese Academy of Sciences.The experimental results demonstrate that the proposed system can not only reduce the number of features to one tenth of its original size, but also improve the recognition rate up to 99.78%. Compared with the unimodal iris and face system, the recognition rate of the proposed system are improved by 11.56% and 2% respectively. The experimental results reveal its performance in the verification mode compared with the existing state-of-the-art systems. The proposed system is satisfactory in addressing face-iris multimodal biometric identification.

Curve Similarity Model for Real-Time Gait Phase Detection Based on Ground Contact Forces

This paper proposed a new novel method to adaptively detect gait patterns in real time through the ground contact forces (GCFs) measured by load cell. The curve similarity model (CSM) is used to identify the division of off-ground and on-ground statuses, and differentiate gait patterns based on the detection rules. Traditionally, published threshold-based methods detect gait patterns by means of setting a fixed threshold to divide the GCFs into on-ground and off-ground statuses. However, the threshold-based methods in the literature are neither an adaptive nor a real-time approach. In this paper, the curve is composed of a series of continuous or discrete ordered GCF data points, and the CSM is built offline to obtain a training template. Then, the testing curve is compared with the training template to figure out the degree of similarity. If the computed degree of similarity is less than a given threshold, they are considered to be similar, which would lead to the division of off-ground and on-ground statuses. Finally, gait patterns could be differentiated according to the status division based on the detection rules. In order to test the detection error rate of the proposed method, a method in the literature is introduced as the reference method to obtain comparative results. The experimental results indicated that the proposed method could be used for real-time gait pattern detection, detect the gait patterns adaptively, and obtain a low error rate compared with the reference method.

Free-view gait recognition

Human gait has been shown to be an effective biometric measure for person identification at a distance. On the other hand, changes in the view angle pose a major challenge for gait recognition as human gait silhouettes are usually different from different view angles. Traditionally, such a multi-view gait recognition problem can be tackled by View Transformation Model (VTM) which transforms gait features from multiple gallery views to the probe view so as to evaluate the gait similarity. In the real-world environment, however, gait sequences may be captured from an uncontrolled scene and the view angle is often unknown, dynamically changing, or does not belong to any predefined views (thus VTM becomes inapplicable). To address this free-view gait recognition problem, we propose an innovative view-adaptive mapping (VAM) approach. The VAM employs a novel walking trajectory fitting (WTF) to estimate the view angles of a gait sequence, and a joint gait manifold (JGM) to find the optimal manifold between the probe data and relevant gallery data for gait similarity evaluation. Additionally, a RankSVM-based algorithm is developed to supplement the gallery data for subjects whose gallery features are only available in predefined views. Extensive experiments on both indoor and outdoor datasets demonstrate that the VAM outperforms several reference methods remarkably in free-view gait recognition.

Assessment of frailty: a survey of quantitative and clinical methods

BACKGROUND

Frailty assessment is a critical approach in assessing the health status of older people. The clinical tools deployed by geriatricians to assess frailty can be grouped into two categories; using a questionnaire-based method or analyzing the physical performance of the subject. In performance analysis, the time taken by a subject to complete a physical task such as walking over a specific distance, typically three meters, is measured. The questionnaire-based method is subjective, and the time-based performance analysis does not necessarily identify the kinematic characteristics of motion and their root causes. However, kinematic characteristics are crucial in measuring the degree of frailty.

RESULTS

The studies reviewed in this paper indicate that the quantitative analysis of activity of daily living, balance and gait are significant methods for assessing frailty in older people. Kinematic parameters (such as gait speed) and sensor-derived parameters are also strong markers of frailty. Seventeen gait parameters are found to be sensitive for discriminating various frailty levels. Gait velocity is the most significant parameter. Short term monitoring of daily activities is a more significant method for frailty assessment than is long term monitoring and can be implemented easily using clinical tests such as sit to stand or stand to sit. The risk of fall can be considered an outcome of frailty.

CONCLUSION

Frailty is a multi-dimensional phenomenon that is defined by various domains; physical, social, psychological and environmental. The physical domain has proven to be essential in the objective determination of the degree of frailty in older people. The deployment of inertial sensor in clinical tests is an effective method for the objective assessment of frailty.

RRV: A Spatiotemporal Descriptor for Rigid Body Motion Recognition

The motion behaviors of a rigid body can be characterized by a six degrees of freedom motion trajectory, which contains the 3-D position vectors of a reference point on the rigid body and 3-D rotations of this rigid body over time. This paper devises a rotation and relative velocity (RRV) descriptor by exploring the local translational and rotational invariants of rigid body motion trajectories, which is insensitive to noise, invariant to rigid transformation and scale. The RRV descriptor is then applied to characterize motions of a human body skeleton modeled as articulated interconnections of multiple rigid bodies. To show the descriptive ability of our RRV descriptor, we explore its potentials and applications in different rigid body motion recognition tasks. The experimental results on benchmark datasets demonstrate that our RRV descriptor learning discriminative motion patterns can achieve superior results for various recognition tasks.

A 3D Laser Profiling System for Rail Surface Defect Detection

Rail surface defects such as the abrasion, scratch and peeling often cause damages to the train wheels and rail bearings. An efficient and accurate detection of rail defects is of vital importance for the safety of railway transportation. In the past few decades, automatic rail defect detection has been studied; however, most developed methods use optic-imaging techniques to collect the rail surface data and are still suffering from a high false recognition rate. In this paper, a novel 3D laser profiling system (3D-LPS) is proposed, which integrates a laser scanner, odometer, inertial measurement unit (IMU) and global position system (GPS) to capture the rail surface profile data. For automatic defect detection, first, the deviation between the measured profile and a standard rail model profile is computed for each laser-imaging profile, and the points with large deviations are marked as candidate defect points. Specifically, an adaptive iterative closest point (AICP) algorithm is proposed to register the point sets of the measured profile with the standard rail model profile, and the registration precision is improved to the sub-millimeter level. Second, all of the measured profiles are combined together to form the rail surface through a high-precision positioning process with the IMU, odometer and GPS data. Third, the candidate defect points are merged into candidate defect regions using the K-means clustering. At last, the candidate defect regions are classified by a decision tree classifier. Experimental results demonstrate the effectiveness of the proposed laser-profiling system in rail surface defect detection and classification.