Context-Unsupervised Adversarial Network for Video Sensors

Foreground object segmentation is a crucial first step for surveillance systems based on networks of video sensors. This problem in the context of dynamic scenes has been widely explored in the last two decades, but it still has open research questions due to challenges such as strong shadows, background clutter and illumination changes. After years of solid work based on statistical background pixel modeling, most current proposals use convolutional neural networks (CNNs) either to model the background or to make the foreground/background decision. Although these new techniques achieve outstanding results, they usually require specific training for each scene, which is unfeasible if we aim at designing software for embedded video systems and smart cameras. Our approach to the problem does not require specific context or scene training, and thus no manual labeling. We propose a network for a refinement step on top of conventional state-of-the-art background subtraction systems. By using a statistical technique to produce a rough mask, we do not need to train the network for each scene. The proposed method can take advantage of the specificity of the classic techniques, while obtaining the highly accurate segmentation that a deep learning system provides. We also show the advantage of using an adversarial network to improve the generalization ability of the network and produce more consistent results than an equivalent non-adversarial network. The results provided were obtained by training the network on a common database, without fine-tuning for specific scenes. Experiments on the unseen part of the CDNet database provided 0.82 a F-score, and 0.87 was achieved for LASIESTA databases, which is a database unrelated to the training one. On this last database, the results outperformed by 8.75% those available in the official table. The results achieved for CDNet are well above those of the methods not based on CNNs, and according to the literature, among the best for the context-unsupervised CNNs systems.

SLAM-OR: Simultaneous Localization, Mapping and Object Recognition Using Video Sensors Data in Open Environments from the Sparse Points Cloud

In this paper, we propose a novel approach that enables simultaneous localization, mapping (SLAM) and objects recognition using visual sensors data in open environments that is capable to work on sparse data point clouds. In the proposed algorithm the ORB-SLAM uses the current and previous monocular visual sensors video frame to determine observer position and to determine a cloud of points that represent objects in the environment, while the deep neural network uses the current frame to detect and recognize objects (OR). In the next step, the sparse point cloud returned from the SLAM algorithm is compared with the area recognized by the OR network. Because each point from the 3D map has its counterpart in the current frame, therefore the filtration of points matching the area recognized by the OR algorithm is performed. The clustering algorithm determines areas in which points are densely distributed in order to detect spatial positions of objects detected by OR. Then by using principal component analysis (PCA)-based heuristic we estimate bounding boxes of detected objects. The image processing pipeline that uses sparse point clouds generated by SLAM in order to determine positions of objects recognized by deep neural network and mentioned PCA heuristic are main novelties of our solution. In contrary to state-of-the-art approaches, our algorithm does not require any additional calculations like generation of dense point clouds for objects positioning, which highly simplifies the task. We have evaluated our research on large benchmark dataset using various state-of-the-art OR architectures (YOLO, MobileNet, RetinaNet) and clustering algorithms (DBSCAN and OPTICS) obtaining promising results. Both our source codes and evaluation data sets are available for download, so our results can be easily reproduced.

Quantitative and Qualitative Running Gait Analysis through an Innovative Video-Based Approach

Quantitative and qualitative running gait analysis allows the early identification and the longitudinal monitoring of gait abnormalities linked to running-related injuries. A promising calibration- and marker-less video sensor-based technology (i.e., Graal), recently validated for walking gait, may also offer a time- and cost-efficient alternative to the gold-standard methods for running. This study aim was to ascertain the validity of an improved version of Graal for quantitative and qualitative analysis of running. In 33 healthy recreational runners (mean age 41 years), treadmill running at self-selected submaximal speed was simultaneously evaluated by a validated photosensor system (i.e., Optogait—the reference methodology) and by the video analysis of a posterior 30-fps video of the runner through the optimized version of Graal. Graal is video analysis software that provides a spectral analysis of the brightness over time for each pixel of the video, in order to identify its frequency contents. The two main frequencies of variation of the pixel’s brightness (i.e., F1 and F2) correspond to the two most important frequencies of gait (i.e., stride frequency and cadence). The Optogait system recorded step length, cadence, and its variability (vCAD, a traditional index of gait quality). Graal provided a direct measurement of F2 (reflecting cadence), an indirect measure of step length, and two indexes of global gait quality (harmony and synchrony index). The correspondence between quantitative indexes (Cadence vs. F2 and step length vs. Graal step length) was tested via paired t-test, correlations, and Bland–Altman plots. The relationship between qualitative indexes (vCAD vs. Harmony and Synchrony Index) was investigated by correlation analysis. Cadence and step length were, respectively, not significantly different from and highly correlated with F2 (1.41 Hz ± 0.09 Hz vs. 1.42 Hz ± 0.08 Hz, p = 0.25, r² = 0.81) and Graal step length (104.70 cm ± 013.27 cm vs. 107.56 cm ± 13.67 cm, p = 0.55, r² = 0.98). Bland–Altman tests confirmed a non-significant bias and small imprecision between methods for both parameters. The vCAD was 1.84% ± 0.66%, and it was significantly correlated with neither the Harmony nor the Synchrony Index (0.21 ± 0.03, p = 0.92, r² = 0.00038; 0.21 ± 0.96, p = 0.87, r² = 0.00122). These findings confirm the validity of the optimized version of Graal for the measurement of quantitative indexes of gait. Hence, Graal constitutes an extremely time- and cost-efficient tool suitable for quantitative analysis of running. However, its validity for qualitative running gait analysis remains inconclusive and will require further evaluation in a wider range of absolute and relative running intensities in different individuals.

Distributed coding/decoding complexity in video sensor networks

Video Sensor Networks (VSNs) are recent communication infrastructures used to capture and transmit dense visual information from an application context. In such large scale environments which include video coding, transmission and display/storage, there are several open problems to overcome in practical implementations. This paper addresses the most relevant challenges posed by VSNs, namely stringent bandwidth usage and processing time/power constraints. In particular, the paper proposes a novel VSN architecture where large sets of visual sensors with embedded processors are used for compression and transmission of coded streams to gateways, which in turn transrate the incoming streams and adapt them to the variable complexity requirements of both the sensor encoders and end-user decoder terminals. Such gateways provide real-time transcoding functionalities for bandwidth adaptation and coding/decoding complexity distribution by transferring the most complex video encoding/decoding tasks to the transcoding gateway at the expense of a limited increase in bit rate. Then, a method to reduce the decoding complexity, suitable for system-on-chip implementation, is proposed to operate at the transcoding gateway whenever decoders with constrained resources are targeted. The results show that the proposed method achieves good performance and its inclusion into the VSN infrastructure provides an additional level of complexity control functionality.