Behavioral Pedestrian Tracking Using a Camera and LiDAR Sensors on a Moving Vehicle

Object Detection, Recognition, and Tracking Algorithms for ADASs-A Study on Recent Trends

Advanced driver assistance systems (ADASs) are becoming increasingly common in modern-day vehicles, as they not only improve safety and reduce accidents but also aid in smoother and easier driving. ADASs rely on a variety of sensors such as cameras, radars, lidars, and a combination of sensors, to perceive their surroundings and identify and track objects on the road. The key components of ADASs are object detection, recognition, and tracking algorithms that allow vehicles to identify and track other objects on the road, such as other vehicles, pedestrians, cyclists, obstacles, traffic signs, traffic lights, etc. This information is then used to warn the driver of potential hazards or used by the ADAS itself to take corrective actions to avoid an accident. This paper provides a review of prominent state-of-the-art object detection, recognition, and tracking algorithms used in different functionalities of ADASs. The paper begins by introducing the history and fundamentals of ADASs followed by reviewing recent trends in various ADAS algorithms and their functionalities, along with the datasets employed. The paper concludes by discussing the future of object detection, recognition, and tracking algorithms for ADASs. The paper also discusses the need for more research on object detection, recognition, and tracking in challenging environments, such as those with low visibility or high traffic density.

Analyzing Factors Influencing Situation Awareness in Autonomous Vehicles-A Survey

Autonomous driving of higher automation levels asks for optimal execution of critical maneuvers in all environments. A crucial prerequisite for such optimal decision-making instances is accurate situation awareness of automated and connected vehicles. For this, vehicles rely on the sensory data captured from onboard sensors and information collected through V2X communication. The classical onboard sensors exhibit different capabilities and hence a heterogeneous set of sensors is required to create better situation awareness. Fusion of the sensory data from such a set of heterogeneous sensors poses critical challenges when it comes to creating an accurate environment context for effective decision-making in AVs. Hence this exclusive survey analyses the influence of mandatory factors like data pre-processing preferably data fusion along with situation awareness toward effective decision-making in the AVs. A wide range of recent and related articles are analyzed from various perceptive, to pick the major hiccups, which can be further addressed to focus on the goals of higher automation levels. A section of the solution sketch is provided that directs the readers to the potential research directions for achieving accurate contextual awareness. To the best of our knowledge, this survey is uniquely positioned for its scope, taxonomy, and future directions.

Development of a Particle Filter-Based Path Tracking Algorithm of Autonomous Trucks with a Single Steering and Driving Module Using a Monocular Camera

Recently, in various fields, research into the path tracking of autonomous vehicles and automated guided vehicles has been conducted to improve worker safety, convenience, and work efficiency. For path tracking of various systems applied to autonomous driving technology, it is necessary to recognize the surrounding environment, determine technology accordingly, and develop control methods. Various sensors and artificial-intelligence-based perception methods have limitations in that they must learn a large amount of data. Therefore, a particle-filter-based path tracking algorithm using a monocular camera was used for the recognition of target RGB. The path tracking errors were calculated and a linear-quadratic-regulator-based desired steering angle were derived. The autonomous trucks were steered and driven using a pulse-width-modulation-based steering and driving motor. Based on an autonomous truck with a single steering and driving module, it was verified that the path tracking could be used in three evaluation scenarios. To compare the LQR-based path tracking control performance proposed in this paper, an elliptical path tracking scenario using a conventional sliding mode control with robust control performance was performed. The results show that the RMS of the lateral preview error of the SMC was approximately 18% larger than that of the LQR-based method.

Cyclist Orientation Estimation Using LiDAR Data

It is crucial for an autonomous vehicle to predict cyclist behavior before decision-making. When a cyclist is on real traffic roads, his or her body orientation indicates the current moving directions, and his or her head orientation indicates his or her intention for checking the road situation before making next movement. Therefore, estimating the orientation of cyclist's body and head is an important factor of cyclist behavior prediction for autonomous driving. This research proposes to estimate cyclist orientation including both body and head orientation using deep neural network with the data from Light Detection and Ranging (LiDAR) sensor. In this research, two different methods are proposed for cyclist orientation estimation. The first method uses 2D images to represent the reflectivity, ambient and range information collected by LiDAR sensor. At the same time, the second method uses 3D point cloud data to represent the information collected from LiDAR sensor. The two proposed methods adopt a model ResNet50, which is a 50-layer convolutional neural network, for orientation classification. Hence, the performances of two methods are compared to achieve the most effective usage of LiDAR sensor data in cyclist orientation estimation. This research developed a cyclist dataset, which includes multiple cyclists with different body and head orientations. The experimental results showed that a model that uses 3D point cloud data has better performance for cyclist orientation estimation compared to the model that uses 2D images. Moreover, in the 3D point cloud data-based method, using reflectivity information has a more accurate estimation result than using ambient information.

Online Self-Calibration of 3D Measurement Sensors Using a Voxel-Based Network

Multi-sensor fusion is important in the field of autonomous driving. A basic prerequisite for multi-sensor fusion is calibration between sensors. Such calibrations must be accurate and need to be performed online. Traditional calibration methods have strict rules. In contrast, the latest online calibration methods based on convolutional neural networks (CNNs) have gone beyond the limits of the conventional methods. We propose a novel algorithm for online self-calibration between sensors using voxels and three-dimensional (3D) convolution kernels. The proposed approach has the following features: (1) it is intended for calibration between sensors that measure 3D space; (2) the proposed network is capable of end-to-end learning; (3) the input 3D point cloud is converted to voxel information; (4) it uses five networks that process voxel information, and it improves calibration accuracy through iterative refinement of the output of the five networks and temporal filtering. We use the KITTI and Oxford datasets to evaluate the calibration performance of the proposed method. The proposed method achieves a rotation error of less than 0.1° and a translation error of less than 1 cm on both the KITTI and Oxford datasets.

Automatic Extrinsic Calibration of 3D LIDAR and Multi-Cameras Based on Graph Optimization

In recent years, multi-sensor fusion technology has made enormous progress in 3D reconstruction, surveying and mapping, autonomous driving, and other related fields, and extrinsic calibration is a necessary condition for multi-sensor fusion applications. This paper proposes a 3D LIDAR-to-camera automatic calibration framework based on graph optimization. The system can automatically identify the position of the pattern and build a set of virtual feature point clouds, and can simultaneously complete the calibration of the LIDAR and multiple cameras. To test this framework, a multi-sensor system is formed using a mobile robot equipped with LIDAR, monocular and binocular cameras, and the pairwise calibration of LIDAR with two cameras is evaluated quantitatively and qualitatively. The results show that this method can produce more accurate calibration results than the state-of-the-art method. The average error on the camera normalization plane is 0.161 mm, which outperforms existing calibration methods. Due to the introduction of graph optimization, the original point cloud is also optimized while optimizing the external parameters between the sensors, which can effectively correct the errors caused during data collection, so it is also robust to bad data.

Target-based calibration of 3D LiDAR and binocular camera on unmanned vehicles

Unmanned vehicles need to make a comprehensive perception of the surrounding environmental information during driving. Perception of automotive information is one of the important information. In the field of automotive perception, the stere-vision of car-detection plays a vital role and stere-vision can calculate the length, width, and height of a car, making the car more specific. However, under the existing technology, it is impossible to obtain accurate detection in a complex environment by relying on a single sensor. Therefore, it is particularly important to study the complex sensing technology based on multi-sensor fusion. This paper proposes a method based on feature point pair matching. Two rectangular planks are used to extract the 3D point cloud of the edge of the board in stereo vision and LiDAR coordinate systems, which is then used to obtain the corner coordinates. Finally, the Kabsch algorithm is used to solve the coordinate transformation between the paired feature points, and a clustering method is used to remove outliers from the multiple measurements and obtain the average value. By setting up an experiment, this method can be implemented on the Nvidia Jetson Tx2 embedded development board, and accurate registration parameters can be obtained, thus verifying the theoretical method’s feasibility. It finishes calibration of the LiDAR and binocular camera based on present methods. The result shows that, it can reduce the effects of noise, and acquire registration parameters accurately of LiDAR and cameras. Compared with the approved method of the same type, our proposed method has less error and good practical value.

Driving Environment Perception Based on the Fusion of Vehicular Wireless Communications and Automotive Remote Sensors

Driving environment perception for automated vehicles is typically achieved by the use of automotive remote sensors such as radars and cameras. A vehicular wireless communication system can be viewed as a new type of remote sensor that plays a central role in connected and automated vehicles (CAVs), which are capable of sharing information with each other and also with the surrounding infrastructure. In this paper, we present the design and implementation of driving environment perception based on the fusion of vehicular wireless communications and automotive remote sensors. A track-to-track fusion of high-level sensor data and vehicular wireless communication data was performed to accurately and reliably locate the remote target in the vehicle surroundings and predict the future trajectory. The proposed approach was implemented and evaluated in vehicle tests conducted at a proving ground. The experimental results demonstrate that using vehicular wireless communications in conjunction with the on-board sensors enables improved perception of the surrounding vehicle located at varying longitudinal and lateral distances. The results also indicate that vehicle future trajectory and potential crash involvement can be reliably predicted with the proposed system in different cut-in driving scenarios.

Multiple-Joint Pedestrian Tracking Using Periodic Models

Estimating accurate positions of multiple pedestrians is a critical task in robotics and autonomous cars. We propose a tracker based on typical human motion patterns to track multiple pedestrians. This paper assumes that the legs' reflection and extension angles are approximately changing periodically during human motion. A Fourier series is fitted in order to describe the moving, such as describing the position and velocity of the hip, knee, and ankle. Our tracker receives the position of the ankle, knee, and hip as measurements. As a proof of concept, we compare our tracker with state-of-the-art methods. The proposed models have been validated by experimental data, the Human Gait Database (HuGaDB), and the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) tracking benchmark. The results indicate that our tracker is able to estimate the reflection and extension angles with a precision of 90.97%. Moreover, the comparison shows that the tracking precision increases up to 1.3% with the proposed tracker when compared to a constant velocity based tracker.

3D car-detection based on a Mobile Deep Sensor Fusion Model and real-scene applications

Unmanned vehicles need to make a comprehensive perception of the surrounding environmental information during driving. Perception of automotive information is of significance. In the field of automotive perception, the sterevision of car-detection plays a vital role and sterevision can calculate the length, width, and height of a car, making the car more specific. However, under the existing technology, it is impossible to obtain accurate detection in a complex environment by relying on a single sensor. Therefore, it is particularly important to study the complex sensing technology based on multi-sensor fusion. Recently, with the development of deep learning in the field of vision, a mobile sensor-fusion method based on deep learning is proposed and applied in this paper——Mobile Deep Sensor Fusion Model (MDSFM). The content of this article is as follows. It does a data processing that projects 3D data to 2D data, which can form a dataset suitable for the model, thereby training data more efficiently. In the modules of LiDAR, it uses a revised squeezeNet structure to lighten the model and reduce parameters. In the modules of cameras, it uses the improved design of detecting module in R-CNN with a Mobile Spatial Attention Module (MSAM). In the fused part, it uses a dual-view deep fusing structure. And then it selects images from the KITTI’s datasets for validation to test this model. Compared with other recognized methods, it shows that our model has a fairly good performance. Finally, it implements a ROS program on the experimental car and our model is in good condition. The result shows that it can improve performance of detecting easy cars significantly through MDSFM. It increases the quality of the detected data and improves the generalized ability of car-detection model. It improves contextual relevance and preserves background information. It remains stable in driverless environments. It is applied in the realistic scenario and proves that the model has a good practical value.

Cooperative Multi-Sensor Tracking of Vulnerable Road Users in the Presence of Missing Detections

This paper presents a vulnerable road user (VRU) tracking algorithm capable of handling noisy and missing detections from heterogeneous sensors. We propose a cooperative fusion algorithm for matching and reinforcing of radar and camera detections using their proximity and positional uncertainty. The belief in the existence and position of objects is then maximized by temporal integration of fused detections by a multi-object tracker. By switching between observation models, the tracker adapts to the detection noise characteristics making it robust to individual sensor failures. The main novelty of this paper is an improved imputation sampling function for updating the state when detections are missing. The proposed function uses a likelihood without association that is conditioned on the sensor information instead of the sensor model. The benefits of the proposed solution are two-fold: firstly, particle updates become computationally tractable and secondly, the problem of imputing samples from a state which is predicted without an associated detection is bypassed. Experimental evaluation shows a significant improvement in both detection and tracking performance over multiple control algorithms. In low light situations, the cooperative fusion outperforms intermediate fusion by as much as 30%, while increases in tracking performance are most significant in complex traffic scenes.

Autonomous Shuttle Bus for Public Transportation: A Review

The rapid evolution of autonomous technology in the field of automotive and information technology (IT) has made it possible to implement autonomous vehicles (AVs) for public passenger transport. Although the shuttle bus transport capacities currently in use are low (maximum 15 people), the use of these transport units in large urban agglomerations is beneficial for society. The current paper is written to review the current AV implementation with respect to shuttle buses with its direct implications in their scientific evolution, with direct links to the legal and social aspects of public transportation all over the world. A critical aspect that is presented in the paper is the legal framework of autonomous driving, which is extremely uneven around the globe, with the direct impact of autonomous shuttle bus exploitation. As the legislation on AVs presents some shortcomings in the approval, registration, and public road implementation of these vehicles, many of the world’s major cities have found ways to integrate them into testing programs, establishing the basis for future comprehensive legislative measures in this highly dynamic scientific domain. The current technological solutions adopted by several autonomous shuttle bus producers will be presented with an exhaustive overview of each major component. The aspects of the control algorithm, with its complicated layers of security and perturbance factors, will be explained in detail. Thus, in some countries/cities, autonomous shuttle buses have been implemented on less-traveled routes where they can travel at speeds up to 25 km/h without hindering the public’s circulation, such as university campuses, industrial areas, airports, and sports bases. Some countries/cities use autonomous shuttle buses for pilot programs related to passenger transport, while others use them in postal transport and others for scientific purposes. In all of these situations, the first step in autonomous driving has been taken. The paper also makes an evaluation of the social factors that are a consequence of the mass introduction of autonomous driving as a means of public transportation. Autonomous shuttle buses are becoming a part of everyday life in big cities. Their acceptance as a strategic means of transport depends on their efficiency in daily services; through its efficiency, this means of transport will become a game-changer once its benefits become not only known but experienced by a large number of users.

Efficient CORDIC Iteration Design of LiDAR Sensors' Point-Cloud Map Reconstruction Technology

In this paper, we propose an efficient COordinate Rotation DIgital Computer (CORDIC) iteration circuit design for Light Detection and Ranging (LiDAR) sensors. A novel CORDIC architecture that achieves the goal of pre-selecting angles and reduces the number of iterations is presented for LiDAR sensors. The value of the trigonometric functions can be found in seven rotations regardless of the number of input N digits. The number of iterations are reduced by more than half. The experimental results show the similarity value to be all 1 and prove that the LiDAR decoded packet results are exactly the same as the ground truth. The total chip area is 1.93 mm × 1.93 mm and the core area is 1.32 mm × 1.32 mm, separately. The number of logic gates is 129,688. The designed chip only takes 0.012 ms and 0.912 ms to decode a packet and a 3D frame of LiDAR sensors, respectively. The throughput of the chip is 8.2105 × 108 bits/sec. The average power consumption is 237.34 mW at a maximum operating frequency of 100 MHz. This design can not only reduce the number of iterations and the computing time but also reduce the chip area. This paper provides an efficient CORDIC iteration design and solution for LiDAR sensors to reconstruct the point-cloud map for autonomous vehicles.

Automatic Multi-Camera Extrinsic Parameter Calibration Based on Pedestrian Torsors

Extrinsic camera calibration is essential for any computer vision task in a camera network. Typically, researchers place a calibration object in the scene to calibrate all the cameras in a camera network. However, when installing cameras in the field, this approach can be costly and impractical, especially when recalibration is needed. This paper proposes a novel, accurate and fully automatic extrinsic calibration framework for camera networks with partially overlapping views. The proposed method considers the pedestrians in the observed scene as the calibration objects and analyzes the pedestrian tracks to obtain extrinsic parameters. Compared to the state of the art, the new method is fully automatic and robust in various environments. Our method detect human poses in the camera images and then models walking persons as vertical sticks. We apply a brute-force method to determines the correspondence between persons in multiple camera images. This information along with 3D estimated locations of the top and the bottom of the pedestrians are then used to compute the extrinsic calibration matrices. We also propose a novel method to calibrate the camera network by only using the top and centerline of the person when the bottom of the person is not available in heavily occluded scenes. We verified the robustness of the method in different camera setups and for both single and multiple walking people. The results show that the triangulation error of a few centimeters can be obtained. Typically, it requires less than one minute of observing the walking people to reach this accuracy in controlled environments. It also just takes a few minutes to collect enough data for the calibration in uncontrolled environments. Our proposed method can perform well in various situations such as multi-person, occlusions, or even at real intersections on the street.

Rapid Motion Segmentation of LiDAR Point Cloud Based on a Combination of Probabilistic and Evidential Approaches for Intelligent Vehicles

Point clouds from light detecting and ranging (LiDAR) sensors represent increasingly important information for environmental object detection and classification of automated and intelligent vehicles. Objects in the driving environment can be classified as either dynamic or static depending on their movement characteristics. A LiDAR point cloud is also segmented into dynamic and static points based on the motion properties of the measured objects. The segmented motion information of a point cloud can be useful for various functions in automated and intelligent vehicles. This paper presents a fast motion segmentation algorithm that segments a LiDAR point cloud into dynamic and static points in real-time. The segmentation algorithm classifies the motion of the latest point cloud based on the LiDAR’s laser beam characteristics and the geometrical relationship between consecutive LiDAR point clouds. To accurately and reliably estimate the motion state of each LiDAR point considering the measurement uncertainty, both probability theory and evidence theory are employed in the segmentation algorithm. The probabilistic and evidential algorithm segments the point cloud into three classes: dynamic, static, and unknown. Points are placed in the unknown class when LiDAR point cloud is not sufficient for motion segmentation. The point motion segmentation algorithm was evaluated quantitatively and qualitatively through experimental comparisons with previous motion segmentation methods.

Tracking and Simulating Pedestrian Movements at Intersections Using Unmanned Aerial Vehicles

For a city to be livable and walkable is the ultimate goal of future cities. However, conflicts among pedestrians, vehicles, and cyclists at traffic intersections are becoming severe in high-density urban transportation areas, especially in China. Correspondingly, the transit time at intersections is becoming prolonged, and pedestrian safety is becoming endangered. Simulating pedestrian movements at complex traffic intersections is necessary to optimize the traffic organization. We propose an unmanned aerial vehicle (UAV)-based method for tracking and simulating pedestrian movements at intersections. Specifically, high-resolution videos acquired by a UAV are used to recognize and position moving targets, including pedestrians, cyclists, and vehicles, using the convolutional neural network. An improved social force-based motion model is proposed, considering the conflicts among pedestrians, cyclists, and vehicles. In addition, maximum likelihood estimation is performed to calibrate an improved social force model. UAV videos of intersections in Shenzhen are analyzed to demonstrate the performance of the presented approach. The results demonstrate that the proposed social force-based motion model can effectively simulate the movement of pedestrians and cyclists at road intersections. The presented approach provides an alternative method to track and simulate pedestrian movements, thus benefitting the organization of pedestrian flow and traffic signals controlling the intersections.

A New Water Level Measurement Method Combining Infrared Sensors and Floats for Applications on Laboratory Scale Channel under Unsteady Flow Regime

In this paper, we studied water transport under an unsteady flow regime in an experimental channel (4 m in length; 3 cm in width). Our experiments implicated some measuring requirements, specifically, a water level (WL) detection technique that is able to measure WL in a range of 2 cm with a precision of 1 mm. The existing WL detection techniques could not meet our measurement requirements. Therefore, we propose a new measurement method that combines two approaches: An "old" water contact technique (float) with a "new" remote non-contact technique (infrared sensor). We used an extruded polystyrene (XPS Foam) that needed some adequate treatment before using it as float in experimental measurements. The combination of IR-sensors with treated float foam lead to a sensitive measurement method that is able to detect flat and sharp flow signals, as well as highly dynamic variations of water surface level. Based on the experimental measurements of WL and outflow at the channel output, we deduced a loop rating curve that is suitable with a power law adjustment. The new measurement method could be extended to larger scale applications like rivers and more complicated cross section geometry of irregular shape.