Deep Learning-Based Thermal Image Analysis for Pavement Defect Detection and Classification Considering Complex Pavement Conditions

Photothermal Radiometry Data Analysis by Using Machine Learning

Photothermal techniques are infrared remote sensing techniques that have been used for biomedical applications, as well as industrial non-destructive testing (NDT). Machine learning is a branch of artificial intelligence, which includes a set of algorithms for learning from past data and analyzing new data, without being explicitly programmed to do so. In this paper, we first review the latest development of machine learning and its applications in photothermal techniques. Next, we present our latest work on machine learning for data analysis in opto-thermal transient emission radiometry (OTTER), which is a type of photothermal technique that has been extensively used in skin hydration, skin hydration depth profiles, skin pigments, as well as topically applied substances and skin penetration measurements. We have investigated different algorithms, such as random forest regression, gradient boosting regression, support vector machine (SVM) regression, and partial least squares regression, as well as deep learning neural network regression. We first introduce the theoretical background, then illustrate its applications with experimental results.

Automatic Damage Detection of Pavement through DarkNet Analysis of Digital, Infrared, and Multi-Spectral Dynamic Imaging Images

It is important to maintain the safety of road driving by automatically performing a series of processes to automatically measure and repair damage to the road pavement. However, road pavements include not only damages such as longitudinal cracks, transverse cracks, alligator cracks, and potholes, but also various elements such as manholes, road marks, oil marks, shadows, and joints. Therefore, in order to separate categories that exist in various road pavements, in this paper, 13,500 digital, IR, and MSX images were collected and nine categories were automatically classified by DarkNet. The DarkNet classification accuracies of digital images, IR images, and MSX images are 97.4%, 80.1%, and 91.1%, respectively. The MSX image is a enhanced image of the IR image and showed an average of 6% lower accuracy than the digital image but an average of 11% higher accuracy than the IR image. Therefore, MSX images can play a complementary role if DarkNet classification is performed together with digital images. In this paper, a method for detecting the directionality of each crack through a two-dimensional wavelet transform is presented, and this result can contribute to future research on detecting cracks in pavements.

Automated Road Defect and Anomaly Detection for Traffic Safety: A Systematic Review

Recently, there has been a substantial increase in the development of sensor technology. As enabling factors, computer vision (CV) combined with sensor technology have made progress in applications intended to mitigate high rates of fatalities and the costs of traffic-related injuries. Although past surveys and applications of CV have focused on subareas of road hazards, there is yet to be one comprehensive and evidence-based systematic review that investigates CV applications for Automated Road Defect and Anomaly Detection (ARDAD). To present ARDAD's state-of-the-art, this systematic review is focused on determining the research gaps, challenges, and future implications from selected papers (N = 116) between 2000 and 2023, relying primarily on Scopus and Litmaps services. The survey presents a selection of artefacts, including the most popular open-access datasets (D = 18), research and technology trends that with reported performance can help accelerate the application of rapidly advancing sensor technology in ARDAD and CV. The produced survey artefacts can assist the scientific community in further improving traffic conditions and safety.

The Development of a Cost-Effective Imaging Device Based on Thermographic Technology

Thermal vision-based devices are nowadays used in a number of industries, ranging from the automotive industry, surveillance, navigation, fire detection, and rescue missions to precision agriculture. This work describes the development of a low-cost imaging device based on thermographic technology. The proposed device uses a miniature microbolometer module, a 32-bit ARM microcontroller, and a high-accuracy ambient temperature sensor. The developed device is capable of enhancing RAW high dynamic thermal readings obtained from the sensor using a computationally efficient image enhancement algorithm and presenting its visual result on the integrated OLED display. The choice of microcontroller, rather than the alternative System on Chip (SoC), offers almost instantaneous power uptime and extremely low power consumption while providing real-time imaging of an environment. The implemented image enhancement algorithm employs the modified histogram equalization, where the ambient temperature sensor helps the algorithm enhance both background objects near ambient temperature and foreground objects (humans, animals, and other heat sources) that actively emit heat. The proposed imaging device was evaluated on a number of environmental scenarios using standard no-reference image quality measures and comparisons against the existing state-of-the-art enhancement algorithms. Qualitative results obtained from the survey of 11 subjects are also provided. The quantitative evaluations show that, on average, images acquired by the developed camera provide better perception quality in 75% of tested cases. According to qualitative evaluations, images acquired by the developed camera provide better perception quality in 69% of tested cases. The obtained results verify the usability of the developed low-cost device for a range of applications where thermal imaging is needed.

Deep Learning Based Infrared Thermal Image Analysis of Complex Pavement Defect Conditions Considering Seasonal Effect

Deep learning techniques underpinned by extensive data sources encompassing complex pavement features have proven effective in early pavement damage detection. With pavement features exhibiting temperature variation, inexpensive infra-red imaging technology in combination with deep learning techniques can detect pavement damages effectively. Previous experiments based on pavement data captured during summer sunny conditions when subjected to SA-ResNet deep learning architecture technique demonstrated 96.47% prediction accuracy. This paper has extended the same deep learning approach to a different dataset comprised of images captured during winter sunny conditions to compare the prediction accuracy, sensitivity and recall score with summer conditions. The results suggest that irrespective of the prevalent weather season, the proposed deep learning algorithm categorises pavement features around 92% accurately (95.18% in summer and 91.67% in winter conditions), suggesting the beneficial replacement of one image type with other. The data captured in sunny conditions during summer and winter show prediction accuracies of DC = 96.47% > MSX = 95.24% > IR-T = 93.83% and DC = 94.14% > MSX = 90.69% > IR-T = 90.173%, respectively. DC images demonstrated a sensitivity of 96.47% and 94.20% for summer and winter conditions, respectively, to demonstrate that reliable categorisation is possible with deep learning techniques irrespective of the weather season. However, summer conditions showing better overall prediction accuracy than winter conditions suggests that inexpensive IR-T imaging cameras with medium resolution levels can still be an economical solution, unlike expensive alternate options, but their usage has to be limited to summer sunny conditions.

Automatic Pavement Defect Detection and Classification Using RGB-Thermal Images Based on Hierarchical Residual Attention Network

A convolutional neural network based on an improved residual structure is proposed to implement a lightweight classification model for the recognition of complex pavement conditions, which uses RGB-thermal as input and embeds an attention module to adjust the spatial, as well as channel, information of the images. The best prediction accuracy of the proposed model is 98.88%, while the RGB-thermal is used as input and an attention mechanism is used. The attention mechanism increases the attention to detail of the image and regulates the use of image channels, which enhances the final performance of the model. It is also compared with state-of-the-art (SOTA) deep learning models, indicating our model has fewer parameters, shorter training time, and higher recognition accuracy compared to existing image classification models. A visualization method incorporating gradient-weighted class activation mapping (Grad-CAM) is proposed to analyze the classification results, comparing the data the model learns from the images under different input data.

Road Condition Monitoring Using Smart Sensing and Artificial Intelligence: A Review

Road condition monitoring (RCM) has been a demanding strategic research area in maintaining a large network of transport infrastructures. With advancements in computer vision and data mining techniques along with high computing resources, several innovative pavement distress evaluation systems have been developed in recent years. The majority of these technologies employ next-generation distributed sensors and vision-based artificial intelligence (AI) methodologies to evaluate, classify and localize pavement distresses using the measured data. This paper presents an exhaustive and systematic literature review of these technologies in RCM that have been published from 2017–2022 by utilizing next-generation sensors, including contact and noncontact measurements. The various methodologies and innovative contributions of the existing literature reviewed in this paper, together with their limitations, promise a futuristic insight for researchers and transport infrastructure owners. The decisive role played by smart sensors and data acquisition platforms, such as smartphones, drones, vehicles integrated with non-intrusive sensors, such as RGB, and thermal cameras, lasers and GPR sensors in the performance of the system are also highlighted. In addition to sensing, a discussion on the prevalent challenges in the development of AI technologies as well as potential areas for further exploration paves the way for an all-inclusive and well-directed futuristic research on RCM.

A Fast Inference Vision Transformer for Automatic Pavement Image Classification and Its Visual Interpretation Method

Traditional automatic pavement distress detection methods using convolutional neural networks (CNNs) require a great deal of time and resources for computing and are poor in terms of interpretability. Therefore, inspired by the successful application of Transformer architecture in natural language processing (NLP) tasks, a novel Transformer method called LeViT was introduced for automatic asphalt pavement image classification. LeViT consists of convolutional layers, transformer stages where Multi-layer Perception (MLP) and multi-head self-attention blocks alternate using the residual connection, and two classifier heads. To conduct the proposed methods, three different sources of pavement image datasets and pre-trained weights based on ImageNet were attained. The performance of the proposed model was compared with six state-of-the-art (SOTA) deep learning models. All of them were trained based on transfer learning strategy. Compared to the tested SOTA methods, LeViT has less than 1/8 of the parameters of the original Vision Transformer (ViT) and 1/2 of ResNet and InceptionNet. Experimental results show that after training for 100 epochs with a 16 batch-size, the proposed method acquired 91.56% accuracy, 91.72% precision, 91.56% recall, and 91.45% F1-score in the Chinese asphalt pavement dataset and 99.17% accuracy, 99.19% precision, 99.17% recall, and 99.17% F1-score in the German asphalt pavement dataset, which is the best performance among all the tested SOTA models. Moreover, it shows superiority in inference speed (86 ms/step), which is approximately 25% of the original ViT method and 80% of some prevailing CNN-based models, including DenseNet, VGG, and ResNet. Overall, the proposed method can achieve competitive performance with fewer computation costs. In addition, a visualization method combining Grad-CAM and Attention Rollout was proposed to analyze the classification results and explore what has been learned in every MLP and attention block of LeViT, which improved the interpretability of the proposed pavement image classification model.