Visual-Based Defect Detection and Classification Approaches for Industrial Applications-A SURVEY

GM-DETR: Research on a Defect Detection Method Based on Improved DETR

Defect detection is an indispensable part of the industrial intelligence process. The introduction of the DETR model marked the successful application of a transformer for defect detection, achieving true end-to-end detection. However, due to the complexity of defective backgrounds, low resolutions can lead to a lack of image detail control and slow convergence of the DETR model. To address these issues, we proposed a defect detection method based on an improved DETR model, called the GM-DETR. We optimized the DETR model by integrating GAM global attention with CNN feature extraction and matching features. This optimization process reduces the defect information diffusion and enhances the global feature interaction, improving the neural network's performance and ability to recognize target defects in complex backgrounds. Next, to filter out unnecessary model parameters, we proposed a layer pruning strategy to optimize the decoding layer, thereby reducing the model's parameter count. In addition, to address the issue of poor sensitivity of the original loss function to small differences in defect targets, we replaced the L1 loss in the original loss function with MSE loss to accelerate the network's convergence speed and improve the model's recognition accuracy. We conducted experiments on a dataset of road pothole defects to further validate the effectiveness of the GM-DETR model. The results demonstrate that the improved model exhibits better performance, with an increase in average precision of 4.9% (mAP@0.5), while reducing the parameter count by 12.9%.

Internal thread defect detection system based on multi-vision

In the realm of industrial inspection, the precise assessment of internal thread quality is crucial for ensuring mechanical integrity and safety. However, challenges such as limited internal space, inadequate lighting, and complex geometry significantly hinder high-precision inspection. In this study, we propose an innovative automated internal thread detection scheme based on machine vision, aimed at addressing the time-consuming and inefficient issues of traditional manual inspection methods. Compared with other existing technologies, this research significantly improves the speed of internal thread image acquisition through the optimization of lighting and image capturing devices. To effectively tackle the challenge of image stitching for complex thread textures, an internal thread image stitching technique based on a cylindrical model is proposed, generating a full-view thread image. The use of the YOLOv8 model for precise defect localization in threads enhances the accuracy and efficiency of detection. This system provides an efficient and intuitive artificial intelligence solution for detecting surface defects on geometric bodies in confined spaces.

A Novel Classification Method: Neighborhood-Based Positive Unlabeled Learning Using Decision Tree (NPULUD)

In a standard binary supervised classification task, the existence of both negative and positive samples in the training dataset are required to construct a classification model. However, this condition is not met in certain applications where only one class of samples is obtainable. To overcome this problem, a different classification method, which learns from positive and unlabeled (PU) data, must be incorporated. In this study, a novel method is presented: neighborhood-based positive unlabeled learning using decision tree (NPULUD). First, NPULUD uses the nearest neighborhood approach for the PU strategy and then employs a decision tree algorithm for the classification task by utilizing the entropy measure. Entropy played a pivotal role in assessing the level of uncertainty in the training dataset, as a decision tree was developed with the purpose of classification. Through experiments, we validated our method over 24 real-world datasets. The proposed method attained an average accuracy of 87.24%, while the traditional supervised learning approach obtained an average accuracy of 83.99% on the datasets. Additionally, it is also demonstrated that our method obtained a statistically notable enhancement (7.74%), with respect to state-of-the-art peers, on average.

Research on Surface Defect Detection of Strip Steel Based on Improved YOLOv7

Surface defect detection of strip steel is an important guarantee for improving the production quality of strip steel. However, due to the diverse types, scales, and texture structures of surface defects on strip steel, as well as the irregular distribution of defects, it is difficult to achieve rapid and accurate detection of strip steel surface defects with existing methods. This article proposes a real-time and high-precision surface defect detection algorithm for strip steel based on YOLOv7. Firstly, Partial Conv is used to replace the conventional convolution blocks of the backbone network to reduce the size of the network model and improve the speed of detection; Secondly, The CA attention mechanism module is added to the ELAN module to enhance the ability of the network to extract detect features and improve the effectiveness of detect detection in complex environments; Finally, The SPD convolution module is introduced at the output end to improve the detection performance of small targets with surface defects on steel. The experimental results on the NEU-DET dataset indicate that the mean average accuracy (mAP@IoU = 0.5) is 80.4%, which is 4.0% higher than the baseline network. The number of parameters is reduced by 8.9%, and the computational load is reduced by 21.9% (GFLOPs). The detection speed reaches 90.9 FPS, which can well meet the requirements of real-time detection.

Image Data-Centric Visual Feature Selection on Roll-to-Roll Slot-Die Coating Systems for Edge Wave Coating Defect Detection

Roll-to-roll (R2R) manufacturing depends on a system's capability to deposit high-quality coatings with precise thickness, width, and uniformity. Therefore, consistent maintenance requires the immediate and accurate detection of coating defects. This study proposes a primary color selection (PCS) method to detect edge defects in R2R systems. This method addresses challenges associated with training data demands, complexity, and defect adaptability through a vision data-centric approach, ensuring precise edge coating defect detection. Using color information, high accuracy was achieved while minimizing data capacity requirements and processing time. Precise edge detection was facilitated by accurately distinguishing coated and noncoated regions by selecting the primary color channel based on color variability. The PCS method achieved superior accuracy (95.8%), outperforming the traditional weighted sum method (78.3%). This method is suitable for real-time detection in manufacturing systems and mitigates edge coating defects, thus facilitating quality control and production optimization.

On Automated Object Grasping for Intelligent Prosthetic Hands Using Machine Learning

Prosthetic technology has witnessed remarkable advancements, yet challenges persist in achieving autonomous grasping control while ensuring the user's experience is not compromised. Current electronic prosthetics often require extensive training for users to gain fine motor control over the prosthetic fingers, hindering their usability and acceptance. To address this challenge and improve the autonomy of prosthetics, this paper proposes an automated method that leverages computer vision-based techniques and machine learning algorithms. In this study, three reinforcement learning algorithms, namely Soft Actor-Critic (SAC), Deep Q-Network (DQN), and Proximal Policy Optimization (PPO), are employed to train agents for automated grasping tasks. The results indicate that the SAC algorithm achieves the highest success rate of 99% among the three algorithms at just under 200,000 timesteps. This research also shows that an object's physical characteristics can affect the agent's ability to learn an optimal policy. Moreover, the findings highlight the potential of the SAC algorithm in developing intelligent prosthetic hands with automatic object-gripping capabilities.

Mixed Receptive Fields Augmented YOLO with Multi-Path Spatial Pyramid Pooling for Steel Surface Defect Detection

Aiming at the problems of low detection efficiency and poor detection accuracy caused by texture feature interference and dramatic changes in the scale of defect on steel surfaces, an improved YOLOv5s model is proposed. In this study, we propose a novel re-parameterized large kernel C3 module, which enables the model to obtain a larger effective receptive field and improve the ability of feature extraction under complex texture interference. Moreover, we construct a feature fusion structure with a multi-path spatial pyramid pooling module to adapt to the scale variation of steel surface defects. Finally, we propose a training strategy that applies different kernel sizes for feature maps of different scales so that the receptive field of the model can adapt to the scale changes of the feature maps to the greatest extent. The experiment on the NEU-DET dataset shows that our model improved the detection accuracy of crazing and rolled in-scale, which contain a large number of weak texture features and are densely distributed by 14.4% and 11.1%, respectively. Additionally, the detection accuracy of inclusion and scratched defects with prominent scale changes and significant shape features was improved by 10.5% and 6.6%, respectively. Meanwhile, the mean average precision value reaches 76.8%, compared with the YOLOv5s and YOLOv8s, which increased by 8.6% and 3.7%, respectively.

Spot Welding Parameter Tuning for Weld Defect Prevention in Automotive Production Lines: An ML-Based Approach

Spot welding is a critical joining process which presents specific challenges in early defect detection, has high rework costs, and consumes excessive amounts of materials, hindering effective, sustainable production. Especially in automotive manufacturing, the welding source’s quality needs to be controlled to increase the efficiency and sustainable performance of the production lines. Using data analytics, manufacturing companies can control and predict the welding parameters causing problems related to resource quality and process performance. In this study, we aimed to define the root cause of welding defects and solve the welding input value range problem using machine learning algorithms. In an automotive production line application, we analyzed real-time IoT data and created variables regarding the best working range of welding input parameters required in the inference analysis for expulsion reduction. The results will help to provide guidelines and parameter selection approaches to model ML-based solutions for the optimization problems associated with welding.

Detecting Machining Defects inside Engine Piston Chamber with Computer Vision and Machine Learning

This paper describes the implementation of a solution for detecting the machining defects from an engine block, in the piston chamber. The solution was developed for an automotive manufacturer and the main goal of the implementation is the replacement of the visual inspection performed by a human operator with a computer vision application. We started by exploring different machine vision applications used in the manufacturing environment for several types of operations, and how machine learning is being used in robotic industrial applications. The solution implementation is re-using hardware that is already available at the manufacturing plant and decommissioned from another system. The re-used components are the cameras, the IO (Input/Output) Ethernet module, sensors, cables, and other accessories. The hardware will be used in the acquisition of the images, and for processing, a new system will be implemented with a human-machine interface, user controls, and communication with the main production line. Main results and conclusions highlight the efficiency of the CCD (charged-coupled device) sensors in the manufacturing environment and the robustness of the machine learning algorithms (convolutional neural networks) implemented in computer vision applications (thresholding and regions of interest).

DSTEELNet: A Real-Time Parallel Dilated CNN with Atrous Spatial Pyramid Pooling for Detecting and Classifying Defects in Surface Steel Strips

Automatic defects inspection and classification demonstrate significant importance in improving quality in the steel industry. This paper proposed and developed DSTEELNet convolution neural network (CNN) architecture to improve detection accuracy and the required time to detect defects in surface steel strips. DSTEELNet includes three parallel stacks of convolution blocks with atrous spatial pyramid pooling. Each convolution block used a different dilation rate that expands the receptive fields, increases the feature resolutions and covers square regions of input 2D image without any holes or missing edges and without increases in computations. This work illustrates the performance of DSTEELNet with a different number of parallel stacks and a different order of dilation rates. The experimental results indicate significant improvements in accuracy and illustrate that the DSTEELNet achieves of 97% mAP in detecting defects in surface steel strips on the augmented dataset GNEU and Severstal datasets and is able to detect defects in a single image in 23ms.

A Robust Deep Learning Ensemble-Driven Model for Defect and Non-Defect Recognition and Classification Using a Weighted Averaging Sequence-Based Meta-Learning Ensembler

The need to overcome the challenges of visual inspections conducted by domain experts drives the recent surge in visual inspection research. Typical manual industrial data analysis and inspection for defects conducted by trained personnel are expensive, time-consuming, and characterized by mistakes. Thus, an efficient intelligent-driven model is needed to eliminate or minimize the challenges of defect identification and elimination in processes to the barest minimum. This paper presents a robust method for recognizing and classifying defects in industrial products using a deep-learning architectural ensemble approach integrated with a weighted sequence meta-learning unification framework. In the proposed method, a unique base model is constructed and fused together with other co-learning pretrained models using a sequence-driven meta-learning ensembler that aggregates the best features learned from the various contributing models for better and superior performance. During experimentation in the study, different publicly available industrial product datasets consisting of the defect and non-defect samples were used to train, validate, and test the introduced model, with remarkable results obtained that demonstrate the viability of the proposed method in tackling the challenges of the manual visual inspection approach.

A Review on Data-Driven Quality Prediction in the Production Process with Machine Learning for Industry 4.0

The quality-control process in manufacturing must ensure the product is free of defects and performs according to the customer’s expectations. Maintaining the quality of a firm’s products at the highest level is very important for keeping an edge over the competition. To maintain and enhance the quality of their products, manufacturers invest a lot of resources in quality control and quality assurance. During the assembly line, parts will arrive at a constant interval for assembly. The quality criteria must first be met before the parts are sent to the assembly line where the parts and subparts are assembled to get the final product. Once the product has been assembled, it is again inspected and tested before it is delivered to the customer. Because manufacturers are mostly focused on visual quality inspection, there can be bottlenecks before and after assembly. The manufacturer may suffer a loss if the assembly line is slowed down by this bottleneck. To improve quality, state-of-the-art sensors are being used to replace visual inspections and machine learning is used to help determine which part will fail. Using machine learning techniques, a review of quality assessment in various production processes is presented, along with a summary of the four industrial revolutions that have occurred in manufacturing, highlighting the need to detect anomalies in assembly lines, the need to detect the features of the assembly line, the use of machine learning algorithms in manufacturing, the research challenges, the computing paradigms, and the use of state-of-the-art sensors in Industry 4.0.

An Image-Based Data-Driven Model for Texture Inspection of Ground Workpieces

Nowadays, the grinding process is mostly automatic, yet post-grinding quality inspection is mostly carried out manually. Although the conventional inspection technique may have cumbersome setup and tuning processes, the data-driven model, with its vision-based dataset, provides an opportunity to automate the inspection process. In this study, a convolutional neural network technique with transfer learning is proposed for three kinds of inspections based on 750-1000 surface raw images of the ground workpieces in each task: classifying the grit number of the abrasive belt that grinds the workpiece, estimating the surface roughness of the ground workpiece, and classifying the degree of wear of the abrasive belts. The results show that a deep convolutional neural network can recognize the texture on the abrasive surface images and that the classification model can achieve an accuracy of 0.9 or higher. In addition, the external coaxial white light was the most suitable light source among the three tested light sources: the external coaxial white light, the high-angle ring light, and the external coaxial red light. Finally, the model that classifies the degree of wear of the abrasive belts can also be utilized as the abrasive belt life estimator.

IA-Mask R-CNN: Improved Anchor Design Mask R-CNN for Surface Defect Detection of Automotive Engine Parts

The detection of surface defects on automotive engine parts is an important part of automobile manufacturing quality assurance. The traditional detection methods rely on manual inspection and can be inaccurate and inefficient, while the existing deep learning-based methods, such as the Mask R-CNN detection method, have insufficient precision for detecting minor defects since the anchor scales design does not consider small defects. To overcome these shortcomings, this paper proposes an IA-Mask R-CNN detection method with an improved anchor scales design. First, an image dataset that contains 560 pictures of surface defects of automotive engine parts is established using a 1080P HDMI high-definition digital microscope capable of recording three million real pixels and labeled manually. Then, the anchor scales suitable for the surface defect detection of automotive engine parts are determined by labeled data analysis and used to improve the anchor design in Mask R-CNN. Finally, the proposed method is compared experimentally with the Faster R-CNN and Mask R-CNN, and qualitative and quantitative analyses are conducted. The experimental results show that, without increasing the number of parameters or training time of the Mask R-CNN, the proposed method performed better in detecting minor, as well as larger defects than the other detection methods.

Detection of Dense Citrus Fruits by Combining Coordinated Attention and Cross-Scale Connection with Weighted Feature Fusion

The accuracy detection of individual citrus fruits in a citrus orchard environments is one of the key steps in realizing precision agriculture applications such as yield estimation, fruit thinning, and mechanical harvesting. This study proposes an improved object detection YOLOv5 model to achieve accurate the identification and counting of citrus fruits in an orchard environment. First, the latest visual attention mechanism coordinated attention module (CA) was inserted into an improved backbone network to focus on fruit-dense regions to recognize small target fruits. Second, an efficient two-way cross-scale connection and weighted feature fusion BiFPN in the neck network were used to replace the PANet multiscale feature fusion network, giving effective feature corresponding weights to fully fuse the high-level and bottom-level features. Finally, the varifocal loss function was used to calculate the model loss for better model training results. The results of the experiments on four varieties of citrus trees showed that our improved model proposed to this study could effectively identify dense small citrus fruits. Specifically, the recognized AP (average precision) reached 98.4%, and the average recognition time was 0.019 s per image. Compared with the original YOLOv5 (including deferent variants of n, s, m, l, and x), the increase in the average accuracy precision of the improved YOLOv5 ranged from 7.5% to 0.8% while maintaining similar average inference time. Four different citrus varieties were also tested to evaluate the generalization performance of the improved model. The method can be further used as a part in a vision system to provide technical support for the real-time and accurate detection of multiple fruit targets during mechanical picking in citrus orchards.

Investigation of an Inline Inspection Method for the Examination of Cylinder-like Specular Surfaces Using Deflectometry

An optical measuring method is presented, with which it is possible to measure and evaluate reflective cylindrical surfaces using a combination of deflectometry and subsequent reconstruction of the surface. The system is set up and tested on rolling elements of cylindrical roller bearings. However, it is not limited to this use case and can be applied to other cylindrical specular surfaces. The system distinguishes itself from existing test methods through the combination of high-resolution three-dimensional defect measurement with a very short recording time, and offers the possibility of introducing tolerance limits in the production of cylindrical specular surfaces. With this method, it is possible to record a defect with the dimensions of 1.3mm by 1.8mm within 5 s and to reconstruct the absolute depth. The resolution of the system is below 10 μm in both X- and Y-direction, and is therefore sufficiently accurate to detect typical surface defects such as scratches, dents, or deformations. To validate the measured values of the system, the results of an artificially generated 10.35 μm deep defect location were compared with those of a highly accurate mechanical stylus measurement.

Improved U-Net++ with Patch Split for Micro-Defect Inspection in Silk Screen Printing

The trend of multi-variety production is leading to a change in the product type of silk screen prints produced at short intervals. The types and locations of defects that usually occur in silk screen prints may vary greatly and thus, it is difficult for operators to conduct quality inspections for minuscule defects. In this paper, an improved U-Net++ is proposed based on patch splits for automated quality inspection of small or tiny defects, hereinafter referred to as ‘fine’ defects. The novelty of the method is that, to better handle defects within an image, patch level inputs are considered instead of using the original image as input. In the existing technique with the original image as input, artificial intelligence (AI) learning is not utilized efficiently, whereas our proposed method learns stably, and the Dice score was 0.728, which is approximately 10% higher than the existing method. The proposed model was applied to an actual silk screen printing process. All of the fine defects in products, such as silk screen prints, could be detected regardless of the product size. In addition, it was shown that quality inspection using the patch-split method-based AI is possible even in situations where there are few prior defective data.

Object Detection Algorithm for Surface Defects Based on a Novel YOLOv3 Model

The surface defects of industrial structural parts have the characteristics of a large-scale span and many small objects, so a novel YOLOv3 model, the YOLOv3-ALL algorithm, is proposed in this paper to solve the problem of precise defect detection. The K-means++ algorithm is combined with the intersection-over-union (IoU) and comparison of the prior box for clustering, which improves the clustering effect. The convolutional block attention module (CBAM) is embedded in the network, thus improving the ability of the network to obtain key information in the image. By adding fourth-scale prediction, the detection capability of a YOLOv3 network for small-object defects is greatly improved. A loss function is designed, which adds the generalized intersection-over-union (GIoU) loss combined with focal loss to solve the problems of L2 loss and class imbalance in samples. Experiments regarding contour-defect detection for stamping parts show that the mean average precision (mAP) of the YOLOV3-ALL algorithm reaches 75.05% in defect detection, which is 25.16% higher than that of the YOLOv3 algorithm. The average detection time is 39 ms/sheet. This proves that the YOLOv3-ALL algorithm has good real-time detection efficiency and high detection accuracy.

Vision-Based Defect Inspection and Condition Assessment for Sewer Pipes: A Comprehensive Survey

Due to the advantages of economics, safety, and efficiency, vision-based analysis techniques have recently gained conspicuous advancements, enabling them to be extensively applied for autonomous constructions. Although numerous studies regarding the defect inspection and condition assessment in underground sewer pipelines have presently emerged, we still lack a thorough and comprehensive survey of the latest developments. This survey presents a systematical taxonomy of diverse sewer inspection algorithms, which are sorted into three categories that include defect classification, defect detection, and defect segmentation. After reviewing the related sewer defect inspection studies for the past 22 years, the main research trends are organized and discussed in detail according to the proposed technical taxonomy. In addition, different datasets and the evaluation metrics used in the cited literature are described and explained. Furthermore, the performances of the state-of-the-art methods are reported from the aspects of processing accuracy and speed.

Unsupervised Anomaly Detection and Segmentation on Dirty Datasets

Industrial quality control is an important task. Most of the existing vision-based unsupervised industrial anomaly detection and segmentation methods require that the training set only consists of normal samples, which is difficult to ensure in practice. This paper proposes an unsupervised framework to solve the industrial anomaly detection and segmentation problem when the training set contains anomaly samples. Our framework uses a model pretrained on ImageNet as a feature extractor to extract patch-level features. After that, we propose a trimming method to estimate a robust Gaussian distribution based on the patch features at each position. Then, with an iterative filtering process, we can iteratively filter out the anomaly samples in the training set and re-estimate the Gaussian distribution at each position. In the prediction phase, the Mahalanobis distance between a patch feature vector and the center of the Gaussian distribution at the corresponding position is used as the anomaly score of this patch. The subsequent anomaly region segmentation is performed based on the patch anomaly score. We tested the proposed method on three datasets containing the anomaly samples and obtained state-of-the-art performance.

Defect Detection on LED Chips Based on Position Pre-Estimation and Feature Enhancement

Light-emitting diode (LED) chips have disordered arrangement and defects with characteristics of low contrast, for which traditional segmentation methods cannot classify surface defects effectively. In this paper, a chip segmentation method based on position pre-estimation and a modified Normalized Correlation Coefficient (NCC) matching algorithm, as well as feature enhancement methods are proposed. The position pre-estimation method is used to avoid the interference introduced by the disordered chip arrangement and the large missing area. By modifying the NCC algorithm, matching speed is improved by eight times compared to traditional NCC while matching result is not affected by brightness change. Furthermore, feature enhancement schemes with higher speed and accuracy were designed to identify low-contrast defects. The experimental results showed that the average accuracy reached 99.54%, improved by 0.66% compared to the state-of-the-art method while the inspection missing rate was 0.03%. In addition, the detection time of a single chip was approximately 1.098 ms, which meets the requirements of online detection, and the smallest defect that could be detected was 2 µm. In summary, the methods proposed in this study meet the requirements of industrial online detection regardless of accuracy, efficiency, or extensibility.

An Automatic Procedure for Overheated Idler Detection in Belt Conveyors Using Fusion of Infrared and RGB Images Acquired during UGV Robot Inspection

Complex mechanical systems used in the mining industry for efficient raw materials extraction require proper maintenance. Especially in a deep underground mine, the regular inspection of machines operating in extremely harsh conditions is challenging, thus, monitoring systems and autonomous inspection robots are becoming more and more popular. In the paper, it is proposed to use a mobile unmanned ground vehicle (UGV) platform equipped with various data acquisition systems for supporting inspection procedures. Although maintenance staff with appropriate experience are able to identify problems almost immediately, due to mentioned harsh conditions such as temperature, humidity, poisonous gas risk, etc., their presence in dangerous areas is limited. Thus, it is recommended to use inspection robots collecting data and appropriate algorithms for their processing. In this paper, the authors propose red-green-blue (RGB) and infrared (IR) image fusion to detect overheated idlers. An original procedure for image processing is proposed, that exploits some characteristic features of conveyors to pre-process the RGB image to minimize non-informative components in the pictures collected by the robot. Then, the authors use this result for IR image processing to improve SNR and finally detect hot spots in IR image. The experiments have been performed on real conveyors operating in industrial conditions.

UTRAD: Anomaly detection and localization with U-Transformer

Anomaly detection is an active research field in industrial defect detection and medical disease detection. However, previous anomaly detection works suffer from unstable training, or non-universal criteria of evaluating feature distribution. In this paper, we introduce UTRAD, a U-TRansformer based Anomaly Detection framework. Deep pre-trained features are regarded as dispersed word tokens, and represented with transformer-based autoencoders. With reconstruction on more informative feature distribution instead of raw images, we achieve a more stable training process and a more precise anomaly detection and localization result. In addition, our proposed UTRAD has a multi-scale pyramidal hierarchy with skip connections that help detect both multi-scale structural and non-structural anomalies. As attention layers are decomposed to multi-level patches, UTRAD significantly reduces the computational cost and memory usage compared with the vanilla transformer. Experiments on industrial dataset MVtec AD and medical datasets Retinal-OCT, Brain-MRI, Head-CT have been conducted. Our proposed UTRAD out-performs all other state-of-the-art methods in the above datasets. Code released at https://github.com/gordon-chenmo/UTRAD.

Infoecology of the deep learning and smart manufacturing: thematic and concept interactions

Purpose

This infecological study mainly aimed to know the thematic and conceptual relationship in published papers in deep learning (DL) and smart manufacturing (SM).

Design/methodology/approach

The research methodology has specific research objectives based on the type and method of research, data analysis tools. In general, description methods are applied by Web of Science (WoS) analysis tools and Voyant tools as a web-based reading and analysis environment for digital texts. The Yewno tool is applied to draw a knowledge map to show the concept's interaction between DL and SM.

Findings

The knowledge map of DL and SM concepts shows that there are currently few concepts interacting with each other, while the rapid growth of technology and the needs of today's society have revealed the need to use more and more DL in SM. The results of this study can provide a coherent and well-mapped road map to the main policymakers of the field of research in DL and SM, through the study of coexistence and interactions of the thematic categories with other thematic areas. In this way, they can design more effective guidelines and strategies to solve the problems of researchers in conducting their studies and direct. The analysis results demonstrated that the information ecosystem of DL and SM studies dynamically developed over time. The continuous conduction flow of scientific studies in this field brought continuous changes into the infoecology of subjects and concepts in this area.

Originality/value

The paper investigated the thematic interaction of the scientific productions in DL and SM and discussed possible implications. We used of the variety tools and techniques to draw our own perspective. Also, we drew arguments from other research work to back up our findings.

A Data-Centric Approach to Design and Analysis of a Surface-Inspection System Based on Deep Learning in the Plastic Injection Molding Industry

Manufacturers are eager to replace the human inspector with automatic inspection systems to improve the competitive advantage by means of quality. However, some manufacturers have failed to apply the traditional vision system because of constraints in data acquisition and feature extraction. In this paper, we propose an inspection system based on deep learning for a tampon applicator producer that uses the applicator’s structural characteristics for data acquisition and uses state-of-the-art models for object detection and instance segmentation, YOLOv4 and YOLACT for feature extraction, respectively. During the on-site trial test, we experienced some False-Positive (FP) cases and found a possible Type I error. We used a data-centric approach to solve the problem by using two different data pre-processing methods, the Background Removal (BR) and Contrast Limited Adaptive Histogram Equalization (CLAHE). We have experimented with analyzing the effect of the methods on the inspection with the self-created dataset. We found that CLAHE increased Recall by 0.1 at the image level, and both CLAHE and BR improved Precision by 0.04–0.06 at the bounding box level. These results support that the data-centric approach might improve the detection rate. However, the data pre-processing techniques deteriorated the metrics used to measure the overall performance, such as F1-score and Average Precision (AP), even though we empirically confirmed that the malfunctions improved. With the detailed analysis of the result, we have found some cases that revealed the ambiguity of the decisions caused by the inconsistency in data annotation. Our research alerts AI practitioners that validating the model based only on the metrics may lead to a wrong conclusion.

Imbalance Modelling for Defect Detection in Ceramic Substrate by Using Convolutional Neural Network

The complexity of defect detection in a ceramic substrate causes interclass and intraclass imbalance problems. Identifying flaws in ceramic substrates has traditionally relied on aberrant material occurrences and characteristic quantities. However, defect substrates in ceramic are typically small and have a wide variety of defect distributions, thereby making defect detection more challenging and difficult. Thus, we propose a method for defect detection based on unsupervised learning and deep learning. First, the proposed method conducts K-means clustering for grouping instances according to their inherent complex characteristics. Second, the distribution of rarely occurring instances is balanced by using augmentation filters. Finally, a convolutional neural network is trained by using the balanced dataset. The effectiveness of the proposed method was validated by comparing the results with those of other methods. Experimental results show that the proposed method outperforms other methods.

Defect Detection on a Wind Turbine Blade Based on Digital Image Processing

Wind power generation is a widely used power generation technology. Among these, the wind turbine blade is an important part of a wind turbine. If the wind turbine blade is damaged, it will cause serious consequences. The traditional methods of defect detection for wind turbine blades are mainly manual detection and acoustic nondestructive detection, which are unsafe and time-consuming, and have low accuracy. In order to detect the defects on wind turbine blades more safely, conveniently, and accurately, this paper studied a defect detection method for wind turbine blades based on digital image processing. Because the log-Gabor filter used needed to extract features through multiple filter templates, the number of output images was large. Firstly, this paper used the Lévy flight strategy to improve the PSO algorithm to create the LPSO algorithm. The improved LPSO algorithm could successfully solve the PSO algorithm’s problem of falling into the local optimal solution. Then, the LPSO algorithm and log-Gabor filter were used to generate an adaptive filter, which could directly output the optimal results in multiple feature extraction images. Finally, a classifier based on HOG + SVM was used to identify and classify the defect types. The method extracted and identified the scratch-type, crack-type, sand-hole-type, and spot-type defects, and the recognition rate was more than 92%.

Classification of Textile Polymer Composites: Recent Trends and Challenges

Polymer based textile composites have gained much attention in recent years and gradually transformed the growth of industries especially automobiles, construction, aerospace and composites. The inclusion of natural polymeric fibres as reinforcement in carbon fibre reinforced composites manufacturing delineates an economic way, enhances their surface, structural and mechanical properties by providing better bonding conditions. Almost all textile-based products are associated with quality, price and consumer’s satisfaction. Therefore, classification of textiles products and fibre reinforced polymer composites is a challenging task. This paper focuses on the classification of various problems in textile processes and fibre reinforced polymer composites by artificial neural networks, genetic algorithm and fuzzy logic. Moreover, their limitations associated with state-of-the-art processes and some relatively new and sequential classification methods are also proposed and discussed in detail in this paper.

Pipeline In-Line Inspection Method, Instrumentation and Data Management

Pipelines play an important role in the national/international transportation of natural gas, petroleum products, and other energy resources. Pipelines are set up in different environments and consequently suffer various damage challenges, such as environmental electrochemical reaction, welding defects, and external force damage, etc. Defects like metal loss, pitting, and cracks destroy the pipeline's integrity and cause serious safety issues. This should be prevented before it occurs to ensure the safe operation of the pipeline. In recent years, different non-destructive testing (NDT) methods have been developed for in-line pipeline inspection. These are magnetic flux leakage (MFL) testing, ultrasonic testing (UT), electromagnetic acoustic technology (EMAT), eddy current testing (EC). Single modality or different kinds of integrated NDT system named Pipeline Inspection Gauge (PIG) or un-piggable robotic inspection systems have been developed. Moreover, data management in conjunction with historic data for condition-based pipeline maintenance becomes important as well. In this study, various inspection methods in association with non-destructive testing are investigated. The state of the art of PIGs, un-piggable robots, as well as instrumental applications, are systematically compared. Furthermore, data models and management are utilized for defect quantification, classification, failure prediction and maintenance. Finally, the challenges, problems, and development trends of pipeline inspection as well as data management are derived and discussed.

Segmentation of Drilled Holes in Texture Wooden Furniture Panels Using Deep Neural Network

Drilling operations are an essential part of furniture from MDF laminated boards required for product assembly. Faults in the process might introduce adverse effects to the furniture. Inspection of the drilling quality can be challenging due to a big variety of board surface textures, dust, or woodchips in the manufacturing process, milling cutouts, and other kinds of defects. Intelligent computer vision methods can be engaged for global contextual analysis with local information attention for automated object detection and segmentation. In this paper, we propose blind and through drilled holes segmentation on textured wooden furniture panel images using the UNet encoder-decoder modifications enhanced with residual connections, atrous spatial pyramid pooling, squeeze and excitation module, and CoordConv layers for better segmentation performance. We show that even a lightweight architecture is capable to perform on a range of complex textures and is able to distinguish the holes drilling operations’ semantical information from the rest of the furniture board and conveyor context. The proposed model configurations yield better results in more complex cases with a not significant or small bump in processing time. Experimental results demonstrate that our best-proposed solution achieves a Dice score of up to 97.89% compared to the baseline U-Net model’s Dice score of 94.50%. Statistical, visual, and computational properties of each convolutional neural network architecture are addressed.

AI Landing for Sheet Metal-Based Drawer Box Defect Detection Using Deep Learning (ALDB-DL)

Sheet metal-based products serve as a major portion of the furniture market and maintain higher quality standards by being competitive. During industrial processes, while converting a sheet metal to an end product, new defects are observed and thus need to be identified carefully. Recent studies have shown scratches, bumps, and pollution/dust are identified, but orange peel defects present overall a new challenge. So our model identifies scratches, bumps, and dust by using computer vision algorithms, whereas orange peel defect detection with deep learning have a better performance. The goal of this paper was to resolve artificial intelligence (AI) as an AI landing challenge faced in identifying various kinds of sheet metal-based product defects by ALDB-DL process automation. Therefore, our system model consists of multiple cameras from two different angles to capture the defects of the sheet metal-based drawer box. The aim of this paper was to solve multiple defects detection as design and implementation of Industrial process integration with AI by Automated Optical Inspection (AOI) for sheet metal-based drawer box defect detection, stated as AI Landing for sheet metal-based Drawer Box defect detection using Deep Learning (ALDB-DL). Therefore, the scope was given as achieving higher accuracy using multi-camera-based image feature extraction using computer vision and deep learning algorithm for defect classification in AOI. We used SHapley Additive exPlanations (SHAP) values for pre-processing, LeNet with a (1 × 1) convolution filter, and a Global Average Pooling (GAP) Convolutional Neural Network (CNN) algorithm to achieve the best results. It has applications for sheet metal-based product industries with improvised quality control for edge and surface detection. The results were competitive as the precision, recall, and area under the curve were 1.00, 0.99, and 0.98, respectively. Successively, the discussion section presents a detailed insight view about the industrial functioning with ALDB-DL experience sharing.

Defective Product Classification System for Smart Factory Based on Deep Learning

Smart factories merge various technologies in a manufacturing environment in order to improve factory performance and product quality. In recent years, these smart factories have received a lot of attention from researchers. In this paper, we introduce a defective product classification system based on deep learning for application in smart factories. The key component of the proposed system is a programmable logic controller (PLC) artificial intelligence (AI) embedded board; we call this an AI Edge-PLC module. A pre-trained defective product classification model is uploaded to a cloud service from where the AI Edge-PLC can access and download it for use on a certain product, in this case, electrical wiring. Next, we setup the system to collect electrical wiring data in a real-world factory environment. Then, we applied preprocessing to the collected data in order to extract a region of interest (ROI) from the images. Due to limitations on the availability of appropriate labeled data, we used the transfer learning method to re-train a classification model for our purposes. The pre-trained models were then optimized for applications on AI Edge-PLC boards. After carrying out classification tasks, on our electrical wire dataset and on a previously published casting dataset, using various deep neural networks including VGGNet, ResNet, DenseNet, and GoogLeNet, we analyzed the results achieved by our system. The experimental results show that our system is able to classify defective products quickly with high accuracy in a real-world manufacturing environment.

Improvement for Convolutional Neural Networks in Image Classification Using Long Skip Connection

In this paper, we examine and research the effect of long skip connection on convolutional neural networks (CNNs) for the tasks of image (surface defect) classification. The standard popular models only apply short skip connection inside blocks (layers with the same size). We apply the long version of residual connection on several proposed models, which aims to reuse the lost spatial knowledge from the layers close to input. For some models, Depthwise Separable Convolution is used rather than traditional convolution in order to reduce both count of parameters and floating-point operations per second (FLOPs). Comparative experiments of the newly upgraded models and some popular models have been carried out on different datasets including Bamboo strips datasets and a reduced version of ImageNet. The modified version of DenseNet 121 (we call MDenseNet 121) achieves higher validation accuracy while it has about 75% of weights and FLOPs in comparison to the original DenseNet 121.

Dropout Deep Belief Network Based Chinese Ancient Ceramic Non-Destructive Identification

A non-destructive identification method was developed here based on dropout deep belief network in multi-spectral data of ancient ceramic. A fractional differential algorithm was proposed to enhance the spectral details by making use of the difference between the first and second-order differential pre-process spectral data. An unsupervised multi-layer restricted Boltzmann machine (RBM) was employed to extract some high-level features during pre-training. Some weight and bias values trained by RBM were used to initialize a back propagation (BP) neural network. The RBM deep belief network was fine-tuned by the BP neural network to promote the initiative performance of network training, which helped to overcome local optimal limitation of the network due to the random initializing weight parameter. The dropout strategy has been put forward into the RBM network to solve the over-fitting of small sample spectral data. The experimental results show that the proposed method has excellent recognition performance of the ceramics by comparisons with some other ones.

Chip Appearance Inspection Method for High-Precision SMT Equipment

In order to meet the defect-detection requirements of chips in high-precision surface mount technology (SMT) equipment widely used in the electronic industry, a chip appearance defect-detection method based on multi-order fractional discrete wavelet packet decomposition (DWPD) is proposed in this paper. First, lead and body regions were extracted from chip images using the image segmentation algorithm with asymmetric Laplace mixture model and connected-component labelling algorithm; then, the texture feature of the region to be inspected was extracted with the multi-order fractional DWPD algorithm and the geometric and gradient features were combined to form image features of the region to be inspected before the subset of features was selected from image features with the feature selection algorithm based on the variational Bayesian Gaussian mixture model; and finally, the support vector machine was used to determine whether the region to be inspected was defective. An experiment was conducted on a data set captured in high-precision SMT equipment. The accuracy of the proposed chip appearance defect-detection method is about 93%, which is more accurate than existing ones.

Surface Defect Inspection in Images Using Statistical Patches Fusion and Deeply Learned Features

Defect detection in images is a challenging task due to the existence of tiny and noisy patterns on surface images. To tackle this challenge, a defect detection approach is proposed in this paper using statistical data fusion. First, the proposed approach breaks a large image that contains multiple separate defects into smaller overlapping patches to detect the existence of defects in each patch, using the conventional convolutional neural network approach. Then, a statistical data fusion approach is proposed to maintain the spatial coherence of cracks in the image and aggregate the information extracted from overlapping patches to enhance the overall performance and robustness of the system. The proposed approach is evaluated using three benchmark datasets to demonstrate its superior performance in terms of both individual patch inspection and the whole image inspection.

Quality Control of PET Bottles Caps with Dedicated Image Calibration and Deep Neural Networks

Product quality control is currently the leading trend in industrial production. It is heading towards the exact analysis of each product before reaching the end customer. Every stage of production control is of particular importance in the food and pharmaceutical industries, where, apart from visual issues, additional safety regulations are demanded. Many production processes can be controlled completely contactless through the use of machine vision cameras and advanced image processing techniques. The most dynamically growing sector of image analysis methods are solutions based on deep neural networks. Their major advantages are fast performance, robustness, and the fact that they can be exploited even in complicated classification problems. However, the use of machine learning methods on high-performance production lines may be limited by inference time or, in the case of multiformated production lines, training time. The article presents a novel data preprocessing (or calibration) method. It uses prior knowledge about the optical system, which enables the use of the lightweight Convolutional Neural Network (CNN) model for product quality control of polyethylene terephthalate (PET) bottle caps. The combination of preprocessing with the lightweight CNN model resulted in at least a five-fold reduction in prediction and training time compared to the lighter standard models tested on ImageNet, without loss of accuracy.

An Unsupervised Generative Adversarial Network-Based Method for Defect Inspection of Texture Surfaces

Recently, deep learning-based defect inspection methods have begun to receive more attention—from both researchers and the industrial community—due to their powerful representation and learning capabilities. These methods, however, require a large number of samples and manual annotation to achieve an acceptable detection rate. In this paper, we propose an unsupervised method of detecting and locating defects on patterned texture surface images which, in the training phase, needs only a moderate number of defect-free samples. An extended deep convolutional generative adversarial network (DCGAN) is utilized to reconstruct input image patches; the resulting residual map can be used to realize the initial segmentation defects. To further improve the accuracy of defect segmentation, a submodule termed “local difference analysis” (LDA) is embedded into the overall module to eliminate false positives. We conduct comparative experiments on a series of datasets and the final results verify the effectiveness of the proposed method.

Using Deep Learning to Detect Defects in Manufacturing: A Comprehensive Survey and Current Challenges

The detection of product defects is essential in quality control in manufacturing. This study surveys stateoftheart deep-learning methods in defect detection. First, we classify the defects of products, such as electronic components, pipes, welded parts, and textile materials, into categories. Second, recent mainstream techniques and deep-learning methods for defects are reviewed with their characteristics, strengths, and shortcomings described. Third, we summarize and analyze the application of ultrasonic testing, filtering, deep learning, machine vision, and other technologies used for defect detection, by focusing on three aspects, namely method and experimental results. To further understand the difficulties in the field of defect detection, we investigate the functions and characteristics of existing equipment used for defect detection. The core ideas and codes of studies related to high precision, high positioning, rapid detection, small object, complex background, occluded object detection and object association, are summarized. Lastly, we outline the current achievements and limitations of the existing methods, along with the current research challenges, to assist the research community on defect detection in setting a further agenda for future studies.

Research Progress of Automated Visual Surface Defect Detection for Industrial Metal Planar Materials

The computer-vision-based surface defect detection of metal planar materials is a research hotspot in the field of metallurgical industry. The high standard of planar surface quality in the metal manufacturing industry requires that the performance of an automated visual inspection system and its algorithms are constantly improved. This paper attempts to present a comprehensive survey on both two-dimensional and three-dimensional surface defect detection technologies based on reviewing over 160 publications for some typical metal planar material products of steel, aluminum, copper plates and strips. According to the algorithm properties as well as the image features, the existing two-dimensional methodologies are categorized into four groups: statistical, spectral, model, and machine learning-based methods. On the basis of three-dimensional data acquisition, the three-dimensional technologies are divided into stereoscopic vision, photometric stereo, laser scanner, and structured light measurement methods. These classical algorithms and emerging methods are introduced, analyzed, and compared in this review. Finally, the remaining challenges and future research trends of visual defect detection are discussed and forecasted at an abstract level.

Defect Detection on Rolling Element Surface Scans Using Neural Image Segmentation

The surface inspection of steel parts like rolling elements for roller bearings is an essential component of the quality assurance process in their production. Existing inspection systems require high maintenance cost and allow little flexibility. In this paper, we propose the use of a rapidly retrainable convolutional neural network. Our approach reduces the development and maintenance cost compared to a manually programmed classification system for steel surface defect detection. One of the main disadvantages of neural network approaches is their high demand for labeled training data. To bypass this, we propose the use of simulated defects. In the production of rolling elements, real defects are a rarity. Collecting a balanced dataset thus costs a lot of time and resources. Simulating defects reduces the time required for data collection. It also allows us to automatically label the dataset. This further eases the data collection process compared to existing approaches. Combined, this allows us to train our system faster and cheaper than existing systems. We will show that our system can be retrained in a matter of minutes, minimizing production downtime, while still allowing high accuracy in defect detection.