1
|
Kong L, Xie K, Niu K, He J, Zhang W. Remote Photoplethysmography and Motion Tracking Convolutional Neural Network with Bidirectional Long Short-Term Memory: Non-Invasive Fatigue Detection Method Based on Multi-Modal Fusion. SENSORS (BASEL, SWITZERLAND) 2024; 24:455. [PMID: 38257546 PMCID: PMC11154312 DOI: 10.3390/s24020455] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/17/2023] [Revised: 01/04/2024] [Accepted: 01/08/2024] [Indexed: 01/24/2024]
Abstract
Existing vision-based fatigue detection methods commonly utilize RGB cameras to extract facial and physiological features for monitoring driver fatigue. These features often include single indicators such as eyelid movement, yawning frequency, and heart rate. However, the accuracy of RGB cameras can be affected by factors like varying lighting conditions and motion. To address these challenges, we propose a non-invasive method for multi-modal fusion fatigue detection called RPPMT-CNN-BiLSTM. This method incorporates a feature extraction enhancement module based on the improved Pan-Tompkins algorithm and 1D-MTCNN. This enhances the accuracy of heart rate signal extraction and eyelid features. Furthermore, we use one-dimensional neural networks to construct two models based on heart rate and PERCLOS values, forming a fatigue detection model. To enhance the robustness and accuracy of fatigue detection, the trained model data results are input into the BiLSTM network. This generates a time-fitting relationship between the data extracted from the CNN, allowing for effective dynamic modeling and achieving multi-modal fusion fatigue detection. Numerous experiments validate the effectiveness of the proposed method, achieving an accuracy of 98.2% on the self-made MDAD (Multi-Modal Driver Alertness Dataset). This underscores the feasibility of the algorithm. In comparison with traditional methods, our approach demonstrates higher accuracy and positively contributes to maintaining traffic safety, thereby advancing the field of smart transportation.
Collapse
Affiliation(s)
- Lingjian Kong
- School of Electronic Information and Electrical Engineering, Yangtze University, Jingzhou 434023, China; (L.K.); (K.N.)
| | - Kai Xie
- School of Electronic Information and Electrical Engineering, Yangtze University, Jingzhou 434023, China; (L.K.); (K.N.)
| | - Kaixuan Niu
- School of Electronic Information and Electrical Engineering, Yangtze University, Jingzhou 434023, China; (L.K.); (K.N.)
| | - Jianbiao He
- School of Computer Science, Central South University, Changsha 410083, China;
| | - Wei Zhang
- School of Electronic Information, Central South University, Changsha 410083, China;
| |
Collapse
|
2
|
Kim D, Park H, Kim T, Kim W, Paik J. Real-time driver monitoring system with facial landmark-based eye closure detection and head pose recognition. Sci Rep 2023; 13:18264. [PMID: 37880264 PMCID: PMC10600215 DOI: 10.1038/s41598-023-44955-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Accepted: 10/13/2023] [Indexed: 10/27/2023] Open
Abstract
This paper introduces a real-time Driver Monitoring System (DMS) designed to monitor driver behavior while driving, employing facial landmark estimation-based behavior recognition. The system utilizes an infrared (IR) camera to capture and analyze video data. Through facial landmark estimation, crucial information about the driver's head posture and eye area is extracted from the detected facial region, obtained via face detection. The proposed method consists of two distinct modules, each focused on recognizing specific behaviors. The first module employs head pose analysis to detect instances of inattention. By monitoring the driver's head movements along the horizontal and vertical axes, this module assesses the driver's attention level. The second module implements an eye-closure recognition filter to identify instances of drowsiness. Depending on the continuity of eye closures, the system categorizes them as either occasional drowsiness or sustained drowsiness. The advantages of the proposed method lie in its efficiency and real-time capabilities, as it solely relies on IR camera video for computation and analysis. To assess its performance, the system underwent evaluation using IR-Datasets, demonstrating its effectiveness in monitoring and recognizing driver behavior accurately. The presented real-time Driver Monitoring System with facial landmark-based behavior recognition offers a practical and robust approach to enhance driver safety and alertness during their journeys.
Collapse
Affiliation(s)
- Dohun Kim
- Electronics and Telecommunications Research Institute, 22, Daewangpangyo-ro 712beon-gil, Bundang-gu, Seongnam-si, Gyeonggi-do, Republic of Korea
- Department of Image, Chung-Ang University, 84 Heukseok-ro, Seoul, 06974, Korea
| | - Hyukjin Park
- TQS Korea, 406ho, B, Jiphyeonjungang 7-ro, Sejong-si, Korea
| | - Tonghyun Kim
- CANLAB, 604ho, Daewootechnopia 296, Sandan-ro, Danwon-gu, Ansan-si, Gyeonggi-do, Korea
| | - Wonjong Kim
- Electronics and Telecommunications Research Institute, 22, Daewangpangyo-ro 712beon-gil, Bundang-gu, Seongnam-si, Gyeonggi-do, Republic of Korea
| | - Joonki Paik
- Department of Image, Chung-Ang University, 84 Heukseok-ro, Seoul, 06974, Korea.
- Department of Artificial Intelligence, Chung-Ang University, 84 Heukseok-ro, Seoul, 06974, Korea.
| |
Collapse
|
3
|
Liang S, Yin L, Zhang D, Su D, Qu HY. ResNet14Attention network for identifying the titration end-point of potassium dichromate. Heliyon 2023; 9:e18992. [PMID: 37609400 PMCID: PMC10440524 DOI: 10.1016/j.heliyon.2023.e18992] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2023] [Revised: 07/21/2023] [Accepted: 08/04/2023] [Indexed: 08/24/2023] Open
Abstract
With the rapid development of industry, the increasing discharge of sewage causes the detection of water quality to be of increasing importance. Potassium dichromate titration is one of the most important testing methods in water quality detection; the ability to accurately identify the titration end-point of potassium dichromate is currently a research challenge. To identify titration end-point quickly and accurately, this study proposes a ResNet14Attention network, which utilizes residual modules that focus on original image information and an attention mechanism that focuses highly on classification targets. The proposed ResNet14Attention network is compared with 12 convolutional neural networks such as ResNet series networks, VGG, and GoogLeNet. The results of comparison experiments reveal that only the proposed ResNet14Attention network has the highest training and testing accuracy of 100% among all convolutional neural networks in the comparison experiment; the proposed ResNet14Attention network has the highest training speed compared to all the networks that over 90% accuracy.
Collapse
Affiliation(s)
- Siwen Liang
- Guangxi Key Laboratory of Power System Optimization and Energy Technology, Guangxi University, Nanning, Guangxi, 530004, China
| | - Linfei Yin
- Guangxi Key Laboratory of Power System Optimization and Energy Technology, Guangxi University, Nanning, Guangxi, 530004, China
| | - Dashui Zhang
- School of Chemistry and Chemical Engineering, Nanning University, Nanning, Guangxi, 530004, China
| | - Dongwei Su
- Guangxi Key Laboratory of Power System Optimization and Energy Technology, Guangxi University, Nanning, Guangxi, 530004, China
| | - Hui-Ying Qu
- School of Chemistry and Chemical Engineering, Nanning University, Nanning, Guangxi, 530004, China
| |
Collapse
|
4
|
Dilek E, Dener M. Computer Vision Applications in Intelligent Transportation Systems: A Survey. SENSORS (BASEL, SWITZERLAND) 2023; 23:2938. [PMID: 36991649 PMCID: PMC10051529 DOI: 10.3390/s23062938] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/24/2023] [Revised: 03/03/2023] [Accepted: 03/06/2023] [Indexed: 06/19/2023]
Abstract
As technology continues to develop, computer vision (CV) applications are becoming increasingly widespread in the intelligent transportation systems (ITS) context. These applications are developed to improve the efficiency of transportation systems, increase their level of intelligence, and enhance traffic safety. Advances in CV play an important role in solving problems in the fields of traffic monitoring and control, incident detection and management, road usage pricing, and road condition monitoring, among many others, by providing more effective methods. This survey examines CV applications in the literature, the machine learning and deep learning methods used in ITS applications, the applicability of computer vision applications in ITS contexts, the advantages these technologies offer and the difficulties they present, and future research areas and trends, with the goal of increasing the effectiveness, efficiency, and safety level of ITS. The present review, which brings together research from various sources, aims to show how computer vision techniques can help transportation systems to become smarter by presenting a holistic picture of the literature on different CV applications in the ITS context.
Collapse
|
5
|
Li R, Gao R, Suganthan P. A Decomposition-Based Hybrid Ensemble CNN Framework for Driver Fatigue Recognition. Inf Sci (N Y) 2023. [DOI: 10.1016/j.ins.2022.12.088] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
|
6
|
Bekhouche SE, Ruichek Y, Dornaika F. Driver drowsiness detection in video sequences using hybrid selection of deep features. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2022.109436] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
|
7
|
Abstract
Nowadays, many cities have problems with traffic congestion at certain peak hours, which produces more pollution, noise and stress for citizens. Neural networks (NN) and machine-learning (ML) approaches are increasingly used to solve real-world problems, overcoming analytical and statistical methods, due to their ability to deal with dynamic behavior over time and with a large number of parameters in massive data. In this paper, machine-learning (ML) and deep-learning (DL) algorithms are proposed for predicting traffic flow at an intersection, thus laying the groundwork for adaptive traffic control, either by remote control of traffic lights or by applying an algorithm that adjusts the timing according to the predicted flow. Therefore, this work only focuses on traffic flow prediction. Two public datasets are used to train, validate and test the proposed ML and DL models. The first one contains the number of vehicles sampled every five minutes at six intersections for 56 days using different sensors. For this research, four of the six intersections are used to train the ML and DL models. The Multilayer Perceptron Neural Network (MLP-NN) obtained better results (R-Squared and EV score of 0.93) and took less training time, followed closely by Gradient Boosting then Recurrent Neural Networks (RNNs), with good metrics results but the longer training time, and finally Random Forest, Linear Regression and Stochastic Gradient. All ML and DL algorithms scored good performance metrics, indicating that they are feasible for implementation on smart traffic light controllers.
Collapse
|