A Study of Deep CNN-Based Classification of Open and Closed Eyes Using a Visible Light Camera Sensor

Eye-LRCN: A Long-Term Recurrent Convolutional Network for Eye Blink Completeness Detection

Computer vision syndrome causes vision problems and discomfort mainly due to dry eye. Several studies show that dry eye in computer users is caused by a reduction in the blink rate and an increase in the prevalence of incomplete blinks. In this context, this article introduces Eye-LRCN, a new eye blink detection method that also evaluates the completeness of the blink. The method is based on a long-term recurrent convolutional network (LRCN), which combines a convolutional neural network (CNN) for feature extraction with a bidirectional recurrent neural network that performs sequence learning and classifies the blinks. A Siamese architecture is used during CNN training to overcome the high-class imbalance present in blink detection and the limited amount of data available to train blink detection models. The method was evaluated on three different tasks: blink detection, blink completeness detection, and eye state detection. We report superior performance to the state-of-the-art methods in blink detection and blink completeness detection, and remarkable results in eye state detection.

Using cascade CNN-LSTM-FCNs to identify AI-altered video based on eye state sequence

Deep learning is notably successful in data analysis, computer vision, and human control. Nevertheless, this approach has inevitably allowed the development of DeepFake video sequences and images that could be altered so that the changes are not easily or explicitly detectable. Such alterations have been recently used to spread false news or disinformation. This study aims to identify Deepfaked videos and images and alert viewers to the possible falsity of the information. The current work presented a novel means of revealing fake face videos by cascading the convolution network with recurrent neural networks and fully connected network (FCN) models. The system detection approach utilizes the eye-blinking state in temporal video frames. Notwithstanding, it is deemed challenging to precisely depict (i) artificiality in fake videos and (ii) spatial information within the individual frame through this physiological signal. Spatial features were extracted using the VGG16 network and trained with the ImageNet dataset. The temporal features were then extracted in every 20 sequences through the LSTM network. On another note, the pre-processed eye-blinking state served as a probability to generate a novel BPD dataset. This newly-acquired dataset was fed to three models for training purposes with each entailing four, three, and six hidden layers, respectively. Every model constitutes a unique architecture and specific dropout value. Resultantly, the model optimally and accurately identified tampered videos within the dataset. The study model was assessed using the current BPD dataset based on one of the most complex datasets (FaceForensic++) with 90.8% accuracy. Such precision was successfully maintained in datasets that were not used in the training process. The training process was also accelerated by lowering the computation prerequisites.

Deep Learning Approach Based on Residual Neural Network and SVM Classifier for Driver’s Distraction Detection

In the last decade, distraction detection of a driver gained a lot of significance due to increases in the number of accidents. Many solutions, such as feature based, statistical, holistic, etc., have been proposed to solve this problem. With the advent of high processing power at cheaper costs, deep learning-based driver distraction detection techniques have shown promising results. The study proposes ReSVM, an approach combining deep features of ResNet-50 with the SVM classifier, for distraction detection of a driver. ReSVM is compared with six state-of-the-art approaches on four datasets, namely: State Farm Distracted Driver Detection, Boston University, DrivFace, and FT-UMT. Experiments demonstrate that ReSVM outperforms the existing approaches and achieves a classification accuracy as high as 95.5%. The study also compares ReSVM with its variants on the aforementioned datasets.

A novel disassembly process of end-of-life lithium-ion batteries enhanced by online sensing and machine learning techniques

An effective lithium-ion battery (LIB) recycling infrastructure is of great importance to alleviate the concerns over the disposal of waste LIBs and the sustainability of critical elements for producing LIB components. The End-of-life (EOL) LIBs are in various sizes and shapes, which create significant challenges to automate a few unit operations (e.g., disassembly at the cell level) of the recycling process. Meanwhile, hazardous and flammable materials are contained in LIBs, posing great threats to the human exposure. Therefore, it is difficult to dismantle the LIBs safely and efficiently to recover critical materials. Automation has become a competitive solution in manufacturing world, which allows for mass production at outstanding speeds and with great repeatability or quality. It is imperative to develop automatic disassembly solution to effectively disassemble the LIBs while safeguarding human workers against the hazards environment. In this work, we demonstrate an automatic battery disassembly platform enhanced by online sensing and machine learning technologies. The computer vision is used to classify different types of batteries based on their brands and sizes. The real-time temperature data is captured from a thermal camera. A data-driven model is built to predict the cutting temperature pattern and the temperature spike can be mitigated by the close-loop control system. Furthermore, quality control is conducted using a neural network model to detect and mitigate the cutting defects. The integrated disassembly platform can realize the real-time diagnosis and closed-loop control of the cutting process to optimize the cutting quality and improve the safety.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s10845-022-01936-x.

Enabling Intelligent Recovery of Critical Materials from Li-Ion Battery through Direct Recycling Process with Internet-of-Things

The rapid market expansion of Li-ion batteries (LIBs) leads to concerns over the appropriate disposal of hazardous battery waste and the sustainability in the supply of critical materials for LIB production. Technologies and strategies to extend the life of LIBs and reuse the materials have long been sought. Direct recycling is a more effective recycling approach than existing ones with respect to cost, energy consumption, and emissions. This approach has become increasingly more feasible due to digitalization and the adoption of the Internet-of-Things (IoT). To address the question of how IoT could enhance direct recycling of LIBs, we first highlight the importance of direct recycling in tackling the challenges in the supply chain of LIB and discuss the characteristics and application of IoT technologies, which could enhance direct recycling. Finally, we share our perspective on a paradigm where IoT could be integrated into the direct recycling process of LIBs to enhance the efficiency, intelligence, and effectiveness of the recycling process.

Recognition of Blinks Activity Patterns during Stress Conditions Using CNN and Markovian Analysis

This paper investigates eye behaviour through blinks activity during stress conditions. Although eye blinking is a semi-voluntary action, it is considered to be affected by one’s emotional states such as arousal or stress. The blinking rate provides information towards this direction, however, the analysis on the entire eye aperture timeseries and the corresponding blinking patterns provide enhanced information on eye behaviour during stress conditions. Thus, two experimental protocols were established to induce affective states (neutral, relaxed and stress) systematically through a variety of external and internal stressors. The study populations included 24 and 58 participants respectively performing 12 experimental affective trials. After the preprocessing phase, the eye aperture timeseries and the corresponding features were extracted. The behaviour of inter-blink intervals (IBI) was investigated using the Markovian Analysis to quantify incidence dynamics in sequences of blinks. Moreover, Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) network models were employed to discriminate stressed versus neutral tasks per cognitive process using the sequence of IBI. The classification accuracy reached a percentage of 81.3% which is very promising considering the unimodal analysis and the noninvasiveness modality used.

Evaluation of convolutional neural network for recognizing uterine contractions with electrohysterogram

Uterine contraction (UC) activity is commonly used to monitor the approach of labour and delivery. Electrohysterograms (EHGs) have recently been used to monitor UC and distinguish between efficient and inefficient contractions. In this study, we aimed to identify UC in EHG signals using a convolutional neural network (CNN). An open-access database (Icelandic 16-electrode EHG database from 45 pregnant women with 122 recordings, DB1) was used to develop a CNN model, and 14000 segments with a length of 45 s (7000 from UCs and 7000 from non-UCs, which were determined with reference to the simultaneously recorded tocography signals) were manually extracted from the 122 EHG recordings. Five-fold cross-validation was applied to evaluate the ability of the CNN to identify UC based on its sensitivity (SE), specificity (SP), accuracy (ACC), and area under the receiver operating characteristic curve (AUC). The CNN model developed using DB1 was then applied to an independent clinical database (DB2) to further test its generalisation for recognizing UCs. The EHG signals in DB2 were recorded from 20 pregnant women using our multi-channel system, and 308 segments (154 from UCs and 154 from non-UCs) were extracted. The CNN model from five-fold cross-validation achieved average SE, SP, ACC, and AUC of 0.87, 0.98, 0.93, and 0.92 for DB1, and 0.88, 0.97, 0.93, and 0.87 for DB2, respectively. In summary, we demonstrated that CNN could effectively identify UCs using EHG signals and could be used as a tool for monitoring maternal and foetal health.

Faster R-CNN and Geometric Transformation-Based Detection of Driver's Eyes Using Multiple Near-Infrared Camera Sensors

Studies are being actively conducted on camera-based driver gaze tracking in a vehicle environment for vehicle interfaces and analyzing forward attention for judging driver inattention. In existing studies on the single-camera-based method, there are frequent situations in which the eye information necessary for gaze tracking cannot be observed well in the camera input image owing to the turning of the driver's head during driving. To solve this problem, existing studies have used multiple-camera-based methods to obtain images to track the driver's gaze. However, this method has the drawback of an excessive computation process and processing time, as it involves detecting the eyes and extracting the features of all images obtained from multiple cameras. This makes it difficult to implement it in an actual vehicle environment. To solve these limitations of existing studies, this study proposes a method that uses a shallow convolutional neural network (CNN) for the images of the driver's face acquired from two cameras to adaptively select camera images more suitable for detecting eye position; faster R-CNN is applied to the selected driver images, and after the driver's eyes are detected, the eye positions of the camera image of the other side are mapped through a geometric transformation matrix. Experiments were conducted using the self-built Dongguk Dual Camera-based Driver Database (DDCD-DB1) including the images of 26 participants acquired from inside a vehicle and the Columbia Gaze Data Set (CAVE-DB) open database. The results confirmed that the performance of the proposed method is superior to those of the existing methods.

Application of Deep Learning System into the Development of Communication Device for Quadriplegic Patient

Objective

Methods

Results

Conclusion

Etracker: A Mobile Gaze-Tracking System with Near-Eye Display Based on a Combined Gaze-Tracking Algorithm

Eye tracking technology has become increasingly important for psychological analysis, medical diagnosis, driver assistance systems, and many other applications. Various gaze-tracking models have been established by previous researchers. However, there is currently no near-eye display system with accurate gaze-tracking performance and a convenient user experience. In this paper, we constructed a complete prototype of the mobile gaze-tracking system 'Etracker' with a near-eye viewing device for human gaze tracking. We proposed a combined gaze-tracking algorithm. In this algorithm, the convolutional neural network is used to remove blinking images and predict coarse gaze position, and then a geometric model is defined for accurate human gaze tracking. Moreover, we proposed using the mean value of gazes to resolve pupil center changes caused by nystagmus in calibration algorithms, so that an individual user only needs to calibrate it the first time, which makes our system more convenient. The experiments on gaze data from 26 participants show that the eye center detection accuracy is 98% and Etracker can provide an average gaze accuracy of 0.53° at a rate of 30⁻60 Hz.

IrisDenseNet: Robust Iris Segmentation Using Densely Connected Fully Convolutional Networks in the Images by Visible Light and Near-Infrared Light Camera Sensors

The recent advancements in computer vision have opened new horizons for deploying biometric recognition algorithms in mobile and handheld devices. Similarly, iris recognition is now much needed in unconstraint scenarios with accuracy. These environments make the acquired iris image exhibit occlusion, low resolution, blur, unusual glint, ghost effect, and off-angles. The prevailing segmentation algorithms cannot cope with these constraints. In addition, owing to the unavailability of near-infrared (NIR) light, iris recognition in visible light environment makes the iris segmentation challenging with the noise of visible light. Deep learning with convolutional neural networks (CNN) has brought a considerable breakthrough in various applications. To address the iris segmentation issues in challenging situations by visible light and near-infrared light camera sensors, this paper proposes a densely connected fully convolutional network (IrisDenseNet), which can determine the true iris boundary even with inferior-quality images by using better information gradient flow between the dense blocks. In the experiments conducted, five datasets of visible light and NIR environments were used. For visible light environment, noisy iris challenge evaluation part-II (NICE-II selected from UBIRIS.v2 database) and mobile iris challenge evaluation (MICHE-I) datasets were used. For NIR environment, the institute of automation, Chinese academy of sciences (CASIA) v4.0 interval, CASIA v4.0 distance, and IIT Delhi v1.0 iris datasets were used. Experimental results showed the optimal segmentation of the proposed IrisDenseNet and its excellent performance over existing algorithms for all five datasets.

Region Based CNN for Foreign Object Debris Detection on Airfield Pavement

In this paper, a novel algorithm based on convolutional neural network (CNN) is proposed to detect foreign object debris (FOD) based on optical imaging sensors. It contains two modules, the improved region proposal network (RPN) and spatial transformer network (STN) based CNN classifier. In the improved RPN, some extra select rules are designed and deployed to generate high quality candidates with fewer numbers. Moreover, the efficiency of CNN detector is significantly improved by introducing STN layer. Compared to faster R-CNN and single shot multiBox detector (SSD), the proposed algorithm achieves better result for FOD detection on airfield pavement in the experiment.