A supervised Actor–Critic approach for adaptive cruise control

Neural H₂ Control Using Continuous-Time Reinforcement Learning

In this article, we discuss continuous-time H₂ control for the unknown nonlinear system. We use differential neural networks to model the system, then apply the H₂ tracking control based on the neural model. Since the neural H₂ control is very sensitive to the neural modeling error, we use reinforcement learning to improve the control performance. The stabilities of the neural modeling and the H₂ tracking control are proven. The convergence of the approach is also given. The proposed method is validated with two benchmark control problems.

Missile guidance with assisted deep reinforcement learning for head-on interception of maneuvering target

In missile guidance, pursuit performance is seriously degraded due to the uncertainty and randomness in target maneuverability, detection delay, and environmental noise. In many methods, accurately estimating the acceleration of the target or the time-to-go is needed to intercept the maneuvering target, which is hard in an environment with uncertainty. In this paper, we propose an assisted deep reinforcement learning (ARL) algorithm to optimize the neural network-based missile guidance controller for head-on interception. Based on the relative velocity, distance, and angle, ARL can control the missile to intercept the maneuvering target and achieve large terminal intercept angle. To reduce the influence of environmental uncertainty, ARL predicts the target’s acceleration as an auxiliary supervised task. The supervised learning task improves the ability of the agent to extract information from observations. To exploit the agent’s good trajectories, ARL presents the Gaussian self-imitation learning to make the mean of action distribution approach the agent’s good actions. Compared with vanilla self-imitation learning, Gaussian self-imitation learning improves the exploration in continuous control. Simulation results validate that ARL outperforms traditional methods and proximal policy optimization algorithm with higher hit rate and larger terminal intercept angle in the simulation environment with noise, delay, and maneuverable target.

Discrete-Time H2 Neural Control Using Reinforcement Learning

In this article, we discuss H₂ control for unknown nonlinear systems in discrete time. A discrete-time recurrent neural network is used to model the nonlinear system, and then, the H₂ tracking control is applied based on the neural model. Since this neural H₂ control is very sensitive to the neural modeling error, we use reinforcement learning and another neural approximator to improve tracking accuracy and robustness of the controller. The stabilities of the neural identifier and the H₂ tracking control are proven. The convergence of the approach is also given. The proposed method is validated with the control of the pan and tilt robot and the surge tank.

Application of Artificial Neural Networks to Streamline the Process of Adaptive Cruise Control

This article deals with the use of neural networks for estimation of deceleration model parameters for the adaptive cruise control unit. The article describes the basic functionality of adaptive cruise control and creates a mathematical model of braking, which is one of the basic functions of adaptive cruise control. Furthermore, an analysis of the influences acting in the braking process is performed, the most significant of which are used in the design of deceleration prediction for the adaptive cruise control unit using neural networks. Such a connection using artificial neural networks using modern sensors can be another step towards full vehicle autonomy. The advantage of this approach is the original use of neural networks, which refines the determination of the deceleration value of the vehicle in front of a static or dynamic obstacle, while including a number of influences that affect the braking process and thus increase driving safety.

NN Reinforcement Learning Adaptive Control for a Class of Nonstrict-Feedback Discrete-Time Systems

This article investigates an adaptive reinforcement learning (RL) optimal control design problem for a class of nonstrict-feedback discrete-time systems. Based on the neural network (NN) approximating ability and RL control design technique, an adaptive backstepping RL optimal controller and a minimal learning parameter (MLP) adaptive RL optimal controller are developed by establishing a novel strategic utility function and introducing external function terms. It is proved that the proposed adaptive RL optimal controllers can guarantee that all signals in the closed-loop systems are semiglobal uniformly ultimately bounded (SGUUB). The main feature is that the proposed schemes can solve the optimal control problem that the previous literature cannot deal with. Furthermore, the proposed MPL adaptive optimal control scheme can reduce the number of adaptive laws, and thus the computational complexity is decreased. Finally, the simulation results illustrate the validity of the proposed optimal control schemes.

Design of a Reinforcement Learning-Based Lane Keeping Planning Agent for Automated Vehicles

Reinforcement learning-based approaches are widely studied in the literature for solving different control tasks for Connected and Autonomous Vehicles, from which this paper deals with the problem of lateral control of a dynamic nonlinear vehicle model, performing the task of lane-keeping. In this area, the appropriate formulation of the goals and environment information is crucial, for which the research outlines the importance of lookahead information, enabling to accomplish maneuvers with complex trajectories. Another critical part is the real-time manner of the problem. On the one hand, optimization or search based methods, such as the presented Monte Carlo Tree Search method, can solve the problem with the trade-off of high numerical complexity. On the other hand, single Reinforcement Learning agents struggle to learn these tasks with high performance, though they have the advantage that after the training process, they can operate in a real-time manner. Two planning agent structures are proposed in the paper to resolve this duality, where the machine learning agents aid the tree search algorithm. As a result, the combined solution provides high performance and low computational needs.

Nucleus and cytoplasm-based segmentation and actor-critic neural network for acute lymphocytic leukaemia detection in single cell blood smear images

Acute lymphoblastic leukaemia (ALL), which is due to the malfunctioning in the bone marrow, is common among people all over the world. The haematologist suffers a lot to discriminate the presence of leukaemia in the patients using the blood smears. To overcome the inaccuracy and reliability issues, this paper proposes an automatic method of leukaemia detection, named chronological Sine Cosine Algorithm-based actor-critic neural network (Chrono-SCA-ACNN). Initially, the blood smear images are segmented using the proposed entropy-based hybrid model, from which the image-level features and statistical features are extracted from the segments. Then, the selected features are applied to the proposed classifier, which detects the leukaemia. In the proposed Chrono-SCA-ACNN, the optimal weights are selected by the proposed Chrono-SCA, which is the integration of the chronological concept in the SCA. Finally, the experimentation is performed using the ALL-IDB2 database, and the effectiveness of the proposed method over the existing methods is evaluated. From the analysis, the accuracy of the proposed method is found to be 0.99, which proves that it outperforms the existing classification methodologies. Graphical abstract Block diagram of proposed Leukaemia detection. The main aim of the paper is to segment and classify the WBCs for ALL detection in single cell blood smear images. Initially, the blood smear image is subjected to pre-processing in order to enhance the quality of the input image so as to make it effective for the further processes associated with Leukaemia detection. The pre-processed image is applied to the segmentation process that segments the cytoplasm and nucleus using the Entropy-based hybrid model. The entropy-based hybrid model is developed using the FCM and active contour to segment the cytoplasm and nucleus that is fused using the entropy. The segments are subjected to the feature extraction that extracts the statistical features and the color histogram-based features from the segments. The features are presented to the Actor-Critic Neural Network and the weights of the Neural Network (NN) are optimally tuned using the proposed Chrono-SCA. The block diagram of the proposed method of leukaemia detection is depicted in Fig. 1.

Optimization driven actor-critic neural network for sentiment analysis in social media

Purpose

Sentiment analysis is the subfield of data mining, which is profusely used for studying the opinions of the users by analyzing their suggestions on the Web platform. It plays an important role in the daily decision-making process, and every decision has a great impact on daily life. Various techniques including machine learning algorithms have been proposed for sentiment analysis, but still, they are inefficient for extracting the sentiment features from the given text. Although the improvement in sentiment analysis approaches, there are several problems, which make the analysis inefficient and inaccurate. This paper aims to develop the sentiment analysis scheme on movie reviews by proposing a novel classifier.

Design/methodology/approach

For the analysis, the movie reviews are collected and subjected to pre-processing. From the pre-processed review, a total of nine sentiment related features are extracted and provided to the proposed exponential-salp swarm algorithm based actor-critic neural network (ESSA-ACNN) classifier for the sentiment classification. The ESSA algorithm is developed by integrating the exponentially weighted moving average (EWMA) and SSA for selecting the optimal weight of ACNN. Finally, the proposed classifier classifies the reviews into positive or negative class.

Findings

The performance of the ESSA-ACNN classifier is analyzed by considering the reviews present in the movie review database. From, the simulation results, it is evident that the proposed ESSA-ACNN classifier has improved performance than the existing works by having the performance of 0.7417, 0.8807 and 0.8119, for sensitivity, specificity and accuracy, respectively.

Originality/value

The proposed classifier can be applicable for real-world problems, such as business, political activities and so on.

Mutual Information based hybrid model and deep learning for Acute Lymphocytic Leukemia detection in single cell blood smear images

BACKGROUND AND OBJECTIVE

Due to the development in digital microscopic imaging, image processing and classification has become an interesting area for diagnostic research. Various techniques are available in the literature for the detection of Acute Lymphocytic Leukemia from the single cell blood smear images. The purpose of this work is to develop an effective method for leukemia detection.

METHODS

This work has developed deep learning based leukemia detection module from the blood smear images. Here, the detection scheme carries out pre-processing, segmentation, feature extraction and classification. The segmentation is done by the proposed Mutual Information (MI) based hybrid model, which combines the segmentation results of the active contour model and fuzzy C means algorithm. Then, from the segmented images, the statistical and the Local Directional Pattern (LDP) features are extracted and provided to the proposed Chronological Sine Cosine Algorithm (SCA) based Deep CNN classifier for the classification.

RESULTS

For the experimentation, the blood smear images are considered from the AA-IDB2 database and evaluated based on metrics, such as True Positive Rate (TPR), True Negative Rate (TNR), and accuracy. Simulation results reveal that the proposed Chronological SCA based Deep CNN classifier has the accuracy of 98.7%.

CONCLUSIONS

The performance of the proposed Chronological SCA-based Deep CNN classifier is compared with the state-of-the-art methods. The analysis shows that the proposed classifier has comparatively improved performance and determines the leukemia from the blood smear images.

A synergetic strategy of automobile intelligent cruise system based on fuzzy control adopting hierarchical structure

To improve automobile travel safety and relieve the intension of driving fatigue, a synergetic strategy on the throttle and brake combined control of automobile cruise system is proposed based on fuzzy control theory. A hierarchical control structure, including the upper fuzzy controller and the lower model match controller, is designed for integrated control function. The upper controller consists of the two input variables, that is, the deviation of the theoretical safe distance and the relative distance and the relative velocity between the host car and the preceding vehicle, and the single output variable, that is, the excepted acceleration of host car. The lower controller has achieved the fast compensation of excepted acceleration by adopting parallel feedforward and feedback compensator to input into the inverse dynamics model of automobile cruise system. Using MATLAB/Simulink, the control function on the proposed synergetic strategy is validated for different working cases. The results show that the synergetic strategy possesses an integrated capability involved automated stop & go control, and following the preceding car to prevent rear-end collision, and the constant velocity cruise whether the driving conditions of the city road or expressway.

Taylor and Gradient Descent-Based Actor Critic Neural Network for the Classification of Privacy Preserved Medical Data

Classification of the privacy preserved medical data is the domain of the researchers as it stirs the importance behind hiding the sensitive data from the third-party authenticator. Ensuring the privacy of the medical records and using the disease prediction mechanisms played a remarkable role in peoples' lives such that the earlier detection of the diseases is required for earlier diagnosis. Accordingly, this article proposes a method, named Taylor gradient descent (TGD)-based actor critic neural network (ACNN), which concentrates on performing the medical data classification. Initially, the privacy of the medical data is ensured by using the key matrix developed based on the privacy utility coefficient matrix using the chronological-Whale optimization algorithm. The privacy protected data are subjected to classification by using ACNN that performs the optimal classification using the proposed TGD algorithm. The proposed TGD algorithm is the integration of Taylor series in the gradient descent algorithm that updates the optimal weight of ACNN based on the weights in the previous iterations. The analysis using the Cleveland, Switzerland, and Hungarian dataset proves that the proposed classification strategy obtains an accuracy of 0.9252, a sensitivity of 0.8419, and a specificity of 0.8387, respectively.

Abdominal-Waving Control of Tethered Bumblebees Based on Sarsa With Transformed Reward

Cyborg insects have attracted great attention as the flight performance they have is incomparable by micro aerial vehicles and play a critical role in supporting extensive applications. Approaches to construct cyborg insects consist of two major issues: 1) the stimulating paradigm and 2) the control policy. At present, most cyborg insects are constructed based on invasive methods, requiring the implantation of electrodes into neural or muscle systems, which would harm the insects. As the control policy is basically manual control, the shortcomings of which lie in the requirement of excessive amount of experiments and focused attention. This paper presents the design and implementation of a noninvasive and much safer cyborg insect system based on visual stimulation. The tethered paradigm is adopted here and we look at controlling the flight behavior of bumblebees, especially the abdominal-waving behavior, in the context of a model-free reinforcement learning problem. The problem is formulated as a finite and deterministic Markov decision process, where the agent is designed to change the abdominal-waving behavior from the initial state to the target state. Sarsa with transformed reward function which can speed up the learning process is employed to learn the optimal control policy. Learned policies are compared to the stochastic one by evaluating the results of ten bumblebees, demonstrating that abdominal-waving state can be modulated to approximate the target state quickly with small deviation.

Fractional Krill–Lion Algorithm Based Actor Critic Neural Network for Face Recognition in Real Time Surveillance Videos

Face recognition from low quality videos is one of the major challenges prevailing in video surveillance system. Several works have been contributed towards the face recognition, but have suffered due to the fact that the low quality videos have face part with low resolution. Also, using traditional feature extraction schemes make the recognition process to be tough. This paper introduces a novel feature descriptor and classification scheme for recognizing the face from low quality videos. Here, the video frames are sequentially provided to the Viola–Jones algorithm for detecting the face part, and the quality of the face part is improved by applying bi-cubic interpolation based super resolution scheme. Now, the features of enhanced video frame are extracted using proposed local direction feature descriptor, namely scattering wavelet-based local directional pattern (SW-LDP). Then, the extracted features are fed as the input to the actor critic neural network, where training is done using the newly developed fractional calculus based krill–lion (fractional KL) algorithm. The proposed fractional KL-ACNN algorithm is experimented using the standard FAMED database. From the analysis, it is evident that the proposed classifier achieved low FAR of 3.89%, and low FRR of 4.04%, and high accuracy value of 95%, respectively.

Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics

This work suggests a solution for the output reference model (ORM) tracking control problem, based on approximate dynamic programming. General nonlinear systems are included in a control system (CS) and subjected to state feedback. By linear ORM selection, indirect CS feedback linearization is obtained, leading to favorable linear behavior of the CS. The Value Iteration (VI) algorithm ensures model-free nonlinear state feedback controller learning, without relying on the process dynamics. From linear to nonlinear parameterizations, a reliable approximate VI implementation in continuous state-action spaces depends on several key parameters such as problem dimension, exploration of the state-action space, the state-transitions dataset size, and a suitable selection of the function approximators. Herein, we find that, given a transition sample dataset and a general linear parameterization of the Q-function, the ORM tracking performance obtained with an approximate VI scheme can reach the performance level of a more general implementation using neural networks (NNs). Although the NN-based implementation takes more time to learn due to its higher complexity (more parameters), it is less sensitive to exploration settings, number of transition samples, and to the selected hyper-parameters, hence it is recommending as the de facto practical implementation. Contributions of this work include the following: VI convergence is guaranteed under general function approximators; a case study for a low-order linear system in order to generalize the more complex ORM tracking validation on a real-world nonlinear multivariable aerodynamic process; comparisons with an offline deep deterministic policy gradient solution; implementation details and further discussions on the obtained results.

Data-Driven Model-Free Tracking Reinforcement Learning Control with VRFT-based Adaptive Actor-Critic

This paper proposes a neural network (NN)-based control scheme in an Adaptive Actor-Critic (AAC) learning framework designed for output reference model tracking, as a representative deep-learning application. The control learning scheme is model-free with respect to the process model. AAC designs usually require an initial controller to start the learning process; however, systematic guidelines for choosing the initial controller are not offered in the literature, especially in a model-free manner. Virtual Reference Feedback Tuning (VRFT) is proposed for obtaining an initially stabilizing NN nonlinear state-feedback controller, designed from input-state-output data collected from the process in open-loop setting. The solution offers systematic design guidelines for initial controller design. The resulting suboptimal state-feedback controller is next improved under the AAC learning framework by online adaptation of a critic NN and a controller NN. The mixed VRFT-AAC approach is validated on a multi-input multi-output nonlinear constrained coupled vertical two-tank system. Discussions on the control system behavior are offered together with comparisons with similar approaches.

MEC--a near-optimal online reinforcement learning algorithm for continuous deterministic systems

In this paper, the first probably approximately correct (PAC) algorithm for continuous deterministic systems without relying on any system dynamics is proposed. It combines the state aggregation technique and the efficient exploration principle, and makes high utilization of online observed samples. We use a grid to partition the continuous state space into different cells to save samples. A near-upper Q operator is defined to produce a near-upper Q function using samples in each cell. The corresponding greedy policy effectively balances between exploration and exploitation. With the rigorous analysis, we prove that there is a polynomial time bound of executing nonoptimal actions in our algorithm. After finite steps, the final policy reaches near optimal in the framework of PAC. The implementation requires no knowledge of systems and has less computation complexity. Simulation studies confirm that it is a better performance than other similar PAC algorithms.