Reinforcement learning algorithms with function approximation: Recent advances and applications

Integration of IoT-Enabled Technologies and Artificial Intelligence (AI) for Smart City Scenario: Recent Advancements and Future Trends

As the global population grows, and urbanization becomes more prevalent, cities often struggle to provide convenient, secure, and sustainable lifestyles due to the lack of necessary smart technologies. Fortunately, the Internet of Things (IoT) has emerged as a solution to this challenge by connecting physical objects using electronics, sensors, software, and communication networks. This has transformed smart city infrastructures, introducing various technologies that enhance sustainability, productivity, and comfort for urban dwellers. By leveraging Artificial Intelligence (AI) to analyze the vast amount of IoT data available, new opportunities are emerging to design and manage futuristic smart cities. In this review article, we provide an overview of smart cities, defining their characteristics and exploring the architecture of IoT. A detailed analysis of various wireless communication technologies employed in smart city applications is presented, with extensive research conducted to determine the most appropriate communication technologies for specific use cases. The article also sheds light on different AI algorithms and their suitability for smart city applications. Furthermore, the integration of IoT and AI in smart city scenarios is discussed, emphasizing the potential contributions of 5G networks coupled with AI in advancing modern urban environments. This article contributes to the existing literature by highlighting the tremendous opportunities presented by integrating IoT and AI, paving the way for the development of smart cities that significantly enhance the quality of life for urban dwellers while promoting sustainability and productivity. By exploring the potential of IoT, AI, and their integration, this review article provides valuable insights into the future of smart cities, demonstrating how these technologies can positively impact urban environments and the well-being of their inhabitants.

Could artificial intelligence revolutionize the development of nanovectors for gene therapy and mRNA vaccines?

Gene therapy enables the introduction of nucleic acids like DNA and RNA into host cells, and is expected to revolutionize the treatment of a wide range of diseases. This growth has been further accelerated by the discovery of CRISPR/Cas technology, which allows accurate genomic editing in a broad range of cells and organisms in vitro and in vivo. Despite many advances in gene delivery and the development of various viral and non-viral gene delivery vectors, the lack of highly efficient non-viral systems with low cellular toxicity remains a challenge. The application of cutting-edge technologies such as artificial intelligence (AI) has great potential to find new paradigms to solve this issue. Herein, we review AI and its major subfields including machine learning (ML), neural networks (NNs), expert systems, deep learning (DL), computer vision and robotics. We discuss the potential of AI-based models and algorithms in the design of targeted gene delivery vehicles capable of crossing extracellular and intracellular barriers by viral mimicry strategies. We finally discuss the role of AI in improving the function of CRISPR/Cas systems, developing novel nanobots, and mRNA vaccine carriers.

Autonomous Navigation of Unmanned Aircraft Using Space Target LOS Measurements and QLEKF

An autonomous navigation method based on the fusion of INS (inertial navigation system) measurements with the line-of-sight (LOS) observations of space targets is presented for unmanned aircrafts. INS/GNSS (global navigation satellite system) integration is the conventional approach to achieving the long-term and high-precision navigation of unmanned aircrafts. However, the performance of INS/GNSS integrated navigation may be degraded gradually in a GNSS-denied environment. INS/CNS (celestial navigation system) integrated navigation has been developed as a supplement to the GNSS. A limitation of traditional INS/CNS integrated navigation is that the CNS is not efficient in suppressing the position error of the INS. To solve the abovementioned problems, we studied a novel integrated navigation method, where the position, velocity and attitude errors of the INS were corrected using a star camera mounted on the aircraft in order to observe the space targets whose absolute positions were available. Additionally, a QLEKF (Q-learning extended Kalman filter) is designed for the performance enhancement of the integrated navigation system. The effectiveness of the presented autonomous navigation method based on the star camera and the IMU (inertial measurement unit) is demonstrated via CRLB (Cramer-Rao lower bounds) analysis and numerical simulations.

A Reinforcement Learning Control in Hot Stamping for Cycle Time Optimization

Hot stamping is a hot metal forming technology increasingly in demand that produces ultra-high strength parts with complex shapes. A major concern in these systems is how to shorten production times to improve production Key Performance Indicators. In this work, we present a Reinforcement Learning approach that can obtain an optimal behavior strategy for dynamically managing the cycle time in hot stamping to optimize manufacturing production while maintaining the quality of the final product. Results are compared with the business-as-usual cycle time control approach and the optimal solution obtained by the execution of a dynamic programming algorithm. Reinforcement Learning control outperforms the business-as-usual behavior by reducing the cycle time and the total batch time in non-stable temperature phases.

Combining CNN and Q-learning for increasing the accuracy of lost gamma source finding

The increasing use of nuclear technology in various fields makes it necessary to provide the required safety to work with this industry. Gamma source is one of the most widely used sources in industry and medicine. Finding a lost gamma source in a gamma irradiation room without human presence is challenging due to the particular arrangements and barriers in the room for radiation shielding and requires an efficient and robust method. In this paper, locating and routing the lost gamma source in the gamma irradiation room containing radiation blocking barriers are done simultaneously by using two methods, convolutional neural network (CNN) and Q-learning, which are powerful algorithms for deep learning and machine learning. Environment simulation with gamma source was performed using Geant4 simulation. The results show that by combining these two methods in geometries with radiation blocking barriers, in addition to locating with 90% accuracy, routing can also be performed. Although the presence of thick barriers in the room reduces the accuracy, increases the time required to finding the lost gamma source or the inefficiency of other methods, nevertheless, the results show that combination of CNN and Q-learning reduces the time and greatly increases the accuracy.

Intelligent scaling for 6G IoE services for resource provisioning

The proposed research motivates the 6G cellular networking for the Internet of Everything's (IoE) usage empowerment that is currently not compatible with 5G. For 6G, more innovative technological resources are required to be handled by Mobile Edge Computing (MEC). Although the demand for change in service from different sectors, the increase in IoE, the limitation of available computing resources of MEC, and intelligent resource solutions are getting much more significant. This research used IScaler, an effective model for intelligent service placement solutions and resource scaling. IScaler is considered to be made for MEC in Deep Reinforcement Learning (DRL). The paper has considered several requirements for making service placement decisions. The research also highlights several challenges geared by architectonics that submerge an Intelligent Scaling and Placement module.

The Missing Link Between Memory and Reinforcement Learning

Reinforcement learning systems usually assume that a value function is defined over all states (or state-action pairs) that can immediately give the value of a particular state or action. These values are used by a selection mechanism to decide which action to take. In contrast, when humans and animals make decisions, they collect evidence for different alternatives over time and take action only when sufficient evidence has been accumulated. We have previously developed a model of memory processing that includes semantic, episodic and working memory in a comprehensive architecture. Here, we describe how this memory mechanism can support decision making when the alternatives cannot be evaluated based on immediate sensory information alone. Instead we first imagine, and then evaluate a possible future that will result from choosing one of the alternatives. Here we present an extended model that can be used as a model for decision making that depends on accumulating evidence over time, whether that information comes from the sequential attention to different sensory properties or from internal simulation of the consequences of making a particular choice. We show how the new model explains both simple immediate choices, choices that depend on multiple sensory factors and complicated selections between alternatives that require forward looking simulations based on episodic and semantic memory structures. In this framework, vicarious trial and error is explained as an internal simulation that accumulates evidence for a particular choice. We argue that a system like this forms the "missing link" between more traditional ideas of semantic and episodic memory, and the associative nature of reinforcement learning.

Integrating Machine Learning with Human Knowledge

Machine learning has been heavily researched and widely used in many disciplines. However, achieving high accuracy requires a large amount of data that is sometimes difficult, expensive, or impractical to obtain. Integrating human knowledge into machine learning can significantly reduce data requirement, increase reliability and robustness of machine learning, and build explainable machine learning systems. This allows leveraging the vast amount of human knowledge and capability of machine learning to achieve functions and performance not available before and will facilitate the interaction between human beings and machine learning systems, making machine learning decisions understandable to humans. This paper gives an overview of the knowledge and its representations that can be integrated into machine learning and the methodology. We cover the fundamentals, current status, and recent progress of the methods, with a focus on popular and new topics. The perspectives on future directions are also discussed.

INVERSE PROBLEM ALGORITHM APPLICATION TO SEMI-QUANTITATIVE ANALYSIS OF 272 PATIENTS WITH ISCHEMIC STROKE SYMPTOMS: CAROTID STENOSIS RISK ASSESSMENT FOR FIVE RISK FACTORS

This study proposes the inverse problem algorithm (IPA) with five risk factors applied to the semi-quantitative analysis of carotid stenosis 272 patients with suspected ischemic stroke. The IPA is known to provide a substantiated machine learning-based prediction of the expected outcomes by solving an inverse matrix of variable coefficients. In case of carotid stenosis prediction, such risk factors as patient’s age, mean arterial pressure (MAP), glucose AC, low-density lipoprotein-cholesterol (LDL-C), and C-Reactive protein (CRP) were assessed for the main group of 217 patients. Their results were processed by the STATISTICA program with a customized loss function ([Formula: see text]), yielding the first-order nonlinear semi-empirical formula with 16 terms. The loss function was calculated via the total mismatch between the theoretical predictions and true carotid stenosis cases (%) for all 217 patients. Thus, the carotid stenosis (%) compromised solution array [[Formula: see text]] was optimized using [Formula: see text] individual data points via the proposed algorithm. The results showed a complete regression with loss function [Formula: see text]=2.3543, variance [Formula: see text]=87.46%, and correlation coefficient [Formula: see text]. The reference group of 55 more patients with the same preliminary diagnosis and symptoms was selected to validate the method predictive feasibility, which was found quite satisfactory. The decreasing order of three dominant risk factors was as follows: CRP, glucose AC, and MAP, whereas age and LDL-C weakly influenced the program computation results. The IPA showed a strong convergence by its default characteristic. The reduction of the number of variables in computation deteriorated the prediction accuracy, exhibiting the algorithm’s high sensitivity to the number of variables.

A Q-learning approach based on human reasoning for navigation in a dynamic environment

A Q-learning approach is often used for navigation in static environments where state space is easy to define. In this paper, a new Q-learning approach is proposed for navigation in dynamic environments by imitating human reasoning. As a model-free method, a Q-learning method does not require the environmental model in advance. The state space and the reward function in the proposed approach are defined according to human perception and evaluation, respectively. Specifically, approximate regions instead of accurate measurements are used to define states. Moreover, due to the limitation of robot dynamics, actions for each state are calculated by introducing a dynamic window that takes robot dynamics into account. The conducted tests show that the obstacle avoidance rate of the proposed approach can reach 90.5% after training, and the robot can always operate below the dynamics limitation.

Neural circuits for learning context-dependent associations of stimuli

The use of reinforcement learning combined with neural networks provides a powerful framework for solving certain tasks in engineering and cognitive science. Previous research shows that neural networks have the power to automatically extract features and learn hierarchical decision rules. In this work, we investigate reinforcement learning methods for performing a context-dependent association task using two kinds of neural network models (using continuous firing rate neurons), as well as a neural circuit gating model. The task allows examination of the ability of different models to extract hierarchical decision rules and generalize beyond the examples presented to the models in the training phase. We find that the simple neural circuit gating model, trained using response-based regulation of Hebbian associations, performs almost at the same level as a reinforcement learning algorithm combined with neural networks trained with more sophisticated back-propagation of error methods. A potential explanation is that hierarchical reasoning is the key to performance and the specific learning method is less important.

Adaptive Critic Design with Local Gaussian Process Models

In this paper, local Gaussian process (GP) approximation is introduced to build the critic network of adaptive dynamic programming (ADP). The sample data are partitioned into local regions, and for each region, an individual GP model is utilized. The nearest local model is used to predict a given state-action point. With the two-phase value iteration method for a Gaussian-kernel (GK)-based critic network which realizes the update of the hyper-parameters and value functions simultaneously, fast value function approximation can be achieved. Combining this critic network with an actor network, we present a local GK-based ADP approach. Simulations were carried out to demonstrate the feasibility of the proposed approach.

Near-Optimal Tracking Control of Mobile Robots Via Receding-Horizon Dual Heuristic Programming

Trajectory tracking control of wheeled mobile robots (WMRs) has been an important research topic in control theory and robotics. Although various tracking control methods with stability have been developed for WMRs, it is still difficult to design optimal or near-optimal tracking controller under uncertainties and disturbances. In this paper, a near-optimal tracking control method is presented for WMRs based on receding-horizon dual heuristic programming (RHDHP). In the proposed method, a backstepping kinematic controller is designed to generate desired velocity profiles and the receding horizon strategy is used to decompose the infinite-horizon optimal control problem into a series of finite-horizon optimal control problems. In each horizon, a closed-loop tracking control policy is successively updated using a class of approximate dynamic programming algorithms called finite-horizon dual heuristic programming (DHP). The convergence property of the proposed method is analyzed and it is shown that the tracking control system based on RHDHP is asymptotically stable by using the Lyapunov approach. Simulation results on three tracking control problems demonstrate that the proposed method has improved control performance when compared with conventional model predictive control (MPC) and DHP. It is also illustrated that the proposed method has lower computational burden than conventional MPC, which is very beneficial for real-time tracking control.

Unregistered biological words recognition by Q-learning with transfer learning

Unregistered biological words recognition is the process of identification of terms that is out of vocabulary. Although many approaches have been developed, the performance approaches are not satisfactory. As the identification process can be viewed as a Markov process, we put forward a Q-learning with transfer learning algorithm to detect unregistered biological words from texts. With the Q-learning, the recognizer can attain the optimal solution of identification during the interaction with the texts and contexts. During the processing, a transfer learning approach is utilized to fully take advantage of the knowledge gained in a source task to speed up learning in a different but related target task. A mapping, required by many transfer learning, which relates features from the source task to the target task, is carried on automatically under the reinforcement learning framework. We examined the performance of three approaches with GENIA corpus and JNLPBA04 data. The proposed approach improved performance in both experiments. The precision, recall rate, and F score results of our approach surpassed those of conventional unregistered word recognizer as well as those of Q-learning approach without transfer learning.