Deep Reinforcement Learning With Quantum-Inspired Experience Replay

A priority experience replay actor-critic algorithm using self-attention mechanism for strategy optimization of discrete problems

In the dynamic field of deep reinforcement learning, the self-attention mechanism has been increasingly recognized. Nevertheless, its application in discrete problem domains has been relatively limited, presenting complex optimization challenges. This article introduces a pioneering deep reinforcement learning algorithm, termed Attention-based Actor-Critic with Priority Experience Replay (A2CPER). A2CPER combines the strengths of self-attention mechanisms with the Actor-Critic framework and prioritized experience replay to enhance policy formulation for discrete problems. The algorithm's architecture features dual networks within the Actor-Critic model-the Actor formulates action policies and the Critic evaluates state values to judge the quality of policies. The incorporation of target networks aids in stabilizing network optimization. Moreover, the addition of self-attention mechanisms bolsters the policy network's capability to focus on critical information, while priority experience replay promotes training stability and reduces correlation among training samples. Empirical experiments on discrete action problems validate A2CPER's adeptness at policy optimization, marking significant performance improvements across tasks. In summary, A2CPER highlights the viability of self-attention mechanisms in reinforcement learning, presenting a robust framework for discrete problem-solving and potential applicability in complex decision-making scenarios.

Offline Reinforcement Learning With Behavior Value Regularization

Offline reinforcement learning (offline RL) aims to find task-solving policies from prerecorded datasets without online environment interaction. It is unfortunate that extrapolation errors can cause over-optimistic Q-value estimates when learning with a fixed dataset, limiting the performance of the learned policy. To tackle this issue, this article proposes an offline actor-critic with behavior value regularization (OAC-BVR) method. In the policy evaluation stage, the difference between the Q-function and the value of the behavior policy is considered as the regularization term, driving the learned value function to approach the value of the behavior policy. The convergence of the proposed policy evaluation with behavior value regularization (PE-BVR) and the value function difference are analyzed, respectively. Compared with existing offline actor-critic methods, the proposed OAC-BVR method integrates the value of the behavior policy, thereby simultaneously alleviating over-optimistic Q-value estimates and reducing Q-function bias. Experimental results on the D4RL MuJoCo and Maze2d datasets demonstrate the validity of the proposed PE-BVR and the performance advantage of OAC-BVR over the state-of-the-art offline RL algorithms. The code of OAC-BVR is available at https://github.com/LongyangHuang/OAC-BVR.

Prioritized experience replay in path planning via multi-dimensional transition priority fusion

Introduction

Deep deterministic policy gradient (DDPG)-based path planning algorithms for intelligent robots struggle to discern the value of experience transitions during training due to their reliance on a random experience replay. This can lead to inappropriate sampling of experience transitions and overemphasis on edge experience transitions. As a result, the algorithm's convergence becomes slower, and the success rate of path planning diminishes.

Methods

We comprehensively examines the impacts of immediate reward, temporal-difference error (TD-error), and Actor network loss function on the training process. It calculates experience transition priorities based on these three factors. Subsequently, using information entropy as a weight, the three calculated priorities are merged to determine the final priority of the experience transition. In addition, we introduce a method for adaptively adjusting the priority of positive experience transitions to focus on positive experience transitions and maintain a balanced distribution. Finally, the sampling probability of each experience transition is derived from its respective priority.

Results

The experimental results showed that the test time of our method is shorter than that of PER algorithm, and the number of collisions with obstacles is less. It indicated that the determined experience transition priority accurately gauges the significance of distinct experience transitions for path planning algorithm training.

Discussion

This method enhances the utilization rate of transition conversion and the convergence speed of the algorithm and also improves the success rate of path planning.

RIS-Assisted Multi-Antenna AmBC Signal Detection Using Deep Reinforcement Learning

Signal detection is one of the most critical and challenging issues in ambient backscatter communication (AmBC) systems. In this paper, a multi-antenna AmBC signal detection method is proposed based on reconfigurable intelligent surface (RIS) and deep reinforcement learning. Firstly, an efficient multi-antenna AmBC system is developed based on RIS, which can achieve information transmission and energy collection simultaneously. Secondly, a smart twin delayed deep deterministic (TD3) AmBC signal detection method is presented, based on deep reinforcement learning. Extensive quantitative and qualitative experiments are performed, which show that the proposed method is more compelling than the outstanding comparison methods.

AI Makes Crypto Evolve

The recent literature reveals a dichotomy formed by a coevolution between cryptography and Artificial Intelligence (AI). This dichotomy consists of two sides, namely Crypto-Influenced AI (CIAI) and AI-Influenced Cryptography (AIIC). While it is pertinent to investigate this dichotomy from both sides, the first side has already been studied. In this review, we focused on AIIC. We identified and analyzed the stages on the evolutionary path of AIIC. Moreover, we attempted to anticipate what the future may hold for AIIC given the impact of quantum computing on the present and the future of AI.

The Dichotomy of Neural Networks and Cryptography: War and Peace

In recent years, neural networks and cryptographic schemes have come together in war and peace; a cross-impact that forms a dichotomy deserving a comprehensive review study. Neural networks can be used against cryptosystems; they can play roles in cryptanalysis and attacks against encryption algorithms and encrypted data. This side of the dichotomy can be interpreted as a war declared by neural networks. On the other hand, neural networks and cryptographic algorithms can mutually support each other. Neural networks can help improve the performance and the security of cryptosystems, and encryption techniques can support the confidentiality of neural networks. The latter side of the dichotomy can be referred to as the peace. There are, to the best of our knowledge, no current surveys that take a comprehensive look at the many ways neural networks are currently interacting with cryptography. This survey aims to fill that niche by providing an overview on the state of the cross-impact between neural networks and cryptography systems. To this end, this paper will highlight the current areas where progress is being made as well as the aspects where there is room for future research to be conducted.

Chaotic Image Encryption: State-of-the-Art, Ecosystem, and Future Roadmap

Recently, many researchers have been interested in the application of chaos in cryptography. Specifically, numerous research works have been focusing on chaotic image encryption. A comprehensive survey can highlight existing trends and shed light on less-studied topics in the area of chaotic image encryption. In addition to such a survey, this paper studies the main challenges in this field, establishes an ecosystem for chaotic image encryption, and develops a future roadmap for further research in this area.