Distributed Q -Learning-Based Online Optimization Algorithm for Unit Commitment and Dispatch in Smart Grid

Finite-horizon optimal secure tracking control under denial-of-service attacks

The finite-horizon optimal secure tracking control (FHOSTC) problem for cyber-physical systems under actuator denial-of-service (DoS) attacks is addressed in this paper. A model-free method based on the Q-function is designed to achieve FHOSTC without the system model information. First, an augmented time-varying Riccati equation (TVRE) is derived by integrating the system with the reference system into a unified augmented system. Then, a lower bound on malicious DoS attacks probability that guarantees the solutions of the TVRE is provided. Third, a Q-function that changes over time (time-varying Q-function, TVQF) is devised. A TVQF-based method is then proposed to solve the TVRE without the need for the knowledge of the augmented system dynamics. The developed method works backward-in-time and uses the least-squares method. To validate the performance and features of the developed method, simulation studies are conducted in the end.

Distributed Actor-Critic Algorithms for Multiagent Reinforcement Learning Over Directed Graphs

Actor-critic (AC) cooperative multiagent reinforcement learning (MARL) over directed graphs is studied in this article. The goal of the agents in MARL is to maximize the globally averaged return in a distributed way, i.e., each agent can only exchange information with its neighboring agents. AC methods proposed in the literature require the communication graphs to be undirected and the weight matrices to be doubly stochastic (more precisely, the weight matrices are row stochastic and their expectation are column stochastic). Differently from these methods, we propose a distributed AC algorithm for MARL over directed graph with fixed topology that only requires the weight matrix to be row stochastic. Then, we also study the MARL over directed graphs (possibly not connected) with changing topologies, proposing a different distributed AC algorithm based on the push-sum protocol that only requires the weight matrices to be column stochastic. Convergence of the proposed algorithms is proven for linear function approximation of the action value function. Simulations are presented to demonstrate the effectiveness of the proposed algorithms.

Event-Triggered Cardinality-Constrained Cooling and Electrical Load Dispatch Based on Collaborative Neurodynamic Optimization

This article addresses event-triggered optimal load dispatching based on collaborative neurodynamic optimization. Two cardinality-constrained global optimization problems are formulated and two event-triggering functions are defined for event-triggered load dispatching in thermal energy and electric power systems. An event-triggered dispatching method is developed in the collaborative neurodynamic optimization framework with multiple projection neural networks and a meta-heuristic updating rule. Experimental results are elaborated to demonstrate the efficacy and superiority of the approach against many existing methods for optimal load dispatching in air conditioning systems and electric power generation systems.

Energy-Saving Optimization Method of Urban Rail Transit Based on Improved Differential Evolution Algorithm

The transformation of railway infrastructure and traction equipment is an ideal way to realize energy savings of urban rail transit trains. However, upgrading railway infrastructure and traction equipment is a high investment and difficult process. To produce energy-savings in the urban rail transit system without changing the existing infrastructure, we propose an energy-saving optimization method by optimizing the traction curve of the train. Firstly, after analyzing the relationship between the idle distance and running energy-savings, an optimization method of traction energy-savings based on the combination of the inertia motion and energy optimization is established by taking the maximum idle distance as the objective; and the maximum allowable running speed, passenger comfort, train timetable, maximum allowable acceleration and kinematics equation as constraints. Secondly, a solution method based on the combination of the adaptive dynamic multimodal differential evolution algorithm and the Q learning algorithm is applied to solve the optimization model of energy-savings. Finally, numeric experiments are conducted to verify the proposed method. Extensive experiments demonstrate the effectiveness of the proposed method. The results show that the method has significant energy-saving properties, saving energy by about 11.2%.

Distributed Q-Learning Algorithm for Dynamic Resource Allocation With Unknown Objective Functions and Application to Microgrid

Dynamic resource allocation problem (DRAP) with unknown cost functions and unknown resource transition functions is studied in this article. The goal of the agents is to minimize the sum of cost functions over given time periods in a distributed way, that is, by only exchanging information with their neighboring agents. First, we propose a distributed Q -learning algorithm for DRAP with unknown cost functions and unknown resource transition functions under discrete local feasibility constraints (DLFCs). It is theoretically proved that the joint policy of agents produced by the distributed Q -learning algorithm can always provide a feasible allocation (FA), that is, satisfying the constraints at each time period. Then, we also study the DRAP with unknown cost functions and unknown resource transition functions under continuous local feasibility constraints (CLFCs), where a novel distributed Q -learning algorithm is proposed based on function approximation and distributed optimization. It should be noted that the update rule of the local policy of each agent can also ensure that the joint policy of agents is an FA at each time period. Such property is of vital importance to execute the ε -greedy policy during the whole training process. Finally, simulations are presented to demonstrate the effectiveness of the proposed algorithms.

Training Deep Neural Network for Optimal Power Allocation in Islanded Microgrid Systems: A Distributed Learning-Based Approach

Currently, numerical optimization methods are used to solve distributed optimal power allocation (OPA) problems for islanded microgrid (MG) systems. Most of them are developed based on rigorous mathematical derivation. However, the complexity of such optimization algorithms inevitably creates a gap between theoretical analysis and real-time implementation. In order to bridge such a gap, in this article we provide a new distributed learning-based framework to solve the real-time OPA problem. Specifically, inspired by the human-thinking scheme, distributed deep neural networks (DNNs) together with a dynamic average consensus algorithm are first employed to obtain an approximate OPA solution in a distributed manner. Then a distributed balance generation and demand algorithm is designed to fine-tune it to obtain the final optimal feasible solution. In addition, it is theoretically proved that the proposed DNN can well approximate one existing OPA algorithm (Guo et al. 2018), where quantitative numbers of at most how many hidden layers and neurons are provided. Several experimental case studies show that our proposed distributed learning framework can achieve similar optimal results to those obtained by using typical existing distributed numerical optimization methods while it is superior in terms of simplicity and real-time capability.