On the improvement of the real time recurrent learning algorithm for recurrent neural networks

Design and Optimization of a Neuro-Fuzzy System for the Control of an Electromechanical Plant

One characteristic of neuro-fuzzy systems is the possibility of incorporating preliminary information in their structure as well as being able to establish an initial configuration to carry out the training. In this regard, the strategy to establish the configuration of the fuzzy system is a relevant aspect. This document displays the design and implementation of a neuro-fuzzy controller based on Boolean relations to regulate the angular position in an electromechanical plant, composed by a motor coupled to inertia with friction (a widely studied plant that serves to show the control system design process). The structure of fuzzy systems based on Boolean relations considers the operation of sensors and actuators present in the control system. In this way, the initial configuration of fuzzy controller can be determined. In order to perform the optimization of the neuro-fuzzy controller, the continuous plant model is converted to discrete time to be included in the closed-loop controller training equations. For the design process, first the optimization of a Proportional Integral (PI) linear controller is carried out. Thus, linear controller parameters are employed to establish the structure and initial configuration of the neuro-fuzzy controller. The optimization process also includes weighting factors for error and control action in such a way that allows having different system responses. Considering the structure of the control system, the optimization algorithm (training algorithm) employed is dynamic back propagation. The results via simulations show that optimization is achieved in the linear and neuro-fuzzy controllers using different weighting values for the error signal and control action. It is also observed that the proposed control strategy allows disturbance rejection.

Optimization of a Fuzzy Automatic Voltage Controller Using Real-Time Recurrent Learning

The automatic voltage regulator is an important component in energy generation systems; therefore, the tuning of this system is a fundamental aspect for the suitable energy conversion. This article shows the optimization of a fuzzy automatic voltage controller for a generation system using real-time recurrent learning, which is a technique conventionally used for the training of recurrent neural networks. The controller used consists of a compact fuzzy system based on Boolean relations, designed having equivalences with PI, PD, PID, and second order controllers. For algorithm implementation, the training equations are deduced considering the structure of the second order compact fuzzy controller. The results show that a closed-loop fuzzy control strategy was successfully implemented using real-time recurrent learning. In order to implement the controllers optimization, different weighting values for error and control action are used. The results show the behavior of the configurations used and its performance considering the steady state error, overshoot, and settling time.

A new gene regulatory network model based on BP algorithm for interrogating differentially expressed genes of Sea Urchin

Background

Computer science and mathematical theories are combined to analyze the complex interactions among genes, which are simplified to a network to establish a theoretical model for the analysis of the structure, module and dynamic properties. In contrast, traditional model of gene regulatory networks often lack an effective method for solving gene expression data because of high durational and spatial complexity. In this paper, we propose a new model for constructing gene regulatory networks using back propagation (BP) neural network based on predictive function and network topology.

Results

Combined with complex nonlinear mapping and self-learning, the BP neural network was mapped into a complex network. Network characteristics were obtained from the parameters of the average path length, average clustering coefficient, average degree, modularity, and map’s density to simulate the real gene network by an artificial network. Through the statistical analysis and comparison of network parameters of Sea Urchin mRNA microarray data under different temperatures, the value of network parameters was observed. Differentially expressed Sea Urchin genes associated with temperature were determined by calculating the difference in the degree of each gene from different networks.

Conclusion

The new model we developed is suitable to simulate gene regulatory network and has capability of determining differentially expressed genes.

Learning in fully recurrent neural networks by approaching tangent planes to constraint surfaces

In this paper we present a new variant of the online real time recurrent learning algorithm proposed by Williams and Zipser (1989). Whilst the original algorithm utilises gradient information to guide the search towards the minimum training error, it is very slow in most applications and often gets stuck in local minima of the search space. It is also sensitive to the choice of learning rate and requires careful tuning. The new variant adjusts weights by moving to the tangent planes to constraint surfaces. It is simple to implement and requires no parameters to be set manually. Experimental results show that this new algorithm gives significantly faster convergence whilst avoiding problems like local minima.

Low-complexity nonlinear adaptive filter based on a pipelined bilinear recurrent neural network

To reduce the computational complexity of the bilinear recurrent neural network (BLRNN), a novel low-complexity nonlinear adaptive filter with a pipelined bilinear recurrent neural network (PBLRNN) is presented in this paper. The PBLRNN, inheriting the modular architectures of the pipelined RNN proposed by Haykin and Li, comprises a number of BLRNN modules that are cascaded in a chained form. Each module is implemented by a small-scale BLRNN with internal dynamics. Since those modules of the PBLRNN can be performed simultaneously in a pipelined parallelism fashion, it would result in a significant improvement of computational efficiency. Moreover, due to nesting module, the performance of the PBLRNN can be further improved. To suit for the modular architectures, a modified adaptive amplitude real-time recurrent learning algorithm is derived on the gradient descent approach. Extensive simulations are carried out to evaluate the performance of the PBLRNN on nonlinear system identification, nonlinear channel equalization, and chaotic time series prediction. Experimental results show that the PBLRNN provides considerably better performance compared to the single BLRNN and RNN models.

Pipelined chebyshev functional link artificial recurrent neural network for nonlinear adaptive filter

A novel nonlinear adaptive filter with pipelined Chebyshev functional link artificial recurrent neural network (PCFLARNN) is presented in this paper, which uses a modification real-time recurrent learning algorithm. The PCFLARNN consists of a number of simple small-scale Chebyshev functional link artificial recurrent neural network (CFLARNN) modules. Compared to the standard recurrent neural network (RNN), those modules of PCFLARNN can simultaneously be performed in a pipelined parallelism fashion, and this would lead to a significant improvement in its total computational efficiency. Furthermore, contrasted with the architecture of a pipelined RNN (PRNN), each module of PCFLARNN is a CFLARNN whose nonlinearity is introduced by enhancing the input pattern with Chebyshev functional expansion, whereas the RNN of each module in PRNN utilizing linear input and first-order recurrent term only fails to utilize the high-order terms of inputs. Therefore, the performance of PCFLARNN can further be improved at the cost of a slightly increased computational complexity. In addition, due to the introduced nonlinear functional expansion of each module in PRNN, the number of input signals can be reduced. Computer simulations have demonstrated that the proposed filter performs better than PRNN and RNN for nonlinear colored signal prediction, nonstationary speech signal prediction, and chaotic time series prediction.

Robust adaptive gradient-descent training algorithm for recurrent neural networks in discrete time domain

For a recurrent neural network (RNN), its transient response is a critical issue, especially for real-time signal processing applications. The conventional RNN training algorithms, such as backpropagation through time (BPTT) and real-time recurrent learning (RTRL), have not adequately addressed this problem because they suffer from low convergence speed. While increasing the learning rate may help to improve the performance of the RNN, it can result in unstable training in terms of weight divergence. Therefore, an optimal tradeoff between RNN training speed and weight convergence is desired. In this paper, a robust adaptive gradient-descent (RAGD) training algorithm of RNN is developed based on a novel RNN hybrid training concept. It switches the training patterns between standard real-time online backpropagation (BP) and RTRL according to the derived convergence and stability conditions. The weight convergence and L(2)-stability of the algorithm are derived via the conic sector theorem. The optimized adaptive learning maximizes the training speed of the RNN for each weight update without violating the stability and convergence criteria. Computer simulations are carried out to demonstrate the applicability of the theoretical results.

A normalized adaptive training of recurrent neural networks with augmented error gradient

For training algorithms of recurrent neural networks (RNN), convergent speed and training error are always two contradictory performances. In this letter, we propose a normalized adaptive recurrent learning (NARL) to obtain a tradeoff between transient and steady-state response. An augmented term is added to error gradient to exactly model the derivative of cost function with respect to hidden layer weight. The influence of the induced gain of activation function on training stability is also taken into consideration. Moreover, adaptive learning rate is employed to improve the robustness of the gradient training. Finally, computer simulations of a model prediction problem are synthesized to give comparisons between NARL and conventional normalized real-time recurrent learning (N-RTRL).