A Survey of Stochastic Computing Neural Networks for Machine Learning Applications

Neural networks (NNs) are effective machine learning models that require significant hardware and energy consumption in their computing process. To implement NNs, stochastic computing (SC) has been proposed to achieve a tradeoff between hardware efficiency and computing performance. In an SC NN, hardware requirements and power consumption are significantly reduced by moderately sacrificing the inference accuracy and computation speed. With recent developments in SC techniques, however, the performance of SC NNs has substantially been improved, making it comparable with conventional binary designs yet by utilizing less hardware. In this article, we begin with the design of a basic SC neuron and then survey different types of SC NNs, including multilayer perceptrons, deep belief networks, convolutional NNs, and recurrent NNs. Recent progress in SC designs that further improve the hardware efficiency and performance of NNs is subsequently discussed. The generality and versatility of SC NNs are illustrated for both the training and inference processes. Finally, the advantages and challenges of SC NNs are discussed with respect to binary counterparts.

On Complete Stability of Recurrent Neural Networks With Time-Varying Delays and General Piecewise Linear Activation Functions

This paper addresses the problem of complete stability of delayed recurrent neural networks with a general class of piecewise linear activation functions. By applying an appropriate partition of the state space and iterating the defined bounding functions, some sufficient conditions are obtained to ensure that an n -neuron neural network is completely stable with exactly ∏_i=1ⁿ(2K_i-1) equilibrium points, among which ∏_i=1ⁿK_i equilibrium points are locally exponentially stable and the others are unstable, where K_i (i=1,…,n) are non-negative integers which depend jointly on activation functions and parameters of neural networks. The results of this paper include the existing works on the stability analysis of recurrent neural networks with piecewise linear functions as special cases and hence can be considered as the improvement and extension of the existing stability results in the literature. A numerical example is provided to illustrate the derived theoretical results.

Neural Networks Application for Processing of the Data from the FMICW Radars

In this paper the results of the Neural Networks and machine learning applications for radar signal processing are presented. The radar output from the primary radar signal processing is represented as a 2D image composed from echoes of the targets and noise background. The Frequency Modulated Interrupted Continuous Wave (FMICW) radar PCDR35 (Portable Cloud Doppler Radar at the frequency 35.4 GHz) was used. Presently, the processing is realized via a National Instruments industrial computer. The neural network of the proposed system is using four or five (optional for the user) signal processing steps. These steps are 2D spectrum filtration, thresholding, unification of the target, target area transforming to the rectangular shape (optional step), and target board line detection. The proposed neural network was tested with sets of four cases (100 tests for every case). This neural network provides image processing of the 2D spectrum. The results obtained from this new system are much better than the results of our previous algorithm.

The correspondence between deterministic and stochastic digital neurons: analysis and methodology

This paper analyzes the criteria for the direct correspondence between a deterministic neural network and its stochastic counterpart, and presents the guidelines that have been derived to establish such a correspondence during the design of a neural network application. In particular, the role of the slope and bias of the neuron activation function and that of the noise of its output have been addressed, thus filling a specific literature gap. This paper presents the results that have been theoretically derived in this regard, together with the simulations of few relevant application examples that have been performed to support them.

Neural network implementation using bit streams

A new method for the parallel hardware implementation of artificial neural networks (ANNs) using digital techniques is presented. Signals are represented using uniformly weighted single-bit streams. Techniques for generating bit streams from analog or multibit inputs are also presented. This single-bit representation offers significant advantages over multibit representations since they mitigate the fan-in and fan-out issues which are typical to distributed systems. To process these bit streams using ANNs concepts, functional elements which perform summing, scaling, and squashing have been implemented. These elements are modular and have been designed such that they can be easily interconnected. Two new architectures which act as monotonically increasing differentiable nonlinear squashing functions have also been presented. Using these functional elements, a multilayer perceptron (MLP) can be easily constructed. Two examples successfully demonstrate the use of bit streams in the implementation of ANNs. Since every functional element is individually instantiated, the implementation is genuinely parallel. The results clearly show that this bit-stream technique is viable for the hardware implementation of a variety of distributed systems and for ANNs in particular.

Real-time neural network inversion on the SRC-6e reconfigurable computer

Implementation of real-time neural network inversion on the SRC-6e, a computer that uses multiple field-programmable gate arrays (FPGAs) as reconfigurable computing elements, is examined using a sonar application as a specific case study. A feedforward multilayer perceptron neural network is used to estimate the performance of the sonar system (Jung et al., 2001). A particle swarm algorithm uses the trained network to perform a search for the control parameters required to optimize the output performance of the sonar system in the presence of imposed environmental constraints (Fox et al., 2002). The particle swarm optimization (PSO) requires repetitive queries of the neural network. Alternatives for implementing neural networks and particle swarm algorithms in reconfigurable hardware are contrasted. The final implementation provides nearly two orders of magnitude of speed increase over a state-of-the-art personal computer (PC), providing a real-time solution.

A new architecture for digital stochastic pulse-mode neurons based on the voting circuit

This paper presents a new kind of architecture for artificial digital neurons based on the voting circuit, which may be considered an improved version of those presented in literature. Stochastic pulse modulation has been used, where the values of the neuron's inputs are coded in terms of bit probabilities. The resulting activation function closely resembles the logistic sigmoid, with a transition slope that can be selected at the architectural level with no additional hardware requirements. The proposed neuron architecture has been simulated in software. Simulation results confirm that the neuron features a sigmoid transfer characteristic similar to that of conventional voting circuits. The resource occupation of the neuron, as obtained from implementation on reconfigurable platforms, has been estimated to be significantly lower than previous implementations. The theoretical analysis of the neuron's behavior is also presented.