Memristive Quantized Neural Networks: A Novel Approach to Accelerate Deep Learning On-Chip

Automatic Learning Rate Adaption for Memristive Deep Learning Systems

As a possible device to further enhance the performance of the hybrid complementary metal oxide semiconductor (CMOS) technology in the hardware, the memristor has attracted widespread attention in implementing efficient and compact deep learning (DL) systems. In this study, an automatic learning rate tuning method for memristive DL systems is presented. Memristive devices are utilized to adjust the adaptive learning rate in deep neural networks (DNNs). The speed of the learning rate adaptation process is fast at first and then becomes slow, which consist of the memristance or conductance adjustment process of the memristors. As a result, no manual tuning of learning rates is required in the adaptive back propagation (BP) algorithm. While cycle-to-cycle and device-to-device variations could be a significant issue in memristive DL systems, the proposed method appears robust to noisy gradients, various architectures, and different datasets. Moreover, fuzzy control methods for adaptive learning are presented for pattern recognition, such that the over-fitting issue can be well addressed. To our best knowledge, this is the first memristive DL system using an adaptive learning rate for image recognition. Another highlight of the presented memristive adaptive DL system is that quantized neural network architecture is utilized, and there is therefore a significant increase in the training efficiency, without the loss of testing accuracy.

The Framework and Memristive Circuit Design for Multisensory Mutual Associative Memory Networks

In this work, we propose a multisensory mutual associative memory networks framework and memristive circuit to mimic the ability of the biological brain to make associations of information received simultaneously. The circuit inspired by neural mechanisms of associative memory cells mainly consists of three modules: 1) the storage neurons module, which encodes external multimodal information into the firing rate of spikes; 2) the synapse module, which uses the nonvolatility memristor to achieve weight adjustment and associative learning; and 3) the retrieval neuron module, which feeds the retrieval signal output from each sensory pathway to other sensory pathways, so that achieve mutual association and retrieval between multiple modalities. Different from other one-to-one or many-to-one unidirectional associative memory work, this circuit achieves bidirectional association from multiple modalities to multiple modalities. In addition, we simulate the acquisition, extinction, recovery, transmission, and consolidation properties of associative memory. The circuit is applied to cross-modal association of image and audio recognition results, and episodic memory is simulated, where multiple images in a scene are intramodal associated. With power and area analysis, the circuit is validated as hardware-friendly. Further research to extend this work into large-scale associative memory networks, combined with visual-auditory-tactile-gustatory sensory sensors, is promising for application in intelligent robotic platforms to facilitate the development of neuromorphic systems and brain-like intelligence.

Research on the Impact of Data Density on Memristor Crossbar Architectures in Neuromorphic Pattern Recognition

Binary memristor crossbars have great potential for use in brain-inspired neuromorphic computing. The complementary crossbar array has been proposed to perform the Exclusive-NOR function for neuromorphic pattern recognition. The single crossbar obtained by shortening the Exclusive-NOR function has more advantages in terms of power consumption, area occupancy, and fault tolerance. In this paper, we present the impact of data density on the single memristor crossbar architecture for neuromorphic image recognition. The impact of data density on the single memristor architecture is mathematically derived from the reduced formula of the Exclusive-NOR function, and then verified via circuit simulation. The complementary and single crossbar architectures are tested by using ten 32 × 32 images with different data densities of 0.25, 0.5, and 0.75. The simulation results showed that the data density of images has a negative effect on the single memristor crossbar architecture while not affecting the complementary memristor crossbar architecture. The maximum output column current produced by the single memristor crossbar array decreases as data density decreases while the complementary memristor crossbar array architecture provides stable maximum output column currents. When recognizing images with data density as low as 0.25, the maximum output column currents of the single memristor crossbar architecture is reduced four-fold compared with the maximum currents from the complementary memristor crossbar architecture. This reduction causes the Winner-take-all circuit to work incorrectly and will reduce the recognition rate of the single memristor crossbar architecture. These simulation results show that the single memristor crossbar architecture has more advantages compared with the complementary crossbar architecture when the images do have not many different densities, and none of the images have very low densities. This work also indicates that the single crossbar architecture must be improved by adding a constant term to deal with images that have low data densities. These are valuable case studies for archiving the advantages of single memristor crossbar architecture in neuromorphic computing applications.

Thermally stable threshold selector based on CuAg alloy for energy-efficient memory and neuromorphic computing applications

As a promising candidate for high-density data storage and neuromorphic computing, cross-point memory arrays provide a platform to overcome the von Neumann bottleneck and accelerate neural network computation. In order to suppress the sneak-path current problem that limits their scalability and read accuracy, a two-terminal selector can be integrated at each cross-point to form the one-selector-one-memristor (1S1R) stack. In this work, we demonstrate a CuAg alloy-based, thermally stable and electroforming-free selector device with tunable threshold voltage and over 7 orders of magnitude ON/OFF ratio. A vertically stacked 64 × 64 1S1R cross-point array is further implemented by integrating the selector with SiO₂-based memristors. The 1S1R devices exhibit extremely low leakage currents and proper switching characteristics, which are suitable for both storage class memory and synaptic weight storage. Finally, a selector-based leaky integrate-and-fire neuron is designed and experimentally implemented, which expands the application prospect of CuAg alloy selectors from synapses to neurons.

Improved Results on Fixed-/Preassigned-Time Synchronization for Memristive Complex-Valued Neural Networks

This article concerns the problems of synchronization in a fixed time or prespecified time for memristive complex-valued neural networks (MCVNNs), in which the state variables, activation functions, rates of neuron self-inhibition, neural connection memristive weights, and external inputs are all assumed to be complex-valued. First, the more comprehensive fixed-time stability theorem and more accurate estimations on settling time (ST) are systematically established by using the comparison principle. Second, by introducing different norms of complex numbers instead of decomposing the complex-valued system into real and imaginary parts, we successfully design several simpler discontinuous controllers to acquire much improved fixed-time synchronization (FXTS) results. Third, based on similar mathematical derivations, the preassigned-time synchronization (PATS) conditions are explored by newly developed new control strategies, in which ST can be prespecified and is independent of initial values and any parameters of neural networks and controllers. Finally, numerical simulations are provided to illustrate the effectiveness and superiority of the improved synchronization methodology.

Noise and Memristance Variation Tolerance of Single Crossbar Architectures for Neuromorphic Image Recognition

We performed a comparative study on the Gaussian noise and memristance variation tolerance of three crossbar architectures, namely the complementary crossbar architecture, the twin crossbar architecture, and the single crossbar architecture, for neuromorphic image recognition and conducted an experiment to determine the performance of the single crossbar architecture for simple pattern recognition. Ten grayscale images with the size of 32 × 32 pixels were used for testing and comparing the recognition rates of the three architectures. The recognition rates of the three memristor crossbar architectures were compared to each other when the noise level of images was varied from -10 to 4 dB and the percentage of memristance variation was varied from 0% to 40%. The simulation results showed that the single crossbar architecture had the best Gaussian noise input and memristance variation tolerance in terms of recognition rate. At the signal-to-noise ratio of -10 dB, the single crossbar architecture produced a recognition rate of 91%, which was 2% and 87% higher than those of the twin crossbar architecture and the complementary crossbar architecture, respectively. When the memristance variation percentage reached 40%, the single crossbar architecture had a recognition rate as high as 67.8%, which was 1.8% and 9.8% higher than the recognition rates of the twin crossbar architecture and the complementary crossbar architecture, respectively. Finally, we carried out an experiment to determine the performance of the single crossbar architecture with a fabricated 3 × 3 memristor crossbar based on carbon fiber and aluminum film. The experiment proved successful implementation of pattern recognition with the single crossbar architecture.

Study of Quantized Hardware Deep Neural Networks Based on Resistive Switching Devices, Conventional versus Convolutional Approaches

A comprehensive analysis of two types of artificial neural networks (ANN) is performed to assess the influence of quantization on the synaptic weights. Conventional multilayer-perceptron (MLP) and convolutional neural networks (CNN) have been considered by changing their features in the training and inference contexts, such as number of levels in the quantization process, the number of hidden layers on the network topology, the number of neurons per hidden layer, the image databases, the number of convolutional layers, etc. A reference technology based on 1T1R structures with bipolar memristors including HfO2 dielectrics was employed, accounting for different multilevel schemes and the corresponding conductance quantization algorithms. The accuracy of the image recognition processes was studied in depth. This type of studies are essential prior to hardware implementation of neural networks. The obtained results support the use of CNNs for image domains. This is linked to the role played by convolutional layers at extracting image features and reducing the data complexity. In this case, the number of synaptic weights can be reduced in comparison to MLPs.