Energy Scaling Advantages of Resistive Memory Crossbar Based Computation and Its Application to Sparse Coding

A High-Precision Voltage-Quantization-Based Current-Mode Computing-in-Memory SRAM

Non-linear distortion of signals is a serious problem in computing-in-memory SRAM (CIM-SRAM) circuits in current mode. This problem greatly limits the performance of calculations and directly affects the computing power of the CIM-SRAM. In this study, the causes of non-linearity and inconsistency were investigated. Based on detailed analyses, we proposed a high-precision, fully dynamic range IV (HFIV) conversion circuit. The HFIV circuit was added to each bit line (BL) for voltage clamping and proportionally mirroring the read current. We applied the structure to numerous prior studies and evaluated them using the 55 nm complementary metal-oxide semiconductor process. The results showed the proposed HFIV circuit could increase the CIM-SRAM's calculation linearity to 99.92% (8~32 SRAM bit-cells) and 99.8% (32~64 SRAM bit-cells) with a 1.2 V supply.

Electrochemical Ionic Synapses: Progress and Perspectives

Artificial neural networks based on crossbar arrays of analog programmable resistors can address the high energy challenge of conventional hardware in artificial intelligence applications. However, state-of-the-art two-terminal resistive switching devices based on conductive filament formation suffer from high variability and poor controllability. Electrochemical ionic synapses are three-terminal devices that operate by electrochemical and dynamic insertion/extraction of ions that control the electronic conductivity of a channel in a single solid-solution phase. They are promising candidates for programmable resistors in crossbar arrays because they have shown uniform and deterministic control of electronic conductivity based on ion doping, with very low energy consumption. Here, the desirable specifications of these programmable resistors are presented. Then, an overview of the current progress of devices based on Li+ , O2- , and H+ ions and material systems is provided. Achieving nanosecond speed, low operation voltage (≈1 V), low energy consumption, with complementary metal-oxide-semiconductor compatibility all simultaneously remains a challenge. Toward this goal, a physical model of the device is constructed to provide guidelines for the desired material properties to overcome the remaining challenges. Finally, an outlook is provided, including strategies to advance materials toward the desirable properties and the future opportunities for electrochemical ionic synapses.

Probabilistic Neural Computing with Stochastic Devices

The brain has effectively proven a powerful inspiration for the development of computing architectures in which processing is tightly integrated with memory, communication is event-driven, and analog computation can be performed at scale. These neuromorphic systems increasingly show an ability to improve the efficiency and speed of scientific computing and artificial intelligence applications. Herein, it is proposed that the brain's ubiquitous stochasticity represents an additional source of inspiration for expanding the reach of neuromorphic computing to probabilistic applications. To date, many efforts exploring probabilistic computing have focused primarily on one scale of the microelectronics stack, such as implementing probabilistic algorithms on deterministic hardware or developing probabilistic devices and circuits with the expectation that they will be leveraged by eventual probabilistic architectures. A co-design vision is described by which large numbers of devices, such as magnetic tunnel junctions and tunnel diodes, can be operated in a stochastic regime and incorporated into a scalable neuromorphic architecture that can impact a number of probabilistic computing applications, such as Monte Carlo simulations and Bayesian neural networks. Finally, a framework is presented to categorize increasingly advanced hardware-based probabilistic computing technologies.

Impact of Asymmetric Weight Update on Neural Network Training With Tiki-Taka Algorithm

Recent progress in novel non-volatile memory-based synaptic device technologies and their feasibility for matrix-vector multiplication (MVM) has ignited active research on implementing analog neural network training accelerators with resistive crosspoint arrays. While significant performance boost as well as area- and power-efficiency is theoretically predicted, the realization of such analog accelerators is largely limited by non-ideal switching characteristics of crosspoint elements. One of the most performance-limiting non-idealities is the conductance update asymmetry which is known to distort the actual weight change values away from the calculation by error back-propagation and, therefore, significantly deteriorates the neural network training performance. To address this issue by an algorithmic remedy, Tiki-Taka algorithm was proposed and shown to be effective for neural network training with asymmetric devices. However, a systematic analysis to reveal the required asymmetry specification to guarantee the neural network performance has been unexplored. Here, we quantitatively analyze the impact of update asymmetry on the neural network training performance when trained with Tiki-Taka algorithm by exploring the space of asymmetry and hyper-parameters and measuring the classification accuracy. We discover that the update asymmetry level of the auxiliary array affects the way the optimizer takes the importance of previous gradients, whereas that of main array affects the frequency of accepting those gradients. We propose a novel calibration method to find the optimal operating point in terms of device and network parameters. By searching over the hyper-parameter space of Tiki-Taka algorithm using interpolation and Gaussian filtering, we find the optimal hyper-parameters efficiently and reveal the optimal range of asymmetry, namely the asymmetry specification. Finally, we show that the analysis and calibration method be applicable to spiking neural networks.

Applying Neuromorphic Computing Simulation in Band Gap Prediction and Chemical Reaction Classification

The rapidly developing artificial intelligence (AI) requires revolutionary computing architectures to break the energy efficiency bottleneck caused by the traditional von Neumann computing architecture. In addition, the emerging brain-machine interface also requires computational circuitry that can conduct large parallel computational tasks with low energy cost and good biocompatibility. Neuromorphic computing, a novel computational architecture emulating human brains, has drawn significant interest for the aforementioned applications due to its low energy cost, capability to parallelly process large-scale data, and biocompatibility. Most efforts in the domain of neuromorphic computing focus on addressing traditional AI problems, such as handwritten digit recognition and file classification. Here, we demonstrate for the first time that current neuromorphic computing techniques can be used to solve key machine learning questions in cheminformatics. We predict the band gaps of small-molecule organic semiconductors and classify chemical reaction types with a simulated neuromorphic circuitry. Our work can potentially guide the design and fabrication of elementary devices and circuitry for neuromorphic computing specialized for chemical purposes.

Algorithm for Training Neural Networks on Resistive Device Arrays

Hardware architectures composed of resistive cross-point device arrays can provide significant power and speed benefits for deep neural network training workloads using stochastic gradient descent (SGD) and backpropagation (BP) algorithm. The training accuracy on this imminent analog hardware, however, strongly depends on the switching characteristics of the cross-point elements. One of the key requirements is that these resistive devices must change conductance in a symmetrical fashion when subjected to positive or negative pulse stimuli. Here, we present a new training algorithm, so-called the "Tiki-Taka" algorithm, that eliminates this stringent symmetry requirement. We show that device asymmetry introduces an unintentional implicit cost term into the SGD algorithm, whereas in the "Tiki-Taka" algorithm a coupled dynamical system simultaneously minimizes the original objective function of the neural network and the unintentional cost term due to device asymmetry in a self-consistent fashion. We tested the validity of this new algorithm on a range of network architectures such as fully connected, convolutional and LSTM networks. Simulation results on these various networks show that the accuracy achieved using the conventional SGD algorithm with symmetric (ideal) device switching characteristics is matched in accuracy achieved using the "Tiki-Taka" algorithm with non-symmetric (non-ideal) device switching characteristics. Moreover, all the operations performed on the arrays are still parallel and therefore the implementation cost of this new algorithm on array architectures is minimal; and it maintains the aforementioned power and speed benefits. These algorithmic improvements are crucial to relax the material specification and to realize technologically viable resistive crossbar arrays that outperform digital accelerators for similar training tasks.

Low-Voltage, CMOS-Free Synaptic Memory Based on LiXTiO2 Redox Transistors

Neuromorphic computers based on analogue neural networks aim to substantially lower computing power by reducing the need to shuttle data between memory and logic units. Artificial synapses containing nonvolatile analogue conductance states enable direct computation using memory elements; however, most nonvolatile analogue memories require high write voltages and large current densities and are accompanied by nonlinear and unpredictable weight updates. Here, we develop an inorganic redox transistor based on electrochemical lithium-ion insertion into Li_XTiO₂ that displays linear weight updates at both low current densities and low write voltages. The write voltage, as low as 200 mV at room temperature, is achieved by minimizing the open-circuit voltage and using a low-voltage diffusive memristor selector. We further show that the Li_XTiO₂ redox transistor can achieve an extremely sharp transistor subthreshold slope of just 40 mV/decade when operating in an electrochemically driven phase transformation regime.

Parallel programming of an ionic floating-gate memory array for scalable neuromorphic computing

Neuromorphic computers could overcome efficiency bottlenecks inherent to conventional computing through parallel programming and readout of artificial neural network weights in a crossbar memory array. However, selective and linear weight updates and <10-nanoampere read currents are required for learning that surpasses conventional computing efficiency. We introduce an ionic floating-gate memory array based on a polymer redox transistor connected to a conductive-bridge memory (CBM). Selective and linear programming of a redox transistor array is executed in parallel by overcoming the bridging threshold voltage of the CBMs. Synaptic weight readout with currents <10 nanoamperes is achieved by diluting the conductive polymer with an insulator to decrease the conductance. The redox transistors endure >1 billion write-read operations and support >1-megahertz write-read frequencies.

Lithium ion trapping mechanism of SiO2 in LiCoO2 based memristors

Pt/LiCoO₂/SiO₂/Si stacks with different SiO₂ thicknesses are fabricated and the influence of SiO₂ on memristive behavior is investigated. It is demonstrated that SiO₂ can serve as Li ion trapping layer benefiting device retention, and the thickness of SiO₂ must be controlled to avoid large SET voltage and state instability. Simulation model based on Nernst potential and diffusion potential is postulated for electromotive force in LiCoO₂ based memristors. The simulation results show that SiO₂ trapping layer decreases the total electromotive field of device and thereby prevents Li ions from migrating back to LiCoO₂. This model shows a good agreement with experimental data and reveals the Li ion trapping mechanism of SiO₂ in LiCoO₂ based memristors.

Ultralow Power Dual-Gated Subthreshold Oxide Neuristors: An Enabler for Higher Order Neuronal Temporal Correlations

Inspired by neural computing, the pursuit of ultralow power neuromorphic architectures with highly distributed memory and parallel processing capability has recently gained more traction. However, emulation of biological signal processing via artificial neuromorphic architectures does not exploit the immense interplay between local activities and global neuromodulations observed in biological neural networks and hence are unable to mimic complex biologically plausible adaptive functions like heterosynaptic plasticity and homeostasis. Here, we demonstrate emulation of complex neuronal behaviors like heterosynaptic plasticity, homeostasis, association, correlation, and coincidence in a single neuristor via a dual-gated architecture. This multiple gating approach allows one gate to capture the effect of local activity correlations and the second gate to represent global neuromodulations, allowing additional modulations which augment their plasticity, enabling higher order temporal correlations at a unitary level. Moreover, the dual-gate operation extends the available dynamic range of synaptic conductance while maintaining symmetry in the weight-update operation, expanding the number of accessible memory states. Finally, operating neuristors in the subthreshold regime enable synaptic weight changes with high gain while maintaining ultralow power consumption of the order of femto-Joules.

Training LSTM Networks With Resistive Cross-Point Devices

In our previous work we have shown that resistive cross point devices, so called resistive processing unit (RPU) devices, can provide significant power and speed benefits when training deep fully connected networks as well as convolutional neural networks. In this work, we further extend the RPU concept for training recurrent neural networks (RNNs) namely LSTMs. We show that the mapping of recurrent layers is very similar to the mapping of fully connected layers and therefore the RPU concept can potentially provide large acceleration factors for RNNs as well. In addition, we study the effect of various device imperfections and system parameters on training performance. Symmetry of updates becomes even more crucial for RNNs; already a few percent asymmetry results in an increase in the test error compared to the ideal case trained with floating point numbers. Furthermore, the input signal resolution to the device arrays needs to be at least 7 bits for successful training. However, we show that a stochastic rounding scheme can reduce the input signal resolution back to 5 bits. Further, we find that RPU device variations and hardware noise are enough to mitigate overfitting, so that there is less need for using dropout. Here we attempt to study the validity of the RPU approach by simulating large scale networks. For instance, the models studied here are roughly 1500 times larger than the more often studied multilayer perceptron models trained on the MNIST dataset in terms of the total number of multiplication and summation operations performed per epoch.

Memristor-Based Analog Computation and Neural Network Classification with a Dot Product Engine

Using memristor crossbar arrays to accelerate computations is a promising approach to efficiently implement algorithms in deep neural networks. Early demonstrations, however, are limited to simulations or small-scale problems primarily due to materials and device challenges that limit the size of the memristor crossbar arrays that can be reliably programmed to stable and analog values, which is the focus of the current work. High-precision analog tuning and control of memristor cells across a 128 × 64 array is demonstrated, and the resulting vector matrix multiplication (VMM) computing precision is evaluated. Single-layer neural network inference is performed in these arrays, and the performance compared to a digital approach is assessed. Memristor computing system used here reaches a VMM accuracy equivalent of 6 bits, and an 89.9% recognition accuracy is achieved for the 10k MNIST handwritten digit test set. Forecasts show that with integrated (on chip) and scaled memristors, a computational efficiency greater than 100 trillion operations per second per Watt is possible.

On-Demand Reconfiguration of Nanomaterials: When Electronics Meets Ionics

Rapid advances in the semiconductor industry, driven largely by device scaling, are now approaching fundamental physical limits and face severe power, performance, and cost constraints. Multifunctional materials and devices may lead to a paradigm shift toward new, intelligent, and efficient computing systems, and are being extensively studied. Herein examines how, by controlling the internal ion distribution in a solid-state film, a material's chemical composition and physical properties can be reversibly reconfigured using an applied electric field, at room temperature and after device fabrication. Reconfigurability is observed in a wide range of materials, including commonly used dielectric films, and has led to the development of new device concepts such as resistive random-access memory. Physical reconfigurability further allows memory and logic operations to be merged in the same device for efficient in-memory computing and neuromorphic computing systems. By directly changing the chemical composition of the material, coupled electrical, optical, and magnetic effects can also be obtained. A survey of recent fundamental material and device studies that reveal the dynamic ionic processes is included, along with discussions on systematic modeling efforts, device and material challenges, and future research directions.

Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices

In a previous work we have detailed the requirements for obtaining maximal deep learning performance benefit by implementing fully connected deep neural networks (DNN) in the form of arrays of resistive devices. Here we extend the concept of Resistive Processing Unit (RPU) devices to convolutional neural networks (CNNs). We show how to map the convolutional layers to fully connected RPU arrays such that the parallelism of the hardware can be fully utilized in all three cycles of the backpropagation algorithm. We find that the noise and bound limitations imposed by the analog nature of the computations performed on the arrays significantly affect the training accuracy of the CNNs. Noise and bound management techniques are presented that mitigate these problems without introducing any additional complexity in the analog circuits and that can be addressed by the digital circuits. In addition, we discuss digitally programmable update management and device variability reduction techniques that can be used selectively for some of the layers in a CNN. We show that a combination of all those techniques enables a successful application of the RPU concept for training CNNs. The techniques discussed here are more general and can be applied beyond CNN architectures and therefore enables applicability of the RPU approach to a large class of neural network architectures.

Sparse coding with memristor networks

Sparse representation of information provides a powerful means to perform feature extraction on high-dimensional data and is of broad interest for applications in signal processing, computer vision, object recognition and neurobiology. Sparse coding is also believed to be a key mechanism by which biological neural systems can efficiently process a large amount of complex sensory data while consuming very little power. Here, we report the experimental implementation of sparse coding algorithms in a bio-inspired approach using a 32 × 32 crossbar array of analog memristors. This network enables efficient implementation of pattern matching and lateral neuron inhibition and allows input data to be sparsely encoded using neuron activities and stored dictionary elements. Different dictionary sets can be trained and stored in the same system, depending on the nature of the input signals. Using the sparse coding algorithm, we also perform natural image processing based on a learned dictionary.