1
|
Zhao R, Gong Z, Liu Y, Chen J. A High-Precision Voltage-Quantization-Based Current-Mode Computing-in-Memory SRAM. MICROMACHINES 2023; 14:2180. [PMID: 38138349 PMCID: PMC10745502 DOI: 10.3390/mi14122180] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/14/2023] [Revised: 11/12/2023] [Accepted: 11/27/2023] [Indexed: 12/24/2023]
Abstract
Non-linear distortion of signals is a serious problem in computing-in-memory SRAM (CIM-SRAM) circuits in current mode. This problem greatly limits the performance of calculations and directly affects the computing power of the CIM-SRAM. In this study, the causes of non-linearity and inconsistency were investigated. Based on detailed analyses, we proposed a high-precision, fully dynamic range IV (HFIV) conversion circuit. The HFIV circuit was added to each bit line (BL) for voltage clamping and proportionally mirroring the read current. We applied the structure to numerous prior studies and evaluated them using the 55 nm complementary metal-oxide semiconductor process. The results showed the proposed HFIV circuit could increase the CIM-SRAM's calculation linearity to 99.92% (8~32 SRAM bit-cells) and 99.8% (32~64 SRAM bit-cells) with a 1.2 V supply.
Collapse
Affiliation(s)
- Ruiyong Zhao
- Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai 200031, China; (R.Z.); (Y.L.)
- University of Chinese Academy of Sciences, Beijing 100049, China
| | - Zhenghui Gong
- Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai 200031, China; (R.Z.); (Y.L.)
| | - Yulan Liu
- Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai 200031, China; (R.Z.); (Y.L.)
- University of Chinese Academy of Sciences, Beijing 100049, China
| | - Jing Chen
- Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai 200031, China; (R.Z.); (Y.L.)
| |
Collapse
|
2
|
Huang M, Schwacke M, Onen M, Del Alamo J, Li J, Yildiz B. Electrochemical Ionic Synapses: Progress and Perspectives. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2023; 35:e2205169. [PMID: 36300807 DOI: 10.1002/adma.202205169] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/08/2022] [Revised: 09/13/2022] [Indexed: 06/16/2023]
Abstract
Artificial neural networks based on crossbar arrays of analog programmable resistors can address the high energy challenge of conventional hardware in artificial intelligence applications. However, state-of-the-art two-terminal resistive switching devices based on conductive filament formation suffer from high variability and poor controllability. Electrochemical ionic synapses are three-terminal devices that operate by electrochemical and dynamic insertion/extraction of ions that control the electronic conductivity of a channel in a single solid-solution phase. They are promising candidates for programmable resistors in crossbar arrays because they have shown uniform and deterministic control of electronic conductivity based on ion doping, with very low energy consumption. Here, the desirable specifications of these programmable resistors are presented. Then, an overview of the current progress of devices based on Li+ , O2- , and H+ ions and material systems is provided. Achieving nanosecond speed, low operation voltage (≈1 V), low energy consumption, with complementary metal-oxide-semiconductor compatibility all simultaneously remains a challenge. Toward this goal, a physical model of the device is constructed to provide guidelines for the desired material properties to overcome the remaining challenges. Finally, an outlook is provided, including strategies to advance materials toward the desirable properties and the future opportunities for electrochemical ionic synapses.
Collapse
Affiliation(s)
- Mantao Huang
- Department of Nuclear Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Miranda Schwacke
- Department of Materials Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Murat Onen
- Microsystems Technology Laboratories, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Jesús Del Alamo
- Microsystems Technology Laboratories, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Ju Li
- Department of Nuclear Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
- Department of Materials Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Bilge Yildiz
- Department of Nuclear Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
- Department of Materials Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| |
Collapse
|
3
|
Misra S, Bland LC, Cardwell SG, Incorvia JAC, James CD, Kent AD, Schuman CD, Smith JD, Aimone JB. Probabilistic Neural Computing with Stochastic Devices. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2023; 35:e2204569. [PMID: 36395387 DOI: 10.1002/adma.202204569] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/20/2022] [Revised: 08/03/2022] [Indexed: 06/16/2023]
Abstract
The brain has effectively proven a powerful inspiration for the development of computing architectures in which processing is tightly integrated with memory, communication is event-driven, and analog computation can be performed at scale. These neuromorphic systems increasingly show an ability to improve the efficiency and speed of scientific computing and artificial intelligence applications. Herein, it is proposed that the brain's ubiquitous stochasticity represents an additional source of inspiration for expanding the reach of neuromorphic computing to probabilistic applications. To date, many efforts exploring probabilistic computing have focused primarily on one scale of the microelectronics stack, such as implementing probabilistic algorithms on deterministic hardware or developing probabilistic devices and circuits with the expectation that they will be leveraged by eventual probabilistic architectures. A co-design vision is described by which large numbers of devices, such as magnetic tunnel junctions and tunnel diodes, can be operated in a stochastic regime and incorporated into a scalable neuromorphic architecture that can impact a number of probabilistic computing applications, such as Monte Carlo simulations and Bayesian neural networks. Finally, a framework is presented to categorize increasingly advanced hardware-based probabilistic computing technologies.
Collapse
Affiliation(s)
- Shashank Misra
- Microsystems Engineering, Science and Applications, Sandia National Laboratories, Albuquerque, NM, 87123, USA
| | - Leslie C Bland
- Department of Physics, Temple University, Philadelphia, PA, 19122-1801, USA
| | - Suma G Cardwell
- Neural Exploration and Research Laboratory, Sandia National Laboratories, Albuquerque, NM, 87123, USA
| | - Jean Anne C Incorvia
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Conrad D James
- Microsystems Engineering, Science and Applications, Sandia National Laboratories, Albuquerque, NM, 87123, USA
| | - Andrew D Kent
- Department of Physics, New York University, New York, NY, 10003, USA
| | - Catherine D Schuman
- Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville, TN, 37996, USA
| | - J Darby Smith
- Neural Exploration and Research Laboratory, Sandia National Laboratories, Albuquerque, NM, 87123, USA
| | - James B Aimone
- Neural Exploration and Research Laboratory, Sandia National Laboratories, Albuquerque, NM, 87123, USA
| |
Collapse
|
4
|
Lee C, Noh K, Ji W, Gokmen T, Kim S. Impact of Asymmetric Weight Update on Neural Network Training With Tiki-Taka Algorithm. Front Neurosci 2022; 15:767953. [PMID: 35069098 PMCID: PMC8770851 DOI: 10.3389/fnins.2021.767953] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2021] [Accepted: 10/26/2021] [Indexed: 11/13/2022] Open
Abstract
Recent progress in novel non-volatile memory-based synaptic device technologies and their feasibility for matrix-vector multiplication (MVM) has ignited active research on implementing analog neural network training accelerators with resistive crosspoint arrays. While significant performance boost as well as area- and power-efficiency is theoretically predicted, the realization of such analog accelerators is largely limited by non-ideal switching characteristics of crosspoint elements. One of the most performance-limiting non-idealities is the conductance update asymmetry which is known to distort the actual weight change values away from the calculation by error back-propagation and, therefore, significantly deteriorates the neural network training performance. To address this issue by an algorithmic remedy, Tiki-Taka algorithm was proposed and shown to be effective for neural network training with asymmetric devices. However, a systematic analysis to reveal the required asymmetry specification to guarantee the neural network performance has been unexplored. Here, we quantitatively analyze the impact of update asymmetry on the neural network training performance when trained with Tiki-Taka algorithm by exploring the space of asymmetry and hyper-parameters and measuring the classification accuracy. We discover that the update asymmetry level of the auxiliary array affects the way the optimizer takes the importance of previous gradients, whereas that of main array affects the frequency of accepting those gradients. We propose a novel calibration method to find the optimal operating point in terms of device and network parameters. By searching over the hyper-parameter space of Tiki-Taka algorithm using interpolation and Gaussian filtering, we find the optimal hyper-parameters efficiently and reveal the optimal range of asymmetry, namely the asymmetry specification. Finally, we show that the analysis and calibration method be applicable to spiking neural networks.
Collapse
Affiliation(s)
- Chaeun Lee
- Department of Materials Science and Engineering, Pohang University of Science and Technology, Pohang-si, South Korea
| | - Kyungmi Noh
- Department of Materials Science and Engineering, Pohang University of Science and Technology, Pohang-si, South Korea
| | - Wonjae Ji
- Department of Materials Science and Engineering, Pohang University of Science and Technology, Pohang-si, South Korea
| | - Tayfun Gokmen
- IBM Research AI, Yorktown Heights, NY, United States
| | - Seyoung Kim
- Department of Materials Science and Engineering, Pohang University of Science and Technology, Pohang-si, South Korea
| |
Collapse
|
5
|
Li B, Sun H, Shu H, Wang X. Applying Neuromorphic Computing Simulation in Band Gap Prediction and Chemical Reaction Classification. ACS OMEGA 2022; 7:168-175. [PMID: 35036688 PMCID: PMC8756567 DOI: 10.1021/acsomega.1c04287] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/10/2021] [Accepted: 12/01/2021] [Indexed: 06/14/2023]
Abstract
The rapidly developing artificial intelligence (AI) requires revolutionary computing architectures to break the energy efficiency bottleneck caused by the traditional von Neumann computing architecture. In addition, the emerging brain-machine interface also requires computational circuitry that can conduct large parallel computational tasks with low energy cost and good biocompatibility. Neuromorphic computing, a novel computational architecture emulating human brains, has drawn significant interest for the aforementioned applications due to its low energy cost, capability to parallelly process large-scale data, and biocompatibility. Most efforts in the domain of neuromorphic computing focus on addressing traditional AI problems, such as handwritten digit recognition and file classification. Here, we demonstrate for the first time that current neuromorphic computing techniques can be used to solve key machine learning questions in cheminformatics. We predict the band gaps of small-molecule organic semiconductors and classify chemical reaction types with a simulated neuromorphic circuitry. Our work can potentially guide the design and fabrication of elementary devices and circuitry for neuromorphic computing specialized for chemical purposes.
Collapse
Affiliation(s)
- Baochen Li
- Department
of Chemical and Biomolecular Engineering, The Ohio State University, Columbus, Ohio 43210, United States
| | - Haibin Sun
- Department
of Chemical and Biomolecular Engineering, The Ohio State University, Columbus, Ohio 43210, United States
| | - Haonian Shu
- Department
of Chemical and Biomolecular Engineering, The Ohio State University, Columbus, Ohio 43210, United States
| | - Xiaoxue Wang
- Department
of Chemical and Biomolecular Engineering, The Ohio State University, Columbus, Ohio 43210, United States
- The
Ohio State University Sustainability Institute, 3018 Smith Lab, 174 W. 18th Avenue, Columbus, Ohio 43210, United States
| |
Collapse
|
6
|
|
7
|
Gokmen T, Haensch W. Algorithm for Training Neural Networks on Resistive Device Arrays. Front Neurosci 2020; 14:103. [PMID: 32174807 PMCID: PMC7054461 DOI: 10.3389/fnins.2020.00103] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2019] [Accepted: 01/27/2020] [Indexed: 11/13/2022] Open
Abstract
Hardware architectures composed of resistive cross-point device arrays can provide significant power and speed benefits for deep neural network training workloads using stochastic gradient descent (SGD) and backpropagation (BP) algorithm. The training accuracy on this imminent analog hardware, however, strongly depends on the switching characteristics of the cross-point elements. One of the key requirements is that these resistive devices must change conductance in a symmetrical fashion when subjected to positive or negative pulse stimuli. Here, we present a new training algorithm, so-called the "Tiki-Taka" algorithm, that eliminates this stringent symmetry requirement. We show that device asymmetry introduces an unintentional implicit cost term into the SGD algorithm, whereas in the "Tiki-Taka" algorithm a coupled dynamical system simultaneously minimizes the original objective function of the neural network and the unintentional cost term due to device asymmetry in a self-consistent fashion. We tested the validity of this new algorithm on a range of network architectures such as fully connected, convolutional and LSTM networks. Simulation results on these various networks show that the accuracy achieved using the conventional SGD algorithm with symmetric (ideal) device switching characteristics is matched in accuracy achieved using the "Tiki-Taka" algorithm with non-symmetric (non-ideal) device switching characteristics. Moreover, all the operations performed on the arrays are still parallel and therefore the implementation cost of this new algorithm on array architectures is minimal; and it maintains the aforementioned power and speed benefits. These algorithmic improvements are crucial to relax the material specification and to realize technologically viable resistive crossbar arrays that outperform digital accelerators for similar training tasks.
Collapse
Affiliation(s)
- Tayfun Gokmen
- IBM Research AI, Yorktown Heights, NY, United States
| | | |
Collapse
|
8
|
Li Y, Fuller EJ, Asapu S, Agarwal S, Kurita T, Yang JJ, Talin AA. Low-Voltage, CMOS-Free Synaptic Memory Based on Li XTiO 2 Redox Transistors. ACS APPLIED MATERIALS & INTERFACES 2019; 11:38982-38992. [PMID: 31559816 DOI: 10.1021/acsami.9b14338] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Neuromorphic computers based on analogue neural networks aim to substantially lower computing power by reducing the need to shuttle data between memory and logic units. Artificial synapses containing nonvolatile analogue conductance states enable direct computation using memory elements; however, most nonvolatile analogue memories require high write voltages and large current densities and are accompanied by nonlinear and unpredictable weight updates. Here, we develop an inorganic redox transistor based on electrochemical lithium-ion insertion into LiXTiO2 that displays linear weight updates at both low current densities and low write voltages. The write voltage, as low as 200 mV at room temperature, is achieved by minimizing the open-circuit voltage and using a low-voltage diffusive memristor selector. We further show that the LiXTiO2 redox transistor can achieve an extremely sharp transistor subthreshold slope of just 40 mV/decade when operating in an electrochemically driven phase transformation regime.
Collapse
Affiliation(s)
- Yiyang Li
- Sandia National Laboratories , Livermore , California 94551 , United States
| | - Elliot J Fuller
- Sandia National Laboratories , Livermore , California 94551 , United States
| | - Shiva Asapu
- Department of Electrical and Computer Engineering , University of Massachusetts , Amherst , Massachusetts 01003 , United States
| | - Sapan Agarwal
- Sandia National Laboratories , Livermore , California 94551 , United States
| | - Tomochika Kurita
- Sandia National Laboratories , Livermore , California 94551 , United States
- Fujitsu Laboratories, Ltd. , Atsugi , Kanagawa 243-0197 , Japan
| | - J Joshua Yang
- Department of Electrical and Computer Engineering , University of Massachusetts , Amherst , Massachusetts 01003 , United States
| | - A Alec Talin
- Sandia National Laboratories , Livermore , California 94551 , United States
| |
Collapse
|
9
|
Fuller EJ, Keene ST, Melianas A, Wang Z, Agarwal S, Li Y, Tuchman Y, James CD, Marinella MJ, Yang JJ, Salleo A, Talin AA. Parallel programming of an ionic floating-gate memory array for scalable neuromorphic computing. Science 2019; 364:570-574. [PMID: 31023890 DOI: 10.1126/science.aaw5581] [Citation(s) in RCA: 248] [Impact Index Per Article: 49.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2019] [Accepted: 04/15/2019] [Indexed: 12/12/2022]
Abstract
Neuromorphic computers could overcome efficiency bottlenecks inherent to conventional computing through parallel programming and readout of artificial neural network weights in a crossbar memory array. However, selective and linear weight updates and <10-nanoampere read currents are required for learning that surpasses conventional computing efficiency. We introduce an ionic floating-gate memory array based on a polymer redox transistor connected to a conductive-bridge memory (CBM). Selective and linear programming of a redox transistor array is executed in parallel by overcoming the bridging threshold voltage of the CBMs. Synaptic weight readout with currents <10 nanoamperes is achieved by diluting the conductive polymer with an insulator to decrease the conductance. The redox transistors endure >1 billion write-read operations and support >1-megahertz write-read frequencies.
Collapse
Affiliation(s)
| | - Scott T Keene
- Department of Materials Science and Engineering, Stanford University, Stanford, CA, USA
| | - Armantas Melianas
- Department of Materials Science and Engineering, Stanford University, Stanford, CA, USA
| | - Zhongrui Wang
- Department of Computer Science and Electrical Engineering, University of Massachusetts Amherst, Amherst, MA, USA
| | | | - Yiyang Li
- Sandia National Laboratories, Livermore, CA, USA
| | - Yaakov Tuchman
- Department of Materials Science and Engineering, Stanford University, Stanford, CA, USA
| | | | | | - J Joshua Yang
- Department of Computer Science and Electrical Engineering, University of Massachusetts Amherst, Amherst, MA, USA
| | - Alberto Salleo
- Department of Materials Science and Engineering, Stanford University, Stanford, CA, USA.
| | - A Alec Talin
- Sandia National Laboratories, Livermore, CA, USA.
| |
Collapse
|
10
|
Lithium ion trapping mechanism of SiO 2 in LiCoO 2 based memristors. Sci Rep 2019; 9:5081. [PMID: 30911041 PMCID: PMC6434038 DOI: 10.1038/s41598-019-41508-3] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2018] [Accepted: 03/11/2019] [Indexed: 11/09/2022] Open
Abstract
Pt/LiCoO2/SiO2/Si stacks with different SiO2 thicknesses are fabricated and the influence of SiO2 on memristive behavior is investigated. It is demonstrated that SiO2 can serve as Li ion trapping layer benefiting device retention, and the thickness of SiO2 must be controlled to avoid large SET voltage and state instability. Simulation model based on Nernst potential and diffusion potential is postulated for electromotive force in LiCoO2 based memristors. The simulation results show that SiO2 trapping layer decreases the total electromotive field of device and thereby prevents Li ions from migrating back to LiCoO2. This model shows a good agreement with experimental data and reveals the Li ion trapping mechanism of SiO2 in LiCoO2 based memristors.
Collapse
|
11
|
John RA, Tiwari N, Yaoyi C, Tiwari N, Kulkarni M, Nirmal A, Nguyen AC, Basu A, Mathews N. Ultralow Power Dual-Gated Subthreshold Oxide Neuristors: An Enabler for Higher Order Neuronal Temporal Correlations. ACS NANO 2018; 12:11263-11273. [PMID: 30395439 DOI: 10.1021/acsnano.8b05903] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
Inspired by neural computing, the pursuit of ultralow power neuromorphic architectures with highly distributed memory and parallel processing capability has recently gained more traction. However, emulation of biological signal processing via artificial neuromorphic architectures does not exploit the immense interplay between local activities and global neuromodulations observed in biological neural networks and hence are unable to mimic complex biologically plausible adaptive functions like heterosynaptic plasticity and homeostasis. Here, we demonstrate emulation of complex neuronal behaviors like heterosynaptic plasticity, homeostasis, association, correlation, and coincidence in a single neuristor via a dual-gated architecture. This multiple gating approach allows one gate to capture the effect of local activity correlations and the second gate to represent global neuromodulations, allowing additional modulations which augment their plasticity, enabling higher order temporal correlations at a unitary level. Moreover, the dual-gate operation extends the available dynamic range of synaptic conductance while maintaining symmetry in the weight-update operation, expanding the number of accessible memory states. Finally, operating neuristors in the subthreshold regime enable synaptic weight changes with high gain while maintaining ultralow power consumption of the order of femto-Joules.
Collapse
Affiliation(s)
- Rohit Abraham John
- School of Materials Science and Engineering , Nanyang Technological University , 50 Nanyang Avenue , Singapore 639798
| | - Nidhi Tiwari
- Energy Research Institute at NTU (ERI@N) , Nanyang Technological University , Singapore 637553
| | - Chen Yaoyi
- School of Materials Science and Engineering , Nanyang Technological University , 50 Nanyang Avenue , Singapore 639798
| | - Naveen Tiwari
- School of Materials Science and Engineering , Nanyang Technological University , 50 Nanyang Avenue , Singapore 639798
| | - Mohit Kulkarni
- School of Materials Science and Engineering , Nanyang Technological University , 50 Nanyang Avenue , Singapore 639798
| | - Amoolya Nirmal
- School of Materials Science and Engineering , Nanyang Technological University , 50 Nanyang Avenue , Singapore 639798
| | - Anh Chien Nguyen
- School of Materials Science and Engineering , Nanyang Technological University , 50 Nanyang Avenue , Singapore 639798
| | - Arindam Basu
- School of Electrical and Electronic Engineering , Nanyang Technological University , 50 Nanyang Avenue , Singapore 639798
| | - Nripan Mathews
- School of Materials Science and Engineering , Nanyang Technological University , 50 Nanyang Avenue , Singapore 639798
- Energy Research Institute at NTU (ERI@N) , Nanyang Technological University , Singapore 637553
| |
Collapse
|
12
|
Gokmen T, Rasch MJ, Haensch W. Training LSTM Networks With Resistive Cross-Point Devices. Front Neurosci 2018; 12:745. [PMID: 30405334 PMCID: PMC6207602 DOI: 10.3389/fnins.2018.00745] [Citation(s) in RCA: 32] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2018] [Accepted: 09/28/2018] [Indexed: 11/24/2022] Open
Abstract
In our previous work we have shown that resistive cross point devices, so called resistive processing unit (RPU) devices, can provide significant power and speed benefits when training deep fully connected networks as well as convolutional neural networks. In this work, we further extend the RPU concept for training recurrent neural networks (RNNs) namely LSTMs. We show that the mapping of recurrent layers is very similar to the mapping of fully connected layers and therefore the RPU concept can potentially provide large acceleration factors for RNNs as well. In addition, we study the effect of various device imperfections and system parameters on training performance. Symmetry of updates becomes even more crucial for RNNs; already a few percent asymmetry results in an increase in the test error compared to the ideal case trained with floating point numbers. Furthermore, the input signal resolution to the device arrays needs to be at least 7 bits for successful training. However, we show that a stochastic rounding scheme can reduce the input signal resolution back to 5 bits. Further, we find that RPU device variations and hardware noise are enough to mitigate overfitting, so that there is less need for using dropout. Here we attempt to study the validity of the RPU approach by simulating large scale networks. For instance, the models studied here are roughly 1500 times larger than the more often studied multilayer perceptron models trained on the MNIST dataset in terms of the total number of multiplication and summation operations performed per epoch.
Collapse
Affiliation(s)
- Tayfun Gokmen
- IBM Research AI, Yorktown Heights, NY, United States
| | | | | |
Collapse
|
13
|
Hu M, Graves CE, Li C, Li Y, Ge N, Montgomery E, Davila N, Jiang H, Williams RS, Yang JJ, Xia Q, Strachan JP. Memristor-Based Analog Computation and Neural Network Classification with a Dot Product Engine. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2018; 30:1705914. [PMID: 29318659 DOI: 10.1002/adma.201705914] [Citation(s) in RCA: 152] [Impact Index Per Article: 25.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/11/2017] [Indexed: 05/24/2023]
Abstract
Using memristor crossbar arrays to accelerate computations is a promising approach to efficiently implement algorithms in deep neural networks. Early demonstrations, however, are limited to simulations or small-scale problems primarily due to materials and device challenges that limit the size of the memristor crossbar arrays that can be reliably programmed to stable and analog values, which is the focus of the current work. High-precision analog tuning and control of memristor cells across a 128 × 64 array is demonstrated, and the resulting vector matrix multiplication (VMM) computing precision is evaluated. Single-layer neural network inference is performed in these arrays, and the performance compared to a digital approach is assessed. Memristor computing system used here reaches a VMM accuracy equivalent of 6 bits, and an 89.9% recognition accuracy is achieved for the 10k MNIST handwritten digit test set. Forecasts show that with integrated (on chip) and scaled memristors, a computational efficiency greater than 100 trillion operations per second per Watt is possible.
Collapse
Affiliation(s)
- Miao Hu
- Hewlett Packard Labs, Hewlett Packard Enterprise, Palo Alto, CA, 94304, USA
| | - Catherine E Graves
- Hewlett Packard Labs, Hewlett Packard Enterprise, Palo Alto, CA, 94304, USA
| | - Can Li
- Department of Electrical and Computer Engineering, University of Massachusetts, Amherst, MA, 01003, USA
| | - Yunning Li
- Department of Electrical and Computer Engineering, University of Massachusetts, Amherst, MA, 01003, USA
| | - Ning Ge
- HP Labs, HP Inc., Palo Alto, CA, 94304, USA
| | - Eric Montgomery
- Hewlett Packard Labs, Hewlett Packard Enterprise, Palo Alto, CA, 94304, USA
| | - Noraica Davila
- Hewlett Packard Labs, Hewlett Packard Enterprise, Palo Alto, CA, 94304, USA
| | - Hao Jiang
- Department of Electrical and Computer Engineering, University of Massachusetts, Amherst, MA, 01003, USA
| | - R Stanley Williams
- Hewlett Packard Labs, Hewlett Packard Enterprise, Palo Alto, CA, 94304, USA
| | - J Joshua Yang
- Department of Electrical and Computer Engineering, University of Massachusetts, Amherst, MA, 01003, USA
| | - Qiangfei Xia
- Department of Electrical and Computer Engineering, University of Massachusetts, Amherst, MA, 01003, USA
| | - John Paul Strachan
- Hewlett Packard Labs, Hewlett Packard Enterprise, Palo Alto, CA, 94304, USA
| |
Collapse
|
14
|
Ren Y, Milo V, Wang Z, Xu H, Ielmini D, Zhao X, Liu Y. Analytical Modeling of Organic-Inorganic CH3
NH3
PbI3
Perovskite Resistive Switching and its Application for Neuromorphic Recognition. ADVANCED THEORY AND SIMULATIONS 2018. [DOI: 10.1002/adts.201700035] [Citation(s) in RCA: 29] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/13/2023]
Affiliation(s)
- Yanyun Ren
- Center for Advanced Optoelectronic Functional Materials Research; Key Laboratory for UV Light-Emitting Materials and Technology (Northeast Normal University); Ministry of Education; Changchun 130024 China
| | - Valerio Milo
- Dipartimento di Elettronica; Informazione e Bioingegneria; Italian Universities Nanoelectronics Team; Politecnico di Milano Milan 20133 Italy
| | - Zhongqiang Wang
- Center for Advanced Optoelectronic Functional Materials Research; Key Laboratory for UV Light-Emitting Materials and Technology (Northeast Normal University); Ministry of Education; Changchun 130024 China
| | - Haiyang Xu
- Center for Advanced Optoelectronic Functional Materials Research; Key Laboratory for UV Light-Emitting Materials and Technology (Northeast Normal University); Ministry of Education; Changchun 130024 China
| | - Daniele Ielmini
- Dipartimento di Elettronica; Informazione e Bioingegneria; Italian Universities Nanoelectronics Team; Politecnico di Milano Milan 20133 Italy
| | - Xiaoning Zhao
- Center for Advanced Optoelectronic Functional Materials Research; Key Laboratory for UV Light-Emitting Materials and Technology (Northeast Normal University); Ministry of Education; Changchun 130024 China
| | - Yichun Liu
- Center for Advanced Optoelectronic Functional Materials Research; Key Laboratory for UV Light-Emitting Materials and Technology (Northeast Normal University); Ministry of Education; Changchun 130024 China
| |
Collapse
|
15
|
Lee J, Lu WD. On-Demand Reconfiguration of Nanomaterials: When Electronics Meets Ionics. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2018; 30. [PMID: 28985005 DOI: 10.1002/adma.201702770] [Citation(s) in RCA: 46] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/17/2017] [Revised: 08/01/2017] [Indexed: 05/04/2023]
Abstract
Rapid advances in the semiconductor industry, driven largely by device scaling, are now approaching fundamental physical limits and face severe power, performance, and cost constraints. Multifunctional materials and devices may lead to a paradigm shift toward new, intelligent, and efficient computing systems, and are being extensively studied. Herein examines how, by controlling the internal ion distribution in a solid-state film, a material's chemical composition and physical properties can be reversibly reconfigured using an applied electric field, at room temperature and after device fabrication. Reconfigurability is observed in a wide range of materials, including commonly used dielectric films, and has led to the development of new device concepts such as resistive random-access memory. Physical reconfigurability further allows memory and logic operations to be merged in the same device for efficient in-memory computing and neuromorphic computing systems. By directly changing the chemical composition of the material, coupled electrical, optical, and magnetic effects can also be obtained. A survey of recent fundamental material and device studies that reveal the dynamic ionic processes is included, along with discussions on systematic modeling efforts, device and material challenges, and future research directions.
Collapse
Affiliation(s)
- Jihang Lee
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI, 48109, USA
- Department of Materials Science and Engineering, University of Michigan, Ann Arbor, MI, 48109, USA
| | - Wei D Lu
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI, 48109, USA
| |
Collapse
|
16
|
Gokmen T, Onen M, Haensch W. Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices. Front Neurosci 2017; 11:538. [PMID: 29066942 PMCID: PMC5641314 DOI: 10.3389/fnins.2017.00538] [Citation(s) in RCA: 94] [Impact Index Per Article: 13.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2017] [Accepted: 09/14/2017] [Indexed: 12/13/2022] Open
Abstract
In a previous work we have detailed the requirements for obtaining maximal deep learning performance benefit by implementing fully connected deep neural networks (DNN) in the form of arrays of resistive devices. Here we extend the concept of Resistive Processing Unit (RPU) devices to convolutional neural networks (CNNs). We show how to map the convolutional layers to fully connected RPU arrays such that the parallelism of the hardware can be fully utilized in all three cycles of the backpropagation algorithm. We find that the noise and bound limitations imposed by the analog nature of the computations performed on the arrays significantly affect the training accuracy of the CNNs. Noise and bound management techniques are presented that mitigate these problems without introducing any additional complexity in the analog circuits and that can be addressed by the digital circuits. In addition, we discuss digitally programmable update management and device variability reduction techniques that can be used selectively for some of the layers in a CNN. We show that a combination of all those techniques enables a successful application of the RPU concept for training CNNs. The techniques discussed here are more general and can be applied beyond CNN architectures and therefore enables applicability of the RPU approach to a large class of neural network architectures.
Collapse
Affiliation(s)
- Tayfun Gokmen
- IBM Thomas J. Watson Research Center, Yorktown Heights, NY, United States
| | - Murat Onen
- IBM Thomas J. Watson Research Center, Yorktown Heights, NY, United States
| | - Wilfried Haensch
- IBM Thomas J. Watson Research Center, Yorktown Heights, NY, United States
| |
Collapse
|
17
|
Sheridan PM, Cai F, Du C, Ma W, Zhang Z, Lu WD. Sparse coding with memristor networks. NATURE NANOTECHNOLOGY 2017; 12:784-789. [PMID: 28530717 DOI: 10.1038/nnano.2017.83] [Citation(s) in RCA: 176] [Impact Index Per Article: 25.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/16/2016] [Accepted: 03/28/2017] [Indexed: 05/04/2023]
Abstract
Sparse representation of information provides a powerful means to perform feature extraction on high-dimensional data and is of broad interest for applications in signal processing, computer vision, object recognition and neurobiology. Sparse coding is also believed to be a key mechanism by which biological neural systems can efficiently process a large amount of complex sensory data while consuming very little power. Here, we report the experimental implementation of sparse coding algorithms in a bio-inspired approach using a 32 × 32 crossbar array of analog memristors. This network enables efficient implementation of pattern matching and lateral neuron inhibition and allows input data to be sparsely encoded using neuron activities and stored dictionary elements. Different dictionary sets can be trained and stored in the same system, depending on the nature of the input signals. Using the sparse coding algorithm, we also perform natural image processing based on a learned dictionary.
Collapse
Affiliation(s)
- Patrick M Sheridan
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Fuxi Cai
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Chao Du
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Wen Ma
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Zhengya Zhang
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Wei D Lu
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan 48109, USA
| |
Collapse
|