1
|
Noh K, Kwak H, Son J, Kim S, Um M, Kang M, Kim D, Ji W, Lee J, Jo H, Woo J, Lee HM, Kim S. Retention-aware zero-shifting technique for Tiki-Taka algorithm-based analog deep learning accelerator. SCIENCE ADVANCES 2024; 10:eadl3350. [PMID: 38875324 PMCID: PMC11177898 DOI: 10.1126/sciadv.adl3350] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/18/2023] [Accepted: 05/10/2024] [Indexed: 06/16/2024]
Abstract
We present the fabrication of 4 K-scale electrochemical random-access memory (ECRAM) cross-point arrays for analog neural network training accelerator and an electrical characteristic of an 8 × 8 ECRAM array with a 100% yield, showing excellent switching characteristics, low cycle-to-cycle, and device-to-device variations. Leveraging the advances of the ECRAM array, we showcase its efficacy in neural network training using the Tiki-Taka version 2 algorithm (TTv2) tailored for non-ideal analog memory devices. Through an experimental study using ECRAM devices, we investigate the influence of retention characteristics on the training performance of TTv2, revealing that the relative location of the retention convergence point critically determines the available weight range and, consequently, affects the training accuracy. We propose a retention-aware zero-shifting technique designed to optimize neural network training performance, particularly in scenarios involving cross-point devices with limited retention times. This technique ensures robust and efficient analog neural network training despite the practical constraints posed by analog cross-point devices.
Collapse
Affiliation(s)
- Kyungmi Noh
- Department of Materials Science and Engineering, Pohang University of Science and Technology, Pohang 37673, Republic of Korea
| | - Hyunjeong Kwak
- Department of Materials Science and Engineering, Pohang University of Science and Technology, Pohang 37673, Republic of Korea
| | - Jeonghoon Son
- Department of Materials Science and Engineering, Pohang University of Science and Technology, Pohang 37673, Republic of Korea
| | - Seungkun Kim
- Department of Materials Science and Engineering, Pohang University of Science and Technology, Pohang 37673, Republic of Korea
| | - Minseong Um
- School of Electrical Engineering, Korea University, Seoul 02841, Republic of Korea
| | - Minil Kang
- Department of Semiconductor System Engineering, Korea University, Seoul 02841, Republic of Korea
| | - Doyoon Kim
- Department of Materials Science and Engineering, Pohang University of Science and Technology, Pohang 37673, Republic of Korea
| | - Wonjae Ji
- Department of Materials Science and Engineering, Pohang University of Science and Technology, Pohang 37673, Republic of Korea
| | - Junyong Lee
- Department of Materials Science and Engineering, Pohang University of Science and Technology, Pohang 37673, Republic of Korea
| | - HwiJeong Jo
- School of Electrical Engineering, Korea University, Seoul 02841, Republic of Korea
| | - Jiyong Woo
- Department of Electronics Engineering, Kyungpook National University, Daegu 41566, Republic of Korea
| | - Hyung-Min Lee
- School of Electrical Engineering, Korea University, Seoul 02841, Republic of Korea
| | - Seyoung Kim
- Department of Materials Science and Engineering, Pohang University of Science and Technology, Pohang 37673, Republic of Korea
| |
Collapse
|
2
|
Won J, Kang J, Hong S, Han N, Kang M, Park Y, Roh Y, Seo HJ, Joe C, Cho U, Kang M, Um M, Lee K, Yang J, Jung M, Lee H, Oh S, Kim S, Kim S. Device-Algorithm Co-Optimization for an On-Chip Trainable Capacitor-Based Synaptic Device with IGZO TFT and Retention-Centric Tiki-Taka Algorithm. ADVANCED SCIENCE (WEINHEIM, BADEN-WURTTEMBERG, GERMANY) 2023; 10:e2303018. [PMID: 37559176 PMCID: PMC10582414 DOI: 10.1002/advs.202303018] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/11/2023] [Revised: 06/28/2023] [Indexed: 08/11/2023]
Abstract
Analog in-memory computing synaptic devices are widely studied for efficient implementation of deep learning. However, synaptic devices based on resistive memory have difficulties implementing on-chip training due to the lack of means to control the amount of resistance change and large device variations. To overcome these shortcomings, silicon complementary metal-oxide semiconductor (Si-CMOS) and capacitor-based charge storage synapses are proposed, but it is difficult to obtain sufficient retention time due to Si-CMOS leakage currents, resulting in a deterioration of training accuracy. Here, a novel 6T1C synaptic device using only n-type indium gaIlium zinc oxide thin film transistor (IGZO TFT) with low leakage current and a capacitor is proposed, allowing not only linear and symmetric weight update but also sufficient retention time and parallel on-chip training operations. In addition, an efficient and realistic training algorithm to compensate for any remaining device non-idealities such as drifting references and long-term retention loss is proposed, demonstrating the importance of device-algorithm co-optimization.
Collapse
Affiliation(s)
- Jongun Won
- Department of Materials Science & EngineeringInter‐university Semiconductor Research CenterResearch Institute of Advanced MaterialsSeoul National UniversitySeoul08826Republic of Korea
| | - Jaehyeon Kang
- Department of Materials Science & EngineeringInter‐university Semiconductor Research CenterResearch Institute of Advanced MaterialsSeoul National UniversitySeoul08826Republic of Korea
| | - Sangjun Hong
- Device SolutionsSamsung ElectronicsPyeongtaek17786Republic of Korea
| | - Narae Han
- Department of Materials Science & EngineeringInter‐university Semiconductor Research CenterResearch Institute of Advanced MaterialsSeoul National UniversitySeoul08826Republic of Korea
| | - Minseung Kang
- Department of Materials Science & EngineeringInter‐university Semiconductor Research CenterResearch Institute of Advanced MaterialsSeoul National UniversitySeoul08826Republic of Korea
| | - Yeaji Park
- Department of Materials Science & EngineeringInter‐university Semiconductor Research CenterResearch Institute of Advanced MaterialsSeoul National UniversitySeoul08826Republic of Korea
| | - Youngchae Roh
- Department of Materials Science & EngineeringInter‐university Semiconductor Research CenterResearch Institute of Advanced MaterialsSeoul National UniversitySeoul08826Republic of Korea
| | - Hyeong Jun Seo
- Department of Materials Science & EngineeringInter‐university Semiconductor Research CenterResearch Institute of Advanced MaterialsSeoul National UniversitySeoul08826Republic of Korea
| | - Changhoon Joe
- Department of Materials Science & EngineeringInter‐university Semiconductor Research CenterResearch Institute of Advanced MaterialsSeoul National UniversitySeoul08826Republic of Korea
| | - Ung Cho
- Department of Materials Science & EngineeringInter‐university Semiconductor Research CenterResearch Institute of Advanced MaterialsSeoul National UniversitySeoul08826Republic of Korea
| | - Minil Kang
- Department of Semiconductor System EngineeringKorea UniversitySeoul02841Republic of Korea
| | - Minseong Um
- School of Electrical EngineeringKorea UniversitySeoul02841Republic of Korea
| | - Kwang‐Hee Lee
- Samsung Advanced Institute of Technology (SAIT)Samsung ElectronicsSuwon‐si16678Republic of Korea
| | - Jee‐Eun Yang
- Samsung Advanced Institute of Technology (SAIT)Samsung ElectronicsSuwon‐si16678Republic of Korea
| | - Moonil Jung
- Samsung Advanced Institute of Technology (SAIT)Samsung ElectronicsSuwon‐si16678Republic of Korea
| | - Hyung‐Min Lee
- School of Electrical EngineeringKorea UniversitySeoul02841Republic of Korea
| | - Saeroonter Oh
- Department of Electrical and Electronic EngineeringHanyang UniversityAnsan15588Republic of Korea
| | - Sangwook Kim
- Samsung Advanced Institute of Technology (SAIT)Samsung ElectronicsSuwon‐si16678Republic of Korea
| | - Sangbum Kim
- Department of Materials Science & EngineeringInter‐university Semiconductor Research CenterResearch Institute of Advanced MaterialsSeoul National UniversitySeoul08826Republic of Korea
| |
Collapse
|
3
|
Abedin M, Gong N, Beckmann K, Liehr M, Saraf I, Van der Straten O, Ando T, Cady N. Material to system-level benchmarking of CMOS-integrated RRAM with ultra-fast switching for low power on-chip learning. Sci Rep 2023; 13:14963. [PMID: 37697024 PMCID: PMC10495451 DOI: 10.1038/s41598-023-42214-x] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2023] [Accepted: 09/06/2023] [Indexed: 09/13/2023] Open
Abstract
Analog hardware-based training provides a promising solution to developing state-of-the-art power-hungry artificial intelligence models. Non-volatile memory hardware such as resistive random access memory (RRAM) has the potential to provide a low power alternative. The training accuracy of analog hardware depends on RRAM switching properties including the number of discrete conductance states and conductance variability. Furthermore, the overall power consumption of the system inversely correlates with the RRAM devices conductance. To study material dependence of these properties, TaOx and HfOx RRAM devices in one-transistor one-RRAM configuration (1T1R) were fabricated using a custom 65 nm CMOS fabrication process. Analog switching performance was studied with a range of initial forming compliance current (200-500 µA) and analog switching tests with ultra-short pulse width (300 ps) was carried out. We report that by utilizing low current during electroforming and high compliance current during analog switching, a large number of RRAM conductance states can be achieved while maintaining low conductance state. While both TaOx and HfOx could be switched to more than 20 distinct states, TaOx devices exhibited 10× lower conductance, which reduces total power consumption for array-level operations. Furthermore, we adopted an analog, fully in-memory training algorithm for system-level training accuracy benchmarking and showed that implementing TaOx 1T1R cells could yield an accuracy of up to 96.4% compared to 97% for the floating-point arithmetic baseline, while implementing HfOx devices would yield a maximum accuracy of 90.5%. Our experimental work and benchmarking approach paves the path for future materials engineering in analog-AI hardware for a low-power environment training.
Collapse
Affiliation(s)
- Minhaz Abedin
- University at Albany, College of Nanotechnology, Science and Engineering, Albany, NY, 12203, USA
- IBM Research, Albany, NY, 12203, USA
| | - Nanbo Gong
- IBM Thomas J. Watson Research Center, Yorktown Heights, NY, 10598, USA
| | - Karsten Beckmann
- University at Albany, College of Nanotechnology, Science and Engineering, Albany, NY, 12203, USA
- NY CREATES, Albany, NY, 12203, USA
| | - Maximilian Liehr
- University at Albany, College of Nanotechnology, Science and Engineering, Albany, NY, 12203, USA
| | | | | | - Takashi Ando
- IBM Thomas J. Watson Research Center, Yorktown Heights, NY, 10598, USA
| | - Nathaniel Cady
- University at Albany, College of Nanotechnology, Science and Engineering, Albany, NY, 12203, USA.
| |
Collapse
|