Nonvolatile Memories in Spiking Neural Network Architectures: Current and Emerging Trends

The 3D Monolithically Integrated Hardware Based Neural System with Enhanced Memory Window of the Volatile and Non-Volatile Devices

3D neuromorphic hardware system is first demonstrated in neuromorphic application as on-chip level by integrating array devices with CMOS circuits after wafer bonding (WB) and interconnection process. The memory window of synaptic device is degraded after WB and 3 Dimesional (3D) integration due to process defects and thermal stress. To address this degradation, Ag diffusion in materials of Ta2O5 and HfO2 is studied in a volatile memristor, furthermore, the interconnection and gate metal Ru are investigated to reduce defective traps of gate interface in non-volatile memory devices. As a result, a memory window is improved over 106 in both types of devices. Improved and 3D integrated 12 × 14 array devices are identified in the synaptic characteristics according to the change of the synaptic weight from the interconnected Test Element Group (TEG) of the Complementary Metal Oxide Semiconductor (CMOS) circuits. The trained array devices present recognizable image of letters, achieving an accuracy rate of 92% when utilizing a convolutional neural network, comparing the normalized accuracy of 93% achieved by an ideal synapse device. This study proposes to modulate the memory windows up to 106 in an integrated hardware-based neural system, considering the possibility of device degradation in both volatile and non-volatile memory devices demonstrated by the hardware neural system.

Exploring the Effects of Caputo Fractional Derivative in Spiking Neural Network Training

Fractional calculus is an emerging topic in artificial neural network training, especially when using gradient-based methods. This paper brings the idea of fractional derivatives to spiking neural network training using Caputo derivative-based gradient calculation. We focus on conducting an extensive investigation of performance improvements via a case study of small-scale networks using derivative orders in the unit interval. With particle swarm optimization we provide an example of handling the derivative order as an optimizable hyperparameter to find viable values for it. Using multiple benchmark datasets we empirically show that there is no single generally optimal derivative order, rather this value is data-dependent. However, statistics show that a range of derivative orders can be determined where the Caputo derivative outperforms first-order gradient descent with high confidence. Improvements in convergence speed and training time are also examined and explained by the reformulation of the Caputo derivative-based training as an adaptive weight normalization technique.