Bayesian Estimation and Inference Using Stochastic Electronics

Implementation of Bayesian networks and Bayesian inference using a Cu0.1Te0.9/HfO2/Pt threshold switching memristor

Bayesian networks and Bayesian inference, which forecast uncertain causal relationships within a stochastic framework, are used in various artificial intelligence applications. However, implementing hardware circuits for the Bayesian inference has shortcomings regarding device performance and circuit complexity. This work proposed a Bayesian network and inference circuit using a Cu0.1Te0.9/HfO2/Pt volatile memristor, a probabilistic bit neuron that can control the probability of being 'true' or 'false.' Nodal probabilities within the network are feasibly sampled with low errors, even with the device's cycle-to-cycle variations. Furthermore, Bayesian inference of all conditional probabilities within the network is implemented with low power (<186 nW) and energy consumption (441.4 fJ), and a normalized mean squared error of ∼7.5 × 10-4 through division feedback logic with a variational learning rate to suppress the inherent variation of the memristor. The suggested memristor-based Bayesian network shows the potential to replace the conventional complementary metal oxide semiconductor-based Bayesian estimation method with power efficiency using a stochastic computing method.

Controllable SiOx Nanorod Memristive Neuron for Probabilistic Bayesian Inference

Modern artificial neural network technology using a deterministic computing framework is faced with a critical challenge in dealing with massive data that are largely unstructured and ambiguous. This challenge demands the advances of an elementary physical device for tackling these uncertainties. Here, we designed and fabricated a SiO_x nanorod memristive device by employing the glancing angle deposition (GLAD) technique, suggesting a controllable stochastic artificial neuron that can mimic the fundamental integrate-and-fire signaling and stochastic dynamics of a biological neuron. The nanorod structure provides the random distribution of multiple nanopores all across the active area, capable of forming a multitude of Si filaments at many SiO_x nanorod edges after the electromigration process, leading to a stochastic switching event with very high dynamic range (≈5.15 × 10¹⁰ ) and low energy (≈4.06 pJ). Different probabilistic activation (ProbAct) functions in a sigmoid form are implemented, showing its controllability with low variation by manufacturing and electrical programming schemes. Furthermore, as an application prospect, based on the suggested memristive neuron, we demonstrated the self-resting neural operation with the local circuit configuration and revealed probabilistic Bayesian inferences for genetic regulatory networks with low normalized mean squared errors (≈2.41 × 10^-2 ) and its robustness to the ProbAct variation.

Brain-Inspired Hardware Solutions for Inference in Bayesian Networks

The implementation of inference (i.e., computing posterior probabilities) in Bayesian networks using a conventional computing paradigm turns out to be inefficient in terms of energy, time, and space, due to the substantial resources required by floating-point operations. A departure from conventional computing systems to make use of the high parallelism of Bayesian inference has attracted recent attention, particularly in the hardware implementation of Bayesian networks. These efforts lead to several implementations ranging from digital circuits, mixed-signal circuits, to analog circuits by leveraging new emerging nonvolatile devices. Several stochastic computing architectures using Bayesian stochastic variables have been proposed, from FPGA-like architectures to brain-inspired architectures such as crossbar arrays. This comprehensive review paper discusses different hardware implementations of Bayesian networks considering different devices, circuits, and architectures, as well as a more futuristic overview to solve existing hardware implementation problems.

Hardware Design for Autonomous Bayesian Networks

Directed acyclic graphs or Bayesian networks that are popular in many AI-related sectors for probabilistic inference and causal reasoning can be mapped to probabilistic circuits built out of probabilistic bits (p-bits), analogous to binary stochastic neurons of stochastic artificial neural networks. In order to satisfy standard statistical results, individual p-bits not only need to be updated sequentially but also in order from the parent to the child nodes, necessitating the use of sequencers in software implementations. In this article, we first use SPICE simulations to show that an autonomous hardware Bayesian network can operate correctly without any clocks or sequencers, but only if the individual p-bits are appropriately designed. We then present a simple behavioral model of the autonomous hardware illustrating the essential characteristics needed for correct sequencer-free operation. This model is also benchmarked against SPICE simulations and can be used to simulate large-scale networks. Our results could be useful in the design of hardware accelerators that use energy-efficient building blocks suited for low-level implementations of Bayesian networks. The autonomous massively parallel operation of our proposed stochastic hardware has biological relevance since neural dynamics in brain is also stochastic and autonomous by nature.

Computing Generalized Matrix Inverse on Spiking Neural Substrate

Emerging neural hardware substrates, such as IBM's TrueNorth Neurosynaptic System, can provide an appealing platform for deploying numerical algorithms. For example, a recurrent Hopfield neural network can be used to find the Moore-Penrose generalized inverse of a matrix, thus enabling a broad class of linear optimizations to be solved efficiently, at low energy cost. However, deploying numerical algorithms on hardware platforms that severely limit the range and precision of representation for numeric quantities can be quite challenging. This paper discusses these challenges and proposes a rigorous mathematical framework for reasoning about range and precision on such substrates. The paper derives techniques for normalizing inputs and properly quantizing synaptic weights originating from arbitrary systems of linear equations, so that solvers for those systems can be implemented in a provably correct manner on hardware-constrained neural substrates. The analytical model is empirically validated on the IBM TrueNorth platform, and results show that the guarantees provided by the framework for range and precision hold under experimental conditions. Experiments with optical flow demonstrate the energy benefits of deploying a reduced-precision and energy-efficient generalized matrix inverse engine on the IBM TrueNorth platform, reflecting 10× to 100× improvement over FPGA and ARM core baselines.

Stochastic Spin-Orbit Torque Devices as Elements for Bayesian Inference

Probabilistic inference from real-time input data is becoming increasingly popular and may be one of the potential pathways at enabling cognitive intelligence. As a matter of fact, preliminary research has revealed that stochastic functionalities also underlie the spiking behavior of neurons in cortical microcircuits of the human brain. In tune with such observations, neuromorphic and other unconventional computing platforms have recently started adopting the usage of computational units that generate outputs probabilistically, depending on the magnitude of the input stimulus. In this work, we experimentally demonstrate a spintronic device that offers a direct mapping to the functionality of such a controllable stochastic switching element. We show that the probabilistic switching of Ta/CoFeB/MgO heterostructures in presence of spin-orbit torque and thermal noise can be harnessed to enable probabilistic inference in a plethora of unconventional computing scenarios. This work can potentially pave the way for hardware that directly mimics the computational units of Bayesian inference.