Artificial neural networks in hardware: A survey of two decades of progress

Comparison of deep and conventional machine learning models for prediction of one supply chain management distribution cost

Strategic supply chain management (SCM) is essential for organizations striving to optimize performance and attain their goals. Prediction of supply chain management distribution cost (SCMDC) is one branch of SCM and it's essential for organizations striving to optimize performance and attain their goals. For this purpose, four machine learning algorithms, including random forest (RF), support vector machine (SVM), multilayer perceptron (MLP) and decision tree (DT), along with deep learning using convolutional neural network (CNN), was used to predict and analyze SCMDC. A comprehensive dataset consisting of 180,519 open-source data points was used for analyze and make the structure of each algorithm. Evaluation based on Root Mean Square Error (RMSE) and Correlation coefficient (R2) show the CNN model has high accuracy in SCMDC prediction than other models. The CNN algorithm demonstrated exceptional accuracy on the test dataset, with an RMSE of RMSE of 0.528 and an R2 value of 0.953. Notable advantages of CNNs include automatic learning of hierarchical features, proficiency in capturing spatial and temporal patterns, computational efficiency, robustness to data variations, minimal preprocessing requirements, end-to-end training capability, scalability, and widespread adoption supported by extensive research. These attributes position the CNN algorithm as the preferred choice for precise and reliable SCMDC predictions, especially in scenarios requiring rapid responses and limited computational resources.

Optical neural networks: progress and challenges

Artificial intelligence has prevailed in all trades and professions due to the assistance of big data resources, advanced algorithms, and high-performance electronic hardware. However, conventional computing hardware is inefficient at implementing complex tasks, in large part because the memory and processor in its computing architecture are separated, performing insufficiently in computing speed and energy consumption. In recent years, optical neural networks (ONNs) have made a range of research progress in optical computing due to advantages such as sub-nanosecond latency, low heat dissipation, and high parallelism. ONNs are in prospect to provide support regarding computing speed and energy consumption for the further development of artificial intelligence with a novel computing paradigm. Herein, we first introduce the design method and principle of ONNs based on various optical elements. Then, we successively review the non-integrated ONNs consisting of volume optical components and the integrated ONNs composed of on-chip components. Finally, we summarize and discuss the computational density, nonlinearity, scalability, and practical applications of ONNs, and comment on the challenges and perspectives of the ONNs in the future development trends.

Robust noise-aware algorithm for randomized neural network and its convergence properties

The concept of randomized neural networks (RNNs), such as the random vector functional link network (RVFL) and extreme learning machine (ELM), is a widely accepted and efficient network method for constructing single-hidden layer feedforward networks (SLFNs). Due to its exceptional approximation capabilities, RNN is being extensively used in various fields. While the RNN concept has shown great promise, its performance can be unpredictable in imperfect conditions, such as weight noises and outliers. Thus, there is a need to develop more reliable and robust RNN algorithms. To address this issue, this paper proposes a new objective function that addresses the combined effect of weight noise and training data outliers for RVFL networks. Based on the half-quadratic optimization method, we then propose a novel algorithm, named noise-aware RNN (NARNN), to optimize the proposed objective function. The convergence of the NARNN is also theoretically validated. We also discuss the way to use the NARNN for ensemble deep RVFL (edRVFL) networks. Finally, we present an extension of the NARNN to concurrently address weight noise, stuck-at-fault, and outliers. The experimental results demonstrate that the proposed algorithm outperforms a number of state-of-the-art robust RNN algorithms.

A comprehensive review of advanced trends: from artificial synapses to neuromorphic systems with consideration of non-ideal effects

A neuromorphic system is composed of hardware-based artificial neurons and synaptic devices, designed to improve the efficiency of neural computations inspired by energy-efficient and parallel operations of the biological nervous system. A synaptic device-based array can compute vector-matrix multiplication (VMM) with given input voltage signals, as a non-volatile memory device stores the weight information of the neural network in the form of conductance or capacitance. However, unlike software-based neural networks, the neuromorphic system unavoidably exhibits non-ideal characteristics that can have an adverse impact on overall system performance. In this study, the characteristics required for synaptic devices and their importance are discussed, depending on the targeted application. We categorize synaptic devices into two types: conductance-based and capacitance-based, and thoroughly explore the operations and characteristics of each device. The array structure according to the device structure and the VMM operation mechanism of each structure are analyzed, including recent advances in array-level implementation of synaptic devices. Furthermore, we reviewed studies to minimize the effect of hardware non-idealities, which degrades the performance of hardware neural networks. These studies introduce techniques in hardware and signal engineering, as well as software-hardware co-optimization, to address these non-idealities through compensation approaches.

ANN-based fault diagnosis of induction motor under stator inter-turn short-circuits and unbalanced supply voltage

Induction motors (IMs) are extensively used in industrial sector. This kind of machine is subjected to several stresses that could interrupt their normal operation. An excessive stress can generate some symptoms before the IM fall in failure situation. Therefore, incipient detection of these symptoms permits the shutdown of IMs in order to avoid total destruction. Fault detection is then the main objective of diagnosis systems. Stator inter-turn short-circuits (SITSC) constitutes an important amount of cause of IM breakdown. However; unbalance supply voltage (USV) is one of the advantageous factors that affect IMs operation. Thus, in order to avoid false alarm induced by USV, the diagnosis system must make difference between USV and SITSC faults. This paper presents an efficient approach to estimate SITSC percentage and detect USV occurrence using Artificial Intelligent (AI) tool. Artificial neuronal network (ANN) plays the key-role of the proposed diagnosis system. A fault Classifier of SITSC and USV is carried out using multi-layer perceptron neuronal network (MLP-NN). The training, testing and validation phases of MLP-NN need the dataset creation. The required data is obtained from both simulated mathematical model of IM and laboratory test-bed. The reached results show the sensitivity and the well-functioning of the proposed diagnosis system.

Dynamic Moore-Penrose Inversion With Unknown Derivatives: Gradient Neural Network Approach

Finding dynamic Moore-Penrose inverses (DMPIs) in real-time is a challenging problem due to the time-varying nature of the inverse. Traditional numerical methods for static Moore-Penrose inverse are not efficient for calculating DMPIs and are restricted by serial processing. The current state-of-the-art method for finding DMPIs is called the zeroing neural network (ZNN) method, which requires that the time derivative of the associated matrix is available all the time during the solution process. However, in practice, the time derivative of the associated dynamic matrix may not be available in a real-time manner or be subject to noises caused by differentiators. In this article, we propose a novel gradient-based neural network (GNN) method for computing DMPIs, which does not need the time derivative of the associated dynamic matrix. In particular, the neural state matrix of the proposed GNN converges to the theoretical DMPI in finite time. The finite-time convergence is kept by simply setting a large parameter when there are additive noises in the implementation of the GNN model. Simulation results demonstrate the efficacy and superiority of the proposed GNN method.

AMITE: A Novel Polynomial Expansion for Analyzing Neural Network Nonlinearities

Polynomial expansions are important in the analysis of neural network nonlinearities. They have been applied thereto addressing well-known difficulties in verification, explainability, and security. Existing approaches span classical Taylor and Chebyshev methods, asymptotics, and many numerical approaches. We find that, while these have useful properties individually, such as exact error formulas, adjustable domain, and robustness to undefined derivatives, there are no approaches that provide a consistent method, yielding an expansion with all these properties. To address this, we develop an analytically modified integral transform expansion (AMITE), a novel expansion via integral transforms modified using derived criteria for convergence. We show the general expansion and then demonstrate an application for two popular activation functions: hyperbolic tangent and rectified linear units. Compared with existing expansions (i.e., Chebyshev, Taylor, and numerical) employed to this end, AMITE is the first to provide six previously mutually exclusive desired expansion properties, such as exact formulas for the coefficients and exact expansion errors. We demonstrate the effectiveness of AMITE in two case studies. First, a multivariate polynomial form is efficiently extracted from a single hidden layer black-box multilayer perceptron (MLP) to facilitate equivalence testing from noisy stimulus-response pairs. Second, a variety of feedforward neural network (FFNN) architectures having between three and seven layers are range bounded using Taylor models improved by the AMITE polynomials and error formulas. AMITE presents a new dimension of expansion methods suitable for the analysis/approximation of nonlinearities in neural networks, opening new directions and opportunities for the theoretical analysis and systematic testing of neural networks.

Dual functional states of working memory realized by memristor-based neural network

Working memory refers to the brain's ability to store and manipulate information for a short period. It is disputably considered to rely on two mechanisms: sustained neuronal firing, and "activity-silent" working memory. To develop a highly biologically plausible neuromorphic computing system, it is anticipated to physically realize working memory that corresponds to both of these mechanisms. In this study, we propose a memristor-based neural network to realize the sustained neural firing and activity-silent working memory, which are reflected as dual functional states within memory. Memristor-based synapses and two types of artificial neurons are designed for the Winner-Takes-All learning rule. During the cognitive task, state transformation between the "focused" state and the "unfocused" state of working memory is demonstrated. This work paves the way for further emulating the complex working memory functions with distinct neural activities in our brains.

Artificial Intelligence and Advanced Materials

Artificial intelligence (AI) is gaining strength, and materials science can both contribute to and profit from it. In a simultaneous progress race, new materials, systems, and processes can be devised and optimized thanks to machine learning (ML) techniques, and such progress can be turned into innovative computing platforms. Future materials scientists will profit from understanding how ML can boost the conception of advanced materials. This review covers aspects of computation from the fundamentals to directions taken and repercussions produced by computation to account for the origins, procedures, and applications of AI. ML and its methods are reviewed to provide basic knowledge of its implementation and its potential. The materials and systems used to implement AI with electric charges are finding serious competition from other information-carrying and processing agents. The impact these techniques have on the inception of new advanced materials is so deep that a new paradigm is developing where implicit knowledge is being mined to conceive materials and systems for functions instead of finding applications to found materials. How far this trend can be carried is hard to fathom, as exemplified by the power to discover unheard of materials or physical laws buried in data.

Energy-Efficient Hardware Implementation of Fully Connected Artificial Neural Networks Using Approximate Arithmetic Blocks

In this paper, we explore efficient hardware implementation of feedforward artificial neural networks (ANNs) using approximate adders and multipliers. Due to a large area requirement in a parallel architecture, the ANNs are implemented under the time-multiplexed architecture where computing resources are re-used in the multiply accumulate (MAC) blocks. The efficient hardware implementation of ANNs is realized by replacing the exact adders and multipliers in the MAC blocks by the approximate ones taking into account the hardware accuracy. Additionally, an algorithm to determine the approximate level of multipliers and adders due to the expected accuracy is proposed. As an application, the MNIST and SVHN databases are considered. To examine the efficiency of the proposed method, various architectures and structures of ANNs are realized. Experimental results show that the ANNs designed using the proposed approximate multiplier have a smaller area and consume less energy than those designed using previously proposed prominent approximate multipliers. It is also observed that the use of both approximate adders and multipliers yields, respectively, up to 50% and 10% reduction in energy consumption and area of the ANN design with a small deviation or better hardware accuracy when compared to the exact adders and multipliers.

Single dendritic neural classification with an effective spherical search-based whale learning algorithm

McCulloch-Pitts neuron-based neural networks have been the mainstream deep learning methods, achieving breakthrough in various real-world applications. However, McCulloch-Pitts neuron is also under longtime criticism of being overly simplistic. To alleviate this issue, the dendritic neuron model (DNM), which employs non-linear information processing capabilities of dendrites, has been widely used for prediction and classification tasks. In this study, we innovatively propose a hybrid approach to co-evolve DNM in contrast to back propagation (BP) techniques, which are sensitive to initial circumstances and readily fall into local minima. The whale optimization algorithm is improved by spherical search learning to perform co-evolution through dynamic hybridizing. Eleven classification datasets were selected from the well-known UCI Machine Learning Repository. Its efficiency in our model was verified by statistical analysis of convergence speed and Wilcoxon sign-rank tests, with receiver operating characteristic curves and the calculation of area under the curve. In terms of classification accuracy, the proposed co-evolution method beats 10 existing cutting-edge non-BP methods and BP, suggesting that well-learned DNMs are computationally significantly more potent than conventional McCulloch-Pitts types and can be employed as the building blocks for the next-generation deep learning methods.

Performance analysis of multiple input single layer neural network hardware chip

An artificial neural network (ANN) is a computational system that is designed to replicate and process the behavior of the human brain using neuron nodes. ANNs are made up of thousands of processing neurons with input and output modules that self-learn and compute data to offer the best results. The hardware realization of the massive neuron system is a difficult task. The research article emphasizes the design and realization of multiple input perceptron chips in Xilinx integrated system environment (ISE) 14.7 software. The proposed single-layer ANN architecture is scalable and accepts variable 64 inputs. The design is distributed in eight parallel blocks of ANN in which one block consists of eight neurons. The performance of the chip is analyzed based on the hardware utilization, memory, combinational delay, and different processing elements with targeted hardware Virtex-5 field-programmable gate array (FPGA). The chip simulation is performed in Modelsim 10.0 software. Artificial intelligence has a wide range of applications, and cutting-edge computing technology has a vast market. Hardware processors that are fast, affordable, and suited for ANN applications and accelerators are being developed by the industries. The novelty of the work is that it provides a parallel and scalable design platform on FPGA for fast switching, which is the current need in the forthcoming neuromorphic hardware.

S3NN: Time step reduction of spiking surrogate gradients for training energy efficient single-step spiking neural networks

As the scales of neural networks increase, techniques that enable them to run with low computational cost and energy efficiency are required. From such demands, various efficient neural network paradigms, such as spiking neural networks (SNNs) or binary neural networks (BNNs), have been proposed. However, they have sticky drawbacks, such as degraded inference accuracy and latency. To solve these problems, we propose a single-step spiking neural network (S³NN), an energy-efficient neural network with low computational cost and high precision. The proposed S³NN processes the information between hidden layers by spikes as SNNs. Nevertheless, it has no temporal dimension so that there is no latency within training and inference phases as BNNs. Thus, the proposed S³NN has a lower computational cost than SNNs that require time-series processing. However, S³NN cannot adopt naïve backpropagation algorithms due to the non-differentiability nature of spikes. We deduce a suitable neuron model by reducing the surrogate gradient for multi-time step SNNs to a single-time step. We experimentally demonstrated that the obtained surrogate gradient allows S³NN to be trained appropriately. We also showed that the proposed S³NN could achieve comparable accuracy to full-precision networks while being highly energy-efficient.

Optical neural ordinary differential equations

Increasing the layer number of on-chip photonic neural networks (PNNs) is essential to improve its model performance. However, the successive cascading of network hidden layers results in larger integrated photonic chip areas. To address this issue, we propose the optical neural ordinary differential equations (ON-ODEs) architecture that parameterizes the continuous dynamics of hidden layers with optical ODE solvers. The ON-ODE comprises the PNNs followed by the photonic integrator and optical feedback loop, which can be configured to represent residual neural networks (ResNets) and implement the function of recurrent neural networks with effectively reduced chip area occupancy. For the interference-based optoelectronic nonlinear hidden layer, the numerical experiments demonstrate that the single hidden layer ON-ODE can achieve approximately the same accuracy as the two-layer optical ResNets in image classification tasks. In addition, the ON-ODE improves the model classification accuracy for the diffraction-based all-optical linear hidden layer. The time-dependent dynamics property of ON-ODE is further applied for trajectory prediction with high accuracy.

Nanofluidic computing makes a splash

Ionic computing raises the possibility of devices that operate similarly to the human brain.

Photonic machine learning with on-chip diffractive optics

Machine learning technologies have been extensively applied in high-performance information-processing fields. However, the computation rate of existing hardware is severely circumscribed by conventional Von Neumann architecture. Photonic approaches have demonstrated extraordinary potential for executing deep learning processes that involve complex calculations. In this work, an on-chip diffractive optical neural network (DONN) based on a silicon-on-insulator platform is proposed to perform machine learning tasks with high integration and low power consumption characteristics. To validate the proposed DONN, we fabricated 1-hidden-layer and 3-hidden-layer on-chip DONNs with footprints of 0.15 mm2 and 0.3 mm2 and experimentally verified their performance on the classification task of the Iris plants dataset, yielding accuracies of 86.7% and 90%, respectively. Furthermore, a 3-hidden-layer on-chip DONN is fabricated to classify the Modified National Institute of Standards and Technology handwritten digit images. The proposed passive on-chip DONN provides a potential solution for accelerating future artificial intelligence hardware with enhanced performance.

Atomically Thin Synapse Networks on Van Der Waals Photo-Memtransistors

A new type of atomically thin synaptic network on van der Waals (vdW) heterostructures is reported, where each ultrasmall cell (≈2 nm thick) built with trilayer WS₂ semiconductor acts as a gate-tunable photoactive synapse, i.e., a photo-memtransistor. A train of UV pulses onto the WS₂ memristor generates dopants in atomic-level precision by direct light-lattice interactions, which, along with the gate tunability, leads to the accurate modulation of the channel conductance for potentiation and depression of the synaptic cells. Such synaptic dynamics can be explained by a parallel atomistic resistor network model. In addition, it is shown that such a device scheme can generally be realized in other 2D vdW semiconductors, such as MoS₂ , MoSe₂ , MoTe₂ , and WSe₂ . Demonstration of these atomically thin photo-memtransistor arrays, where the synaptic weights can be tuned for the atomistic defect density, provides implications for a new type of artificial neural networks for parallel matrix computations with an ultrahigh integration density.

Machine fault detection methods based on machine learning algorithms: A review

Preventive identification of mechanical parts failures has always played a crucial role in machine maintenance. Over time, as the processing cycles are repeated, the machinery in the production system is subject to wear with a consequent loss of technical efficiency compared to optimal conditions. These conditions can, in some cases, lead to the breakage of the elements with consequent stoppage of the production process pending the replacement of the element. This situation entails a large loss of turnover on the part of the company. For this reason, it is crucial to be able to predict failures in advance to try to replace the element before its wear can cause a reduction in machine performance. Several systems have recently been developed for the preventive faults detection that use a combination of low-cost sensors and algorithms based on machine learning. In this work the different methodologies for the identification of the most common mechanical failures are examined and the most widely applied algorithms based on machine learning are analyzed: Support Vector Machine (SVM) solutions, Artificial Neural Network (ANN) algorithms, Convolutional Neural Network (CNN) model, Recurrent Neural Network (RNN) applications, and Deep Generative Systems. These topics have been described in detail and the works most appreciated by the scientific community have been reviewed to highlight the strengths in identifying faults and to outline the directions for future challenges.

Noise-mitigation strategies in physical feedforward neural networks

Physical neural networks are promising candidates for next generation artificial intelligence hardware. In such architectures, neurons and connections are physically realized and do not leverage digital concepts with their practically infinite signal-to-noise ratio to encode, transduce, and transform information. They, therefore, are prone to noise with a variety of statistical and architectural properties, and effective strategies leveraging network-inherent assets to mitigate noise in a hardware-efficient manner are important in the pursuit of next generation neural network hardware. Based on analytical derivations, we here introduce and analyze a variety of different noise-mitigation approaches. We analytically show that intra-layer connections in which the connection matrix's squared mean exceeds the mean of its square fully suppress uncorrelated noise. We go beyond and develop two synergistic strategies for noise that is uncorrelated and correlated across populations of neurons. First, we introduce the concept of ghost neurons, where each group of neurons perturbed by correlated noise has a negative connection to a single neuron, yet without receiving any input information. Second, we show that pooling of neuron populations is an efficient approach to suppress uncorrelated noise. As such, we developed a general noise-mitigation strategy leveraging the statistical properties of the different noise terms most relevant in analog hardware. Finally, we demonstrate the effectiveness of this combined approach for a trained neural network classifying the modified National Institute of Standards and Technology handwritten digits, for which we achieve a fourfold improvement of the output signal-to-noise ratio. Our noise mitigation lifts the 92.07% classification accuracy of the noisy neural network to 97.49%, which is essentially identical to the 97.54% of the noise-free network.

Neural Schrödinger Equation: Physical Law as Deep Neural Network

We show a new family of neural networks based on the Schrödinger equation (SE-NET). In this analogy, the trainable weights of the neural networks correspond to the physical quantities of the Schrödinger equation. These physical quantities can be trained using the complex-valued adjoint method. Since the propagation of the SE-NET can be described by the evolution of physical systems, its outputs can be computed by using a physical solver. The trained network is transferable to actual optical systems. As a demonstration, we implemented the SE-NET with the Crank-Nicolson finite difference method on Pytorch. From the results of numerical simulations, we found that the performance of the SE-NET becomes better when the SE-NET becomes wider and deeper. However, the training of the SE-NET was unstable due to gradient explosions when SE-NET becomes deeper. Therefore, we also introduced phase-only training, which only updates the phase of the potential field (refractive index) in the Schrödinger equation. This enables stable training even for the deep SE-NET model because the unitarity of the system is kept under the training. In addition, the SE-NET enables a joint optimization of physical structures and digital neural networks. As a demonstration, we performed a numerical demonstration of end-to-end machine learning (ML) with an optical frontend toward a compact spectrometer. Our results extend the application field of ML to hybrid physical-digital optimizations.

Reservoir Computing for Temporal Data Classification Using a Dynamic Solid Electrolyte ZnO Thin Film Transistor

The processing of sequential and temporal data is essential to computer vision and speech recognition, two of the most common applications of artificial intelligence (AI). Reservoir computing (RC) is a branch of AI that offers a highly efficient framework for processing temporal inputs at a low training cost compared to conventional Recurrent Neural Networks (RNNs). However, despite extensive effort, two-terminal memristor-based reservoirs have, until now, been implemented to process sequential data by reading their conductance states only once, at the end of the entire sequence. This method reduces the dimensionality, related to the number of signals from the reservoir and thereby lowers the overall performance of reservoir systems. Higher dimensionality facilitates the separation of originally inseparable inputs by reading out from a larger set of spatiotemporal features of inputs. Moreover, memristor-based reservoirs either use multiple pulse rates, fast or slow read (immediately or with a delay introduced after the end of the sequence), or excitatory pulses to enhance the dimensionality of reservoir states. This adds to the complexity of the reservoir system and reduces power efficiency. In this paper, we demonstrate the first reservoir computing system based on a dynamic three terminal solid electrolyte ZnO/Ta2O5 Thin-film Transistor fabricated at less than 100°C. The inherent nonlinearity and dynamic memory of the device lead to a rich separation property of reservoir states that results in, to our knowledge, the highest accuracy of 94.44%, using electronic charge-based system, for the classification of hand-written digits. This improvement is attributed to an increase in the dimensionality of the reservoir by reading the reservoir states after each pulse rather than at the end of the sequence. The third terminal enables a read operation in the off state, that is when no pulse is applied at the gate terminal, via a small read pulse at the drain. This fundamentally allows multiple read operations without increasing energy consumption, which is not possible in the conventional two-terminal memristor counterpart. Further, we have also shown that devices do not saturate even after multiple write pulses which demonstrates the device’s ability to process longer sequences.

Stochastic Resonance in Organic Electronic Devices

Stochastic Resonance (SR) is a phenomenon in which noise improves the performance of a system. With the addition of noise, a weak input signal to a nonlinear system, which may exceed its threshold, is transformed into an output signal. In the other words, noise-driven signal transfer is achieved. SR has been observed in nonlinear response systems, such as biological and artificial systems, and this review will focus mainly on examples of previous studies of mathematical models and experimental realization of SR using poly(hexylthiophene)-based organic field-effect transistors (OFETs). This phenomenon may contribute to signal processing with low energy consumption. However, the generation of SR requires a noise source. Therefore, the focus is on OFETs using materials such as organic materials with unstable electrical properties and critical elements due to unidirectional signal transmission, such as neural synapses. It has been reported that SR can be observed in OFETs by application of external noise. However, SR does not occur under conditions where the input signal exceeds the OFET threshold without external noise. Here, we present an example of a study that analyzes the behavior of SR in OFET systems and explain how SR can be made observable. At the same time, the role of internal noise in OFETs will be explained.

Understanding and mitigating noise in trained deep neural networks

Deep neural networks unlocked a vast range of new applications by solving tasks of which many were previously deemed as reserved to higher human intelligence. One of the developments enabling this success was a boost in computing power provided by special purpose hardware, such as graphic or tensor processing units. However, these do not leverage fundamental features of neural networks like parallelism and analog state variables. Instead, they emulate neural networks relying on binary computing, which results in unsustainable energy consumption and comparatively low speed. Fully parallel and analogue hardware promises to overcome these challenges, yet the impact of analogue neuron noise and its propagation, i.e. accumulation, threatens rendering such approaches inept. Here, we determine for the first time the propagation of noise in deep neural networks comprising noisy nonlinear neurons in trained fully connected layers. We study additive and multiplicative as well as correlated and uncorrelated noise, and develop analytical methods that predict the noise level in any layer of symmetric deep neural networks or deep neural networks trained with back propagation. We find that noise accumulation is generally bound, and adding additional network layers does not worsen the signal to noise ratio beyond a limit. Most importantly, noise accumulation can be suppressed entirely when neuron activation functions have a slope smaller than unity. We therefore developed the framework for noise in fully connected deep neural networks implemented in analog systems, and identify criteria allowing engineers to design noise-resilient novel neural network hardware.

Digital Implementation of Oscillatory Neural Network for Image Recognition Applications

Computing paradigm based on von Neuman architectures cannot keep up with the ever-increasing data growth (also called “data deluge gap”). This has resulted in investigating novel computing paradigms and design approaches at all levels from materials to system-level implementations and applications. An alternative computing approach based on artificial neural networks uses oscillators to compute or Oscillatory Neural Networks (ONNs). ONNs can perform computations efficiently and can be used to build a more extensive neuromorphic system. Here, we address a fundamental problem: can we efficiently perform artificial intelligence applications with ONNs? We present a digital ONN implementation to show a proof-of-concept of the ONN approach of “computing-in-phase” for pattern recognition applications. To the best of our knowledge, this is the first attempt to implement an FPGA-based fully-digital ONN. We report ONN accuracy, training, inference, memory capacity, operating frequency, hardware resources based on simulations and implementations of 5 × 3 and 10 × 6 ONNs. We present the digital ONN implementation on FPGA for pattern recognition applications such as performing digits recognition from a camera stream. We discuss practical challenges and future directions in implementing digital ONN.

Adversarial parameter defense by multi-step risk minimization

Previous studies demonstrate DNNs' vulnerability to adversarial examples and adversarial training can establish a defense to adversarial examples. In addition, recent studies show that deep neural networks also exhibit vulnerability to parameter corruptions. The vulnerability of model parameters is of crucial value to the study of model robustness and generalization. In this work, we introduce the concept of parameter corruption and propose to leverage the loss change indicators for measuring the flatness of the loss basin and the parameter robustness of neural network parameters. On such basis, we analyze parameter corruptions and propose the multi-step adversarial corruption algorithm. To enhance neural networks, we propose the adversarial parameter defense algorithm that minimizes the average risk of multiple adversarial parameter corruptions. Experimental results show that the proposed algorithm can improve both the parameter robustness and accuracy of neural networks.

Deep neural networks using a single neuron: folded-in-time architecture using feedback-modulated delay loops

Deep neural networks are among the most widely applied machine learning tools showing outstanding performance in a broad range of tasks. We present a method for folding a deep neural network of arbitrary size into a single neuron with multiple time-delayed feedback loops. This single-neuron deep neural network comprises only a single nonlinearity and appropriately adjusted modulations of the feedback signals. The network states emerge in time as a temporal unfolding of the neuron's dynamics. By adjusting the feedback-modulation within the loops, we adapt the network's connection weights. These connection weights are determined via a back-propagation algorithm, where both the delay-induced and local network connections must be taken into account. Our approach can fully represent standard Deep Neural Networks (DNN), encompasses sparse DNNs, and extends the DNN concept toward dynamical systems implementations. The new method, which we call Folded-in-time DNN (Fit-DNN), exhibits promising performance in a set of benchmark tasks.

An Efficient Hardware Design for a Low-Latency Traffic Flow Prediction System Using an Online Neural Network

Neural networks are computing systems inspired by the biological neural networks in human brains. They are trained in a batch learning mode; hence, the whole training data should be ready before the training task. However, this is not applicable for many real-time applications where data arrive sequentially such as online topic-detection in social communities, traffic flow prediction, etc. In this paper, an efficient hardware implementation of a low-latency online neural network system is proposed for a traffic flow prediction application. The proposed model is implemented with different Machine Learning (ML) algorithms to predict the traffic flow with high accuracy where the Hedge Backpropagation (HBP) model achieves the least mean absolute error (MAE) of 0.001. The proposed system is implemented using floating point and fixed point arithmetics on Field Programmable Gate Array (FPGA) part of the ZedBoard. The implementation is provided using BRAM architecture and distributed memory in FPGA in order to achieve the best trade-off between latency, the consumption of area, and power. Using the fixed point approach, the prediction times using the distributed memory and BRAM architectures are 150 ns and 420 ns, respectively. The area delay product (ADP) of the proposed system is reduced by 17 × compared with the hardware implementation of the latest proposed system in the literature. The execution time of the proposed hardware system is improved by 200 × compared with the software implemented on a dual core Intel i7-7500U CPU at 2.9 GHz. Consequently, the proposed hardware model is faster than the software model and more suitable for time-critical online machine learning models.

Reservoir computing with biocompatible organic electrochemical networks for brain-inspired biosignal classification

Early detection of malign patterns in patients' biological signals can save millions of lives. Despite the steady improvement of artificial intelligence-based techniques, the practical clinical application of these methods is mostly constrained to an offline evaluation of the patients' data. Previous studies have identified organic electrochemical devices as ideal candidates for biosignal monitoring. However, their use for pattern recognition in real time was never demonstrated. Here, we produce and characterize brain-inspired networks composed of organic electrochemical transistors and use them for time-series predictions and classification tasks using the reservoir computing approach. To show their potential use for biofluid monitoring and biosignal analysis, we classify four classes of arrhythmic heartbeats with an accuracy of 88%. The results of this study introduce a previously unexplored paradigm for biocompatible computational platforms and may enable development of ultralow-power consumption hardware-based artificial neural networks capable of interacting with body fluids and biological tissues.

Prediction of Permeate Flux in Ultrafiltration Processes: A Review of Modeling Approaches

In any membrane filtration, the prediction of permeate flux is critical to calculate the membrane surface required, which is an essential parameter for scaling-up, equipment sizing, and cost determination. For this reason, several models based on phenomenological or theoretical derivation (such as gel-polarization, osmotic pressure, resistance-in-series, and fouling models) and non-phenomenological models have been developed and widely used to describe the limiting phenomena as well as to predict the permeate flux. In general, the development of models or their modifications is done for a particular synthetic model solution and membrane system that shows a good capacity of prediction. However, in more complex matrices, such as fruit juices, those models might not have the same performance. In this context, the present work shows a review of different phenomenological and non-phenomenological models for permeate flux prediction in UF, and a comparison, between selected models, of the permeate flux predictive capacity. Selected models were tested with data from our previous work reported for three fruit juices (bergamot, kiwi, and pomegranate) processed in a cross-flow system for 10 h. The validation of each selected model's capacity of prediction was performed through a robust statistical examination, including a residual analysis. The results obtained, within the statistically validated models, showed that phenomenological models present a high variability of prediction (values of R-square in the range of 75.91-99.78%), Mean Absolute Percentage Error (MAPE) in the range of 3.14-51.69, and Root Mean Square Error (RMSE) in the range of 0.22-2.01 among the investigated juices. The non-phenomenological models showed a great capacity to predict permeate flux with R-squares higher than 97% and lower MAPE (0.25-2.03) and RMSE (3.74-28.91). Even though the estimated parameters have no physical meaning and do not shed light into the fundamental mechanistic principles that govern these processes, these results suggest that non-phenomenological models are a useful tool from a practical point of view to predict the permeate flux, under defined operating conditions, in membrane separation processes. However, the phenomenological models are still a proper tool for scaling-up and for an understanding the UF process.

Neuromorphic Binarized Polariton Networks

The rapid development of artificial neural networks and applied artificial intelligence has led to many applications. However, current software implementation of neural networks is severely limited in terms of performance and energy efficiency. It is believed that further progress requires the development of neuromorphic systems, in which hardware directly mimics the neuronal network structure of a human brain. Here, we propose theoretically and realize experimentally an optical network of nodes performing binary operations. The nonlinearity required for efficient computation is provided by semiconductor microcavities in the strong quantum light-matter coupling regime, which exhibit exciton-polariton interactions. We demonstrate the system performance against a pattern recognition task, obtaining accuracy on a par with state-of-the-art hardware implementations. Our work opens the way to ultrafast and energy-efficient neuromorphic systems taking advantage of ultrastrong optical nonlinearity of polaritons.

Advances in Halide Perovskite Memristor from Lead-Based to Lead-Free Materials

Memristors have attracted considerable attention as one of the four basic circuit elements besides resistors, capacitors, and inductors. Especially, the nonvolatile memory devices have become a promising candidate for the new-generation information storage, due to their excellent write, read, and erase rates, in addition to the low-energy consumption, multistate storage, and high scalability. Among them, halide perovskite (HP) memristors have great potential to achieve low-cost practical information storage and computing. However, the usual lead-based HP memristors face serious problems of high toxicity and low stability. To alleviate the above issues, great effort has been devoted to develop lead-free HP memristors. Here, we have summarized and discussed the advances in HP memristors from lead-based to lead-free materials including memristive properties, stability, neural network applications, and memristive mechanism. Finally, the challenges and prospects of lead-free HP memristors have been discussed.

Effects of memristive synapse radiation interactions on learning in spiking neural networks

This study uses advanced modeling and simulation to explore the effects of external events such as radiation interactions on the synaptic devices in an electronic spiking neural network. Specifically, the networks are trained using the spike-timing-dependent plasticity (STDP) learning rule to recognize spatio-temporal patterns (STPs) representing 25 and 100-pixel characters. Memristive synapses based on a TiO2 non-linear drift model designed in Verilog-A are utilized, with STDP learning behavior achieved through bi-phasic pre- and post-synaptic action potentials. The models are modified to include experimentally observed state-altering and ionizing radiation effects on the device. It is found that radiation interactions tend to make the connection between afferents stronger by increasing the conductance of synapses overall, subsequently distorting the STDP learning curve. In the absence of consistent STPs, these effects accumulate over time and make the synaptic weight evolutions unstable. With STPs at lower flux intensities, the network can recover and relearn with constant training. However, higher flux can overwhelm the leaky integrate-and-fire post-synaptic neuron circuits and reduce stability of the network.

Hardware implementation of radial-basis neural networks with Gaussian activation functions on FPGA

This article introduces a method for realizing the Gaussian activation function of radial-basis (RBF) neural networks with their hardware implementation on field-programmable gaits area (FPGAs). The results of modeling of the Gaussian function on FPGA chips of different families have been presented. RBF neural networks of various topologies have been synthesized and investigated. The hardware component implemented by this algorithm is an RBF neural network with four neurons of the latent layer and one neuron with a sigmoid activation function on an FPGA using 16-bit numbers with a fixed point, which took 1193 logic matrix gate (LUTs—LookUpTable). Each hidden layer neuron of the RBF network is designed on an FPGA as a separate computing unit. The speed as a total delay of the combination scheme of the block RBF network was 101.579 ns. The implementation of the Gaussian activation functions of the hidden layer of the RBF network occupies 106 LUTs, and the speed of the Gaussian activation functions is 29.33 ns. The absolute error is ± 0.005. The Spartan 3 family of chips for modeling has been used to get these results. Modeling on chips of other series has been also introduced in the article. RBF neural networks of various topologies have been synthesized and investigated. Hardware implementation of RBF neural networks with such speed allows them to be used in real-time control systems for high-speed objects.

[Artificial intelligence in cardiology : Relevance, current applications, and future developments]

Big data and applications of artificial intelligence (AI), such as machine learning or deep learning, will enrich healthcare in the future and become increasingly important. Among other things, they have the potential to avoid unnecessary examinations as well as diagnostic and therapeutic errors. They could enable improved, early and accelerated decision-making. In the article, the authors provide an overview of current AI-based applications in cardiology. The examples describe innovative solutions for risk assessment, diagnosis and therapy support up to patient self-management. Big data and AI serve as a basis for efficient, predictive, preventive and personalised medicine. However, the examples also show that research is needed to further develop the solutions for the benefit of the patient and the medical profession, to demonstrate the effectiveness and benefits in health care and to establish legal and ethical standards.

Real-time Neural Networks Implementation Proposal for Microcontrollers

The adoption of intelligent systems with Artificial Neural Networks (ANNs) embedded in hardware for real-time applications currently faces a growing demand in fields such as the Internet of Things (IoT) and Machine to Machine (M2M). However, the application of ANNs in this type of system poses a significant challenge due to the high computational power required to process its basic operations. This paper aims to show an implementation strategy of a Multilayer Perceptron (MLP)-type neural network, in a microcontroller (a low-cost, low-power platform). A modular matrix-based MLP with the full classification process was implemented as was the backpropagation training in the microcontroller. The testing and validation were performed through Hardware-In-the-Loop (HIL) of the Mean Squared Error (MSE) of the training process, classification results, and the processing time of each implementation module. The results revealed a linear relationship between the values of the hyperparameters and the processing time required for classification, also the processing time concurs with the required time for many applications in the fields mentioned above. These findings show that this implementation strategy and this platform can be applied successfully in real-time applications that require the capabilities of ANNs.

FPGA-Based Implementation of Stochastic Configuration Networks for Regression Prediction

The implementation of neural network regression prediction based on digital circuits is one of the challenging problems in the field of machine learning and cognitive recognition, and it is also an effective way to relieve the pressure of the Internet in the era of intelligence. As a nonlinear network, the stochastic configuration network (SCN) is considered to be an effective method for regression prediction due to its good performance in learning and generalization. Therefore, in this paper, we adapt the SCN to regression analysis, and design and verify the field programmable gate array (FPGA) framework to implement SCN model for the first time. In addition, in order to improve the performance of the SCN model based on the FPGA, the implementation of the nonlinear activation function on the FPGA is optimized, which effectively improves the prediction accuracy while considering the utilization rate of hardware resources. Experimental results based on the simulation data set and the real data set prove that the proposed FPGA framework successfully implements the SCN regression prediction model, and the improved SCN model has higher accuracy and a more stable performance. Compared with the extreme learning machine (ELM), the prediction performance of the proposed SCN implementation model based on the FPGA for the simulation data set and the real data set is improved by 56.37% and 17.35%, respectively.

Polaritonic Neuromorphic Computing Outperforms Linear Classifiers

Machine learning software applications are ubiquitous in many fields of science and society for their outstanding capability to solve computationally vast problems like the recognition of patterns and regularities in big data sets. In spite of these impressive achievements, such processors are still based on the so-called von Neumann architecture, which is a bottleneck for faster and power-efficient neuromorphic computation. Therefore, one of the main goals of research is to conceive physical realizations of artificial neural networks capable of performing fully parallel and ultrafast operations. Here we show that lattices of exciton-polariton condensates accomplish neuromorphic computing with outstanding accuracy thanks to their high optical nonlinearity. We demonstrate that our neural network significantly increases the recognition efficiency compared with the linear classification algorithms on one of the most widely used benchmarks, the MNIST problem, showing a concrete advantage from the integration of optical systems in neural network architectures.

EANN: Energy Adaptive Neural Networks

This paper proposes an Energy Adaptive Feedforward Neural Network (EANN). It uses multiple approximation techniques in the hardware implementation of the neuron unit. The used techniques are precision scaling, approximate multiplier, computation skipping, neuron skipping, activation function approximation and truncated accumulation. The proposed EANN system applies the partial dynamic reconfiguration (PDR) feature supported by the FPGA platform to reconfigure the hardware elements of the neural network based on the energy budget. The PDR technique enables the EANN system to remain functioning when the available energy budget is reduced by factors of 46.2% to 79.8% of the total energy of the unapproximated neural network. Unlike the conventional operation that only uses certain amount of energy and cannot function properly if the energy budget falls below that energy level, the EANN system remains functioning for longer time after energy drop at the expense of less accuracy. The proposed EANN system is highly recommended in limited-energy applications as it adapts the hardware units to the degraded energy at the expense of some accuracy loss.

l2-l∞ state estimation for delayed artificial neural networks under high-rate communication channels with Round-Robin protocol

In this paper, the l₂-l_∞ state estimation problem is addressed for a class of delayed artificial neural networks under high-rate communication channels with Round-Robin (RR) protocol. To estimate the state of the artificial neural networks, numerous sensors are deployed to measure the artificial neural networks. The sensors communicate with the remote state estimator through a shared high-rate communication channel. In the high-rate communication channel, the RR protocol is utilized to schedule the transmission sequence of the numerous sensors. The aim of this paper is to design an estimator such that, under the high-rate communication channel and the RR protocol, the exponential stability of the estimation error dynamics as well as the l₂-l_∞ performance constraint are ensured. First, sufficient conditions are given which guarantee the existence of the desired l₂-l_∞ state estimator. Then, the estimator gains are obtained by solving two sets of matrix inequalities. Finally, numerical examples are provided to verify the effectiveness of the developed l₂-l_∞ state estimation scheme.

A Digital Hardware System for Spiking Network of Tactile Afferents

In the present research, we explore the possibility of utilizing a hardware-based neuromorphic approach to develop a tactile sensory system at the level of first-order afferents, which are slowly adapting type 1 (SA-I) and fast adapting type 1 (FA-I) afferents. Four spiking models are used to mimic neural signals of both SA-I and FA-I primary afferents. Next, a digital circuit is designed for each spiking model for both afferents to be implemented on the field-programmable gate array (FPGA). The four different digital circuits are then compared from source utilization point of view to find the minimum cost circuit for creating a population of digital afferents. In this way, the firing responses of both SA-I and FA-I afferents are physically measured in hardware. Finally, a population of 243 afferents consisting of 90 SA-I and 153 FA-I digital neuromorphic circuits are implemented on the FPGA. The FPGA also receives nine inputs from the force sensors through an interfacing board. Therefore, the data of multiple inputs are processed by the spiking network of tactile afferents, simultaneously. Benefiting from parallel processing capabilities of FPGA, the proposed architecture offers a low-cost neuromorphic structure for tactile information processing. Applying machine learning algorithms on the artificial spiking patterns collected from FPGA, we successfully classified three different objects based on the firing rate paradigm. Consequently, the proposed neuromorphic system provides the opportunity for development of new tactile processing component for robotic and prosthetic applications.

Heuristic recurrent algorithms for photonic Ising machines

The inability of conventional electronic architectures to efficiently solve large combinatorial problems motivates the development of novel computational hardware. There has been much effort toward developing application-specific hardware across many different fields of engineering, such as integrated circuits, memristors, and photonics. However, unleashing the potential of such architectures requires the development of algorithms which optimally exploit their fundamental properties. Here, we present the Photonic Recurrent Ising Sampler (PRIS), a heuristic method tailored for parallel architectures allowing fast and efficient sampling from distributions of arbitrary Ising problems. Since the PRIS relies on vector-to-fixed matrix multiplications, we suggest the implementation of the PRIS in photonic parallel networks, which realize these operations at an unprecedented speed. The PRIS provides sample solutions to the ground state of Ising models, by converging in probability to their associated Gibbs distribution. The PRIS also relies on intrinsic dynamic noise and eigenvalue dropout to find ground states more efficiently. Our work suggests speedups in heuristic methods via photonic implementations of the PRIS.

Artificial Synapse Emulated through Fully Aqueous Solution-Processed Low-Voltage In2O3 Thin-Film Transistor with Gd2O3 Solid Electrolyte

Brain-like neuromorphic computing system provides an alternative approach for the future computer for its characteristics of high-efficiency, power-efficient, self-learning, and parallel computing. Therefore, the imitation of synapse behavior based on microelectronics is particularly important. Recently, the synaptic transistors have received widespread attention. Among them, solid oxide-based synaptic transistors are more compatible with the large-scale fabrication than the liquid and organic-based transistors. So the development of oxide synaptic transistor is required. Here, a novel aqueous solution-processed Gd₂O₃ is suggested to be the solid electrolyte for synaptic transistors. The microstructure and the dielectric properties of Gd₂O₃ film are investigated, which show the potential for the simulation of synaptic transmission. Then, the fully aqueous solution-processed In₂O₃/Gd₂O₃ thin-film transistor (TFT) is fabricated. The device exhibits an acceptable electrical performance with a small threshold voltage of 1.24 V, and a small subthreshold swing of 0.12 V/decade. The artificial synapse behavior is stimulated and the short-term plasticity of In₂O₃/Gd₂O₃ TFT is studied. The dependence of its excitatory postsynaptic current on presynaptic pulse magnitude, width, and frequency is verified. Besides, the synapse behavior of devices under continuous illumination stresses is investigated. The lights with different photon energy have different effects on the synaptic transmission, which is related to the ionization of oxygen vacancies. Our results demonstrate that fully aqueous solution-processed In₂O₃ TFT with Gd₂O₃ solid electrolyte is a candidate for the synaptic transistor.