Artificial Neural Network-Based Mechanism to Detect Security Threats in Wireless Sensor Networks

Wireless sensor networks (WSNs) are essential in many areas, from healthcare to environmental monitoring. However, WSNs are vulnerable to routing attacks that might jeopardize network performance and data integrity due to their inherent vulnerabilities. This work suggests a unique method for enhancing WSN security through the detection of routing threats using feed-forward artificial neural networks (ANNs). The proposed solution makes use of ANNs' learning capabilities to model the network's dynamic behavior and recognize routing attacks like black-hole, gray-hole, and wormhole attacks. CICIDS2017 is a heterogeneous dataset that was used to train and test the proposed system in order to guarantee its robustness and adaptability. The system's ability to recognize both known and novel attack patterns enhances its efficacy in real-world deployment. Experimental assessments using an NS2 simulator show how well the proposed method works to improve routing protocol security. The proposed system's performance was assessed using a confusion matrix. The simulation and analysis demonstrated how much better the proposed system performs compared to the existing methods for routing attack detection. With an average detection rate of 99.21% and a high accuracy of 99.49%, the proposed system minimizes the rate of false positives. The study advances secure communication in WSNs and provides a reliable means of protecting sensitive data in resource-constrained settings.

An Agent-Based Model to Reproduce the Boolean Logic Behaviour of Neuronal Self-Organised Communities through Pulse Delay Modulation and Generation of Logic Gates

The human brain is arguably the most complex "machine" to ever exist. Its detailed functioning is yet to be fully understood, let alone modelled. Neurological processes have logical signal-processing and biophysical aspects, and both affect the brain's structure, functioning and adaptation. Mathematical approaches based on both information and graph theory have been extensively used in an attempt to approximate its biological functioning, along with Artificial Intelligence frameworks inspired by its logical functioning. In this article, an approach to model some aspects of the brain learning and signal processing is presented, mimicking the metastability and backpropagation found in the real brain while also accounting for neuroplasticity. Several simulations are carried out with this model to demonstrate how dynamic neuroplasticity, neural inhibition and neuron migration can reshape the brain's logical connectivity to synchronise signal processing and obtain certain target latencies. This work showcases the importance of dynamic logical and biophysical remodelling in brain plasticity. Combining mathematical (agents, graph theory, topology and backpropagation) and biomedical ingredients (metastability, neuroplasticity and migration), these preliminary results prove complex brain phenomena can be reproduced-under pertinent simplifications-via affordable computations, which can be construed as a starting point for more ambitiously accurate simulations.

Deep convolutional neural network and IoT technology for healthcare

Background

Deep Learning is an AI technology that trains computers to analyze data in an approach similar to the human brain. Deep learning algorithms can find complex patterns in images, text, audio, and other data types to provide accurate predictions and conclusions. Neuronal networks are another name for Deep Learning. These layers are the input, the hidden, and the output of a deep learning model. First, data is taken in by the input layer, and then it is processed by the output layer. Deep Learning has many advantages over traditional machine learning algorithms like a KA-nearest neighbor, support vector algorithms, and regression approaches. Deep learning models can read more complex data than traditional machine learning methods.

Objectives

This research aims to find the ideal number of best-hidden layers for the neural network and different activation function variations. The article also thoroughly analyzes how various frameworks can be used to create a comparison or fast neural networks. The final goal of the article is to investigate all such innovative techniques that allow us to speed up the training of neural networks without losing accuracy.

Methods

A sample data Set from 2001 was collected by www.Kaggle.com. We can reduce the total number of layers in the deep learning model. This will enable us to use our time. To perform the ReLU activation, we will make use of two layers that are completely connected. If the value being supplied is larger than zero, the ReLU activation will return 0, and else it will output the value being input directly.

Results

We use multiple parameters to determine the most effective method to test how well our method works. In the next paragraph, we'll discuss how the calculation changes secret-shared Values. By adopting 19 train set features, we train our reliable model to predict healthcare cost's (numerical) target feature. We found that 0.89503 was the best choice because it gave us a good fit (R2) and let us set enough coefficients to 0. To develop our stable model with this Set of parameters, we require 26 iterations. We use an R2 of 0.89503, an MSE of 0.01094, an RMSE of 0.10458, a mean residual deviance of 0.01094, a mean absolute error of 0.07452, and a root mean squared log error of 0.07207. After training the model on the train set, we applied the same parameters to the test set and obtained an R2 of 0.90707, MSE of 0.01045, RMSE of 0.10224, mean residual deviation of 0.01045, MAE of 0.06954, and RMSE of 0.07051, validating our solution approach. The objective value of our secured model is higher than that of the scikit-learn model, although the former performs better on goodness-of-fit criteria. As a result, our protected model performs quite well, marginally outperforming the (very optimized) scikit-learn model. Using a backpropagation algorithm and stochastic gradient descent, deep Learning develops artificial neural systems with several interconnected layers. There may be hidden layers of neurons in the network that have the tanh, rectification, and max-out hyperparameters. Modern features like momentum training, dropout, active learning rate, rate annealed, and L1 or L2 regularization provide exceptional prediction performance. The worldwide model's parameters are multi-threadedly (asynchronously) trained on the data from that node, and the model-based data is then gradually augmented by model averaging over the entire network. The method is executed on a single-node, direct H2O cluster initiated by the operator. The operation is parallel despite there just being a single node involved. The number of threads may be adjusted in the settings menu under Preferences and General. The optimal number of threads for the system is used automatically. Successful predictions in the healthcare data sets are made using the H2O Deep Learning operator. There will be a classification done since its label is binomial. The Splitting Validation operator creates test and training datasets to evaluate the model. By default, the settings of the Deep Learning activator are used. To put it another way, we'll construct two hidden layers, each containing 50 neurons. The Accuracy measure is computed by linking the annotated Sample Set with a Performer (Binominal Classification) operator. Table 3 displays the Deep Learning Model, the labeled data, and the Performance Vector that resulted from the technique.

Conclusions

Deep learning algorithms can be used to design systems that report data on patients and deliver warnings to medical applications or electronic health information if there are changes in the patient's health. These systems could be created using deep Learning. This helps verify that patients get the proper effective care at the proper time for each specific patient. A healthcare decision support system was presented using the Internet of Things and deep learning methods. In the proposed system, we examined the capability of integrating deep learning technology into automatic diagnosis and IoT capabilities for faster message exchange over the Internet. We have selected the suitable Neural Network structure (number of best-hidden layers and activation function classes) to construct the e-health system. In addition, the e-health system relied on data from doctors to understand the Neural Network. In the validation method, the total evaluation of the proposed healthcare system for diagnostics provides dependability under various patient conditions. Based on evaluation and simulation findings, a dual hidden layer of feed-forward NN and its neurons store the tanh function more effectively than other NN. To overcome challenges, this study will integrate artificial intelligence with IoT. This study aims to determine the NN's optimal layer counts and activation function variations.

Biologically plausible local synaptic learning rules robustly implement deep supervised learning

In deep neural networks, representational learning in the middle layer is essential for achieving efficient learning. However, the currently prevailing backpropagation learning rules (BP) are not necessarily biologically plausible and cannot be implemented in the brain in their current form. Therefore, to elucidate the learning rules used by the brain, it is critical to establish biologically plausible learning rules for practical memory tasks. For example, learning rules that result in a learning performance worse than that of animals observed in experimental studies may not be computations used in real brains and should be ruled out. Using numerical simulations, we developed biologically plausible learning rules to solve a task that replicates a laboratory experiment where mice learned to predict the correct reward amount. Although the extreme learning machine (ELM) and weight perturbation (WP) learning rules performed worse than the mice, the feedback alignment (FA) rule achieved a performance equal to that of BP. To obtain a more biologically plausible model, we developed a variant of FA, FA_Ex-100%, which implements direct dopamine inputs that provide error signals locally in the layer of focus, as found in the mouse entorhinal cortex. The performance of FA_Ex-100% was comparable to that of conventional BP. Finally, we tested whether FA_Ex-100% was robust against rule perturbations and biologically inevitable noise. FA_Ex-100% worked even when subjected to perturbations, presumably because it could calibrate the correct prediction error (e.g., dopaminergic signals) in the next step as a teaching signal if the perturbation created a deviation. These results suggest that simplified and biologically plausible learning rules, such as FA_Ex-100%, can robustly facilitate deep supervised learning when the error signal, possibly conveyed by dopaminergic neurons, is accurate.

Design and Simulation of a Multilayer Chemical Neural Network That Learns via Backpropagation

The design and implementation of adaptive chemical reaction networks, capable of adjusting their behavior over time in response to experience, is a key goal for the fields of molecular computing and DNA nanotechnology. Mainstream machine learning research offers powerful tools for implementing learning behavior that could one day be realized in a wet chemistry system. Here we develop an abstract chemical reaction network model that implements the backpropagation learning algorithm for a feedforward neural network whose nodes employ the nonlinear "leaky rectified linear unit" transfer function. Our network directly implements the mathematics behind this well-studied learning algorithm, and we demonstrate its capabilities by training the system to learn a linearly inseparable decision surface, specifically, the XOR logic function. We show that this simulation quantitatively follows the definition of the underlying algorithm. To implement this system, we also report ProBioSim, a simulator that enables arbitrary training protocols for simulated chemical reaction networks to be straightforwardly defined using constructs from the host programming language. This work thus provides new insight into the capabilities of learning chemical reaction networks and also develops new computational tools to simulate their behavior, which could be applied in the design and implementations of adaptive artificial life.

Learning cortical hierarchies with temporal Hebbian updates

A key driver of mammalian intelligence is the ability to represent incoming sensory information across multiple abstraction levels. For example, in the visual ventral stream, incoming signals are first represented as low-level edge filters and then transformed into high-level object representations. Similar hierarchical structures routinely emerge in artificial neural networks (ANNs) trained for object recognition tasks, suggesting that similar structures may underlie biological neural networks. However, the classical ANN training algorithm, backpropagation, is considered biologically implausible, and thus alternative biologically plausible training methods have been developed such as Equilibrium Propagation, Deep Feedback Control, Supervised Predictive Coding, and Dendritic Error Backpropagation. Several of those models propose that local errors are calculated for each neuron by comparing apical and somatic activities. Notwithstanding, from a neuroscience perspective, it is not clear how a neuron could compare compartmental signals. Here, we propose a solution to this problem in that we let the apical feedback signal change the postsynaptic firing rate and combine this with a differential Hebbian update, a rate-based version of classical spiking time-dependent plasticity (STDP). We prove that weight updates of this form minimize two alternative loss functions that we prove to be equivalent to the error-based losses used in machine learning: the inference latency and the amount of top-down feedback necessary. Moreover, we show that the use of differential Hebbian updates works similarly well in other feedback-based deep learning frameworks such as Predictive Coding or Equilibrium Propagation. Finally, our work removes a key requirement of biologically plausible models for deep learning and proposes a learning mechanism that would explain how temporal Hebbian learning rules can implement supervised hierarchical learning.

SUMOylation of NaV1.2 channels regulates the velocity of backpropagating action potentials in cortical pyramidal neurons

Voltage-gated sodium channels located in axon initial segments (AIS) trigger action potentials (AP) and play pivotal roles in the excitability of cortical pyramidal neurons. The differential electrophysiological properties and distributions of NaV1.2 and NaV1.6 channels lead to distinct contributions to AP initiation and propagation. While NaV1.6 at the distal AIS promotes AP initiation and forward propagation, NaV1.2 at the proximal AIS promotes the backpropagation of APs to the soma. Here, we show the small ubiquitin-like modifier (SUMO) pathway modulates Na+ channels at the AIS to increase neuronal gain and the speed of backpropagation. Since SUMO does not affect NaV1.6, these effects were attributed to SUMOylation of NaV1.2. Moreover, SUMO effects were absent in a mouse engineered to express NaV1.2-Lys38Gln channels that lack the site for SUMO linkage. Thus, SUMOylation of NaV1.2 exclusively controls INaP generation and AP backpropagation, thereby playing a prominent role in synaptic integration and plasticity.

Constrain Bias Addition to Train Low-Latency Spiking Neural Networks

In recent years, a third-generation neural network, namely, spiking neural network, has received plethora of attention in the broad areas of Machine learning and Artificial Intelligence. In this paper, a novel differential-based encoding method is proposed and new spike-based learning rules for backpropagation is derived by constraining the addition of bias voltage in spiking neurons. The proposed differential encoding method can effectively exploit the correlation between the data and improve the performance of the proposed model, and the new learning rule can take complete advantage of the modulation properties of bias on the spike firing threshold. We experiment with the proposed model on the environmental sound dataset RWCP and the image dataset MNIST and Fashion-MNIST, respectively, and assign various conditions to test the learning ability and robustness of the proposed model. The experimental results demonstrate that the proposed model achieves near-optimal results with a smaller time step by maintaining the highest accuracy and robustness with less training data. Among them, in MNIST dataset, compared with the original spiking neural network with the same network structure, we achieved a 0.39% accuracy improvement.

Brain-inspired Predictive Coding Improves the Performance of Machine Challenging Tasks

Backpropagation has been regarded as the most favorable algorithm for training artificial neural networks. However, it has been criticized for its biological implausibility because its learning mechanism contradicts the human brain. Although backpropagation has achieved super-human performance in various machine learning applications, it often shows limited performance in specific tasks. We collectively referred to such tasks as machine-challenging tasks (MCTs) and aimed to investigate methods to enhance machine learning for MCTs. Specifically, we start with a natural question: Can a learning mechanism that mimics the human brain lead to the improvement of MCT performances? We hypothesized that a learning mechanism replicating the human brain is effective for tasks where machine intelligence is difficult. Multiple experiments corresponding to specific types of MCTs where machine intelligence has room to improve performance were performed using predictive coding, a more biologically plausible learning algorithm than backpropagation. This study regarded incremental learning, long-tailed, and few-shot recognition as representative MCTs. With extensive experiments, we examined the effectiveness of predictive coding that robustly outperformed backpropagation-trained networks for the MCTs. We demonstrated that predictive coding-based incremental learning alleviates the effect of catastrophic forgetting. Next, predictive coding-based learning mitigates the classification bias in long-tailed recognition. Finally, we verified that the network trained with predictive coding could correctly predict corresponding targets with few samples. We analyzed the experimental result by drawing analogies between the properties of predictive coding networks and those of the human brain and discussing the potential of predictive coding networks in general machine learning.

Combining backpropagation with Equilibrium Propagation to improve an Actor-Critic reinforcement learning framework

Backpropagation (BP) has been used to train neural networks for many years, allowing them to solve a wide variety of tasks like image classification, speech recognition, and reinforcement learning tasks. But the biological plausibility of BP as a mechanism of neural learning has been questioned. Equilibrium Propagation (EP) has been proposed as a more biologically plausible alternative and achieves comparable accuracy on the CIFAR-10 image classification task. This study proposes the first EP-based reinforcement learning architecture: an Actor-Critic architecture with the actor network trained by EP. We show that this model can solve the basic control tasks often used as benchmarks for BP-based models. Interestingly, our trained model demonstrates more consistent high-reward behavior than a comparable model trained exclusively by BP.

Artificial neural networks for prediction of COVID-19 in India by using backpropagation

The COVID-19 pandemic has affected thousands of people around the world. In this study, we used artificial neural network (ANN) models to forecast the COVID-19 outbreak for policymakers based on 1st January to 31st October 2021 of positive cases in India. In the confirmed cases of COVID-19 in India, it's critical to use an estimating model with a high degree of accuracy to get a clear understanding of the situation. Two explicit mathematical prediction models were used in this work to anticipate the COVID-19 epidemic in India. A Boltzmann Function-based model and Beesham's prediction model are among these methods and also estimated using the advanced ANN-BP models. The COVID-19 information was partitioned into two sections: training and testing. The former was utilized for training the ANN-BP models, and the latter was used to test them. The information examination uncovers critical day-by-day affirmed case changes, yet additionally unmistakable scopes of absolute affirmed cases revealed across the time span considered. The ANN-BP model that takes into consideration the preceding 14-days outperforms the others based on the archived results. In forecasting the COVID-19 pandemic, this comparison provides the maximum incubation period, in India. Mean square error, and mean absolute percent error have been treated as the forecast model performs more accurately and gets good results. In view of the findings, the ANN-BP model that considers the past 14-days for the forecast is proposed to predict everyday affirmed cases, especially in India that have encountered the main pinnacle of the COVID-19 outbreak. This work has not just demonstrated the relevance of the ANN-BP techniques for the expectation of the COVID-19 outbreak yet additionally showed that considering the incubation time of COVID-19 in forecast models might produce more accurate assessments.

Backpropagation with biologically plausible spatiotemporal adjustment for training deep spiking neural networks

The spiking neural network (SNN) mimics the information-processing operation in the human brain. Directly applying backpropagation to the training of the SNN still has a performance gap compared with traditional deep neural networks. To address the problem, we propose a biologically plausible spatial adjustment that rethinks the relationship between membrane potential and spikes and realizes a reasonable adjustment of gradients to different time steps. It precisely controls the backpropagation of the error along the spatial dimension. Secondly, we propose a biologically plausible temporal adjustment to make the error propagate across the spikes in the temporal dimension, which overcomes the problem of the temporal dependency within a single spike period of traditional spiking neurons. We have verified our algorithm on several datasets, and the experimental results have shown that our algorithm greatly reduces network latency and energy consumption while also improving network performance.

Backpropagation With Sparsity Regularization for Spiking Neural Network Learning

The spiking neural network (SNN) is a possible pathway for low-power and energy-efficient processing and computing exploiting spiking-driven and sparsity features of biological systems. This article proposes a sparsity-driven SNN learning algorithm, namely backpropagation with sparsity regularization (BPSR), aiming to achieve improved spiking and synaptic sparsity. Backpropagation incorporating spiking regularization is utilized to minimize the spiking firing rate with guaranteed accuracy. Backpropagation realizes the temporal information capture and extends to the spiking recurrent layer to support brain-like structure learning. The rewiring mechanism with synaptic regularization is suggested to further mitigate the redundancy of the network structure. Rewiring based on weight and gradient regulates the pruning and growth of synapses. Experimental results demonstrate that the network learned by BPSR has synaptic sparsity and is highly similar to the biological system. It not only balances the accuracy and firing rate, but also facilitates SNN learning by suppressing the information redundancy. We evaluate the proposed BPSR on the visual dataset MNIST, N-MNIST, and CIFAR10, and further test it on the sensor dataset MIT-BIH and gas sensor. Results bespeak that our algorithm achieves comparable or superior accuracy compared to related works, with sparse spikes and synapses.

SpikePropamine: Differentiable Plasticity in Spiking Neural Networks

The adaptive changes in synaptic efficacy that occur between spiking neurons have been demonstrated to play a critical role in learning for biological neural networks. Despite this source of inspiration, many learning focused applications using Spiking Neural Networks (SNNs) retain static synaptic connections, preventing additional learning after the initial training period. Here, we introduce a framework for simultaneously learning the underlying fixed-weights and the rules governing the dynamics of synaptic plasticity and neuromodulated synaptic plasticity in SNNs through gradient descent. We further demonstrate the capabilities of this framework on a series of challenging benchmarks, learning the parameters of several plasticity rules including BCM, Oja's, and their respective set of neuromodulatory variants. The experimental results display that SNNs augmented with differentiable plasticity are sufficient for solving a set of challenging temporal learning tasks that a traditional SNN fails to solve, even in the presence of significant noise. These networks are also shown to be capable of producing locomotion on a high-dimensional robotic learning task, where near-minimal degradation in performance is observed in the presence of novel conditions not seen during the initial training period.

Spiking Autoencoders With Temporal Coding

Spiking neural networks with temporal coding schemes process information based on the relative timing of neuronal spikes. In supervised learning tasks, temporal coding allows learning through backpropagation with exact derivatives, and achieves accuracies on par with conventional artificial neural networks. Here we introduce spiking autoencoders with temporal coding and pulses, trained using backpropagation to store and reconstruct images with high fidelity from compact representations. We show that spiking autoencoders with a single layer are able to effectively represent and reconstruct images from the neuromorphically-encoded MNIST and FMNIST datasets. We explore the effect of different spike time target latencies, data noise levels and embedding sizes, as well as the classification performance from the embeddings. The spiking autoencoders achieve results similar to or better than conventional non-spiking autoencoders. We find that inhibition is essential in the functioning of the spiking autoencoders, particularly when the input needs to be memorised for a longer time before the expected output spike times. To reconstruct images with a high target latency, the network learns to accumulate negative evidence and to use the pulses as excitatory triggers for producing the output spikes at the required times. Our results highlight the potential of spiking autoencoders as building blocks for more complex biologically-inspired architectures. We also provide open-source code for the model.

Application of artificial neural networks for predicting the physical composition of municipal solid waste: An assessment of the impact of seasonal variation

Sustainable planning of waste management is contingent on reliable data on waste characteristics and their variation across the seasons owing to the consequential environmental impact of such variation. Traditional waste characterization techniques in most developing countries are time-consuming and expensive; hence the need to address the issue from a modelling approach arises. In modelling the complexity within the system, a paradigm shift from the classical models to the intelligent models has been observed. The application of artificial intelligence models in waste management is gaining traction; however its application in predicting the physical composition of waste is still lacking. This study aims at investigating the optimal combinations of network architecture, training algorithm and activation functions that accurately predict the fraction of physical waste streams from meteorological parameters using artificial neural networks. The city of Johannesburg was used as a case study. Maximum temperature, minimum temperature, wind speed and humidity were used as input variables to predict the percentage composition of organic, paper, plastics and textile waste streams. Several sub-models were stimulated with combination of nine training algorithms and four activation functions in each single hidden layer topology with a range of 1-15 neurons. Performance metrics used to evaluate the accuracy of the system are, root mean square error, mean absolute deviation, mean absolute percentage error and correlation coefficient (R). Optimal architectures in the order of input layer-number of neurons in the hidden layer-output layer for predicting organic, paper, plastics and textile waste were 4-10-1, 4-14-1, 4-5-1 and 4-8-1 with R-values of 0.916, 0.862, 0.834 and 0.826, respectively at the testing phase. The result of the study verifies that waste composition prediction can be done in a single hidden-layer satisfactorily.

Locating Ships Using Time Reversal and Matrix Pencil Method by Their Underwater Acoustic Signals

This paper presents a technique, based on the matrix pencil method (MPM), for the compression of underwater acoustic signals produced by boat engines. The compressed signal, represented by its complex resonance expansion, is intended to be sent over a low-bit-rate wireless communication channel. We demonstrate that the method can provide data compression greater than 60%, ensuring a correlation greater than 93% between the reconstructed and the original signal, at a sampling frequency of 2.2 kHz. Once the signal was reconstituted, a localization process was carried out with the time reversal method (TR) using information from four different sensors in a simulation environment. This process sought to achieve the identification of the position of the ship using only passive sensors, considering two different sensor arrangements.

Target Classification in Synthetic Aperture Radar Images Using Quantized Wavelet Scattering Networks

The need to classify targets and features in high-resolution imagery is of interest in applications such as detection of landmines in ground penetrating radar and tumors in medical ultrasound images. Convolutional neural networks (CNNs) trained using extensive datasets are being investigated recently. However, large CNNs and wavelet scattering networks (WSNs), which share similar properties, have extensive memory requirements and are not readily extendable to other datasets and architectures-and especially in the context of adaptive and online learning. In this paper, we quantitatively study several quantization schemes on WSNs designed for target classification using X-band synthetic aperture radar (SAR) data and investigate their robustness to low signal-to-noise ratio (SNR) levels. A detailed study was conducted on the tradeoffs involved between the various quantization schemes and the means of maximizing classification performance for each case. Thus, the WSN-based quantization studies performed in this investigation provide a good benchmark and important guidance for the design of quantized neural networks architectures for target classification.

Supervised Learning With First-to-Spike Decoding in Multilayer Spiking Neural Networks

Experimental studies support the notion of spike-based neuronal information processing in the brain, with neural circuits exhibiting a wide range of temporally-based coding strategies to rapidly and efficiently represent sensory stimuli. Accordingly, it would be desirable to apply spike-based computation to tackling real-world challenges, and in particular transferring such theory to neuromorphic systems for low-power embedded applications. Motivated by this, we propose a new supervised learning method that can train multilayer spiking neural networks to solve classification problems based on a rapid, first-to-spike decoding strategy. The proposed learning rule supports multiple spikes fired by stochastic hidden neurons, and yet is stable by relying on first-spike responses generated by a deterministic output layer. In addition to this, we also explore several distinct, spike-based encoding strategies in order to form compact representations of presented input data. We demonstrate the classification performance of the learning rule as applied to several benchmark datasets, including MNIST. The learning rule is capable of generalizing from the data, and is successful even when used with constrained network architectures containing few input and hidden layer neurons. Furthermore, we highlight a novel encoding strategy, termed "scanline encoding," that can transform image data into compact spatiotemporal patterns for subsequent network processing. Designing constrained, but optimized, network structures and performing input dimensionality reduction has strong implications for neuromorphic applications.

Bio-Inspired Architectures Substantially Reduce the Memory Requirements of Neural Network Models

We propose a neural network model for the jumping escape response behavior observed in the cricket cercal sensory system. This sensory system processes low-intensity air currents in the animal's immediate environment generated by predators, competitors, and mates. Our model is inspired by decades of physiological and anatomical studies. We compare the performance of our model with a model derived through a universal approximation, or a generic deep learning, approach, and demonstrate that, to achieve the same performance, these models required between one and two orders of magnitude more parameters. Furthermore, since the architecture of the bio-inspired model is defined by a set of logical relations between neurons, we find that the model is open to interpretation and can be understood. This work demonstrates the potential of incorporating bio-inspired architectural motifs, which have evolved in animal nervous systems, into memory efficient neural network models.

Learning Without Feedback: Fixed Random Learning Signals Allow for Feedforward Training of Deep Neural Networks

While the backpropagation of error algorithm enables deep neural network training, it implies (i) bidirectional synaptic weight transport and (ii) update locking until the forward and backward passes are completed. Not only do these constraints preclude biological plausibility, but they also hinder the development of low-cost adaptive smart sensors at the edge, as they severely constrain memory accesses and entail buffering overhead. In this work, we show that the one-hot-encoded labels provided in supervised classification problems, denoted as targets, can be viewed as a proxy for the error sign. Therefore, their fixed random projections enable a layerwise feedforward training of the hidden layers, thus solving the weight transport and update locking problems while relaxing the computational and memory requirements. Based on these observations, we propose the direct random target projection (DRTP) algorithm and demonstrate that it provides a tradeoff between accuracy and computational cost that is suitable for adaptive edge computing devices.

Novel Method of Semantic Segmentation Applicable to Augmented Reality

This paper proposes a novel method of semantic segmentation, consisting of modified dilated residual network, atrous pyramid pooling module, and backpropagation, that is applicable to augmented reality (AR). In the proposed method, the modified dilated residual network extracts a feature map from the original images and maintains spatial information. The atrous pyramid pooling module places convolutions in parallel and layers feature maps in a pyramid shape to extract objects occupying small areas in the image; these are converted into one channel using a 1 × 1 convolution. Backpropagation compares the semantic segmentation obtained through convolution from the final feature map with the ground truth provided by a database. Losses can be reduced by applying backpropagation to the modified dilated residual network to change the weighting. The proposed method was compared with other methods on the Cityscapes and PASCAL VOC 2012 databases. The proposed method achieved accuracies of 82.8 and 89.8 mean intersection over union (mIOU) and frame rates of 61 and 64.3 frames per second (fps) for the Cityscapes and PASCAL VOC 2012 databases, respectively. These results prove the applicability of the proposed method for implementing natural AR applications at actual speeds because the frame rate is greater than 60 fps.

Spike-Train Level Direct Feedback Alignment: Sidestepping Backpropagation for On-Chip Training of Spiking Neural Nets

Spiking neural networks (SNNs) present a promising computing model and enable bio-plausible information processing and event-driven based ultra-low power neuromorphic hardware. However, training SNNs to reach the same performances of conventional deep artificial neural networks (ANNs), particularly with error backpropagation (BP) algorithms, poses a significant challenge due to inherent complex dynamics and non-differentiable spike activities of spiking neurons. In this paper, we present the first study on realizing competitive spike-train level backpropagation (BP) like algorithms to enable on-chip training of SNNs. We propose a novel spike-train level direct feedback alignment (ST-DFA) algorithm, which is much more bio-plausible and hardware friendly than BP. Algorithm and hardware co-optimization and efficient online neural signal computation are explored for on-chip implementation of ST-DFA. On the Xilinx ZC706 FPGA board, the proposed hardware-efficient ST-DFA shows excellent performance vs. overhead tradeoffs for real-world speech and image classification applications. SNN neural processors with on-chip ST-DFA training show competitive classification accuracy of 96.27% for the MNIST dataset with 4× input resolution reduction and 84.88% for the challenging 16-speaker TI46 speech corpus, respectively. Compared to the hardware implementation of the state-of-the-art BP algorithm HM2-BP, the design of the proposed ST-DFA reduces functional resources by 76.7% and backward training latency by 31.6% while gracefully trading off classification performance.

Optimal memory-aware backpropagation of deep join networks

Deep learning training memory needs can prevent the user from considering large models and large batch sizes. In this work, we propose to use techniques from memory-aware scheduling and automatic differentiation (AD) to execute a backpropagation graph with a bounded memory requirement at the cost of extra recomputations. The case of a single homogeneous chain, i.e. the case of a network whose stages are all identical and form a chain, is well understood and optimal solutions have been proposed in the AD literature. The networks encountered in practice in the context of deep learning are much more diverse, both in terms of shape and heterogeneity. In this work, we define the class of backpropagation graphs, and extend those on which one can compute in polynomial time a solution that minimizes the total number of recomputations. In particular, we consider join graphs which correspond to models such as siamese or cross-modal networks. This article is part of a discussion meeting issue 'Numerical algorithms for high-performance computational science'.

Learning in Feedforward Neural Networks Accelerated by Transfer Entropy

Current neural networks architectures are many times harder to train because of the increasing size and complexity of the used datasets. Our objective is to design more efficient training algorithms utilizing causal relationships inferred from neural networks. The transfer entropy (TE) was initially introduced as an information transfer measure used to quantify the statistical coherence between events (time series). Later, it was related to causality, even if they are not the same. There are only few papers reporting applications of causality or TE in neural networks. Our contribution is an information-theoretical method for analyzing information transfer between the nodes of feedforward neural networks. The information transfer is measured by the TE of feedback neural connections. Intuitively, TE measures the relevance of a connection in the network and the feedback amplifies this connection. We introduce a backpropagation type training algorithm that uses TE feedback connections to improve its performance.

Estimation of magnetohydrodynamic radiative nanofluid flow over a porous non-linear stretching surface: application in biomedical research

The present study investigates the effects of thermal radiation and chemical reaction on magnetohydrodynamic flow, heat, and mass transfer characteristics of nanofluids such as Cu-water and Ag-water over a non-linear porous stretching surface in the presence of viscous dissipation and heat generation. Using similarity transformation, the governing boundary layer equations of the problem are transformed into non-linear ordinary differential equations and solved numerically by the shooting method along with the Runge-Kutta-Fehlberg fourth-fifth-order integration scheme. The influences of various parameters on velocity, temperature, and concentration profiles of the flow field are analysed and the results are plotted graphically. A backpropagation neural network is applied to predict the skin friction coefficient, Nusselt number, and Sherwood number and these results are presented through graphs. The present numerical results are compared with the existing results and are found to be in good agreement. The results of artificial neural network and the obtained numerical values agree well with an error <5%.

Cellular and Network Mechanisms for Temporal Signal Propagation in a Cortical Network Model

The mechanisms underlying an effective propagation of high intensity information over a background of irregular firing and response latency in cognitive processes remain unclear. Here we propose a SSCCPI circuit to address this issue. We hypothesize that when a high-intensity thalamic input triggers synchronous spike events (SSEs), dense spikes are scattered to many receiving neurons within a cortical column in layer IV, many sparse spike trains are propagated in parallel along minicolumns at a substantially high speed and finally integrated into an output spike train toward or in layer Va. We derive the sufficient conditions for an effective (fast, reliable, and precise) SSCCPI circuit: (i) SSEs are asynchronous (near synchronous); (ii) cortical columns prevent both repeatedly triggering SSEs and incorrectly synaptic connections between adjacent columns; and (iii) the propagator in interneurons is temporally complete fidelity and reliable. We encode the membrane potential responses to stimuli using the non-linear autoregressive integrated process derived by applying Newton's second law to stochastic resilience systems. We introduce a multithreshold decoder to correct encoding errors. Evidence supporting an effective SSCCPI circuit includes that for the condition, (i) time delay enhances SSEs, suggesting that response latency induces SSEs in high-intensity stimuli; irregular firing causes asynchronous SSEs; asynchronous SSEs relate to healthy neurons; and rigorous SSEs relate to brain disorders. For the condition (ii) neurons within a given minicolumn are stereotypically interconnected in the vertical dimension, which prevents repeated triggering SSEs and ensures signal parallel propagation; columnar segregation avoids incorrect synaptic connections between adjacent columns; and signal propagation across layers overwhelmingly prefers columnar direction. For the condition (iii), accumulating experimental evidence supports temporal transfer precision with millisecond fidelity and reliability in interneurons; homeostasis supports a stable fixed-point encoder by regulating changes to synaptic size, synaptic strength, and ion channel function in the membrane; together all-or-none modulation, active backpropagation, additive effects of graded potentials, and response variability functionally support the multithreshold decoder; our simulations demonstrate that the encoder-decoder is temporally complete fidelity and reliable in special intervals contained within the stable fixed-point range. Hence, the SSCCPI circuit provides a possible mechanism of effective signal propagation in cortical networks.

Differences in Sodium Channel Densities in the Apical Dendrites of Pyramidal Cells of the Electrosensory Lateral Line Lobe

Heterogeneity of neural properties within a given neural class is ubiquitous in the nervous system and permits different sub-classes of neurons to specialize for specific purposes. This principle has been thoroughly investigated in the hindbrain of the weakly electric fish A. leptorhynchus in the primary electrosensory area, the Electrosensory Lateral Line lobe (ELL). The pyramidal cells (PCs) that receive inputs from tuberous electroreceptors are organized in three maps in distinct segments of the ELL. The properties of these cells vary greatly across maps due to differences in connectivity, receptor expression, and ion channel composition. These cells are a seminal example of bursting neurons and their bursting dynamic relies on the presence of voltage-gated Na⁺ channels in the extensive apical dendrites of the superficial PCs. Other ion channels can affect burst generation and their expression varies across ELL neurons and segments. For example, SK channels cause hyperpolarizing after-potentials decreasing the likelihood of bursting, yet bursting propensity is similar across segments. We question whether the depolarizing mechanism that generates the bursts presents quantitative differences across segments that could counterbalance other differences having the opposite effect. Although their presence and role are established, the distribution and density of the apical dendrites' Na⁺ channels have not been quantified and compared across ELL maps. Therefore, we test the hypothesis that Na⁺ channel density varies across segment by quantifying their distribution in the apical dendrites of immunolabeled ELL sections. We found the Na⁺ channels to be two-fold denser in the lateral segment (LS) than in the centro-medial segment (CMS), the centro-lateral segment (CLS) being intermediate. Our results imply that this differential expression of voltage-gated Na⁺ channels could counterbalance or interact with other aspects of neuronal physiology that vary across segments (e.g., SK channels). We argue that burst coding of sensory signals, and the way the network regulates bursting, should be influenced by these variations in Na⁺ channel density.

Direct Feedback Alignment With Sparse Connections for Local Learning

Recent advances in deep neural networks (DNNs) owe their success to training algorithms that use backpropagation and gradient-descent. Backpropagation, while highly effective on von Neumann architectures, becomes inefficient when scaling to large networks. Commonly referred to as the weight transport problem, each neuron's dependence on the weights and errors located deeper in the network require exhaustive data movement which presents a key problem in enhancing the performance and energy-efficiency of machine-learning hardware. In this work, we propose a bio-plausible alternative to backpropagation drawing from advances in feedback alignment algorithms in which the error computation at a single synapse reduces to the product of three scalar values. Using a sparse feedback matrix, we show that a neuron needs only a fraction of the information previously used by the feedback alignment algorithms. Consequently, memory and compute can be partitioned and distributed whichever way produces the most efficient forward pass so long as a single error can be delivered to each neuron. We evaluate our algorithm using standard datasets, including ImageNet, to address the concern of scaling to challenging problems. Our results show orders of magnitude improvement in data movement and 2× improvement in multiply-and-accumulate operations over backpropagation. Like previous work, we observe that any variant of feedback alignment suffers significant losses in classification accuracy on deep convolutional neural networks. By transferring trained convolutional layers and training the fully connected layers using direct feedback alignment, we demonstrate that direct feedback alignment can obtain results competitive with backpropagation. Furthermore, we observe that using an extremely sparse feedback matrix, rather than a dense one, results in a small accuracy drop while yielding hardware advantages. All the code and results are available under https://github.com/bcrafton/ssdfa.

Unsupervised learning by competing hidden units

Despite great success of deep learning a question remains to what extent the computational properties of deep neural networks are similar to those of the human brain. The particularly nonbiological aspect of deep learning is the supervised training process with the backpropagation algorithm, which requires massive amounts of labeled data, and a nonlocal learning rule for changing the synapse strengths. This paper describes a learning algorithm that does not suffer from these two problems. It learns the weights of the lower layer of neural networks in a completely unsupervised fashion. The entire algorithm utilizes local learning rules which have conceptual biological plausibility.

It is widely believed that end-to-end training with the backpropagation algorithm is essential for learning good feature detectors in early layers of artificial neural networks, so that these detectors are useful for the task performed by the higher layers of that neural network. At the same time, the traditional form of backpropagation is biologically implausible. In the present paper we propose an unusual learning rule, which has a degree of biological plausibility and which is motivated by Hebb’s idea that change of the synapse strength should be local—i.e., should depend only on the activities of the pre- and postsynaptic neurons. We design a learning algorithm that utilizes global inhibition in the hidden layer and is capable of learning early feature detectors in a completely unsupervised way. These learned lower-layer feature detectors can be used to train higher-layer weights in a usual supervised way so that the performance of the full network is comparable to the performance of standard feedforward networks trained end-to-end with a backpropagation algorithm on simple tasks.

Deep Supervised Learning Using Local Errors

Error backpropagation is a highly effective mechanism for learning high-quality hierarchical features in deep networks. Updating the features or weights in one layer, however, requires waiting for the propagation of error signals from higher layers. Learning using delayed and non-local errors makes it hard to reconcile backpropagation with the learning mechanisms observed in biological neural networks as it requires the neurons to maintain a memory of the input long enough until the higher-layer errors arrive. In this paper, we propose an alternative learning mechanism where errors are generated locally in each layer using fixed, random auxiliary classifiers. Lower layers could thus be trained independently of higher layers and training could either proceed layer by layer, or simultaneously in all layers using local error information. We address biological plausibility concerns such as weight symmetry requirements and show that the proposed learning mechanism based on fixed, broad, and random tuning of each neuron to the classification categories outperforms the biologically-motivated feedback alignment learning technique on the CIFAR10 dataset, approaching the performance of standard backpropagation. Our approach highlights a potential biological mechanism for the supervised, or task-dependent, learning of feature hierarchies. In addition, we show that it is well suited for learning deep networks in custom hardware where it can drastically reduce memory traffic and data communication overheads. Code used to run all learning experiments is available under https://gitlab.com/hesham-mostafa/learning-using-local-erros.git.

Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks

Spiking neural networks (SNNs) are promising in ascertaining brain-like behaviors since spikes are capable of encoding spatio-temporal information. Recent schemes, e.g., pre-training from artificial neural networks (ANNs) or direct training based on backpropagation (BP), make the high-performance supervised training of SNNs possible. However, these methods primarily fasten more attention on its spatial domain information, and the dynamics in temporal domain are attached less significance. Consequently, this might lead to the performance bottleneck, and scores of training techniques shall be additionally required. Another underlying problem is that the spike activity is naturally non-differentiable, raising more difficulties in supervised training of SNNs. In this paper, we propose a spatio-temporal backpropagation (STBP) algorithm for training high-performance SNNs. In order to solve the non-differentiable problem of SNNs, an approximated derivative for spike activity is proposed, being appropriate for gradient descent training. The STBP algorithm combines the layer-by-layer spatial domain (SD) and the timing-dependent temporal domain (TD), and does not require any additional complicated skill. We evaluate this method through adopting both the fully connected and convolutional architecture on the static MNIST dataset, a custom object detection dataset, and the dynamic N-MNIST dataset. Results bespeak that our approach achieves the best accuracy compared with existing state-of-the-art algorithms on spiking networks. This work provides a new perspective to investigate the high-performance SNNs for future brain-like computing paradigm with rich spatio-temporal dynamics.

Effective heart disease prediction system using data mining techniques

The health care industries collect huge amounts of data that contain some hidden information, which is useful for making effective decisions. For providing appropriate results and making effective decisions on data, some advanced data mining techniques are used. In this study, an effective heart disease prediction system (EHDPS) is developed using neural network for predicting the risk level of heart disease. The system uses 15 medical parameters such as age, sex, blood pressure, cholesterol, and obesity for prediction. The EHDPS predicts the likelihood of patients getting heart disease. It enables significant knowledge, eg, relationships between medical factors related to heart disease and patterns, to be established. We have employed the multilayer perceptron neural network with backpropagation as the training algorithm. The obtained results have illustrated that the designed diagnostic system can effectively predict the risk level of heart diseases.

Variable Action Potential Backpropagation during Tonic Firing and Low-Threshold Spike Bursts in Thalamocortical But Not Thalamic Reticular Nucleus Neurons

Backpropagating action potentials (bAPs) are indispensable in dendritic signaling. Conflicting Ca²⁺-imaging data and an absence of dendritic recording data means that the extent of backpropagation in thalamocortical (TC) and thalamic reticular nucleus (TRN) neurons remains unknown. Because TRN neurons signal electrically through dendrodendritic gap junctions and possibly via chemical dendritic GABAergic synapses, as well as classical axonal GABA release, this lack of knowledge is problematic. To address this issue, we made two-photon targeted patch-clamp recordings from rat TC and TRN neuron dendrites to measure bAPs directly. These recordings reveal that “tonic”' and low-threshold-spike (LTS) “burst” APs in both cell types are always recorded first at the soma before backpropagating into the dendrites while undergoing substantial distance-dependent dendritic amplitude attenuation. In TC neurons, bAP attenuation strength varies according to firing mode. During LTS bursts, somatic AP half-width increases progressively with increasing spike number, allowing late-burst spikes to propagate more efficiently into the dendritic tree compared with spikes occurring at burst onset. Tonic spikes have similar somatic half-widths to late burst spikes and undergo similar dendritic attenuation. In contrast, in TRN neurons, AP properties are unchanged between LTS bursts and tonic firing and, as a result, distance-dependent dendritic attenuation remains consistent across different firing modes. Therefore, unlike LTS-associated global electrical and calcium signals, the spatial influence of bAP signaling in TC and TRN neurons is more restricted, with potentially important behavioral-state-dependent consequences for synaptic integration and plasticity in thalamic neurons.

SIGNIFICANCE STATEMENT In most neurons, action potentials (APs) initiate in the axosomatic region and propagate into the dendritic tree to provide a retrograde signal that conveys information about the level of cellular output to the locations that receive most input: the dendrites. In thalamocortical and thalamic reticular nucleus neurons, the site of AP generation and the true extent of backpropagation remain unknown. Using patch-clamp recordings, this study measures dendritic propagation of APs directly in these neurons. In either cell type, high-frequency low-threshold spike burst or lower-frequency tonic APs undergo substantial voltage attenuation as they spread into the dendritic tree. Therefore, backpropagating spikes in these cells can only influence signaling in the proximal part of the dendritic tree.

A Circuit-Based Neural Network with Hybrid Learning of Backpropagation and Random Weight Change Algorithms

A hybrid learning method of a software-based backpropagation learning and a hardware-based RWC learning is proposed for the development of circuit-based neural networks. The backpropagation is known as one of the most efficient learning algorithms. A weak point is that its hardware implementation is extremely difficult. The RWC algorithm, which is very easy to implement with respect to its hardware circuits, takes too many iterations for learning. The proposed learning algorithm is a hybrid one of these two. The main learning is performed with a software version of the BP algorithm, firstly, and then, learned weights are transplanted on a hardware version of a neural circuit. At the time of the weight transplantation, a significant amount of output error would occur due to the characteristic difference between the software and the hardware. In the proposed method, such error is reduced via a complementary learning of the RWC algorithm, which is implemented in a simple hardware. The usefulness of the proposed hybrid learning system is verified via simulations upon several classical learning problems.

Regular Cycles of Forward and Backward Signal Propagation in Prefrontal Cortex and in Consciousness

This paper addresses two fundamental questions: (1) Is it possible to develop mathematical neural network models which can explain and replicate the way in which higher-order capabilities like intelligence, consciousness, optimization, and prediction emerge from the process of learning (Werbos, 1994, 2016a; National Science Foundation, 2008)? and (2) How can we use and test such models in a practical way, to track, to analyze and to model high-frequency (≥ 500 hz) many-channel data from recording the brain, just as econometrics sometimes uses models grounded in the theory of efficient markets to track real-world time-series data (Werbos, 1990)? This paper first reviews some of the prior work addressing question (1), and then reports new work performed in MATLAB analyzing spike-sorted and burst-sorted data on the prefrontal cortex from the Buzsaki lab (Fujisawa et al., 2008, 2015) which is consistent with a regular clock cycle of about 153.4 ms and with regular alternation between a forward pass of network calculations and a backwards pass, as in the general form of the backpropagation algorithm which one of us first developed in the period 1968-1974 (Werbos, 1994, 2006; Anderson and Rosenfeld, 1998). In business and finance, it is well known that adjustments for cycles of the year are essential to accurate prediction of time-series data (Box and Jenkins, 1970); in a similar way, methods for identifying and using regular clock cycles offer large new opportunities in neural time-series analysis. This paper demonstrates a few initial footprints on the large "continent" of this type of neural time-series analysis, and discusses a few of the many further possibilities opened up by this new approach to "decoding" the neural code (Heller et al., 1995).

Training Deep Spiking Neural Networks Using Backpropagation

Deep spiking neural networks (SNNs) hold the potential for improving the latency and energy efficiency of deep neural networks through data-driven event-based computation. However, training such networks is difficult due to the non-differentiable nature of spike events. In this paper, we introduce a novel technique, which treats the membrane potentials of spiking neurons as differentiable signals, where discontinuities at spike times are considered as noise. This enables an error backpropagation mechanism for deep SNNs that follows the same principles as in conventional deep networks, but works directly on spike signals and membrane potentials. Compared with previous methods relying on indirect training and conversion, our technique has the potential to capture the statistics of spikes more precisely. We evaluate the proposed framework on artificially generated events from the original MNIST handwritten digit benchmark, and also on the N-MNIST benchmark recorded with an event-based dynamic vision sensor, in which the proposed method reduces the error rate by a factor of more than three compared to the best previous SNN, and also achieves a higher accuracy than a conventional convolutional neural network (CNN) trained and tested on the same data. We demonstrate in the context of the MNIST task that thanks to their event-driven operation, deep SNNs (both fully connected and convolutional) trained with our method achieve accuracy equivalent with conventional neural networks. In the N-MNIST example, equivalent accuracy is achieved with about five times fewer computational operations.

Associative spike timing-dependent potentiation of the basal dendritic excitatory synapses in the hippocampus in vivo

Spike timing-dependent plasticity in the hippocampus has rarely been studied in vivo. Using extracellular potential and current source density analysis in urethane-anesthetized adult rats, we studied synaptic plasticity at the basal dendritic excitatory synapse in CA1 after excitation-spike (ES) pairing; E was a weak basal dendritic excitation evoked by stratum oriens stimulation, and S was a population spike evoked by stratum radiatum apical dendritic excitation. We hypothesize that positive ES pairing-generating synaptic excitation before a spike-results in long-term potentiation (LTP) while negative ES pairing results in long-term depression (LTD). Pairing (50 pairs at 5 Hz) at ES intervals of -10 to 0 ms resulted in significant input-specific LTP of the basal dendritic excitatory sink, lasting 60-120 min. Pairing at +10- to +20-ms ES intervals, or unpaired 5-Hz stimulation, did not induce significant basal dendritic or apical dendritic LTP or LTD. No basal dendritic LTD was found after stimulation of stratum oriens with 200 pairs of high-intensity pulses at 25-ms interval. Pairing-induced LTP was abolished by pretreatment with an N-methyl-d-aspartate receptor antagonist, 3-(2-carboxypiperazin-4-yl)-propyl-1-phosphonic acid (CPP), which also reduced spike bursting during 5-Hz pairing. Pairing at 0.5 Hz did not induce spike bursts or basal dendritic LTP. In conclusion, ES pairing at 5 Hz resulted in input-specific basal dendritic LTP at ES intervals of -10 ms to 0 ms but no LTD at ES intervals of -20 to +20 ms. Associative LTP likely occurred because of theta-rhythmic coincidence of subthreshold excitation with a backpropagated spike burst, which are conditions that can occur naturally in the hippocampus.

ODTbrain: a Python library for full-view, dense diffraction tomography

BACKGROUND

Analyzing the three-dimensional (3D) refractive index distribution of a single cell makes it possible to describe and characterize its inner structure in a marker-free manner. A dense, full-view tomographic data set is a set of images of a cell acquired for multiple rotational positions, densely distributed from 0 to 360 degrees. The reconstruction is commonly realized by projection tomography, which is based on the inversion of the Radon transform. The reconstruction quality of projection tomography is greatly improved when first order scattering, which becomes relevant when the imaging wavelength is comparable to the characteristic object size, is taken into account. This advanced reconstruction technique is called diffraction tomography. While many implementations of projection tomography are available today, there is no publicly available implementation of diffraction tomography so far.

RESULTS

We present a Python library that implements the backpropagation algorithm for diffraction tomography in 3D. By establishing benchmarks based on finite-difference time-domain (FDTD) simulations, we showcase the superiority of the backpropagation algorithm over the backprojection algorithm. Furthermore, we discuss how measurment parameters influence the reconstructed refractive index distribution and we also give insights into the applicability of diffraction tomography to biological cells.

CONCLUSION

The present software library contains a robust implementation of the backpropagation algorithm. The algorithm is ideally suited for the application to biological cells. Furthermore, the implementation is a drop-in replacement for the classical backprojection algorithm and is made available to the large user community of the Python programming language.

Tonic firing rate controls dendritic Ca2+ signaling and synaptic gain in substantia nigra dopamine neurons

Substantia nigra dopamine neurons fire tonically resulting in action potential backpropagation and dendritic Ca(2+) influx. Using Ca(2+) imaging in acute mouse brain slices, we find a surprisingly steep relationship between tonic firing rate and dendritic Ca(2+). Increasing the tonic rate from 1 to 6 Hz generated Ca(2+) signals up to fivefold greater than predicted by linear summation of single spike-evoked Ca(2+)-transients. This "Ca(2+) supralinearity" was produced largely by depolarization of the interspike voltage leading to activation of subthreshold Ca(2+) channels and was present throughout the proximal and distal dendrites. Two-photon glutamate uncaging experiments show somatic depolarization enhances NMDA receptor-mediated Ca(2+) signals >400 μm distal to the soma, due to unusually tight electrotonic coupling of the soma to distal dendrites. Consequently, we find that fast tonic firing intensifies synaptically driven burst firing output in dopamine neurons. These results show that modulation of background firing rate precisely tunes dendritic Ca(2+) signaling and provides a simple yet powerful mechanism to dynamically regulate the gain of synaptic input.

The Dropout Learning Algorithm

Dropout is a recently introduced algorithm for training neural network by randomly dropping units during training to prevent their co-adaptation. A mathematical analysis of some of the static and dynamic properties of dropout is provided using Bernoulli gating variables, general enough to accommodate dropout on units or connections, and with variable rates. The framework allows a complete analysis of the ensemble averaging properties of dropout in linear networks, which is useful to understand the non-linear case. The ensemble averaging properties of dropout in non-linear logistic networks result from three fundamental equations: (1) the approximation of the expectations of logistic functions by normalized geometric means, for which bounds and estimates are derived; (2) the algebraic equality between normalized geometric means of logistic functions with the logistic of the means, which mathematically characterizes logistic functions; and (3) the linearity of the means with respect to sums, as well as products of independent variables. The results are also extended to other classes of transfer functions, including rectified linear functions. Approximation errors tend to cancel each other and do not accumulate. Dropout can also be connected to stochastic neurons and used to predict firing rates, and to backpropagation by viewing the backward propagation as ensemble averaging in a dropout linear network. Moreover, the convergence properties of dropout can be understood in terms of stochastic gradient descent. Finally, for the regularization properties of dropout, the expectation of the dropout gradient is the gradient of the corresponding approximation ensemble, regularized by an adaptive weight decay term with a propensity for self-consistent variance minimization and sparse representations.

Development of dendritic tonic GABAergic inhibition regulates excitability and plasticity in CA1 pyramidal neurons

Synaptic plasticity rules change during development: while hippocampal synapses can be potentiated by a single action potential pairing protocol in young neurons, mature neurons require burst firing to induce synaptic potentiation. An essential component for spike timing-dependent plasticity is the backpropagating action potential (BAP). BAP along the dendrites can be modulated by morphology and ion channel composition, both of which change during late postnatal development. However, it is unclear whether these dendritic changes can explain the developmental changes in synaptic plasticity induction rules. Here, we show that tonic GABAergic inhibition regulates dendritic action potential backpropagation in adolescent, but not preadolescent, CA1 pyramidal neurons. These developmental changes in tonic inhibition also altered the induction threshold for spike timing-dependent plasticity in adolescent neurons. This GABAergic regulatory effect on backpropagation is restricted to distal regions of apical dendrites (>200 μm) and mediated by α5-containing GABA(A) receptors. Direct dendritic recordings demonstrate α5-mediated tonic GABA(A) currents in adolescent neurons which can modulate BAPs. These developmental modulations in dendritic excitability could not be explained by concurrent changes in dendritic morphology. To explain our data, model simulations propose a distally increasing or localized distal expression of dendritic α5 tonic inhibition in mature neurons. Overall, our results demonstrate that dendritic integration and plasticity in more mature dendrites are significantly altered by tonic α5 inhibition in a dendritic region-specific and developmentally regulated manner.

Layer-specific high-frequency action potential spiking in the prefrontal cortex of awake rats

Cortical pyramidal neurons show irregular in vivo action potential (AP) spiking with high-frequency bursts occurring on sparse background activity. Somatic APs can backpropagate from soma into basal and apical dendrites and locally generate dendritic calcium spikes. The critical AP frequency for generation of such dendritic calcium spikes can be very different depending on cell type or brain area involved. Previously, it was shown in vitro that calcium electrogenesis can be induced in L(ayer) 5 pyramidal neurons of prefrontal cortex (PFC). It remains an open question whether somatic burst spiking and the resulting dendritic calcium electrogenesis also occur in morphologically more compact L2/3 pyramidal neurons. Furthermore, it is not known whether critical frequencies that trigger dendritic calcium electrogenesis occur in PFC under awake conditions in vivo. Here, we addressed these issues and found that pyramidal neurons in both PFC L2/3 and L5 in awake rats spike APs in short bursts but with different probabilities. The critical frequency (CF) for calcium electrogenesis in vitro was layer-specific and lower in L5 neurons compared to L2/3. Taking the in vitro CF as a predictive measure for dendritic electrogenesis during in vivo spontaneous activity, supracritical bursts in vivo were observed in a larger fraction of L5 neurons compared to L2/3 neurons but with similar incidence within these subpopulations. Together, these results show that in PFC of awake rats, AP spiking occurs at frequencies that are relevant for dendritic calcium electrogenesis and suggest that in awake rat PFC, dendritic calcium electrogenesis may be involved in neuronal computation.

Mitral cell spike synchrony modulated by dendrodendritic synapse location

On their long lateral dendrites, mitral cells of the olfactory bulb form dendrodendritic synapses with large populations of granule cell interneurons. The mitral-granule cell microcircuit operating through these reciprocal synapses has been implicated in inducing synchrony between mitral cells. However, the specific mechanisms of mitral cell synchrony operating through this microcircuit are largely unknown and are complicated by the finding that distal inhibition on the lateral dendrites does not modulate mitral cell spikes. In order to gain insight into how this circuit synchronizes mitral cells within its spatial constraints, we built on a reduced circuit model of biophysically realistic multi-compartment mitral and granule cells to explore systematically the roles of dendrodendritic synapse location and mitral cell separation on synchrony. The simulations showed that mitral cells can synchronize when separated at arbitrary distances through a shared set of granule cells, but synchrony is optimally attained when shared granule cells form two balanced subsets, each subset clustered near to a soma of the mitral cell pairs. Another constraint for synchrony is that the input magnitude must be balanced. When adjusting the input magnitude driving a particular mitral cell relative to another, the mitral-granule cell circuit served to normalize spike rates of the mitral cells while inducing a phase shift or delay in the more weakly driven cell. This shift in phase is absent when the granule cells are removed from the circuit. Our results indicate that the specific distribution of dendrodendritic synaptic clusters is critical for optimal synchronization of mitral cell spikes in response to their odor input.

Imaging membrane potential in dendritic spines

Dendritic spines mediate most excitatory inputs in the brain. Although it is clear that spines compartmentalize calcium, it is still unknown what role, if any, they play in integrating synaptic inputs. To investigate the electrical function of spines directly, we used second harmonic generation (SHG) imaging of membrane potential in pyramidal neurons from hippocampal cultures and neocortical brain slices. With FM 4-64 as an intracellular SHG chromophore, we imaged membrane potential in the soma, dendritic branches, and spines. The SHG response to voltage was linear and seemed based on an electro-optic mechanism. The SHG sensitivity of the chromophore in spines was similar to that of the parent dendritic shaft and the soma. Backpropagation of somatic action potentials generated SHG signals at spines with similar amplitude and kinetics to somatic ones. Our optical measurements of membrane potential from spines demonstrate directly that backpropagating action potentials invade the spines.

Artificial neural networks for diagnosis and survival prediction in colon cancer

ANNs are nonlinear regression computational devices that have been used for over 45 years in classification and survival prediction in several biomedical systems, including colon cancer. Described in this article is the theory behind the three-layer free forward artificial neural networks with backpropagation error, which is widely used in biomedical fields, and a methodological approach to its application for cancer research, as exemplified by colon cancer. Review of the literature shows that applications of these networks have improved the accuracy of colon cancer classification and survival prediction when compared to other statistical or clinicopathological methods. Accuracy, however, must be exercised when designing, using and publishing biomedical results employing machine-learning devices such as ANNs in worldwide literature in order to enhance confidence in the quality and reliability of reported data.

Supralinear Ca2+ influx into dendritic tufts of layer 2/3 neocortical pyramidal neurons in vitro and in vivo

Pyramidal neurons in layer 2/3 of the neocortex are central to cortical circuitry, but the intrinsic properties of their dendrites are poorly understood. Here we study layer 2/3 apical dendrites in parallel experiments in acute brain slices and in anesthetized rats using whole-cell recordings and Ca2+ imaging. We find that backpropagation of action potentials into the dendritic arbor is actively supported by Na+ channels both in vitro and in vivo. Single action potentials evoke substantial Ca2+ influx in the apical trunk but little or none in the dendritic tuft. Supralinear Ca2+ influx is produced in the tuft, however, when an action potential is paired with synaptic input. This dendritic supralinearity enables layer 2/3 neurons to integrate ascending sensory input from layer 4 and associative input to layer 1.

Axonal dopamine receptors activate peripheral spike initiation in a stomatogastric motor neuron

We studied the effects of dopamine on the stomatogastric ganglion (STG) of the lobster, Homarus americanus. The two pyloric dilator (PD) neurons are active in the pyloric rhythm, have somata in the STG, and send axons many centimeters to innervate muscles of the stomach. Dopamine application to the stomatogastric nervous system when the PD neurons were rhythmically active evoked additional action potentials during the PD neuron interburst intervals. These action potentials were peripherally generated at a region between the STG and the first bilateral branch, approximately 1 cm away from the STG, and traveled antidromically to the neuropil and orthodromically to the pyloric dilator muscles. Focal applications of dopamine to the nerves showed that spikes could be initiated in almost the entire peripheral axon of the PD neurons. Dopamine also evoked spikes in isolated peripheral axons. The concentration threshold for peripheral spike initiation was at or below 10-9 m dopamine. Thus, the peripheral axon can play an important role in shaping the output signaling to the muscles by the motor neuron.

The contribution of dendritic Kv3 K+ channels to burst threshold in a sensory neuron

Voltage-gated ion channels localized to dendritic membranes can shape signal processing in central neurons. This study describes the distribution and functional role of a high voltage-activating K(+) channel in the electrosensory lobe (ELL) of an apteronotid weakly electric fish. We identify a homolog of the Kv3.3 K(+) channel, AptKv3.3, that exhibits a high density of mRNA expression and immunolabel that is distributed over the entire soma-dendritic axis of ELL pyramidal cells. The kinetics and pharmacology of native K(+) channels recorded in pyramidal cell somata and apical dendrites match those of AptKv3.3 channels expressed in a heterologous expression system. The functional role of AptKv3.3 channels was assessed using focal drug ejections in somatic and dendritic regions of an in vitro slice preparation. Local blockade of AptKv3.3 channels slows the repolarization of spikes in pyramidal cell somata as well as spikes backpropagating into apical dendrites. The resulting increase in dendritic spike duration lowers the threshold for a gamma-frequency burst discharge that is driven by inward current associated with backpropagating dendritic spikes. Thus, dendritic AptKv3.3 K(+) channels influence the threshold for a form of burst discharge that has an established role in feature extraction of sensory input.

Calcium-dependent persistent facilitation of spike backpropagation in the CA1 pyramidal neurons

Sodium-dependent action potentials initiated near the soma are known to backpropagate over the dendrites of CA1 pyramidal neurons in an activity-dependent manner. Consequently, later spikes in a train have smaller amplitude when recorded in the apical dendrites. We found that depolarization and resultant Ca(2+) influx into dendrites caused a persistent facilitation of spike backpropagation. Dendritic patch recordings were made from CA1 pyramidal neurons in mouse hippocampal slices under blockade of fast excitatory and inhibitory synaptic inputs. Trains of 10 backpropagating action potentials induced by antidromic stimulation showed a clear decrement in the amplitude of later spikes when recorded in the middle apical dendrites. After several depolarizing current pulses, the amplitude of later spikes increased persistently, and all spikes in a train became almost equal in size. BAPTA (10 mm) contained in the pipette or low-Ca(2+) perfusing solution abolished this depolarization-induced facilitation, indicating that Ca(2+) influx is required. This facilitation was present in Galpha(q) knock-out mice that lack the previously reported muscarinic receptor-mediated enhancement of spike backpropagation. Therefore, these two forms of facilitation are clearly distinct in their intracellular mechanisms. Intracellular injection of either calmodulin binding domain (100 micrometer) or Ca(2+)/calmodulin-kinase II (CaMKII) inhibitor 281-301 (10 micrometer) blocked the depolarization-induced facilitation. Bath application of a membrane-permeable CaMKII inhibitor KN-93 (10 micrometer) also blocked the facilitation, but KN-92 (10 micrometer), an inactive isomer of KN-93, had no effect. These results suggest that increases in [Ca(2+)](i) cause persistent facilitation of spike backpropagation in the apical dendrite of CA1 pyramidal neuron by CaMKII-dependent mechanisms.