A general backpropagation algorithm for feedforward neural networks learning

Innovative rapid liquid concentration measurement based on thermal lens effect and machine learning

This study addresses the critical need for rapid and online measurement of liquid concentrations in industrial applications. Although the thermal lens effect (TLE) is extensively explored in laser systems for determining thermal lens focal lengths, its application in quantifying solution concentrations remains underexplored. This research explores the relationship between various liquid concentrations and the interference fringes induced by the TLE. A novel approach is introduced, utilizing TLE to measure solution concentrations, with integration of image processing and discrete Fourier transform (DFT) techniques for feature extraction from interference rings. Further, machine learning, specifically backpropagation artificial neural network (BP-ANN), is employed to model concentration measurement. The model demonstrates high accuracy, evidenced by low root mean square error (RMSE) values of 3.055 and 5.396 for the training and test sets, respectively. This enables precise, real-time determination of soy sauce concentration, offering significant implications for industrial testing, environmental monitoring, and other related fields.

Analysis on the inherent noise tolerance of feedforward network and one noise-resilient structure

In the past few decades, feedforward neural networks have gained much attraction in their hardware implementations. However, when we realize a neural network in analog circuits, the circuit-based model is sensitive to hardware nonidealities. The nonidealities, such as random offset voltage drifts and thermal noise, may lead to variation in hidden neurons and further affect neural behaviors. This paper considers that time-varying noise exists at the input of hidden neurons, with zero-mean Gaussian distribution. First, we derive lower and upper bounds on the mean square error loss to estimate the inherent noise tolerance of a noise-free trained feedforward network. Then, the lower bound is extended for any non-Gaussian noise cases based on the Gaussian mixture model concept. The upper bound is generalized for any non-zero-mean noise case. As the noise could degrade the neural performance, a new network architecture is designed to suppress the noise effect. This noise-resilient design does not require any training process. We also discuss its limitation and give a closed-form expression to describe the noise tolerance when the limitation is exceeded.

Machine Learning for Wireless Sensor Networks Security: An Overview of Challenges and Issues

Energy and security are major challenges in a wireless sensor network, and they work oppositely. As security complexity increases, battery drain will increase. Due to the limited power in wireless sensor networks, options to rely on the security of ordinary protocols embodied in encryption and key management are futile due to the nature of communication between sensors and the ever-changing network topology. Therefore, machine learning algorithms are one of the proposed solutions for providing security services in this type of network by including monitoring and decision intelligence. Machine learning algorithms present additional hurdles in terms of training and the amount of data required for training. This paper provides a convenient reference for wireless sensor network infrastructure and the security challenges it faces. It also discusses the possibility of benefiting from machine learning algorithms by reducing the security costs of wireless sensor networks in several domains; in addition to the challenges and proposed solutions to improving the ability of sensors to identify threats, attacks, risks, and malicious nodes through their ability to learn and self-development using machine learning algorithms. Furthermore, this paper discusses open issues related to adapting machine learning algorithms to the capabilities of sensors in this type of network.

Development of artificial neural networks software for arsenic adsorption from an aqueous environment

Arsenic contamination is a global problem, as it affects the health of millions of people. For this study, data-driven artificial neural network (ANN) software was developed to predict and validate the removal of As(V) from an aqueous solution using graphene oxide (GO) under various experimental conditions. A reliable model for wastewater treatment is essential in order to predict its overall performance and to provide an idea of how to control its operation. This model considered the adsorption process parameters (initial concentration, adsorbent dosage, pH, and residence time) as the input variables and arsenic removal as the only output. The ANN model predicted the adsorption efficiency with high accuracy for both training and testing datasets, when compared with the available response surface methodology (RSM) model. Based on the best model synaptic weights, user-friendly ANN software was created to predict and analyze arsenic removal as a function of adsorption process parameters. We developed various graphical user interfaces (GUI) for easy use of the developed model. Thus, a researcher can efficiently operate the software without an understanding of programming or artificial neural networks. Sensitivity analysis and quantitative estimation were carried out to study the function of adsorption process parameter variables on As(V) removal efficiency, using the GUI of the model. The model prediction shows that the adsorbent dosages, initial concentration, and pH are the most influential parameters. The efficiency was increased as the adsorbent dosages increased, decreasing with initial concentration and pH. The result show that the pH 2.0-5.0 is optimal for adsorbent efficiency (%).

Modelling of adsorption of anionic azo dye using Strychnos potatorum Linn seeds (SPS) from aqueous solution with artificial neural network (ANN)

Synthetic dyes used in the textile and paper industries pose a major threat to the environment. In the present research work, the adsorption efficiency of the natural adsorbent Strychnos potatorum Linn (Fam: Loganiaceae) seeds were examined against the reactive orange-M2R dye from aqueous solution by varying the process conditions such as contact time, pH, adsorbent dosage, and initial dye concentration on adsorption of anionic azo dye. This study compares different types of artificial neural networks which are feedforward artificial neural network (FANN) and nonlinear autoregressive exogenous (NARX) model to predict the efficiency of a cost-effective natural adsorbent Strychnos potatorum Linn seeds on removing reactive orange-M2R dye from aqueous solution. Twelve training algorithms of neural network were compared, and the prediction on the adsorption performance of anionic azo dye from aqueous solution using Strychnos potatonum Linn seeds was evaluated by using the root mean squared error (RMSE), mean absolute error (MAE), coefficient of determination (R²), and accuracy. For FANN model, Levenberg-Marquardt (LM) backpropagation with 19 hidden neurons was selected as the optimum FANN model, with R² of 0.994 and accuracy of 87.20%, 98.21%, and 66.60% for training, testing, and validation datasets, respectively. For NARX model, LM with 8 hidden neurons was selected as the most suitable training algorithm, with R² value of more than 0.99 and accuracy of 88.00%, 90.91%, and 75.00% for training, testing, and validation datasets, respectively. NARX model accurately predicted the adsorption of anionic azo dye from aqueous solution using Strychnos potatonum Linn seeds with better performance than FANN model.

Acoustic Characterization of Rooms Using Reverberation Time Estimation Based on Supervised Learning Algorithm

The measurement of reverberation time is an essential procedure for the characterization of the acoustic performance of rooms. The values returned by these measurements allow us to predict how the sound will be transformed by the walls and furnishings of the rooms. The measurement of the reverberation time is not an easy procedure to carry out and requires the use of a space in an exclusive way. In fact, it is necessary to use instruments that reproduce a sound source and instruments for recording the response of the space. In this work, an automatic procedure for estimating the reverberation time based on the use of artificial neural networks was developed. Previously selected sounds were played, and joint sound recordings were made. The recorded sounds were processed with the extraction of characteristics, then they were labeled by associating to each sound the value of the reverberation time in octave bands of that specific room. The obtained dataset was used as input for the training of an algorithm based on artificial neural networks. The results returned by the predictive model suggest using this methodology to estimate the reverberation time of any closed space, using simple audio recordings without having to perform standard measurements or calculate the integration explicitly.

Road Extraction of High-Resolution Remote Sensing Images Derived from DenseUNet

Road network extraction is one of the significant assignments for disaster emergency response, intelligent transportation systems, and real-time updating road network. Road extraction base on high-resolution remote sensing images has become a hot topic. Presently, most of the researches are based on traditional machine learning algorithms, which are complex and computational because of impervious surfaces such as roads and buildings that are discernible in the images. Given the above problems, we propose a new method to extract the road network from remote sensing images using a DenseUNet model with few parameters and robust characteristics. DenseUNet consists of dense connection units and skips connections, which strengthens the fusion of different scales by connections at various network layers. The performance of the advanced method is validated on two datasets of high-resolution images by comparison with three classical semantic segmentation methods. The experimental results show that the method can be used for road extraction in complex scenes.

An Intelligent Fire Warning Application Using IoT and an Adaptive Neuro-Fuzzy Inference System

In the recent past, a few fire warning and alarm systems have been presented based on a combination of a smoke sensor and an alarm device to design a life-safety system. However, such fire alarm systems are sometimes error-prone and can react to non-actual indicators of fire presence classified as false warnings. There is a need for high-quality and intelligent fire alarm systems that use multiple sensor values (such as a signal from a flame detector, humidity, heat, and smoke sensors, etc.) to detect true incidents of fire. An Adaptive neuro-fuzzy Inference System (ANFIS) is used in this paper to calculate the maximum likelihood of the true presence of fire and generate fire alert. The novel idea proposed in this paper is to use ANFIS for the identification of a true fire incident by using change rate of smoke, the change rate of temperature, and humidity in the presence of fire. The model consists of sensors to collect vital data from sensor nodes where Fuzzy logic converts the raw data in a linguistic variable which is trained in ANFIS to get the probability of fire occurrence. The proposed idea also generates alerts with a message sent directly to the user’s smartphone. Our system uses small size, cost-effective sensors and ensures that this solution is reproducible. MATLAB-based simulation is used for the experiments and the results show a satisfactory output.

Integrating a Hybrid Back Propagation Neural Network and Particle Swarm Optimization for Estimating Soil Heavy Metal Contents Using Hyperspectral Data

Soil heavy metals affect human life and the environment, and thus, it is very necessary to monitor their contents. Substantial research has been conducted to estimate and map soil heavy metals in large areas using hyperspectral data and machine learning methods (such as neural network), however, lower estimation accuracy is often obtained. In order to improve the estimation accuracy, in this study, a back propagation neural network (BPNN) was combined with the particle swarm optimization (PSO), which led to an integrated PSO-BPNN method used to estimate the contents of soil heavy metals: Cd, Hg, and As. This study was conducted in Guangdong, China, based on the soil heavy metal contents and hyperspectral data collected from 90 soil samples. The prediction accuracies from BPNN and PSO-BPNN were compared using field observations. The results showed that, 1) the sample averages of Cd, Hg, and As were 0.174 mg/kg, 0.132 mg/kg, and 9.761 mg/kg, respectively, with the corresponding maximum values of 0.570 mg/kg, 0.310 mg/kg, and 68.600 mg/kg being higher than the environment baseline values; 2) the transformed and combined spectral variables had higher correlations with the contents of the soil heavy metals than the original spectral data; 3) PSO-BPNN significantly improved the estimation accuracy of the soil heavy metal contents, with the decrease in the mean relative error (MRE) and relative root mean square error (RRMSE) by 68% to 71%, and 64% to 67%, respectively. This indicated that the PSO-BPNN provided great potential to estimate the soil heavy metal contents; and 4) with the PSO-BPNN, the Cd content could also be mapped using HuanJing-1A Hyperspectral Imager (HSI) data with a RRMSE value of 36%, implying that the PSO-BPNN method could be utilized to map the heavy metal content in soil, using both field spectral data and hyperspectral imagery for the large area.

Automatic segmentation of vertebrae in 3D CT images using adaptive fast 3D pulse coupled neural networks

Two systems are presented for segmentation of vertebrae in a 3D computed tomography (CT) image. The first method extracts seven features from each voxel and uses a multi-layer perceptron neural network (MLPNN) to classify the voxel as vertebrae or background. In the second method, the segmentation is completed in two steps: first, a newly developed adaptive pulse coupled neural network (APCNN) directly applied to a given image segments vertebrae, then the result is refined using a median filter. In the developed APCNN, the values for the user-defined parameters of the pulse coupled neural networks (PCNN) are adaptively adjusted for each image individually, instead of using one value for all images as in conventional PCNN. The performance of both systems in terms of Dice index (DI) was evaluated and compared against the state-of-the-art segmentation methods using seventeen clinical and standard CT images. Overall, both systems demonstrated statistically similar and promising performance with average DI > 95%. Compared to existing PCNN-based segmentation algorithms, the accuracy of the proposed APCNN improved by 29.3% on average. The developed APCNN-based system is more accurate than MLPNN-based system and existing PCNN-based algorithms in segmentation of vertebrae with blurred and weak boundaries and in the images contaminated by salt- and- pepper noise. In terms of computation time, the APCNN-based system is 16 times faster than the MLPNN-based system. Consequently, the presented APCNN-based algorithm is both accurate and fast and could be used in clinical environment for segmentation of vertebrae in 3D CT images.

Deep learning enables automated scoring of liver fibrosis stages

Current liver fibrosis scoring by computer-assisted image analytics is not fully automated as it requires manual preprocessing (segmentation and feature extraction) typically based on domain knowledge in liver pathology. Deep learning-based algorithms can potentially classify these images without the need for preprocessing through learning from a large dataset of images. We investigated the performance of classification models built using a deep learning-based algorithm pre-trained using multiple sources of images to score liver fibrosis and compared them against conventional non-deep learning-based algorithms - artificial neural networks (ANN), multinomial logistic regression (MLR), support vector machines (SVM) and random forests (RF). Automated feature classification and fibrosis scoring were achieved by using a transfer learning-based deep learning network, AlexNet-Convolutional Neural Networks (CNN), with balanced area under receiver operating characteristic (AUROC) values of up to 0.85–0.95 versus ANN (AUROC of up to 0.87–1.00), MLR (AUROC of up to 0.73–1.00), SVM (AUROC of up to 0.69–0.99) and RF (AUROC of up to 0.94–0.99). Results indicate that a deep learning-based algorithm with transfer learning enables the construction of a fully automated and accurate prediction model for scoring liver fibrosis stages that is comparable to other conventional non-deep learning-based algorithms that are not fully automated.

ECG authentication system design incorporating a convolutional neural network and generalized S-Transformation

Electrocardiogram (ECG) is gaining increased attention as a biometric method in a wide range of applications, such as access control and security/privacy requirements. The majority of reported investigations using the ECG biometric method are usually based on fiducial or nonfiducial methods, which are always accompanied by a series of issues, such as locating fiducial points accurately is difficult, feature selection is subjective, and classifiers are limited by the quantity and structure of data. This paper proposes a new biometric authentication system for human identification that uses ECG signals as a biometric trait and integrates a generalized S-transformation and a convolutional neural network (CNN). Specifically, we first introduce a blind segmentation strategy that effectively avoids difficult data-specific heartbeat recognition and segmentation techniques. Then, a generalized S-transformation is performed on the blind signal-processed ECG signal, capturing the ECG trajectory at each time point in the frequency domain. Next, the getframe technology is used to capture an image of the ECG trajectories and convert the one-dimensional signal to a two-dimensional image, which serves as the input layer of the CNN, thus fully reflecting the changing trend in the ECG signal spectrum characteristics over a continuous period. Finally, the CNN is used for automatic discriminative feature learning and representations, which avoids a tedious feature extraction algorithm. In addition, considering the possible impact of ECG signals with different signal behaviors on identification, experiments are performed on three ECG databases with diverse features, comprising normal individuals, atrial fibrillation patients, and a noisy database, to evaluate the effectiveness of the proposed algorithm. Promising identification rates of 99%, 98%, and 99% were achieved, respectively. Thus, our proposed ECG authentication system can be effectively used for identity recognition under various conditions.

Multiclass classification of obstructive sleep apnea/hypopnea based on a convolutional neural network from a single-lead electrocardiogram

OBJECTIVE

In this paper, we propose a convolutional neural network (CNN)-based deep learning architecture for multiclass classification of obstructive sleep apnea and hypopnea (OSAH) using single-lead electrocardiogram (ECG) recordings. OSAH is the most common sleep-related breathing disorder. Many subjects who suffer from OSAH remain undiagnosed; thus, early detection of OSAH is important.

APPROACH

In this study, automatic classification of three classes-normal, hypopnea, and apnea-based on a CNN is performed. An optimal six-layer CNN model is trained on a training dataset (45 096 events) and evaluated on a test dataset (11 274 events). The training set (69 subjects) and test set (17 subjects) were collected from 86 subjects with length of approximately 6 h and segmented into 10 s durations.

MAIN RESULTS

The proposed CNN model reaches a mean [Formula: see text]-score of 93.0 for the training dataset and 87.0 for the test dataset.

SIGNIFICANCE

Thus, proposed deep learning architecture achieved a high performance for multiclass classification of OSAH using single-lead ECG recordings. The proposed method can be employed in screening of patients suspected of having OSAH.

A Generalized Hopfield Network for Nonsmooth Constrained Convex Optimization: Lie Derivative Approach

This paper proposes a generalized Hopfield network for solving general constrained convex optimization problems. First, the existence and the uniqueness of solutions to the generalized Hopfield network in the Filippov sense are proved. Then, the Lie derivative is introduced to analyze the stability of the network using a differential inclusion. The optimality of the solution to the nonsmooth constrained optimization problems is shown to be guaranteed by the enhanced Fritz John conditions. The convergence rate of the generalized Hopfield network can be estimated by the second-order derivative of the energy function. The effectiveness of the proposed network is evaluated on several typical nonsmooth optimization problems and used to solve the hierarchical and distributed model predictive control four-tank benchmark.

Improved Neural Networks with Random Weights for Short-Term Load Forecasting

An effective forecasting model for short-term load plays a significant role in promoting the management efficiency of an electric power system. This paper proposes a new forecasting model based on the improved neural networks with random weights (INNRW). The key is to introduce a weighting technique to the inputs of the model and use a novel neural network to forecast the daily maximum load. Eight factors are selected as the inputs. A mutual information weighting algorithm is then used to allocate different weights to the inputs. The neural networks with random weights and kernels (KNNRW) is applied to approximate the nonlinear function between the selected inputs and the daily maximum load due to the fast learning speed and good generalization performance. In the application of the daily load in Dalian, the result of the proposed INNRW is compared with several previously developed forecasting models. The simulation experiment shows that the proposed model performs the best overall in short-term load forecasting.

Artificial neural network modeling of the water quality index for Kinta River (Malaysia) using water quality variables as predictors

This article describes design and application of feed-forward, fully-connected, three-layer perceptron neural network model for computing the water quality index (WQI)(1) for Kinta River (Malaysia). The modeling efforts showed that the optimal network architecture was 23-34-1 and that the best WQI predictions were associated with the quick propagation (QP) training algorithm; a learning rate of 0.06; and a QP coefficient of 1.75. The WQI predictions of this model had significant, positive, very high correlation (r=0.977, p<0.01) with the measured WQI values, implying that the model predictions explain around 95.4% of the variation in the measured WQI values. The approach presented in this article offers useful and powerful alternative to WQI computation and prediction, especially in the case of WQI calculation methods which involve lengthy computations and use of various sub-index formulae for each value, or range of values, of the constituent water quality variables.

A unified self-stabilizing neural network algorithm for principal and minor components extraction

Recently, many unified learning algorithms have been developed for principal component analysis and minor component analysis. These unified algorithms can be used to extract principal components and, if altered simply by the sign, can also serve as a minor component extractor. This is of practical significance in the implementations of algorithms. This paper proposes a unified self-stabilizing neural network learning algorithm for principal and minor components extraction, and studies the stability of the proposed unified algorithm via the fixed-point analysis method. The proposed unified self-stabilizing algorithm for principal and minor components extraction is extended for tracking the principal subspace (PS) and minor subspace (MS). The averaging differential equation and the energy function associated with the unified algorithm for tracking PS and MS are given. It is shown that the averaging differential equation will globally asymptotically converge to an invariance set, and the corresponding energy function exhibit a unique global minimum attained if and only if its state matrices span the PS or MS of the autocorrelation matrix of a vector data stream. It is concluded that the proposed unified algorithm for tracking PS and MS can efficiently track an orthonormal basis of the PS or MS. Simulations are carried out to further illustrate the theoretical results achieved.

Uniformly stable backpropagation algorithm to train a feedforward neural network

Neural networks (NNs) have numerous applications to online processes, but the problem of stability is rarely discussed. This is an extremely important issue because, if the stability of a solution is not guaranteed, the equipment that is being used can be damaged, which can also cause serious accidents. It is true that in some research papers this problem has been considered, but this concerns continuous-time NN only. At the same time, there are many systems that are better described in the discrete time domain such as population of animals, the annual expenses in an industry, the interest earned by a bank, or the prediction of the distribution of loads stored every hour in a warehouse. Therefore, it is of paramount importance to consider the stability of the discrete-time NN. This paper makes several important contributions. 1) A theorem is stated and proven which guarantees uniform stability of a general discrete-time system. 2) It is proven that the backpropagation (BP) algorithm with a new time-varying rate is uniformly stable for online identification and the identification error converges to a small zone bounded by the uncertainty. 3) It is proven that the weights' error is bounded by the initial weights' error, i.e., overfitting is eliminated in the proposed algorithm. 4) The BP algorithm is applied to predict the distribution of loads that a transelevator receives from a trailer and places in the deposits in a warehouse every hour, so that the deposits in the warehouse are reserved in advance using the prediction results. 5) The BP algorithm is compared with the recursive least square (RLS) algorithm and with the Takagi-Sugeno type fuzzy inference system in the problem of predicting the distribution of loads in a warehouse, giving that the first and the second are stable and the third is unstable. 6) The BP algorithm is compared with the RLS algorithm and with the Kalman filter algorithm in a synthetic example.

On Adaptive Learning Rate That Guarantees Convergence in Feedforward Networks

This paper investigates new learning algorithms (LF I and LF II) based on Lyapunov function for the training of feedforward neural networks. It is observed that such algorithms have interesting parallel with the popular backpropagation (BP) algorithm where the fixed learning rate is replaced by an adaptive learning rate computed using convergence theorem based on Lyapunov stability theory. LF II, a modified version of LF I, has been introduced with an aim to avoid local minima. This modification also helps in improving the convergence speed in some cases. Conditions for achieving global minimum for these kind of algorithms have been studied in detail. The performances of the proposed algorithms are compared with BP algorithm and extended Kalman filtering (EKF) on three bench-mark function approximation problems: XOR, 3-bit parity, and 8-3 encoder. The comparisons are made in terms of number of learning iterations and computational time required for convergence. It is found that the proposed algorithms (LF I and II) are much faster in convergence than other two algorithms to attain same accuracy. Finally, the comparison is made on a complex two-dimensional (2-D) Gabor function and effect of adaptive learning rate for faster convergence is verified. In a nutshell, the investigations made in this paper help us better understand the learning procedure of feedforward neural networks in terms of adaptive learning rate, convergence speed, and local minima.

Study of a fast discriminative training algorithm for pattern recognition

Discriminative training refers to an approach to pattern recognition based on direct minimization of a cost function commensurate with the performance of the recognition system. This is in contrast to the procedure of probability distribution estimation as conventionally required in Bayes' formulation of the statistical pattern recognition problem. Currently, most discriminative training algorithms for nonlinear classifier designs are based on gradient-descent (GD) methods for cost minimization. These algorithms are easy to derive and effective in practice, but are slow in training speed and have difficulty selecting the learning rates. To address the problem, we present our study on a fast discriminative training algorithm. The algorithm initializes the parameters by the expectation-maximization (EM) algorithm, and then uses a set of closed-form formulas derived in this paper to further optimize a proposed objective of minimizing error rate. Experiments in speech applications show that the algorithm provides better recognition accuracy in a fewer iterations than the EM algorithm and a neural network trained by hundreds of GD iterations. Although some convergent properties need further research, the proposed objective and derived formulas can benefit further study of the problem.