Estimating the Number of Hidden Neurons in a Feedforward Network Using the Singular Value Decomposition

Hybrid deep modeling of a CHO-K1 fed-batch process: combining first-principles with deep neural networks

Introduction: Hybrid modeling combining First-Principles with machine learning is becoming a pivotal methodology for Biopharma 4.0 enactment. Chinese Hamster Ovary (CHO) cells, being the workhorse for industrial glycoproteins production, have been the object of several hybrid modeling studies. Most previous studies pursued a shallow hybrid modeling approach based on three-layered Feedforward Neural Networks (FFNNs) combined with macroscopic material balance equations. Only recently, the hybrid modeling field is incorporating deep learning into its framework with significant gains in descriptive and predictive power. Methods: This study compares, for the first time, deep and shallow hybrid modeling in a CHO process development context. Data of 24 fed-batch cultivations of a CHO-K1 cell line expressing a target glycoprotein, comprising 30 measured state variables over time, were used to compare both methodologies. Hybrid models with varying FFNN depths (3-5 layers) were systematically compared using two training methodologies. The classical training is based on the Levenberg-Marquardt algorithm, indirect sensitivity equations and cross-validation. The deep learning is based on the Adaptive Moment Estimation Method (ADAM), stochastic regularization and semidirect sensitivity equations. Results and conclusion: The results point to a systematic generalization improvement of deep hybrid models over shallow hybrid models. Overall, the training and testing errors decreased by 14.0% and 23.6% respectively when applying the deep methodology. The Central Processing Unit (CPU) time for training the deep hybrid model increased by 31.6% mainly due to the higher FFNN complexity. The final deep hybrid model is shown to predict the dynamics of the 30 state variables within the error bounds in every test experiment. Notably, the deep hybrid model could predict the metabolic shifts in key metabolites (e.g., lactate, ammonium, glutamine and glutamate) in the test experiments. We expect deep hybrid modeling to accelerate the deployment of high-fidelity digital twins in the biopharma sector in the near future.

Multiple ECG segments denoising autoencoder model

OBJECTIVES

Denoising autoencoder (DAE) with a single hidden layer of neurons can recode a signal, i.e., converting the original signal into a noise-reduced signal. The DAE approach has shown a good performance in denoising bio-signals, like electrocardiograms (ECG). In this paper, we study the effect of correlated, uncorrelated and jittered datasets on the performance of the DAE model.

METHODS

Vectors of multiple concatenated ECG segments of simultaneously recorded Einthoven recordings I, II, III are considered to establish the following dataset cases: (1) correlated, (2) uncorrelated, and (3) jittered. We consider our previous work in finding the optimal number of hidden neurons receiving the input signal with respect to signal quality and computational burden by applying Akaike's information criterion. To evaluate DAE, these datasets are corrupted with six types of noise, namely mix noise (MX), motion artifact noise (MA), electrode movement (EM), baseline wander (BW), Gaussian white noise (GWN) and high-frequency noise (HFN), to simulate real case scenario. Spectral analysis is used to study the effects of noise whose power spectrum may overlap with the power spectrum of the wanted signal on DAE performance.

RESULTS

The simulation results show (a) that the number of hidden neurons to denoise multiple correlated ECG is much lower than for jittered signals, (b) QRS-complex based ECG alignment preferable, (c) noises with slightly overlapping power spectrum, like BW and HFN, can be easily removed with sufficient number of neurons, while the noise with completely overlapping spectrum, like GWN, requires a very low-dimensional and thus coarser reduction to recover the signal.

CONCLUSIONS

The performance of DAE model in terms of signal-to-noise ratio improvement and the required number of hidden neurons can be improved by utilizing the correlation among simultaneous Einthoven I, II, III records.

Socio-Economic Factors Affecting Member's Satisfaction towards National Health Insurance: An Evidence from the Philippines

The National Health Insurance, "PhilHealth", is the healthcare provider for Filipino citizens in the Philippines. The study focused on determining the effects of members' satisfaction with PhilHealth among Filipino members. The study utilized 10 latent variables from the integrated Service Quality (SERVQUAL) and Expectation-Confirmation Theory (ECT). There are 500 respondents that are used and analyzed through Structural Equation Modeling (SEM) and a Deep Learning Neural Network (DLNN). Utilizing SEM, it was revealed that Reliability, Responsiveness, Socio-Economic Factors, Expectation, Perceived Performance, Confirmation of Beliefs, and Members' Satisfaction are significant factors in the satisfaction of PhilHealth members. Utilizing DLNN, it was found that Expectation (EX) is the most significant factor, and it is consistent with the results of the SEM. The government can use the findings of this study for the improvement of PhilHealth. The framework that is used for the analysis can be extended and can apply to future research with regard to its provided services. The overall results, framework, and concept utilized may be applied by other service industries worldwide.

Recurrent Neural Network-Based Multimodal Deep Learning for Estimating Missing Values in Healthcare

This estimation method operates by integrating the input values that are redundantly collected from heterogeneous devices through the selection of a representative value and estimating missing values by using a multimodal RNN. Users use a heterogeneous healthcare platform mainly in a mobile environment. Users who pay a relatively large amount of attention to healthcare possess various types of healthcare devices and collect data through their mobile devices. The collected data may be duplicated depending on the types of these devices. This data duplication causes an ambiguity issue in that it is difficult to determine which value among multiple data should be taken as the user’s actual value. Accordingly, it is necessary to create a neural network structure that considers the data value at the time previous to the current time. RNNs are appropriate for handling data with a time series characteristic. To learn an RNN-based neural network, learning data that have the same time step are required. Therefore, an RNN in which one variable becomes single-modal was designed for each learning run. In the RNN, a cell is a gated recurrent unit (GRU) cell that presents sufficient accuracy in the small resource environment of mobile devices. The RNNs that are learned according to the variables can each operate without additional learning, even if the situation of the user’s mobile device changes. In a heterogeneous environment, missing values are generated by various types of errors, including errors caused by battery charge and discharge, sensor failure, equipment exchange, and near-field communication errors. The higher the missing value ratio, the greater the number of errors that are likely to occur. For this reason, to achieve a more stable heterogeneous health platform, missing values must be considered. In this study, a missing value was estimated by means of multimodal deep learning; that is, a multimodal deep learning method was designed with one neural network that was connected with each learned single-modal RNN using a fully connected network (FCN). Each RNN input value delivers mutual influence through the weights of the FCN, and thereby, it is possible to estimate an output value even if any one of the input values is missing. According to the evaluation in terms of representative value selection, when a representative value was selected by using the mean or median, the most stable service was achieved. As a result of the evaluation according to the estimation method, the accuracy of the RNN-based multimodal deep learning method is 3.91%p higher than that of the SVD method.

Optimization of the structural complexity of artificial neural network for hardware-driven neuromorphic computing application

This work focuses on the optimization of the structural complexity of a single-layer feedforward neural network (SLFN) for neuromorphic hardware implementation. The singular value decomposition (SVD) method is used for the determination of the effective number of neurons in the hidden layer for Modified National Institute of Standards and Technology (MNIST) dataset classification. The proposed method is also verified on a SLFN using weights derived from a synaptic transistor device. The effectiveness of this methodology in estimating the reduced number of neurons in the hidden layer makes this method highly useful in optimizing complex neural network architectures for their hardware realization.

Neural Network Model for Predicting Student Failure in the Academic Leveling Course of Escuela Politécnica Nacional

The purpose of this study is to train an artificial neural network model for predicting student failure in the academic leveling course of the Escuela Politécnica Nacional of Ecuador, based on academic and socioeconomic information. For this, 1308 higher education students participated, 69.0% of whom failed the academic leveling course; besides, 93.7% of the students self-identified as mestizo, 83.9% came from the province of Pichincha, and 92.4% belonged to general population. As a first approximation, a neural network model was trained with twelve variables containing students’ academic and socioeconomic information. Then, a dimensionality reduction process was performed from which a new neural network was modeled. This dimension reduced model was trained with the variables application score, vulnerability index, regime, gender, and population segment, which were the five variables that explained more than 80% of the first model. The classification accuracy of the dimension reduced model was 0.745, while precision and recall were 0.883 and 0.778, respectively. The area under ROC curve was 0.791. This model could be used as a guide to lead intervention policies so that the failure rate in the academic leveling course would decrease.

Machine Learning in Predictive Toxicology: Recent Applications and Future Directions for Classification Models

In recent times, machine learning has become increasingly prominent in predictive toxicology as it has shifted from in vivo studies toward in silico studies. Currently, in vitro methods together with other computational methods such as quantitative structure-activity relationship modeling and absorption, distribution, metabolism, and excretion calculations are being used. An overview of machine learning and its applications in predictive toxicology is presented here, including support vector machines (SVMs), random forest (RF) and decision trees (DTs), neural networks, regression models, naïve Bayes, k-nearest neighbors, and ensemble learning. The recent successes of these machine learning methods in predictive toxicology are summarized, and a comparison of some models used in predictive toxicology is presented. In predictive toxicology, SVMs, RF, and DTs are the dominant machine learning methods due to the characteristics of the data available. Lastly, this review describes the current challenges facing the use of machine learning in predictive toxicology and offers insights into the possible areas of improvement in the field.

A Hybrid Spatio-Temporal Prediction Model for Solar Photovoltaic Generation Using Numerical Weather Data and Satellite Images

Precise and accurate prediction of solar photovoltaic (PV) generation plays a major role in developing plans for the supply and demand of power grid systems. Most previous studies on the prediction of solar PV generation employed only weather data composed of numerical text data. The numerical text weather data can reflect temporal factors, however, they cannot consider the movement features related to the wind direction of the spatial characteristics, which include the amount of both clouds and particulate matter (PM) among other weather features. This study aims developing a hybrid spatio-temporal prediction model by combining general weather data and data extracted from satellite images having spatial characteristics. A model for hourly prediction of solar PV generation is proposed using data collected from a solar PV power plant in Incheon, South Korea. To evaluate the performance of the prediction model, we compared and performed ARIMAX analysis, which is a traditional statistical time-series analysis method, and SVR, ANN, and DNN, which are based on machine learning algorithms. The models that reflect the temporal and spatial characteristics exhibited better performance than those using only the general weather numerical data or the satellite image data.

Early Dropout Prediction Model: A Case Study of University Leveling Course Students

The school-dropout problem is a serious issue that affects both a country’s education system and its economy, given the substantial investment in education made by national governments. One strategy for counteracting the problem at an early stage is to identify students at risk of dropping out. The present study introduces a model to predict student dropout rates in the Escuela Politécnica Nacional leveling course. Data related to 2097 higher education students were analyzed; a logistic regression model and an artificial neural network model were trained using four variables, which incorporated student academic and socio-economic information. After comparing the two models, the neural network, with an experimentally defined architecture of 4–7–1 architecture and a logistic activation function, was selected as the model that should be applied to early predict dropout in the leveling course. The study findings show that students with the highest risk of dropping out are those in vulnerable situations, with low application grades, from the Costa regime, who are enrolled in the leveling course for technical degrees. This model can be used by the university authorities to identify possible dropout cases, as well as to establish policies to reduce university dropout and failure rates.

Appearance and characterization of fruit image textures for quality sorting using wavelet transform and genetic algorithms

Images of four qualities of mangoes and guavas are evaluated for color and textural features to characterize and classify them, and to model the fruit appearance grading. The paper discusses three approaches to identify most discriminating texture features of both the fruits. In the first approach, fruit's color and texture features are selected using Mahalanobis distance. A total of 20 color features and 40 textural features are extracted for analysis. Using Mahalanobis distance and feature intercorrelation analyses, one best color feature (mean of a* [L*a*b* color space]) and two textural features (energy a*, contrast of H*) are selected as features for Guava while two best color features (R std, H std) and one textural features (energy b*) are selected as features for mangoes with the highest discriminate power. The second approach studies some common wavelet families for searching the best classification model for fruit quality grading. The wavelet features extracted from five basic mother wavelets (db, bior, rbior, Coif, Sym) are explored to characterize fruits texture appearance. In third approach, genetic algorithm is used to select only those color and wavelet texture features that are relevant to the separation of the class, from a large universe of features. The study shows that image color and texture features which were identified using a genetic algorithm can distinguish between various qualities classes of fruits. The experimental results showed that support vector machine classifier is elected for Guava grading with an accuracy of 97.61% and artificial neural network is elected from Mango grading with an accuracy of 95.65%.

PRACTICAL APPLICATIONS

The proposed method is nondestructive fruit quality assessment method. The experimental results has proven that Genetic algorithm along with wavelet textures feature has potential to discriminate fruit quality. Finally, it can be concluded that discussed method is an accurate, reliable, and objective tool to determine fruit quality namely Mango and Guava, and might be applicable to in-line sorting systems.

Generalized single-hidden layer feedforward networks for regression problems

In this paper, traditional single-hidden layer feedforward network (SLFN) is extended to novel generalized SLFN (GSLFN) by employing polynomial functions of inputs as output weights connecting randomly generated hidden units with corresponding output nodes. The significant contributions of this paper are as follows: 1) a primal GSLFN (P-GSLFN) is implemented using randomly generated hidden nodes and polynomial output weights whereby the regression matrix is augmented by full or partial input variables and only polynomial coefficients are to be estimated; 2) a simplified GSLFN (S-GSLFN) is realized by decomposing the polynomial output weights of the P-GSLFN into randomly generated polynomial nodes and tunable output weights; 3) both P- and S-GSLFN are able to achieve universal approximation if the output weights are tuned by ridge regression estimators; and 4) by virtue of the developed batch and online sequential ridge ELM (BR-ELM and OSR-ELM) learning algorithms, high performance of the proposed GSLFNs in terms of generalization and learning speed is guaranteed. Comprehensive simulation studies and comparisons with standard SLFNs are carried out on real-world regression benchmark data sets. Simulation results demonstrate that the innovative GSLFNs using BR-ELM and OSR-ELM are superior to standard SLFNs in terms of accuracy, training speed, and structure compactness.

Combination of uniform design with artificial neural network coupling genetic algorithm: an effective way to obtain high yield of biomass and algicidal compound of a novel HABs control actinomycete

Controlling harmful algae blooms (HABs) using microbial algicides is cheap, efficient and environmental-friendly. However, obtaining high yield of algicidal microbes to meet the need of field test is still a big challenge since qualitative and quantitative analysis of algicidal compounds is difficult. In this study, we developed a protocol to increase the yield of both biomass and algicidal compound present in a novel algicidal actinomycete Streptomyces alboflavus RPS, which kills Phaeocystis globosa. To overcome the problem in algicidal compound quantification, we chose algicidal ratio as the index and used artificial neural network to fit the data, which was appropriate for this nonlinear situation. In this protocol, we firstly determined five main influencing factors through single factor experiments and generated the multifactorial experimental groups with a U₁₅(15⁵) uniform-design-table. Then, we used the traditional quadratic polynomial stepwise regression model and an accurate, fully optimized BP-neural network to simulate the fermentation. Optimized with genetic algorithm and verified using experiments, we successfully increased the algicidal ratio of the fermentation broth by 16.90% and the dry mycelial weight by 69.27%. These results suggested that this newly developed approach is a viable and easy way to optimize the fermentation conditions for algicidal microorganisms.

A new formulation for feedforward neural networks

Feedforward neural network is one of the most commonly used function approximation techniques and has been applied to a wide variety of problems arising from various disciplines. However, neural networks are black-box models having multiple challenges/difficulties associated with training and generalization. This paper initially looks into the internal behavior of neural networks and develops a detailed interpretation of the neural network functional geometry. Based on this geometrical interpretation, a new set of variables describing neural networks is proposed as a more effective and geometrically interpretable alternative to the traditional set of network weights and biases. Then, this paper develops a new formulation for neural networks with respect to the newly defined variables; this reformulated neural network (ReNN) is equivalent to the common feedforward neural network but has a less complex error response surface. To demonstrate the learning ability of ReNN, in this paper, two training methods involving a derivative-based (a variation of backpropagation) and a derivative-free optimization algorithms are employed. Moreover, a new measure of regularization on the basis of the developed geometrical interpretation is proposed to evaluate and improve the generalization ability of neural networks. The value of the proposed geometrical interpretation, the ReNN approach, and the new regularization measure are demonstrated across multiple test problems. Results show that ReNN can be trained more effectively and efficiently compared to the common neural networks and the proposed regularization measure is an effective indicator of how a network would perform in terms of generalization.