Activation functions in deep learning: A comprehensive survey and benchmark

Dynamics of heterogeneous Hopfield neural network with adaptive activation function based on memristor

Memristor and activation function are two important nonlinear factors of the memristive Hopfield neural network. The effects of different memristors on the dynamics of Hopfield neural networks have been studied by many researchers. However, less attention has been paid to the activation function. In this paper, we present a heterogeneous memristive Hopfield neural network with neurons using different activation functions. The activation functions include fixed activation functions and an adaptive activation function, where the adaptive activation function is based on a memristor. The theoretical and experimental study of the neural network's dynamics has been conducted using phase portraits, bifurcation diagrams, and Lyapunov exponents spectras. Numerical results show that complex dynamical behaviors such as multi-scroll chaos, transient chaos, state jumps and multi-type coexisting attractors can be observed in the heterogeneous memristive Hopfield neural network. In addition, the hardware implementation of memristive Hopfield neural network with adaptive activation function is designed and verified. The experimental results are in good agreement with those obtained using numerical simulations.

Statistical and machine learning models for location-specific crop yield prediction using weather indices

Crop yield prediction gains growing importance for all stakeholders in agriculture. Since the growth and development of crops are fully connected with many weather factors, it is inevitable to incorporate meteorological information into yield prediction mechanism. The changes in climate-yield relationship are more pronounced at a local level than across relatively large regions. Hence, district or sub-region-level modeling may be an appropriate approach. To obtain a location- and crop-specific model, different models with different functional forms have to be explored. This systematic review aims to discuss research papers related to statistical and machine-learning models commonly used to predict crop yield using weather factors. It was found that Artificial Neural Network (ANN) and Multiple Linear Regression were the most applied models. Support Vector Regression (SVR) model has a high success ratio as it performed well in most of the cases. The optimization options in ANN and SVR models allow us to tune models to specific patterns of association between weather conditions of a location and crop yield. ANN model can be trained using different activation functions with optimized learning rate and number of hidden layer neurons. Similarly, the SVR model can be trained with different kernel functions and various combinations of hyperparameters. Penalized regression models namely, LASSO and Elastic Net are better alternatives to simple linear regression. The nonlinear machine learning models namely, SVR and ANN were found to perform better in most of the cases which indicates there exists a nonlinear complex association between crop yield and weather factors.

HRIDM: Hybrid Residual/Inception-Based Deeper Model for Arrhythmia Detection from Large Sets of 12-Lead ECG Recordings

Heart diseases such as cardiovascular and myocardial infarction are the foremost reasons of death in the world. The timely, accurate, and effective prediction of heart diseases is crucial for saving lives. Electrocardiography (ECG) is a primary non-invasive method to identify cardiac abnormalities. However, manual interpretation of ECG recordings for heart disease diagnosis is a time-consuming and inaccurate process. For the accurate and efficient detection of heart diseases from the 12-lead ECG dataset, we have proposed a hybrid residual/inception-based deeper model (HRIDM). In this study, we have utilized ECG datasets from various sources, which are multi-institutional large ECG datasets. The proposed model is trained on 12-lead ECG data from over 10,000 patients. We have compared the proposed model with several state-of-the-art (SOTA) models, such as LeNet-5, AlexNet, VGG-16, ResNet-50, Inception, and LSTM, on the same training and test datasets. To show the effectiveness of the computational efficiency of the proposed model, we have only trained over 20 epochs without GPU support and we achieved an accuracy of 50.87% on the test dataset for 27 categories of heart abnormalities. We found that our proposed model outperformed the previous studies which participated in the official PhysioNet/CinC Challenge 2020 and achieved fourth place as compared with the 41 official ranking teams. The result of this study indicates that the proposed model is an implying new method for predicting heart diseases using 12-lead ECGs.

Improved weight initialization for deep and narrow feedforward neural network

Appropriate weight initialization settings, along with the ReLU activation function, have become cornerstones of modern deep learning, enabling the training and deployment of highly effective and efficient neural network models across diverse areas of artificial intelligence. The problem of "dying ReLU," where ReLU neurons become inactive and yield zero output, presents a significant challenge in the training of deep neural networks with ReLU activation function. Theoretical research and various methods have been introduced to address the problem. However, even with these methods and research, training remains challenging for extremely deep and narrow feedforward networks with ReLU activation function. In this paper, we propose a novel weight initialization method to address this issue. We establish several properties of our initial weight matrix and demonstrate how these properties enable the effective propagation of signal vectors. Through a series of experiments and comparisons with existing methods, we demonstrate the effectiveness of the novel initialization method.

Improving Nowcasting of Intense Convective Precipitation by Incorporating Dual-Polarization Radar Variables into Generative Adversarial Networks

The nowcasting of strong convective precipitation is highly demanded and presents significant challenges, as it offers meteorological services to diverse socio-economic sectors to prevent catastrophic weather events accompanied by strong convective precipitation from causing substantial economic losses and human casualties. With the accumulation of dual-polarization radar data, deep learning models based on data have been widely applied in the nowcasting of precipitation. Deep learning models exhibit certain limitations in the nowcasting approach: The evolutionary method is prone to accumulate errors throughout the iterative process (where multiple autoregressive models generate future motion fields and intensity residuals and then implicitly iterate to yield predictions), and the "regression to average" issue of autoregressive model leads to the "blurring" phenomenon. The evolution method's generator is a two-stage model: In the initial stage, the generator employs the evolution method to generate the provisional forecasted data; in the subsequent stage, the generator reprocesses the provisional forecasted data. Although the evolution method's generator is a generative adversarial network, the adversarial strategy adopted by this model ignores the significance of temporary prediction data. Therefore, this study proposes an Adversarial Autoregressive Network (AANet): Firstly, the forecasted data are generated via the two-stage generators (where FURENet directly produces the provisional forecasted data, and the Semantic Synthesis Model reprocesses the provisional forecasted data); Subsequently, structural similarity loss (SSIM loss) is utilized to mitigate the influence of the "regression to average" issue; Finally, the two-stage adversarial (Tadv) strategy is adopted to assist the two-stage generators to generate more realistic and highly similar generated data. It has been experimentally verified that AANet outperforms NowcastNet in the nowcasting of the next 1 h, with a reduction of 0.0763 in normalized error (NE), 0.377 in root mean square error (RMSE), and 4.2% in false alarm rate (FAR), as well as an enhancement of 1.45 in peak signal-to-noise ratio (PSNR), 0.0208 in SSIM, 5.78% in critical success index (CSI), 6.25% in probability of detection (POD), and 5.7% in F1.

Compressed Deep Learning Models for Wearable Atrial Fibrillation Detection through Attention

Deep learning (DL) models have shown promise for the accurate detection of atrial fibrillation (AF) from electrocardiogram/photoplethysmography (ECG/PPG) data, yet deploying these on resource-constrained wearable devices remains challenging. This study proposes integrating a customized channel attention mechanism to compress DL neural networks for AF detection, allowing the model to focus only on the most salient time-series features. The results demonstrate that applying compression through channel attention significantly reduces the total number of model parameters and file size while minimizing loss in detection accuracy. Notably, after compression, performance increases for certain model variants in key AF databases (ADB and C2017DB). Moreover, analyzing the learned channel attention distributions after training enhances the explainability of the AF detection models by highlighting the salient temporal ECG/PPG features most important for its diagnosis. Overall, this research establishes that integrating attention mechanisms is an effective strategy for compressing large DL models, making them deployable on low-power wearable devices. We show that this approach yields compressed, accurate, and explainable AF detectors ideal for wearables. Incorporating channel attention enables simpler yet more accurate algorithms that have the potential to provide clinicians with valuable insights into the salient temporal biomarkers of AF. Our findings highlight that the use of attention is an important direction for the future development of efficient, high-performing, and interpretable AF screening tools for wearable technology.

Contrastive Self-supervised Learning for Neurodegenerative Disorder Classification

Neurodegenerative diseases such as Alzheimer's disease (AD) or frontotemporal lobar degeneration (FTLD) involve specific loss of brain volume, detectable in vivo using T1-weighted MRI scans. Supervised machine learning approaches classifying neurodegenerative diseases require diagnostic-labels for each sample. However, it can be difficult to obtain expert labels for a large amount of data. Self-supervised learning (SSL) offers an alternative for training machine learning models without data-labels. We investigated if the SSL models can applied to distinguish between different neurodegenerative disorders in an interpretable manner. Our method comprises a feature extractor and a downstream classification head. A deep convolutional neural network trained in a contrastive self-supervised way serves as the feature extractor, learning latent representation, while the classifier head is a single-layer perceptron. We used N=2694 T1-weighted MRI scans from four data cohorts: two ADNI datasets, AIBL and FTLDNI, including cognitively normal controls (CN), cases with prodromal and clinical AD, as well as FTLD cases differentiated into its sub-types. Our results showed that the feature extractor trained in a self-supervised way provides generalizable and robust representations for the downstream classification. For AD vs. CN, our model achieves 82% balanced accuracy on the test subset and 80% on an independent holdout dataset. Similarly, the Behavioral variant of frontotemporal dementia (BV) vs. CN model attains an 88% balanced accuracy on the test subset. The average feature attribution heatmaps obtained by the Integrated Gradient method highlighted hallmark regions, i.e., temporal gray matter atrophy for AD, and insular atrophy for BV. In conclusion, our models perform comparably to state-of-the-art supervised deep learning approaches. This suggests that the SSL methodology can successfully make use of unannotated neuroimaging datasets as training data while remaining robust and interpretable.

Self-gating stochastic-resonance-based autoencoder for unsupervised learning

Incorporating additive noise components to an ensemble of McCulloch-Pitts neurons can enhance the information representation of the input, asymptotically approaching the average firing probability for large enough ensembles. We further multiply the input by the average firing probability to control the higher probability of self-gating, thereby forming a unified noise-boosted activation model with learnable noise-related hyperparameters. This gating strategy plays a crucial role in improving the performance of neural networks, as evidenced by the optimization of the autoencoder loss at nonzero optimal-noise-scaling hyperparameters, a phenomenon termed self-gating stochastic resonance. Experiments with designed autoencoders using noise-boosted activation functions demonstrate the potential applications of the self-gating stochastic resonance effect in the field of unsupervised learning.

Peak response regularization for localization

Deep convolutional neural networks approaches often assume that the feature response has a Gaussian distribution with target-centered peak response, which can be used to guide the target location and classification. Nevertheless, such an assumption is implausible when there is progressive interference from other targets and/or background noise, which produces sub-peaks on the tracking response map and causes model drift. In this paper, we propose a feature response regularization approach for sub-peak response suppression and peak response enforcement and aim to handle progressive interference systematically. Our approach, referred to as Peak Response Regularization (PRR), applies simple-yet-efficient method to aggregate and align discriminative features, which convert local extremal response in discrete feature space to extremal response in continuous space, which enforces the localization and representation capability of convolutional features. Experiments on human pose detection, object detection, object tracking, and image classification demonstrate that PRR improves the performance of image tasks with a negligible computational cost.

Toward the design of persuasive systems for a healthy workplace: a real-time posture detection

Persuasive technologies, in connection with human factor engineering requirements for healthy workplaces, have played a significant role in ensuring a change in human behavior. Healthy workplaces suggest different best practices applicable to body posture, proximity to the computer system, movement, lighting conditions, computer system layout, and other significant psychological and cognitive aspects. Most importantly, body posture suggests how users should sit or stand in workplaces in line with best and healthy practices. In this study, we developed two study phases (pilot and main) using two deep learning models: convolutional neural networks (CNN) and Yolo-V3. To train the two models, we collected posture datasets from creative common license YouTube videos and Kaggle. We classified the dataset into comfortable and uncomfortable postures. Results show that our YOLO-V3 model outperformed CNN model with a mean average precision of 92%. Based on this finding, we recommend that YOLO-V3 model be integrated in the design of persuasive technologies for a healthy workplace. Additionally, we provide future implications for integrating proximity detection taking into consideration the ideal number of centimeters users should maintain in a healthy workplace.

Nondestructive testing of runny salted egg yolk based on improved ConvNeXt-T

Salted egg yolks from salted duck eggs are widely utilized in the domestic and international food industry as both raw materials and ingredients. When salted egg yolks are not fully cured and matured, they exist in a fluid state, with a mixture of solid and liquid internally. Due to this composition, they are susceptible to deterioration during storage and usage, necessitating their detection and classification. In this study, a dataset specifically for salted egg yolks was established, and the ConvNeXt-T model, employed as the benchmark model, underwent two notable improvements. First, a lightweight location-aware circular convolution (ParC) was introduced, utilizing a ParC-block to replace a portion of the original ConvNeXt-T block. This enhancement aimed to overcome the limitations of convolution in extracting global feature information while integrating the global sensing capability of vision transformer and the localization capability of convolution. Additionally, the activation function was modified through substitution. These improvements resulted in the final model. Experimental results indicate that the enhanced model exhibits faster convergence on the custom salted egg yolk dataset compared to the baseline model. Furthermore, a significant reduction of model parameters by a factor of 4 led to a 2.167 percentage point improvement in the accuracy of the test set. The ParC-ConvNeXt-SMU-T model achieved an accuracy of 96.833% with 26.8 million parameters. Notably, the improved model demonstrates exceptional effectiveness in recognizing salted egg yolks. PRACTICAL APPLICATION: This study can be widely applied in the process of salted egg yolk production and quality inspection, which can improve the actual sorting efficiency of salted egg yolks and reduce the labor cost at the same time. It can also be used for nondestructive testing of salted egg yolks by governmental enterprises and other regulatory authorities.

Unveiling the potential of machine learning in cost-effective degradation of pharmaceutically active compounds: A stirred photo-reactor study

In this study, neural networks and support vector regression (SVR) were employed to predict the degradation over three pharmaceutically active compounds (PhACs): Ibuprofen (IBP), diclofenac (DCF), and caffeine (CAF) within a stirred reactor featuring a flotation cell with two non-concentric ultraviolet lamps. A total of 438 datapoints were collected from published works and distributed into 70% training and 30% test datasets while cross-validation was utilized to assess the training reliability. The models incorporated 15 input variables concerning reaction kinetics, molecular properties, hydrodynamic information, presence of radiation, and catalytic properties. It was observed that the Support Vector Regression (SVR) presented a poor performance as the ε hyperparameter ignored large error over low concentration levels. Meanwhile, the Artificial Neural Networks (ANN) model was able to provide rough estimations on the expected degradation of the pollutants without requiring information regarding reaction rate constants. The multi-objective optimization analysis suggested a leading role due to ozone kinetic for a rapid degradation of the contaminants and most of the results required intensification with hydrogen peroxide and Fenton process. Although both models were affected by accuracy limitations, this work provided a lightweight model to evaluate different Advanced Oxidation Processes (AOPs) by providing general information regarding the process operational conditions as well as know molecular and catalytic properties.

Deep Learning Based Over-the-Air Training of Wireless Communication Systems without Feedback

In trainable wireless communications systems, the use of deep learning for over-the-air training aims to address the discontinuity in backpropagation learning caused by the channel environment. The primary methods supporting this learning procedure either directly approximate the backpropagation gradients using techniques derived from reinforcement learning, or explicitly model the channel environment by training a generative channel model. In both cases, over-the-air training of transmitter and receiver requires a feedback channel to sound the channel environment and obtain measurements of the learning objective. The use of continuous feedback not only demands extra system resources but also makes the training process more susceptible to adversarial attacks. Conversely, opting for a feedback-free approach to train the models over the forward link, exclusively on the receiver side, could pose challenges to reliably end the training process without intermittent testing over the actual channel environment. In this article, we propose a novel method for the over-the-air training of wireless communication systems that does not require a feedback channel to train the transmitter and receiver. Random samples are transmitted through the channel environment to train a mixture density network to approximate the channel distribution on the receiver side of the network. The transmitter and receiver models are trained with the resulting channel model, and the transmitter can be deployed after training. We show that the block error rate measurements obtained with the simulated channel are suitable for monitoring as a stopping criterion during the training process. The resulting method is demonstrated to have equivalent performance to the end-to-end autoencoder training on small message sequences.

SSDL-an automated semi-supervised deep learning approach for patient-specific 3D reconstruction of proximal femur from QCT images

Deep Learning (DL) techniques have recently been used in medical image segmentation and the reconstruction of 3D anatomies of a human body. In this work, we propose a semi-supervised DL (SSDL) approach utilizing a CNN-based 3D U-Net model for femur segmentation from sparsely annotated quantitative computed tomography (QCT) slices. Specifically, QCT slices at the proximal end of the femur forming ball and socket joint with acetabulum were annotated for precise segmentation, where a segmenting binary mask was generated using a 3D U-Net model to segment the femur accurately. A total of 5474 QCT slices were considered for training among which 2316 slices were annotated. 3D femurs were further reconstructed from segmented slices employing polynomial spline interpolation. Both qualitative and quantitative performance of segmentation and 3D reconstruction were satisfactory with more than 90% accuracy achieved for all of the standard performance metrics considered. The spatial overlap index and reproducibility validation metric for segmentation-Dice Similarity Coefficient was 91.8% for unseen patients and 99.2% for validated patients. An average relative error of 12.02% and 10.75% for volume and surface area, respectively, were computed for 3D reconstructed femurs. The proposed approach demonstrates its effectiveness in accurately segmenting and reconstructing 3D femur from QCT slices.

Enhanced accuracy through machine learning-based simultaneous evaluation: a case study of RBS analysis of multinary materials

We address the high accuracy and precision demands for analyzing large in situ or in operando spectral data sets. A dual-input artificial neural network (ANN) algorithm enables the compositional and depth-sensitive analysis of multinary materials by simultaneously evaluating spectra collected under multiple experimental conditions. To validate the developed algorithm, a case study was conducted analyzing complex Rutherford backscattering spectrometry (RBS) spectra collected in two scattering geometries. The dual-input ANN analysis excelled in providing a systematic analysis and precise results, showcasing its robustness in handling complex data and minimizing user bias. A comprehensive comparison with human supervision analysis and conventional single-input ANN analysis revealed a reduced susceptibility of the dual-input ANN analysis to inaccurately known setup parameters, a common challenge in material characterization. The developed multi-input approach can be extended to a wide range of analytical techniques, in which the combined analysis of measurements performed under different experimental conditions is beneficial for disentangling details of the material properties.

Learning pharmacometric covariate model structures with symbolic regression networks

Efficiently finding covariate model structures that minimize the need for random effects to describe pharmacological data is challenging. The standard approach focuses on identification of relevant covariates, and present methodology lacks tools for automatic identification of covariate model structures. Although neural networks could potentially be used to approximate covariate-parameter relationships, such approximations are not human-readable and come at the risk of poor generalizability due to high model complexity.In the present study, a novel methodology for the simultaneous selection of covariate model structure and optimization of its parameters is proposed. It is based on symbolic regression, posed as an optimization problem with a smooth loss function. This enables training of the model through back-propagation using efficient gradient computations.Feasibility and effectiveness are demonstrated by application to a clinical pharmacokinetic data set for propofol, containing infusion and blood sample time series from 1031 individuals. The resulting model is compared to a published state-of-the-art model for the same data set. Our methodology finds a covariate model structure and corresponding parameter values with a slightly better fit, while relying on notably fewer covariates than the state-of-the-art model. Unlike contemporary practice, finding the covariate model structure is achieved without an iterative procedure involving manual interactions.

Optimizing Neural Networks for Chemical Reaction Prediction: Insights from Methylene Blue Reduction Reactions

This paper offers a thorough investigation of hyperparameter tuning for neural network architectures using datasets encompassing various combinations of Methylene Blue (MB) Reduction by Ascorbic Acid (AA) reactions with different solvents and concentrations. The aim is to predict coefficients of decay plots for MB absorbance, shedding light on the complex dynamics of chemical reactions. Our findings reveal that the optimal model, determined through our investigation, consists of five hidden layers, each with sixteen neurons and employing the Swish activation function. This model yields an NMSE of 0.05, 0.03, and 0.04 for predicting the coefficients A, B, and C, respectively, in the exponential decay equation A + B · e-x/C. These findings contribute to the realm of drug design based on machine learning, providing valuable insights into optimizing chemical reaction predictions.

High-throughput microfluidic systems accelerated by artificial intelligence for biomedical applications

High-throughput microfluidic systems are widely used in biomedical fields for tasks like disease detection, drug testing, and material discovery. Despite the great advances in automation and throughput, the large amounts of data generated by the high-throughput microfluidic systems generally outpace the abilities of manual analysis. Recently, the convergence of microfluidic systems and artificial intelligence (AI) has been promising in solving the issue by significantly accelerating the process of data analysis as well as improving the capability of intelligent decision. This review offers a comprehensive introduction on AI methods and outlines the current advances of high-throughput microfluidic systems accelerated by AI, covering biomedical detection, drug screening, and automated system control and design. Furthermore, the challenges and opportunities in this field are critically discussed as well.

Predictive modeling based on artificial neural networks for membrane fouling in a large pilot-scale anaerobic membrane bioreactor for treating real municipal wastewater

Membrane fouling is the primary obstacle to applying anaerobic membrane bioreactors (AnMBRs) in municipal wastewater treatment. This issue holds critical significance as efficient wastewater treatment serves as a cornerstone for achieving environmental sustainability. This study uses machine learning to predict membrane fouling, taking advantage of rapid computational and algorithmic advances. Based on the 525-day operation data of a large pilot-scale AnMBR for treating real municipal wastewater, regression prediction was realized using multilayer perceptron (MLP) and long short-term memory (LSTM) artificial neural networks under substantial variations in operating conditions. The models involved employing the organic loading rate, suspended solids concentration, protein concentration in extracellular polymeric substance (EPSp), polysaccharide concentration in EPS (EPSc), reactor temperature, and flux as input features, and transmembrane pressure as the prediction target output. Hyperparameter optimization enhanced the regression prediction accuracies, and the rationality and utility of the MLP model for predicting large-scale AnMBR membrane fouling were confirmed at global and local levels of interpretability analysis. This work not only provides a methodological advance but also underscores the importance of merging environmental engineering with computational advancements to address pressing environmental challenges.

sscNOVA: a semi-supervised convolutional neural network for predicting functional regulatory variants in autoimmune diseases

Genome-wide association studies (GWAS) have identified thousands of variants in the human genome with autoimmune diseases. However, identifying functional regulatory variants associated with autoimmune diseases remains challenging, largely because of insufficient experimental validation data. We adopt the concept of semi-supervised learning by combining labeled and unlabeled data to develop a deep learning-based algorithm framework, sscNOVA, to predict functional regulatory variants in autoimmune diseases and analyze the functional characteristics of these regulatory variants. Compared to traditional supervised learning methods, our approach leverages more variants' data to explore the relationship between functional regulatory variants and autoimmune diseases. Based on the experimentally curated testing dataset and evaluation metrics, we find that sscNOVA outperforms other state-of-the-art methods. Furthermore, we illustrate that sscNOVA can help to improve the prioritization of functional regulatory variants from lead single-nucleotide polymorphisms and the proxy variants in autoimmune GWAS data.

Exploring Convolutional Neural Network Architectures for EEG Feature Extraction

The main purpose of this paper is to provide information on how to create a convolutional neural network (CNN) for extracting features from EEG signals. Our task was to understand the primary aspects of creating and fine-tuning CNNs for various application scenarios. We considered the characteristics of EEG signals, coupled with an exploration of various signal processing and data preparation techniques. These techniques include noise reduction, filtering, encoding, decoding, and dimension reduction, among others. In addition, we conduct an in-depth analysis of well-known CNN architectures, categorizing them into four distinct groups: standard implementation, recurrent convolutional, decoder architecture, and combined architecture. This paper further offers a comprehensive evaluation of these architectures, covering accuracy metrics, hyperparameters, and an appendix that contains a table outlining the parameters of commonly used CNN architectures for feature extraction from EEG signals.

Investigating bromide incorporation factor (BIF) and model development for predicting THMs in drinking water using machine learning

Many disinfection byproducts (DBPs) in drinking water can pose cancer risks to humans while several DBPs including trihalomethanes are typically regulated. Although trihalomethanes are regulated, brominated fractions (bromodichloromethane, dibromochloromethane and bromoform) are more toxic to humans than the chlorinated ones (chloroform). To date, >100 models have been reported to predict DBPs. However, models to predict individual trihalomethanes are very limited, indicating the needs of such models. Various factors including natural organic matter (NOM), bromide ions (Br-), disinfectants (e.g., chlorine dose), pH, temperature and reaction time affect the formation and distribution of trihalomethanes in drinking water. In this study, NOM was fractionated into four groups based on the molecular weight (MW) cutoff values and their respective contributions to dissolved organic carbon (DOC), trihalomethanes and bromide incorporation factors (BIF) were investigated. Models were developed for predicting chloroform, bromodichloromethane, dibromochloromethane, bromoform and trihalomethanes. Three machine learning techniques: Support Vector Regressor (SVR), Random Forest Regressor (RFR) and Artificial Neural Networks (ANN) were adopted for training and testing the models. The normalized BIFs were in the ranges of 0.08-0.16 and 0.07-0.15 per mg/L of DOC for pH 6.0 and 8.5 respectively. The BIFs were higher for lower pH and MW values while increase of bromide to chlorine ratios increased BIFs. The models showed excellent predictive performances in training (R2 = 0.889-0.998) and testing (R2 = 0.870-0.988) datasets. The SVR and RFR models showed the best performances with lower RMSE and MAE in most cases. These models can be used to better control different trihalomethanes in drinking water to maintain regulatory compliance, and to minimize the risks to humans.

Neural functional theory for inhomogeneous fluids: Fundamentals and applications

We present a hybrid scheme based on classical density functional theory and machine learning for determining the equilibrium structure and thermodynamics of inhomogeneous fluids. The exact functional map from the density profile to the one-body direct correlation function is represented locally by a deep neural network. We substantiate the general framework for the hard sphere fluid and use grand canonical Monte Carlo simulation data of systems in randomized external environments during training and as reference. Functional calculus is implemented on the basis of the neural network to access higher-order correlation functions via automatic differentiation and the free energy via functional line integration. Thermal Noether sum rules are validated explicitly. We demonstrate the use of the neural functional in the self-consistent calculation of density profiles. The results outperform those from state-of-the-art fundamental measure density functional theory. The low cost of solving an associated Euler-Lagrange equation allows to bridge the gap from the system size of the original training data to macroscopic predictions upon maintaining near-simulation microscopic precision. These results establish the machine learning of functionals as an effective tool in the multiscale description of soft matter.

Cross-Parallel Transformer: Parallel ViT for Medical Image Segmentation

Medical image segmentation primarily utilizes a hybrid model consisting of a Convolutional Neural Network and sequential Transformers. The latter leverage multi-head self-attention mechanisms to achieve comprehensive global context modelling. However, despite their success in semantic segmentation, the feature extraction process is inefficient and demands more computational resources, which hinders the network's robustness. To address this issue, this study presents two innovative methods: PTransUNet (PT model) and C-PTransUNet (C-PT model). The C-PT module refines the Vision Transformer by substituting a sequential design with a parallel one. This boosts the feature extraction capabilities of Multi-Head Self-Attention via self-correlated feature attention and channel feature interaction, while also streamlining the Feed-Forward Network to lower computational demands. On the Synapse public dataset, the PT and C-PT models demonstrate improvements in DSC accuracy by 0.87% and 3.25%, respectively, in comparison with the baseline model. As for the parameter count and FLOPs, the PT model aligns with the baseline model. In contrast, the C-PT model shows a decrease in parameter count by 29% and FLOPs by 21.4% relative to the baseline model. The proposed segmentation models in this study exhibit benefits in both accuracy and efficiency.

Deep Learning Approaches for Quantifying Ventilation Defects in Hyperpolarized Gas Magnetic Resonance Imaging of the Lung: A Review

This paper provides an in-depth overview of Deep Neural Networks and their application in the segmentation and analysis of lung Magnetic Resonance Imaging (MRI) scans, specifically focusing on hyperpolarized gas MRI and the quantification of lung ventilation defects. An in-depth understanding of Deep Neural Networks is presented, laying the groundwork for the exploration of their use in hyperpolarized gas MRI and the quantification of lung ventilation defects. Five distinct studies are examined, each leveraging unique deep learning architectures and data augmentation techniques to optimize model performance. These studies encompass a range of approaches, including the use of 3D Convolutional Neural Networks, cascaded U-Net models, Generative Adversarial Networks, and nnU-net for hyperpolarized gas MRI segmentation. The findings highlight the potential of deep learning methods in the segmentation and analysis of lung MRI scans, emphasizing the need for consensus on lung ventilation segmentation methods.

Low-Cost Optimized U-Net Model with GMM Automatic Labeling Used in Forest Semantic Segmentation

Currently, Convolutional Neural Networks (CNN) are widely used for processing and analyzing image or video data, and an essential part of state-of-the-art studies rely on training different CNN architectures. They have broad applications, such as image classification, semantic segmentation, or face recognition. Regardless of the application, one of the important factors influencing network performance is the use of a reliable, well-labeled dataset in the training stage. Most of the time, especially if we talk about semantic classification, labeling is time and resource-consuming and must be done manually by a human operator. This article proposes an automatic label generation method based on the Gaussian mixture model (GMM) unsupervised clustering technique. The other main contribution of this paper is the optimization of the hyperparameters of the traditional U-Net model to achieve a balance between high performance and the least complex structure for implementing a low-cost system. The results showed that the proposed method decreased the resources needed, computation time, and model complexity while maintaining accuracy. Our methods have been tested in a deforestation monitoring application by successfully identifying forests in aerial imagery.

Ensemble model for predicting chronic non-communicable diseases using Latin square extraction and fuzzy-artificial neural networks from 2013 to 2019

Background

The presented study tracks the increase or decrease in the prevalence of seventeen different chronic non-communicable diseases in Serbia. This analysis considers factors such as region, age, and gender and is based on data from two national cross-sectional studies conducted in 2013 and 2019. The research aims to accurately identify the regions with the highest percentage of affected individuals, as well as their respective age and gender groups. The ultimate goal is to facilitate organized, free preventive screenings for these population categories within a very short time-frame in the future.

Materials and methods

The study analyzed two cross-sectional studies conducted between 2013 and 2019, using data obtained from the Institute of Public Health of Serbia. Both studies involved a total of 27801 participants. The study compared the performance of Decision Tree and Support Vector Regressor models with artificial neural network (ANN) models that employed two encoding functions. The new methodology for the ANN-L36 model was based on artificial neural networks constructed using a Latin square (L36) design, incorporating Taguchi's robust design optimization.

Results

The results of the analysis from three different models have shown that cardiovascular diseases are the most prevalent illnesses among the population in Serbia, with hypertension as the leading condition in all regions, particularly among individuals aged 64 to 75 years, and more prevalent among females. In 2019, there was a decrease in the percentage of the leading disease, hypertension, compared to 2013, with a decrease from 34.0% to 32.2%. The ANN-L36 model with Fuzzy encoding function demonstrated the highest precision, achieving the smallest relative error of 0.1%.

Conclusion

To date, no studies have been conducted at the national level in Serbia to comprehensively track and identify chronic diseases in the manner proposed by this study. The model presented in this research will be implemented in practice and is set to significantly contribute to the future healthcare framework in Serbia, shaping and advancing the approach towards addressing these conditions. Furthermore, experimental evidence has shown that Taguchi's optimization approach yields the best results for identifying various chronic non-communicable diseases.

Automated breast tumor ultrasound image segmentation with hybrid UNet and classification using fine-tuned CNN model

Introduction

Breast cancer stands as the second most deadly form of cancer among women worldwide. Early diagnosis and treatment can significantly mitigate mortality rates.

Purpose

The study aims to classify breast ultrasound images into benign and malignant tumors. This approach involves segmenting the breast's region of interest (ROI) employing an optimized UNet architecture and classifying the ROIs through an optimized shallow CNN model utilizing an ablation study.

Method

Several image processing techniques are utilized to improve image quality by removing text, artifacts, and speckle noise, and statistical analysis is done to check the enhanced image quality is satisfactory. With the processed dataset, the segmentation of breast tumor ROI is carried out, optimizing the UNet model through an ablation study where the architectural configuration and hyperparameters are altered. After obtaining the tumor ROIs from the fine-tuned UNet model (RKO-UNet), an optimized CNN model is employed to classify the tumor into benign and malignant classes. To enhance the CNN model's performance, an ablation study is conducted, coupled with the integration of an attention unit. The model's performance is further assessed by classifying breast cancer with mammogram images.

Result

The proposed classification model (RKONet-13) results in an accuracy of 98.41 %. The performance of the proposed model is further compared with five transfer learning models for both pre-segmented and post-segmented datasets. K-fold cross-validation is done to assess the proposed RKONet-13 model's performance stability. Furthermore, the performance of the proposed model is compared with previous literature, where the proposed model outperforms existing methods, demonstrating its effectiveness in breast cancer diagnosis. Lastly, the model demonstrates its robustness for breast cancer classification, delivering an exceptional performance of 96.21 % on a mammogram dataset.

Conclusion

The efficacy of this study relies on image pre-processing, segmentation with hybrid attention UNet, and classification with fine-tuned robust CNN model. This comprehensive approach aims to determine an effective technique for detecting breast cancer within ultrasound images.

Deep Learning for Counting People from UWB Channel Impulse Response Signals

The use of higher frequency bands compared to other wireless communication protocols enhances the capability of accurately determining locations from ultra-wideband (UWB) signals. It can also be used to estimate the number of people in a room based on the waveform of the channel impulse response (CIR) from UWB transceivers. In this paper, we apply deep neural networks to UWB CIR signals for the purpose of estimating the number of people in a room. We especially focus on empirically investigating the various network architectures for classification from single UWB CIR data, as well as from various ensemble configurations. We present our processes for acquiring and preprocessing CIR data, our designs of the different network architectures and ensembles that were applied, and the comparative experimental evaluations. We demonstrate that deep neural networks can accurately classify the number of people within a Line of Sight (LoS), thereby achieving an 99% performance and efficiency with respect to both memory size and FLOPs (Floating Point Operations Per Second).

Deep Reinforcement Learning for Workload Prediction in Federated Cloud Environments

The Federated Cloud Computing (FCC) paradigm provides scalability advantages to Cloud Service Providers (CSP) in preserving their Service Level Agreement (SLA) as opposed to single Data Centers (DC). However, existing research has primarily focused on Virtual Machine (VM) placement, with less emphasis on energy efficiency and SLA adherence. In this paper, we propose a novel solution, Federated Cloud Workload Prediction with Deep Q-Learning (FEDQWP). Our solution addresses the complex VM placement problem, energy efficiency, and SLA preservation, making it comprehensive and beneficial for CSPs. By leveraging the capabilities of deep learning, our FEDQWP model extracts underlying patterns and optimizes resource allocation. Real-world workloads are extensively evaluated to demonstrate the efficacy of our approach compared to existing solutions. The results show that our DQL model outperforms other algorithms in terms of CPU utilization, migration time, finished tasks, energy consumption, and SLA violations. Specifically, our QLearning model achieves efficient CPU utilization with a median value of 29.02, completes migrations in an average of 0.31 units, finishes an average of 699 tasks, consumes the least energy with an average of 1.85 kWh, and exhibits the lowest number of SLA violations with an average of 0.03 violations proportionally. These quantitative results highlight the superiority of our proposed method in optimizing performance in FCC environments.

Heart rate estimation from ballistocardiogram signals processing via low-cost telemedicine architectures: a comparative performance evaluation

Medical devices (MDs) have been designed for monitoring the parameters of patients in many sectors. Nonetheless, despite being high-performing and reliable, they often turn out to be expensive and intrusive. In addition, MDs are almost exclusively used in controlled, hospital-based environments. Paving a path of technological innovation in the clinical field, a very active line of research is currently dealing with the possibility to rely on non-medical-graded low-cost devices, to develop unattended telemedicine (TM) solutions aimed at non-invasively gathering data, signals, and images. In this article, a TM solution is proposed for monitoring the heart rate (HR) of patients during sleep. A remote patient monitoring system (RPMS) featuring a smart belt equipped with pressure sensors for ballistocardiogram (BCG) signals sampling was deployed. A field trial was then conducted over a 2-month period on 24 volunteers, who also agreed to wear a finger pulse oximeter capable of producing a photoplethysmography (PPG) signal as the gold standard, to examine the feasibility of the solution via the estimation of HR values from the collected BCG signals. For this purpose, two of the highest-performing approaches for HR estimation from BCG signals, one algorithmic and the other based on a convolutional neural network (CNN), were retrieved from the literature and updated for a TM-related use case. Finally, HR estimation performances were assessed in terms of patient-wise mean absolute error (MAE). Results retrieved from the literature (controlled environment) outperformed those achieved in the experimentation (TM environment) by 29% (MAE = 4.24 vs. 5.46, algorithmic approach) and 52% (MAE = 2.32 vs. 3.54, CNN-based approach), respectively. Nonetheless, a low packet loss ratio, restrained elaboration time of the collected biomedical big data, low-cost deployment, and positive feedback from the users, demonstrate the robustness, reliability, and applicability of the proposed TM solution. In light of this, further steps will be planned to fulfill new targets, such as evaluation of respiratory rate (RR), and pattern assessment of the movement of the participants overnight.

Frugal Heart Rate Correction Method for Scalable Health and Safety Monitoring in Construction Sites

Continuous, real-time monitoring of occupational health and safety in high-risk workplaces such as construction sites can substantially improve the safety of workers. However, introducing such systems in practice is associated with a number of challenges, such as scaling up the solution while keeping its cost low. In this context, this work investigates the use of an off-the-shelf, low-cost smartwatch to detect health issues based on heart rate monitoring in a privacy-preserving manner. To improve the smartwatch's low measurement quality, a novel, frugal machine learning method is proposed that corrects measurement errors, along with a new dataset for this task. This method's integration with the smartwatch and the remaining parts of the health and safety monitoring system (built on the ASSIST-IoT reference architecture) are presented. This method was evaluated in a laboratory environment in terms of its accuracy, computational requirements, and frugality. With an experimentally established mean absolute error of 8.19 BPM, only 880 bytes of required memory, and a negligible impact on the performance of the device, this method meets all relevant requirements and is expected to be field-tested in the coming months. To support reproducibility and to encourage alternative approaches, the dataset, the trained model, and its implementation on the smartwatch were published under free licenses.

Personalized movie recommendations based on deep representation learning

Personalized recommendation is a technical means to help users quickly and efficiently obtain interesting content from massive information. However, the traditional recommendation algorithm is difficult to solve the problem of sparse data and cold-start and does not make reasonable use of the user-item rating matrix. In this article, a personalized recommendation method based on deep belief network (DBN) and softmax regression is proposed to address the issues with traditional recommendation algorithms. In this method, the DBN is used to learn the deep representation of users and items, and the user-item rating matrix is maximized. Then softmax regression is used to learn multiple categories in the feature space to predict the probability of interaction between users and items. Finally, the method is applied to the area of movie recommendation. The key to this method is the negative sampling mechanism, which greatly improves the effectiveness of the recommendations, as a result, creates an accurate list of recommendations. This method was verified and evaluated on Douban and several movielens datasets of different sizes. The experimental results demonstrate that the recommended performance of this model, which has high accuracy and generalization ability, is much better than typical baseline models such as singular value decomposition (SVD), and the mean absolute error (MAE) value is 98%, which is lower than the best baseline model.

Predicting System Degradation with a Guided Neural Network Approach

Evaluating the physical degradation behavior and estimating the lifetime of engineering systems and structures is crucial to ensure their safe and reliable operation. However, measuring lifetime through actual operating conditions can be a difficult and slow process. While valuable and quick in measuring lifetimes, accelerated life testing is often oversimplified and does not provide accurate simulations of the exact operating environment. This paper proposes a data-driven framework for time-efficient modeling of field degradation using sensor measurements from short-term actual operating conditions degradation tests. The framework consists of two neural networks: a physics discovery neural network and a predictive neural network. The former models the underlying physics of degradation, while the latter makes probabilistic predictions for degradation intensity. The physics discovery neural network guides the predictive neural network for better life estimations. The proposed framework addresses two main challenges associated with applying neural networks for lifetime estimation: incorporating the underlying physics of degradation and requirements for extensive training data. This paper demonstrates the effectiveness of the proposed approach through a case study of atmospheric corrosion of steel test samples in a marine environment. The results show the proposed framework's effectiveness, where the mean absolute error of the predictions is lower by up to 76% compared to a standard neural network. By employing the proposed data-driven framework for lifetime prediction, systems safety and reliability can be evaluated efficiently, and maintenance activities can be optimized.

Non-uniform image reconstruction for fast photoacoustic microscopy of histology imaging

Photoacoustic microscopic imaging utilizes the characteristic optical absorption properties of pigmented materials in tissues to enable label-free observation of fine morphological and structural features. Since DNA/RNA can strongly absorb ultraviolet light, ultraviolet photoacoustic microscopy can highlight the cell nucleus without complicated sample preparations such as staining, which is comparable to the standard pathological images. Further improvements in the imaging acquisition speed are critical to advancing the clinical translation of photoacoustic histology imaging technology. However, improving the imaging speed with additional hardware is hampered by considerable costs and complex design. In this work, considering heavy redundancy in the biological photoacoustic images that overconsume the computing power, we propose an image reconstruction framework called non-uniform image reconstruction (NFSR), which exploits an object detection network to reconstruct low-sampled photoacoustic histology images into high-resolution images. The sampling speed of photoacoustic histology imaging is significantly improved, saving 90% of the time cost. Furthermore, NFSR focuses on the reconstruction of the region of interest while maintaining high PSNR and SSIM evaluation indicators of more than 99% but reducing the overall computation by 60%.

Deep Learning Based Multiresponse Optimization Methodology for Dual-Axis MEMS Accelerometer

This paper presents a deep neural network (DNN) based design optimization methodology for dual-axis microelectromechanical systems (MEMS) capacitive accelerometer. The proposed methodology considers the geometric design parameters and operating conditions of the MEMS accelerometer as input parameters and allows to analyze the effect of the individual design parameters on the output responses of the sensor using a single model. Moreover, a DNN-based model allows to simultaneously optimize the multiple output responses of the MEMS accelerometers in an efficient manner. The efficiency of the proposed DNN-based optimization model is compared with the design of the computer experiments (DACE) based multiresponse optimization methodology presented in the Literature, which showed a better performance in terms of two output performance metrics, i.e., mean absolute error (MAE) and root mean squared error (RMSE).

The assessment of response surface methodology (RSM) and artificial neural network (ANN) modeling in dry flue gas desulfurization at low temperatures

The performance of a flue gas desulfurization (FGD) system is characterized by SO₂ removal efficiency (Y1) and reagent conversion (Y2). Achieving a near-perfect reaction environment has been of concern in dry FGD (DFGD) due to the low reactivity compared to the wet and semi-dry units. This study will appraise output responses using modeling by response surface methodology (RSM) and artificial neural networks (ANN) approaches. The impacts of input parameters like hydration time, hydration temperature, diatomite to hydrated lime (Ca(OH)₂), sulfation temperature and inlet gas concentration will be studied using a randomized central composite design (CCD). ANN fitting tool mapped the CCD metadata using the Levenberg-Marquardt (LM) algorithm activated by the hyperbolic tangent (tansig) function. The hidden cells ranged from 7 to 10 to ascertain the effect node architecture on modeling accuracy. Validation of each procedure was assessed using root mean square error (RMSE), mean square error (MSE) and R-Squared studies. The outcomes presented a more accurate 5-10-2 ANN model in the mapping of the DFGD from R² data of Y1 = 0.993 and Y2 = 0.9986 with a mapping deviation from the RMSE values of Y1 = 0.48465; Y2 = 0.44971 and MSE results of Y1 = 0.23488; Y2.= 0.20229.

LSTM-Autoencoder for Vibration Anomaly Detection in Vertical Carousel Storage and Retrieval System (VCSRS)

Industry 5.0, also known as the "smart factory", is an evolution of manufacturing technology that utilizes advanced data analytics and machine learning techniques to optimize production processes. One key aspect of Industry 5.0 is using vibration data to monitor and detect anomalies in machinery and equipment. In the case of a vertical carousel storage and retrieval system (VCSRS), vibration data can be collected and analyzed to identify potential issues with the system's operation. A correlation coefficient model was used to detect anomalies accurately in the vertical carousel system to ascertain the optimal sensor placement position. This model utilized the Fisher information matrix (FIM) and effective independence (EFI) methods to optimize the sensor placement for maximum accuracy and reliability. An LSTM-autoencoder (long short-term memory) model was used for training and testing further to enhance the accuracy of the anomaly detection process. This machine-learning technique allowed for detecting patterns and trends in the vibration data that may not have been evident using traditional methods. The combination of the correlation coefficient model and the LSTM-autoencoder resulted in an accuracy rate of 97.70% for detecting anomalies in the vertical carousel system.

Crop genomic selection with deep learning and environmental data: A survey

Machine learning techniques for crop genomic selections, especially for single-environment plants, are well-developed. These machine learning models, which use dense genome-wide markers to predict phenotype, routinely perform well on single-environment datasets, especially for complex traits affected by multiple markers. On the other hand, machine learning models for predicting crop phenotype, especially deep learning models, using datasets that span different environmental conditions, have only recently emerged. Models that can accept heterogeneous data sources, such as temperature, soil conditions and precipitation, are natural choices for modeling GxE in multi-environment prediction. Here, we review emerging deep learning techniques that incorporate environmental data directly into genomic selection models.

Optimal control by deep learning techniques and its applications on epidemic models

We represent the optimal control functions by neural networks and solve optimal control problems by deep learning techniques. Adjoint sensitivity analysis is applied to train the neural networks embedded in differential equations. This method can not only be applied in classic epidemic control problems, but also in epidemic forecasting, discovering unknown mechanisms, and the ideas behind can give new insights to traditional mathematical epidemiological problems.

Evaluating the Performance of Deep Learning Frameworks for Malaria Parasite Detection Using Microscopic Images of Peripheral Blood Smears

Malaria is a significant health concern in many third-world countries, especially for pregnant women and young children. It accounted for about 229 million cases and 600,000 mortality globally in 2019. Hence, rapid and accurate detection is vital. This study is focused on achieving three goals. The first is to develop a deep learning framework capable of automating and accurately classifying malaria parasites using microscopic images of thin and thick peripheral blood smears. The second is to report which of the two peripheral blood smears is the most appropriate for use in accurately detecting malaria parasites in peripheral blood smears. Finally, we evaluate the performance of our proposed model with commonly used transfer learning models. We proposed a convolutional neural network capable of accurately predicting the presence of malaria parasites using microscopic images of thin and thick peripheral blood smears. Model evaluation was carried out using commonly used evaluation metrics, and the outcome proved satisfactory. The proposed model performed better when thick peripheral smears were used with accuracy, precision, and sensitivity of 96.97%, 97.00%, and 97.00%. Identifying the most appropriate peripheral blood smear is vital for improved accuracy, rapid smear preparation, and rapid diagnosis of patients, especially in regions where malaria is endemic.

Rapid Identification of Wild Gentiana Genus in Different Geographical Locations Based on FT-IR and an Improved Neural Network Structure Double-Net

Gentiana Genus, a herb mainly distributed in Asia and Europe, has been used to treat the damp heat disease of the liver for over 2000 years in China. Previous studies have shown significant differences in the compositional contents of wild Gentiana Genus samples from different geographical origins. Therefore, the traceable geographic locations of the wild Gentiana Genus samples are essential to ensure practical medicinal value. Over the last few years, the developments in chemometrics have facilitated the analysis of the composition of medicinal herbs via spectroscopy. Notably, FT-IR spectroscopy is widely used because of its benefit of allowing rapid, nondestructive measurements. In this paper, we collected wild Gentiana Genus samples from seven different provinces (222 samples in total). Twenty-one different FT-IR spectral pre-processing methods that were used in our experiments. Meanwhile, we also designed a neural network, Double-Net, to predict the geographical locations of wild Gentiana Genus plants via FT-IR spectroscopy. The experiments showed that the accuracy of the neural network structure Double-Net we designed can reach 100%, and the F1_score can reach 1.0.