Multivariate Temporal Convolutional Network: A Deep Neural Networks Approach for Multivariate Time Series Forecasting

Performance Analysis and Prediction of 5G Round-Trip Time Based on the VMD-LSTM Method

With the increasing level of industrial informatization, massive industrial data require real-time and high-fidelity wireless transmission. Although some industrial wireless network protocols have been designed over the last few decades, most of them have limited coverage and narrow bandwidth. They cannot always ensure the certainty of information transmission, making it especially difficult to meet the requirements of low latency in industrial manufacturing fields. The 5G technology is characterized by a high transmission rate and low latency; therefore, it has good prospects in industrial applications. To apply 5G technology to factory environments with low latency requirements for data transmission, in this study, we analyze the statistical performance of the round-trip time (RTT) in a 5G-R15 communication system. The results indicate that the average value of 5G RTT is about 11 ms, which is less than the 25 ms of WIA-FA. We then consider 5G RTT data as a group of time series, utilizing the augmented Dickey-Fuller (ADF) test method to analyze the stability of the RTT data. We conclude that the RTT data are non-stationary. Therefore, firstly, the original 5G RTT series are subjected to first-order differencing to obtain differential sequences with stronger stationarity. Then, a time series analysis-based variational mode decomposition-long short-term memory (VMD-LSTM) method is proposed to separately predict each differential sequence. Finally, the predicted results are subjected to inverse difference to obtain the predicted value of 5G RTT, and a predictive error of 4.481% indicates that the method performs better than LSTM and other methods. The prediction results could be used to evaluate network performance based on business requirements, reduce the impact of instruction packet loss, and improve the robustness of control algorithms. The proposed early warning accuracy metrics for control issues can also be used to indicate when to retrain the model and to indicate the setting of the control cycle. The field of industrial control, especially in the manufacturing industry, which requires low latency, will benefit from this analysis. It should be noted that the above analysis and prediction methods are also applicable to the R16 and R17 versions.

MONDEP: A unified SpatioTemporal MONitoring Framework for National DEPression Forecasting

Depression has become a prevalent mental disorder that significantly affects a person's emotions, behaviors, physical health, ability to perform daily tasks, and ability to maintain healthy relationships. Untreated depression can escalate the risk of suicide, making the situation even worse. Despite an abundance of models previously proposed for forecasting depression, the issue of foretelling the overall number of patients at each administrative level remains under-investigated. Therefore, in this paper, we propose a simple but effective SpatioTemporal Monitoring Framework for National Depression Forecasting (MONDEP). In particular, we analyze national depression statistics data in Thailand as a case study and create prediction models for a real-time depression forecasting system using machine learning and deep learning approaches. In order to forecast the prevalence of depression at various administrative levels, we use the hierarchical structure of depression aggregation. The proposed framework consists of three modules: Data Pre-processing to extract and pre-process the raw data, Exploratory Data Analysis (EDA) to visualize and analyze the data to get insight, and Model Training and Testing to predict future depression cases. The objective of our research is to construct a comprehensive MONDEP framework that utilizes machine learning and deep learning to predict depression profiles at the district and national levels using multivariate time series across various administrative levels. Our study illustrates the considerable association between a spatial-temporal component and demonstrates how depression profiles may be represented by employing lower administrative-level data to estimate the general level of mental health across the nation. Additionally, the best performance across all criteria is obtained when a deep learning model is used to exploit multivariate time series, showing a 13% improvement in MAE measure compared to the SARIMAX baseline. We believe the proposed framework could be used as a point of reference for decision-making regarding the management of depression and has the potential to be incredibly helpful for policymakers in successfully managing mental health services on time.

Weibull analysis of ceramics and related materials: A review

It has been realized throughout the years that an ideal combination of high toughness, hardness and strength is required in many engineering applications that need load-bearing capabilities. Ceramics and related materials have significant constraints for structural and particular non-structural applications due to their low toughness and limited strength while having substantially superior hardness than typical metallic materials. For example, hydroxyapatite (HAp) has gained attention for applications in orthopaedic implants, dental materials, drug delivery, etc. Researchers have continued to strive to produce HAp materials with reliable properties within the acceptable Weibull modulus (m) for load bearing. The Weibull analysis (WA) is a statistical analysis adopted widely in reliability applications to detect failure periods. Researchers have confirmed it to be an effective technique to get results on the reliability of materials at a moderately low rate with assured reliability of the material or component. This review summarizes the WA and the steps in the Weibull method for its reliability analysis to predict the failure rate of ceramics like HAp and other related materials. Also, the applications of WA for these materials were reviewed. From the review, it was discovered that Weibull distribution is proven to confer to the feeblest-link concept. For brittle materials, it was revealed that the Weibull Modulus ranges from 2 to 40, and environment, production processes, and comparative factors are well-thought-out contributing factors for reliability. In addition, the confidence interval can be up to 95 %. The frequently used technique for reliability valuation is to syndicate the Weibull statistics. Also, a very narrow distribution is desirable to offer the expected likelihood. Furthermore, when paired with trials, Monte Carlo simulations prove to be a very helpful tool for forecasting the dependability of different estimate techniques and their optimization. Finally, if the equivalent m is anticipated to be high, it signifies that the material has a high degree of homogeneity of properties and high reliability. WA can find application in predicting the dependability and lifetime of materials, making it widely utilized in engineering and other disciplines. It is especially useful for analysing data in which the likelihood of failure per unit of time varies over time.

Load Prediction in Double-Channel Residual Self-Attention Temporal Convolutional Network with Weight Adaptive Updating in Cloud Computing

When resource demand increases and decreases rapidly, container clusters in the cloud environment need to respond to the number of containers in a timely manner to ensure service quality. Resource load prediction is a prominent challenge issue with the widespread adoption of cloud computing. A novel cloud computing load prediction method has been proposed, the Double-channel residual Self-attention Temporal convolutional Network with Weight adaptive updating (DSTNW), in order to make the response of the container cluster more rapid and accurate. A Double-channel Temporal Convolution Network model (DTN) has been developed to capture long-term sequence dependencies and enhance feature extraction capabilities when the model handles long load sequences. Double-channel dilated causal convolution has been adopted to replace the single-channel dilated causal convolution in the DTN. A residual temporal self-attention mechanism (SM) has been proposed to improve the performance of the network and focus on features with significant contributions from the DTN. DTN and SM jointly constitute a dual-channel residual self-attention temporal convolutional network (DSTN). In addition, by evaluating the accuracy aspects of single and stacked DSTNs, an adaptive weight strategy has been proposed to assign corresponding weights for the single and stacked DSTNs, respectively. The experimental results highlight that the developed method has outstanding prediction performance for cloud computing in comparison with some state-of-the-art methods. The proposed method achieved an average improvement of 24.16% and 30.48% on the Container dataset and Google dataset, respectively.

Prototype Learning for Medical Time Series Classification via Human-Machine Collaboration

Deep neural networks must address the dual challenge of delivering high-accuracy predictions and providing user-friendly explanations. While deep models are widely used in the field of time series modeling, deciphering the core principles that govern the models' outputs remains a significant challenge. This is crucial for fostering the development of trusted models and facilitating domain expert validation, thereby empowering users and domain experts to utilize them confidently in high-risk decision-making contexts (e.g., decision-support systems in healthcare). In this work, we put forward a deep prototype learning model that supports interpretable and manipulable modeling and classification of medical time series (i.e., ECG signal). Specifically, we first optimize the representation of single heartbeat data by employing a bidirectional long short-term memory and attention mechanism, and then construct prototypes during the training phase. The final classification outcomes (i.e., normal sinus rhythm, atrial fibrillation, and other rhythm) are determined by comparing the input with the obtained prototypes. Moreover, the proposed model presents a human-machine collaboration mechanism, allowing domain experts to refine the prototypes by integrating their expertise to further enhance the model's performance (contrary to the human-in-the-loop paradigm, where humans primarily act as supervisors or correctors, intervening when required, our approach focuses on a human-machine collaboration, wherein both parties engage as partners, enabling more fluid and integrated interactions). The experimental outcomes presented herein delineate that, within the realm of binary classification tasks-specifically distinguishing between normal sinus rhythm and atrial fibrillation-our proposed model, albeit registering marginally lower performance in comparison to certain established baseline models such as Convolutional Neural Networks (CNNs) and bidirectional long short-term memory with attention mechanisms (Bi-LSTMAttns), evidently surpasses other contemporary state-of-the-art prototype baseline models. Moreover, it demonstrates significantly enhanced performance relative to these prototype baseline models in the context of triple classification tasks, which encompass normal sinus rhythm, atrial fibrillation, and other rhythm classifications. The proposed model manifests a commendable prediction accuracy of 0.8414, coupled with macro precision, recall, and F1-score metrics of 0.8449, 0.8224, and 0.8235, respectively, achieving both high classification accuracy as well as good interpretability.

Employing hybrid deep learning for near-real-time forecasts of sensor-based algal parameters in a Microcystis bloom-dominated lake

Harmful cyanobacterial blooms (CyanoHABs) are increasingly impacting the ecosystem of lakes, reservoirs and estuaries globally. The integration of real-time monitoring and deep learning technology has opened up new horizons for early warnings of CyanoHABs. However, unlike traditional methods such as pigment quantification or microscopy counting, the high-frequency data from in-situ fluorometric sensors display unpredictable fluctuations and variability, posing a challenge for predictive models to discern underlying trends within the time-series sequence. This study introduces a hybrid framework for near-real-time CyanoHABs predictions in a cyanobacterium Microcystis-dominated lake - Lake Dianchi, China. The proposed model was validated using hourly Chlorophyll-a (Chl a) concentrations and algal cell densities. Our results demonstrate that applying decomposition-based singular spectrum analysis (SSA) significantly enhances the prediction accuracy of subsequent CyanoHABs models, particularly in the case of temporal convolutional network (TCN). Comparative experiments revealed that the SSA-TCN model outperforms other SSA-based deep learning models for predicting Chl a (R2 = 0.45-0.93, RMSE = 2.29-5.89 μg/L) and algal cell density (R2 = 0.63-0.89, RMSE = 9489.39-16,015.37 cells/mL) at one to four steps ahead predictions. The forecast of bloom intensities achieved a remarkable accuracy of 98.56 % and an average precision rate of 94.04 % ± 0.05 %. In addition, scenarios involving various input combinations of environmental factors demonstrated that water temperature emerged as the most effective driver for CyanoHABs predictions, with a mean RMSE of 2.94 ± 0.12 μg/L, MAE of 1.55 ± 0.09 μg/L, and R2 of 0.83 ± 0.01. Overall, the newly developed approach underscores the potential of a well-designed hybrid deep-learning framework for accurately predicting sensor-based algal parameters. It offers novel perspectives for managing CyanoHABs through online monitoring and artificial intelligence in aquatic ecosystems.

GT-LSTM: A spatio-temporal ensemble network for traffic flow prediction

Traffic flow prediction plays an instrumental role in modern intelligent transportation systems. Numerous existing studies utilize inter-embedded fusion routes to extract the intrinsic patterns of traffic flow with a single temporal learning approach, which relies heavily on constructing graphs and has low training efficiency. Different from existing studies, this paper proposes a spatio-temporal ensemble network that aims to leverage the strengths of different sequential capturing approaches to obtain the intrinsic dependencies of traffic flow. Specifically, we propose a novel model named graph temporal convolutional long short-term memory network (GT-LSTM), which mainly consists of features splicing and patterns capturing. In features splicing, the spatial dependencies of traffic flow are captured by employing self-adaptive graph convolutional network (GCN), and a non-inter-embedded approach is designed to integrate the spatial and temporal states. Further, the aggregated spatio-temporal states are fed into patterns capturing, which can effectively exploit the advantages of temporal convolutional network (TCN) and bidirectional long short-term memory network (Bi-LSTM) to extract the intrinsic patterns of traffic flow. Extensive experiments conducted on four real-world datasets demonstrate that the proposed network obtains excellent performance in both forecasting accuracy and training efficiency.

A hybrid neural network based model for blood donation forecasting

OBJECTIVE

To develop a hybrid neural network-based blood donation prediction method, via this predictive model, we can obtain the best estimate of whole blood in Beijing Tongzhou District Central Blood Station and help managers smoothly solve the allocation problem under fluctuating hospital demand and limited resources.

METHOD

Inspired by the practical problems faced by blood stations providing transfusion services to several hospitals, a hybrid model based on a time-series prediction method and neural network, SARIMAX-TCN-LSTM is proposed for the prediction of daily whole blood donations. The experiment was performed at the central blood station in Tongzhou district, where we used whole blood donations from January 1, 2015, to November 14, 2021, as the subject, supplemented by meteorological and epidemic factors affecting blood donation, to predict daily blood donations for the next two weeks.

RESULT

The hybrid model significantly outperformed the traditional time series forecasting method on multiple regression metrics, with twice as effective fitting as the baseline and a 33% reduction in Root Mean Squared Error (RMSE). Results indicate that the proposed model can improve the prediction accuracy of daily blood donations, and the co-validity of the structure was evidenced in an ablation experiment.

CONCLUSION

Development and evaluation of a hybrid neural network-based model structure improve the prediction of daily blood donations. This intelligent forecasting method can help managers to overcome the challenges of sudden blood demand and contribute to the optimization of resource allocation tasks.

Application of robust deep learning models to predict mine water inflow: Implication for groundwater environment management

Traditional mine water inflow prediction is characterized by a high degree of uncertainty in model parameters and complex mechanisms involved in the water inflow process. Data-driven models play a key role in predicting inflow mechanisms without considering physical changes. However, the existing models are limited by nonlinearity and non-stationarity. Thus, the principal objective of this study was to propose two robust models, the DIFF-TCN model and the DIFF-LSTM model, for predicting the average water inflow per day. The models consist of three methods, namely Difference Method (DIFF), Temporal Convolutional Neural Network (TCN), and Long Short-Term Memory Neural Network (LSTM). When applied to the Tingnan Coal Mine, Shanxi Province, China, the DIFF-TCN performs better in predicting the average daily water inflow, the model has a MAE of 5.88 m³/h, RMSE of 6.85 m³/h and R² of 0.96 in the test stage of the water inflow event. Comparison with the other deep learning models (with similar complex structures) and traditional time series model shows the superiority of our proposed DIFF-TCN model. The SHAP value is used to explain the contribution of each model input to the predicted values, and it indicates that the historical time of water inflow data are the most important input, and the advance distance and the groundwater level data also contribute to the model predictions, but groundwater level data for some periods in the past may have a detrimental effect on the model. The findings of this study can provide better understanding about potential of robust deep learning models for smart hydrological forecasting, and they can also provide technical guidance for mining safety production and protection of water resources and water environment around the mining area.

ProtInteract: A deep learning framework for predicting protein-protein interactions

Proteins mainly perform their functions by interacting with other proteins. Protein-protein interactions underpin various biological activities such as metabolic cycles, signal transduction, and immune response. However, due to the sheer number of proteins, experimental methods for finding interacting and non-interacting protein pairs are time-consuming and costly. We therefore developed the ProtInteract framework to predict protein-protein interaction. ProtInteract comprises two components: first, a novel autoencoder architecture that encodes each protein's primary structure to a lower-dimensional vector while preserving its underlying sequence attributes. This leads to faster training of the second network, a deep convolutional neural network (CNN) that receives encoded proteins and predicts their interaction under three different scenarios. In each scenario, the deep CNN predicts the class of a given encoded protein pair. Each class indicates different ranges of confidence scores corresponding to the probability of whether a predicted interaction occurs or not. The proposed framework features significantly low computational complexity and relatively fast response. The contributions of this work are twofold. First, ProtInteract assimilates the protein's primary structure into a pseudo-time series. Therefore, we leverage the nature of the time series of proteins and their physicochemical properties to encode a protein's amino acid sequence into a lower-dimensional vector space. This approach enables extracting highly informative sequence attributes while reducing computational complexity. Second, the ProtInteract framework utilises this information to identify protein interactions with other proteins based on its amino acid configuration. Our results suggest that the proposed framework performs with high accuracy and efficiency in predicting protein-protein interactions.

Predictive End-to-End Enterprise Process Network Monitoring

Ever-growing data availability combined with rapid progress in analytics has laid the foundation for the emergence of business process analytics. Organizations strive to leverage predictive process analytics to obtain insights. However, current implementations are designed to deal with homogeneous data. Consequently, there is limited practical use in an organization with heterogeneous data sources. The paper proposes a method for predictive end-to-end enterprise process network monitoring leveraging multi-headed deep neural networks to overcome this limitation. A case study performed with a medium-sized German manufacturing company highlights the method’s utility for organizations.

An explainable deep learning-based algorithm with an attention mechanism for predicting the live birth potential of mouse embryos

In assisted reproductive technology (ART), embryos produced by in vitro fertilization (IVF) are graded according to their live birth potential, and high-grade embryos are preferentially transplanted. However, rates of live birth following clinical ART remain low worldwide. Grading is based on the embryo shape at a limited number of stages and does not consider the shape of embryos and intracellular structures, e.g., nuclei, at various stages important for normal embryogenesis. Here, we developed a Normalized Multi-View Attention Network (NVAN) that directly predicts live birth potential from the nuclear structure in live-cell fluorescence images of mouse embryos from zygote to across a wide range of stages. The input is morphological features of cell nuclei, which were extracted as multivariate time-series data by using the segmentation algorithm for mouse embryos. The classification accuracy of our method (83.87%) greatly exceeded that of existing machine-learning methods and that of visual inspection by embryo culture specialists. Our method also has a new attention mechanism that allows us to determine which values of multivariate time-series data, used to describe nuclear morphology, were the basis for the prediction. By visualizing the features that contributed most to the prediction of live birth potential, we found that the size and shape of the nucleus at the morula stage and at the time of cell division were important for live birth prediction. We anticipate that our method will help ART and developmental engineering as a new basic technology for IVF embryo selection.

Optimal Compensation of MEMS Gyroscope Noise Kalman Filter Based on Conv-DAE and MultiTCN-Attention Model in Static Base Environment

Errors in microelectromechanical systems (MEMS) inertial measurement units (IMUs) are large, complex, nonlinear, and time varying. The traditional noise reduction and compensation methods based on traditional models are not applicable. This paper proposes a noise reduction method based on multi-layer combined deep learning for the MEMS gyroscope in the static base state. In this method, the combined model of MEMS gyroscope is constructed by Convolutional Denoising Auto-Encoder (Conv-DAE) and Multi-layer Temporal Convolutional Neural with the Attention Mechanism (MultiTCN-Attention) model. Based on the robust data processing capability of deep learning, the noise features are obtained from the past gyroscope data, and the parameter optimization of the Kalman filter (KF) by the Particle Swarm Optimization algorithm (PSO) significantly improves the filtering and noise reduction accuracy. The experimental results show that, compared with the original data, the noise standard deviation of the filtering effect of the combined model proposed in this paper decreases by 77.81% and 76.44% on the x and y axes, respectively; compared with the existing MEMS gyroscope noise compensation method based on the Autoregressive Moving Average with Kalman filter (ARMA-KF) model, the noise standard deviation of the filtering effect of the combined model proposed in this paper decreases by 44.00% and 46.66% on the x and y axes, respectively, reducing the noise impact by nearly three times.

Time-Series Forecasting of a CO2-EOR and CO2 Storage Project Using a Data-Driven Approach

This study aims to develop a predictive and reliable data-driven model for forecasting the fluid production (oil, gas, and water) of existing wells and future infill wells for CO2-enhanced oil recovery (EOR) and CO2 storage projects. Several models were investigated, such as auto-regressive (AR), multilayer perceptron (MLP), and long short-term memory (LSTM) networks. The models were trained based on static and dynamic parameters and daily fluid production while considering the inverse distance of neighboring wells. The developed models were evaluated using walk-forward validation and compared based on the quality metrics, span, and variation in the forecasting horizon. The AR model demonstrates a convincing generalization performance across various time series datasets with a long but varied forecasting horizon across eight wells. The LSTM model has a shorter forecasting horizon but strong generalizability and robustness in forecasting horizon consistency. MLP has the shortest and most varied forecasting horizon compared to the other models. The LSTM model exhibits promising performance in forecasting the fluid production of future infill wells when the model is developed from an existing well with similar features to an infill well. This study offers an alternative to the physics-driven model when traditional modeling is costly and laborious.

A Multivariate Temporal Convolutional Attention Network for Time-Series Forecasting

Multivariate time-series forecasting is one of the crucial and persistent challenges in time-series forecasting tasks. As a kind of data with multivariate correlation and volatility, multivariate time series impose highly nonlinear time characteristics on the forecasting model. In this paper, a new multivariate time-series forecasting model, multivariate temporal convolutional attention network (MTCAN), based on a self-attentive mechanism is proposed. MTCAN is based on the Convolution Neural Network (CNN) model, using 1D dilated convolution as the basic unit to construct asymmetric blocks, and then, the feature extraction is performed by the self-attention mechanism to finally obtain the prediction results. The input and output lengths of this network can be determined flexibly. The validation of the method is carried out with three different multivariate time-series datasets. The reliability and accuracy of the prediction results are compared with Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), Convolutional Long Short-Term Memory (ConvLSTM), and Temporal Convolutional Network (TCN). The prediction results show that the model proposed in this paper has significantly improved prediction accuracy and generalization.

A Fusion Transformer for Multivariable Time Series Forecasting: The Mooney Viscosity Prediction Case

Multivariable time series forecasting is an important topic of machine learning, and it frequently involves a complex mix of inputs, including static covariates and exogenous time series input. A targeted investigation of this input data is critical for improving prediction performance. In this paper, we propose the fusion transformer (FusFormer), a transformer-based model for forecasting time series data, whose framework fuses various computation modules for time series input and static covariates. To be more precise, the model calculation consists of two parallel stages. First, it employs a temporal encoder–decoder framework for extracting dynamic temporal features from time series data input, which analyzes and integrates the relative position information of sequence elements into the attention mechanism. Simultaneously, the static covariates are fed to the static enrichment module, which is inspired by gated linear units, to suppress irrelevant information and control the extent of nonlinear processing. Finally, the prediction results are calculated by fusing the outputs of the above two stages. Using Mooney viscosity forecasting as a case study, we demonstrate considerable forecasting performance improvements over existing methodologies and verify the effectiveness of each component of FusFormer via ablation analysis, and an interpretability use case is conducted to visualize temporal patterns of time series. The experimental results prove that FusFormer can achieve accurate Mooney viscosity prediction and improve the efficiency of the tire production process.

DeepExpress: Heterogeneous and Coupled Sequence Modeling for Express Delivery Prediction

The prediction of express delivery sequence, i.e., modeling and estimating the volumes of daily incoming and outgoing parcels for delivery, is critical for online business, logistics, and positive customer experience, and specifically for resource allocation optimization and promotional activity arrangement. A precise estimate of consumer delivery requests has to involve sequential factors such as shopping behaviors, weather conditions, events, business campaigns, and their couplings. Despite that various methods have integrated external features to enhance the effects, extant works fail to address complex feature-sequence couplings in the following aspects: weaken the inter-dependencies when processing heterogeneous data, and ignore the cumulative and evolving situation of coupling relationships. To address these issues, we propose DeepExpress - a deep-learning based express delivery sequence prediction model, which extends the classic seq2seq framework to learn feature-sequence couplings. DeepExpress leverages an express delivery seq2seq learning, a carefully-designed heterogeneous feature representation, and a novel joint training attention mechanism to adaptively handle heterogeneity issues, and capture feature-sequence couplings for accurate prediction. Experimental results on real-world data demonstrate that the proposed method outperforms both shallow and deep baseline models.

Neural Networks for Directed Connectivity Estimation in Source-Reconstructed EEG Data

Directed connectivity between brain sources identified from scalp electroencephalography (EEG) can shed light on the brain’s information flows and provide a biomarker of neurological disorders. However, as volume conductance results in scalp activity being a mix of activities originating from multiple sources, the correct interpretation of their connectivity is a formidable challenge despite source localization being applied with some success. Traditional connectivity approaches rely on statistical assumptions that usually do not hold for EEG, calling for a model-free approach. We investigated several types of Artificial Neural Networks in estimating Directed Connectivity between Reconstructed EEG Sources and assessed their accuracy with respect to several ground truths. We show that a Long Short-Term Memory neural network with Non-Uniform Embedding yields the most promising results due to its relative robustness to differing dipole locations. We conclude that certain network architectures can compete with the already established methods for brain connectivity analysis.

Prediction of Liquid Magnetization Series Data in Agriculture Based on Enhanced CGAN

The magnetized water and fertilizer liquid can produce biological effect of magnetic field on crops, but its residual magnetic field strength is difficult to be expressed quantitatively in real time, and accurate prediction of it is helpful to define the scope of action of liquid magnetization. In this paper, a prediction model for liquid magnetization series data is presented. It consists of conditional generative adversarial network (CGAN) and projected gradient descent (PGD) algorithm. First, the real training dataset is used as the input of PGD attack algorithm to generate antagonistic samples. These samples are added to the training of CGAN as true samples for data enhancement. Second, the training dataset is used as both the generator and discriminator input of CGAN to constrain the model, capture distribution of the real data. Third, a network model with three layers of CNN is built and trained inside CGAN. The input model is constructed by using the structure of two-dimensional convolution model to predict data. Lastly, the performance of the model is evaluated by the error between the final generated predicted value and the real value, and the model is compared with other prediction models. The experimental results show that, with limited data samples, by combining PGD attack with CGAN, the distribution of the real data can be more accurately captured and the data can be generated to meet the actual needs.

News Sentiment Informed Time-series Analyzing AI (SITALA) to curb the spread of COVID-19 in Houston

Coronavirus disease (COVID-19) has evolved into a pandemic with many unknowns. Houston, located in the Harris County of Texas, is becoming the next hotspot of this pandemic. With a severe decline in international and inter-state travel, a model at the county level is needed as opposed to the state or country level. Existing approaches have a few drawbacks. Firstly, the data used is the number of COVID-19 positive cases instead of positivity. The former is a function of the number of tests carried out while the number of tests normalizes the latter. Positivity gives a better picture of the spread of this pandemic as, with time, more tests are being administered. Positivity under 5% has been desired for the reopening of businesses to almost 100% capacity. Secondly, the data used by models like SEIRD (Susceptible, Exposed, Infectious, Recovered, and Deceased) lacks information about the sentiment of people concerning coronavirus. Thirdly, models that make use of social media posts might have too much noise and misinformation. On the other hand, news sentiment can capture long-term effects of hidden variables like public policy, opinions of local doctors, and disobedience of state-wide mandates. The present study introduces a new artificial intelligence (i.e., AI) model, viz., Sentiment Informed Time-series Analyzing AI (SITALA), trained on COVID-19 test positivity data and news sentiment from over 2750 news articles for Harris county. The news sentiment was obtained using IBM Watson Discovery News. SITALA is inspired by Google-Wavenet architecture and makes use of TensorFlow. The mean absolute error for the training dataset of 66 consecutive days is 2.76, and that for the test dataset of 22 consecutive days is 9.6. A cone of uncertainty is provided within which future COVID-19 test positivity has been shown to fall with high accuracy. The model predictions fare better than a published Bayesian-based SEIRD model. The model forecasts that in order to curb the spread of coronavirus in Houston, a sustained negative news sentiment (e.g., death count for COVID-19 will grow at an alarming rate in Houston if mask orders are not followed) will be desirable. Public policymakers may use SITALA to set the tone of the local policies and mandates.

A Deep Learning-Based Classification Scheme for False Data Injection Attack Detection in Power System

A smart grid improves power grid efficiency by using modern information and communication technologies. However, at the same time, due to the dependence on information technology and the deep integration of electrical components and computing information in cyber space, the system might become increasingly vulnerable to cyber-attacks. Among various emerging security problems, a false data injection attack (FDIA) is a new type of attack against the state estimation. In this article, a deep learning-based identification scheme is developed to detect and mitigate information corruption. The scheme implements a conditional deep belief network (CDBN) to analyze time-series input data and leverages captured features to detect the FDIA. The performance of our detection mechanism is validated by using the IEEE 14-bus test system for simulation. Different attack scenarios and parameters are set to demonstrate the feasibility and effectiveness of the developed scheme. Compared with the artificial neural network (ANN) and the support vector machine (SVM), the experimental analyses indicate that the results of our detection mechanism are better than those of the other two in terms of FDIA detection accuracy and robustness.

CrashNet: an encoder–decoder architecture to predict crash test outcomes

Destructive car crash tests are an elaborate, time-consuming, and expensive necessity of the automotive development process. Today, finite element method (FEM) simulations are used to reduce costs by simulating car crashes computationally. We propose CrashNet, an encoder–decoder deep neural network architecture that reduces costs further and models specific outcomes of car crashes very accurately. We achieve this by formulating car crash events as time series prediction enriched with a set of scalar features. Traditional sequence-to-sequence models are usually composed of convolutional neural network (CNN) and CNN transpose layers. We propose to concatenate those with an MLP capable of learning how to inject the given scalars into the output time series. In addition, we replace the CNN transpose with 2D CNN transpose layers in order to force the model to process the hidden state of the set of scalars as one time series. The proposed CrashNet model can be trained efficiently and is able to process scalars and time series as input in order to infer the results of crash tests. CrashNet produces results faster and at a lower cost compared to destructive tests and FEM simulations. Moreover, it represents a novel approach in the car safety management domain.

Machine Learning-Based Algorithms to Knowledge Extraction from Time Series Data: A Review

To predict the future behavior of a system, we can exploit the information collected in the past, trying to identify recurring structures in what happened to predict what could happen, if the same structures repeat themselves in the future as well. A time series represents a time sequence of numerical values observed in the past at a measurable variable. The values are sampled at equidistant time intervals, according to an appropriate granular frequency, such as the day, week, or month, and measured according to physical units of measurement. In machine learning-based algorithms, the information underlying the knowledge is extracted from the data themselves, which are explored and analyzed in search of recurring patterns or to discover hidden causal associations or relationships. The prediction model extracts knowledge through an inductive process: the input is the data and, possibly, a first example of the expected output, the machine will then learn the algorithm to follow to obtain the same result. This paper reviews the most recent work that has used machine learning-based techniques to extract knowledge from time series data.

A Novel Recursive Model Based on a Convolutional Long Short-Term Memory Neural Network for Air Pollution Prediction

Deep learning provides a promising approach for air pollution prediction. The existing deep learning-based predicted models generally consider either the temporal correlations of air quality monitoring stations or the nonlinear relationship between the PM2.5 (particulate matter with an aerodynamic diameter of less than 2.5 μm) concentrations and explanatory variables. Spatial correlation has not been effectively incorporated into prediction models, therefore exhibiting poor performance in PM2.5 prediction tasks. Additionally, determining the manner by which to expand longer-term prediction tasks is still challenging. In this paper, to allow for spatiotemporal correlations, a spatiotemporal convolutional recursive long short-term memory (CR-LSTM) neural network model is proposed for predicting the PM2.5 concentrations in long-term prediction tasks by combining a convolutional long short-term memory (ConvLSTM) neural network and a recursive strategy. Herein, the ConvLSTM network was used to capture the complex spatiotemporal correlations and to predict the future PM2.5 concentrations; the recursive strategy was used for expanding the long-term prediction tasks. The CR-LSTM model was used to realize the prediction of the future 24 h of PM2.5 concentrations for 12 air quality monitoring stations in Beijing by configuring both the appropriate time lag derived from the temporal correlations and the spatial neighborhood, including the hourly historical PM2.5 concentrations, the daily mean meteorological data, and the annual nighttime light and normalized difference vegetation index (NDVI). The results showed that the proposed CR-LSTM model achieved better performance (coefficient of determination (R2) = 0.74; root mean square error (RMSE) = 18.96 μg/m3) than other common models, such as multiple linear regression (MLR), support vector regression (SVR), the conventional LSTM model, the LSTM extended (LSTME) model, and the temporal sliding LSTM extended (TS-LSTME) model. The proposed CR-LSTM model, implementing a combination of geographical rules, recursive strategy, and deep learning, shows improved performance in longer-term prediction tasks.

Transfer learning driven sequential forecasting and ventilation control of PM2.5 associated health risk levels in underground public facilities

Particulate matter with aerodynamic diameter less than 2.5 µm (PM_2.5) has become a major public concern in closed indoor environments, such as subway stations. Forecasting platform PM_2.5 concentrations is significant in developing early warning systems, and regulating ventilation systems to ensure commuter health. However, the performance of existing forecasting approaches relies on a considerable amount of historical sensor data, which is usually not available in practical situations due to hostile monitoring environments or newly installed equipment. Transfer learning (TL) provides a solution to the scant data problem, as it leverages the knowledge learned from well-measured subway stations to facilitate predictions on others. This paper presents a TL-based residual neural network framework for sequential forecast of health risk levels traced by subway platform PM_2.5 levels. Experiments are conducted to investigate the potential of the proposed methodology under different data availability scenarios. The TL-framework outperforms the RNN structures with a determination coefficient (R²) improvement of 42.84%, and in comparison, to stand-alone models the prediction errors (RMSE) are reduced up to 40%. Additionally, the forecasted data by TL-framework under limited data scenario allowed the ventilation system to maintain IAQ at healthy levels, and reduced PM_2.5 concentrations by 29.21% as compared to stand-alone network.

An Experimental Review on Deep Learning Architectures for Time Series Forecasting

In recent years, deep learning techniques have outperformed traditional models in many machine learning tasks. Deep neural networks have successfully been applied to address time series forecasting problems, which is a very important topic in data mining. They have proved to be an effective solution given their capacity to automatically learn the temporal dependencies present in time series. However, selecting the most convenient type of deep neural network and its parametrization is a complex task that requires considerable expertise. Therefore, there is a need for deeper studies on the suitability of all existing architectures for different forecasting tasks. In this work, we face two main challenges: a comprehensive review of the latest works using deep learning for time series forecasting and an experimental study comparing the performance of the most popular architectures. The comparison involves a thorough analysis of seven types of deep learning models in terms of accuracy and efficiency. We evaluate the rankings and distribution of results obtained with the proposed models under many different architecture configurations and training hyperparameters. The datasets used comprise more than 50,000 time series divided into 12 different forecasting problems. By training more than 38,000 models on these data, we provide the most extensive deep learning study for time series forecasting. Among all studied models, the results show that long short-term memory (LSTM) and convolutional networks (CNN) are the best alternatives, with LSTMs obtaining the most accurate forecasts. CNNs achieve comparable performance with less variability of results under different parameter configurations, while also being more efficient.

An Advanced CNN-LSTM Model for Cryptocurrency Forecasting

Nowadays, cryptocurrencies are established and widely recognized as an alternative exchange currency method. They have infiltrated most financial transactions and as a result cryptocurrency trade is generally considered one of the most popular and promising types of profitable investments. Nevertheless, this constantly increasing financial market is characterized by significant volatility and strong price fluctuations over a short-time period therefore, the development of an accurate and reliable forecasting model is considered essential for portfolio management and optimization. In this research, we propose a multiple-input deep neural network model for the prediction of cryptocurrency price and movement. The proposed forecasting model utilizes as inputs different cryptocurrency data and handles them independently in order to exploit useful information from each cryptocurrency separately. An extensive empirical study was performed using three consecutive years of cryptocurrency data from three cryptocurrencies with the highest market capitalization i.e., Bitcoin (BTC), Etherium (ETH), and Ripple (XRP). The detailed experimental analysis revealed that the proposed model has the ability to efficiently exploit mixed cryptocurrency data, reduces overfitting and decreases the computational cost in comparison with traditional fully-connected deep neural networks.

Industrial Facility Electricity Consumption Forecast Using Artificial Neural Networks and Incremental Learning

Society’s concerns with electricity consumption have motivated researchers to improve on the way that energy consumption management is done. The reduction of energy consumption and the optimization of energy management are, therefore, two major aspects to be considered. Additionally, load forecast provides relevant information with the support of historical data allowing an enhanced energy management, allowing energy costs reduction. In this paper, the proposed consumption forecast methodology uses an Artificial Neural Network (ANN) and incremental learning to increase the forecast accuracy. The ANN is retrained daily, providing an updated forecasting model. The case study uses 16 months of data, split in 5-min periods, from a real industrial facility. The advantages of using the proposed method are illustrated with the numerical results.

Spatio-Temporal Abnormal Behavior Prediction in Elderly Persons Using Deep Learning Models

The ability to identify and accurately predict abnormal behavior is important for health monitoring systems in smart environments. Specifically, for elderly persons wishing to maintain their independence and comfort in their living spaces, abnormal behaviors observed during activities of daily living are a good indicator that the person is more likely to have health and behavioral problems that need intervention and assistance. In this paper, we investigate a variety of deep learning models such as Long Short Term Memory (LSTM), Convolutional Neural Network (CNN), CNN-LSTM and Autoencoder-CNN-LSTM for identifying and accurately predicting the abnormal behaviors of elderly people. The temporal information and spatial sequences collected over time are used to generate models, which can be fitted to the training data and the fitted model can be used to make a prediction. We present an experimental evaluation of these models performance in identifying and predicting elderly persons abnormal behaviors in smart homes, via extensive testing on two public data sets, taking into account different models architectures and tuning the hyperparameters for each model. The performance evaluation is focused on accuracy measure.

Short-Term Photovoltaic Power Forecasting Using a Convolutional Neural Network–Salp Swarm Algorithm

The high utilization of renewable energy to manage climate change and provide green energy requires short-term photovoltaic (PV) power forecasting. In this paper, a novel forecasting strategy that combines a convolutional neural network (CNN) and a salp swarm algorithm (SSA) is proposed to forecast PV power output. First, the historical PV power data and associated weather information are classified into five weather types, such as rainy, heavy cloudy, cloudy, light cloudy and sunny. The CNN classification is then used to determine the prediction for the next day’s weather type. Five models of CNN regression are established to accommodate the prediction for different weather types. Each CNN regression is optimized using a salp swarm algorithm (SSA) to tune the best parameter. To evaluate the performance of the proposed method, comparisons were made to the SSA based support vector machine (SVM-SSA) and long short-term memory neural network (LSTM-SSA) methods. The proposed method was tested on a PV power generation system with a 500 kWp capacity located in south Taiwan. The results showed that the proposed CNN-SSA could accommodate the actual generation pattern better than the SVM-SSA and LSTM-SSA methods.

Multi-Step Short-Term Wind Speed Prediction Using a Residual Dilated Causal Convolutional Network with Nonlinear Attention

Wind energy is the most used renewable energy worldwide second only to hydropower. However, the stochastic nature of wind speed makes it harder for wind farms to manage the future power production and maintenance schedules efficiently. Many wind speed prediction models exist that focus on advance neural networks and/or preprocessing techniques to improve the accuracy. Since most of these models require a large amount of historic wind data and are validated using the data split method, the application to real-world scenarios cannot be determined. In this paper, we present a multi-step univariate prediction model for wind speed data inspired by the residual U-net architecture of the convolutional neural network (CNN). We propose a residual dilated causal convolutional neural network (Res-DCCNN) with nonlinear attention for multi-step-ahead wind speed forecasting. Our model can outperform long-term short-term memory networks (LSTM), gated recurrent units (GRU), and Res-DCCNN using sliding window validation techniques for 50-step-ahead wind speed prediction. We tested the performance of the proposed model on six real-world wind speed datasets with different probability distributions to confirm its effectiveness, and using several error metrics, we demonstrated that our proposed model was robust, precise, and applicable to real-world cases.

Temporal Convolutional Networks Applied to Energy-Related Time Series Forecasting

Modern energy systems collect high volumes of data that can provide valuable information about energy consumption. Electric companies can now use historical data to make informed decisions on energy production by forecasting the expected demand. Many deep learning models have been proposed to deal with these types of time series forecasting problems. Deep neural networks, such as recurrent or convolutional, can automatically capture complex patterns in time series data and provide accurate predictions. In particular, Temporal Convolutional Networks (TCN) are a specialised architecture that has advantages over recurrent networks for forecasting tasks. TCNs are able to extract long-term patterns using dilated causal convolutions and residual blocks, and can also be more efficient in terms of computation time. In this work, we propose a TCN-based deep learning model to improve the predictive performance in energy demand forecasting. Two energy-related time series with data from Spain have been studied: the national electric demand and the power demand at charging stations for electric vehicles. An extensive experimental study has been conducted, involving more than 1900 models with different architectures and parametrisations. The TCN proposal outperforms the forecasting accuracy of Long Short-Term Memory (LSTM) recurrent networks, which are considered the state-of-the-art in the field.

Deep Learning Applications with Practical Measured Results in Electronics Industries

This editorial introduces the Special Issue, entitled “Deep Learning Applications with Practical Measured Results in Electronics Industries”, of Electronics. Topics covered in this issue include four main parts: (I) environmental information analyses and predictions, (II) unmanned aerial vehicle (UAV) and object tracking applications, (III) measurement and denoising techniques, and (IV) recommendation systems and education systems. Four papers on environmental information analyses and predictions are as follows: (1) “A Data-Driven Short-Term Forecasting Model for Offshore Wind Speed Prediction Based on Computational Intelligence” by Panapakidis et al.; (2) “Multivariate Temporal Convolutional Network: A Deep Neural Networks Approach for Multivariate Time Series Forecasting” by Wan et al.; (3) “Modeling and Analysis of Adaptive Temperature Compensation for Humidity Sensors” by Xu et al.; (4) “An Image Compression Method for Video Surveillance System in Underground Mines Based on Residual Networks and Discrete Wavelet Transform” by Zhang et al. Three papers on UAV and object tracking applications are as follows: (1) “Trajectory Planning Algorithm of UAV Based on System Positioning Accuracy Constraints” by Zhou et al.; (2) “OTL-Classifier: Towards Imaging Processing for Future Unmanned Overhead Transmission Line Maintenance” by Zhang et al.; (3) “Model Update Strategies about Object Tracking: A State of the Art Review” by Wang et al. Five papers on measurement and denoising techniques are as follows: (1) “Characterization and Correction of the Geometric Errors in Using Confocal Microscope for Extended Topography Measurement. Part I: Models, Algorithms Development and Validation” by Wang et al.; (2) “Characterization and Correction of the Geometric Errors Using a Confocal Microscope for Extended Topography Measurement, Part II: Experimental Study and Uncertainty Evaluation” by Wang et al.; (3) “Deep Transfer HSI Classification Method Based on Information Measure and Optimal Neighborhood Noise Reduction” by Lin et al.; (4) “Quality Assessment of Tire Shearography Images via Ensemble Hybrid Faster Region-Based ConvNets” by Chang et al.; (5) “High-Resolution Image Inpainting Based on Multi-Scale Neural Network” by Sun et al. Two papers on recommendation systems and education systems are as follows: (1) “Deep Learning-Enhanced Framework for Performance Evaluation of a Recommending Interface with Varied Recommendation Position and Intensity Based on Eye-Tracking Equipment Data Processing” by Sulikowski et al. and (2) “Generative Adversarial Network Based Neural Audio Caption Model for Oral Evaluation” by Zhang et al.