Deep Learning for Time Series Forecasting: A Survey

Machine learning model with output correction: Towards reliable bradycardia detection in neonates

Bradycardia is a commonly occurring condition in premature infants, often causing serious consequences and cardiovascular complications. Reliable and accurate detection of bradycardia events is pivotal for timely intervention and effective treatment. Excessive false alarms pose a critical problem in bradycardia event detection, eroding trust in machine learning (ML)-based clinical decision support tools designed for such detection. This could result in disregarding the algorithm's accurate recommendations and disrupting workflows, potentially compromising the quality of patient care. This article introduces an ML-based approach incorporating an output correction element, designed to minimise false alarms. The approach has been applied to bradycardia detection in preterm infants. We applied five ML-based autoencoder techniques, using recurrent neural network (RNN), long-short-term memory (LSTM), gated recurrent unit (GRU), 1D convolutional neural network (1D CNN), and a combination of 1D CNN and LSTM. The analysis is performed on ∼440 hours of real-time preterm infant data. The proposed approach achieved 0.978, 0.73, 0.992, 0.671 and 0.007 in AUC-ROC, AUC-PRC, recall, F1 score, and false positive rate (FPR) respectively and a false alarms reduction of 36% when compared with methods without the correction approach. This study underscores the imperative of cultivating solutions that alleviate alarm fatigue and encourage active engagement among healthcare professionals.

Machine Learning: A Potential Therapeutic Tool to Facilitate Neonatal Therapeutic Decision Making

Bacterial infection is one of the major causes of neonatal morbidity and mortality worldwide. Finding rapid and reliable methods for early recognition and diagnosis of bacterial infections and early individualization of antibacterial drug administration are essential to eradicate these infections and prevent serious complications. However, this is often difficult to perform due to non-specific clinical presentations, low accuracy of current diagnostic methods, and limited knowledge of neonatal pharmacokinetics. Although neonatal medicine has been relatively late to embrace the benefits of machine learning (ML), there have been some initial applications of ML for the early prediction of neonatal sepsis and individualization of antibiotics. This article provides a brief introduction to ML and discusses the current state of the art in diagnosing and treating neonatal bacterial infections, gaps, potential uses of ML, and future directions to address the limitations of current studies. Neonatal bacterial infections involve a combination of physiologic development, disease expression, and treatment response outcomes. To address this complex relationship, future models could consider appropriate ML algorithms to capture time series features while integrating influences from the host, microbes, and drugs to optimize antimicrobial drug use in neonates. All models require prospective clinical trials to validate their clinical utility before clinical use.

Application of Residual Structure Time Convolutional Network Based on Attention Mechanism in Remaining Useful Life Interval Prediction of Bearings

In the context of Industry 4.0, bearings, as critical components of machinery, play a vital role in ensuring operational reliability. The detection of their health status is thus of paramount importance. Existing predictive models often focus on point predictions of bearing lifespan, lacking the ability to quantify uncertainty and having room for improvement in accuracy. To accurately predict the long-term remaining useful life (RUL) of bearings, a novel time convolutional network model with an attention mechanism-based soft thresholding decision residual structure for quantifying the lifespan interval of bearings, namely TCN-AM-GPR, is proposed. Firstly, a spatio-temporal graph is constructed from the bearing sensor signals as the input to the prediction model. Secondly, a residual structure based on a soft threshold decision with a self-attention mechanism is established to further suppress noise in the collected bearing lifespan signals. Thirdly, the extracted features pass through an interval quantization layer to obtain the RUL and its confidence interval of the bearings. The proposed methodology has been verified using the PHM2012 bearing dataset, and the comparison of simulation experiment results shows that TCN-AM-GPR achieved the best point prediction evaluation index, with a 2.17% improvement in R2 compared to the second-best performance from TCN-GPR. At the same time, it also has the best interval prediction comprehensive evaluation index, with a relative decrease of 16.73% in MWP compared to the second-best performance from TCN-GPR. The research results indicate that TCN-AM-GPR can ensure the accuracy of point estimates, while having superior advantages and practical significance in describing prediction uncertainty.

Harnessing LSTM and XGBoost algorithms for storm prediction

Storms can cause significant damage, severe social disturbance and loss of human life, but predicting them is challenging due to their infrequent occurrence. To overcome this problem, a novel deep learning and machine learning approach based on long short-term memory (LSTM) and Extreme Gradient Boosting (XGBoost) was applied to predict storm characteristics and occurrence in Western France. A combination of data from buoys and a storm database between 1996 and 2020 was processed for model training and testing. The models were trained and validated with the dataset from January 1996 to December 2015 and the trained models were then used to predict storm characteristics and occurrence from January 2016 to December 2020. The LSTM model used to predict storm characteristics showed great accuracy in forecasting temperature and pressure, with challenges observed in capturing extreme values for wave height and wind speed. The trained XGBoost model, on the other hand, performed extremely well in predicting storm occurrence. The methodology adopted can help reduce the impact of storms on humans and objects.

Glu-Ensemble: An ensemble deep learning framework for blood glucose forecasting in type 2 diabetes patients

Diabetes is a chronic metabolic disorder characterized by elevated blood glucose levels, posing significant health risks such as cardiovascular disease, and nerve, kidney, and eye damage. Effective management of blood glucose is essential for individuals with diabetes to mitigate these risks. This study introduces the Glu-Ensemble, a deep learning framework designed for precise blood glucose forecasting in patients with type 2 diabetes. Unlike other predictive models, Glu-Ensemble addresses challenges related to small sample sizes, data quality issues, reliance on strict statistical assumptions, and the complexity of models. It enhances prediction accuracy and model generalizability by utilizing larger datasets and reduces bias inherent in many predictive models. The framework's unified approach, as opposed to patient-specific models, eliminates the need for initial calibration time, facilitating immediate blood glucose predictions for new patients. The obtained results indicate that Glu-Ensemble surpasses traditional methods in accuracy, as measured by root mean square error, mean absolute error, and error grid analysis. The Glu-Ensemble framework emerges as a promising tool for blood glucose level prediction in type 2 diabetes patients, warranting further investigation in clinical settings for its practical application.

Forecasting with an N-dimensional Langevin equation and a neural-ordinary differential equation

Accurate prediction of electricity day-ahead prices is essential in competitive electricity markets. Although stationary electricity-price forecasting techniques have received considerable attention, research on non-stationary methods is comparatively scarce, despite the common prevalence of non-stationary features in electricity markets. Specifically, existing non-stationary techniques will often aim to address individual non-stationary features in isolation, leaving aside the exploration of concurrent multiple non-stationary effects. Our overarching objective here is the formulation of a framework to systematically model and forecast non-stationary electricity-price time series, encompassing the broader scope of non-stationary behavior. For this purpose, we develop a data-driven model that combines an N-dimensional Langevin equation (LE) with a neural-ordinary differential equation (NODE). The LE captures fine-grained details of the electricity-price behavior in stationary regimes but is inadequate for non-stationary conditions. To overcome this inherent limitation, we adopt a NODE approach to learn, and at the same time predict, the difference between the actual electricity-price time series and the simulated price trajectories generated by the LE. By learning this difference, the NODE reconstructs the non-stationary components of the time series that the LE is not able to capture. We exemplify the effectiveness of our framework using the Spanish electricity day-ahead market as a prototypical case study. Our findings reveal that the NODE nicely complements the LE, providing a comprehensive strategy to tackle both stationary and non-stationary electricity-price behavior. The framework's dependability and robustness is demonstrated through different non-stationary scenarios by comparing it against a range of basic naïve methods.

Thin-film transistor for temporal self-adaptive reservoir computing with closed-loop architecture

Reservoir computing is a powerful neural network-based computing paradigm for spatiotemporal signal processing. Recently, physical reservoirs have been explored based on various electronic devices with outstanding efficiency. However, the inflexible temporal dynamics of these reservoirs have posed fundamental restrictions in processing spatiotemporal signals with various timescales. Here, we fabricated thin-film transistors with controllable temporal dynamics, which can be easily tuned with electrical operation signals and showed excellent cycle-to-cycle uniformity. Based on this, we constructed a temporal adaptive reservoir capable of extracting temporal information of multiple timescales, thereby achieving improved accuracy in the human-activity-recognition task. Moreover, by leveraging the former computing output to modify the hyperparameters, we constructed a closed-loop architecture that equips the reservoir computing system with temporal self-adaptability according to the current input. The adaptability is demonstrated by accurate real-time recognition of objects moving at diverse speed levels. This work provides an approach for reservoir computing systems to achieve real-time processing of spatiotemporal signals with compound temporal characteristics.

Long- and Short-Term Memory Model of Cotton Price Index Volatility Risk Based on Explainable Artificial Intelligence

Market uncertainty greatly interferes with the decisions and plans of market participants, thus increasing the risk of decision-making, leading to compromised interests of decision-makers. Cotton price index (hereinafter referred to as cotton price) volatility is highly noisy, nonlinear, and stochastic and is susceptible to supply and demand, climate, substitutes, and other policy factors, which are subject to large uncertainties. To reduce decision risk and provide decision support for policymakers, this article integrates 13 factors affecting cotton price index volatility based on existing research and further divides them into transaction data and interaction data. A long- and short-term memory (LSTM) model is constructed, and a comparison experiment is implemented to analyze the cotton price index volatility. To make the constructed model explainable, we use explainable artificial intelligence (XAI) techniques to perform statistical analysis of the input features. The experimental results show that the LSTM model can accurately analyze the cotton price index fluctuation trend but cannot accurately predict the actual price of cotton; the transaction data plus interaction data are more sensitive than the transaction data in analyzing the cotton price fluctuation trend and can have a positive effect on the cotton price fluctuation analysis. This study can accurately reflect the fluctuation trend of the cotton market, provide reference to the state, enterprises, and cotton farmers for decision-making, and reduce the risk caused by frequent fluctuation of cotton prices. The analysis of the model using XAI techniques builds the confidence of decision-makers in the model.

ECG Forecasting System Based on Long Short-Term Memory

Worldwide, cardiovascular diseases are some of the primary causes of death; yet the early detection and diagnosis of such diseases have the potential to save many lives. Technological means of detection are becoming increasingly essential and numerous techniques have been created for this purpose, such as forecasting. Of these techniques, the time series forecasting technique seeks to predict future events. The long-term time series forecasting of physiological data could assist medical professionals in predicting and treating patients based on very early diagnosis. This article presents a model that utilizes a deep learning technique to predict long-term ECG signals. The forecasting model can learn signals' nonlinearity, nonstationarity, and complexity based on a long short-term memory architecture. However, this is not a trivial task as the correct forecasting of a signal that closely resembles the original complex signal's structure and behavior while minimizing any differences in amplitude continues to pose challenges. To achieve this goal, we used a dataset available on the Physio net database, called MIT-BIH, with 48 ECG recordings of 30 min each. The developed model starts with pre-processing to reduce interference in the original signals, then applies a deep learning algorithm, based on a long short-term memory (LTSM) neural network with two hidden layers. Next, we applied the root mean square error (RMSE) and mean absolute error (MAE) metrics to evaluate the performance of the model and obtained an average RMSE of 0.0070±0.0028 and an average MAE of 0.0522±0.0098 across all simulations. The results indicate that the proposed LSTM model is a promising technique for ECG forecasting, considering the trends of the changes in the original data series, most notably in R-peak amplitude. Given the model's accuracy and the features of the physiological signals, the system could be used to improve existing predictive healthcare systems for cardiovascular monitoring.

Green IoT Event Detection for Carbon-Emission Monitoring in Sensor Networks

This research addresses the intersection of low-power microcontroller technology and binary classification of events in the context of carbon-emission reduction. The study introduces an innovative approach leveraging microcontrollers for real-time event detection in a homogeneous hardware/firmware manner and faced with limited resources. This showcases their efficiency in processing sensor data and reducing power consumption without the need for extensive training sets. Two case studies focusing on landfill CO2 emissions and home energy usage demonstrate the feasibility and effectiveness of this approach. The findings highlight significant power savings achieved by minimizing data transmission during non-event periods (94.8-99.8%), in addition to presenting a sustainable alternative to traditional resource-intensive AI/ML platforms that comparatively draw and produce 20,000 times the amount of power and carbon emissions, respectively.

Utilizing Deep Learning Algorithms for Signal Processing in Electrochemical Biosensors: From Data Augmentation to Detection and Quantification of Chemicals of Interest

Nanomaterial-based aptasensors serve as useful instruments for detecting small biological entities. This work utilizes data gathered from three electrochemical aptamer-based sensors varying in receptors, analytes of interest, and lengths of signals. Our ultimate objective was the automatic detection and quantification of target analytes from a segment of the signal recorded by these sensors. Initially, we proposed a data augmentation method using conditional variational autoencoders to address data scarcity. Secondly, we employed recurrent-based networks for signal extrapolation, ensuring uniform signal lengths. In the third step, we developed seven deep learning classification models (GRU, unidirectional LSTM (ULSTM), bidirectional LSTM (BLSTM), ConvGRU, ConvULSTM, ConvBLSTM, and CNN) to identify and quantify specific analyte concentrations for six distinct classes, ranging from the absence of analyte to 10 μM. Finally, the second classification model was created to distinguish between abnormal and normal data segments, detect the presence or absence of analytes in the sample, and, if detected, identify the specific analyte and quantify its concentration. Evaluating the time series forecasting showed that the GRU-based network outperformed two other ULSTM and BLSTM networks. Regarding classification models, it turned out signal extrapolation was not effective in improving the classification performance. Comparing the role of the network architectures in classification performance, the result showed that hybrid networks, including both convolutional and recurrent layers and CNN networks, achieved 82% to 99% accuracy across all three datasets. Utilizing short-term Fourier transform (STFT) as the preprocessing technique improved the performance of all datasets with accuracies from 84% to 99%. These findings underscore the effectiveness of suitable data preprocessing methods in enhancing neural network performance, enabling automatic analyte identification and quantification from electrochemical aptasensor signals.

Assessing the impacts of climate change on streamflow dynamics: A machine learning perspective

This study investigates changes in river flow patterns, in the Hunza Basin, Pakistan, attributed to climate change. Given the anticipated rise in extreme weather events, accurate streamflow predictions are increasingly vital. We assess three machine learning (ML) models - artificial neural network (ANN), recurrent neural network (RNN), and adaptive fuzzy neural inference system (ANFIS) - for streamflow prediction under the Coupled Model Intercomparison Project 6 (CMIP6) Shared Socioeconomic Pathways (SSPs), specifically SSP245 and SSP585. Four key performance indicators, mean square error (MSE), root mean square error (RMSE), mean absolute error (MAE), and coefficient of determination (R2), guide the evaluation. These models employ monthly precipitation, maximum and minimum temperatures as inputs, and discharge as the output, spanning 1985-2014. The ANN model with a 3-10-1 architecture outperforms RNN and ANFIS, displaying lower MSE, RMSE, MAE, and higher R2 values for both training (MSE = 20417, RMSE = 142, MAE = 71, R2 = 0.94) and testing (MSE = 9348, RMSE = 96, MAE = 108, R2 = 0.92) datasets. Subsequently, the superior ANN model predicts streamflow up to 2100 using SSP245 and SSP585 scenarios. These results underscore the potential of ANN models for robust futuristic streamflow estimation, offering valuable insights for water resource management and planning.

Cycle-level traffic conflict prediction at signalized intersections with LiDAR data and Bayesian deep learning

Real-time safety prediction models are vital in proactive road safety management strategies. This study develops models to predict traffic conflicts at signalized intersections at the signal cycle level, using advanced Bayesian deep learning techniques and efficient LiDAR points. The modeling framework contains three phases, which are data preprocessing, base deep learning model development, and Bayesian deep learning model development. The core of the framework is the long short-term memory (LSTM) employed to predict the conflict frequency of a cycle by using traffic features of the previous five cycles (e.g., dynamic traffic parameters, traffic conflict frequency). Four Bayesian deep learning models were developed, including Bayesian-Standard LSTM, Bayesian-Hybrid-LSTM, Bayesian-Stacked-LSTM Encoder-Decoder, and Bayesian-Multi-head Stacked-LSTM Encoder-Decoder. The developed models were applied to traffic conflicts extracted from LiDAR points that were collected from a signalized intersection in Harbin, China with a total duration of seven days. Traffic conflicts, measured by the modified time-to-collision conflict indicator, were identified using the peak over threshold approach. The models were thoroughly evaluated from the aspects of reliability, transferability, sensitivity, and robustness. The results show that the developed four models can predict traffic conflict frequency per cycle per lane simultaneously with its uncertainty. Moreover, the two Bayesian encoder-decoder models perform better than Bayesian-Standard LSTM and Bayesian-Hybrid-LSTM in the four tests. Bayesian-Multi-head Stacked-LSTM Encoder-Decoder is suggested as the optimal model for its high reliability under uncertainty, good transferability in three scenarios, low sensitivity to different parameters, and sound robustness against small noise. The proposed framework could benefit studies on the state-of-the-art data-driven approach for real-time safety prediction.

Continual Deep Learning for Time Series Modeling

The multi-layer structures of Deep Learning facilitate the processing of higher-level abstractions from data, thus leading to improved generalization and widespread applications in diverse domains with various types of data. Each domain and data type presents its own set of challenges. Real-world time series data may have a non-stationary data distribution that may lead to Deep Learning models facing the problem of catastrophic forgetting, with the abrupt loss of previously learned knowledge. Continual learning is a paradigm of machine learning to handle situations when the stationarity of the datasets may no longer be true or required. This paper presents a systematic review of the recent Deep Learning applications of sensor time series, the need for advanced preprocessing techniques for some sensor environments, as well as the summaries of how to deploy Deep Learning in time series modeling while alleviating catastrophic forgetting with continual learning methods. The selected case studies cover a wide collection of various sensor time series applications and can illustrate how to deploy tailor-made Deep Learning, advanced preprocessing techniques, and continual learning algorithms from practical, real-world application aspects.

Deep learning for deterioration prediction of COVID-19 patients based on time-series of three vital signs

Unrecognized deterioration of COVID-19 patients can lead to high morbidity and mortality. Most existing deterioration prediction models require a large number of clinical information, typically collected in hospital settings, such as medical images or comprehensive laboratory tests. This is infeasible for telehealth solutions and highlights a gap in deterioration prediction models based on minimal data, which can be recorded at a large scale in any clinic, nursing home, or even at the patient's home. In this study, we develop and compare two prognostic models that predict if a patient will experience deterioration in the forthcoming 3 to 24 h. The models sequentially process routine triadic vital signs: (a) oxygen saturation, (b) heart rate, and (c) temperature. These models are also provided with basic patient information, including sex, age, vaccination status, vaccination date, and status of obesity, hypertension, or diabetes. The difference between the two models is the way that the temporal dynamics of the vital signs are processed. Model #1 utilizes a temporally-dilated version of the Long-Short Term Memory model (LSTM) for temporal processes, and Model #2 utilizes a residual temporal convolutional network (TCN) for this purpose. We train and evaluate the models using data collected from 37,006 COVID-19 patients at NYU Langone Health in New York, USA. The convolution-based model outperforms the LSTM based model, achieving a high AUROC of 0.8844-0.9336 for 3 to 24 h deterioration prediction on a held-out test set. We also conduct occlusion experiments to evaluate the importance of each input feature, which reveals the significance of continuously monitoring the variation of the vital signs. Our results show the prospect for accurate deterioration forecast using a minimum feature set that can be relatively easily obtained using wearable devices and self-reported patient information.

Data Science Methods and Tools for Industry 4.0: A Systematic Literature Review and Taxonomy

The Fourth Industrial Revolution, also named Industry 4.0, is leveraging several modern computing fields. Industry 4.0 comprises automated tasks in manufacturing facilities, which generate massive quantities of data through sensors. These data contribute to the interpretation of industrial operations in favor of managerial and technical decision-making. Data science supports this interpretation due to extensive technological artifacts, particularly data processing methods and software tools. In this regard, the present article proposes a systematic literature review of these methods and tools employed in distinct industrial segments, considering an investigation of different time series levels and data quality. The systematic methodology initially approached the filtering of 10,456 articles from five academic databases, 103 being selected for the corpus. Thereby, the study answered three general, two focused, and two statistical research questions to shape the findings. As a result, this research found 16 industrial segments, 168 data science methods, and 95 software tools explored by studies from the literature. Furthermore, the research highlighted the employment of diverse neural network subvariations and missing details in the data composition. Finally, this article organized these results in a taxonomic approach to synthesize a state-of-the-art representation and visualization, favoring future research studies in the field.

Enhancing travel time prediction with deep learning on chronological and retrospective time order information of big traffic data

With growing environmental concerns and the exploitation of ubiquitous big data, smart transportation is transforming logistics business and operations into a more sustainable approach. To answer questions in intelligent transportation planning, such as which data are feasible, which methods are applicable for intelligent prediction of such data, and what are the available operations for prediction, this paper offers a new deep learning approach called bi-directional isometric-gated recurrent unit (BDIGRU). It is merged to the deep learning framework of neural networks for predictive analysis of travel time and business adoption for route planning. The proposed new method directly learns high-level features from big traffic data and reconstructs them by its own attention mechanism drawn by temporal orders to complete the learning process recursively in an end-to-end manner. After deriving the computational algorithm with stochastic gradient descent, we use the proposed method to perform predictive analysis of stochastic travel time under various traffic conditions (especially for congestions) and then determine the optimal vehicle route with the shortest travel time under future uncertainty. Based on empirical results with big traffic data, we show that the proposed BDIGRU method can (1) significantly improve the predictive accuracy of one-step 30 min ahead travel time compared to several conventional (data-driven, model-driven, hybrid, and heuristics) methods measured with several performance criteria, and (2) efficiently determine the optimal vehicle route in relation to the predictive variability under uncertainty.

Statistical actuarial estimation of the Capitation Payment Unit from copula functions and deep learning: historical comparability analysis for the Colombian health system, 2015-2021

The Capitation Payment Unit (CPU) financing mechanism constitutes more than 70% of health spending in Colombia, with a budget allocation of close to 60 trillion Colombian pesos for the year 2022 (approximately 15.7 billion US dollars). This article estimates actuarially, using modern techniques, the CPU for the contributory regime of the General System of Social Security in Health in Colombia, and compares it with what is estimated by the Ministry of Health and Social Protection. Using freely available information systems, by means of statistical copulas functions and artificial neural networks, pure risk premiums are calculated between 2015 and 2021. The study concludes that the weights by risk category are systematically different, showing historical pure premiums surpluses in the group of 0-1 years and deficits (for the regions normal and cities) in the groups over 54 years of age.

A dual attention-based fusion network for long- and short-term multivariate vehicle exhaust emission prediction

The increasing number of vehicles is one main cause of atmospheric environment pollution problems. Timely and accurate long- and short-term (LST) prediction of the on-road vehicle exhaust emission could contribute to atmospheric pollution prevention, public health protection, and government decision-making for environmental management. Vehicle exhaust emission has strong non-stationary and nonlinear characteristics due to the inherent randomness and imbalance nature of meteorological factors and traffic flow. Therefore accurate LST vehicle exhaust emission prediction encounters many challenges, such as the LST temporal dependencies and complicated nonlinear correlation on various emission gases, including carbon monoxide (CO), hydrocarbon (HC), and nitric oxide (NO), and external influence factors. To resolve these challenging issues, we propose a novel hybrid deep learning framework, namely Dual Attention-based Fusion Network (DAFNet), to effectively predict LST multivariate vehicle exhaust emission with the temporal convolutional network, convolutional neural network, long short term memory (LSTM)-skip based on recurrent neural network, dual attention mechanism, and autoregressive decomposition model. The proposed DAFNet consists of three major parts: 1) a nonlinear component to effectively capture the dynamic LST temporal dependency of multivariate gas by the temporal convolutional network, convolutional neural network, and LSTM-skip. Moreover, the above two networks employ an attention mechanism to model the internal relevance of the LST temporal patterns and multivariate gas, respectively. 2) a linear component to tackle the scale-insensitive problem of the neural network model by an autoregressive decomposition model. 3) the external components are taken to compensate the impact of external factors on vehicle exhaust emission by the multilayer perceptron model. Finally, the proposed DAFNet is evaluated on two real-world vehicle emission datasets in Zibo and Hefei, China. Experimental results demonstrate that the proposed DAFNet is a powerful tool to provide highly accurate prediction for LST multivariate vehicle exhaust emission in the field of vehicle environmental management.

rAAV Manufacturing: The Challenges of Soft Sensing during Upstream Processing

Recombinant adeno-associated virus (rAAV) is the most effective viral vector technology for directly translating the genomic revolution into medicinal therapies. However, the manufacturing of rAAV viral vectors remains challenging in the upstream processing with low rAAV yield in large-scale production and high cost, limiting the generalization of rAAV-based treatments. This situation can be improved by real-time monitoring of critical process parameters (CPP) that affect critical quality attributes (CQA). To achieve this aim, soft sensing combined with predictive modeling is an important strategy that can be used for optimizing the upstream process of rAAV production by monitoring critical process variables in real time. However, the development of soft sensors for rAAV production as a fast and low-cost monitoring approach is not an easy task. This review article describes four challenges and critically discusses the possible solutions that can enable the application of soft sensors for rAAV production monitoring. The challenges from a data scientist's perspective are (i) a predictor variable (soft-sensor inputs) set without AAV viral titer, (ii) multi-step forecasting, (iii) multiple process phases, and (iv) soft-sensor development composed of the mechanistic model.

Data-driven multi-joint waveguide bending sensor based on time series neural network

Due to the bulky interrogation devices, traditional fiber optic sensing system is mainly connected by wire or equipped only for large facilities. However, the advancement in neural network algorithms and flexible materials has broadened its application scenarios to bionics. In this paper, a multi-joint waveguide bending sensor based on color dyed filters is designed to detect bending angles, directions and positions. The sensors are fabricated by casting method using soft silicone rubber. Besides, required optical properties of sensor materials are characterized to better understand principles of the sensor design. Time series neural networks are utilized to predict bending position and angle quantitatively. The results confirm that the waveguide sensor demodulated by the data-driven neural network algorithm performs well and can be used for engineering applications.

A Hybrid Seasonal Autoregressive Integrated Moving Average and Denoising Autoencoder Model for Atmospheric Temperature Profile Prediction

Atmospheric parameter profile plays a vital role in the studies on meteorology and quantitative remote sensing. Besides measurement of the radiosonde and satellite remote sensing, the atmospheric temperature profile can be predicted by the time series model. However, the performance of the time series model is limited by the weak correlation of temperature data in the stratosphere. In this study, to predict monthly mean atmospheric temperature profile, a hybrid model named seasonal autoregressive integrated moving average and denoising autoencoder (SARIMA-DAE), which combines the traditional time series model with an artificial neural network (ANN), is proposed. The SARIMA is first used to predict the temperature at each altitude level independently; then, the DAE is employed to denoise the predicted temperature profile by SARIMA. The data set used is the L3 monthly mean gridded product of atmospheric infrared sounder, an instrument aboard NASA's Aqua satellite. Taking Maoming, a city in China, as an example, the absolute errors are within 1 K. The mean squared error and mean absolute percentage error on the test set are 0.12 and 0.0012, respectively. Compared with SARIMA, support vector machine regression, and ANN models, experimental results show that the SARIMA-DAE model improves the prediction accuracy at almost all altitude levels. Especially in the stratosphere, the advantage of the hybrid model is more pronounced. The model we proposed should, therefore, be of value to estimate missing data and predict atmospheric parameter profiles.

An interpretable DIC risk prediction model based on convolutional neural networks with time series data

Disseminated intravascular coagulation (DIC) is a complex, life-threatening syndrome associated with the end-stage of different coagulation disorders. Early prediction of the risk of DIC development is an urgent clinical need to reduce adverse outcomes. However, effective approaches and models to identify early DIC are still lacking. In this study, a novel interpretable deep learning based time series is used to predict the risk of DIC. The study cohort included ICU patients from a 4300-bed academic hospital between January 1, 2019, and January 1, 2022. Experimental results show that our model achieves excellent performance (AUC: 0.986, Accuracy: 95.7%, and F1:0.935). Gradient-weighted Class Activation Mapping (Grad-CAM) was used to explain how predictive models identified patients with DIC. The decision basis of the model was displayed in the form of a heat map. The model can be used to identify high-risk patients with DIC early, which will help in the early intervention of DIC patients and improve the treatment effect.

A Hybrid Model Based on Improved Transformer and Graph Convolutional Network for COVID-19 Forecasting

The coronavirus disease 2019 (COVID-19) has spread rapidly around the world since 2020, with a significant fatality rate. Until recently, numerous countries were unable to adequately control the pandemic. As a result, COVID-19 trend prediction has become a hot topic in academic circles. Both traditional models and existing deep learning (DL) models have the problem of low prediction accuracy. In this paper, we propose a hybrid model based on an improved Transformer and graph convolution network (GCN) for COVID-19 forecasting. The salient feature of the model in this paper is that rich temporal sequence information is extracted by the multi-head attention mechanism, and then the correlation of temporal sequence information is further aggregated by GCN. In addition, to solve the problem of the high time complexity of the existing Transformer, we use the cosine function to replace the softmax calculation, so that the calculation of query, key and value can be split, and the time complexity is reduced from the original O(N2) to O(N). We only concentrated on three states in the United States, one of which was the most affected, one of which was the least affected, and one intermediate state, in order to make our predictions more meaningful. We use mean absolute percentage error and mean absolute error as evaluation indexes. The experimental results show that the proposed time series model has a better predictive performance than the current DL models and traditional models. Additionally, our model's convergence outperforms that of the current DL models, offering a more precise benchmark for the control of epidemics.

EP-ADTA: Edge Prediction-Based Adaptive Data Transfer Algorithm for Underwater Wireless Sensor Networks (UWSNs)

The underwater wireless sensor network is an important component of the underwater three-dimensional monitoring system. Due to the high bit error rate, high delay, low bandwidth, limited energy, and high dynamic of underwater networks, it is very difficult to realize efficient and reliable data transmission. Therefore, this paper posits that it is not enough to design the routing algorithm only from the perspective of the transmission environment; the comprehensive design of the data transmission algorithm should also be combined with the application. An edge prediction-based adaptive data transmission algorithm (EP-ADTA) is proposed that can dynamically adapt to the needs of underwater monitoring applications and the changes in the transmission environment. EP-ADTA uses the end–edge–cloud architecture to define the underwater wireless sensor networks. The algorithm uses communication nodes as the agents, realizes the monitoring data prediction and compression according to the edge prediction, dynamically selects the transmission route, and controls the data transmission accuracy based on reinforcement learning. The simulation results show that EP-ADTA can meet the accuracy requirements of underwater monitoring applications, dynamically adapt to the changes in the transmission environment, and ensure efficient and reliable data transmission in underwater wireless sensor networks.

Time-Series Forecasting of a CO2-EOR and CO2 Storage Project Using a Data-Driven Approach

This study aims to develop a predictive and reliable data-driven model for forecasting the fluid production (oil, gas, and water) of existing wells and future infill wells for CO2-enhanced oil recovery (EOR) and CO2 storage projects. Several models were investigated, such as auto-regressive (AR), multilayer perceptron (MLP), and long short-term memory (LSTM) networks. The models were trained based on static and dynamic parameters and daily fluid production while considering the inverse distance of neighboring wells. The developed models were evaluated using walk-forward validation and compared based on the quality metrics, span, and variation in the forecasting horizon. The AR model demonstrates a convincing generalization performance across various time series datasets with a long but varied forecasting horizon across eight wells. The LSTM model has a shorter forecasting horizon but strong generalizability and robustness in forecasting horizon consistency. MLP has the shortest and most varied forecasting horizon compared to the other models. The LSTM model exhibits promising performance in forecasting the fluid production of future infill wells when the model is developed from an existing well with similar features to an infill well. This study offers an alternative to the physics-driven model when traditional modeling is costly and laborious.

A Review on Deep Sequential Models for Forecasting Time Series Data

Deep sequential (DS) models are extensively employed for forecasting time series data since the dawn of the deep learning era, and they provide forecasts for the values required in subsequent time steps. DS models, unlike other traditional statistical models for forecasting time series data, can learn hidden patterns in temporal sequences and have the memorizing data from prior time points. Given the widespread usage of deep sequential models in several domains, a comprehensive study describing their applications is necessary. This work presents a comprehensive review of contemporary deep learning time series models, their performance in diverse domains, and an investigation of the models that were employed in various applications. Three deep sequential models, namely, artificial neural network (ANN), long short-term memory (LSTM), and temporal-conventional neural network (TCNN) along with their applications for forecasting time series data, are elaborated. We showed a comprehensive comparison between such models in terms of application fields, model structure and activation functions, optimizers, and implementation, with a goal of learning more about the optimal model used. Furthermore, the challenges and perspectives of future development of deep sequential models are presented and discussed. We conclude that the LSTM model is widely employed, particularly in the form of a hybrid model, in which the most accurate predictions are made when the shape of hybrids is used as the model.

Learning models for forecasting hospital resource utilization for COVID-19 patients in Canada

Hospitals in Canada are facing a crisis-level shortage of critical supplies and equipment during the COVID-19 pandemic. This motivates us to create predictive models that can use Canada COVID-19 data and pandemic-related factors to accurately forecast 5 quantities-three related to hospital resource utilization (i.e., the number of hospital beds, ICU beds, and ventilators that will be needed by COVID-19 patients) and two to the pandemic progress (i.e., the number of COVID-19 cases and COVID-19 deaths)-several weeks in advance. We developed a machine learning method that can use information (i.e., resource utilization, pandemic progress, population mobility, weather condition, and public policy) currently known about a region since March 2020, to learn multiple temporal convolutional network (TCN) models every week; each used for forecasting the weekly average of one of these 5 quantities in Canada (respectively, in six specific provinces) for each, in the next 1 (resp., 2,3,4) weeks. To validate the effectiveness of our method, we compared our method, versus other standard models, on the COVID-19 data and hospital resource data, on the tasks of predicting the 116 values (for Canada and its six most populated provinces), every week from Oct 2020 to July 2021, and the 20 values (only for Canada) for four specific times within 9 July to 31 Dec 2021. Experimental results show that our 4640 TCN models (each forecasting a regional target for a specific future time, on a specific date) can produce accurate 1,2,3,4-week forecasts of the utilization of every hospital resource and pandemic progress for each week from 2 Oct 2020 to 2 July 2021, as well as 80 TCN models for each of the four specified times within 9 July and 31 Dec 2021. Compared to other baseline and state-of-the-art predictive models, our TCN models yielded the best forecasts, with the lowest mean absolute percentage error (MAPE). Additional experiments, on the IHME COVID-19 data, demonstrate the effectiveness of our TCN models, in comparison with IHME forecasts. Each of our TCN models used a pre-defined set of features; we experimentally validate the effectiveness of these features by showing that these models perform better than other models that instead used other features. Overall, these experimental results demonstrate that our method can accurately forecast hospital resource utilization and pandemic progress for Canada and for each of the six provinces.

An Ensemble-Based Multiclass Classifier for Intrusion Detection Using Internet of Things

Internet of Things (IoT) is the fastest growing technology that has applications in various domains such as healthcare, transportation. It interconnects trillions of smart devices through the Internet. A secure network is the basic necessity of the Internet of Things. Due to the increasing rate of interconnected and remotely accessible smart devices, more and more cybersecurity issues are being witnessed among cyber-physical systems. A perfect intrusion detection system (IDS) can probably identify various cybersecurity issues and their sources. In this article, using various telemetry datasets of different Internet of Things scenarios, we exhibit that external users can access the IoT devices and infer the victim user's activity by sniffing the network traffic. Further, the article presents the performance of various bagging and boosting ensemble decision tree techniques of machine learning in the design of an efficient IDS. Most of the previous IDSs just focused on good accuracy and ignored the execution speed that must be improved to optimize the performance of an ID model. Most of the earlier pieces of research focused on binary classification. This study attempts to evaluate the performance of various ensemble machine learning multiclass classification algorithms by deploying on openly available "TON-IoT" datasets of IoT and Industrial IoT (IIoT) sensors.

A Novel Encoder-Decoder Model for Multivariate Time Series Forecasting

The time series is a kind of complex structure data, which contains some special characteristics such as high dimension, dynamic, and high noise. Moreover, multivariate time series (MTS) has become a crucial study in data mining. The MTS utilizes the historical data to forecast its variation trend and has turned into one of the hotspots. In the era of rapid information development and big data, accurate prediction of MTS has attracted much attention. In this paper, a novel deep learning architecture based on the encoder-decoder framework is proposed for MTS forecasting. In this architecture, firstly, the gated recurrent unit (GRU) is taken as the main unit structure of both the procedures in encoding and decoding to extract the useful successive feature information. Then, different from the existing models, the attention mechanism (AM) is introduced to exploit the importance of different historical data for reconstruction at the decoding stage. Meanwhile, feature reuse is realized by skip connections based on the residual network for alleviating the influence of previous features on data reconstruction. Finally, in order to enhance the performance and the discriminative ability of the new MTS, the convolutional structure and fully connected module are established. Furthermore, to better validate the effectiveness of MTS forecasting, extensive experiments are executed on two different types of MTS such as stock data and shared bicycle data, respectively. The experimental results adequately demonstrate the effectiveness and the feasibility of the proposed method.

A deep LSTM network for the Spanish electricity consumption forecasting

Nowadays, electricity is a basic commodity necessary for the well-being of any modern society. Due to the growth in electricity consumption in recent years, mainly in large cities, electricity forecasting is key to the management of an efficient, sustainable and safe smart grid for the consumer. In this work, a deep neural network is proposed to address the electricity consumption forecasting in the short-term, namely, a long short-term memory (LSTM) network due to its ability to deal with sequential data such as time-series data. First, the optimal values for certain hyper-parameters have been obtained by a random search and a metaheuristic, called coronavirus optimization algorithm (CVOA), based on the propagation of the SARS-Cov-2 virus. Then, the optimal LSTM has been applied to predict the electricity demand with 4-h forecast horizon. Results using Spanish electricity data during nine years and half measured with 10-min frequency are presented and discussed. Finally, the performance of the proposed LSTM using random search and the LSTM using CVOA is compared, on the one hand, with that of recently published deep neural networks (such as a deep feed-forward neural network optimized with a grid search) and temporal fusion transformers optimized with a sampling algorithm, and, on the other hand, with traditional machine learning techniques, such as a linear regression, decision trees and tree-based ensemble techniques (gradient-boosted trees and random forest), achieving the smallest prediction error below 1.5%.

Hydropower production prediction using artificial neural networks: an Ecuadorian application case

Hydropower is among the most efficient technologies to produce renewable electrical energy. Hydropower systems present multiple advantages since they provide sustainable and controllable energy. However, hydropower plants’ effectiveness is affected by multiple factors such as river/reservoir inflows, temperature, electricity price, among others. The mentioned factors make the prediction and recommendation of a station’s operational output a difficult challenge. Therefore, reliable and accurate energy production forecasts are vital and of great importance for capacity planning, scheduling, and power systems operation. This research aims to develop and apply artificial neural network (ANN) models to predict hydroelectric production in Ecuador’s short and medium term, considering historical data such as hydropower production and precipitations. For this purpose, two scenarios based on the prediction horizon have been considered, i.e., one-step and multi-step forecasted problems. Sixteen ANN structures based on multilayer perceptron (MLP), long short-term memory (LSTM), and sequence-to-sequence (seq2seq) LSTM were designed. More than 3000 models were configured, trained, and validated using a grid search algorithm based on hyperparameters. The results show that the MLP univariate and differentiated model of one-step scenario outperforms the other architectures analyzed in both scenarios. The obtained model can be an important tool for energy planning and decision-making for sustainable hydropower production.

Application of Deep Learning Architectures for Satellite Image Time Series Prediction: A Review

Satellite image time series (SITS) is a sequence of satellite images that record a given area at several consecutive times. The aim of such sequences is to use not only spatial information but also the temporal dimension of the data, which is used for multiple real-world applications, such as classification, segmentation, anomaly detection, and prediction. Several traditional machine learning algorithms have been developed and successfully applied to time series for predictions. However, these methods have limitations in some situations, thus deep learning (DL) techniques have been introduced to achieve the best performance. Reviews of machine learning and DL methods for time series prediction problems have been conducted in previous studies. However, to the best of our knowledge, none of these surveys have addressed the specific case of works using DL techniques and satellite images as datasets for predictions. Therefore, this paper concentrates on the DL applications for SITS prediction, giving an overview of the main elements used to design and evaluate the predictive models, namely the architectures, data, optimization functions, and evaluation metrics. The reviewed DL-based models are divided into three categories, namely recurrent neural network-based models, hybrid models, and feed-forward-based models (convolutional neural networks and multi-layer perceptron). The main characteristics of satellite images and the major existing applications in the field of SITS prediction are also presented in this article. These applications include weather forecasting, precipitation nowcasting, spatio-temporal analysis, and missing data reconstruction. Finally, current limitations and proposed workable solutions related to the use of DL for SITS prediction are also highlighted.

Wind Power Forecasting with Deep Learning Networks: Time-Series Forecasting

Studies have demonstrated that changes in the climate affect wind power forecasting under different weather conditions. Theoretically, accurate prediction of both wind power output and weather changes using statistics-based prediction models is difficult. In practice, traditional machine learning models can perform long-term wind power forecasting with a mean absolute percentage error (MAPE) of 10% to 17%, which does not meet the engineering requirements for our renewable energy project. Deep learning networks (DLNs) have been employed to obtain the correlations between meteorological features and power generation using a multilayer neural convolutional architecture with gradient descent algorithms to minimize estimation errors. This has wide applicability to the field of wind power forecasting. Therefore, this study aimed at the long-term (24–72-h ahead) prediction of wind power with an MAPE of less than 10% by using the Temporal Convolutional Network (TCN) algorithm of DLNs. In our experiment, we performed TCN model pretraining using historical weather data and the power generation outputs of a wind turbine from a Scada wind power plant in Turkey. The experimental results indicated an MAPE of 5.13% for 72-h wind power prediction, which is adequate within the constraints of our project. Finally, we compared the performance of four DLN-based prediction models for power forecasting, namely, the TCN, long short-term memory (LSTM), recurrent neural network (RNN), and gated recurrence unit (GRU) models. We validated that the TCN outperforms the other three models for wind power prediction in terms of data input volume, stability of error reduction, and forecast accuracy.

A Data-Driven Multi-Regime Approach for Predicting Energy Consumption

There has been increasing interest in reducing carbon footprints globally in recent years. Hence increasing share of green energy and energy efficiency are promoted by governments. Therefore, optimizing energy consumption is becoming more critical for people, companies, industries, and the environment. Predicting energy consumption more precisely means that future energy management planning can be more effective. To date, most research papers have focused on predicting residential building energy consumption; however, a large portion of the energy is consumed by industrial machines. Prediction of energy consumption of large industrial machines in real time is challenging due to concept drift, in which prediction performance deteriorates over time. In this research, a novel data-driven method multi-regime approach (MRA) was developed to better predict the energy consumption for industrial machines. Whereas most papers have focused on finding an excellent prediction model that contradicts the no-free-lunch theorem, this study concentrated on adding potential concept drift points into the prediction process. A real-world dataset was collected from a semi-autonomous grinding (SAG) mill used as a data source, and a deep neural network was utilized as a prediction model for the MRA method. The results proved that the MRA method enables the detection of multi-regimes over time and provides a highly accurate prediction performance, thanks to the dynamic model approach.

Deep learning approach towards accurate state of charge estimation for lithium-ion batteries using self-supervised transformer model

Accurate state of charge (SOC) estimation of lithium-ion (Li-ion) batteries is crucial in prolonging cell lifespan and ensuring its safe operation for electric vehicle applications. In this article, we propose the deep learning-based transformer model trained with self-supervised learning (SSL) for end-to-end SOC estimation without the requirements of feature engineering or adaptive filtering. We demonstrate that with the SSL framework, the proposed deep learning transformer model achieves the lowest root-mean-square-error (RMSE) of 0.90% and a mean-absolute-error (MAE) of 0.44% at constant ambient temperature, and RMSE of 1.19% and a MAE of 0.7% at varying ambient temperature. With SSL, the proposed model can be trained with as few as 5 epochs using only 20% of the total training data and still achieves less than 1.9% RMSE on the test data. Finally, we also demonstrate that the learning weights during the SSL training can be transferred to a new Li-ion cell with different chemistry and still achieve on-par performance compared to the models trained from scratch on the new cell.

Gap Reconstruction in Optical Motion Capture Sequences Using Neural Networks

Optical motion capture is a mature contemporary technique for the acquisition of motion data; alas, it is non-error-free. Due to technical limitations and occlusions of markers, gaps might occur in such recordings. The article reviews various neural network architectures applied to the gap-filling problem in motion capture sequences within the FBM framework providing a representation of body kinematic structure. The results are compared with interpolation and matrix completion methods. We found out that, for longer sequences, simple linear feedforward neural networks can outperform the other, sophisticated architectures, but these outcomes might be affected by the small amount of data availabe for training. We were also able to identify that the acceleration and monotonicity of input sequence are the parameters that have a notable impact on the obtained results.

A Chaotic Neural Network Model for English Machine Translation Based on Big Data Analysis

In this paper, the chaotic neural network model of big data analysis is used to conduct in-depth analysis and research on the English translation. Firstly, under the guidance of the translation strategy of text type theory, the translation generated by the machine translation system is edited after translation, and then professionals specializing in computer and translation are invited to confirm the translation. After that, the errors in the translations generated by the machine translation system are classified based on the Double Quantum Filter-Muttahida Quami Movement (DQF-MQM) error type classification framework. Due to the characteristics of the source text as an informative academic text, long and difficult sentences, passive voice, and terminology translation are the main causes of machine translation errors. In view of the rigorous logic of the source text and the fixed language steps, this research proposes corresponding post-translation editing strategies for each type of error. It is suggested that translators should maintain the logic of the source text by converting implicit connections into explicit connections, maintain the academic accuracy of the source text by adding subjects and adjusting the word order to deal with the passive voice, and deal with semitechnical terms by appropriately selecting word meanings in postediting. The errors of machine translation in computer science and technology text abstracts are systematically categorized, and the corresponding post-translation editing strategies are proposed to provide reference suggestions for translators in this field, to improve the quality of machine translation in this field.

Stock Price Movement Prediction Based on a Deep Factorization Machine and the Attention Mechanism

The prediction of stock price movement is a popular area of research in academic and industrial fields due to the dynamic, highly sensitive, nonlinear and chaotic nature of stock prices. In this paper, we constructed a convolutional neural network model based on a deep factorization machine and attention mechanism (FA-CNN) to improve the prediction accuracy of stock price movement via enhanced feature learning. Unlike most previous studies, which focus only on the temporal features of financial time series data, our model also extracts intraday interactions among input features. Further, in data representation, we used the sub-industry index as supplementary information for the current state of the stock, since there exists stock price co-movement between individual stocks and their industry index. The experiments were carried on the individual stocks in three industries. The results showed that the additional inputs of (a) the intraday interactions among input features and (b) the sub-industry index information effectively improved the prediction accuracy. The highest prediction accuracy of the proposed FA-CNN model is 64.81%. It is 7.38% higher than that of traditional LSTM, and 3.71% higher than that of the model without sub-industry index as additional input features.