Predictive Modular Neural Networks for Time Series Classification

Multivariate LSTM-FCNs for time series classification

Over the past decade, multivariate time series classification has received great attention. We propose transforming the existing univariate time series classification models, the Long Short Term Memory Fully Convolutional Network (LSTM-FCN) and Attention LSTM-FCN (ALSTM-FCN), into a multivariate time series classification model by augmenting the fully convolutional block with a squeeze-and-excitation block to further improve accuracy. Our proposed models outperform most state-of-the-art models while requiring minimum preprocessing. The proposed models work efficiently on various complex multivariate time series classification tasks such as activity recognition or action recognition. Furthermore, the proposed models are highly efficient at test time and small enough to deploy on memory constrained systems.

Reacting to different types of concept drift: the Accuracy Updated Ensemble algorithm

Data stream mining has been receiving increased attention due to its presence in a wide range of applications, such as sensor networks, banking, and telecommunication. One of the most important challenges in learning from data streams is reacting to concept drift, i.e., unforeseen changes of the stream's underlying data distribution. Several classification algorithms that cope with concept drift have been put forward, however, most of them specialize in one type of change. In this paper, we propose a new data stream classifier, called the Accuracy Updated Ensemble (AUE2), which aims at reacting equally well to different types of drift. AUE2 combines accuracy-based weighting mechanisms known from block-based ensembles with the incremental nature of Hoeffding Trees. The proposed algorithm is experimentally compared with 11 state-of-the-art stream methods, including single classifiers, block-based and online ensembles, and hybrid approaches in different drift scenarios. Out of all the compared algorithms, AUE2 provided best average classification accuracy while proving to be less memory consuming than other ensemble approaches. Experimental results show that AUE2 can be considered suitable for scenarios, involving many types of drift as well as static environments.

FUZZY C-MEANS CLUSTERING MODELS FOR MULTIVARIATE TIME-VARYING DATA: DIFFERENT APPROACHES

The classification of multivariate time-varying data finds application in several fields, such as economics, finance, marketing research, psychometrics, bioinformatics, medicine, signal processing, pattern recognition, etc. In this paper, by considering an exploratory formalization, we propose different unsupervised clustering models for multivariate data time arrays (objects×quantitative variables×times). These models can be classified in two different approaches: the cross sectional and the longitudinal approach. In the first case, after the objects, observed at each time, have been classified, comparison among the classifications made in different time instants will be done. In the second approach, we cluster the time trajectories of the objects; then, we obtain only one classification by comparing the instantaneous and evolutive features of the trajectories of the objects. In particular, in this work, the second approach is analyzed in detail, with reference to the so-called single and double step procedures. Geometric, correlative, instantaneous, evolutive and trend characteristics of the multivariate time arrays are taken into account in the different proposed clustering models. Furthermore, the fuzzy approach, that is particularly suitable in the dynamic classification problem, has been considered. Extensions of a cluster-validity criterion for the proposed fuzzy dynamic clustering models are also suggested. A socio-economic example concludes the paper.

Predictive modular neural networks for unsupervised segmentation of switching time series: the data allocation problem

In this paper, we explore some aspects of the problem of online unsupervised learning of a switching time series, i.e., a time series which is generated by a combination of several alternately activated sources. This learning problem can be solved by a two-stage approach: 1) separating and assigning each incoming datum to a specific dataset (one dataset corresponding to each source) and 2) developing one model per dataset (i.e., one model per source). We introduce a general data allocation (DA) methodology, which combines the two steps into an iterative scheme: existing models compete for the incoming data; data assigned to each model are used to refine the model. We distinguish between two modes of DA: in parallel DA, every incoming datablock is allocated to the model with lowest prediction error; in serial DA, the incoming datablock is allocated to the first model with prediction error below a prespecified threshold. We present sufficient conditions for asymptotically correct allocation of the data. We also present numerical experiments to support our theoretical analysis.

Cooperative recurrent modular neural networks for constrained optimization: a survey of models and applications

Constrained optimization problems arise in a wide variety of scientific and engineering applications. Since several single recurrent neural networks when applied to solve constrained optimization problems for real-time engineering applications have shown some limitations, cooperative recurrent neural network approaches have been developed to overcome drawbacks of these single recurrent neural networks. This paper surveys in details work on cooperative recurrent neural networks for solving constrained optimization problems and their engineering applications, and points out their standing models from viewpoint of both convergence to the optimal solution and model complexity. We provide examples and comparisons to shown advantages of these models in the given applications.

Epicenter location by analysis of interictal spikes: a case study for the use of artificial neural networks in biomedical engineering

Artificial neural network (ANN) technology is finding increasing application in medicine and biomedical engineering. This paper supplies necessary background in ANN technology for researchers unfamiliar with this rapidly emerging discipline. This introduction to ANN application is cast in the context of epileptic seizure epicenter location. This is a very real problem faced by neurosurgeons every day. Precise location of the area of excision is currently determined with a network of surgically implanted subdural electrodes. This means that the cure entails two surgical procedures: one to implant the electrode array that precisely locates the epicenter, and another to remove the epicenter. This paper outlines an experimental diagnostic software system (DSS) that uses artificial neural network (ANN) analysis of magnetoencephalographic (MEG) data to eliminate the first of these surgical procedures. The MEG recording is a quick and painless process that requires no surgery. This approach has the potential to save time, reduce patient discomfort, and eliminate a painful and potentially dangerous surgical step in the treatment procedure.

Hybrid model building methodology using unsupervised fuzzy clustering and supervised neural networks

This paper suggests a model building methodology for dealing with new processes. The methodology, called Hybrid Fuzzy Neural Networks (HFNN), combines unsupervised fuzzy clustering and supervised neural networks in order to create simple and flexible models. Fuzzy clustering was used to define relevant domains on the input space. Then, sets of multilayer perceptrons (MLP) were trained (one for each domain) to map input-output relations, creating, in the process, a set of specified sub-models. The estimated output of the model was obtained by fusing the different sub-model outputs weighted by their predicted possibilities. On-line reinforcement learning enabled improvement of the model. The determination of the optimal number of clusters is fundamental to the success of the HFNN approach. The effectiveness of several validity measures was compared to the generalization capability of the model and information criteria. The validity measures were tested with fermentation simulations and real fermentations of a yeast-like fungus, Aureobasidium pullulans. The results outline the criteria limitations. The learning capability of the HFNN was tested with the fermentation data. The results underline the advantages of HFNN over a single neural network.

Time-Series Segmentation Using Predictive Modular Neural Networks

A predictive modular neural network method is applied to the problem of unsupervised time-series segmentation. The method consists of the concurrent application of two algorithms: one for source identification, the other for time-series classification. The source identification algorithm discovers the sources generating the time series, assigns data to each source, and trains one predictor for each source. The classification algorithm recursively computes a credit function for each source, based on the competition of the respective predictors, according to their predictive accuracy; the credit function is used for classification of the time-series observation at each time step. The method is tested by numerical experiments.