Nonlinear Principal Component Analysis: Neural Network Models and Applications

Missing value imputation on gene expression data using bee-based algorithm to improve classification performance

Existing missing value imputation methods focused on imputing the data regarding actual values towards a completion of datasets as an input for machine learning tasks. This work proposes an imputation of missing values towards improvement of accuracy performance for classification. The proposed method was based on bee algorithm and the use of k-nearest neighborhood with linear regression to guide on finding the appropriate solution in prevention of randomness. Among the processes, GINI importance score was utilized in selecting values for imputation. The imputed values thus reflected on improving a discriminative power in classification tasks instead of replicating the actual values from the original dataset. In this study, we evaluated the proposed method against frequently used imputation methods such as k-nearest neighborhood, principal components analysis, nonlinear principal, and component analysis to compare root mean square error results and accuracy of using imputed datasets in a classification task. The experimental results indicated that our proposed method obtained the best accuracy results from all datasets comparing to other methods. In comparison to original dataset, the classification model from imputed datasets yielded 15-25% higher accuracy in class prediction. From analysis, the results showed that feature ranking used in a classification process was affected and lead to noticeably change in informativeness as the imputed data from the proposed method played the role to boost a discriminating power.

Reconstructing rodent brain signals during euthanasia with eigensystem realization algorithm (ERA)

We accurately reconstruct the Local Field Potential time series obtained from anesthetized and awake rats, both before and during CO2 euthanasia. We apply the Eigensystem Realization Algorithm to identify an underlying linear dynamical system capable of generating the observed data. Time series exhibiting more intricate dynamics typically lead to systems of higher dimensions, offering a means to assess the complexity of the brain throughout various phases of the experiment. Our results indicate that anesthetized brains possess complexity levels similar to awake brains before CO2 administration. This resemblance undergoes significant changes following euthanization, as signals from the awake brain display a more resilient complexity profile, implying a state of heightened neuronal activity or a last fight response during the euthanasia process. In contrast, anesthetized brains seem to enter a more subdued state early on. Our data-driven techniques can likely be applied to a broader range of electrophysiological recording modalities.

Nonlinear manifolds underlie neural population activity during behaviour

There is rich variety in the activity of single neurons recorded during behaviour. Yet, these diverse single neuron responses can be well described by relatively few patterns of neural co-modulation. The study of such low-dimensional structure of neural population activity has provided important insights into how the brain generates behaviour. Virtually all of these studies have used linear dimensionality reduction techniques to estimate these population-wide co-modulation patterns, constraining them to a flat "neural manifold". Here, we hypothesised that since neurons have nonlinear responses and make thousands of distributed and recurrent connections that likely amplify such nonlinearities, neural manifolds should be intrinsically nonlinear. Combining neural population recordings from monkey, mouse, and human motor cortex, and mouse striatum, we show that: 1) neural manifolds are intrinsically nonlinear; 2) their nonlinearity becomes more evident during complex tasks that require more varied activity patterns; and 3) manifold nonlinearity varies across architecturally distinct brain regions. Simulations using recurrent neural network models confirmed the proposed relationship between circuit connectivity and manifold nonlinearity, including the differences across architecturally distinct regions. Thus, neural manifolds underlying the generation of behaviour are inherently nonlinear, and properly accounting for such nonlinearities will be critical as neuroscientists move towards studying numerous brain regions involved in increasingly complex and naturalistic behaviours.

Evaluation of particulate organic carbon from MODIS-Aqua in a marine-coastal water body

La Paz Bay (LPB) in Mexico is one of the largest marine-coastal bodies of water in the Gulf of California (GC) and is ecologically important for the feeding, reproduction, and refuge of marine species. Although particulate organic carbon (POC) is an important reservoir of oceanic carbon and an indicator of productivity in the euphotic zone, studies in this region are scarce. This study evaluates the performance of satellite-derived POC in LPB from January 2003 to December 2020. The metrics obtained for COP ( RMSE = 33.8 mg m - 3 ;P bias = 29.6 % yr P = 0.4 con p < 0.05 ), Chla-a ( RMSE = 0.23 mg m - 3 ;P bias = - 4.3 % yr P = 0.94 con p < 0.05 ), and SST ( RMSE = 2 . 3 ∘ C ;P bias = - 2.2 % yr P = 0.92 con p < 0.05 ) establish that although in some cases there was a slight over/underestimation, the satellite estimates consistently represent the variability and average values measured in situ. On the other hand, the spatio-temporal analysis of the POC allowed us to identify two seasons with their respective transition periods and five subregions in which the POC is characterized by having its maximum variability; two of these coincide with the locations of the eddies reported for the winter and summer seasons in the LPB, while the following three are located: one in the coastal zone and in the two areas in which the LPB interacts with the GC. The associations, variability nodes, and multiple linear regression analysis suggest that POC fluctuations in the LPB respond mainly to biological processes and, to some extent, to the seasonality of SST and wind. Finally, our results justify the use of the MODIS-Aqua satellite POC for studies in marine-coastal water bodies with similar characteristics to the LPB and suggest that this water body can be considered a reservoir for the marine region of northwestern Mexico.

Capabilities and limits of autoencoders for extracting collective variables in atomistic materials science

Free energy calculations in materials science are routinely hindered by the need to provide reaction coordinates that can meaningfully partition atomic configuration space, a prerequisite for most enhanced sampling approaches. Recent studies on molecular systems have highlighted the possibility of constructing appropriate collective variables directly from atomic motions through deep learning techniques. Here we extend this class of approaches to condensed matter problems, for which we encode the finite temperature collective variable by an iterative procedure starting from 0 K features of the energy landscape i.e. activation events or migration mechanisms given by a minimum - saddle point - minimum sequence. We employ the autoencoder neural networks in order to build a scalar collective variable for use with the adaptive biasing force method. Particular attention is given to design choices required for application to crystalline systems with defects, including the filtering of thermal motions which otherwise dominate the autoencoder input. The machine-learning workflow is tested on body-centered cubic iron and its common defects, such as small vacancy or self-interstitial clusters and screw dislocations. For localized defects, excellent collective variables as well as derivatives, necessary for free energy sampling, are systematically obtained. However, the approach has a limited accuracy when dealing with reaction coordinates that include atomic displacements of a magnitude comparable to thermal motions, e.g. the ones produced by the long-range elastic field of dislocations. We then combine the extraction of collective variables by autoencoders with an adaptive biasing force free energy method based on Bayesian inference. Using a vacancy migration as an example, we demonstrate the performance of coupling these two approaches for simultaneous discovery of reaction coordinates and free energy sampling in systems with localized defects.

On Sampling Minimum Energy Path

Sampling the minimum energy path (MEP) between two minima of a system is often hindered by the presence of an energy barrier separating the two metastable states. As a consequence, direct sampling based on molecular dynamics or Markov Chain Monte Carlo methods becomes inefficient, the crossing of the energy barrier being associated to a rare event. Augmented sampling methods based on the definition of collective variables or reaction coordinates allow us to circumvent this limitation at the price of an arbitrary choice of the dimensionality reduction algorithm. We couple the statistical sampling techniques, namely, metadynamics and invertible neural networks, with autoencoders so as to gradually learn the MEP and the collective variable at the same time. Learning is achieved through a succession of two steps: statistical sampling of the most probable path between the two minima and redefinition of the collective variable from the updated data points. The prototypical Mueller potential with nearly orthogonal minima is employed to demonstrate the ability of such coupling to unravel a complex MEP.

Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices

While promising for high-capacity machine learning accelerators, memristor devices have non-idealities that prevent software-equivalent accuracies when used for online training. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average gradients, stochastic rounding to avoid vanishing weight updates, and decomposition methods to keep the memory overhead low during mini-batch training. Since the weight update has to be transferred to the memristor matrices efficiently, we also investigate the impact of reconstructing the gradient matrixes both internally (rank-seq) and externally (rank-sum) to the memristor array. Our results show that streaming batch principal component analysis (streaming batch PCA) and non-negative matrix factorization (NMF) decomposition algorithms can achieve near MBGD accuracy in a memristor-based multi-layer perceptron trained on the MNIST (Modified National Institute of Standards and Technology) database with only 3 to 10 ranks at significant memory savings. Moreover, NMF rank-seq outperforms streaming batch PCA rank-seq at low-ranks making it more suitable for hardware implementation in future memristor-based accelerators.

Missing data estimation in extreme rainfall indices for the Metropolitan area of Cali - Colombia: An approach based on artificial neural networks

Changes observed in the current climate and projected for the future significantly concern researchers, decision-makers, and the general public. Climate indices of extreme rainfall events are a trend assessment tool to detect climate variability and change signals, which have an average reliability at least in the short term and given climatic inertia. This paper shows 12 climate indices of extreme rainfall events for annual and seasonal scales for 12 climate stations between 1969 to 2019 in the Metropolitan area of Cali (southwestern Colombia). The construction of the indices starts from daily rainfall time series, which although have between 0.5% and 5.4% of missing data, can affect the estimation of the indices. Here, we propose a methodology to complete missing data of the extreme event indices that model the peaks in the time series. This methodology uses an artificial neural network approach known as Non-Linear Principal Component Analysis (NLPCA). The approach reconstructs the time series by modulating the extreme values of the indices, a fundamental feature when evaluating extreme rainfall events in a region. The accuracy in the indices estimation shows values close to 1 in the Pearson's Correlation Coefficient and in the Bi-weighting Correlation. Moreover, values close to 0 in the percent bias and RMSE-observations standard deviation ratio. The database provided here is an essential input in future evaluation studies of extreme rainfall events in the Metropolitan area of Cali, the third most crucial urban conglomerate in Colombia with more than 3.9 million inhabitants.

Dimension-reduction simplifies the analysis of signal crosstalk in a bacterial quorum sensing pathway

Many pheromone sensing bacteria produce and detect more than one chemically distinct signal, or autoinducer. The pathways that detect these signals are typically noisy and interlocked through crosstalk and feedback. As a result, the sensing response of individual cells is described by statistical distributions that change under different combinations of signal inputs. Here we examine how signal crosstalk reshapes this response. We measure how combinations of two homoserine lactone (HSL) input signals alter the statistical distributions of individual cell responses in the AinS/R- and LuxI/R-controlled branches of the Vibrio fischeri bioluminescence pathway. We find that, while the distributions of pathway activation in individual cells vary in complex fashion with environmental conditions, these changes have a low-dimensional representation. For both the AinS/R and LuxI/R branches, the distribution of individual cell responses to mixtures of the two HSLs is effectively one-dimensional, so that a single tuning parameter can capture the full range of variability in the distributions. Combinations of crosstalking HSL signals extend the range of responses for each branch of the circuit, so that signals in combination allow population-wide distributions that are not available under a single HSL input. Dimension reduction also simplifies the problem of identifying the HSL conditions to which the pathways and their outputs are most sensitive. A comparison of the maximum sensitivity HSL conditions to actual HSL levels measured during culture growth indicates that the AinS/R and LuxI/R branches lack sensitivity to population density except during the very earliest and latest stages of growth respectively.

A level-wise spine registration framework to account for large pose changes

PURPOSES

Accurate and efficient spine registration is crucial to success of spine image guidance. However, changes in spine pose cause intervertebral motion that can lead to significant registration errors. In this study, we develop a geometrical rectification technique via nonlinear principal component analysis (NLPCA) to achieve level-wise vertebral registration that is robust to large changes in spine pose.

METHODS

We used explanted porcine spines and live pigs to develop and test our technique. Each sample was scanned with preoperative CT (pCT) in an initial pose and rescanned with intraoperative stereovision (iSV) in a different surgical posture. Patient registration rectified arbitrary spinal postures in pCT and iSV into a common, neutral pose through a parameterized moving-frame approach. Topologically encoded depth projection 2D images were then generated to establish invertible point-to-pixel correspondences. Level-wise point correspondences between pCT and iSV vertebral surfaces were generated via 2D image registration. Finally, closed-form vertebral level-wise rigid registration was obtained by directly mapping 3D surface point pairs. Implanted mini-screws were used as fiducial markers to measure registration accuracy.

RESULTS

In seven explanted porcine spines and two live animal surgeries (maximum in-spine pose change of 87.5 mm and 32.7 degrees averaged from all spines), average target registration errors (TRE) of 1.70  ±  0.15 mm and 1.85 ± 0.16 mm were achieved, respectively. The automated spine rectification took 3-5 min, followed by an additional 30 secs for depth image projection and level-wise registration.

CONCLUSIONS

Accuracy and efficiency of the proposed level-wise spine registration support its application in human open spine surgeries. The registration framework, itself, may also be applicable to other intraoperative imaging modalities such as ultrasound and MRI, which may expand utility of the approach in spine registration in general.

Building an adaptive interface via unsupervised tracking of latent manifolds

In human-machine interfaces, decoder calibration is critical to enable an effective and seamless interaction with the machine. However, recalibration is often necessary as the decoder off-line predictive power does not generally imply ease-of-use, due to closed loop dynamics and user adaptation that cannot be accounted for during the calibration procedure. Here, we propose an adaptive interface that makes use of a non-linear autoencoder trained iteratively to perform online manifold identification and tracking, with the dual goal of reducing the need for interface recalibration and enhancing human-machine joint performance. Importantly, the proposed approach avoids interrupting the operation of the device and it neither relies on information about the state of the task, nor on the existence of a stable neural or movement manifold, allowing it to be applied in the earliest stages of interface operation, when the formation of new neural strategies is still on-going. In order to more directly test the performance of our algorithm, we defined the autoencoder latent space as the control space of a body-machine interface. After an initial offline parameter tuning, we evaluated the performance of the adaptive interface versus that of a static decoder in approximating the evolving low-dimensional manifold of users simultaneously learning to perform reaching movements within the latent space. Results show that the adaptive approach increased the representational efficiency of the interface decoder. Concurrently, it significantly improved users' task-related performance, indicating that the development of a more accurate internal model is encouraged by the online co-adaptation process.

Unexpected complexity of everyday manual behaviors

How does the brain control an effector as complex and versatile as the hand? One possibility is that neural control is simplified by limiting the space of hand movements. Indeed, hand kinematics can be largely described within 8 to 10 dimensions. This oft replicated finding has been construed as evidence that hand postures are confined to this subspace. A prediction from this hypothesis is that dimensions outside of this subspace reflect noise. To address this question, we track the hand of human participants as they perform two tasks-grasping and signing in American Sign Language. We apply multiple dimension reduction techniques and replicate the finding that most postural variance falls within a reduced subspace. However, we show that dimensions outside of this subspace are highly structured and task dependent, suggesting they too are under volitional control. We propose that hand control occupies a higher dimensional space than previously considered.

A multivariate-based variable selection framework for clustering traffic conflicts in a brazilian freeway

More than one million people die or suffer non-fatal injuries annually due to road accidents around the world. Understanding the causes that give rise to different types of conflict events, as well as their characteristics, can help researchers and traffic authorities to draw up strategies aimed at mitigating collision risks. This paper proposes a framework for grouping traffic conflicts relying on similar profiles and factors that contribute to conflict occurrence using self-organizing maps (SOM). In order to improve the quality of the formed groups, we developed a novel variable importance index relying on the outputs of the nonlinear principal component analysis (NLPCA) that intends to identify the most informative variables for grouping collision events. Such index guides a backward variable selection procedure in which less relevant variables are removed one-by-one; after each removal, the clustering quality is assessed via the Davies-Bouldin (DB) index. The proposed framework was applied to a real-time dataset collected from a Brazilian highway aimed at allocating traffic conflicts into groups presenting similar profiles. The selected variables suggest that lower average speeds, which are typically verified during congestion events, contribute to conflict occurrence. Higher variability on speed (denoted by high standard deviation, and speed's coefficient of variation levels on that variable), which are also perceived in the assessed freeway near to congestion periods, also contribute to conflicts.

Microbial life cycles link global modularity in regulation to mosaic evolution

Microbes are exposed to changing environments, to which they can respond by adopting various lifestyles such as swimming, colony formation or dormancy. These lifestyles are often studied in isolation, thereby giving a fragmented view of the life cycle as a whole. Here, we study lifestyles in the context of this whole. We first use machine learning to reconstruct the expression changes underlying life cycle progression in the bacterium Bacillus subtilis, based on hundreds of previously acquired expression profiles. This yields a timeline that reveals the modular organization of the life cycle. By analysing over 380 Bacillales genomes, we then show that life cycle modularity gives rise to mosaic evolution in which life stages such as motility and sporulation are conserved and lost as discrete units. We postulate that this mosaic conservation pattern results from habitat changes that make these life stages obsolete or detrimental. Indeed, when evolving eight distinct Bacillales strains and species under laboratory conditions that favour colony growth, we observe rapid and parallel losses of the sporulation life stage across species, induced by mutations that affect the same global regulator. We conclude that a life cycle perspective is pivotal to understanding the causes and consequences of modularity in both regulation and evolution.

Recovery of Protein Folding Funnels from Single-Molecule Time Series by Delay Embeddings and Manifold Learning

The stability and folding of proteins is governed by the underlying single-molecule free energy surface (smFES) mapping the free energy of the molecule as a function of configurational state. Ascertaining the smFES is of great value in understanding and engineering protein structure and function. By integrating tools from dynamical systems theory and nonlinear manifold learning, we describe an approach to reconstruct the multidimensional smFES for a protein from a time series in a single experimentally measurable observable. We employ Takens' delay embeddings to project the time series into a high-dimensional space in which the projected dynamics are C¹-equivalent to the true system dynamics and employ diffusion maps to recover a low-dimensional reconstruction of the smFES that is equivalent to the true smFES up to a smooth and invertible transformation. We validate the approach in molecular dynamics simulations of Trp-cage, Villin, and BBA to demonstrate that landscapes recovered from univariate time series in the head-to-tail distance are topologically identical-they precisely preserve the metastable states and folding pathways-and topographically approximate-the free energy barrier heights and well depths are approximately preserved-to the true landscapes determined from complete knowledge of all atomic coordinates. We go on to show that the reconstructed landscapes reliably predict temperature denaturation and identify point mutations and groups of mutations critical to folding. These results demonstrate that protein folding funnels can be reconstructed from experimentally measurable time series and used to understand and engineer folding.

Molecular enhanced sampling with autoencoders: On-the-fly collective variable discovery and accelerated free energy landscape exploration

Macromolecular and biomolecular folding landscapes typically contain high free energy barriers that impede efficient sampling of configurational space by standard molecular dynamics simulation. Biased sampling can artificially drive the simulation along prespecified collective variables (CVs), but success depends critically on the availability of good CVs associated with the important collective dynamical motions. Nonlinear machine learning techniques can identify such CVs but typically do not furnish an explicit relationship with the atomic coordinates necessary to perform biased sampling. In this work, we employ auto-associative artificial neural networks ("autoencoders") to learn nonlinear CVs that are explicit and differentiable functions of the atomic coordinates. Our approach offers substantial speedups in exploration of configurational space, and is distinguished from existing approaches by its capacity to simultaneously discover and directly accelerate along data-driven CVs. We demonstrate the approach in simulations of alanine dipeptide and Trp-cage, and have developed an open-source and freely available implementation within OpenMM. © 2018 Wiley Periodicals, Inc.

A Mobile Health System for Neurocognitive Impairment Evaluation Based on P300 Detection

A new mobile healthcare system for neuro-cognitive function monitoring and treatment is presented. The architecture of the system features sensors to measure the brain potential, localized data analysis and filtering, and in-cloud distribution to specialized medical personnel. As such, it presents tradeoffs typical of other cyber-physical systems, where hardware, algorithms, and software implementations have to come together in a coherent fashion. The system is based on spatio-temporal detection and characterization of a specific brain potential called P300. The diagnosis of cognitive deficit is achieved by analyzing the data collected by the system with a new algorithm called tuned-Residue Iteration Decomposition (t-RIDE). The system has been tested on 17 subjects ( n = 12 healthy, n = 3 mildly cognitive impaired, and n = 2 with Alzheimer's disease involved in three different cognitive tasks with increasing difficulty. The system allows fast diagnosis of cognitive deficit, including mild and heavy cognitive impairment: t-RIDE convergence is achieved in 79 iterations (i.e., 1.95s), yielding an 80% accuracy in P300 amplitude evaluation with only 13 trials on a single EEG channel.

Collective variable discovery and enhanced sampling using autoencoders: Innovations in network architecture and error function design

Auto-associative neural networks ("autoencoders") present a powerful nonlinear dimensionality reduction technique to mine data-driven collective variables from molecular simulation trajectories. This technique furnishes explicit and differentiable expressions for the nonlinear collective variables, making it ideally suited for integration with enhanced sampling techniques for accelerated exploration of configurational space. In this work, we describe a number of sophistications of the neural network architectures to improve and generalize the process of interleaved collective variable discovery and enhanced sampling. We employ circular network nodes to accommodate periodicities in the collective variables, hierarchical network architectures to rank-order the collective variables, and generalized encoder-decoder architectures to support bespoke error functions for network training to incorporate prior knowledge. We demonstrate our approach in blind collective variable discovery and enhanced sampling of the configurational free energy landscapes of alanine dipeptide and Trp-cage using an open-source plugin developed for the OpenMM molecular simulation package.

Machine learning and data science in soft materials engineering

In many branches of materials science it is now routine to generate data sets of such large size and dimensionality that conventional methods of analysis fail. Paradigms and tools from data science and machine learning can provide scalable approaches to identify and extract trends and patterns within voluminous data sets, perform guided traversals of high-dimensional phase spaces, and furnish data-driven strategies for inverse materials design. This topical review provides an accessible introduction to machine learning tools in the context of soft and biological materials by 'de-jargonizing' data science terminology, presenting a taxonomy of machine learning techniques, and surveying the mathematical underpinnings and software implementations of popular tools, including principal component analysis, independent component analysis, diffusion maps, support vector machines, and relative entropy. We present illustrative examples of machine learning applications in soft matter, including inverse design of self-assembling materials, nonlinear learning of protein folding landscapes, high-throughput antimicrobial peptide design, and data-driven materials design engines. We close with an outlook on the challenges and opportunities for the field.

Distinct Variation Pattern Discovery Using Alternating Nonlinear Principal Component Analysis

Autoassociative neural networks (ANNs) have been proposed as a nonlinear extension of principal component analysis (PCA), which is commonly used to identify linear variation patterns in high-dimensional data. While principal component scores represent uncorrelated features, standard backpropagation methods for training ANNs provide no guarantee of producing distinct features, which is important for interpretability and for discovering the nature of the variation patterns in the data. Here, we present an alternating nonlinear PCA method, which encourages learning of distinct features in ANNs. A new measure motivated by the condition of orthogonal loadings in PCA is proposed for measuring the extent to which the nonlinear principal components represent distinct variation patterns. We demonstrate the effectiveness of our method using a simulated point cloud data set as well as a subset of the MNIST handwritten digits data. The results show that standard ANNs consistently mix the true variation sources in the low-dimensional representation learned by the model, whereas our alternating method produces solutions where the patterns are better separated in the low-dimensional space.

Principal polynomial analysis

This paper presents a new framework for manifold learning based on a sequence of principal polynomials that capture the possibly nonlinear nature of the data. The proposed Principal Polynomial Analysis (PPA) generalizes PCA by modeling the directions of maximal variance by means of curves, instead of straight lines. Contrarily to previous approaches, PPA reduces to performing simple univariate regressions, which makes it computationally feasible and robust. Moreover, PPA shows a number of interesting analytical properties. First, PPA is a volume-preserving map, which in turn guarantees the existence of the inverse. Second, such an inverse can be obtained in closed form. Invertibility is an important advantage over other learning methods, because it permits to understand the identified features in the input domain where the data has physical meaning. Moreover, it allows to evaluate the performance of dimensionality reduction in sensible (input-domain) units. Volume preservation also allows an easy computation of information theoretic quantities, such as the reduction in multi-information after the transform. Third, the analytical nature of PPA leads to a clear geometrical interpretation of the manifold: it allows the computation of Frenet-Serret frames (local features) and of generalized curvatures at any point of the space. And fourth, the analytical Jacobian allows the computation of the metric induced by the data, thus generalizing the Mahalanobis distance. These properties are demonstrated theoretically and illustrated experimentally. The performance of PPA is evaluated in dimensionality and redundancy reduction, in both synthetic and real datasets from the UCI repository.