Neural Network with Optimal Neuron Activation Functions Based on Additive Gaussian Process Regression

Computational Investigation of the Potential and Limitations of Machine Learning with Neural Network Circuits Based on Synaptic Transistors

Synaptic transistors have been proposed to implement neuron activation functions of neural networks (NNs). While promising to enable compact, fast, inexpensive, and energy-efficient dedicated NN circuits, they also have limitations compared to digital NNs (realized as codes for digital processors), including shape choices of the activation function using particular types of transistor implementation, and instabilities due to noise and other factors present in analog circuits. We present a computational study of the effects of these factors on NN performance and find that, while accuracy competitive with traditional NNs can be realized for many applications, there is high sensitivity to the instability in the shape of the activation function, suggesting that, when highly accurate NNs are required, high-precision circuitry should be developed beyond what has been reported for synaptic transistors to date.

Degeneration of kernel regression with Matern kernels into low-order polynomial regression in high dimension

Kernel methods such as kernel ridge regression and Gaussian process regression with Matern-type kernels have been increasingly used, in particular, to fit potential energy surfaces (PES) and density functionals, and for materials informatics. When the dimensionality of the feature space is high, these methods are used with necessarily sparse data. In this regime, the optimal length parameter of a Matern-type kernel may become so large that the method effectively degenerates into a low-order polynomial regression and, therefore, loses any advantage over such regression. This is demonstrated theoretically as well as numerically in the examples of six- and fifteen-dimensional molecular PES using squared exponential and simple exponential kernels. The results shed additional light on the success of polynomial approximations such as PIP for medium-size molecules and on the importance of orders-of-coupling-based models for preserving the advantages of kernel methods with Matern-type kernels of on the use of physically motivated (reproducing) kernels.

A controlled study of the effect of deviations from symmetry of the potential energy surface (PES) on the accuracy of the vibrational spectrum computed with collocation

Symmetry, in particular permutational symmetry, of a potential energy surface (PES) is a useful property in quantum chemical calculations. It facilitates, in particular, state labelling and identification of degenerate states. In many practically important applications, however, these issues are unimportant. The imposition of exact symmetry and the perception that it is necessary create additional methodological requirements narrowing or complicating algorithmic choices that are thereby biased against methods and codes that by default do not incorporate symmetry, including most off-the-shelf machine learning methods that cannot be directly used if exact symmetry is demanded. By introducing symmetric and unsymmetric errors into the PES of H2CO in a controlled way and computing the vibrational spectrum with collocation using symmetric and nonsymmetric collocation point sets, we show that when the deviations from an ideal PES are random, imposition of exact symmetry does not bring any practical advantages. Moreover, a calculation ignoring symmetry may be more accurate. We also compare machine-learned PESs with and without symmetrization and demonstrate that there is no advantage of imposing exact symmetry for the accuracy of the vibrational spectrum.