Energy landscape of the finite-size spherical three-spin glass model

The Loss Surface of Deep Linear Networks Viewed Through the Algebraic Geometry Lens

By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of deep linear neural network models. After providing clarification on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous symmetries of the deep linear networks, can be straightforwardly removed by a generalized L₂-regularization. Then, we establish upper bounds on the number of isolated stationary points of these networks with the help of algebraic geometry. Combining these upper bounds with a method in numerical algebraic geometry, we find all stationary points for modest depth and matrix size. We demonstrate that, in the presence of the non-zero regularization, deep linear networks can indeed possess local minima which are not global minima. Finally, we show that even though the number of stationary points increases as the number of neurons (regularization parameters) increases (decreases), higher index saddles are surprisingly rare.

Barriers, trapping times, and overlaps between local minima in the dynamics of the disordered Ising p-spin model

We study the low-temperature out-of-equilibrium Monte Carlo dynamics of the disordered Ising p-spin Model with p=3 and a small number of spin variables. We focus on sequences of configurations that are stable against single spin flips obtained by instantaneous gradient descent from persistent ones. We analyze the statistics of energy gaps, energy barriers, and trapping times on subsequences such that the overlap between consecutive configurations does not overcome a threshold. We compare our results to the predictions of various trap models finding the best agreement with the step model when the p-spin configurations are constrained to be uncorrelated.

Loss surface of XOR artificial neural networks

Training an artificial neural network involves an optimization process over the landscape defined by the cost (loss) as a function of the network parameters. We explore these landscapes using optimization tools developed for potential energy landscapes in molecular science. The number of local minima and transition states (saddle points of index one), as well as the ratio of transition states to minima, grow rapidly with the number of nodes in the network. There is also a strong dependence on the regularization parameter, with the landscape becoming more convex (fewer minima) as the regularization term increases. We demonstrate that in our formulation, stationary points for networks with N_{h} hidden nodes, including the minimal network required to fit the XOR data, are also stationary points for networks with N_{h}+1 hidden nodes when all the weights involving the additional node are zero. Hence, smaller networks trained on XOR data are embedded in the landscapes of larger networks. Our results clarify certain aspects of the classification and sensitivity (to perturbations in the input data) of minima and saddle points for this system, and may provide insight into dropout and network compression.

Energy landscapes for machine learning

Machine learning techniques are being increasingly used as flexible non-linear fitting and prediction tools in the physical sciences. Fitting functions that exhibit multiple solutions as local minima can be analysed in terms of the corresponding machine learning landscape. Methods to explore and visualise molecular potential energy landscapes can be applied to these machine learning landscapes to gain new insight into the solution space involved in training and the nature of the corresponding predictions. In particular, we can define quantities analogous to molecular structure, thermodynamics, and kinetics, and relate these emergent properties to the structure of the underlying landscape. This Perspective aims to describe these analogies with examples from recent applications, and suggest avenues for new interdisciplinary research.

Turning intractable counting into sampling: Computing the configurational entropy of three-dimensional jammed packings

We present a numerical calculation of the total number of disordered jammed configurations Ω of N repulsive, three-dimensional spheres in a fixed volume V. To make these calculations tractable, we increase the computational efficiency of the approach of Xu et al. [Phys. Rev. Lett. 106, 245502 (2011)10.1103/PhysRevLett.106.245502] and Asenjo et al. [Phys. Rev. Lett. 112, 098002 (2014)10.1103/PhysRevLett.112.098002] and we extend the method to allow computation of the configurational entropy as a function of pressure. The approach that we use computes the configurational entropy by sampling the absolute volume of basins of attraction of the stable packings in the potential energy landscape. We find a surprisingly strong correlation between the pressure of a configuration and the volume of its basin of attraction in the potential energy landscape. This relation is well described by a power law. Our methodology to compute the number of minima in the potential energy landscape should be applicable to a wide range of other enumeration problems in statistical physics, string theory, cosmology, and machine learning that aim to find the distribution of the extrema of a scalar cost function that depends on many degrees of freedom.

Index statistical properties of sparse random graphs

Using the replica method, we develop an analytical approach to compute the characteristic function for the probability P(N)(K,λ) that a large N×N adjacency matrix of sparse random graphs has K eigenvalues below a threshold λ. The method allows to determine, in principle, all moments of P(N)(K,λ), from which the typical sample-to-sample fluctuations can be fully characterized. For random graph models with localized eigenvectors, we show that the index variance scales linearly with N≫1 for |λ|>0, with a model-dependent prefactor that can be exactly calculated. Explicit results are discussed for Erdös-Rényi and regular random graphs, both exhibiting a prefactor with a nonmonotonic behavior as a function of λ. These results contrast with rotationally invariant random matrices, where the index variance scales only as lnN, with an universal prefactor that is independent of λ. Numerical diagonalization results confirm the exactness of our approach and, in addition, strongly support the Gaussian nature of the index fluctuations.

Exploring the potential energy landscape of the Thomson problem via Newton homotopies

Locating the stationary points of a real-valued multivariate potential energy function is an important problem in many areas of science. This task generally amounts to solving simultaneous nonlinear systems of equations. While there are several numerical methods that can find many or all stationary points, they each exhibit characteristic problems. Moreover, traditional methods tend to perform poorly near degenerate stationary points with additional zero Hessian eigenvalues. We propose an efficient and robust implementation of the Newton homotopy method, which is capable of quickly sampling a large number of stationary points of a wide range of indices, as well as degenerate stationary points. We demonstrate our approach by applying it to the Thomson problem. We also briefly discuss a possible connection between the present work and Smale's 7th problem.

Algebraic geometrization of the Kuramoto model: Equilibria and stability analysis

Finding equilibria of the finite size Kuramoto model amounts to solving a nonlinear system of equations, which is an important yet challenging problem. We translate this into an algebraic geometry problem and use numerical methods to find all of the equilibria for various choices of coupling constants K, natural frequencies, and on different graphs. We note that for even modest sizes (N ~ 10-20), the number of equilibria is already more than 100 000. We analyze the stability of each computed equilibrium as well as the configuration of angles. Our exploration of the equilibrium landscape leads to unexpected and possibly surprising results including non-monotonicity in the number of equilibria, a predictable pattern in the indices of equilibria, counter-examples to conjectures, multi-stable equilibrium landscapes, scenarios with only unstable equilibria, and multiple distinct extrema in the stable equilibrium distribution as a function of the number of cycles in the graph.

Energy landscape of the finite-size mean-field 2-spin spherical model and topology trivialization

Motivated by the recently observed phenomenon of topology trivialization of potential energy landscapes (PELs) for several statistical mechanics models, we perform a numerical study of the finite-size 2-spin spherical model using both numerical polynomial homotopy continuation and a reformulation via non-Hermitian matrices. The continuation approach computes all of the complex stationary points of this model while the matrix approach computes the real stationary points. Using these methods, we compute the average number of stationary points while changing the topology of the PEL as well as the variance. Histograms of these stationary points are presented along with an analysis regarding the complex stationary points. This work connects topology trivialization to two different branches of mathematics: algebraic geometry and catastrophe theory, which is fertile ground for further interdisciplinary research.

Certification and the potential energy landscape

Typically, there is no guarantee that a numerical approximation obtained using standard nonlinear equation solvers is indeed an actual solution, meaning that it lies in the quadratic convergence basin. Instead, it may lie only in the linear convergence basin, or even in a chaotic region, and hence not converge to the corresponding stationary point when further optimization is attempted. In some cases, these non-solutions could be misleading. Proving that a numerical approximation will quadratically converge to a stationary point is termed certification. In this report, we provide details of how Smale's α-theory can be used to certify numerically obtained stationary points of a potential energy landscape, providing a mathematical proof that the numerical approximation does indeed correspond to an actual stationary point, independent of the precision employed.

Potential energy landscape of the two-dimensional XY model: higher-index stationary points

The application of numerical techniques to the study of energy landscapes of large systems relies on sufficient sampling of the stationary points. Since the number of stationary points is believed to grow exponentially with system size, we can only sample a small fraction. We investigate the interplay between this restricted sample size and the physical features of the potential energy landscape for the two-dimensional XY model in the absence of disorder with up to N = 100 spins. Using an eigenvector-following technique, we numerically compute stationary points with a given Hessian index I for all possible values of I. We investigate the number of stationary points, their energy and index distributions, and other related quantities, with particular focus on the scaling with N. The results are used to test a number of conjectures and approximate analytic results for the general properties of energy landscapes.

Efficient retrieval of landscape Hessian: forced optimal covariance adaptive learning

Knowledge of the Hessian matrix at the landscape optimum of a controlled physical observable offers valuable information about the system robustness to control noise. The Hessian can also assist in physical landscape characterization, which is of particular interest in quantum system control experiments. The recently developed landscape theoretical analysis motivated the compilation of an automated method to learn the Hessian matrix about the global optimum without derivative measurements from noisy data. The current study introduces the forced optimal covariance adaptive learning (FOCAL) technique for this purpose. FOCAL relies on the covariance matrix adaptation evolution strategy (CMA-ES) that exploits covariance information amongst the control variables by means of principal component analysis. The FOCAL technique is designed to operate with experimental optimization, generally involving continuous high-dimensional search landscapes (≳30) with large Hessian condition numbers (≳10^{4}). This paper introduces the theoretical foundations of the inverse relationship between the covariance learned by the evolution strategy and the actual Hessian matrix of the landscape. FOCAL is presented and demonstrated to retrieve the Hessian matrix with high fidelity on both model landscapes and quantum control experiments, which are observed to possess nonseparable, nonquadratic search landscapes. The recovered Hessian forms were corroborated by physical knowledge of the systems. The implications of FOCAL extend beyond the investigated studies to potentially cover other physically motivated multivariate landscapes.

An inversion-relaxation approach for sampling stationary points of spin model Hamiltonians

Sampling the stationary points of a complicated potential energy landscape is a challenging problem. Here, we introduce a sampling method based on relaxation from stationary points of the highest index of the Hessian matrix. We illustrate how this approach can find all the stationary points for potentials or Hamiltonians bounded from above, which includes a large class of important spin models, and we show that it is far more efficient than previous methods. For potentials unbounded from above, the relaxation part of the method is still efficient in finding minima and transition states, which are usually the primary focus of attention for atomistic systems.

Potential energy landscapes for the 2D XY model: minima, transition states, and pathways

We describe a numerical study of the potential energy landscape for the two-dimensional XY model (with no disorder), considering up to 100 spins and central processing unit and graphics processing unit implementations of local optimization, focusing on minima and saddles of index one (transition states). We examine both periodic and anti-periodic boundary conditions, and show that the number of stationary points located increases exponentially with increasing lattice size. The corresponding disconnectivity graphs exhibit funneled landscapes; the global minima are readily located because they exhibit relatively large basins of attraction compared to the higher energy minima as the lattice size increases.