Structural analysis of high-dimensional basins of attraction

100 Years of the Lennard-Jones Potential

It is now 100 years since Lennard-Jones published his first paper introducing the now famous potential that bears his name. It is therefore timely to reflect on the many achievements, as well as the limitations, of this potential in the theory of atomic and molecular interactions, where applications range from descriptions of intermolecular forces to molecules, clusters, and condensed matter.

Basins with Tentacles

To explore basin geometry in high-dimensional dynamical systems, we consider a ring of identical Kuramoto oscillators. Many attractors coexist in this system; each is a twisted periodic orbit characterized by a winding number q, with basin size proportional to e^{-kq^{2}}. We uncover the geometry behind this size distribution and find the basins are octopuslike, with nearly all their volume in the tentacles, not the head of the octopus (the ball-like region close to the attractor). We present a simple geometrical reason why basins with tentacles should be common in high-dimensional systems.

Geometry of Energy Landscapes and the Optimizability of Deep Neural Networks

Deep neural networks are workhorse models in machine learning with multiple layers of nonlinear functions composed in series. Their loss function is highly nonconvex, yet empirically even gradient descent minimization is sufficient to arrive at accurate and predictive models. It is hitherto unknown why deep neural networks are easily optimizable. We analyze the energy landscape of a spin glass model of deep neural networks using random matrix theory and algebraic geometry. We analytically show that the multilayered structure holds the key to optimizability: Fixing the number of parameters and increasing network depth, the number of stationary points in the loss function decreases, minima become more clustered in parameter space, and the trade-off between the depth and width of minima becomes less severe. Our analytical results are numerically verified through comparison with neural networks trained on a set of classical benchmark datasets. Our model uncovers generic design principles of machine learning models.

Exploring Energy Landscapes

Recent advances in the potential energy landscapes approach are highlighted, including both theoretical and computational contributions. Treating the high dimensionality of molecular and condensed matter systems of contemporary interest is important for understanding how emergent properties are encoded in the landscape and for calculating these properties while faithfully representing barriers between different morphologies. The pathways characterized in full dimensionality, which are used to construct kinetic transition networks, may prove useful in guiding such calculations. The energy landscape perspective has also produced new procedures for structure prediction and analysis of thermodynamic properties. Basin-hopping global optimization, with alternative acceptance criteria and generalizations to multiple metric spaces, has been used to treat systems ranging from biomolecules to nanoalloy clusters and condensed matter. This review also illustrates how all this methodology, developed in the context of chemical physics, can be transferred to landscapes defined by cost functions associated with machine learning.

Energy landscapes for machine learning

Machine learning techniques are being increasingly used as flexible non-linear fitting and prediction tools in the physical sciences. Fitting functions that exhibit multiple solutions as local minima can be analysed in terms of the corresponding machine learning landscape. Methods to explore and visualise molecular potential energy landscapes can be applied to these machine learning landscapes to gain new insight into the solution space involved in training and the nature of the corresponding predictions. In particular, we can define quantities analogous to molecular structure, thermodynamics, and kinetics, and relate these emergent properties to the structure of the underlying landscape. This Perspective aims to describe these analogies with examples from recent applications, and suggest avenues for new interdisciplinary research.

Monte Carlo sampling for stochastic weight functions

Conventional Monte Carlo simulations are stochastic in the sense that the acceptance of a trial move is decided by comparing a computed acceptance probability with a random number, uniformly distributed between 0 and 1. Here, we consider the case that the weight determining the acceptance probability itself is fluctuating. This situation is common in many numerical studies. We show that it is possible to construct a rigorous Monte Carlo algorithm that visits points in state space with a probability proportional to their average weight. The same approach may have applications for certain classes of high-throughput experiments and the analysis of noisy datasets.

Affine and topogical structural entropies in granular statistical mechanics: Explicit calculations and equation of state

We identify two orthogonal sources of structural entropy in rattler-free granular systems: affine, involving structural changes that only deform the contact network, and topological, corresponding to different topologies of the contact network. We show that a recently developed connectivity-based granular statistical mechanics separates the two naturally by identifying the structural degrees of freedom with spanning trees on the graph of the contact network. We extend the connectivity-based formalism to include constraints on, and correlations between, degrees of freedom as interactions between branches of the spanning tree. We then use the statistical mechanics formalism to calculate the partition function generally and the different entropies in the high-angoricity limit. We also calculate the degeneracy of the affine entropy and a number of expectation values. From the latter, we derive an equipartition principle and an equation of state relating the macroscopic volume and boundary stress to the analog of the temperature, the contactivity.