Energy landscapes for a machine learning application to series data

Subaging in underparametrized Deep Neural Networks

In this work, we consider a simple classification problem to show that the dynamics of finite--width Deep Neural Networks in the underparametrized regime gives rise to effects similar to those associated with glassy systems, namely a slow evolution of the loss function and aging. Remarkably, the aging is sublinear in the waiting time (subaging) and the power--law exponent characterizing it is robust to different architectures under the constraint of a constant total number of parameters. Our results are maintained in the more complex scenario of the MNIST database. We find that for this database there is a unique exponent ruling the subaging behavior in the whole phase.

Modeling microsolvation clusters with electronic-structure calculations guided by analytical potentials and predictive machine learning techniques

We propose a new methodology to study, at the density functional theory (DFT) level, the clusters resulting from the microsolvation of alkali-metal ions with rare-gas atoms. The workflow begins with a global optimization search to generate a pool of low-energy minimum structures for different cluster sizes. This is achieved by employing an analytical potential energy surface (PES) and an evolutionary algorithm (EA). The next main stage of the methodology is devoted to establish an adequate DFT approach to treat the microsolvation system, through a systematic benchmark study involving several combinations of functionals and basis sets, in order to characterize the global minimum structures of the smaller clusters. In the next stage, we apply machine learning (ML) classification algorithms to predict how the low-energy minima of the analytical PES map to the DFT ones. An early and accurate detection of likely DFT local minima is extremely important to guide the choice of the most promising low-energy minima of large clusters to be re-optimized at the DFT level of theory. In this work, the methodology was applied to the Li+Krn (n = 2-14 and 16) microsolvation clusters for which the most competitive DFT approach was found to be the B3LYP-D3/aug-pcseg-1. Additionally, the ML classifier was able to accurately predict most of the solutions to be re-optimized at the DFT level of theory, thereby greatly enhancing the efficiency of the process and allowing its applicability to larger clusters.

Archetypal landscapes for deep neural networks

The predictive capabilities of deep neural networks (DNNs) continue to evolve to increasingly impressive levels. However, it is still unclear how training procedures for DNNs succeed in finding parameters that produce good results for such high-dimensional and nonconvex loss functions. In particular, we wish to understand why simple optimization schemes, such as stochastic gradient descent, do not end up trapped in local minima with high loss values that would not yield useful predictions. We explain the optimizability of DNNs by characterizing the local minima and transition states of the loss-function landscape (LFL) along with their connectivity. We show that the LFL of a DNN in the shallow network or data-abundant limit is funneled, and thus easy to optimize. Crucially, in the opposite low-data/deep limit, although the number of minima increases, the landscape is characterized by many minima with similar loss values separated by low barriers. This organization is different from the hierarchical landscapes of structural glass formers and explains why minimization procedures commonly employed by the machine-learning community can navigate the LFL successfully and reach low-lying solutions.

Perspective: new insights from loss function landscapes of neural networks

We investigate the structure of the loss function landscape for neural networks subject to dataset mislabelling, increased training set diversity, and reduced node connectivity, using various techniques developed for energy landscape exploration. The benchmarking models are classification problems for atomic geometry optimisation and hand-written digit prediction. We consider the effect of varying the size of the atomic configuration space used to generate initial geometries and find that the number of stationary points increases rapidly with the size of the training configuration space. We introduce a measure of node locality to limit network connectivity and perturb permutational weight symmetry, and examine how this parameter affects the resulting landscapes. We find that highly-reduced systems have low capacity and exhibit landscapes with very few minima. On the other hand, small amounts of reduced connectivity can enhance network expressibility and can yield more complex landscapes. Investigating the effect of deliberate classification errors in the training data, we find that the variance in testing AUC, computed over a sample of minima, grows significantly with the training error, providing new insight into the role of the variance-bias trade-off when training under noise. Finally, we illustrate how the number of local minima for networks with two and three hidden layers, but a comparable number of variable edge weights, increases significantly with the number of layers, and as the number of training data decreases. This work helps shed further light on neural network loss landscapes and provides guidance for future work on neural network training and optimisation.

Geometry of Energy Landscapes and the Optimizability of Deep Neural Networks

Deep neural networks are workhorse models in machine learning with multiple layers of nonlinear functions composed in series. Their loss function is highly nonconvex, yet empirically even gradient descent minimization is sufficient to arrive at accurate and predictive models. It is hitherto unknown why deep neural networks are easily optimizable. We analyze the energy landscape of a spin glass model of deep neural networks using random matrix theory and algebraic geometry. We analytically show that the multilayered structure holds the key to optimizability: Fixing the number of parameters and increasing network depth, the number of stationary points in the loss function decreases, minima become more clustered in parameter space, and the trade-off between the depth and width of minima becomes less severe. Our analytical results are numerically verified through comparison with neural networks trained on a set of classical benchmark datasets. Our model uncovers generic design principles of machine learning models.

The Structure of Adamantane Clusters: Atomistic vs. Coarse-Grained Predictions From Global Optimization

Candidate structures for the global minima of adamantane clusters, (C₁₀H₁₆)_N, are presented. Based on a rigid model for individual molecules with atom-atom pairwise interactions that include Lennard-Jones and Coulomb contributions, low-energy structures were obtained up to N = 42 using the basin-hopping method. The results indicate that adamantane clusters initially grow accordingly with an icosahedral packing scheme, followed above N = 14 by a structural transition toward face-centered cubic structures. The special stabilities obtained at N = 13, 19, and 38 are consistent with these two structural families, and agree with recent mass spectrometry measurements on cationic adamantane clusters. Coarse-graining the intermolecular potential by averaging over all possible orientations only partially confirm the all-atom results, the magic numbers at 13 and 38 being preserved. However, the details near the structural transition are not captured well, because despite their high symmetry the adamantane molecules are still rather anisotropic.

Exploring Energy Landscapes

Recent advances in the potential energy landscapes approach are highlighted, including both theoretical and computational contributions. Treating the high dimensionality of molecular and condensed matter systems of contemporary interest is important for understanding how emergent properties are encoded in the landscape and for calculating these properties while faithfully representing barriers between different morphologies. The pathways characterized in full dimensionality, which are used to construct kinetic transition networks, may prove useful in guiding such calculations. The energy landscape perspective has also produced new procedures for structure prediction and analysis of thermodynamic properties. Basin-hopping global optimization, with alternative acceptance criteria and generalizations to multiple metric spaces, has been used to treat systems ranging from biomolecules to nanoalloy clusters and condensed matter. This review also illustrates how all this methodology, developed in the context of chemical physics, can be transferred to landscapes defined by cost functions associated with machine learning.

Loss surface of XOR artificial neural networks

Training an artificial neural network involves an optimization process over the landscape defined by the cost (loss) as a function of the network parameters. We explore these landscapes using optimization tools developed for potential energy landscapes in molecular science. The number of local minima and transition states (saddle points of index one), as well as the ratio of transition states to minima, grow rapidly with the number of nodes in the network. There is also a strong dependence on the regularization parameter, with the landscape becoming more convex (fewer minima) as the regularization term increases. We demonstrate that in our formulation, stationary points for networks with N_{h} hidden nodes, including the minimal network required to fit the XOR data, are also stationary points for networks with N_{h}+1 hidden nodes when all the weights involving the additional node are zero. Hence, smaller networks trained on XOR data are embedded in the landscapes of larger networks. Our results clarify certain aspects of the classification and sensitivity (to perturbations in the input data) of minima and saddle points for this system, and may provide insight into dropout and network compression.

Energy landscapes for machine learning

Machine learning techniques are being increasingly used as flexible non-linear fitting and prediction tools in the physical sciences. Fitting functions that exhibit multiple solutions as local minima can be analysed in terms of the corresponding machine learning landscape. Methods to explore and visualise molecular potential energy landscapes can be applied to these machine learning landscapes to gain new insight into the solution space involved in training and the nature of the corresponding predictions. In particular, we can define quantities analogous to molecular structure, thermodynamics, and kinetics, and relate these emergent properties to the structure of the underlying landscape. This Perspective aims to describe these analogies with examples from recent applications, and suggest avenues for new interdisciplinary research.

Machine learning landscapes and predictions for patient outcomes

The theory and computational tools developed to interpret and explore energy landscapes in molecular science are applied to the landscapes defined by local minima for neural networks. These machine learning landscapes correspond to fits of training data, where the inputs are vital signs and laboratory measurements for a database of patients, and the objective is to predict a clinical outcome. In this contribution, we test the predictions obtained by fitting to single measurements, and then to combinations of between 2 and 10 different patient medical data items. The effect of including measurements over different time intervals from the 48 h period in question is analysed, and the most recent values are found to be the most important. We also compare results obtained for neural networks as a function of the number of hidden nodes, and for different values of a regularization parameter. The predictions are compared with an alternative convex fitting function, and a strong correlation is observed. The dependence of these results on the patients randomly selected for training and testing decreases systematically with the size of the database available. The machine learning landscapes defined by neural network fits in this investigation have single-funnel character, which probably explains why it is relatively straightforward to obtain the global minimum solution, or a fit that behaves similarly to this optimal parameterization.

Nanothermodynamics of iron clusters: Small clusters, icosahedral and fcc-cuboctahedral structures

The study of the thermodynamics and structures of iron clusters has been carried on, focusing on small clusters and initial icosahedral and fcc-cuboctahedral structures. Two combined tools are used. First, energy intervals are explored by the Monte Carlo algorithm, called σ-mapping, detailed in the work of Soudan et al. [J. Chem. Phys. 135, 144109 (2011), Paper I]. In its flat histogram version, it provides the classical density of states, g_p(E_p), in terms of the potential energy of the system. Second, the iron system is described by a potential which is called "corrected EAM" (cEAM), explained in the work of Basire et al. [J. Chem. Phys. 141, 104304 (2014), Paper II]. Small clusters from 3 to 12 atoms in their ground state have been compared first with published Density Functional Theory (DFT) calculations, giving a complete agreement of geometries. The series of 13, 55, 147, and 309 atom icosahedrons is shown to be the most stable form for the cEAM potential. However, the 147 atom cluster has a special behaviour, since decreasing the energy from the liquid zone leads to the irreversible trapping of the cluster in a reproducible amorphous state, 7.38 eV higher in energy than the icosahedron. This behaviour is not observed at the higher size of 309 atoms. The heat capacity of the 55, 147, and 309 atom clusters revealed a pronounced peak in the solid zone, related to a solid-solid transition, prior to the melting peak. The corresponding series of 13, 55, and 147 atom cuboctahedrons has been compared, underscoring the unstability towards the icosahedral structure. This unstability occurs clearly in several steps for the 147 atom cluster, with a sudden transformation at a transition state. This illustrates the concerted icosahedron-cuboctahedron transformation of Buckminster Fuller-Mackay, which is calculated for the cEAM potential. Two other clusters of initial fcc structures with 24 and 38 atoms have been studied, as well as a 302 atom cluster. Each one relaxes towards a more stable structure without regularity. The 38 atom cluster exhibits a nearly glassy relaxation, through a cascade of six metastable states of long life. This behaviour, as that of the 147 atom cluster towards the amorphous state, shows that difficulties to reach ergodicity in the lower half of the solid zone are related to particular features of the potential energy landscape, and not necessarily to a too large size of the system. Comparisons of the cEAM iron system with published results about Lennard-Jones systems and DFT calculations are made. The results of the previous clusters have been combined with that of Paper II to plot the cohesive energy E_c and the melting temperature T_m in terms of the cluster atom number N_at. The N_at^-1/3 linear dependence of the melting temperature (Pawlow law) is observed again for N_at > 150. In contrast, for N_at < 150, the curve diverges strongly from the Pawlow law, giving it an overall V-shape, with a linear increase of T_m when N_at goes from 55 to 13 atoms. Surprisingly, the 38 atom cluster is anomalously below the overall curve.

Decoding heat capacity features from the energy landscape

A general scheme is derived to connect transitions in configuration space with features in the heat capacity. A formulation in terms of occupation probabilities for local minima that define the potential energy landscape provides a quantitative description of how contributions arise from competition between different states. The theory does not rely on a structural interpretation for the local minima, so it is equally applicable to molecular energy landscapes and the landscapes defined by abstract functions. Applications are presented for low-temperature solid-solid transitions in atomic clusters, which involve just a few local minima with different morphologies, and for cluster melting, which is driven by the landscape entropy associated with the more numerous high-energy minima. Analyzing these features in terms of the balance between states with increasing and decreasing occupation probabilities provides a direct interpretation of the underlying transitions. This approach enables us to identify a qualitatively different transition that is caused by a single local minimum associated with an exceptionally large catchment volume in configuration space for a machine learning landscape.

Structural analysis of high-dimensional basins of attraction

We propose an efficient Monte Carlo method for the computation of the volumes of high-dimensional bodies with arbitrary shape. We start with a region of known volume within the interior of the manifold and then use the multistate Bennett acceptance-ratio method to compute the dimensionless free-energy difference between a series of equilibrium simulations performed within this object. The method produces results that are in excellent agreement with thermodynamic integration, as well as a direct estimate of the associated statistical uncertainties. The histogram method also allows us to directly obtain an estimate of the interior radial probability density profile, thus yielding useful insight into the structural properties of such a high-dimensional body. We illustrate the method by analyzing the effect of structural disorder on the basins of attraction of mechanically stable packings of soft repulsive spheres.

Energy landscapes for a machine-learning prediction of patient discharge

The energy landscapes framework is applied to a configuration space generated by training the parameters of a neural network. In this study the input data consists of time series for a collection of vital signs monitored for hospital patients, and the outcomes are patient discharge or continued hospitalisation. Using machine learning as a predictive diagnostic tool to identify patterns in large quantities of electronic health record data in real time is a very attractive approach for supporting clinical decisions, which have the potential to improve patient outcomes and reduce waiting times for discharge. Here we report some preliminary analysis to show how machine learning might be applied. In particular, we visualize the fitting landscape in terms of locally optimal neural networks and the connections between them in parameter space. We anticipate that these results, and analogues of thermodynamic properties for molecular systems, may help in the future design of improved predictive tools.