Anomalous diffusion dynamics of learning in deep neural networks

Brain-inspired chaotic spiking backpropagation

Spiking neural networks (SNNs) have superior energy efficiency due to their spiking signal transmission, which mimics biological nervous systems, but they are difficult to train effectively. Although surrogate gradient-based methods offer a workable solution, trained SNNs frequently fall into local minima because they are still primarily based on gradient dynamics. Inspired by the chaotic dynamics in animal brain learning, we propose a chaotic spiking backpropagation (CSBP) method that introduces a loss function to generate brain-like chaotic dynamics and further takes advantage of the ergodic and pseudo-random nature to make SNN learning effective and robust. From a computational viewpoint, we found that CSBP significantly outperforms current state-of-the-art methods on both neuromorphic data sets (e.g. DVS-CIFAR10 and DVS-Gesture) and large-scale static data sets (e.g. CIFAR100 and ImageNet) in terms of accuracy and robustness. From a theoretical viewpoint, we show that the learning process of CSBP is initially chaotic, then subject to various bifurcations and eventually converges to gradient dynamics, consistently with the observation of animal brain activity. Our work provides a superior core tool for direct SNN training and offers new insights into understanding the learning process of a biological brain.

Star-Shaped Space of Solutions of the Spherical Negative Perceptron

Empirical studies on the landscape of neural networks have shown that low-energy configurations are often found in complex connected structures, where zero-energy paths between pairs of distant solutions can be constructed. Here, we consider the spherical negative perceptron, a prototypical nonconvex neural network model framed as a continuous constraint satisfaction problem. We introduce a general analytical method for computing energy barriers in the simplex with vertex configurations sampled from the equilibrium. We find that in the overparametrized regime the solution manifold displays simple connectivity properties. There exists a large geodesically convex component that is attractive for a wide range of optimization dynamics. Inside this region we identify a subset of atypical high-margin solutions that are geodesically connected with most other solutions, giving rise to a star-shaped geometry. We analytically characterize the organization of the connected space of solutions and show numerical evidence of a transition, at larger constraint densities, where the aforementioned simple geodesic connectivity breaks down.

Experimental observations of fractal landscape dynamics in a dense emulsion

Many soft and biological materials display so-called 'soft glassy' dynamics; their constituents undergo anomalous random motions and complex cooperative rearrangements. A recent simulation model of one soft glassy material, a coarsening foam, suggested that the random motions of its bubbles are due to the system configuration moving over a fractal energy landscape in high-dimensional space. Here we show that the salient geometrical features of such high-dimensional fractal landscapes can be explored and reliably quantified, using empirical trajectory data from many degrees of freedom, in a model-free manner. For a mayonnaise-like dense emulsion, analysis of the observed trajectories of oil droplets quantitatively reproduces the high-dimensional fractal geometry of the configuration path and its associated local energy minima generated using a computational model. That geometry in turn drives the droplets' complex random motion observed in real space. Our results indicate that experimental studies can elucidate whether the similar dynamics in different soft and biological materials may also be due to fractal landscape dynamics.

Exploring canyons in glassy energy landscapes using metadynamics

The complex physics of glass-forming systems is controlled by the structure of the low-energy portions of their potential energy landscapes. Here we report that a modified metadynamics algorithm efficiently explores and samples low-energy regions of such high-dimensional landscapes. In the energy landscape for a model foam, our algorithm finds and descends meandering canyons in the landscape, which contain dense clusters of energy minima along their floors. Similar canyon structures in the energy landscapes of two model glass formers—hard sphere fluids and the Kob–Andersen glass—allow us to reach high densities and low energies, respectively. In the hard sphere system, fluid configurations are found to form continuous regions that cover the canyon floors up to densities well above the jamming transition. For the Kob–Andersen glass former, our technique samples low-energy states with modest computational effort, with the lowest energies found approaching the predicted Kauzmann limit.

Subaging in underparametrized Deep Neural Networks

In this work, we consider a simple classification problem to show that the dynamics of finite--width Deep Neural Networks in the underparametrized regime gives rise to effects similar to those associated with glassy systems, namely a slow evolution of the loss function and aging. Remarkably, the aging is sublinear in the waiting time (subaging) and the power--law exponent characterizing it is robust to different architectures under the constraint of a constant total number of parameters. Our results are maintained in the more complex scenario of the MNIST database. We find that for this database there is a unique exponent ruling the subaging behavior in the whole phase.