1
|
Lucibello C, Mézard M. Exponential Capacity of Dense Associative Memories. PHYSICAL REVIEW LETTERS 2024; 132:077301. [PMID: 38427855 DOI: 10.1103/physrevlett.132.077301] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/26/2023] [Revised: 11/29/2023] [Accepted: 01/11/2024] [Indexed: 03/03/2024]
Abstract
Recent generalizations of the Hopfield model of associative memories are able to store a number P of random patterns that grows exponentially with the number N of neurons, P=exp(αN). Besides the huge storage capacity, another interesting feature of these networks is their connection to the attention mechanism which is part of the Transformer architecture widely applied in deep learning. In this work, we study a generic family of pattern ensembles using a statistical mechanics analysis which gives exact asymptotic thresholds for the retrieval of a typical pattern, α_{1}, and lower bounds for the maximum of the load α for which all patterns can be retrieved, α_{c}, as well as sizes of attraction basins. We discuss in detail the cases of Gaussian and spherical patterns, and show that they display rich and qualitatively different phase diagrams.
Collapse
Affiliation(s)
- Carlo Lucibello
- Department of Computing Sciences, Bocconi University, Milano 20136, Italy and Bocconi Institute for Data Science and Analytics (BIDSA), Milano 20136, Italy
| | - Marc Mézard
- Department of Computing Sciences, Bocconi University, Milano 20136, Italy and Bocconi Institute for Data Science and Analytics (BIDSA), Milano 20136, Italy
| |
Collapse
|
2
|
Le Doussal P. Dynamics at the edge for independent diffusing particles. Phys Rev E 2024; 109:024101. [PMID: 38491623 DOI: 10.1103/physreve.109.024101] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2023] [Accepted: 01/02/2024] [Indexed: 03/18/2024]
Abstract
We study the dynamics of the outliers for a large number of independent Brownian particles in one dimension. We derive the multitime joint distribution of the position of the rightmost particle, by two different methods. We obtain the two-time joint distribution of the maximum and second maximum positions, and we study the counting statistics at the edge. Finally, we derive the multitime joint distribution of the running maximum, as well as the joint distribution of the arrival time of the first particle at several space points.
Collapse
Affiliation(s)
- Pierre Le Doussal
- Laboratoire de Physique de l'École Normale Supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université de Paris, 75005 Paris, France
| |
Collapse
|
3
|
Chen WK, Handschy M, Lerman G. Phase transition in random tensors with multiple independent spikes. ANN APPL PROBAB 2021. [DOI: 10.1214/20-aap1636] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
4
|
Baik J, Lee JO. Free energy of bipartite spherical Sherrington–Kirkpatrick model. ANNALES DE L'INSTITUT HENRI POINCARÉ, PROBABILITÉS ET STATISTIQUES 2020. [DOI: 10.1214/20-aihp1062] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
5
|
|
6
|
Maillard P, Pain M. 1-stable fluctuations in branching Brownian motion at critical temperature I: The derivative martingale. ANN PROBAB 2019. [DOI: 10.1214/18-aop1329] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
7
|
Grabchak M, Molchanov SA. Limit Theorems for Random Exponentials: The Bounded Support Case. THEORY OF PROBABILITY AND ITS APPLICATIONS 2019. [DOI: 10.1137/s0040585x97t989295] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|
8
|
|
9
|
A central limit theorem and a law of the iterated logarithm for the Biggins martingale of the supercritical branching random walk. J Appl Probab 2016. [DOI: 10.1017/jpr.2016.73] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Abstract
Let (Wn(θ))n∈ℕ0 be the Biggins martingale associated with a supercritical branching random walk, and denote by W_∞(θ) its limit. Assuming essentially that the martingale (Wn(2θ))n∈ℕ0 is uniformly integrable and that var W1(θ) is finite, we prove a functional central limit theorem for the tail process (W∞(θ)-Wn+r(θ))r∈ℕ0 and a law of the iterated logarithm for W∞(θ)-Wn(θ) as n→∞.
Collapse
|
10
|
Alberts T, Rifkind B. Diffusions of multiplicative cascades. Stoch Process Their Appl 2014. [DOI: 10.1016/j.spa.2013.10.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
|
11
|
Kabluchko Z, Klimovsky A. Complex random energy model: zeros and fluctuations. Probab Theory Relat Fields 2013. [DOI: 10.1007/s00440-013-0480-5] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/27/2022]
|
12
|
Meiners R, Reichenbachs A. On the accuracy of the normal approximation for the free energy in the Random Energy Model. ELECTRONIC COMMUNICATIONS IN PROBABILITY 2013. [DOI: 10.1214/ecp.v18-2377] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
13
|
Suárez A, Silbey R, Oppenheim I. Phase transition in the Jarzynski estimator of free energy differences. PHYSICAL REVIEW. E, STATISTICAL, NONLINEAR, AND SOFT MATTER PHYSICS 2012; 85:051108. [PMID: 23004704 DOI: 10.1103/physreve.85.051108] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/29/2011] [Revised: 02/07/2012] [Indexed: 06/01/2023]
Abstract
The transition between a regime in which thermodynamic relations apply only to ensembles of small systems coupled to a large environment and a regime in which they can be used to characterize individual macroscopic systems is analyzed in terms of the change in behavior of the Jarzynski estimator of equilibrium free energy differences from nonequilibrium work measurements. Given a fixed number of measurements, the Jarzynski estimator is unbiased for sufficiently small systems. In these systems the directionality of time is poorly defined and the configurations that dominate the empirical average, but which are in fact typical of the reverse process, are sufficiently well sampled. As the system size increases the arrow of time becomes better defined. The dominant atypical fluctuations become rare and eventually cannot be sampled with the limited resources that are available. Asymptotically, only typical work values are measured. The Jarzynski estimator becomes maximally biased and approaches the exponential of minus the average work, which is the result that is expected from standard macroscopic thermodynamics. In the proper scaling limit, this regime change has been recently described in terms of a phase transition in variants of the random energy model. In this paper this correspondence is further demonstrated in two examples of physical interest: the sudden compression of an ideal gas and adiabatic quasistatic volume changes in a dilute real gas.
Collapse
Affiliation(s)
- Alberto Suárez
- Department of Chemistry, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, Massachusetts 02139, USA.
| | | | | |
Collapse
|
14
|
Palassini M, Ritort F. Improving free-energy estimates from unidirectional work measurements: theory and experiment. PHYSICAL REVIEW LETTERS 2011; 107:060601. [PMID: 21902307 DOI: 10.1103/physrevlett.107.060601] [Citation(s) in RCA: 22] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/14/2009] [Revised: 07/05/2011] [Indexed: 05/31/2023]
Abstract
We derive analytical expressions for the bias of the Jarzynski free-energy estimator from N nonequilibrium work measurements, for a generic work distribution. To achieve this, we map the estimator onto the random energy model in a suitable scaling limit parametrized by (logN)/μ, where μ measures the width of the lower tail of the work distribution, and then compute the finite-N corrections to this limit with different approaches for different regimes of (logN)/μ. We show that these expressions describe accurately the bias for a wide class of work distributions and exploit them to build an improved free-energy estimator from unidirectional work measurements. We apply the method to optical tweezers unfolding and refolding experiments on DNA hairpins of varying loop size and dissipation, displaying both near-Gaussian and non-Gaussian work distributions.
Collapse
Affiliation(s)
- Matteo Palassini
- Departament de FÃsica Fonamental, Universitat de Barcelona, Spain
| | | |
Collapse
|
15
|
Kabluchko Z. Functional limit theorems for sums of independent geometric Lévy processes. BERNOULLI 2011. [DOI: 10.3150/10-bej299] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
16
|
Limit Laws for Sums of Independent Random Products: the Lattice Case. J THEOR PROBAB 2010. [DOI: 10.1007/s10959-010-0296-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
17
|
|
18
|
Monthus C, Garel T. Critical weight statistics of the random energy model and of the directed polymer on the Cayley tree. PHYSICAL REVIEW. E, STATISTICAL, NONLINEAR, AND SOFT MATTER PHYSICS 2007; 75:051119. [PMID: 17677034 DOI: 10.1103/physreve.75.051119] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/02/2007] [Indexed: 05/16/2023]
Abstract
We consider the critical point of two mean-field disordered models: (i) the random energy model (REM), introduced by Derrida as a mean-field spin-glass model of N spins and (ii) the directed polymer of length N on a Cayley Tree (DPCT) with random bond energies. Both models are known to exhibit a freezing transition between a high-temperature phase where the entropy is extensive and a low-temperature phase of finite entropy, where the weight statistics coincides with the weight statistics of Lévy sums with index mu=TT{c}<1 . In this paper, we study the weight statistics at criticality via the entropy S=Sigma w{i}lnw{i} and the generalized moments Y{k}= Sigma w{i}{k} , where the w{i} are the Boltzmann weights of the 2{N} configurations. In the REM, we find that the critical weight statistics is governed by the finite-size exponent nu=2 : the entropy scales as S[over]{N}(T{c}) approximately N{12} , the typical values e{lnY{k}[over]} decay as N{-k2} , and the disorder-averaged values Y{k}[over] are governed by rare events and decay as N{-12} for any k>1 . For the DPCT, we find that the entropy scales similarly as S[over]{N}(T{c}) approximately N{12} , whereas another exponent nu'=1 governs the Y{k} statistics: the typical values e{lnY{k}[over]} decay as N{-k} , and the disorder-averaged values Y{k}[over] decay as N{-1} for any k>1 . As a consequence, the asymptotic probability distribution pi[over]{N=infinity}(q) of the overlap q , in addition to the delta function delta(q) , which bears the whole normalization, contains an isolated point at q=1 , as a memory of the delta peak (1-TT{c})delta(q-1) of the low-temperature phase T<T{c} . The associated value pi[over]{N=infinity}(q=1) is finite for the DPCT, and diverges as pi[over]{N=infinity}(q=1) approximately N{12} for the REM.
Collapse
Affiliation(s)
- Cécile Monthus
- Service de Physique Théorique, CEA/DSM/SPhT, CNRS, 91191 Gif-sur-Yvette cedex, France
| | | |
Collapse
|
19
|
Hanen A. Un théorème limite pour les covariances des spins dans le modèle de Sherrington–Kirkpatrick avec champ externe. ANN PROBAB 2007. [DOI: 10.1214/009117906000000665] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
20
|
|
21
|
Ben Arous G, Bogachev LV, Molchanov SA. Limit theorems for sums of random exponentials. Probab Theory Relat Fields 2005. [DOI: 10.1007/s00440-004-0406-3] [Citation(s) in RCA: 54] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/01/2022]
|
22
|
Comets F, Kurkova I, Trashorras J. Fluctuations of the free energy in the high temperature Hopfield model. Stoch Process Their Appl 2004. [DOI: 10.1016/j.spa.2004.03.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
|
23
|
Affiliation(s)
- Marie F. Kratz
- U.F.R. de Mathématiques et Informatique, Université René Descartes, Paris V
| | | |
Collapse
|