1
|
Skinner DJ, Lemaire P, Mani M. Physical modeling of embryonic transcriptomes identifies collective modes of gene expression. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.07.26.605398. [PMID: 39131269 PMCID: PMC11312445 DOI: 10.1101/2024.07.26.605398] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/13/2024]
Abstract
Starting from one totipotent cell, complex multicellular organisms form through a series of differentiation and morphogenetic events, culminating in a multitude of cell types arranged in a functional and intricate spatial pattern. To do so, cells coordinate with each other, resulting in dynamics which follow a precise developmental trajectory, constraining the space of possible embryo-to-embryo variation. Using recent single-cell sequencing data of early ascidian embryos, we leverage natural variation together with modeling and inference techniques from statistical physics to investigate development at the level of a complete interconnected embryo - an embryonic transcriptome. After developing a robust and biophysically motivated approach to identifying distinct transcriptomic states or cell types, a statistical analysis reveals correlations within embryos and across cell types demonstrating the presence of collective variation. From these intra-embryo correlations, we infer minimal networks of cell-cell interactions, which reveal the collective modes of gene expression. Our work demonstrates how the existence and nature of spatial interactions along with the collective modes of expression that they give rise to can be inferred from single-cell gene expression measurements, opening up a wider range of biological questions that can be addressed using sequencing-based modalities.
Collapse
Affiliation(s)
- Dominic J Skinner
- Center for Computational Biology, Flatiron Institute, 162 5th Ave, New York, NY 10010, USA
- NSF-Simons Center for Quantitative Biology, Northwestern University, 2205 Tech Drive, Evanston, IL 60208, USA
| | - Patrick Lemaire
- CRBM, Université de Montpellier, CNRS, 34293 Montpellier, France
| | - Madhav Mani
- NSF-Simons Center for Quantitative Biology, Northwestern University, 2205 Tech Drive, Evanston, IL 60208, USA
- Department of Engineering Sciences and Applied Mathematics, Northwestern University, Evanston, IL 60208, USA
| |
Collapse
|
2
|
Kloucek MB, Machon T, Kajimura S, Royall CP, Masuda N, Turci F. Biases in inverse Ising estimates of near-critical behavior. Phys Rev E 2023; 108:014109. [PMID: 37583208 DOI: 10.1103/physreve.108.014109] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2023] [Accepted: 04/27/2023] [Indexed: 08/17/2023]
Abstract
Inverse Ising inference allows pairwise interactions of complex binary systems to be reconstructed from empirical correlations. Typical estimators used for this inference, such as pseudo-likelihood maximization (PLM), are biased. Using the Sherrington-Kirkpatrick model as a benchmark, we show that these biases are large in critical regimes close to phase boundaries, and they may alter the qualitative interpretation of the inferred model. In particular, we show that the small-sample bias causes models inferred through PLM to appear closer to criticality than one would expect from the data. Data-driven methods to correct this bias are explored and applied to a functional magnetic resonance imaging data set from neuroscience. Our results indicate that additional care should be taken when attributing criticality to real-world data sets.
Collapse
Affiliation(s)
- Maximilian B Kloucek
- School of Physics, HH Wills Physics Laboratory, University of Bristol, Tyndall Avenue, Bristol BS8 1TL, United Kingdom
- Bristol Centre for Functional Nanomaterials, HH Wills Physics Laboratory, University of Bristol, Tyndall Avenue, Bristol BS8 1TL, United Kingdom
| | - Thomas Machon
- School of Physics, HH Wills Physics Laboratory, University of Bristol, Tyndall Avenue, Bristol BS8 1TL, United Kingdom
| | - Shogo Kajimura
- Faculty of Information and Human Sciences, Kyoto Institute of Technology, Kyoto 606-8585, Japan
| | - C Patrick Royall
- Gulliver UMR CNRS 7083, ESPCI Paris, Université PSL, 75005 Paris, France
| | - Naoki Masuda
- Department of Mathematics, State University of New York at Buffalo, Buffalo, New York 14260-2900, USA
- Computational and Data-Enabled Science and Engineering Program, State University of New York at Buffalo, Buffalo, New York 14260-5030, USA
| | - Francesco Turci
- School of Physics, HH Wills Physics Laboratory, University of Bristol, Tyndall Avenue, Bristol BS8 1TL, United Kingdom
| |
Collapse
|
3
|
Yasuda M, Takahashi C. Free energy evaluation using marginalized annealed importance sampling. Phys Rev E 2022; 106:024127. [PMID: 36110019 DOI: 10.1103/physreve.106.024127] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2022] [Accepted: 08/03/2022] [Indexed: 06/15/2023]
Abstract
The evaluation of the free energy of a stochastic model is considered a significant issue in various fields of physics and machine learning. However, the exact free energy evaluation is computationally infeasible because the free energy expression includes an intractable partition function. Annealed importance sampling (AIS) is a type of importance sampling based on the Markov chain Monte Carlo method that is similar to a simulated annealing and can effectively approximate the free energy. This study proposes an AIS-based approach, which is referred to as marginalized AIS (mAIS). The statistical efficiency of mAIS is investigated in detail based on theoretical and numerical perspectives. Based on the investigation, it is proved that mAIS is more effective than AIS under a certain condition.
Collapse
Affiliation(s)
- Muneki Yasuda
- Graduate School of Science and Engineering, Yamagata University, Yonezawa, Yamagata 992-8510, Japan
| | - Chako Takahashi
- Graduate School of Science and Engineering, Yamagata University, Yonezawa, Yamagata 992-8510, Japan
| |
Collapse
|
4
|
Ngampruetikorn V, Sachdeva V, Torrence J, Humplik J, Schwab DJ, Palmer SE. Inferring couplings in networks across order-disorder phase transitions. PHYSICAL REVIEW RESEARCH 2022; 4:023240. [PMID: 37576946 PMCID: PMC10421637 DOI: 10.1103/physrevresearch.4.023240] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 08/15/2023]
Abstract
Statistical inference is central to many scientific endeavors, yet how it works remains unresolved. Answering this requires a quantitative understanding of the intrinsic interplay between statistical models, inference methods, and the structure in the data. To this end, we characterize the efficacy of direct coupling analysis (DCA) - a highly successful method for analyzing amino acid sequence data-in inferring pairwise interactions from samples of ferromagnetic Ising models on random graphs. Our approach allows for physically motivated exploration of qualitatively distinct data regimes separated by phase transitions. We show that inference quality depends strongly on the nature of data-generating distributions: optimal accuracy occurs at an intermediate temperature where the detrimental effects from macroscopic order and thermal noise are minimal. Importantly our results indicate that DCA does not always outperform its local-statistics-based predecessors; while DCA excels at low temperatures, it becomes inferior to simple correlation thresholding at virtually all temperatures when data are limited. Our findings offer insights into the regime in which DCA operates so successfully, and more broadly, how inference interacts with the structure in the data.
Collapse
Affiliation(s)
- Vudtiwat Ngampruetikorn
- Initiative for the Theoretical Sciences, The Graduate Center, CUNY, New York, New York 10016, USA
| | - Vedant Sachdeva
- Department of Organismal Biology and Anatomy and Department of Physics, University of Chicago, Chicago, Illinois 60637, USA
| | - Johanna Torrence
- Department of Organismal Biology and Anatomy and Department of Physics, University of Chicago, Chicago, Illinois 60637, USA
| | - Jan Humplik
- Institute of Science and Technology Austria, 3400 Klosterneuburg, Austria
| | - David J Schwab
- Initiative for the Theoretical Sciences, The Graduate Center, CUNY, New York, New York 10016, USA
| | - Stephanie E Palmer
- Department of Organismal Biology and Anatomy and Department of Physics, University of Chicago, Chicago, Illinois 60637, USA
| |
Collapse
|
5
|
Optimal Population Coding for Dynamic Input by Nonequilibrium Networks. ENTROPY 2022; 24:e24050598. [PMID: 35626482 PMCID: PMC9140425 DOI: 10.3390/e24050598] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/10/2022] [Revised: 04/07/2022] [Accepted: 04/19/2022] [Indexed: 12/04/2022]
Abstract
The efficient coding hypothesis states that neural response should maximize its information about the external input. Theoretical studies focus on optimal response in single neuron and population code in networks with weak pairwise interactions. However, more biological settings with asymmetric connectivity and the encoding for dynamical stimuli have not been well-characterized. Here, we study the collective response in a kinetic Ising model that encodes the dynamic input. We apply gradient-based method and mean-field approximation to reconstruct networks given the neural code that encodes dynamic input patterns. We measure network asymmetry, decoding performance, and entropy production from networks that generate optimal population code. We analyze how stimulus correlation, time scale, and reliability of the network affect optimal encoding networks. Specifically, we find network dynamics altered by statistics of the dynamic input, identify stimulus encoding strategies, and show optimal effective temperature in the asymmetric networks. We further discuss how this approach connects to the Bayesian framework and continuous recurrent neural networks. Together, these results bridge concepts of nonequilibrium physics with the analyses of dynamics and coding in networks.
Collapse
|
6
|
Enhancing computational enzyme design by a maximum entropy strategy. Proc Natl Acad Sci U S A 2022; 119:2122355119. [PMID: 35135886 PMCID: PMC8851541 DOI: 10.1073/pnas.2122355119] [Citation(s) in RCA: 17] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/03/2022] [Indexed: 01/16/2023] Open
Abstract
Although computational enzyme design is of great importance, the advances utilizing physics-based approaches have been slow, and further progress is urgently needed. One promising direction is using machine learning, but such strategies have not been established as effective tools for predicting the catalytic power of enzymes. Here, we show that the statistical energy inferred from homologous sequences with the maximum entropy (MaxEnt) principle significantly correlates with enzyme catalysis and stability at the active site region and the more distant region, respectively. This finding decodes enzyme architecture and offers a connection between enzyme evolution and the physical chemistry of enzyme catalysis, and it deepens our understanding of the stability-activity trade-off hypothesis for enzymes. Overall, the strong correlations found here provide a powerful way of guiding enzyme design.
Collapse
|
7
|
Abstract
The “Ising model” refers to both the statistical and the theoretical use of the same equation. In this article, we introduce both uses and contrast their differences. We accompany the conceptual introduction with a survey of Ising-related software packages in R. Since the model’s different uses are best understood through simulations, we make this process easily accessible with fully reproducible examples. Using simulations, we show how the theoretical Ising model captures local-alignment dynamics. Subsequently, we present it statistically as a likelihood function for estimating empirical network models from binary data. In this process, we give recommendations on when to use traditional frequentist estimators as well as novel Bayesian options.
Collapse
|
8
|
Equity Market Description under High and Low Volatility Regimes Using Maximum Entropy Pairwise Distribution. ENTROPY 2021; 23:e23101307. [PMID: 34682031 PMCID: PMC8534913 DOI: 10.3390/e23101307] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 09/12/2021] [Revised: 09/29/2021] [Accepted: 10/01/2021] [Indexed: 11/17/2022]
Abstract
The financial market is a complex system in which the assets influence each other, causing, among other factors, price interactions and co-movement of returns. Using the Maximum Entropy Principle approach, we analyze the interactions between a selected set of stock assets and equity indices under different high and low return volatility episodes at the 2008 Subprime Crisis and the 2020 COVID-19 outbreak. We carry out an inference process to identify the interactions, in which we implement the a pairwise Ising distribution model describing the first and second moments of the distribution of the discretized returns of each asset. Our results indicate that second-order interactions explain more than 80% of the entropy in the system during the Subprime Crisis and slightly higher than 50% during the COVID-19 outbreak independently of the period of high or low volatility analyzed. The evidence shows that during these periods, slight changes in the second-order interactions are enough to induce large changes in assets correlations but the proportion of positive and negative interactions remains virtually unchanged. Although some interactions change signs, the proportion of these changes are the same period to period, which keeps the system in a ferromagnetic state. These results are similar even when analyzing triadic structures in the signed network of couplings.
Collapse
|
9
|
Yasuda M, Sekimoto K. Spatial Monte Carlo integration with annealed importance sampling. Phys Rev E 2021; 103:052118. [PMID: 34134233 DOI: 10.1103/physreve.103.052118] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/25/2020] [Accepted: 04/13/2021] [Indexed: 11/07/2022]
Abstract
Evaluating expectations on an Ising model (or Boltzmann machine) is essential for various applications, including statistical machine learning. However, in general, the evaluation is computationally difficult because it involves intractable multiple summations or integrations; therefore, it requires approximation. Monte Carlo integration (MCI) is a well-known approximation method; a more effective MCI-like approximation method was proposed recently, called spatial Monte Carlo integration (SMCI). However, the estimations obtained using SMCI (and MCI) exhibit a low accuracy in Ising models under a low temperature owing to degradation of the sampling quality. Annealed importance sampling (AIS) is a type of importance sampling based on Markov chain Monte Carlo methods that can suppress performance degradation in low-temperature regions with the force of importance weights. In this study, a method is proposed to evaluate the expectations on Ising models combining AIS and SMCI. The proposed method performs efficiently in both high- and low-temperature regions, which is demonstrated theoretically and numerically.
Collapse
Affiliation(s)
- Muneki Yasuda
- Graduate School of Science and Engineering, Yamagata University, Japan
| | - Kaiji Sekimoto
- Graduate School of Science and Engineering, Yamagata University, Japan
| |
Collapse
|
10
|
Aguilera M, Moosavi SA, Shimazaki H. A unifying framework for mean-field theories of asymmetric kinetic Ising systems. Nat Commun 2021; 12:1197. [PMID: 33608507 PMCID: PMC7895831 DOI: 10.1038/s41467-021-20890-5] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2020] [Accepted: 12/29/2020] [Indexed: 01/15/2023] Open
Abstract
Kinetic Ising models are powerful tools for studying the non-equilibrium dynamics of complex systems. As their behavior is not tractable for large networks, many mean-field methods have been proposed for their analysis, each based on unique assumptions about the system’s temporal evolution. This disparity of approaches makes it challenging to systematically advance mean-field methods beyond previous contributions. Here, we propose a unifying framework for mean-field theories of asymmetric kinetic Ising systems from an information geometry perspective. The framework is built on Plefka expansions of a system around a simplified model obtained by an orthogonal projection to a sub-manifold of tractable probability distributions. This view not only unifies previous methods but also allows us to develop novel methods that, in contrast with traditional approaches, preserve the system’s correlations. We show that these new methods can outperform previous ones in predicting and assessing network properties near maximally fluctuating regimes. Many mean-field theories are proposed for studying the non-equilibrium dynamics of complex systems, each based on specific assumptions about the system’s temporal evolution. Here, Aguilera et al. propose a unified framework for mean-field theories of asymmetric kinetic Ising systems to study non-equilibrium dynamics.
Collapse
Affiliation(s)
- Miguel Aguilera
- IAS-Research Center for Life, Mind, and Society, Department of Logic and Philosophy of Science, University of the Basque Country, Donostia, Spain. .,Department of Informatics & Sussex Neuroscience, University of Sussex, Falmer, Brighton, UK. .,ISAAC Lab, Aragón Institute of Engineering Research (I3A), University of Zaragoza, Zaragoza, Spain.
| | - S Amin Moosavi
- Graduate School of Informatics, Kyoto University, Kyoto, Japan.,Department of Neuroscience, Brown University, Providence, RI, USA
| | - Hideaki Shimazaki
- Graduate School of Informatics, Kyoto University, Kyoto, Japan.,Center for Human Nature, Artificial Intelligence, and Neuroscience (CHAIN), Hokkaido University, Sapporo, Hokkaido, Japan
| |
Collapse
|
11
|
Ibanez-Berganza M, Amico A, Lancia GL, Maggiore F, Monechi B, Loreto V. Unsupervised inference approach to facial attractiveness. PeerJ 2020; 8:e10210. [PMID: 33194411 PMCID: PMC7602690 DOI: 10.7717/peerj.10210] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2020] [Accepted: 09/28/2020] [Indexed: 11/20/2022] Open
Abstract
The perception of facial attractiveness is a complex phenomenon which depends on how the observer perceives not only individual facial features, but also their mutual influence and interplay. In the machine learning community, this problem is typically tackled as a problem of regression of the subject-averaged rating assigned to natural faces. However, it has been conjectured that this approach does not capture the complexity of the phenomenon. It has recently been shown that different human subjects can navigate the face-space and "sculpt" their preferred modification of a reference facial portrait. Here we present an unsupervised inference study of the set of sculpted facial vectors in such experiments. We first infer minimal, interpretable and accurate probabilistic models (through Maximum Entropy and artificial neural networks) of the preferred facial variations, that encode the inter-subject variance. The application of such generative models to the supervised classification of the gender of the subject that sculpted the face reveals that it may be predicted with astonishingly high accuracy. We observe that the classification accuracy improves by increasing the order of the non-linear effective interaction. This suggests that the cognitive mechanisms related to facial discrimination in the brain do not involve the positions of single facial landmarks only, but mainly the mutual influence of couples, and even triplets and quadruplets of landmarks. Furthermore, the high prediction accuracy of the subjects' gender suggests that much relevant information regarding the subjects may influence (and be elicited from) their facial preference criteria, in agreement with the multiple motive theory of attractiveness proposed in previous works.
Collapse
Affiliation(s)
| | - Ambra Amico
- Chair of Systems Design, Swiss Federal Institute of Technology, Zurich, Switzerland
| | - Gian Luca Lancia
- Department of Physics, University of Roma “La Sapienza”, Rome, Italy
| | - Federico Maggiore
- Department of Physics, University of Roma “La Sapienza”, Rome, Italy
| | | | - Vittorio Loreto
- Department of Physics, University of Roma “La Sapienza”, Rome, Italy
- SONY Computer Science Laboratories, Paris, France
- Complexity Science Hub, Vienna, Austria
| |
Collapse
|
12
|
Zanoci C, Dehghani N, Tegmark M. Ensemble inhibition and excitation in the human cortex: An Ising-model analysis with uncertainties. Phys Rev E 2019; 99:032408. [PMID: 30999501 DOI: 10.1103/physreve.99.032408] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/22/2018] [Indexed: 11/07/2022]
Abstract
The pairwise maximum entropy model, also known as the Ising model, has been widely used to analyze the collective activity of neurons. However, controversy persists in the literature about seemingly inconsistent findings, whose significance is unclear due to lack of reliable error estimates. We therefore develop a method for accurately estimating parameter uncertainty based on random walks in parameter space using adaptive Markov-chain Monte Carlo after the convergence of the main optimization algorithm. We apply our method to the activity patterns of excitatory and inhibitory neurons recorded with multielectrode arrays in the human temporal cortex during the wake-sleep cycle. Our analysis shows that the Ising model captures neuronal collective behavior much better than the independent model during wakefulness, light sleep, and deep sleep when both excitatory (E) and inhibitory (I) neurons are modeled; ignoring the inhibitory effects of I neurons dramatically overestimates synchrony among E neurons. Furthermore, information-theoretic measures reveal that the Ising model explains about 80-95% of the correlations, depending on sleep state and neuron type. Thermodynamic measures show signatures of criticality, although we take this with a grain of salt as it may be merely a reflection of long-range neural correlations.
Collapse
Affiliation(s)
- Cristian Zanoci
- Department of Physics and Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
| | - Nima Dehghani
- Department of Physics and Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
| | - Max Tegmark
- Department of Physics and Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
| |
Collapse
|
13
|
Fasoli D, Panzeri S. Optimized brute-force algorithms for the bifurcation analysis of a binary neural network model. Phys Rev E 2019; 99:012316. [PMID: 30780305 DOI: 10.1103/physreve.99.012316] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2018] [Indexed: 06/09/2023]
Abstract
Bifurcation theory is a powerful tool for studying how the dynamics of a neural network model depends on its underlying neurophysiological parameters. However, bifurcation theory of neural networks has been developed mostly for mean-field limits of infinite-size spin-glass models, for finite-size dynamical systems whose units have a graded, continuous output, and for models with discrete-output neurons that evolve in continuous time. To allow progress on understanding the dynamics of some widely used classes of neural network models with discrete units and finite size, which could not be studied thoroughly with the previous methodology, here we introduced algorithms that perform a semianalytical bifurcation analysis of a finite-size firing-rate neural network model with binary firing rates and discrete-time evolution. In particular, we focus on the case of small networks composed of tens of neurons, to which existing statistical methods are not applicable. Our approach is based on a numerical brute-force search of the stationary and oscillatory solutions of the model, from which we derive analytical expressions of its bifurcation structure by means of the state-to-state transition probability matrix. Our algorithms determine how the network parameters affect the degree of multistability, the emergence and the period of the neural oscillations, and the formation of spontaneous symmetry breaking in the neural populations. While this technique can be applied to networks with arbitrary (generally asymmetric) connectivity matrices, in particular we introduce a highly efficient algorithm for the bifurcation analysis of sparse networks. We also provide some examples of the obtained bifurcation diagrams and a python implementation of the algorithms.
Collapse
Affiliation(s)
- Diego Fasoli
- Laboratory of Neural Computation, Center for Neuroscience and Cognitive Systems @UniTn, Istituto Italiano di Tecnologia, 38068 Rovereto, Italy
- Center for Brain and Cognition, Computational Neuroscience Group, Universitat Pompeu Fabra, 08002 Barcelona, Spain
| | - Stefano Panzeri
- Laboratory of Neural Computation, Center for Neuroscience and Cognitive Systems @UniTn, Istituto Italiano di Tecnologia, 38068 Rovereto, Italy
| |
Collapse
|
14
|
Learning Algorithm of Boltzmann Machine Based on Spatial Monte Carlo Integration Method. ALGORITHMS 2018. [DOI: 10.3390/a11040042] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
|
15
|
Lokhov AY, Vuffray M, Misra S, Chertkov M. Optimal structure and parameter learning of Ising models. SCIENCE ADVANCES 2018; 4:e1700791. [PMID: 29556527 PMCID: PMC5856491 DOI: 10.1126/sciadv.1700791] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/15/2017] [Accepted: 02/06/2018] [Indexed: 05/10/2023]
Abstract
Reconstruction of the structure and parameters of an Ising model from binary samples is a problem of practical importance in a variety of disciplines, ranging from statistical physics and computational biology to image processing and machine learning. The focus of the research community shifted toward developing universal reconstruction algorithms that are both computationally efficient and require the minimal amount of expensive data. We introduce a new method, interaction screening, which accurately estimates model parameters using local optimization problems. The algorithm provably achieves perfect graph structure recovery with an information-theoretically optimal number of samples, notably in the low-temperature regime, which is known to be the hardest for learning. The efficacy of interaction screening is assessed through extensive numerical tests on synthetic Ising models of various topologies with different types of interactions, as well as on real data produced by a D-Wave quantum computer. This study shows that the interaction screening method is an exact, tractable, and optimal technique that universally solves the inverse Ising problem.
Collapse
Affiliation(s)
- Andrey Y. Lokhov
- Center for Nonlinear Studies, Los Alamos National Laboratory, Los Alamos, NM 87545, USA
- Theoretical Division T-4, Los Alamos National Laboratory, Los Alamos, NM 87545, USA
- Corresponding author.
| | - Marc Vuffray
- Theoretical Division T-4, Los Alamos National Laboratory, Los Alamos, NM 87545, USA
| | - Sidhant Misra
- Theoretical Division T-5, Los Alamos National Laboratory, Los Alamos, NM 87545, USA
| | - Michael Chertkov
- Center for Nonlinear Studies, Los Alamos National Laboratory, Los Alamos, NM 87545, USA
- Theoretical Division T-4, Los Alamos National Laboratory, Los Alamos, NM 87545, USA
- Skolkovo Institute of Science and Technology, 143026 Moscow, Russia
| |
Collapse
|
16
|
Shirazi AH, Saberi AA, Hosseiny A, Amirzadeh E, Toranj Simin P. Non-criticality of interaction network over system's crises: A percolation analysis. Sci Rep 2017; 7:15855. [PMID: 29158531 PMCID: PMC5696469 DOI: 10.1038/s41598-017-16223-6] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2017] [Accepted: 10/23/2017] [Indexed: 11/29/2022] Open
Abstract
Extraction of interaction networks from multi-variate time-series is one of the topics of broad interest in complex systems. Although this method has a wide range of applications, most of the previous analyses have focused on the pairwise relations. Here we establish the potential of such a method to elicit aggregated behavior of the system by making a connection with the concepts from percolation theory. We study the dynamical interaction networks of a financial market extracted from the correlation network of indices, and build a weighted network. In correspondence with the percolation model, we find that away from financial crises the interaction network behaves like a critical random network of Erdős-Rényi, while close to a financial crisis, our model deviates from the critical random network and behaves differently at different size scales. We perform further analysis to clarify that our observation is not a simple consequence of the growth in correlations over the crises.
Collapse
Affiliation(s)
- Amir Hossein Shirazi
- Department of Physics, Shahid Beheshti University, G.C., Evin, Tehran, 19839, Iran.
| | - Abbas Ali Saberi
- Department of Physics, University of Tehran, Tehran, 14395-547, Iran.
- Institut für Theoretische Physik, Universitat zu Köln, Zülpicher Strasse 77, 50937, Köln, Germany.
- School of Physics and Accelerators, Institute for research in Fundamental Science (IPM) PO Box, 19395-5531, Tehran, Iran.
| | - Ali Hosseiny
- Department of Physics, Shahid Beheshti University, G.C., Evin, Tehran, 19839, Iran
- School of Physics and Accelerators, Institute for research in Fundamental Science (IPM) PO Box, 19395-5531, Tehran, Iran
| | - Ehsan Amirzadeh
- Department of Physics, Shahid Beheshti University, G.C., Evin, Tehran, 19839, Iran
| | - Pourya Toranj Simin
- Department of Physics, Shahid Beheshti University, G.C., Evin, Tehran, 19839, Iran
| |
Collapse
|
17
|
Rostami V, Porta Mana P, Grün S, Helias M. Bistability, non-ergodicity, and inhibition in pairwise maximum-entropy models. PLoS Comput Biol 2017; 13:e1005762. [PMID: 28968396 PMCID: PMC5645158 DOI: 10.1371/journal.pcbi.1005762] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2016] [Revised: 10/17/2017] [Accepted: 09/05/2017] [Indexed: 11/30/2022] Open
Abstract
Pairwise maximum-entropy models have been used in neuroscience to predict the activity of neuronal populations, given only the time-averaged correlations of the neuron activities. This paper provides evidence that the pairwise model, applied to experimental recordings, would produce a bimodal distribution for the population-averaged activity, and for some population sizes the second mode would peak at high activities, that experimentally would be equivalent to 90% of the neuron population active within time-windows of few milliseconds. Several problems are connected with this bimodality: 1. The presence of the high-activity mode is unrealistic in view of observed neuronal activity and on neurobiological grounds. 2. Boltzmann learning becomes non-ergodic, hence the pairwise maximum-entropy distribution cannot be found: in fact, Boltzmann learning would produce an incorrect distribution; similarly, common variants of mean-field approximations also produce an incorrect distribution. 3. The Glauber dynamics associated with the model is unrealistically bistable and cannot be used to generate realistic surrogate data. This bimodality problem is first demonstrated for an experimental dataset from 159 neurons in the motor cortex of macaque monkey. Evidence is then provided that this problem affects typical neural recordings of population sizes of a couple of hundreds or more neurons. The cause of the bimodality problem is identified as the inability of standard maximum-entropy distributions with a uniform reference measure to model neuronal inhibition. To eliminate this problem a modified maximum-entropy model is presented, which reflects a basic effect of inhibition in the form of a simple but non-uniform reference measure. This model does not lead to unrealistic bimodalities, can be found with Boltzmann learning, and has an associated Glauber dynamics which incorporates a minimal asymmetric inhibition. Networks of interacting units are ubiquitous in various fields of biology; e.g. gene regulatory networks, neuronal networks, social structures. If a limited set of observables is accessible, maximum-entropy models provide a way to construct a statistical model for such networks, under particular assumptions. The pairwise maximum-entropy model only uses the first two moments among those observables, and can be interpreted as a network with only pairwise interactions. If correlations are on average positive, we here show that the maximum entropy distribution tends to become bimodal. In the application to neuronal activity this is a problem, because the bimodality is an artefact of the statistical model and not observed in real data. This problem could also affect other fields in biology. We here explain under which conditions bimodality arises and present a solution to the problem by introducing a collective negative feedback, corresponding to a modified maximum-entropy model. This result may point to the existence of a homeostatic mechanism active in the system that is not part of our set of observable units.
Collapse
Affiliation(s)
- Vahid Rostami
- Institute of Neuroscience and Medicine (INM-6) and Institute for Advanced Simulation (IAS-6) and JARA BRAIN Institute I, Jülich Research Centre, Jülich, Germany
- * E-mail:
| | - PierGianLuca Porta Mana
- Institute of Neuroscience and Medicine (INM-6) and Institute for Advanced Simulation (IAS-6) and JARA BRAIN Institute I, Jülich Research Centre, Jülich, Germany
| | - Sonja Grün
- Institute of Neuroscience and Medicine (INM-6) and Institute for Advanced Simulation (IAS-6) and JARA BRAIN Institute I, Jülich Research Centre, Jülich, Germany
- Theoretical Systems Neurobiology, RWTH Aachen University, Aachen, Germany
| | - Moritz Helias
- Institute of Neuroscience and Medicine (INM-6) and Institute for Advanced Simulation (IAS-6) and JARA BRAIN Institute I, Jülich Research Centre, Jülich, Germany
- Department of Physics, Faculty 1, RWTH Aachen University, Aachen, Germany
| |
Collapse
|
18
|
Martirosyan A, Marsili M, De Martino A. Translating ceRNA Susceptibilities into Correlation Functions. Biophys J 2017; 113:206-213. [PMID: 28700919 DOI: 10.1016/j.bpj.2017.05.042] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2017] [Revised: 05/09/2017] [Accepted: 05/26/2017] [Indexed: 12/25/2022] Open
Abstract
Competition to bind microRNAs induces an effective positive cross talk between their targets, which are therefore known as "competing endogenous RNAs" (ceRNAs). Although such an effect is known to play a significant role in specific situations, estimating its strength from data and experimentally in physiological conditions appears to be far from simple. Here, we show that the susceptibility of ceRNAs to different types of perturbations affecting their competitors (and hence their tendency to cross talk) can be encoded in quantities as intuitive and as simple to measure as correlation functions. This scenario is confirmed by extensive numerical simulations and validated by re-analyzing phosphatase and tensin homolog's cross-talk pattern from The Cancer Genome Atlas breast cancer database. These results clarify the links between different quantities used to estimate the intensity of ceRNA cross talk and provide, to our knowledge, new keys to analyze transcriptional data sets and effectively probe ceRNA networks in silico.
Collapse
Affiliation(s)
- Araks Martirosyan
- Dipartimento di Fisica, Sapienza Università di Roma, Rome, Italy; VIB-KU Leuven Center for Brain and Disease Research, Leuven, Belgium
| | - Matteo Marsili
- The Abdus Salam International Centre for Theoretical Physics, Trieste, Italy
| | - Andrea De Martino
- Soft and Living Matter Lab, Institute of Nanotechnology (CNR-NANOTEC), Consiglio Nazionale delle Ricerche, Rome, Italy; Italian Institute for Genomic Medicine, Turin, Italy.
| |
Collapse
|
19
|
Donner C, Obermayer K, Shimazaki H. Approximate Inference for Time-Varying Interactions and Macroscopic Dynamics of Neural Populations. PLoS Comput Biol 2017; 13:e1005309. [PMID: 28095421 PMCID: PMC5283755 DOI: 10.1371/journal.pcbi.1005309] [Citation(s) in RCA: 15] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2016] [Revised: 01/31/2017] [Accepted: 12/12/2016] [Indexed: 11/29/2022] Open
Abstract
The models in statistical physics such as an Ising model offer a convenient way to characterize stationary activity of neural populations. Such stationary activity of neurons may be expected for recordings from in vitro slices or anesthetized animals. However, modeling activity of cortical circuitries of awake animals has been more challenging because both spike-rates and interactions can change according to sensory stimulation, behavior, or an internal state of the brain. Previous approaches modeling the dynamics of neural interactions suffer from computational cost; therefore, its application was limited to only a dozen neurons. Here by introducing multiple analytic approximation methods to a state-space model of neural population activity, we make it possible to estimate dynamic pairwise interactions of up to 60 neurons. More specifically, we applied the pseudolikelihood approximation to the state-space model, and combined it with the Bethe or TAP mean-field approximation to make the sequential Bayesian estimation of the model parameters possible. The large-scale analysis allows us to investigate dynamics of macroscopic properties of neural circuitries underlying stimulus processing and behavior. We show that the model accurately estimates dynamics of network properties such as sparseness, entropy, and heat capacity by simulated data, and demonstrate utilities of these measures by analyzing activity of monkey V4 neurons as well as a simulated balanced network of spiking neurons. Simultaneous analysis of large-scale neural populations is necessary to understand coding principles of neurons because they concertedly process information. Methods of thermodynamics and statistical mechanics are useful to understand collective phenomena of the interacting elements, and they have been successfully used to understand diverse activity of neurons. However, most analysis methods assume stationary data, in which activity rates of neurons and their correlations are constant over time. This assumption is easily violated in the data recorded from awake animals. Neural correlations likely organize dynamically during behavior and cognition, and this may be independent from the modulated activity rates of individual neurons. Recently several methods were proposed to simultaneously estimate dynamics of neural interactions. However, these methods are applicable to up to about 10 neurons. Here by combining multiple analytic approximation methods, we made it possible to estimate time-varying interactions of much larger neural populations. The method allows us to trace dynamic macroscopic properties of neural circuitries such as sparseness, entropy, and sensitivity. Using these statistics, researchers can now quantify to what extent neurons are correlated or de-correlated, and test if neural systems are susceptible within a specific behavioral period.
Collapse
Affiliation(s)
- Christian Donner
- Bernstein Center for Computational Neuroscience, Berlin, Germany
- Neural Information Processing Group, Department of Electrical Engineering and Computer Science, Technische Universität Berlin, Berlin, Germany
- Group for Methods of Artificial Intelligence, Department of Electrical Engineering and Computer Science, Technische Universität Berlin, Berlin, Germany
| | - Klaus Obermayer
- Bernstein Center for Computational Neuroscience, Berlin, Germany
- Neural Information Processing Group, Department of Electrical Engineering and Computer Science, Technische Universität Berlin, Berlin, Germany
| | | |
Collapse
|
20
|
Stein RR, Marks DS, Sander C. Inferring Pairwise Interactions from Biological Data Using Maximum-Entropy Probability Models. PLoS Comput Biol 2015. [PMID: 26225866 PMCID: PMC4520494 DOI: 10.1371/journal.pcbi.1004182] [Citation(s) in RCA: 66] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/02/2022] Open
Abstract
Maximum entropy-based inference methods have been successfully used to infer direct interactions from biological datasets such as gene expression data or sequence ensembles. Here, we review undirected pairwise maximum-entropy probability models in two categories of data types, those with continuous and categorical random variables. As a concrete example, we present recently developed inference methods from the field of protein contact prediction and show that a basic set of assumptions leads to similar solution strategies for inferring the model parameters in both variable types. These parameters reflect interactive couplings between observables, which can be used to predict global properties of the biological system. Such methods are applicable to the important problems of protein 3-D structure prediction and association of gene–gene networks, and they enable potential applications to the analysis of gene alteration patterns and to protein design.
Collapse
Affiliation(s)
- Richard R. Stein
- Computational Biology Program, Sloan Kettering Institute, Memorial Sloan Kettering Cancer Center, New York, New York, United States of America
- * E-mail: (RRS); (CS)
| | - Debora S. Marks
- Department of Systems Biology, Harvard Medical School, Boston, Massachusetts, United States of America
| | - Chris Sander
- Computational Biology Program, Sloan Kettering Institute, Memorial Sloan Kettering Cancer Center, New York, New York, United States of America
- * E-mail: (RRS); (CS)
| |
Collapse
|
21
|
Inferring synaptic structure in presence of neural interaction time scales. PLoS One 2015; 10:e0118412. [PMID: 25807389 PMCID: PMC4373808 DOI: 10.1371/journal.pone.0118412] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2014] [Accepted: 01/16/2015] [Indexed: 12/04/2022] Open
Abstract
Biological networks display a variety of activity patterns reflecting a web of interactions that is complex both in space and time. Yet inference methods have mainly focused on reconstructing, from the network’s activity, the spatial structure, by assuming equilibrium conditions or, more recently, a probabilistic dynamics with a single arbitrary time-step. Here we show that, under this latter assumption, the inference procedure fails to reconstruct the synaptic matrix of a network of integrate-and-fire neurons when the chosen time scale of interaction does not closely match the synaptic delay or when no single time scale for the interaction can be identified; such failure, moreover, exposes a distinctive bias of the inference method that can lead to infer as inhibitory the excitatory synapses with interaction time scales longer than the model’s time-step. We therefore introduce a new two-step method, that first infers through cross-correlation profiles the delay-structure of the network and then reconstructs the synaptic matrix, and successfully test it on networks with different topologies and in different activity regimes. Although step one is able to accurately recover the delay-structure of the network, thus getting rid of any a priori guess about the time scales of the interaction, the inference method introduces nonetheless an arbitrary time scale, the time-bin dt used to binarize the spike trains. We therefore analytically and numerically study how the choice of dt affects the inference in our network model, finding that the relationship between the inferred couplings and the real synaptic efficacies, albeit being quadratic in both cases, depends critically on dt for the excitatory synapses only, whilst being basically independent of it for the inhibitory ones.
Collapse
|
22
|
Barber RF, Drton M. High-dimensional Ising model selection with Bayesian information criteria. Electron J Stat 2015. [DOI: 10.1214/15-ejs1012] [Citation(s) in RCA: 46] [Impact Index Per Article: 5.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
23
|
Mann JK, Barton JP, Ferguson AL, Omarjee S, Walker BD, Chakraborty A, Ndung'u T. The fitness landscape of HIV-1 gag: advanced modeling approaches and validation of model predictions by in vitro testing. PLoS Comput Biol 2014; 10:e1003776. [PMID: 25102049 PMCID: PMC4125067 DOI: 10.1371/journal.pcbi.1003776] [Citation(s) in RCA: 94] [Impact Index Per Article: 9.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2013] [Accepted: 06/29/2014] [Indexed: 11/20/2022] Open
Abstract
Viral immune evasion by sequence variation is a major hindrance to HIV-1 vaccine design. To address this challenge, our group has developed a computational model, rooted in physics, that aims to predict the fitness landscape of HIV-1 proteins in order to design vaccine immunogens that lead to impaired viral fitness, thus blocking viable escape routes. Here, we advance the computational models to address previous limitations, and directly test model predictions against in vitro fitness measurements of HIV-1 strains containing multiple Gag mutations. We incorporated regularization into the model fitting procedure to address finite sampling. Further, we developed a model that accounts for the specific identity of mutant amino acids (Potts model), generalizing our previous approach (Ising model) that is unable to distinguish between different mutant amino acids. Gag mutation combinations (17 pairs, 1 triple and 25 single mutations within these) predicted to be either harmful to HIV-1 viability or fitness-neutral were introduced into HIV-1 NL4-3 by site-directed mutagenesis and replication capacities of these mutants were assayed in vitro. The predicted and measured fitness of the corresponding mutants for the original Ising model (r = −0.74, p = 3.6×10−6) are strongly correlated, and this was further strengthened in the regularized Ising model (r = −0.83, p = 3.7×10−12). Performance of the Potts model (r = −0.73, p = 9.7×10−9) was similar to that of the Ising model, indicating that the binary approximation is sufficient for capturing fitness effects of common mutants at sites of low amino acid diversity. However, we show that the Potts model is expected to improve predictive power for more variable proteins. Overall, our results support the ability of the computational models to robustly predict the relative fitness of mutant viral strains, and indicate the potential value of this approach for understanding viral immune evasion, and harnessing this knowledge for immunogen design. At least 70 million people have been infected with HIV since the beginning of the epidemic and an effective vaccine remains elusive. The high mutation rate and diversity of HIV strains enables the virus to effectively evade host immune responses, presenting a significant challenge for HIV vaccine design. We have developed an approach to translate clinical databases of HIV sequences into mathematical models quantifying the capacity of the virus to replicate as a function of mutations within its genome. We have previously shown how such “fitness landscapes” can be used to guide the design of vaccines to attack vulnerable regions from which it is difficult for the virus to escape by mutation. Here, using new modeling approaches, we have improved on our previous models of HIV fitness landscape by accounting for undersampling of HIV sequences and the specific identity of mutant amino acids. We experimentally tested the accuracy of the improved models to predict the fitness of HIV with multiple mutations in the Gag protein. The experimental data are in strong agreement with model predictions, supporting the value of these models as a novel approach for determining mutational vulnerabilities of HIV-1, which, in turn, can inform vaccine design.
Collapse
Affiliation(s)
- Jaclyn K. Mann
- HIV Pathogenesis Programme, Doris Duke Medical Research Institute, Nelson R. Mandela School of Medicine, University of KwaZulu-Natal, Durban, South Africa
- KwaZulu-Natal Research Institute for Tuberculosis and HIV (K-RITH), Nelson R Mandela School of Medicine, University of KwaZulu-Natal, Durban, South Africa
| | - John P. Barton
- Department of Chemical Engineering, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
- Ragon Institute of Massachusetts General Hospital, Massachusetts Institute of Technology and Harvard University, Boston, Massachusetts, United States of America
| | - Andrew L. Ferguson
- Department of Materials Science and Engineering, University of Illinois at Urbana-Champaign, Urbana, Illinois, United States of America
| | - Saleha Omarjee
- HIV Pathogenesis Programme, Doris Duke Medical Research Institute, Nelson R. Mandela School of Medicine, University of KwaZulu-Natal, Durban, South Africa
- KwaZulu-Natal Research Institute for Tuberculosis and HIV (K-RITH), Nelson R Mandela School of Medicine, University of KwaZulu-Natal, Durban, South Africa
| | - Bruce D. Walker
- HIV Pathogenesis Programme, Doris Duke Medical Research Institute, Nelson R. Mandela School of Medicine, University of KwaZulu-Natal, Durban, South Africa
- Ragon Institute of Massachusetts General Hospital, Massachusetts Institute of Technology and Harvard University, Boston, Massachusetts, United States of America
- Howard Hughes Medical Institute, Chevy Chase, Maryland, United States of America
| | - Arup Chakraborty
- Department of Chemical Engineering, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
- Ragon Institute of Massachusetts General Hospital, Massachusetts Institute of Technology and Harvard University, Boston, Massachusetts, United States of America
- Departments of Chemistry and Physics, Massachusetts Institute of Technology, Boston, Massachusetts, United States of America
- * E-mail: (AC); (TN)
| | - Thumbi Ndung'u
- HIV Pathogenesis Programme, Doris Duke Medical Research Institute, Nelson R. Mandela School of Medicine, University of KwaZulu-Natal, Durban, South Africa
- KwaZulu-Natal Research Institute for Tuberculosis and HIV (K-RITH), Nelson R Mandela School of Medicine, University of KwaZulu-Natal, Durban, South Africa
- Ragon Institute of Massachusetts General Hospital, Massachusetts Institute of Technology and Harvard University, Boston, Massachusetts, United States of America
- Max Planck Institute for Infection Biology, Berlin, Germany
- * E-mail: (AC); (TN)
| |
Collapse
|
24
|
Kiwata H. Estimation of quenched random fields in the inverse Ising problem using a diagonal matching method. PHYSICAL REVIEW. E, STATISTICAL, NONLINEAR, AND SOFT MATTER PHYSICS 2014; 89:062135. [PMID: 25019752 DOI: 10.1103/physreve.89.062135] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/28/2014] [Indexed: 06/03/2023]
Abstract
We consider a method for the accurate estimation of quenched random fields in the inverse Ising problem. Approximations such as the mean-field or Bethe methods are applied to estimate quenched random coupling parameters and external fields. A diagonal matching method is introduced to ensure consistency of the diagonal part of the susceptibility, and the method yields an accurate estimation of the external fields. We introduce the diagonal matching method into the mean-field, Thouless-Anderson-Palmer, and Bethe approximations, and we investigate the effect of the diagonal matching method on the accuracy of estimation of the external fields.
Collapse
Affiliation(s)
- Hirohito Kiwata
- Division of Natural Science, Osaka Kyoiku University, Kashiwara, Osaka 582-8582, Japan
| |
Collapse
|
25
|
Watanabe T, Kan S, Koike T, Misaki M, Konishi S, Miyauchi S, Miyahsita Y, Masuda N. Network-dependent modulation of brain activity during sleep. Neuroimage 2014; 98:1-10. [PMID: 24814208 DOI: 10.1016/j.neuroimage.2014.04.079] [Citation(s) in RCA: 29] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2014] [Revised: 04/23/2014] [Accepted: 04/29/2014] [Indexed: 10/25/2022] Open
Abstract
Brain activity dynamically changes even during sleep. A line of neuroimaging studies has reported changes in functional connectivity and regional activity across different sleep stages such as slow-wave sleep (SWS) and rapid-eye-movement (REM) sleep. However, it remains unclear whether and how the large-scale network activity of human brains changes within a given sleep stage. Here, we investigated modulation of network activity within sleep stages by applying the pairwise maximum entropy model to brain activity obtained by functional magnetic resonance imaging from sleeping healthy subjects. We found that the brain activity of individual brain regions and functional interactions between pairs of regions significantly increased in the default-mode network during SWS and decreased during REM sleep. In contrast, the network activity of the fronto-parietal and sensory-motor networks showed the opposite pattern. Furthermore, in the three networks, the amount of the activity changes throughout REM sleep was negatively correlated with that throughout SWS. The present findings suggest that the brain activity is dynamically modulated even in a sleep stage and that the pattern of modulation depends on the type of the large-scale brain networks.
Collapse
Affiliation(s)
- Takamitsu Watanabe
- Department of Physiology, The University of Tokyo, School of Medicine, Tokyo, 113-0033, Japan; Institute of Cognitive Neuroscience, University College London, London, WC1N 3AR, UK.
| | - Shigeyuki Kan
- Advanced ICT Research Institute, National Institute of Information and Communications Technology, Hyogo, 651-2492, Japan
| | - Takahiko Koike
- Advanced ICT Research Institute, National Institute of Information and Communications Technology, Hyogo, 651-2492, Japan
| | - Masaya Misaki
- Advanced ICT Research Institute, National Institute of Information and Communications Technology, Hyogo, 651-2492, Japan
| | - Seiki Konishi
- Department of Physiology, The University of Tokyo, School of Medicine, Tokyo, 113-0033, Japan
| | - Satoru Miyauchi
- Advanced ICT Research Institute, National Institute of Information and Communications Technology, Hyogo, 651-2492, Japan
| | - Yasushi Miyahsita
- Department of Physiology, The University of Tokyo, School of Medicine, Tokyo, 113-0033, Japan
| | - Naoki Masuda
- Department of Mathematical Informatics, The University of Tokyo, Tokyo, 113-8656, Japan.
| |
Collapse
|
26
|
Decelle A, Ricci-Tersenghi F. Pseudolikelihood decimation algorithm improving the inference of the interaction network in a general class of Ising models. PHYSICAL REVIEW LETTERS 2014; 112:070603. [PMID: 24579583 DOI: 10.1103/physrevlett.112.070603] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/11/2013] [Indexed: 05/22/2023]
Abstract
In this Letter we propose a new method to infer the topology of the interaction network in pairwise models with Ising variables. By using the pseudolikelihood method (PLM) at high temperature, it is generally possible to distinguish between zero and nonzero couplings because a clear gap separate the two groups. However at lower temperatures the PLM is much less effective and the result depends on subjective choices, such as the value of the ℓ1 regularizer and that of the threshold to separate nonzero couplings from null ones. We introduce a decimation procedure based on the PLM that recursively sets to zero the less significant couplings, until the variation of the pseudolikelihood signals that relevant couplings are being removed. The new method is fully automated and does not require any subjective choice by the user. Numerical tests have been performed on a wide class of Ising models, having different topologies (from random graphs to finite dimensional lattices) and different couplings (both diluted ferromagnets in a field and spin glasses). These numerical results show that the new algorithm performs better than standard PLM.
Collapse
Affiliation(s)
- Aurélien Decelle
- Dipartimento di Fisica, Università La Sapienza, Piazzale Aldo Moro 5, I-00185 Roma, Italy
| | - Federico Ricci-Tersenghi
- Dipartimento di Fisica, Università La Sapienza, Piazzale Aldo Moro 5, I-00185 Roma, Italy and INFN-Sezione di Roma 1 and CNR-IPCF, UOS di Roma, Piazzale Aldo Moro 5, I-00185 Roma, Italy
| |
Collapse
|
27
|
Tkačik G, Marre O, Amodei D, Schneidman E, Bialek W, Berry MJ. Searching for collective behavior in a large network of sensory neurons. PLoS Comput Biol 2014; 10:e1003408. [PMID: 24391485 PMCID: PMC3879139 DOI: 10.1371/journal.pcbi.1003408] [Citation(s) in RCA: 123] [Impact Index Per Article: 12.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2013] [Accepted: 11/05/2013] [Indexed: 11/30/2022] Open
Abstract
Maximum entropy models are the least structured probability distributions that exactly reproduce a chosen set of statistics measured in an interacting network. Here we use this principle to construct probabilistic models which describe the correlated spiking activity of populations of up to 120 neurons in the salamander retina as it responds to natural movies. Already in groups as small as 10 neurons, interactions between spikes can no longer be regarded as small perturbations in an otherwise independent system; for 40 or more neurons pairwise interactions need to be supplemented by a global interaction that controls the distribution of synchrony in the population. Here we show that such "K-pairwise" models--being systematic extensions of the previously used pairwise Ising models--provide an excellent account of the data. We explore the properties of the neural vocabulary by: 1) estimating its entropy, which constrains the population's capacity to represent visual information; 2) classifying activity patterns into a small set of metastable collective modes; 3) showing that the neural codeword ensembles are extremely inhomogenous; 4) demonstrating that the state of individual neurons is highly predictable from the rest of the population, allowing the capacity for error correction.
Collapse
Affiliation(s)
- Gašper Tkačik
- Institute of Science and Technology Austria, Klosterneuburg, Austria
| | - Olivier Marre
- Institut de la Vision, INSERM U968, UPMC, CNRS U7210, CHNO Quinze-Vingts, Paris, France
- Department of Molecular Biology, Princeton Neuroscience Institute, Princeton University, Princeton, New Jersey, United States of America
| | - Dario Amodei
- Department of Molecular Biology, Princeton Neuroscience Institute, Princeton University, Princeton, New Jersey, United States of America
- Joseph Henry Laboratories of Physics, Princeton University, Princeton, New Jersey, United States of America
| | - Elad Schneidman
- Department of Neurobiology, Weizmann Institute of Science, Rehovot, Israel
| | - William Bialek
- Joseph Henry Laboratories of Physics, Princeton University, Princeton, New Jersey, United States of America
- Lewis–Sigler Institute for Integrative Genomics, Princeton University, Princeton, New Jersey, United States of America
| | - Michael J. Berry
- Department of Molecular Biology, Princeton Neuroscience Institute, Princeton University, Princeton, New Jersey, United States of America
| |
Collapse
|
28
|
Haslinger R, Ba D, Galuske R, Williams Z, Pipa G. Missing mass approximations for the partition function of stimulus driven Ising models. Front Comput Neurosci 2013; 7:96. [PMID: 23898262 PMCID: PMC3721091 DOI: 10.3389/fncom.2013.00096] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2013] [Accepted: 06/24/2013] [Indexed: 11/13/2022] Open
Abstract
Ising models are routinely used to quantify the second order, functional structure of neural populations. With some recent exceptions, they generally do not include the influence of time varying stimulus drive. Yet if the dynamics of network function are to be understood, time varying stimuli must be taken into account. Inclusion of stimulus drive carries a heavy computational burden because the partition function becomes stimulus dependent and must be separately calculated for all unique stimuli observed. This potentially increases computation time by the length of the data set. Here we present an extremely fast, yet simply implemented, method for approximating the stimulus dependent partition function in minutes or seconds. Noting that the most probable spike patterns (which are few) occur in the training data, we sum partition function terms corresponding to those patterns explicitly. We then approximate the sum over the remaining patterns (which are improbable, but many) by casting it in terms of the stimulus modulated missing mass (total stimulus dependent probability of all patterns not observed in the training data). We use a product of conditioned logistic regression models to approximate the stimulus modulated missing mass. This method has complexity of roughly O(LNNpat) where is L the data length, N the number of neurons and N pat the number of unique patterns in the data, contrasting with the O(L2 (N) ) complexity of alternate methods. Using multiple unit recordings from rat hippocampus, macaque DLPFC and cat Area 18 we demonstrate our method requires orders of magnitude less computation time than Monte Carlo methods and can approximate the stimulus driven partition function more accurately than either Monte Carlo methods or deterministic approximations. This advance allows stimuli to be easily included in Ising models making them suitable for studying population based stimulus encoding.
Collapse
Affiliation(s)
- Robert Haslinger
- Martinos Center for Biomedical Imaging, Massachusetts General Hospital Charlestown, MA, USA ; Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology Cambridge, MA, USA
| | | | | | | | | |
Collapse
|
29
|
Zeng HL, Alava M, Aurell E, Hertz J, Roudi Y. Maximum likelihood reconstruction for Ising models with asynchronous updates. PHYSICAL REVIEW LETTERS 2013; 110:210601. [PMID: 23745850 DOI: 10.1103/physrevlett.110.210601] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/11/2012] [Revised: 11/26/2012] [Indexed: 06/02/2023]
Abstract
We describe how the couplings in an asynchronous kinetic Ising model can be inferred. We consider two cases: one in which we know both the spin history and the update times and one in which we know only the spin history. For the first case, we show that one can average over all possible choices of update times to obtain a learning rule that depends only on spin correlations and can also be derived from the equations of motion for the correlations. For the second case, the same rule can be derived within a further decoupling approximation. We study all methods numerically for fully asymmetric Sherrington-Kirkpatrick models, varying the data length, system size, temperature, and external field. Good convergence is observed in accordance with the theoretical expectations.
Collapse
Affiliation(s)
- Hong-Li Zeng
- Department of Applied Physics, Aalto University, FIN-00076 Aalto, Finland.
| | | | | | | | | |
Collapse
|
30
|
Granot-Atedgi E, Tkačik G, Segev R, Schneidman E. Stimulus-dependent maximum entropy models of neural population codes. PLoS Comput Biol 2013; 9:e1002922. [PMID: 23516339 PMCID: PMC3597542 DOI: 10.1371/journal.pcbi.1002922] [Citation(s) in RCA: 64] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2012] [Accepted: 12/28/2012] [Indexed: 11/18/2022] Open
Abstract
Neural populations encode information about their stimulus in a collective fashion, by joint activity patterns of spiking and silence. A full account of this mapping from stimulus to neural activity is given by the conditional probability distribution over neural codewords given the sensory input. For large populations, direct sampling of these distributions is impossible, and so we must rely on constructing appropriate models. We show here that in a population of 100 retinal ganglion cells in the salamander retina responding to temporal white-noise stimuli, dependencies between cells play an important encoding role. We introduce the stimulus-dependent maximum entropy (SDME) model-a minimal extension of the canonical linear-nonlinear model of a single neuron, to a pairwise-coupled neural population. We find that the SDME model gives a more accurate account of single cell responses and in particular significantly outperforms uncoupled models in reproducing the distributions of population codewords emitted in response to a stimulus. We show how the SDME model, in conjunction with static maximum entropy models of population vocabulary, can be used to estimate information-theoretic quantities like average surprise and information transmission in a neural population.
Collapse
Affiliation(s)
| | - Gašper Tkačik
- Institute of Science and Technology Austria, Am Campus 1, Klosterneuburg, Austria
| | - Ronen Segev
- Faculty of Natural Sciences, Department of Life Sciences and Zlotowski Center for Neuroscience, Ben Gurion University of the Negev, Be'er Sheva, Israel
| | - Elad Schneidman
- Department of Neurobiology, Weizmann Institute of Science, Rehovot, Israel
| |
Collapse
|
31
|
Dunn B, Roudi Y. Learning and inference in a nonequilibrium Ising model with hidden nodes. PHYSICAL REVIEW. E, STATISTICAL, NONLINEAR, AND SOFT MATTER PHYSICS 2013; 87:022127. [PMID: 23496479 DOI: 10.1103/physreve.87.022127] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/14/2012] [Revised: 01/17/2013] [Indexed: 06/01/2023]
Abstract
We study inference and reconstruction of couplings in a partially observed kinetic Ising model. With hidden spins, calculating the likelihood of a sequence of observed spin configurations requires performing a trace over the configurations of the hidden ones. This, as we show, can be represented as a path integral. Using this representation, we demonstrate that systematic approximate inference and learning rules can be derived using dynamical mean-field theory. Although naive mean-field theory leads to an unstable learning rule, taking into account Gaussian corrections allows learning the couplings involving hidden nodes. It also improves learning of the couplings between the observed nodes compared to when hidden nodes are ignored.
Collapse
Affiliation(s)
- Benjamin Dunn
- The Kavli Institue for Systems Neuroscience, NTNU, 7030 Trondheim.
| | | |
Collapse
|
32
|
Ekeberg M, Lövkvist C, Lan Y, Weigt M, Aurell E. Improved contact prediction in proteins: using pseudolikelihoods to infer Potts models. PHYSICAL REVIEW. E, STATISTICAL, NONLINEAR, AND SOFT MATTER PHYSICS 2013; 87:012707. [PMID: 23410359 DOI: 10.1103/physreve.87.012707] [Citation(s) in RCA: 393] [Impact Index Per Article: 35.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/23/2012] [Indexed: 05/24/2023]
Abstract
Spatially proximate amino acids in a protein tend to coevolve. A protein's three-dimensional (3D) structure hence leaves an echo of correlations in the evolutionary record. Reverse engineering 3D structures from such correlations is an open problem in structural biology, pursued with increasing vigor as more and more protein sequences continue to fill the data banks. Within this task lies a statistical inference problem, rooted in the following: correlation between two sites in a protein sequence can arise from firsthand interaction but can also be network-propagated via intermediate sites; observed correlation is not enough to guarantee proximity. To separate direct from indirect interactions is an instance of the general problem of inverse statistical mechanics, where the task is to learn model parameters (fields, couplings) from observables (magnetizations, correlations, samples) in large systems. In the context of protein sequences, the approach has been referred to as direct-coupling analysis. Here we show that the pseudolikelihood method, applied to 21-state Potts models describing the statistical properties of families of evolutionarily related proteins, significantly outperforms existing approaches to the direct-coupling analysis, the latter being based on standard mean-field techniques. This improved performance also relies on a modified score for the coupling strength. The results are verified using known crystal structures of specific sequence instances of various protein families. Code implementing the new method can be found at http://plmdca.csc.kth.se/.
Collapse
Affiliation(s)
- Magnus Ekeberg
- Engineering Physics Program, KTH Royal Institute of Technology, 100 44 Stockholm, Sweden
| | | | | | | | | |
Collapse
|
33
|
Nguyen HC, Berg J. Mean-field theory for the inverse Ising problem at low temperatures. PHYSICAL REVIEW LETTERS 2012; 109:050602. [PMID: 23006160 DOI: 10.1103/physrevlett.109.050602] [Citation(s) in RCA: 11] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/08/2012] [Indexed: 06/01/2023]
Abstract
The large amounts of data from molecular biology and neuroscience have lead to a renewed interest in the inverse Ising problem: how to reconstruct parameters of the Ising model (couplings between spins and external fields) from a number of spin configurations sampled from the Boltzmann measure. To invert the relationship between model parameters and observables (magnetizations and correlations), mean-field approximations are often used, allowing the determination of model parameters from data. However, all known mean-field methods fail at low temperatures with the emergence of multiple thermodynamic states. Here, we show how clustering spin configurations can approximate these thermodynamic states and how mean-field methods applied to thermodynamic states allow an efficient reconstruction of Ising models also at low temperatures.
Collapse
Affiliation(s)
- H Chau Nguyen
- Institute for Theoretical Physics, University of Cologne, Köln, Germany.
| | | |
Collapse
|
34
|
Onken A, Dragoi V, Obermayer K. A maximum entropy test for evaluating higher-order correlations in spike counts. PLoS Comput Biol 2012; 8:e1002539. [PMID: 22685392 PMCID: PMC3369943 DOI: 10.1371/journal.pcbi.1002539] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2011] [Accepted: 04/10/2012] [Indexed: 11/19/2022] Open
Abstract
Evaluating the importance of higher-order correlations of neural spike counts has been notoriously hard. A large number of samples are typically required in order to estimate higher-order correlations and resulting information theoretic quantities. In typical electrophysiology data sets with many experimental conditions, however, the number of samples in each condition is rather small. Here we describe a method that allows to quantify evidence for higher-order correlations in exactly these cases. We construct a family of reference distributions: maximum entropy distributions, which are constrained only by marginals and by linear correlations as quantified by the Pearson correlation coefficient. We devise a Monte Carlo goodness-of-fit test, which tests - for a given divergence measure of interest - whether the experimental data lead to the rejection of the null hypothesis that it was generated by one of the reference distributions. Applying our test to artificial data shows that the effects of higher-order correlations on these divergence measures can be detected even when the number of samples is small. Subsequently, we apply our method to spike count data which were recorded with multielectrode arrays from the primary visual cortex of anesthetized cat during an adaptation experiment. Using mutual information as a divergence measure we find that there are spike count bin sizes at which the maximum entropy hypothesis can be rejected for a substantial number of neuronal pairs. These results demonstrate that higher-order correlations can matter when estimating information theoretic quantities in V1. They also show that our test is able to detect their presence in typical in-vivo data sets, where the number of samples is too small to estimate higher-order correlations directly. Populations of neurons signal information by their joint activity. Dependencies between the activity of multiple neurons are typically described by the linear correlation coefficient. However, this description of the dependencies is not complete. Dependencies beyond the linear correlation coefficient, so-called higher-order correlations, are often neglected because too many experimental samples are required in order to estimate them reliably. Evaluating the importance of higher-order correlations for the neural representation has therefore been notoriously hard. We devise a statistical test that can quantify evidence for higher-order correlations without estimating higher-order correlations directly. The test yields reliable results even when the number of experimental samples is small. The power of the method is demonstrated on data which were recorded from a population of neurons in the primary visual cortex of cat during an adaptation experiment. We show that higher-order correlations can have a substantial impact on the encoded stimulus information which, moreover, is modulated by stimulus adaptation.
Collapse
Affiliation(s)
- Arno Onken
- Technische Universität Berlin, Berlin, Germany.
| | | | | |
Collapse
|
35
|
Aurell E, Ekeberg M. Inverse Ising inference using all the data. PHYSICAL REVIEW LETTERS 2012; 108:090201. [PMID: 22463617 DOI: 10.1103/physrevlett.108.090201] [Citation(s) in RCA: 39] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/29/2011] [Revised: 12/12/2011] [Indexed: 05/31/2023]
Abstract
We show that a method based on logistic regression, using all the data, solves the inverse Ising problem far better than mean-field calculations relying only on sample pairwise correlation functions, while still computationally feasible for hundreds of nodes. The largest improvement in reconstruction occurs for strong interactions. Using two examples, a diluted Sherrington-Kirkpatrick model and a two-dimensional lattice, we also show that interaction topologies can be recovered from few samples with good accuracy and that the use of l(1) regularization is beneficial in this process, pushing inference abilities further into low-temperature regimes.
Collapse
Affiliation(s)
- Erik Aurell
- ACCESS Linnaeus Centre, KTH, Stockholm, Sweden.
| | | |
Collapse
|
36
|
Marks DS, Colwell LJ, Sheridan R, Hopf TA, Pagnani A, Zecchina R, Sander C. Protein 3D structure computed from evolutionary sequence variation. PLoS One 2011; 6:e28766. [PMID: 22163331 PMCID: PMC3233603 DOI: 10.1371/journal.pone.0028766] [Citation(s) in RCA: 748] [Impact Index Per Article: 57.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2011] [Accepted: 11/14/2011] [Indexed: 11/19/2022] Open
Abstract
The evolutionary trajectory of a protein through sequence space is constrained by its function. Collections of sequence homologs record the outcomes of millions of evolutionary experiments in which the protein evolves according to these constraints. Deciphering the evolutionary record held in these sequences and exploiting it for predictive and engineering purposes presents a formidable challenge. The potential benefit of solving this challenge is amplified by the advent of inexpensive high-throughput genomic sequencing. In this paper we ask whether we can infer evolutionary constraints from a set of sequence homologs of a protein. The challenge is to distinguish true co-evolution couplings from the noisy set of observed correlations. We address this challenge using a maximum entropy model of the protein sequence, constrained by the statistics of the multiple sequence alignment, to infer residue pair couplings. Surprisingly, we find that the strength of these inferred couplings is an excellent predictor of residue-residue proximity in folded structures. Indeed, the top-scoring residue couplings are sufficiently accurate and well-distributed to define the 3D protein fold with remarkable accuracy. We quantify this observation by computing, from sequence alone, all-atom 3D structures of fifteen test proteins from different fold classes, ranging in size from 50 to 260 residues., including a G-protein coupled receptor. These blinded inferences are de novo, i.e., they do not use homology modeling or sequence-similar fragments from known structures. The co-evolution signals provide sufficient information to determine accurate 3D protein structure to 2.7–4.8 Å Cα-RMSD error relative to the observed structure, over at least two-thirds of the protein (method called EVfold, details at http://EVfold.org). This discovery provides insight into essential interactions constraining protein evolution and will facilitate a comprehensive survey of the universe of protein structures, new strategies in protein and drug design, and the identification of functional genetic variants in normal and disease genomes.
Collapse
Affiliation(s)
- Debora S Marks
- Department of Systems Biology, Harvard Medical School, Boston, Massachusetts, United States of America.
| | | | | | | | | | | | | |
Collapse
|
37
|
Stephens GJ, Osborne LC, Bialek W. Searching for simplicity in the analysis of neurons and behavior. Proc Natl Acad Sci U S A 2011; 108 Suppl 3:15565-71. [PMID: 21383186 PMCID: PMC3176616 DOI: 10.1073/pnas.1010868108] [Citation(s) in RCA: 26] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022] Open
Abstract
What fascinates us about animal behavior is its richness and complexity, but understanding behavior and its neural basis requires a simpler description. Traditionally, simplification has been imposed by training animals to engage in a limited set of behaviors, by hand scoring behaviors into discrete classes, or by limiting the sensory experience of the organism. An alternative is to ask whether we can search through the dynamics of natural behaviors to find explicit evidence that these behaviors are simpler than they might have been. We review two mathematical approaches to simplification, dimensionality reduction and the maximum entropy method, and we draw on examples from different levels of biological organization, from the crawling behavior of Caenorhabditis elegans to the control of smooth pursuit eye movements in primates, and from the coding of natural scenes by networks of neurons in the retina to the rules of English spelling. In each case, we argue that the explicit search for simplicity uncovers new and unexpected features of the biological system and that the evidence for simplification gives us a language with which to phrase new questions for the next generation of experiments. The fact that similar mathematical structures succeed in taming the complexity of very different biological systems hints that there is something more general to be discovered.
Collapse
Affiliation(s)
- Greg J Stephens
- Joseph Henry Laboratories of Physics, Lewis-Sigler Institute for Integrative Genomics and Princeton Center for Theoretical Sciences, Princeton University, Princeton, NJ 08544, USA.
| | | | | |
Collapse
|
38
|
Schaub MT, Schultz SR. The Ising decoder: reading out the activity of large neural ensembles. J Comput Neurosci 2011; 32:101-18. [PMID: 21667155 DOI: 10.1007/s10827-011-0342-z] [Citation(s) in RCA: 28] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/09/2010] [Revised: 04/28/2011] [Accepted: 05/22/2011] [Indexed: 11/26/2022]
Abstract
The Ising model has recently received much attention for the statistical description of neural spike train data. In this paper, we propose and demonstrate its use for building decoders capable of predicting, on a millisecond timescale, the stimulus represented by a pattern of neural activity. After fitting to a training dataset, the Ising decoder can be applied "online" for instantaneous decoding of test data. While such models can be fit exactly using Boltzmann learning, this approach rapidly becomes computationally intractable as neural ensemble size increases. We show that several approaches, including the Thouless-Anderson-Palmer (TAP) mean field approach from statistical physics, and the recently developed Minimum Probability Flow Learning (MPFL) algorithm, can be used for rapid inference of model parameters in large-scale neural ensembles. Use of the Ising model for decoding, unlike other problems such as functional connectivity estimation, requires estimation of the partition function. As this involves summation over all possible responses, this step can be limiting. Mean field approaches avoid this problem by providing an analytical expression for the partition function. We demonstrate these decoding techniques by applying them to simulated neural ensemble responses from a mouse visual cortex model, finding an improvement in decoder performance for a model with heterogeneous as opposed to homogeneous neural tuning and response properties. Our results demonstrate the practicality of using the Ising model to read out, or decode, spatial patterns of activity comprised of many hundreds of neurons.
Collapse
Affiliation(s)
- Michael T Schaub
- Department of Bioengineering, Imperial College London, South Kensington, London SW72AZ, UK
| | | |
Collapse
|
39
|
Braunstein A, Ramezanpour A, Zecchina R, Zhang P. Inference and learning in sparse systems with multiple states. PHYSICAL REVIEW. E, STATISTICAL, NONLINEAR, AND SOFT MATTER PHYSICS 2011; 83:056114. [PMID: 21728612 DOI: 10.1103/physreve.83.056114] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/04/2011] [Revised: 03/18/2011] [Indexed: 05/31/2023]
Abstract
We discuss how inference can be performed when data are sampled from the nonergodic phase of systems with multiple attractors. We take as a model system the finite connectivity Hopfield model in the memory phase and suggest a cavity method approach to reconstruct the couplings when the data are separately sampled from few attractor states. We also show how the inference results can be converted into a learning protocol for neural networks in which patterns are presented through weak external fields. The protocol is simple and fully local, and is able to store patterns with a finite overlap with the input patterns without ever reaching a spin-glass phase where all memories are lost.
Collapse
Affiliation(s)
- A Braunstein
- Human Genetics Foundation, Via Nizza 52, I-10126 Torino, Italy.
| | | | | | | |
Collapse
|
40
|
Zeng HL, Aurell E, Alava M, Mahmoudi H. Network inference using asynchronously updated kinetic Ising model. PHYSICAL REVIEW. E, STATISTICAL, NONLINEAR, AND SOFT MATTER PHYSICS 2011; 83:041135. [PMID: 21599143 DOI: 10.1103/physreve.83.041135] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/29/2010] [Indexed: 05/30/2023]
Abstract
Network structures are reconstructed from dynamical data by respectively naive mean field (nMF) and Thouless-Anderson-Palmer (TAP) approximations. TAP approximation adds simple corrections to the nMF approximation, taking into account the effect of the focused spin on itself via its influence on other neighboring spins. For TAP approximation, we use two methods to reconstruct the network: (a) iterative method; (b) casting the inference formula to a set of cubic equations and solving it directly. We investigate inference of the asymmetric Sherrington-Kirkpatrick (aS-K) model using asynchronous update. The solutions of the set of cubic equations depend on temperature T in the aS-K model, and a critical temperature T(c)≈2.1 is found. The two methods for TAP approximation produce the same results when the iterative method is convergent. Compared to nMF, TAP is somewhat better at low temperatures, but approaches the same performance as temperature increases. Both nMF and TAP approximation reconstruct better for longer data length L, but for the degree of improvement, TAP performs better than nMF.
Collapse
Affiliation(s)
- Hong-Li Zeng
- Department of Applied Physics, Aalto University, FIN-00076 Aalto, Finland.
| | | | | | | |
Collapse
|
41
|
Fitzgerald JD, Sincich LC, Sharpee TO. Minimal models of multidimensional computations. PLoS Comput Biol 2011; 7:e1001111. [PMID: 21455284 PMCID: PMC3063722 DOI: 10.1371/journal.pcbi.1001111] [Citation(s) in RCA: 26] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2010] [Accepted: 02/17/2011] [Indexed: 11/18/2022] Open
Abstract
The multidimensional computations performed by many biological systems are often characterized with limited information about the correlations between inputs and outputs. Given this limitation, our approach is to construct the maximum noise entropy response function of the system, leading to a closed-form and minimally biased model consistent with a given set of constraints on the input/output moments; the result is equivalent to conditional random field models from machine learning. For systems with binary outputs, such as neurons encoding sensory stimuli, the maximum noise entropy models are logistic functions whose arguments depend on the constraints. A constraint on the average output turns the binary maximum noise entropy models into minimum mutual information models, allowing for the calculation of the information content of the constraints and an information theoretic characterization of the system's computations. We use this approach to analyze the nonlinear input/output functions in macaque retina and thalamus; although these systems have been previously shown to be responsive to two input dimensions, the functional form of the response function in this reduced space had not been unambiguously identified. A second order model based on the logistic function is found to be both necessary and sufficient to accurately describe the neural responses to naturalistic stimuli, accounting for an average of 93% of the mutual information with a small number of parameters. Thus, despite the fact that the stimulus is highly non-Gaussian, the vast majority of the information in the neural responses is related to first and second order correlations. Our results suggest a principled and unbiased way to model multidimensional computations and determine the statistics of the inputs that are being encoded in the outputs.
Collapse
Affiliation(s)
- Jeffrey D. Fitzgerald
- Computational Neurobiology Laboratory, The Salk Institute for Biological Studies, La Jolla, California, United States of America
- Center for Theoretical Biological Physics and Department of Physics, University of California, San Diego, La Jolla, California, United States of America
| | - Lawrence C. Sincich
- Beckman Vision Center, University of California, San Francisco, San Francisco, California, United States of America
| | - Tatyana O. Sharpee
- Computational Neurobiology Laboratory, The Salk Institute for Biological Studies, La Jolla, California, United States of America
- Center for Theoretical Biological Physics and Department of Physics, University of California, San Diego, La Jolla, California, United States of America
- * E-mail:
| |
Collapse
|
42
|
Huang H. Message passing algorithms for the Hopfield network reconstruction: threshold behavior and limitation. PHYSICAL REVIEW. E, STATISTICAL, NONLINEAR, AND SOFT MATTER PHYSICS 2010; 82:056111. [PMID: 21230549 DOI: 10.1103/physreve.82.056111] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/01/2010] [Revised: 10/25/2010] [Indexed: 05/30/2023]
Abstract
The Hopfield network is reconstructed as an inverse Ising problem by passing messages. The applied susceptibility propagation algorithm is shown to improve significantly on other mean-field-type methods and extends well into the low-temperature region. However, this iterative algorithm is limited by the nature of the supplied data. Its performance deteriorates as the data become highly magnetized, and this method finally fails in the presence of the frozen type data where at least two of its magnetizations are equal to 1 in absolute value. On the other hand, a threshold behavior is observed for the susceptibility propagation algorithm and the transition from good reconstruction to poor one becomes sharper as the network size increases.
Collapse
Affiliation(s)
- Haiping Huang
- Key Laboratory of Frontiers in Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China
| |
Collapse
|
43
|
Hertz JA, Roudi Y, Thorning A, Tyrcha J, Aurell E, Zeng HL. Inferring network connectivity using kinetic Ising models. BMC Neurosci 2010. [PMCID: PMC3090939 DOI: 10.1186/1471-2202-11-s1-p51] [Citation(s) in RCA: 9] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022] Open
|
44
|
Huang H. Reconstructing the Hopfield network as an inverse Ising problem. PHYSICAL REVIEW. E, STATISTICAL, NONLINEAR, AND SOFT MATTER PHYSICS 2010; 81:036104. [PMID: 20365812 DOI: 10.1103/physreve.81.036104] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/13/2009] [Indexed: 05/29/2023]
Abstract
We test four fast mean-field-type algorithms on Hopfield networks as an inverse Ising problem. The equilibrium behavior of Hopfield networks is simulated through Glauber dynamics. In the low-temperature regime, the simulated annealing technique is adopted. Although performances of these network reconstruction algorithms on the simulated network of spiking neurons are extensively studied recently, the analysis of Hopfield networks is lacking so far. For the Hopfield network, we found that, in the retrieval phase favored when the network wants to memory one of stored patterns, all the reconstruction algorithms fail to extract interactions within a desired accuracy, and the same failure occurs in the spin-glass phase where spurious minima show up, while in the paramagnetic phase, albeit unfavored during the retrieval dynamics, the algorithms work well to reconstruct the network itself. This implies that, as an inverse problem, the paramagnetic phase is conversely useful for reconstructing the network while the retrieval phase loses all the information about interactions in the network except for the case where only one pattern is stored. The performances of algorithms are studied with respect to the system size, memory load, and temperature; sample-to-sample fluctuations are also considered.
Collapse
Affiliation(s)
- Haiping Huang
- Key Laboratory of Frontiers in Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China
| |
Collapse
|
45
|
|