REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.

An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability

Traditionally, scientists have placed more emphasis on communicating inferential uncertainty (i.e., the precision of statistical estimates) compared to outcome variability (i.e., the predictability of individual outcomes). Here, we show that this can lead to sizable misperceptions about the implications of scientific results. Specifically, we present three preregistered, randomized experiments where participants saw the same scientific findings visualized as showing only inferential uncertainty, only outcome variability, or both and answered questions about the size and importance of findings they were shown. Our results, composed of responses from medical professionals, professional data scientists, and tenure-track faculty, show that the prevalent form of visualizing only inferential uncertainty can lead to significant overestimates of treatment effects, even among highly trained experts. In contrast, we find that depicting both inferential uncertainty and outcome variability leads to more accurate perceptions of results while appearing to leave other subjective impressions of the results unchanged, on average.

Integrating explanation and prediction in computational social science

Computational social science is more than just large repositories of digital data and the computational methods needed to construct and analyse them. It also represents a convergence of different fields with different ways of thinking about and doing science. The goal of this Perspective is to provide some clarity around how these approaches differ from one another and to propose how they might be productively integrated. Towards this end we make two contributions. The first is a schema for thinking about research activities along two dimensions-the extent to which work is explanatory, focusing on identifying and estimating causal effects, and the degree of consideration given to testing predictions of outcomes-and how these two priorities can complement, rather than compete with, one another. Our second contribution is to advocate that computational social scientists devote more attention to combining prediction and explanation, which we call integrative modelling, and to outline some practical suggestions for realizing this goal.

Prediction and explanation in social systems

Historically, social scientists have sought out explanations of human and social phenomena that provide interpretable causal mechanisms, while often ignoring their predictive accuracy. We argue that the increasingly computational nature of social science is beginning to reverse this traditional bias against prediction; however, it has also highlighted three important issues that require resolution. First, current practices for evaluating predictions must be better standardized. Second, theoretical limits to predictive accuracy in complex social systems must be better characterized, thereby setting expectations for what can be predicted or explained. Third, predictive accuracy and interpretability must be recognized as complements, not substitutes, when evaluating explanations. Resolving these three issues will lead to better, more replicable, and more useful social science.

Graphical models for inferring single molecule dynamics

BACKGROUND

The recent explosion of experimental techniques in single molecule biophysics has generated a variety of novel time series data requiring equally novel computational tools for analysis and inference. This article describes in general terms how graphical modeling may be used to learn from biophysical time series data using the variational Bayesian expectation maximization algorithm (VBEM). The discussion is illustrated by the example of single-molecule fluorescence resonance energy transfer (smFRET) versus time data, where the smFRET time series is modeled as a hidden Markov model (HMM) with Gaussian observables. A detailed description of smFRET is provided as well.

RESULTS

The VBEM algorithm returns the model's evidence and an approximating posterior parameter distribution given the data. The former provides a metric for model selection via maximum evidence (ME), and the latter a description of the model's parameters learned from the data. ME/VBEM provide several advantages over the more commonly used approach of maximum likelihood (ML) optimized by the expectation maximization (EM) algorithm, the most important being a natural form of model selection and a well-posed (non-divergent) optimization problem.

CONCLUSIONS

The results demonstrate the utility of graphical modeling for inference of dynamic processes in single molecule biophysics.

Learning rates and states from biophysical time series: a Bayesian approach to model selection and single-molecule FRET data

Time series data provided by single-molecule Förster resonance energy transfer (smFRET) experiments offer the opportunity to infer not only model parameters describing molecular complexes, e.g., rate constants, but also information about the model itself, e.g., the number of conformational states. Resolving whether such states exist or how many of them exist requires a careful approach to the problem of model selection, here meaning discrimination among models with differing numbers of states. The most straightforward approach to model selection generalizes the common idea of maximum likelihood--selecting the most likely parameter values--to maximum evidence: selecting the most likely model. In either case, such an inference presents a tremendous computational challenge, which we here address by exploiting an approximation technique termed variational Bayesian expectation maximization. We demonstrate how this technique can be applied to temporal data such as smFRET time series; show superior statistical consistency relative to the maximum likelihood approach; compare its performance on smFRET data generated from experiments on the ribosome; and illustrate how model selection in such probabilistic or generative modeling can facilitate analysis of closely related temporal data currently prevalent in biophysics. Source code used in this analysis, including a graphical user interface, is available open source via http://vbFRET.sourceforge.net.

Quantification of cell edge velocities and traction forces reveals distinct motility modules during cell spreading

Actin-based cell motility and force generation are central to immune response, tissue development, and cancer metastasis, and understanding actin cytoskeleton regulation is a major goal of cell biologists. Cell spreading is a commonly used model system for motility experiments -- spreading fibroblasts exhibit stereotypic, spatially-isotropic edge dynamics during a reproducible sequence of functional phases: 1) During early spreading, cells form initial contacts with the surface. 2) The middle spreading phase exhibits rapidly increasing attachment area. 3) Late spreading is characterized by periodic contractions and stable adhesions formation. While differences in cytoskeletal regulation between phases are known, a global analysis of the spatial and temporal coordination of motility and force generation is missing. Implementing improved algorithms for analyzing edge dynamics over the entire cell periphery, we observed that a single domain of homogeneous cytoskeletal dynamics dominated each of the three phases of spreading. These domains exhibited a unique combination of biophysical and biochemical parameters -- a motility module. Biophysical characterization of the motility modules revealed that the early phase was dominated by periodic, rapid membrane blebbing; the middle phase exhibited continuous protrusion with very low traction force generation; and the late phase was characterized by global periodic contractions and high force generation. Biochemically, each motility module exhibited a different distribution of the actin-related protein VASP, while inhibition of actin polymerization revealed different dependencies on barbed-end polymerization. In addition, our whole-cell analysis revealed that many cells exhibited heterogeneous combinations of motility modules in neighboring regions of the cell edge. Together, these observations support a model of motility in which regions of the cell edge exhibit one of a limited number of motility modules that, together, determine the overall motility function. Our data and algorithms are publicly available to encourage further exploration.

Bayesian approach to network modularity

We present an efficient, principled, and interpretable technique for inferring module assignments and for identifying the optimal number of modules in a given network. We show how several existing methods for finding modules can be described as variant, special, or limiting cases of our work, and how the method overcomes the resolution limit problem, accurately recovering the true number of modules. Our approach is based on Bayesian methods for model selection which have been used with success for almost a century, implemented using a variational technique developed only in the past decade. We apply the technique to synthetic and real networks and outline how the method naturally allows selection among competing models.

Opposing Effects of PKCθ and WASp on Symmetry Breaking and Relocation of the Immunological Synapse

The immunological synapse (IS) is a junction between the T cell and antigen-presenting cell and is composed of supramolecular activation clusters (SMACs). No studies have been published on naive T cell IS dynamics. Here, we find that IS formation during antigen recognition comprises cycles of stable IS formation and autonomous naive T cell migration. The migration phase is driven by PKCtheta, which is localized to the F-actin-dependent peripheral (p)SMAC. PKCtheta(-/-) T cells formed hyperstable IS in vitro and in vivo and, like WT cells, displayed fast oscillations in the distal SMAC, but they showed reduced slow oscillations in pSMAC integrity. IS reformation is driven by the Wiscott Aldrich Syndrome protein (WASp). WASp(-/-) T cells displayed normal IS formation but were unable to reform IS after migration unless PKCtheta was inhibited. Thus, opposing effects of PKCtheta and WASp control IS stability through pSMAC symmetry breaking and reformation.

Nonmuscle myosin IIA-dependent force inhibits cell spreading and drives F-actin flow

Nonmuscle myosin IIA (NMM-IIA) is involved in the formation of focal adhesions and neurite retraction. However, the role of NMM-IIA in these functions remains largely unknown. Using RNA interference as a tool to decrease NMM-IIA expression, we have found that NMM-IIA is the major myosin involved in traction force generation and retrograde F-actin flow in mouse embryonic fibroblast cells. Quantitative analyses revealed that approximately 60% of traction force on fibronectin-coated surfaces is contributed by NMM-IIA and approximately 30% by NMM-IIB. The retrograde F-actin flow decreased dramatically in NMM-IIA-depleted cells, but seemed unaffected by NMM-IIB deletion. In addition, we found that depletion of NMM-IIA caused cells to spread at a higher rate and to a greater area on fibronectin substrates during the early spreading period, whereas deletion of NMM-IIB appeared to have no effect on spreading. The distribution of NMM-IIA was concentrated on the dorsal surface and approached the ventral surface in the periphery, whereas NMM-IIB was primarily concentrated around the nucleus and to a lesser extent at the ventral surface in cell periphery. Our results suggest that NMM-IIA is involved in generating a coherent cytoplasmic contractile force from one side of the cell to the other through the cross-linking and the contraction of dorsal actin filaments.

Lateral membrane waves constitute a universal dynamic pattern of motile cells

We have monitored active movements of the cell circumference on specifically coated substrates for a variety of cells including mouse embryonic fibroblasts and T cells, as well as wing disk cells from fruit flies. Despite having different functions and being from multiple phyla, these cell types share a common spatiotemporal pattern in their normal membrane velocity; we show that protrusion and retraction events are organized in lateral waves along the cell membrane. These wave patterns indicate both spatial and temporal long-range periodic correlations of the actomyosin gel.

The small GTPase R-Ras regulates organization of actin and drives membrane protrusions through the activity of PLCepsilon

R-Ras, an atypical member of the Ras subfamily of small GTPases, enhances integrin-mediated adhesion and signaling through a poorly understood mechanism. Dynamic analysis of cell spreading by total internal reflection fluorescence (TIRF) microscopy demonstrated that active R-Ras lengthened the duration of initial membrane protrusion, and promoted the formation of a ruffling lamellipod, rich in branched actin structures and devoid of filopodia. By contrast, dominant-negative R-Ras enhanced filopodia formation. Moreover, RNA interference (RNAi) approaches demonstrated that endogenous R-Ras contributed to cell spreading. These observations suggest that R-Ras regulates membrane protrusions through organization of the actin cytoskeleton. Our results suggest that phospholipase Cepsilon (PLCepsilon) is a novel R-Ras effector mediating the effects of R-Ras on the actin cytoskeleton and membrane protrusion, because R-Ras was co-precipitated with PLCepsilon and increased its activity. Knockdown of PLCepsilon with siRNA reduced the formation of the ruffling lamellipod in R-Ras cells. Consistent with this pathway, inhibitors of PLC activity, or chelating intracellular Ca2+ abolished the ability of R-Ras to promote membrane protrusions and spreading. Overall, these data suggest that R-Ras signaling regulates the organization of the actin cytoskeleton to sustain membrane protrusion through the activity of PLCepsilon.

Effective viscosity of dense colloidal crystals

An exact scheme is presented to determine the effective viscosity tensor for periodic arrays of hard spherical particles suspended in a Newtonian fluid. In the highly symmetric case of cubic lattices this tensor is characterized by only two parameters. These parameters are calculated numerically for the three cubic lattice types and for the whole range of volume fractions. The correctness of the present method and its numerical implementation is confirmed by a comparison with the numerical and analytical results known from the literature. Some regular terms are determined that enter singular perturbation expansions suitable for high concentrations. Previous results for these terms are shown to be highly inaccurate. The modified expansions approach the exact numerical results over a range of densities extending to relatively low concentrations. The effective viscosity is examined for simple tetragonal (st) lattices and the results for various structures of the st type can be qualitatively understood on the basis of the motion of the spheres in response to the ambient shear flow. The angular velocity of the spheres-relative to the shear flow-is shown to be nonzero for certain orientations of the st lattice with respect to the shear flow, in contrast to what has been known for cubic arrays. Finite viscosities are found in most cases where the particles are in contact as they are allowed to move in either rigid planar or linelike structures, or they can perform a smooth rolling motion. The only occurrence where the viscosity diverges for a st structure, or equally any other Bravais lattice, is for the case of close packing. Moreover, the concentration-dependent shear viscosity is determined for a variety of microstructures and the results are compared with recent data obtained from experiments on ordered hard-sphere suspensions.