Hierarchical approximate Bayesian computation

A Modeling Framework to Examine Psychological Processes Underlying Ordinal Responses and Response Times of Psychometric Data

This article presents a joint modeling framework of ordinal responses and response times (RTs) for the measurement of latent traits. We integrate cognitive theories of decision-making and confidence judgments with psychometric theories to model individual-level measurement processes. The model development starts with the sequential sampling framework which assumes that when an item is presented, a respondent accumulates noisy evidence over time to respond to the item. Several cognitive and psychometric theories are reviewed and integrated, leading us to three psychometric process models with different representations of the cognitive processes underlying the measurement. We provide simulation studies that examine parameter recovery and show the relationships between latent variables and data distributions. We further test the proposed models with empirical data measuring three traits related to motivation. The results show that all three models provide reasonably good descriptions of observed response proportions and RT distributions. Also, different traits favor different process models, which implies that psychological measurement processes may have heterogeneous structures across traits. Our process of model building and examination illustrates how cognitive theories can be incorporated into psychometric model development to shed light on the measurement process, which has had little attention in traditional psychometric models.

d[Formula: see text]: Sensitivity at the optimal criterion location

Signal detection theory (SDT) was developed to provide a measure of the discriminability of a signal against background noise, independently of response bias. However, equal discriminability over a range of bias is only achieved by the traditional signal detection measure d[Formula: see text] under a narrow set of conditions - i.e., binormal noise and signal distributions of equal variance and base rates. In response to observed departures from these conditions, more robust alternative measures of d[Formula: see text] have been developed, including da and, more recently, d[Formula: see text]. Each of these alternatives addresses some, but not all, of the difficulties that arise when the assumptions of SDT are violated. Moreover, none of these measures directly follow from a central idea of discriminability by an observer that adopts a minimize error count (MEC) strategy. I propose a new d[Formula: see text] alternative, d[Formula: see text], that is robust to violations of the standard signal detection assumptions, remains consistent with varying bias, and is grounded in the principle of discriminability following a MEC strategy. Simulations illustrate how d[Formula: see text] is similar to the recently developed d[Formula: see text] when the observer optimizes their criterion placement to minimize the number of errors but, unlike d[Formula: see text], remains consistent irrespective of the observer's criterion placement Moreover, unlike da, d[Formula: see text] reflects changes in discriminability related to base rates of signal vs. noise presentations. The use of d[Formula: see text] also has implications for the interpretation of bias metrics, such as β and c, which are examined at the optimal criterion under a variety of conditions.

Efficient Bayesian inference for stochastic agent-based models

The modelling of many real-world problems relies on computationally heavy simulations of randomly interacting individuals or agents. However, the values of the parameters that underlie the interactions between agents are typically poorly known, and hence they need to be inferred from macroscopic observations of the system. Since statistical inference rests on repeated simulations to sample the parameter space, the high computational expense of these simulations can become a stumbling block. In this paper, we compare two ways to mitigate this issue in a Bayesian setting through the use of machine learning methods: One approach is to construct lightweight surrogate models to substitute the simulations used in inference. Alternatively, one might altogether circumvent the need for Bayesian sampling schemes and directly estimate the posterior distribution. We focus on stochastic simulations that track autonomous agents and present two case studies: tumour growths and the spread of infectious diseases. We demonstrate that good accuracy in inference can be achieved with a relatively small number of simulations, making our machine learning approaches orders of magnitude faster than classical simulation-based methods that rely on sampling the parameter space. However, we find that while some methods generally produce more robust results than others, no algorithm offers a one-size-fits-all solution when attempting to infer model parameters from observations. Instead, one must choose the inference technique with the specific real-world application in mind. The stochastic nature of the considered real-world phenomena poses an additional challenge that can become insurmountable for some approaches. Overall, we find machine learning approaches that create direct inference machines to be promising for real-world applications. We present our findings as general guidelines for modelling practitioners.

The quest for simplicity in human learning: Identifying the constraints on attention

For better or worse, humans live a resource-constrained existence; only a fraction of physical sensations ever reach conscious awareness, and we store a shockingly small subset of these experiences in memory for later use. Here, we examined the effects of attention constraints on learning. Among models that frame selective attention as an optimization problem, attention orients toward information that will reduce errors. Using this framing as a basis, we developed a suite of models with a range of constraints on the attention available during each learning event. We fit these models to both choice and eye-fixation data from four benchmark category-learning data sets, and choice data from another dynamic categorization data set. We found consistent evidence for computations we refer to as "simplicity", where attention is deployed to as few dimensions of information as possible during learning, and "competition", where dimensions compete for selective attention via lateral inhibition.

Intraobject and extraobject memory binding across early development

The ability to bind, or link, different aspects of an experience in memory undergoes protracted development across childhood. Most studies of memory binding development have assessed extraobject binding between an object and some external element such as another object, whereas little work has examined the development of intraobject binding, such as between shape and color features within the same object. In this work, we investigate the development of intra- and extraobject memory binding in five-year-olds, eight-year-olds, and young adults with a memory interference paradigm. Between two experiments, we manipulate whether stimuli are presented as coherent objects (Experiment 1: n5-year-olds = 32, 19 males, 13 females; n8-year-olds = 30, 15 males, 15 females; nadults = 30, 15 males, 15 females), requiring intraobject binding between shape and color features, or as spatially separated features (Experiment 2: n5-year-olds = 24, 16 males, 8 females; n8-year-olds = 41, 19 males, 22 females; nadults = 31, 13 males, 18 females), requiring extraobject binding. To estimate the contributions of different binding structures to performance, we present a novel computational model that mathematically instantiates the memory binding, forgetting, and retrieval processes we hypothesize to underlie performance on the task. The results provide evidence of substantial developmental improvements in both intraobject and extraobject binding of shape and color features between 5 and 8 years of age, as well as stronger intraobject compared with extraobject binding of features in all age groups. These findings provide key insights into memory binding across early development. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Individual Differences in Cue Weighting in Sentence Comprehension: An Evaluation Using Approximate Bayesian Computation

Cue-based retrieval theories of sentence processing assume that syntactic dependencies are resolved through a content-addressable search process. An important recent claim is that in certain dependency types, the retrieval cues are weighted such that one cue dominates. This cue-weighting proposal aims to explain the observed average behavior, but here we show that there is systematic individual-level variation in cue weighting. Using the Lewis and Vasishth cue-based retrieval model, we estimated individual-level parameters for reading speed and cue weighting using 13 published datasets; hierarchical approximate Bayesian computation (ABC) was used to estimate the parameters. The modeling reveals a nuanced picture of cue weighting: we find support for the idea that some participants weight cues differentially, but not all participants do. Only fast readers tend to have the predicted higher weighting for structural cues, suggesting that reading proficiency (approximated here by reading speed) might be associated with cue weighting. A broader achievement of the work is to demonstrate how individual differences can be investigated in computational models of sentence processing without compromising the complexity of the model.

Componentwise approximate Bayesian computation via Gibbs-like steps

Approximate Bayesian computation methods are useful for generative models with intractable likelihoods. These methods are, however, sensitive to the dimension of the parameter space, requiring exponentially increasing resources as this dimension grows. To tackle this difficulty we explore a Gibbs version of the approximate Bayesian computation approach that runs component-wise approximate Bayesian computation steps aimed at the corresponding conditional posterior distributions, and based on summary statistics of reduced dimensions. While lacking the standard justifications for the Gibbs sampler, the resulting Markov chain is shown to converge in distribution under some partial independence conditions. The associated stationary distribution can further be shown to be close to the true posterior distribution, and some hierarchical versions of the proposed mechanism enjoy a closed-form limiting distribution. Experiments also demonstrate the gain in efficiency brought by the Gibbs version over the standard solution.

Hierarchical Heterogeneity across Human Cortex Shapes Large-Scale Neural Dynamics

The large-scale organization of dynamical neural activity across cortex emerges through long-range interactions among local circuits. We hypothesized that large-scale dynamics are also shaped by heterogeneity of intrinsic local properties across cortical areas. One key axis along which microcircuit properties are specialized relates to hierarchical levels of cortical organization. We developed a large-scale dynamical circuit model of human cortex that incorporates heterogeneity of local synaptic strengths, following a hierarchical axis inferred from magnetic resonance imaging (MRI)-derived T1- to T2-weighted (T1w/T2w) mapping and fit the model using multimodal neuroimaging data. We found that incorporating hierarchical heterogeneity substantially improves the model fit to functional MRI (fMRI)-measured resting-state functional connectivity and captures sensory-association organization of multiple fMRI features. The model predicts hierarchically organized higher-frequency spectral power, which we tested with resting-state magnetoencephalography. These findings suggest circuit-level mechanisms linking spatiotemporal levels of analysis and highlight the importance of local properties and their hierarchical specialization on the large-scale organization of human cortical dynamics.

On the Neural and Mechanistic Bases of Self-Control

Intertemporal choice requires a dynamic interaction between valuation and deliberation processes. While evidence identifying candidate brain areas for each of these processes is well established, the precise mechanistic role carried out by each brain region is still debated. In this article, we present a computational model that clarifies the unique contribution of frontoparietal cortex regions to intertemporal decision making. The model we develop samples reward and delay information stochastically on a moment-by-moment basis. As preference for the choice alternatives evolves, dynamic inhibitory processes are executed by way of asymmetric lateral inhibition. We find that it is these lateral inhibition processes that best explain the contribution of frontoparietal regions to intertemporal decision making exhibited in our data.

Approximating Bayesian Inference through Model Simulation

The ultimate test of the validity of a cognitive theory is its ability to predict patterns of empirical data. Cognitive models formalize this test by making specific processing assumptions that yield mathematical predictions, and the mathematics allow the models to be fitted to data. As the field of cognitive science has grown to address increasingly complex problems, so too has the complexity of models increased. Some models have become so complex that the mathematics detailing their predictions are intractable, meaning that the model can only be simulated. Recently, new Bayesian techniques have made it possible to fit these simulation-based models to data. These techniques have even allowed simulation-based models to transition into neuroscience, where tests of cognitive theories can be biologically substantiated.

Bayesian statistical approaches to evaluating cognitive models

Cognitive models aim to explain complex human behavior in terms of hypothesized mechanisms of the mind. These mechanisms can be formalized in terms of mathematical structures containing parameters that are theoretically meaningful. For example, in the case of perceptual decision making, model parameters might correspond to theoretical constructs like response bias, evidence quality, response caution, and the like. Formal cognitive models go beyond verbal models in that cognitive mechanisms are instantiated in terms of mathematics and they go beyond statistical models in that cognitive model parameters are psychologically interpretable. We explore three key elements used to formally evaluate cognitive models: parameter estimation, model prediction, and model selection. We compare and contrast traditional approaches with Bayesian statistical approaches to performing each of these three elements. Traditional approaches rely on an array of seemingly ad hoc techniques, whereas Bayesian statistical approaches rely on a single, principled, internally consistent system. We illustrate the Bayesian statistical approach to evaluating cognitive models using a running example of the Linear Ballistic Accumulator model of decision making (Brown SD, Heathcote A. The simplest complete model of choice response time: linear ballistic accumulation. Cogn Psychol 2008, 57:153-178). WIREs Cogn Sci 2018, 9:e1458. doi: 10.1002/wcs.1458 This article is categorized under: Neuroscience > Computation Psychology > Reasoning and Decision Making Psychology > Theory and Methods.

Testing the validity of conflict drift-diffusion models for use in estimating cognitive processes: A parameter-recovery study

Researchers and clinicians are interested in estimating individual differences in the ability to process conflicting information. Conflict processing is typically assessed by comparing behavioral measures like RTs or error rates from conflict tasks. However, these measures are hard to interpret because they can be influenced by additional processes like response caution or bias. This limitation can be circumvented by employing cognitive models to decompose behavioral data into components of underlying decision processes, providing better specificity for investigating individual differences. A new class of drift-diffusion models has been developed for conflict tasks, presenting a potential tool to improve analysis of individual differences in conflict processing. However, measures from these models have not been validated for use in experiments with limited data collection. The present study assessed the validity of these models with a parameter-recovery study to determine whether and under what circumstances the models provide valid measures of cognitive processing. Three models were tested: the dual-stage two-phase model (Hübner, Steinhauser, & Lehle, Psychological Review, 117(3), 759-784, 2010), the shrinking spotlight model (White, Ratcliff, & Starns, Cognitive Psychology, 63(4), 210-238, 2011), and the diffusion model for conflict tasks (Ulrich, Schröter, Leuthold, & Birngruber, Cogntive Psychology, 78, 148-174, 2015). The validity of the model parameters was assessed using different methods of fitting the data and different numbers of trials. The results show that each model has limitations in recovering valid parameters, but they can be mitigated by adding constraints to the model. Practical recommendations are provided for when and how each model can be used to analyze data and provide measures of processing in conflict tasks.

Approximate Bayesian computation techniques for optical characterization of nanoparticle clusters

Characterization of nanoparticle aggregates from observed scattered light leads to a highly complex inverse problem. Even the forward model is so complex that it prohibits the use of classical likelihood-based inference methods. In this study, we compare four so-called likelihood-free methods based on approximate Bayesian computation (ABC) that requires only numeric simulation of the forward model without the need of evaluating a likelihood. In particular, rejection, Markov chain Monte Carlo, population Monte Carlo, and adaptive population Monte Carlo (APMC) are compared in terms of accuracy. In the current model, we assume that the nanoparticle aggregates are mutually well separated and made up of particles of same size. Filippov's particle-cluster algorithm is used to generate aggregates, and discrete dipole approximation is used to estimate scattering behavior. It is found that the APMC algorithm is superior to others in terms of time and acceptance rates, although all algorithms produce similar posterior distributions. Using ABC techniques and utilizing unpolarized light experiments at 266 nm wavelength, characterization of soot aggregates is performed with less than 2 nm deviation in nanoparticle radius and 3-4 deviation in number of nanoparticles forming the monodisperse aggregates. Promising results are also observed for the polydisperse aggregate with log-normal particle size distribution.

Bayesian statistics to test Bayes optimality

We agree with the authors that putting forward specific models and examining their agreement with experimental data are the best approach for understanding the nature of decision making. Although the authors only consider the likelihood function, prior, cost function, and decision rule (LPCD) framework, other choices are available. Bayesian statistics can be used to estimate essential parameters and assess the degree of optimality.

Evidence synthesis from aggregate recurrent event data for clinical trial design and analysis

Information from historical trials is important for the design, interim monitoring, analysis, and interpretation of clinical trials. Meta-analytic models can be used to synthesize the evidence from historical data, which are often only available in aggregate form. We consider evidence synthesis methods for trials with recurrent event endpoints, which are common in many therapeutic areas. Such endpoints are typically analyzed by negative binomial regression. However, the individual patient data necessary to fit such a model are usually unavailable for historical trials reported in the medical literature. We describe approaches for back-calculating model parameter estimates and their standard errors from available summary statistics with various techniques, including approximate Bayesian computation. We propose to use a quadratic approximation to the log-likelihood for each historical trial based on 2 independent terms for the log mean rate and the log of the dispersion parameter. A Bayesian hierarchical meta-analysis model then provides the posterior predictive distribution for these parameters. Simulations show this approach with back-calculated parameter estimates results in very similar inference as using parameter estimates from individual patient data as an input. We illustrate how to design and analyze a new randomized placebo-controlled exacerbation trial in severe eosinophilic asthma using data from 11 historical trials.

Strategies for improving approximate Bayesian computation tests for synchronous diversification

Background

Estimating the variability in isolation times across co-distributed taxon pairs that may have experienced the same allopatric isolating mechanism is a core goal of comparative phylogeography. The use of hierarchical Approximate Bayesian Computation (ABC) and coalescent models to infer temporal dynamics of lineage co-diversification has been a contentious topic in recent years. Key issues that remain unresolved include the choice of an appropriate prior on the number of co-divergence events (Ψ), as well as the optimal strategies for data summarization.

Methods

Through simulation-based cross validation we explore the impact of the strategy for sorting summary statistics and the choice of prior on Ψ on the estimation of co-divergence variability. We also introduce a new setting (β) that can potentially improve estimation of Ψ by enforcing a minimal temporal difference between pulses of co-divergence. We apply this new method to three empirical datasets: one dataset each of co-distributed taxon pairs of Panamanian frogs and freshwater fishes, and a large set of Neotropical butterfly sister-taxon pairs.

Results

We demonstrate that the choice of prior on Ψ has little impact on inference, but that sorting summary statistics yields substantially more reliable estimates of co-divergence variability despite violations of assumptions about exchangeability. We find the implementation of β improves estimation of Ψ, with improvement being most dramatic given larger numbers of taxon pairs. We find equivocal support for synchronous co-divergence for both of the Panamanian groups, but we find considerable support for asynchronous divergence among the Neotropical butterflies.

Conclusions

Our simulation experiments demonstrate that using sorted summary statistics results in improved estimates of the variability in divergence times, whereas the choice of hyperprior on Ψ has negligible effect. Additionally, we demonstrate that estimating the number of pulses of co-divergence across co-distributed taxon-pairs is improved by applying a flexible buffering regime over divergence times. This improves the correlation between Ψ and the true variability in isolation times and allows for more meaningful interpretation of this hyperparameter. This will allow for more accurate identification of the number of temporally distinct pulses of co-divergence that generated the diversification pattern of a given regional assemblage of sister-taxon-pairs.

Electronic supplementary material

The online version of this article (doi:10.1186/s12862-017-1052-6) contains supplementary material, which is available to authorized users.

Approaches to Analysis in Model-based Cognitive Neuroscience

Our understanding of cognition has been advanced by two traditionally nonoverlapping and non-interacting groups. Mathematical psychologists rely on behavioral data to evaluate formal models of cognition, whereas cognitive neuroscientists rely on statistical models to understand patterns of neural activity, often without any attempt to make a connection to the mechanism supporting the computation. Both approaches suffer from critical limitations as a direct result of their focus on data at one level of analysis (cf. Marr, 1982), and these limitations have inspired researchers to attempt to combine both neural and behavioral measures in a cross-level integrative fashion. The importance of solving this problem has spawned several entirely new theoretical and statistical frameworks developed by both mathematical psychologists and cognitive neuroscientists. However, with each new approach comes a particular set of limitations and benefits. In this article, we survey and characterize several approaches for linking brain and behavioral data. We organize these approaches on the basis of particular cognitive modeling goals: (1) using the neural data to constrain a behavioral model, (2) using the behavioral model to predict neural data, and (3) fitting both neural and behavioral data simultaneously. Within each goal, we highlight a few particularly successful approaches for accomplishing that goal, and discuss some applications. Finally, we provide a conceptual guide to choosing among various analytic approaches in performing model-based cognitive neuroscience.

A generalized, likelihood-free method for posterior estimation

Recent advancements in Bayesian modeling have allowed for likelihood-free posterior estimation. Such estimation techniques are crucial to the understanding of simulation-based models, whose likelihood functions may be difficult or even impossible to derive. However, current approaches are limited by their dependence on sufficient statistics and/or tolerance thresholds. In this article, we provide a new approach that requires no summary statistics, error terms, or thresholds and is generalizable to all models in psychology that can be simulated. We use our algorithm to fit a variety of cognitive models with known likelihood functions to ensure the accuracy of our approach. We then apply our method to two real-world examples to illustrate the types of complex problems our method solves. In the first example, we fit an error-correcting criterion model of signal detection, whose criterion dynamically adjusts after every trial. We then fit two models of choice response time to experimental data: the linear ballistic accumulator model, which has a known likelihood, and the leaky competing accumulator model, whose likelihood is intractable. The estimated posterior distributions of the two models allow for direct parameter interpretation and model comparison by means of conventional Bayesian statistics-a feat that was not previously possible.