Parsimonious estimation of signal detection models from confidence ratings

A hierarchical signal detection model with unequal variance for binary responses

Gaussian signal detection models with equal variance are commonly used in simple yes-no detection and discrimination tasks whereas more flexible models with unequal variance require additional information. Here, a hierarchical Bayesian model with equal variance is extended to an unequal-variance model by exploiting variability of hit and false-alarm rates in a random sample of participants. This hierarchical model is investigated analytically, in simulations and in applications to existing data sets. The results suggest that signal variance and other parameters can be accurately estimated if plausible assumptions are met. It is concluded that the model provides a promising alternative to the ubiquitous equal-variance model for binary data.

pyWitness 1.0: A python eyewitness identification analysis toolkit

pyWitness is a python toolkit for recognition memory experiments, with a focus on eyewitness identification (ID) data analysis and model fitting. The current practice is for researchers to use different statistical packages to analyze a single dataset. pyWitness streamlines the process. In addition to conducting key data analyses (e.g., receiver operating characteristic analysis, confidence accuracy characteristic analysis), statistical comparisons, signal-detection-based model fits, simulated data generation, and power analyses are also possible. We describe the package implementation and provide detailed instructions and tutorials with datasets so that users can follow. There is also an online manual that is regularly updated. We developed pyWitness to be user-friendly, reduce human interaction with pre-processing and processing of data and model fits, and produce publication-ready plots. All pyWitness features align with open science practices, such that the algorithms, fits, and methods are reproducible and documented. While pyWitness is a python toolkit, it can also be used from R for users more accustomed to this environment.

d[Formula: see text]: Sensitivity at the optimal criterion location

Signal detection theory (SDT) was developed to provide a measure of the discriminability of a signal against background noise, independently of response bias. However, equal discriminability over a range of bias is only achieved by the traditional signal detection measure d[Formula: see text] under a narrow set of conditions - i.e., binormal noise and signal distributions of equal variance and base rates. In response to observed departures from these conditions, more robust alternative measures of d[Formula: see text] have been developed, including da and, more recently, d[Formula: see text]. Each of these alternatives addresses some, but not all, of the difficulties that arise when the assumptions of SDT are violated. Moreover, none of these measures directly follow from a central idea of discriminability by an observer that adopts a minimize error count (MEC) strategy. I propose a new d[Formula: see text] alternative, d[Formula: see text], that is robust to violations of the standard signal detection assumptions, remains consistent with varying bias, and is grounded in the principle of discriminability following a MEC strategy. Simulations illustrate how d[Formula: see text] is similar to the recently developed d[Formula: see text] when the observer optimizes their criterion placement to minimize the number of errors but, unlike d[Formula: see text], remains consistent irrespective of the observer's criterion placement Moreover, unlike da, d[Formula: see text] reflects changes in discriminability related to base rates of signal vs. noise presentations. The use of d[Formula: see text] also has implications for the interpretation of bias metrics, such as β and c, which are examined at the optimal criterion under a variety of conditions.

Rating norms should be calculated from cumulative link mixed effects models

Studies which provide norms of Likert ratings typically report per-item summary statistics. Traditionally, these summary statistics comprise the mean and the standard deviation (SD) of the ratings, and the number of observations. Such summary statistics can preserve the rank order of items, but provide distorted estimates of the relative distances between items because of the ordinal nature of Likert ratings. Inter-item relations in such ordinal scales can be more appropriately modelled by cumulative link mixed effects models (CLMMs). In a series of simulations, and with a reanalysis of an existing rating norms dataset, we show that CLMMs can be used to more accurately norm items, and can provide summary statistics analogous to the traditionally reported means and SDs, but which are disentangled from participants' response biases. CLMMs can be applied to solve important statistical issues that exist for more traditional analyses of rating norms.

Toward a more comprehensive modeling of sequential lineups

Sequential lineups are one of the most commonly used procedures in police departments across the USA. Although this procedure has been the target of much experimental research, there has been comparatively little work formally modeling it, especially the sequential nature of the judgments that it elicits. There are also important gaps in our understanding of how informative different types of judgments can be (binary responses vs. confidence ratings), and the severity of the inferential risks incurred when relying on different aggregate data structures. Couched in a signal detection theory (SDT) framework, the present work directly addresses these issues through a reanalysis of previously published data alongside model simulations. Model comparison results show that SDT modeling can provide elegant characterizations of extant data, despite some discrepancies across studies, which we attempt to address. Additional analyses compare the merits of sequential lineups (with and without a stopping rule) relative to showups and delineate the conditions in which distinct modeling approaches can be informative. Finally, we identify critical issues with the removal of the stopping rule from sequential lineups as an approach to capture within-subject differences and sidestep the risk of aggregation biases.

A Bayesian inference model for metamemory

The dual-basis theory of metamemory suggests that people evaluate their memory performance based on both processing experience during the memory process and their prior beliefs about overall memory ability. However, few studies have proposed a formal computational model to quantitatively characterize how processing experience and prior beliefs are integrated during metamemory monitoring. Here, we introduce a Bayesian inference model for metamemory (BIM) which provides a theoretical and computational framework for the metamemory monitoring process. BIM assumes that when people evaluate their memory performance, they integrate processing experience and prior beliefs via Bayesian inference. We show that BIM can be fitted to recall or recognition tasks with confidence ratings on either a continuous or discrete scale. Results from data simulation indicate that BIM can successfully recover a majority of generative parameter values, and demonstrate a systematic relationship between parameters in BIM and previous computational models of metacognition such as the stochastic detection and retrieval model (SDRM) and the meta-d' model. We also show examples of fitting BIM to empirical data sets from several experiments, which suggest that the predictions of BIM are consistent with previous studies on metamemory. In addition, when compared with SDRM, BIM could more parsimoniously account for the data of judgments of learning (JOLs) and memory performance from recall tasks. Finally, we discuss an extension of BIM which accounts for belief updating, and conclude with a discussion of how BIM may benefit metamemory research. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully

Large-scale collaborative projects recently demonstrated that several key findings from the social-science literature could not be replicated successfully. Here, we assess the extent to which a finding’s replication success relates to its intuitive plausibility. Each of 27 high-profile social-science findings was evaluated by 233 people without a Ph.D. in psychology. Results showed that these laypeople predicted replication success with above-chance accuracy (i.e., 59%). In addition, when participants were informed about the strength of evidence from the original studies, this boosted their prediction performance to 67%. We discuss the prediction patterns and apply signal detection theory to disentangle detection ability from response bias. Our study suggests that laypeople’s predictions contain useful information for assessing the probability that a given finding will be replicated successfully.