Beyond ROC curvature: Strength effects and response time data support continuous-evidence models of recognition memory

Sometimes memory misleads: Variants of the error-speed effect strengthen the evidence for systematically misleading memory signals in recognition memory

The error-speed effect describes the observation that the speed of recognition errors in a first binary recognition task predicts the response accuracy in a subsequent two-alternative forced-choice (2AFC) task that comprises the erroneously judged items of the first task. So far, the effect has been primarily explained by the assumption that some error responses result from misleading memory evidence. However, it is also possible that the effect arises because participants remember and use their response times from the binary task to solve the 2AFC task. Furthermore, the phenomenon is quite new and its robustness or generalizability across other recognition tasks (e.g., a confidence-rating task) remains to be demonstrated. The aim of the present study is to address these limitations by introducing a new variant of the error-speed effect, replacing the 2AFC task with a confidence-rating task (Experiment 1), and by reversing task order (Experiment 2) to test whether participants employ a response-time strategy. In both experiments, we collected data using a sequential probability ratio t-test procedure and found evidence in favor of the hypothesis that the speed of binary recognition errors predicts confidence ratings for the same stimulus. These results attest to the robustness and generalizability of the error-speed effect and reveal that at least some errors must be due to systematically misleading memory evidence.

ROC measures of memory accessibility

Laming has recently proposed a way to measure the accessibility (as opposed to availability) of memories via recognition testing. His measure "Accessibility" is calculated by subtracting the hit rate and false alarm rate that fall at the point where the receiver-operating characteristic (ROC) curve's derivative is 1. I prove that, if one works within the framework of Unequal-Variance Signal Detection Theory (UVSDT), as Laming does, the measure "Accessibility" depends on the location of the response criterion (though always with a neutral likelihood ratio). Furthermore, I prove that the measure varies with the underlying variances of UVSDT regardless of which definition of bias (criterion or likelihood ratio) is used and, crucially, this holds even when the accuracy of discrimination performance or "sensitivity" (da) in UVSDT is constant. As such, from the standpoint of (at least) UVSDT, it is questionable whether or to what extent the new measure of "Accessibility" actually measures the accessibility of any memory.

Closing the door to false memory: the effects of levels-of-processing and stimulus type on the rejection of perceptually vs. semantically dissimilar distractors

False recognition memory for nonstudied items that share features with targets can be reduced by retrieval monitoring mechanisms. The recall-to-reject process, for example, involves the recollection of information about studied items that disqualifies inconsistent test probes. Monitoring for specific features during retrieval may be enhanced by an encoding orientation that is recapitulated during retrieval. In two experiments, we used concrete words or door scenes as materials and manipulated the level of processing at study and the type of distractors presented at test. We showed that for the verbal material, semantic level of processing at study results in an effective rejection of semantically inconsistent distractors. However, for the pictorial material, the perceptual level of processing leads to an effective rejection of perceptually inconsistent distractors. For targets, the effect of levels of processing was observed for words but not for pictures. The results suggest that retrieval monitoring mechanisms depend on interactions between encoding orientation, study materials, and differentiating features of distractors.

A circular diffusion model of continuous-outcome source memory retrieval: Contrasting continuous and threshold accounts

A circular analogue of the diffusion model adapted for continuous response tasks is applied to a continuous-outcome source memory task. In contrast to existing models of source retrieval that attribute all of the variability in responding to memory, the circular diffusion model decomposes noise into variability arising from memory and from decision processes. We compared three models: (1) a single diffusion process with trial-to-trial variability in drift rate, (2) a mixture of two diffusion processes, one with positive drift that does not vary from trial-to-trial, and a second zero-drift process that represents discrete guessing, and (3) a hybrid model that mixed positive and zero-drift processes with trial-to-trial variability in the positive drift process. Comparison of model fits to joint response error and response-time (RT) data suggest that a memory strength threshold under which no information is retrieved appears to underlie responding in a continuous-report source memory task. Additionally, we also conditioned participants' source responding on their confidence in an old/new recognition task, ruling out the possibility that participant guessing was only due to unrecognized items. Overall, our findings support an all-or-none or some-or none view of source memory retrieval and pose a challenge to continuous models of source memory.

rtmpt: An R package for fitting response-time extended multinomial processing tree models

Response-time extended multinomial processing tree models (RT-MPT; Klauer and Kellen, Journal of Mathematical Psychology, 82, 111-130 2018) provide estimates of process-completion times for cognitive processes modeled by means of multinomial processing tree (MPT) models (Batchelder and Riefer, Psychonomic Bulletin & Review, 6, 57-86 1999). We present the R package rtmpt with which it is possible to fit RT-MPT models easily. The package is free and open source, it can be used with two established MPT syntaxes, and has a number of useful features, such as suppressing process-completion times for specific process outcomes, holding process probabilities constant, and changing some prior parameters. In the background of the R package, an altered version of the original C++ code is used for the MCMC sampling. We provide a guide to using rtmpt, validate the underlying hierarchical Bayesian algorithm of rtmpt using simulation-based calibration and show that previously reported results can be reproduced using rtmpt.

Interaction effects on common measures of sensitivity: choice of measure, type I error, and power

Here we use simulation to assess previously unaddressed problems in the assessment of statistical interactions in detection and recognition tasks. The proportion of hits and false-alarms made by an observer on such tasks is affected by both their sensitivity and bias, and numerous measures have been developed to separate out these two factors. Each of these measures makes different assumptions regarding the underlying process and different predictions as to how false-alarm and hit rates should covary. Previous simulations have shown that choice of an inappropriate measure can lead to inflated type I error rates, or reduced power, for main effects, provided there are differences in response bias between the conditions being compared. Interaction effects pose a particular problem in this context. We show that spurious interaction effects in analysis of variance can be produced, or true interactions missed, even in the absence of variation in bias. Additional simulations show that variation in bias complicates patterns of type I error and power further. This under-appreciated fact has the potential to greatly distort the assessment of interactions in detection and recognition experiments. We discuss steps researchers can take to mitigate their chances of making an error.

Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis

Scientific advances across a range of disciplines hinge on the ability to make inferences about unobservable theoretical entities on the basis of empirical data patterns. Accurate inferences rely on both discovering valid, replicable data patterns and accurately interpreting those patterns in terms of their implications for theoretical constructs. The replication crisis in science has led to widespread efforts to improve the reliability of research findings, but comparatively little attention has been devoted to the validity of inferences based on those findings. Using an example from cognitive psychology, we demonstrate a blinded-inference paradigm for assessing the quality of theoretical inferences from data. Our results reveal substantial variability in experts’ judgments on the very same data, hinting at a possible inference crisis.

Familiarity, recollection, and receiver-operating characteristic (ROC) curves in recognition memory

The Atkinson-Shiffrin theory describes and explains some of the processes involved in storing and retrieving information in human memory. Here we examine predictions of related models for search and decision processes in recognizing information in long-term memory. In some models, recognition is presumably based on a test item's familiarity judgment, and subsequent decisions follow from the sensitivity and decision parameters of signal detection theory. Other models dispense with the continuous notion of familiarity and base recognition on discrete internal states such as relative certainty that an item has or has not been previously studied, with an intermediate state of uncertainty that produces guesses. Still others are hybrid models with two criteria located along a familiarity continuum defining areas for rapid decisions based on high or low familiarities. For intermediate familiarity values, the decision can be delayed pending the results of search for, and occasional recollection of, relevant episodic information. Here we present the results from a study of human recognition memory for lists of words using both response time and error data to construct receiver-operating characteristic (ROC) curves derived from three standard methods based on the same data set. Models are evaluated against, and parameters estimated from, group as well as individual subjects' behavior. We report substantially different ROC curves when they are based on variations in target-word frequency, confidence judgments, and response latencies. The results indicate that individual versus group data must be used with caution in determining the appropriate theoretical interpretation of recognition memory performance.

Task effects determine whether recognition memory is mediated discretely or continuously

How recognition memory is mediated has been of interest to researchers for decades. But the apparent consensus implicating continuous mediation has been challenged. McAdoo, Key, and Gronlund (Journal of Experimental Psychology: Learning, Memory, and Cognition,2018. Advanced online publication) demonstrated that recognition memory can be mediated by either discrete or continuous evidence, depending on target-filler similarity. The present paper expands on this research by showing that different recognition tasks also can be mediated by different evidence. Specifically, recognition memory was mediated by continuous evidence in a ranking task, but by discrete evidence in a confidence-rating task. We posit that participants utilize a control process that induces a reliance on discrete or continuous evidence as a function of efficiency (Malmberg, 2008) to suit the demands of the task.

Characterizing belief bias in syllogistic reasoning: A hierarchical Bayesian meta-analysis of ROC data

The belief-bias effect is one of the most-studied biases in reasoning. A recent study of the phenomenon using the signal detection theory (SDT) model called into question all theoretical accounts of belief bias by demonstrating that belief-based differences in the ability to discriminate between valid and invalid syllogisms may be an artifact stemming from the use of inappropriate linear measurement models such as analysis of variance (Dube et al., Psychological Review, 117(3), 831-863, 2010). The discrepancy between Dube et al.'s, Psychological Review, 117(3), 831-863 (2010) results and the previous three decades of work, together with former's methodological criticisms suggests the need to revisit earlier results, this time collecting confidence-rating responses. Using a hierarchical Bayesian meta-analysis, we reanalyzed a corpus of 22 confidence-rating studies (N = 993). The results indicated that extensive replications using confidence-rating data are unnecessary as the observed receiver operating characteristic functions are not systematically asymmetric. These results were subsequently corroborated by a novel experimental design based on SDT's generalized area theorem. Although the meta-analysis confirms that believability does not influence discriminability unconditionally, it also confirmed previous results that factors such as individual differences mediate the effect. The main point is that data from previous and future studies can be safely analyzed using appropriate hierarchical methods that do not require confidence ratings. More generally, our results set a new standard for analyzing data and evaluating theories in reasoning. Important methodological and theoretical considerations for future work on belief bias and related domains are discussed.

Generalized Processing Tree Models: Jointly Modeling Discrete and Continuous Variables

Multinomial processing tree models assume that discrete cognitive states determine observed response frequencies. Generalized processing tree (GPT) models extend this conceptual framework to continuous variables such as response times, process-tracing measures, or neurophysiological variables. GPT models assume finite-mixture distributions, with weights determined by a processing tree structure, and continuous components modeled by parameterized distributions such as Gaussians with separate or shared parameters across states. We discuss identifiability, parameter estimation, model testing, a modeling syntax, and the improved precision of GPT estimates. Finally, a GPT version of the feature comparison model of semantic categorization is applied to computer-mouse trajectories.

The speed of memory errors shows the influence of misleading information: Testing the diffusion model and discrete-state models

In this report, we evaluate single-item and forced-choice recognition memory for the same items and use the resulting accuracy and reaction time data to test the predictions of discrete-state and continuous models. For the single-item trials, participants saw a word and indicated whether or not it was studied on a previous list. The forced-choice trials had one studied and one non-studied word that both appeared in the earlier single-item trials and both received the same response. Thus, forced-choice trials always had one word with a previous correct response and one with a previous error. Participants were asked to select the studied word regardless of whether they previously called both words "studied" or "not studied." The diffusion model predicts that forced-choice accuracy should be lower when the word with a previous error had a fast versus a slow single-item RT, because fast errors are associated with more compelling misleading memory retrieval. The two-high-threshold (2HT) model does not share this prediction because all errors are guesses, so error RT is not related to memory strength. A low-threshold version of the discrete state approach predicts an effect similar to the diffusion model, because errors are a mixture of responses based on misleading retrieval and guesses, and the guesses should tend to be slower. Results showed that faster single-trial errors were associated with lower forced-choice accuracy, as predicted by the diffusion and low-threshold models.

Assessing recognition memory using confidence ratings and response times

Classification of stimuli into categories (such as 'old' and 'new' in tests of recognition memory or 'present' versus 'absent' in signal detection tasks) requires the mapping of internal signals to discrete responses. Introspective judgements about a given choice response are regularly employed in research, legal and clinical settings in an effort to measure the signal that is thought to be the basis of the classification decision. Correlations between introspective judgements and task performance suggest that such ratings often do convey information about internal states that are relevant for a given task, but well-known limitations of introspection call the fidelity of this information into question. We investigated to what extent response times can reveal information usually assessed with explicit confidence ratings. We quantitatively compared response times to confidence ratings in their ability to qualify recognition memory decisions and found convergent results suggesting that much of the information from confidence ratings can be obtained from response times.

Further Evidence for Discrete-State Mediation in Recognition Memory

The two high threshold model (2HTM) of recognition memory makes strong predictions regarding differences between receiver operating characteristics (ROC) functions across strength manipulations. Province and Rouder (2012) tested these predictions and showed that the 2HTM provided a better account of the data than a continuous signal detection model using an extended two-alternative forced-choice task. The present study replicates and extends Province and Rouder’s findings at the level of confidence-rating responses as well as their associated response times. Model-mimicry simulations are also reported, ascertaining that the models can be well discriminated in this experimental design. Supplemental files for this article are available at osf.io/zadt6/

Examining Event-Related Potential (ERP) correlates of decision bias in recognition memory judgments

Memory judgments can be based on accurate memory information or on decision bias (the tendency to report that an event is part of episodic memory when one is in fact unsure). Event related potentials (ERP) correlates are important research tools for elucidating the dynamics underlying memory judgments but so far have been established only for investigations of accurate old/new discrimination. To identify the ERP correlates of bias, and observe how these interact with ERP correlates of memory, we conducted three experiments that manipulated decision bias within participants via instructions during recognition memory tests while their ERPs were recorded. In Experiment 1, the bias manipulation was performed between blocks of trials (automatized bias) and compared to trial-by-trial shifts of bias in accord with an external cue (flexibly controlled bias). In Experiment 2, the bias manipulation was performed at two different levels of accurate old/new discrimination as the memory strength of old (studied) items was varied. In Experiment 3, the bias manipulation was added to another, bottom-up driven manipulation of bias induced via familiarity. In the first two Experiments, and in the low familiarity condition of Experiment 3, we found evidence of an early frontocentral ERP component at 320 ms poststimulus (the FN320) that was sensitive to the manipulation of bias via instruction, with more negative amplitudes indexing more liberal bias. By contrast, later during the trial (500–700 ms poststimulus), bias effects interacted with old/new effects across all three experiments. Results suggest that the decision criterion is typically activated early during recognition memory trials, and is integrated with retrieved memory signals and task-specific processing demands later during the trial. More generally, the findings demonstrate how ERPs can help to specify the dynamics of recognition memory processes under top-down and bottom-up controlled retrieval conditions.

Using response time modeling to distinguish memory and decision processes in recognition and source tasks

Receiver operating characteristic (ROC) functions are often used to make inferences about memory processes, such as claiming that memory strength is more variable for studied versus nonstudied items. However, decision processes can produce the ROC patterns that are usually attributed to memory, so independent forms of data are needed to support strong conclusions. The present experiments tested ROC-based claims about the variability of memory evidence by modeling response time (RT) data with the diffusion model. To ensure that the model can correctly discriminate equal- and unequal-variance distributions, Experiment 1 used a numerousity discrimination task that had a direct manipulation of evidence variability. Fits of the model produced correct conclusions about evidence variability in all cases. Experiments 2 and 3 explored the effect of repeated learning trials on evidence variability in recognition and source memory tasks, respectively. Fits of the diffusion model supported the same conclusions about variability as the ROC literature. For recognition, evidence variability was higher for targets than for lures, but it did not differ on the basis of the number of learning trials for target items. For source memory, evidence variability was roughly equal for source 1 and source 2 items, and variability increased for items with additional learning attempts. These results demonstrate that RT modeling can help resolve ambiguities regarding the processes that produce different patterns in ROC data. The results strengthen the evidence that memory strength distributions have unequal variability across item types in recognition and source memory tasks.

Validating the unequal-variance assumption in recognition memory using response time distributions instead of ROC functions: A diffusion model analysis

Recognition memory z-transformed Receiver Operating Characteristic (zROC) functions have a slope less than 1. One way to accommodate this finding is to assume that memory evidence is more variable for studied (old) items than non-studied (new) items. This assumption has been implemented in signal detection models, but this approach cannot accommodate the time course of decision making. We tested the unequal-variance assumption by fitting the diffusion model to accuracy and response time (RT) distributions from nine old/new recognition data sets comprising previously-published data from 376 participants. The η parameter in the diffusion model measures between-trial variability in evidence based on accuracy and the RT distributions for correct and error responses. In fits to nine data sets, η estimates were higher for targets than lures in all cases, and fitting results rejected an equal-variance version of the model in favor of an unequal-variance version. Parameter recovery simulations showed that the variability differences were not produced by biased estimation of the η parameter. Estimates of the other model parameters were largely consistent between the equal- and unequal-variance versions of the model. Our results provide independent support for the unequal-variance assumption without using zROC data.

Discrete-slots models of visual working-memory response times

Much recent research has aimed to establish whether visual working memory (WM) is better characterized by a limited number of discrete all-or-none slots or by a continuous sharing of memory resources. To date, however, researchers have not considered the response-time (RT) predictions of discrete-slots versus shared-resources models. To complement the past research in this field, we formalize a family of mixed-state, discrete-slots models for explaining choice and RTs in tasks of visual WM change detection. In the tasks under investigation, a small set of visual items is presented, followed by a test item in 1 of the studied positions for which a change judgment must be made. According to the models, if the studied item in that position is retained in 1 of the discrete slots, then a memory-based evidence-accumulation process determines the choice and the RT; if the studied item in that position is missing, then a guessing-based accumulation process operates. Observed RT distributions are therefore theorized to arise as probabilistic mixtures of the memory-based and guessing distributions. We formalize an analogous set of continuous shared-resources models. The model classes are tested on individual subjects with both qualitative contrasts and quantitative fits to RT-distribution data. The discrete-slots models provide much better qualitative and quantitative accounts of the RT and choice data than do the shared-resources models, although there is some evidence for "slots plus resources" when memory set size is very small.

Internal reinstatement hides cuing effects in source memory tasks

Reinstating source details at test often has no impact on source memory. We tested the proposition that participants internally reinstate source cues when such cues are not provided by the experimenter, thus making the external cues redundant. Participants studied words paired with either a male or a female face and were later asked to specify the gender of the face studied with each word. To disrupt the ability to internally reinstate sources, some participants saw eight male faces and eight female faces throughout the study list (multiple-face condition), making it difficult to determine which face should be internally reinstated for uncued test trials. Other participants saw only a single face for each gender (single-face condition), which should facilitate internal reinstatement. Across three experiments, participants in the multiple-face condition showed improved source discrimination when the studied faces were reinstated at test, as compared to uncued trials. In contrast, participants in the single-face condition showed no effect of the face cues. Moreover, the cuing effect for the multiple-face condition disappeared when the test structure facilitated internal reinstatement. Overall, the experiments support the contention that internal reinstatement is a natural part of source retrieval that can mask the effects of external cues.

Unequal-strength source zROC slopes reflect criteria placement and not (necessarily) memory processes

Source memory zROC slopes change from below 1 to above 1 depending on which source gets the strongest learning. This effect has been attributed to memory processes, either in terms of a threshold source recollection process or changes in the variability of continuous source evidence. We propose 2 decision mechanisms that can produce the slope effect, and we test them in 3 experiments. The evidence mixing account assumes that people change how they weight item versus source evidence based on which source is stronger, and the converging criteria account assumes that participants become more willing to make high confidence source responses for test probes that have higher item strength. Results failed to support the evidence mixing account, in that the slope effect emerged even when item evidence was not informative for the source judgment (i.e., in tests that included strong and weak items from both sources). In contrast, results showed strong support for the converging criteria account. This account not only accommodated the unequal-strength slope effect but also made a prediction for unstudied (new) items that was empirically confirmed: participants made more high confidence source responses for new items when they were more confident that the item was studied. The converging criteria account has an advantage over accounts based on source recollection or evidence variability, as the latter accounts do not predict the relationship between recognition and source confidence for new items.