Bayesian adaptive estimation of arbitrary points on a psychometric function

Model Based or Model Free? Comparing Adaptive Methods for Estimating Thresholds in Neuroscience

The quantification of human perception through the study of psychometric functions Ψ is one of the pillars of experimental psychophysics. In particular, the evaluation of the threshold is at the heart of many neuroscience and cognitive psychology studies, and a wide range of adaptive procedures has been developed to improve its estimation. However, these procedures are often implicitly based on different mathematical assumptions on the psychometric function, and unfortunately, these assumptions cannot always be validated prior to data collection. This raises questions about the accuracy of the estimator produced using the different procedures. In the study we examine in this letter, we compare five adaptive procedures commonly used in psychophysics to estimate the threshold: Dichotomous Optimistic Search (DOS), Staircase, PsiMethod, Gaussian Processes, and QuestPlus. These procedures range from model-based methods, such as the PsiMethod, which relies on strong assumptions regarding the shape of Ψ, to model-free methods, such as DOS, for which assumptions are minimal. The comparisons are performed using simulations of multiple experiments, with psychometric functions of various complexity. The results show that while model-based methods perform well when Ψ is an ideal psychometric function, model-free methods rapidly outshine them when Ψ deviates from this model, as, for instance, when Ψ is a beta cumulative distribution function. Our results highlight the importance of carefully choosing the most appropriate method depending on the context.

Order effects in two-alternative forced-choice tasks invalidate adaptive threshold estimates

Adaptive psychophysical methods are widely used for the quick estimation of percentage points (thresholds) on psychometric functions for two-alternative forced-choice (2AFC) tasks. The use of adaptive methods is supported by numerous simulation studies documenting their performance, which have shown that thresholds can be reasonably estimated with them when their founding assumptions hold. One of these assumptions is that the psychometric function is invariant, but empirical evidence is mounting that human performance in 2AFC tasks needs to be described by two different psychometric functions, one that holds when the test stimulus is presented first in the 2AFC trial and a different one that holds when the test is presented second. The same holds when presentations are instead simultaneous at two spatial locations rather than sequential. We re-evaluated the performance of adaptive methods in the presence of these order effects via simulation studies and an empirical study with human observers. The simulation study showed that thresholds are severely overestimated by adaptive methods in these conditions, and the empirical study corroborated these findings. These results question the validity of threshold estimates obtained with adaptive methods that incorrectly assume the psychometric function to be invariant with presentation order. Alternative ways in which thresholds can be accurately estimated in the presence of order effects are discussed.

A novel Bayesian adaptive method for mapping the visual field

Measuring visual functions such as light and contrast sensitivity, visual acuity, reading speed, and crowding across retinal locations provides visual-field maps (VFMs) that are extremely valuable for detecting and managing eye diseases. Although mapping light sensitivity is a standard glaucoma test, the measurement is often noisy (Keltner et al., 2000). Mapping other visual functions is even more challenging. To improve the precision of light-sensitivity mapping and enable other VFM assessments, we developed a novel hybrid Bayesian adaptive testing framework, the qVFM method. The method combines a global module for preliminary assessment of the VFM's shape and a local module for assessing individual visual-field locations. This study validates the qVFM method in measuring light sensitivity across the visual field. In both simulation and psychophysics studies, we sampled 100 visual-field locations (60° × 60°) and compared the performance of qVFM with the qYN procedure (Lesmes et al., 2015) that measured light sensitivity at each location independently. In the simulations, a simulated observer was tested monocularly for 1,000 runs with 1,200 trials/run, to compare the accuracy and precision of the two methods. In the experiments, data were collected from 12 eyes (six left, six right) of six human subjects. Subjects were cued to report the presence or absence of a target stimulus, with the luminance and location of the target adaptively selected in each trial. Both simulations and a psychological experiment showed that the qVFM method can provide accurate, precise, and efficient mapping of light sensitivity. This method can be extended to map other visual functions, with potential clinical signals for monitoring vision loss, evaluating therapeutic interventions, and developing effective rehabilitation for low vision.

Assessing reading performance in the periphery with a Bayesian adaptive approach: The qReading method

Reading is a crucial visual activity and a fundamental skill in daily life. Rapid Serial Visual Presentation (RSVP) is a text-presentation paradigm that has been extensively used in the laboratory to study basic characteristics of reading performance. However, measuring reading function (reading speed vs. print size) is time-consuming for RSVP reading using conventional testing procedures. In this study, we develop a novel method, qReading, utilizing the Bayesian adaptive testing framework to measure reading function in the periphery. We perform both a psychophysical experiment and computer simulations to validate the qReading method. In the experiment, words are presented using an RSVP paradigm at 10° in the lower visual field. The reading function obtained from the qReading method with 50 trials exhibits good agreement (i.e., high accuracy) with the reading function obtained from a conventional method (method of constant stimuli [MCS]) with 186 trials (mean root mean square error: 0.12 log10 units). Simulations further confirm that the qReading method provides an unbiased measure. The qReading procedure also demonstrates excellent precision (half width of 68.2% credible interval: 0.02 log10 units with 50 trials) compared to the MCS method (0.03 log10 units with 186 trials). This investigation establishes that the qReading method can adequately measure the reading function in the normal periphery with high accuracy, precision, and efficiency, and is a potentially valuable tool for both research and clinical assessments.

Determining thresholds using adaptive procedures and psychometric fits: evaluating efficiency using theory, simulations, and human experiments

When measuring thresholds, careful selection of stimulus amplitude can increase efficiency by increasing the precision of psychometric fit parameters (e.g., decreasing the fit parameter error bars). To find efficient adaptive algorithms for psychometric threshold ("sigma") estimation, we combined analytic approaches, Monte Carlo simulations, and human experiments for a one-interval, binary forced-choice, direction-recognition task. To our knowledge, this is the first time analytic results have been combined and compared with either simulation or human results. Human performance was consistent with theory and not significantly different from simulation predictions. Our analytic approach provides a bound on efficiency, which we compared against the efficiency of standard staircase algorithms, a modified staircase algorithm with asymmetric step sizes, and a maximum likelihood estimation (MLE) procedure. Simulation results suggest that optimal efficiency at determining threshold is provided by the MLE procedure targeting a fraction correct level of 0.92, an asymmetric 4-down, 1-up staircase targeting between 0.86 and 0.92 or a standard 6-down, 1-up staircase. Psychometric test efficiency, computed by comparing simulation and analytic results, was between 41 and 58% for 50 trials for these three algorithms, reaching up to 84% for 200 trials. These approaches were 13-21% more efficient than the commonly used 3-down, 1-up symmetric staircase. We also applied recent advances to reduce accuracy errors using a bias-reduced fitting approach. Taken together, the results lend confidence that the assumptions underlying each approach are reasonable and that human threshold forced-choice decision making is modeled well by detection theory models and mimics simulations based on detection theory models.

Developing Bayesian adaptive methods for estimating sensitivity thresholds (d') in Yes-No and forced-choice tasks

Motivated by Signal Detection Theory (SDT), we developed a family of novel adaptive methods that estimate the sensitivity threshold—the signal intensity corresponding to a pre-defined sensitivity level (d′ = 1)—in Yes-No (YN) and Forced-Choice (FC) detection tasks. Rather than focus stimulus sampling to estimate a single level of %Yes or %Correct, the current methods sample psychometric functions more broadly, to concurrently estimate sensitivity and decision factors, and thereby estimate thresholds that are independent of decision confounds. Developed for four tasks—(1) simple YN detection, (2) cued YN detection, which cues the observer's response state before each trial, (3) rated YN detection, which incorporates a Not Sure response, and (4) FC detection—the qYN and qFC methods yield sensitivity thresholds that are independent of the task's decision structure (YN or FC) and/or the observer's subjective response state. Results from simulation and psychophysics suggest that 25 trials (and sometimes less) are sufficient to estimate YN thresholds with reasonable precision (s.d. = 0.10–0.15 decimal log units), but more trials are needed for FC thresholds. When the same subjects were tested across tasks of simple, cued, rated, and FC detection, adaptive threshold estimates exhibited excellent agreement with the method of constant stimuli (MCS), and with each other. These YN adaptive methods deliver criterion-free thresholds that have previously been exclusive to FC methods.

Aniseikonia Tests: The Role of Viewing Mode, Response Bias, and Size-Color Illusions

PURPOSE

To identify the factors responsible for the poor validity of the most common aniseikonia tests, which involve size comparisons of red-green stimuli presented haploscopically.

METHODS

Aniseikonia was induced by afocal size lenses placed before one eye. Observers compared the sizes of semicircles presented haploscopically via color filters. The main factor under study was viewing mode (free viewing versus short presentations under central fixation). To eliminate response bias, a three-response format allowed observers to respond if the left, the right, or neither semicircle appeared larger than the other. To control decisional (criterion) bias, measurements were taken with the lens-magnified stimulus placed on the left and on the right. To control for size-color illusions, measurements were made with color filters in both arrangements before the eyes and under binocular vision (without color filters).

RESULTS

Free viewing resulted in a systematic underestimation of lens-induced aniseikonia that was absent with short presentations. Significant size-color illusions and decisional biases were found that would be mistaken for aniseikonia unless appropriate action is taken.

CONCLUSIONS

To improve their validity, aniseikonia tests should use short presentations and include control conditions to prevent contamination from decisional/response biases. If anaglyphs are used, presence of size-color illusions must be checked for.

TRANSLATIONAL RELEVANCE

We identified optimal conditions for administration of aniseikonia tests and appropriate action for differential diagnosis of aniseikonia in the presence of response biases or size-color illusions. Our study has clinical implications for aniseikonia management.

Does time ever fly or slow down? The difficult interpretation of psychophysical data on time perception

Time perception is studied with subjective or semi-objective psychophysical methods. With subjective methods, observers provide quantitative estimates of duration and data depict the psychophysical function relating subjective duration to objective duration. With semi-objective methods, observers provide categorical or comparative judgments of duration and data depict the psychometric function relating the probability of a certain judgment to objective duration. Both approaches are used to study whether subjective and objective time run at the same pace or whether time flies or slows down under certain conditions. We analyze theoretical aspects affecting the interpretation of data gathered with the most widely used semi-objective methods, including single-presentation and paired-comparison methods. For this purpose, a formal model of psychophysical performance is used in which subjective duration is represented via a psychophysical function and the scalar property. This provides the timing component of the model, which is invariant across methods. A decisional component that varies across methods reflects how observers use subjective durations to make judgments and give the responses requested under each method. Application of the model shows that psychometric functions in single-presentation methods are uninterpretable because the various influences on observed performance are inextricably confounded in the data. In contrast, data gathered with paired-comparison methods permit separating out those influences. Prevalent approaches to fitting psychometric functions to data are also discussed and shown to be inconsistent with widely accepted principles of time perception, implicitly assuming instead that subjective time equals objective time and that observed differences across conditions do not reflect differences in perceived duration but criterion shifts. These analyses prompt evidence-based recommendations for best methodological practice in studies on time perception.

Joint Bayesian inference reveals model properties shared between multiple experimental conditions

Statistical modeling produces compressed and often more easily interpretable descriptions of experimental data in form of model parameters. When experimental manipulations target selected parameters, it is necessary for their interpretation that other model components remain constant. For example, psychophysicists use dose rate models to describe how behavior changes as a function of a single stimulus variable. The main interest is on shifts of this function induced by experimental manipulation, assuming invariance in other aspects of the function. Combining several experimental conditions in a joint analysis that takes such invariance constraints into account can result in a complex model for which no robust standard procedures are available. We formulate a solution for the joint analysis through repeated applications of standard procedures by allowing an additional assumption. This way, experimental conditions can be analyzed separately such that all conditions are implicitly taken into account. We investigate the validity of the supplementary assumption through simulations. Furthermore, we present a natural way to check whether a joint treatment is appropriate. We illustrate the method for the specific case of the psychometric function; however the procedure applies to other models that encompass multiple experimental conditions.

Statistical conclusion validity: some common threats and simple remedies

The ultimate goal of research is to produce dependable knowledge or to provide the evidence that may guide practical decisions. Statistical conclusion validity (SCV) holds when the conclusions of a research study are founded on an adequate analysis of the data, generally meaning that adequate statistical methods are used whose small-sample behavior is accurate, besides being logically capable of providing an answer to the research question. Compared to the three other traditional aspects of research validity (external validity, internal validity, and construct validity), interest in SCV has recently grown on evidence that inadequate data analyses are sometimes carried out which yield conclusions that a proper analysis of the data would not have supported. This paper discusses evidence of three common threats to SCV that arise from widespread recommendations or practices in data analysis, namely, the use of repeated testing and optional stopping without control of Type-I error rates, the recommendation to check the assumptions of statistical tests, and the use of regression whenever a bivariate relation or the equivalence between two variables is studied. For each of these threats, examples are presented and alternative practices that safeguard SCV are discussed. Educational and editorial changes that may improve the SCV of published research are also discussed.

A cautionary note on the use of the adaptive up-down method

Up-down staircases with equal sizes for the steps up and down are widely used to estimate detection and discrimination thresholds in psychoacoustics, but the conventional average-of-reversals estimator does not converge on its presumed percent point in Yes-No tasks or in two-alternative forced-choice detection tasks. The particular percent point of convergence is partly determined by the relative size of the steps with respect to the spread (inverse of slope) of the underlying psychometric function. In particular, threshold is increasingly underestimated as the spread of the psychometric function decreases. This characteristic may have serious consequences when thresholds estimated via up-down staircases are compared across conditions in which the spread of the psychometric function varies, because then these thresholds do not represent comparable measures of performance. This paper documents the misbehavior of the average-of-reversals estimator under up-down rules and types of forced-choice task that are in common use in psychoacoustics but which have not been studied before in simulations. It is also shown that a relatively simple modification of the up-down design (namely, using steps up and down of different size and in a certain ratio depending only on the task and the up-down rule being used) stabilizes the performance of these staircases.

Psychometric functions for detection and discrimination with and without flankers

Recent studies have reported that flanking stimuli broaden the psychometric function and lower detection thresholds. In the present study, we measured psychometric functions for detection and discrimination with and without flankers to investigate whether these effects occur throughout the contrast continuum. Our results confirm that lower detection thresholds with flankers are accompanied by broader psychometric functions. Psychometric functions for discrimination reveal that discrimination thresholds with and without flankers are similar across standard levels, and that the broadening of psychometric functions with flankers disappears as standard contrast increases, to the point that psychometric functions at high standard levels are virtually identical with or without flankers. Threshold-versus-contrast (TvC) curves with flankers only differ from TvC curves without flankers in occasional shallower dippers and lower branches on the left of the dipper, but they run virtually superimposed at high standard levels. We discuss differences between our results and other results in the literature, and how they are likely attributed to the differential vulnerability of alternative psychophysical procedures to the effects of presentation order. We show that different models of flanker facilitation can fit the data equally well, which stresses that succeeding at fitting a model does not validate it in any sense.

A clínica da psicofísica

Apresentamos a Psicofísica como uma ciência aplicada nas investigações e nas abordagens e diagnósticos clínicos. Inicialmente, introduzimos algo dos aspectos epistemológicos e teóricos da área, passamos para as abordagens que a Psicofísica pode apresentar na aplicabilidade clínica e, por fim, discutimos os avanços recentes da aplicação clínica, apresentamos as experiências de nosso laboratório de pesquisa clínica em psicofísica, finalizando com as perspectivas de ampliação do uso da psicofísica para investigações clínicas de funções perceptuais mais complexas.

Bayesian adaptive estimation of the contrast sensitivity function: the quick CSF method

The contrast sensitivity function (CSF) predicts functional vision better than acuity, but long testing times prevent its psychophysical assessment in clinical and practical applications. This study presents the quick CSF (qCSF) method, a Bayesian adaptive procedure that applies a strategy developed to estimate multiple parameters of the psychometric function (A. B. Cobo-Lewis, 1996; L. L. Kontsevich & C. W. Tyler, 1999). Before each trial, a one-step-ahead search finds the grating stimulus (defined by frequency and contrast) that maximizes the expected information gain (J. V. Kujala & T. J. Lukka, 2006; L. A. Lesmes et al., 2006), about four CSF parameters. By directly estimating CSF parameters, data collected at one spatial frequency improves sensitivity estimates across all frequencies. A psychophysical study validated that CSFs obtained with 100 qCSF trials ( approximately 10 min) exhibited good precision across spatial frequencies (SD < 2-3 dB) and excellent agreement with CSFs obtained independently (mean RMSE = 0.86 dB). To estimate the broad sensitivity metric provided by the area under the log CSF (AULCSF), only 25 trials were needed to achieve a coefficient of variation of 15-20%. The current study demonstrates the method's value for basic and clinical investigations. Further studies, applying the qCSF to measure wider ranges of normal and abnormal vision, will determine how its efficiency translates to clinical assessment.

Empirical performance of optimal Bayesian adaptive estimation

Simulation studies have shown how Bayesian adaptive estimation methods should be set up for optimal performance. We assessed the extent to which these results hold up for human observers, who are more subject to failure than simulation subjects. Discrimination and detection experiments with two-alternative forced-choice (2AFC) tasks were used for that purpose. Forty estimates of the point of subjective equality (PSE, or the 50% correct point on the psychometric function for discrimination) and 32 estimates of detection threshold (the 80% correct point on the psychometric function for detection) were taken for each of four observers with the optimal Bayesian method, while data for fitting the psychometric function psi were gathered concurrently with an adaptive method of constant stimuli governed by fixed-step-size staircases. The estimated parameters of the psychometric function served as a criterion for comparison. In the discrimination task, PSEs for each observer were distributed around the independently estimated 50% correct point on psi and their variability was occasionally minimally larger than simulation results indicated it should be. In the detection task, the distribution of threshold estimates was consistently above the independently estimated 80% correct point on psi and their variability was as expected from simulations. A close analysis of these results suggests that the optimal Bayesian method is affected by growing inattention or fatigue in detection tasks (factors that are not considered in simulations), and limits the practical applicability of Bayesian estimation of detection thresholds.