Converging evidence that common timing processes underlie temporal-order and simultaneity judgments: a model-based analysis

Temporal correspondence in perceptual organization: Reciprocal interactions between temporal sensitivity and figure-ground segregation

How do visual representations account for time? Is it the case that they represent time by themselves possessing temporal properties (temporal mirroring) or by atemporal markers/tags (temporal tagging)? This question has been asked for the past 5 decades and more, in neuroscience, philosophy, and psychology. To address this debate, we designed a study to test temporal correspondence. We tested whether a temporal property (flicker frequency) could influence figure-ground segregation, and in turn, reciprocally, whether a figure-ground segregation would alter a temporal property (here, temporal resolution). We manipulated flicker frequency of dots on either side of an ambiguous edge in Experiment 1 and asked participants to indicate the figural region. In Experiment 2, we measured temporal sensitivity using a temporal order judgment (TOJ) task in both figural and ground regions. We showed temporal correspondence by showing specifically that figure-ground segregation depends on flicker frequency differences between two regions in ambiguous displays, where slow-flickering regions are seen as figural (Experiment 1). Reciprocally, in Experiment 2, we showed that participants performed a temporal-order judgment task better when the task had to be performed on a region seen as background compared with the same region seen as a figure. We show how relatively slower flickering regions are seen as figural, and correspondingly, seeing a region as figural is associated with a poorer temporal resolution. Our results collectively allow us to demonstrate a tight temporal correspondence in figure-ground perception, which could be explained using the parvocellular and magnocellular pathways, the two major retino-geniculo-cortical pathways.

Audiovisual simultaneity windows reflect temporal sensory uncertainty

The ability to judge the temporal alignment of visual and auditory information is a prerequisite for multisensory integration and segregation. However, each temporal measurement is subject to error. Thus, when judging whether a visual and auditory stimulus were presented simultaneously, observers must rely on a subjective decision boundary to distinguish between measurement error and truly misaligned audiovisual signals. Here, we tested whether these decision boundaries are relaxed with increasing temporal sensory uncertainty, i.e., whether participants make the same type of adjustment an ideal observer would make. Participants judged the simultaneity of audiovisual stimulus pairs with varying temporal offset, while being immersed in different virtual environments. To obtain estimates of participants' temporal sensory uncertainty and simultaneity criteria in each environment, an independent-channels model was fitted to their simultaneity judgments. In two experiments, participants' simultaneity decision boundaries were predicted by their temporal uncertainty, which varied unsystematically with the environment. Hence, observers used a flexibly updated estimate of their own audiovisual temporal uncertainty to establish subjective criteria of simultaneity. This finding implies that, under typical circumstances, audiovisual simultaneity windows reflect an observer's cross-modal temporal uncertainty.

The Role of Awareness on Motor-Sensory Temporal Recalibration

Temporal recalibration (TR) may arise to realign asynchronous stimuli after exposure to a short, constant delay between voluntary movement and sensory stimulus. The objective of this study was to determine if awareness of the temporal lag between a motor response (i.e., a keypress) and a sensory event (i.e., a visual flash) is necessary for TR to occur. We further investigated whether manipulating the required motor and perceptual judgment tasks modified the influence of awareness on TR. Participants (n = 48) were randomly divided between two groups (Group 1: Aware and Group 2: Unaware). The Aware group was told of the temporal lag between their keypress and visual flash at the beginning of the experiment, whereas the Unaware group was not. All participants completed eight blocks of trials, in which the motor task (single or repetitive tap), perceptual judgment task (judging the temporal order of the keypress in relation to the visual flash or judging whether the two stimuli were simultaneous or not), and fixed temporal lag between keypress and visual flash (0 or 100 ms) varied. TR was determined by comparing judgments between corresponding blocks of trials in which the temporal lag was 0 or 100 ms. Results revealed that both the Aware and Unaware groups demonstrated a similar magnitude of TR across all motor and perceptual judgment tasks, such that the magnitude of TR did not vary between Aware and Unaware participants. These results suggest that awareness of a temporal lag does not influence the magnitude of TR achieved and that motor and perceptual judgment task demands do not modulate the influence of awareness on TR.

Visual field differences in temporal synchrony processing for audio-visual stimuli

Audio-visual integration relies on temporal synchrony between visual and auditory inputs. However, differences in traveling and transmitting speeds between visual and auditory stimuli exist; therefore, audio-visual synchrony perception exhibits flexible functions. The processing speed of visual stimuli affects the perception of audio-visual synchrony. The present study examined the effects of visual fields, in which visual stimuli are presented, for the processing of audio-visual temporal synchrony. The point of subjective simultaneity, the temporal binding window, and the rapid recalibration effect were measured using temporal order judgment, simultaneity judgment, and stream/bounce perception, because different mechanisms of temporal processing have been suggested among these three paradigms. The results indicate that auditory stimuli should be presented earlier for visual stimuli in the central visual field than in the peripheral visual field condition in order to perceive subjective simultaneity in the temporal order judgment task conducted in this study. Meanwhile, the subjective simultaneity bandwidth was broader in the central visual field than in the peripheral visual field during the simultaneity judgment task. In the stream/bounce perception task, neither the point of subjective simultaneity nor the temporal binding window differed between the two types of visual fields. Moreover, rapid recalibration occurred in both visual fields during the simultaneity judgment tasks. However, during the temporal order judgment task and stream/bounce perception, rapid recalibration occurred only in the central visual field. These results suggest that differences in visual processing speed based on the visual field modulate the temporal processing of audio-visual stimuli. Furthermore, these three tasks, temporal order judgment, simultaneity judgment, and stream/bounce perception, each have distinct functional characteristics for audio-visual synchrony perception. Future studies are necessary to confirm the effects of compensation regarding differences in the temporal resolution of the visual field in later cortical visual pathways on visual field differences in audio-visual temporal synchrony.

Body-environment integration: Temporal processing of tactile and auditory inputs along the schizophrenia continuum

According to the dimensional approach to psychosis, there is a continuum from low schizotypy to schizophrenia patients. The temporal aspect of sensory processing seems to be compromised across such continuum, as suggested by different studies separately investigating unisensory or multisensory domains. Most of these studies have so far focused primarily on the temporal processing of visual and auditory stimuli, either in schizotypy or schizophrenia, while leaving the tactile domain and the integration of touch with other senses mostly unexplored. Given the relevance of body-related perceptual abnormalities for psychosis proneness, we aimed at filling this gap in the literature across two studies. We asked participants with increasing levels of schizotypy (study 1) and schizophrenia patients (study 2) to perform three simultaneity judgement tasks: a unimodal tactile task, a unimodal auditory task and a bimodal audio-tactile task. Each task allowed estimating a simultaneity range (SR), as a proxy of the individual tolerance to asynchronies in the tactile, auditory and audio-tactile domains, respectively. Results showed larger SRs as the level of schizotypy increases. Specifically, the linear effect of schizotypy levels on the audio-tactile task was stronger than on the auditory task, which in turn was greater than the effect on the tactile task (study 1). Differently, schizophrenia patients showed larger SRs than controls in all the three tasks (study 2). The current study is the first empirical investigation across the continuum from low schizotypy to schizophrenia of the tolerance to asynchronies in the processing of external (auditory) and body-related (tactile) inputs.

The Do's and Don'ts of Psychophysical Methods for Interpretability of Psychometric Functions and Their Descriptors

Many areas of research require measuring psychometric functions or their descriptors (thresholds, slopes, etc.). Data for this purpose are collected with psychophysical methods of various types and justification for the interpretation of results arises from a model of performance grounded in signal detection theory. Decades of research have shown that psychophysical data display features that are incompatible with such framework, questioning the validity of interpretations obtained under it and revealing that psychophysical performance is more complex than this framework entertains. This paper describes the assumptions and formulation of the conventional framework for the two major classes of psychophysical methods (single- and dual-presentation methods) and presents various lines of empirical evidence that the framework is inconsistent with. An alternative framework is then described and shown to account for all the characteristics that the conventional framework regards as anomalies. This alternative process model explicitly separates the sensory, decisional, and response components of performance and represents them via parameters whose estimation characterizes the corresponding processes. Retrospective and prospective evidence of the validity of the alternative framework is also presented. A formal analysis also reveals that some psychophysical methods and response formats are unsuitable for separation of the three components of observed performance. Recommendations are thus given regarding practices that should be avoided and those that should be followed to ensure interpretability of the psychometric function, or descriptors (detection threshold, difference limen, point of subjective equality, etc.) obtained with shortcut methods that do not require estimation of psychometric functions.

Nonparametric tests for equality of psychometric functions

Many empirical studies measure psychometric functions (curves describing how observers' performance varies with stimulus magnitude) because these functions capture the effects of experimental conditions. To assess these effects, parametric curves are often fitted to the data and comparisons are carried out by testing for equality of mean parameter estimates across conditions. This approach is parametric and, thus, vulnerable to violations of the implied assumptions. Furthermore, testing for equality of means of parameters may be misleading: Psychometric functions may vary meaningfully across conditions on an observer-by-observer basis with no effect on the mean values of the estimated parameters. Alternative approaches to assess equality of psychometric functions per se are thus needed. This paper compares three nonparametric tests that are applicable in all situations of interest: The existing generalized Mantel-Haenszel test, a generalization of the Berry-Mielke test that was developed here, and a split variant of the generalized Mantel-Haenszel test also developed here. Their statistical properties (accuracy and power) are studied via simulation and the results show that all tests are indistinguishable as to accuracy but they differ non-uniformly as to power. Empirical use of the tests is illustrated via analyses of published data sets and practical recommendations are given. The computer code in MATLAB and R to conduct these tests is available as Electronic Supplemental Material.

I before U: Temporal order judgements reveal bias for self-owned objects

A multitude of studies demonstrate that self-relevant stimuli influence attention. Self-owned objects are a special class of self-relevant stimuli. If a self-owned object can indeed be characterised as a self-relevant stimulus then, consistent with theoretical predictions, a behavioural effect of ownership on attention should be present. To test this prediction, a task was selected that is known to be particularly sensitive measure of the prioritisation of visual information: the temporal order judgement. Participants completed temporal order judgements with pictures of "own" and "experimenter" owned objects (mugs) presented on either side of a central fixation cross. There was a variable onset delay between each picture, ranging between 0 ms and 105 ms, and participants were asked to indicate which mug appeared first. The results indicated a reliable change in the point of subjective simultaneity (PSS) in favour of their own mug. Such a change in the PSS was not observed for two groups of participants who were exposed to a mug but did not keep the mug. A further experiment indicated that the source of the bias in PSS was more consistent with a criterion shift or top-down attentional prioritisation rather than a perceptual bias. These findings suggest that ownership, beyond mere-touch, mere-choice, or familiarity, leads to prioritised processing and responses, but the mechanism underlying the effect is not likely to be perceptual in nature.

When uncertain, does human self-motion decision-making fully utilize complete information?

When forced to choose humans often feel uncertain. Investigations of human perceptual decision-making often employ signal detection theory, which assumes that even when uncertain all available information is fully utilized. However, other studies have suggested or assumed that, when uncertain, human subjects guess totally at random, ignoring available information. When uncertain, do humans simply guess totally at random? Or do humans fully utilize complete information? Or does behavior fall between these two extremes yielding "above chance" performance without fully utilizing complete information? While it is often assumed complete information is fully utilized, even when uncertain, to our knowledge this has never been experimentally confirmed. To answer this question, we combined numerical simulations, theoretical analyses, and human studies performed using a self-motion direction-recognition perceptual decision-making task (did I rotate left or right?). Subjects were instructed to make forced-choice binary (left/right) and trinary (left/right/uncertain) decisions when cued following each stimulus. Our results show that humans 1) do not guess at random when uncertain and 2) make binary and trinary decisions equally well. These findings show that humans fully utilize complete information when uncertain for our perceptual decision-making task. This helps unify signal detection theory and other models of forced-choice decision-making which allow for uncertain responses. NEW & NOTEWORTHY Humans make many perceptual decisions every day. But what if we are uncertain? While many studies assume that humans fully utilize complete information, other studies have suggested and/or assumed that when we're uncertain and forced to decide, information is not fully utilized. While humans tend to perform above chance when uncertain, no earlier study has tested whether available information is fully utilized. Our results show that humans make fully informed decisions even when uncertain.

Learned value and object perception: Accelerated perception or biased decisions?

Learned value is known to bias visual search toward valued stimuli. However, some uncertainty exists regarding the stage of visual processing that is modulated by learned value. Here, we directly tested the effect of learned value on preattentive processing using temporal order judgments. Across four experiments, we imbued some stimuli with high value and some with low value, using a nonmonetary reward task. In Experiment 1, we replicated the value-driven distraction effect, validating our nonmonetary reward task. Experiment 2 showed that high-value stimuli, but not low-value stimuli, exhibit a prior-entry effect. Experiment 3, which reversed the temporal order judgment task (i.e., reporting which stimulus came second), showed no prior-entry effect, indicating that although a response bias may be present for high-value stimuli, they are still reported as appearing earlier. However, Experiment 4, using a simultaneity judgment task, showed no shift in temporal perception. Overall, our results support the conclusion that learned value biases perceptual decisions about valued stimuli without speeding preattentive stimulus processing.

Thou Shalt Not Bear False Witness Against Null Hypothesis Significance Testing

Null hypothesis significance testing (NHST) has been the subject of debate for decades and alternative approaches to data analysis have been proposed. This article addresses this debate from the perspective of scientific inquiry and inference. Inference is an inverse problem and application of statistical methods cannot reveal whether effects exist or whether they are empirically meaningful. Hence, raising conclusions from the outcomes of statistical analyses is subject to limitations. NHST has been criticized for its misuse and the misconstruction of its outcomes, also stressing its inability to meet expectations that it was never designed to fulfil. Ironically, alternatives to NHST are identical in these respects, something that has been overlooked in their presentation. Three of those alternatives are discussed here (estimation via confidence intervals and effect sizes, quantification of evidence via Bayes factors, and mere reporting of descriptive statistics). None of them offers a solution to the problems that NHST is purported to have, all of them are susceptible to misuse and misinterpretation, and some bring around their own problems (e.g., Bayes factors have a one-to-one correspondence with p values, but they are entirely deprived of an inferential framework). Those alternatives also fail to cover a broad area of inference not involving distributional parameters, where NHST procedures remain the only (and suitable) option. Like knives or axes, NHST is not inherently evil; only misuse and misinterpretation of its outcomes needs to be eradicated.

A Matched Comparison Across Three Different Sensory Pairs of Cross-Modal Temporal Recalibration From Sustained and Transient Adaptation

Sustained exposure to an asynchronous multisensory signal causes perceived simultaneity to shift in the direction of the leading component of the adapting stimulus. This is known as temporal recalibration, and recent evidence suggests that it can occur very rapidly, even after a single asynchronous audiovisual (AV) stimulus. However, this form of rapid recalibration appears to be unique to AV stimuli, in contrast to recalibration following sustained asynchronies which occurs with audiotactile (AT) and visuotactile (VT) stimuli. This study examines temporal recalibration to AV, VT and AT asynchrony with spatially collocated stimuli using a design that produces both sustained and inter-trial recalibration by combining the traditional sustained adaptation approach with an inter-trial analysis of sequential dependencies in an extended test period. Thus, we compare temporal recalibration to both sustained and transient asynchrony in three crossmodal combinations using the same design, stimuli and observers. The results reveal that prolonged exposure to asynchrony produced equivalent temporal recalibration for all combinations: AV, AT and VT. The pattern for rapid, inter-trial recalibration was very different. Rapid recalibration occurred strongly for AV stimuli, weakly for AT and did not occur at all for VT. For all sensory pairings, recalibration from sustained asynchrony decayed to baseline during the test phase while inter-trial recalibration was present and stable throughout testing, suggesting different mechanisms may underlie adaptation at long and short timescales.

Estimation and Identifiability of Model Parameters in Human Nociceptive Processing Using Yes-No Detection Responses to Electrocutaneous Stimulation

Healthy or pathological states of nociceptive subsystems determine different stimulus-response relations measured from quantitative sensory testing. In turn, stimulus-response measurements may be used to assess these states. In a recently developed computational model, six model parameters characterize activation of nerve endings and spinal neurons. However, both model nonlinearity and limited information in yes-no detection responses to electrocutaneous stimuli challenge to estimate model parameters. Here, we address the question whether and how one can overcome these difficulties for reliable parameter estimation. First, we fit the computational model to experimental stimulus-response pairs by maximizing the likelihood. To evaluate the balance between model fit and complexity, i.e., the number of model parameters, we evaluate the Bayesian Information Criterion. We find that the computational model is better than a conventional logistic model regarding the balance. Second, our theoretical analysis suggests to vary the pulse width among applied stimuli as a necessary condition to prevent structural non-identifiability. In addition, the numerically implemented profile likelihood approach reveals structural and practical non-identifiability. Our model-based approach with integration of psychophysical measurements can be useful for a reliable assessment of states of the nociceptive system.

A single mechanism account of duration and rate processing via the pacemaker-accumulator and beat frequency models

Time is an essential dimension of our environment that allows us to extract meaningful information about speed of movement, speech, motor actions and fine motor control. Traditionally, models of time have tried to quantify how the brain might process the duration of an event. The most commonly cited are the pacemaker-accumulator model and the beat frequency model of interval timing, which explain how duration is perceived, represented and encoded. Here we posit such models as providing a powerful tool for simultaneously extracting, representing and encoding stimulus rate information. That is, any model that can process duration has all the information needed to code stimulus rate. We explore different processing strategies which would enable rate to be read off from both the pacemaker-accumulator and beat frequency model of interval timing. Finally we explore open questions that, when answered, will shed light upon potential mechanisms for duration and rate estimation.

A Roving Dual-Presentation Simultaneity-Judgment Task to Estimate the Point of Subjective Simultaneity

The most popular tasks with which to investigate the perception of subjective synchrony are the temporal order judgment (TOJ) and the simultaneity judgment (SJ). Here, we discuss a complementary approach—a dual-presentation (2x) SJ task—and focus on appropriate analysis methods for a theoretically desirable “roving” design. Two stimulus pairs are presented on each trial and the observer must select the most synchronous. To demonstrate this approach, in Experiment 1 we tested the 2xSJ task alongside TOJ, SJ, and simple reaction-time (RT) tasks using audiovisual stimuli. We interpret responses from each task using detection-theoretic models, which assume variable arrival times for sensory signals at critical brain structures for timing perception. All tasks provide similar estimates of the point of subjective simultaneity (PSS) on average, and PSS estimates from some tasks were correlated on an individual basis. The 2xSJ task produced lower and more stable estimates of model-based (and thus comparable) sensory/decision noise than the TOJ. In Experiment 2 we obtained similar results using RT, TOJ, ternary, and 2xSJ tasks for all combinations of auditory, visual, and tactile stimuli. In Experiment 3 we investigated attentional prior entry, using both TOJs and 2xSJs. We found that estimates of prior-entry magnitude correlated across these tasks. Overall, our study establishes the practicality of the roving dual-presentation SJ task, but also illustrates the additional complexity of the procedure. We consider ways in which this task might complement more traditional procedures, particularly when it is important to estimate both PSS and sensory/decisional noise.

Simultaneity and Temporal Order Judgments Exhibit Distinct Reaction Times and Training Effects

A considerable body of sensory research has addressed the rules governing simultaneity judgments (SJs) and temporal order judgments (TOJs). In principle, neural events that register stimulus-arrival-time differences at an early sensory stage could set the limit on SJs and TOJs alike. Alternatively, distinct limits on SJs and TOJs could arise from task-specific neural events occurring after the stimulus-driven stage. To distinguish between these possibilities, we developed a novel reaction-time (RT) measure and tested it in a perceptual-learning procedure. The stimuli comprised dual-stream Rapid Serial Visual Presentation (RSVP) displays. Participants judged either the simultaneity or temporal order of red-letter and black-number targets presented in opposite lateral hemifield streams of black-letter distractors. Despite identical visual stimulation across-tasks, the SJ and TOJ tasks generated distinct RT patterns. SJs exhibited significantly faster RTs to synchronized targets than to subtly asynchronized targets; TOJs exhibited the opposite RT pattern. These task-specific RT patterns cannot be attributed to the early, stimulus-driven stage and instead match what one would predict if the limits on SJs and TOJs arose from task-specific decision spaces. That is, synchronized targets generate strong evidence for simultaneity, which hastens SJ RTs. By contrast, synchronized targets provide no information about temporal order, which slows TOJ RTs. Subtly asynchronizing the targets reverses this information pattern, and the corresponding RT patterns. In addition to investigating RT patterns, we also investigated training-transfer between the tasks. Training to improve SJ precision failed to improve TOJ precision, and vice versa, despite identical visual stimulation across tasks. This, too, argues against early, stimulus-driven limits on SJs and TOJs. Taken together, the present study offers novel evidence that distinct rules set the limits on SJs and TOJs.

Visual and Auditory Components in the Perception of Asynchronous Audiovisual Speech

Research on asynchronous audiovisual speech perception manipulates experimental conditions to observe their effects on synchrony judgments. Probabilistic models establish a link between the sensory and decisional processes underlying such judgments and the observed data, via interpretable parameters that allow testing hypotheses and making inferences about how experimental manipulations affect such processes. Two models of this type have recently been proposed, one based on independent channels and the other using a Bayesian approach. Both models are fitted here to a common data set, with a subsequent analysis of the interpretation they provide about how experimental manipulations affected the processes underlying perceived synchrony. The data consist of synchrony judgments as a function of audiovisual offset in a speech stimulus, under four within-subjects manipulations of the quality of the visual component. The Bayesian model could not accommodate asymmetric data, was rejected by goodness-of-fit statistics for 8/16 observers, and was found to be nonidentifiable, which renders uninterpretable parameter estimates. The independent-channels model captured asymmetric data, was rejected for only 1/16 observers, and identified how sensory and decisional processes mediating asynchronous audiovisual speech perception are affected by manipulations that only alter the quality of the visual component of the speech signal.