Potential performance theory (PPT): describing a methodology for analyzing task performance

Stability, change, and reliable individual differences in electroencephalography measures: a lifespan perspective on progress and opportunities

Electroencephalographic (EEG) methods have great potential to serve both basic and clinical science approaches to understand individual differences in human neural function. Importantly, the psychometric properties of EEG data, such as internal consistency and test-retest reliability, constrain their ability to differentiate individuals successfully. Rapid and recent technological and computational advancements in EEG research make it timely to revisit the topic of psychometric reliability in the context of individual difference analyses. Moreover, pediatric and clinical samples provide some of the most salient and urgent opportunities to apply individual difference approaches, but the changes these populations experience over time also provide unique challenges from a psychometric perspective. Here we take a developmental neuroscience perspective to consider progress and new opportunities for parsing the reliability and stability of individual differences in EEG measurements across the lifespan. We first conceptually map the different profiles of measurement reliability expected for different types of individual difference analyses over the lifespan. Next, we summarize and evaluate the state of the field's empirical knowledge and need for testing measurement reliability, both internal consistency and test-retest reliability, across EEG measures of power, event-related potentials, nonlinearity, and functional connectivity across ages. Finally, we highlight how standardized pre-processing software for EEG denoising and empirical metrics of individual data quality may be used to further improve EEG-based individual differences research moving forward. We also include recommendations and resources throughout that individual researchers can implement to improve the utility and reproducibility of individual differences analyses with EEG across the lifespan.

Some Implications of Distinguishing Between Unexplained Variance That Is Systematic or Random

Because error variance alternatively can be considered to be the sum of systematic variance associated with unknown variables and randomness, a tripartite assumption is proposed that total variance in the dependent variable can be partitioned into three variance components. These are variance in the dependent variable that is explained by the independent variable, variance in the dependent variable that is unexplained but systematic (associated with variance in unknown variables), and random variance. Based on the tripartite assumption, classical measurement theory, and simple mathematics, it is shown that these components can be estimated using observable data. Mathematical and computer simulations illustrate some of the important issues and implications.

Is consistency a domain-general individual differences characteristic?

We explored randomness in responding in two ways across six experiments. First, we predicted that people would differ from each other in randomness in a stable way when tested in the same domain across two sessions; people who responded more randomly in a particular domain in one session also should respond more randomly in a second session whereas people who responded less randomly in one session also should respond less randomly in a second session. Second, we predicted that there would be some domain general randomness; people's randomness in one domain should predict their randomness in another domain. We used consistency coefficients across blocks of a session as an inverse measure of randomness and found (a) consistency coefficients correlated across sessions within the same domain and (b) consistency coefficients in one domain correlated with consistency coefficients in other domains.

Attachment-related individual differences in the consistency of relationship behavior interpretation

The consistency with which people interpret relationship-based information has important implications for attachment theory and research. Our objective is to determine whether there are attachment-related individual differences in the manner and the consistency with which individuals interpret hypothetical relationship behaviors. In two studies (N = 629, 79% female, 63% American, M(age)  = 29; N = 820, 78% female, 65% American, M(age)  = 29), we assessed participants' ability and consistency in relationship behavior interpretation across two blocks and estimated how they would have performed had they interpreted information perfectly consistently. Secure participants were generally more consistent in their interpretations relative to insecure participants. Estimates of perfectly consistent interpretation revealed that improvements to both systematic factors related to behavior interpretation (e.g., working models) and consistency would have led to a more secure interpretation style for participants of all attachment styles. Results imply that both secure and insecure individuals process relationship-based information according to secure scripts, but insecure individuals do so inconsistently. Our results imply that, due to the inconsistent behavioral responses that may occur as a result of inconsistent information processing, the consistency with which people process relationship-related information will be related to relationship satisfaction. Further directions for future research are discussed.

Using potential performance theory to analyze systematic and random factors in enumeration tasks

Prior research has shown that as the number of items being enumerated increases, performance decreases, especially when the amount of time is limited. Researchers studying nonverbal enumeration have found that random noise increases as a function of the number of items presented. Over a series of 2 experiments, the authors used potential performance theory to expand these findings and discover precisely how much random noise actually influences observed performance and what performance might look like in the absence of random factors. Participants briefly viewed a visual stimulus comprising a set of 4 to 9 dots presented horizontally (Experiment 1) or randomly (Experiment 2) on a computer monitor. Findings from both experiments indicate that the decrease in performance for larger set sizes resulted almost entirely from a reduction in consistency (or an increase in random noise), whereas potential performance remained fairly constant until the maximum set size.

Time pressure heuristics can improve performance due to increased consistency

Our goal is to demonstrate that potential performance theory (PPT) provides a unique type of methodology for studying the use of heuristics under time pressure. While most theories tend to focus on different types of strategies, PPT distinguishes between random and nonrandom effects on performance. We argue that the use of a heuristic under time pressure actually can increase performance by decreasing randomness in responding. We conducted an experiment where participants performed a task under time pressure or not. In turn, PPT equations make it possible to parse the observed change in performance from the unspeeded to the speeded condition into that which is due to a change in the participant's randomness in responding versus that which is due to a change in systematic factors. We found that the change in randomness was slightly more important than the change in systematic factors.

Using PPT to account for randomness in perception

According to many theories of decision making, of which signal detection theory is the most prominent, randomness is the main factor responsible for imperfect performance. These theories imply that correcting for attenuation due to randomness should result in perfect scores as long as the participants use nonextreme decision criteria. On the basis of a recent advance termed potential performance theory (Trafimow & Rice, Psychological Review 115:447-462, 2008), we performed auditory and visual detection experiments and corrected the scores for attenuation. Most participants in both experiments tended to perform at a less-than-perfect level, even after their scores were corrected. The findings demonstrate that at least one systematic factor influences detection that is not included in signal detection theory.

The role of mechanisms, integration, and unification in science and psychology

I assume the importance of unifying theories in science but suggest that psychologists do not understand the concept. To clarify, I distinguish unification from both mechanisms and from integration. The psychology literature is replete with mechanisms and also integration, but there is insufficient unification. Finally, I use potential performance theory as an example of unification in a small way in psychology. Given that this theory provides small-scale unification, there is reason to hope for large-scale unification in the future—a goal to which psychologists can and should aspire.

The assumption of a reliable instrument and other pitfalls to avoid when considering the reliability of data

The purpose of this article is to help researchers avoid common pitfalls associated with reliability including incorrectly assuming that (a) measurement error always attenuates observed score correlations, (b) different sources of measurement error originate from the same source, and (c) reliability is a function of instrumentation. To accomplish our purpose, we first describe what reliability is and why researchers should care about it with focus on its impact on effect sizes. Second, we review how reliability is assessed with comment on the consequences of cumulative measurement error. Third, we consider how researchers can use reliability generalization as a prescriptive method when designing their research studies to form hypotheses about whether or not reliability estimates will be acceptable given their sample and testing conditions. Finally, we discuss options that researchers may consider when faced with analyzing unreliable data.

Using a sharp instrument to parse apart strategy and consistency: an evaluation of PPT and its assumptions

Potential Performance Theory (PPT) is a general theory for parsing observed performance into the underlying strategy and the consistency with which it is used. Although empirical research has supported that PPT is useful, it is desirable to have more information about the bias and standard errors of PPT findings. It also is beneficial to know the effects of violations of PPT assumptions. The authors present computer simulations that evaluate bias and standard errors at varying levels of strategy, consistency, and number of trials per participant. The simulations show that, when the assumptions are true, there is very little bias and the standard errors are low when there are moderate or large numbers of trials per participant (e.g., N=50 or N=100). But when the independence assumption is violated, PPT provides biased findings, although the bias is quite small unless the violations are large.