Sometimes More Is Better, and Sometimes Less Is Better: Task Complexity Moderates the Response Time Accuracy Correlation

Should Intelligence Tests Be Speeded or Unspeeded? A Brief Review of the Effects of Time Pressure on Response Processes and an Experimental Study with Raven's Matrices

Intelligence tests are often performed under time constraints for practical reasons, but the effects of time pressure on reasoning performance are poorly understood. The first part of this work provides a brief review of major expected effects of time pressure, which includes forcing participants to skip items, convoking a mental speed factor, constraining response times, qualitatively altering cognitive processing, affecting anxiety and motivation, and interacting with individual differences. The second part presents data collected with Raven's matrices under three conditions of speededness to provide further insight into the complex effects of time pressure, with three major findings. First, even mild time pressure (with enough time available for all participants to complete the task at a leisurely pace) induced speeding throughout the whole task, starting with the very first item, and participants sped up more than was actually required. Second, time pressure came with lower confidence and poorer strategy use and a substantial decrease of accuracy (d = 0.35), even when controlling for response time at the item level-indicating a detrimental effect on cognitive processing beyond speeding. Third, time pressure disproportionately reduced response times for difficult items and participants with high ability, working memory capacity, or need for cognition, although this did not differentially affect ability estimates. Overall, both the review and empirical sections show that the effects of time pressure go well beyond forcing participants to speed or skip the last few items and make even mild time constraints inadvisable when attempting to measure maximal performance, especially for high-performing samples.

Testing Replicability and Generalizability of the Time on Task Effect

The time on task (ToT) effect describes the relationship of the time spent on a cognitive task and the probability of successful task completion. The effect has been shown to vary in size and direction across tests and even within tests, depending on the test taker and item characteristics. Specifically, investing more time has a positive effect on response accuracy for difficult items and low ability test-takers, but a negative effect for easy items and high ability test-takers. The present study sought to test the replicability of this result pattern of the ToT effect across samples independently drawn from the same populations of persons and items. Furthermore, its generalizability was tested in terms of differential correlations across ability tests. To this end, ToT effects were estimated for three different reasoning tests and one test measuring natural sciences knowledge in 10 comparable subsamples with a total N = 2640. Results for the subsamples were highly similar, demonstrating that ToT effects are estimated with sufficient reliability. Generally, faster answers tended to be more accurate, suggesting a relatively effortless processing style. However, with increasing item difficulty and decreasing person ability, the effect flipped to the opposite direction, i.e., higher accuracy with longer processing times. The within-task moderation of the ToT effect can be reconciled with an account on effortful processing or cognitive load. By contrast, the generalizability of the ToT effect across different tests was only moderate. Cross-test relations were stronger in relative terms if performance in the respective tasks was more strongly related. This suggests that individual differences in the ToT effect depend on test characteristics such as their reliabilities but also similarities and differences of their processing requirements.

Task Switching: On the Relation of Cognitive Flexibility with Cognitive Capacity

The task-switching paradigm is deemed a measure of cognitive flexibility. Previous research has demonstrated that individual differences in task-switch costs are moderately inversely related to cognitive ability. However, current theories emphasize multiple component processes of task switching, such as task-set preparation and task-set inertia. The relations of task-switching processes with cognitive ability were investigated in the current study. Participants completed a task-switching paradigm with geometric forms and a visuospatial working memory capacity (WMC) task. The task-switch effect was decomposed with the diffusion model. Effects of task-switching and response congruency were estimated as latent differences using structural equation modeling. Their magnitudes and relations with visuospatial WMC were investigated. Effects in the means of parameter estimates replicated previous findings, namely increased non-decision time in task-switch trials. Further, task switches and response incongruency had independent effects on drift rates, reflecting their differential effects on task readiness. Findings obtained with the figural tasks employed in this study revealed that WMC was inversely related to the task-switch effect in non-decision time. Relations with drift rates were inconsistent. Finally, WMC was moderately inversely related to response caution. These findings suggest that more able participants either needed less time for task-set preparation or that they invested less time for task-set preparation.

You Can Play the Game Without Knowing the Rules – But You’re Better Off Knowing Them

Abstract. Due to their high item difficulties and excellent psychometric properties, construction-based figural matrices tasks are of particular interest when it comes to high-stakes testing. An important prerequisite is that test preparation – which is likely to occur in this context – does not impair test fairness or item properties. The goal of this study was to provide initial evidence concerning the influence of test preparation. We administered test items to a sample of N = 882 participants divided into two groups, but only one group was given information about the rules employed in the test items. The probability of solving the items was significantly higher in the test preparation group than in the control group ( M = 0.61, SD = 0.19 vs. M = 0.41, SD = 0.25; t(54) = 3.42, p = .001; d = .92). Nevertheless, a multigroup confirmatory factor analysis, as well as a differential item functioning analysis, indicated no differences between the item properties in the two groups. The results suggest that construction-based figural matrices are suitable in the context of high-stakes testing when all participants are provided with test preparation material so that test fairness is ensured.

Explaining the high working memory capacity of gifted children: Contributions of processing skills and executive control

Intellectually gifted children tend to demonstrate especially high working memory capacity, an ability that holds a critical role in intellectual functioning. What could explain the differences in working memory performance between intellectually gifted and nongifted children? We investigated this issue by measuring working memory capacity with complex spans in a sample of 55 gifted and 55 nongifted children. Based on prior studies, we expected the higher working memory capacity of intellectually gifted children to be driven by more effective executive control, as measured using the Attention Network Test. The findings confirmed that intellectually gifted children had higher working memory capacity than typical children, as well as more effective executive attention. Surprisingly, however, working memory differences between groups were not mediated by differences in executive attention. Instead, it appears that gifted children resolve problems faster in the processing phase of the working memory task, which leaves them more time to refresh to-be-remembered items. This faster problem solving speed mediated their advantage in working memory capacity. Importantly, this effect was specific to speed on complex problems: low-level processing speed, as measured with the Attention Network Test, did not contribute to the higher working memory capacity of gifted children.

Reflections and New Perspectives on Face Cognition as a Specific Socio-Cognitive Ability

The study of socio-cognitive abilities emerged from intelligence research, and their specificity remains controversial until today. In recent years, the psychometric structure of face cognition (FC)-a basic facet of socio-cognitive abilities-was extensively studied. In this review, we summarize and discuss the divergent psychometric structures of FC in easy and difficult tasks. While accuracy in difficult tasks was consistently shown to be face-specific, the evidence for easy tasks was inconsistent. The structure of response speed in easy tasks was mostly-but not always-unitary across object categories, including faces. Here, we compare studies to identify characteristics leading to face specificity in easy tasks. The following pattern emerges: in easy tasks, face specificity is found when modeling speed in a single task; however, when modeling speed across multiple, different easy tasks, only a unitary factor structure is reported. In difficult tasks, however, face specificity occurs in both single task approaches and task batteries. This suggests different cognitive mechanisms behind face specificity in easy and difficult tasks. In easy tasks, face specificity relies on isolated cognitive sub-processes such as face identity recognition. In difficult tasks, face-specific and task-independent cognitive processes are employed. We propose a descriptive model and argue for FC to be integrated into common taxonomies of intelligence.

Validating Test Score Interpretations Using Time Information

A validity approach is proposed that uses processing times to collect validity evidence for the construct interpretation of test scores. The rationale of the approach is based on current research of processing times and on classical validity approaches, providing validity evidence based on relationships with other variables. Within the new approach, convergent validity evidence is obtained if a component skill, that is expected to underlie the task solution process in the target construct, positively moderates the relationship between effective speed and effective ability in the corresponding target construct. Discriminant validity evidence is provided if a component skill, that is not expected to underlie the task solution process in the target construct, does indeed not moderate the speed-ability relation in this target construct. Using data from a study that follows up the German PIAAC sample, this approach was applied to reading competence, assessed with PIAAC literacy items, and to quantitative reasoning, assessed with Number Series. As expected from theory, the effect of speed on ability in the target construct was only moderated by the respective underlying component skill, that is, word meaning activation skill as an underlying component skill of reading competence, and perceptual speed as an underlying component skill of reasoning. Accordingly, no positive interactions were found for the component skill that should not underlie the task solution process, that is, word meaning activation for reasoning and perceptual speed for reading. Furthermore, the study shows the suitability of the proposed validation approach. The use of time information in association with task results brings construct validation closer to the actual response process than widely used correlations of test scores.

Response Time Reduction Due to Retesting in Mental Speed Tests: A Meta-Analysis

As retest effects in cognitive ability tests have been investigated by various primary and meta-analytic studies, most studies from this area focus on score gains as a result of retesting. To the best of our knowledge, no meta-analytic study has been reported that provides sizable estimates of response time (RT) reductions due to retesting. This multilevel meta-analysis focuses on mental speed tasks, for which outcome measures often consist of RTs. The size of RT reduction due to retesting in mental speed tasks for up to four test administrations was analyzed based on 36 studies including 49 samples and 212 outcomes for a total sample size of 21,810. Significant RT reductions were found, which increased with the number of test administrations, without reaching a plateau. Larger RT reductions were observed in more complex mental speed tasks compared to simple ones, whereas age and test-retest interval mostly did not moderate the size of the effect. Although a high heterogeneity of effects exists, retest effects were shown to occur for mental speed tasks regarding RT outcomes and should thus be more thoroughly accounted for in applied and research settings.

Is the Correlation between Storage Capacity and Matrix Reasoning Driven by the Storage of Partial Solutions? A Pilot Study of an Experimental Approach

Working memory capacity (WMC) and reasoning abilities-as assessed by figural matrices tests-are substantially correlated. It is controversially discussed whether this correlation is only caused by controlled attention or also by storage capacity. This study aims at investigating storage of partial solutions as a possible mechanism by which storage capacity may contribute to solving figural matrices tests. For this purpose, we analyzed how an experimental manipulation of storage demands changes the pattern of correlations between WMC and performance in a matrix task. We manipulated the storage demands by applying two test formats: one providing the externalization of partial solutions and one without the possibility of externalization. Storage capacity was assessed by different types of change detection tasks. We found substantial correlations between storage capacity and matrices test performance, but they were of comparable size for both test formats. We take this as evidence that the necessity to store partial solutions is not the limiting factor which causes the association between storage capacity and matrices test. It is discussed how this approach can be used to investigate alternative mechanisms by that storage may influence performance in matrices tests.