The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences

A comprehensive evaluation of the neurocognitive predictors of problematic alcohol use, eating, pornography, and internet use: A 6-month longitudinal study

Background and aims

Cognitive control and reward-related abnormalities are centrally implicated in addiction. However, findings from longitudinal studies addressing neurocognitive predictors of addictive behaviors are mixed. Further, little work has been conducted predicting non-substance-related addictive behaviors. Our study aimed to assess predictors of substance and non-substance addictive behaviors in a community sample, systematically evaluating each neurocognitive function's independent influence on addictive behavior.

Methods

Australians (N = 294; 51.7% female; M[SD] age = 24.8[4.7] years) completed online neurocognitive tasks and surveys at baseline and 3-month follow-up. Self-report scales assessed problematic alcohol use, addictive eating (AE), problematic pornography use (PPU), and problematic internet use (PUI) at 3- and 6-month follow-ups. Linear regressions with bootstrapping assessed neurocognitive predictors for each addictive behavior across a 6-month period.

Results

Neurocognition at baseline did not predict AE or PUI severity at 6-month follow-up. Less delay discounting at baseline predicted higher PPU at 6-month follow-up (β = -0.16, p = 0.005). Poorer performance monitoring at baseline predicted higher AE at 3-month follow-up (β = -0.16, p = 0.004), and more reward-related attentional capture at 3-months predicted higher AE at 6-month follow-up (β = 0.14, p = 0.033). Less reward-related attentional capture (β = -0.14, p = 0.003) and less risk-taking under ambiguity (β = -0.11, p = 0.029) at baseline predicted higher PUI at 3-month follow-up. All findings were of small effect size. None of the neurocognitive variables predicted problematic alcohol use.

Discussion and conclusions

We were unable to identify a core set of specific neurocognitive functions that reliably predict multiple addictive behavior types. However, our findings indicate both cognitive control and reward-related functions predict non-substance addictive behaviors in different ways. Findings suggest that there may be partially distinct neurocognitive mechanisms contributing to addiction depending on the specific addictive behavior.

Executive functions involved in thought suppression: An attempt to integrate research in two paradigms

There are two main thought suppression research paradigms: the White Bear and Think/No-Think paradigms. In Think/No-Think research, thought suppression is effective and is considered to be mediated by prepotent response inhibition. Conversely, in White Bear studies, thought suppression is counterproductive and appears to engage resistance to proactive interference. However, findings regarding the involvement of these executive functions in each task are mixed. In the current study, two thought suppression procedures were compared. Using Friedman and Miyake's inhibitory functions model (2004) it was investigated whether the differences between thought suppression tasks can be explained by involvement of different executive functions. The results showed that the suppression phases of both procedures were correlated, but the outcomes of suppression were unrelated. There was no evidence supporting the involvement of the examined executive functions in either thought suppression task. Commonalities and discrepancies of the two tasks are discussed along with their external validity.

Estimating the Reliability and Stability of Cognitive Processes Contributing to Responses on the Implicit Association Test

Implicit measures were initially assumed to assess stable individual differences, but other perspectives posit that they reflect context-dependent processes. This pre-registered research investigates whether the processes contributing to responses on the race Implicit Association Test are temporally stable and reliably measured using multinomial processing tree modeling. We applied two models-the Quad model and the Process Dissociation Procedure-to six datasets (N = 2,036), each collected over two occasions, examined the within-measurement reliability and between-measurement stability of model parameters, and meta-analyzed the results. Parameters reflecting accuracy-oriented processes demonstrate adequate stability and reliability, which suggests these processes are relatively stable within individuals. Parameters reflecting evaluative associations demonstrate poor stability but modest reliability, which suggests that associations are either context-dependent or stable but noisily measured. These findings suggest that processes contributing to racial bias on implicit measures differ in temporal stability, which has practical implications for predicting behavior using the Implicit Association Test.

Comparing self-other distinction across motor, cognitive and affective domains

The self-other distinction (SOD) is a process by which humans disentangle self from other-related mental representations. This online study investigated two unresolved questions: (i) whether partially the same processes underpin SOD for motor, cognitive and affective representations, and (ii) whether SOD overlaps with domain-general cognitive control processes. Participants (N = 243) performed three SOD tasks (motor: automatic imitation inhibition (AIT); cognitive: visual perspective-taking (VPT); affective: emotional egocentricity bias (av-EEB) tasks) and two cognitive control tasks (Stroop and stop-signal reaction time (SSRT) tasks). Correlation analyses showed no associations among the motor, cognitive and affective SOD indexes. Similarly, distinct SOD clusters emerged in the hierarchical clustering dendrogram, indicating clear separations among SODs. However, the results of multidimensional scaling suggested a tendency towards two clusters, as evidenced by the proximity of AIT and VPT indexes in relation to EEB indexes. AIT spatial laterality and Stroop domain-general cognitive control confounded AIT and VPT indexes, albeit slightly differently depending on the analysis method used. SSRT showed neither associations with SODs nor with other domain-general indexes. These findings underscore the complexity of SOD processes and have notable implications for basic and applied research, e.g. in the domain of clinical disorders affected by deficiencies in SOD.

Impaired Reactive Control But Preserved Proactive Control in Hyperactive Children

OBJECTIVE

To examine the manifestation of cognitive control deficit of children with different levels of hyperactivity, an "at risk" dimension for ADHD.

METHOD

A group of children with high hyperactivity (N = 40) and another group of children with low levels of hyperactivity (N = 38) performed a modified stop-signal anticipation task, a revised Go/NoGo task, and the AX-continuous performance test (AX-CPT).

RESULTS

Children with higher levels of hyperactivity displayed: (1) significantly prolonged stop signal reaction time (SSRT) in the modified stop-signal anticipation task; (2) no notable differences in commission errors in the revised Go/NoGo task; (3) increased reaction time (RT) in stop-signal task and Go/NoGo task with increased probabilities of stop or NoGo signal; and (4) positive proactive behavioral index scores in AX-CPT.

CONCLUSION

The results suggested that children with heightened hyperactivity exhibited impaired reactive control, especially for responses already underway, but preserved proactive control. Further studies concerning these children are warranted.

Openness to experience predicts eye movement behavior during scene viewing

Individuals' abilities to perform goal-directed spatial deployments of attention are distinguishable from their broader preferences for how they use spatial attention when circumstances do not compel a specific deployment strategy. Although these preferences are likely to play a major role in how we interact with the visual world during daily life, they remain relatively understudied. This exploratory study investigated two key questions about these preferences: firstly, are individuals consistent in their preferences for how they deploy their spatial attention when making shifts of attention versus adopting an attentional breadth? Secondly, which other factors are associated with these preferences? Across two experiments, we measured how participants preferred to deploy both attentional breadth (using an adapted Navon task) and eye movements (using a free-viewing task). We also measured participants' working memory capacities (Experiment 1), and their personalities and world beliefs (Experiment 2). In both experiments, there were consistent individual differences in preference for attentional breadth and eye movement characteristics, but these two kinds of preference were unrelated to each other. Working memory capacity was not linked to these preferences. Conversely, the personality trait of Openness to Experience robustly predicted two aspects of eye movement behavior preference, such that higher levels of Openness predicted smaller saccades and shorter scan paths. This suggests that personality dimensions may predict preferences for more absorbed engagement with visual information. However, it appears that individuals' preferences for shifts of attention during scene viewing do not necessarily relate to the breadth of attention they choose to adopt.

The Ambiguous Cue Task: Measurement reliability of an experimental paradigm for the assessment of interpretation bias and associations with mental health

Interpretation biases in the processing of ambiguous affective information are assumed to play an important role in the onset and maintenance of emotional disorders. Reports of low reliability for experimental measures of cognitive biases have called into question previous findings on the association of these measures with markers of mental health and demonstrated the need to systematically evaluate measurement reliability for measures of cognitive biases. We evaluated reliability and correlations with self-report measures of mental health for interpretation bias scores derived from the Ambiguous Cue Task (ACT), an experimental paradigm for the assessment of approach-avoidance behavior towards ambiguous affective stimuli. For a non-clinical sample, the measurement of an interpretation bias with the ACT showed high internal consistency (rSB = .91 - .96, N = 354) and acceptable 2-week test-retest correlations (rPearson = .61 - .65, n = 109). Correlations between the ACT interpretation bias scores and mental health-related self-report measures of personality and well-being were generally small (r ≤ |.11|) and statistically not significant when correcting for multiple comparisons. These findings suggest that in non-clinical populations, individual differences in the interpretation of ambiguous affective information as assessed with the ACT do not show a clear association with self-report markers of mental health. However, in allowing for a highly reliable measurement of interpretation bias, the ACT provides a valuable tool for studies considering potentially small effect sizes in non-clinical populations by studying bigger samples as well as for work on clinical populations, for which potentially greater effects can be expected.

Consistency within change: Evaluating the psychometric properties of a widely used predictive-inference task

Rapid adaptation to sudden changes in the environment is a hallmark of flexible human behaviour. Many computational, neuroimaging, and even clinical investigations studying this cognitive process have relied on a behavioural paradigm known as the predictive-inference task. However, the psychometric quality of this task has never been examined, leaving unanswered whether it is indeed suited to capture behavioural variation on a within- and between-subject level. Using a large-scale test-retest design (T1: N = 330; T2: N = 219), we assessed the internal (internal consistency) and temporal (test-retest reliability) stability of the task's most used measures. We show that the main measures capturing flexible belief and behavioural adaptation yield good internal consistency and overall satisfying test-retest reliability. However, some more complex markers of flexible behaviour show lower psychometric quality. Our findings have implications for the large corpus of previous studies using this task and provide clear guidance as to which measures should and should not be used in future studies.

Functional brain connectivity changes associated with day-to-day fluctuations in affective states

Affective neuroscience has traditionally relied on cross-sectional studies to uncover the brain correlates of affects, emotions, and moods. Such findings obfuscate intraindividual variability that may reveal meaningful changing affect states. The few functional magnetic resonance imaging longitudinal studies that have linked changes in brain function to the ebbs and flows of affective states over time have mostly investigated a single individual. In this study, we explored how the functional connectivity of brain areas associated with affective processes can explain within-person fluctuations in self-reported positive and negative affects across several subjects. To do so, we leveraged the Day2day dataset that includes 40 to 50 resting-state functional magnetic resonance imaging scans along self-reported positive and negative affectivity from a sample of six healthy participants. Sparse multivariate mixed-effect linear models could explain 15% and 11% of the within-person variation in positive and negative affective states, respectively. Evaluation of these models' generalizability to new data demonstrated the ability to predict approximately 5% and 2% of positive and negative affect variation. The functional connectivity of limbic areas, such as the amygdala, hippocampus, and insula, appeared most important to explain the temporal dynamics of affects over days, weeks, and months.

A meta-analysis on the relationship between subjective cognitive failures as measured by the cognitive failures questionnaire (CFQ) and objective performance on executive function tasks

The Cognitive Failures Questionnaire (CFQ) has been widely used as a measure of subjective cognitive function in everyday life for decades. However, the evidence on how it relates to objective performance on executive function tasks is mixed. One possible reason for these mixed results is that the CFQ has selective relationships with some aspects of executive function and not others. Here, therefore, we classified tasks according to an influential framework of executive functions-switching, updating, inhibition, and we also considered the Sustained Attention to Response Task (SART) as a category because it was custom designed to gauge cognitive failures. We synthesized a large body of available evidence and performed four Bayesian meta-analyses on the relationship between CFQ scores and objective performance on executive function tasks in these four categories. Results suggested that CFQ scores were associated with objective performance on SART (18 effect sizes, μ = -.19, BF10 = 18.03, i.e., 18.03 times more evidence of a relationship versus no relationship), updating working memory (49 effect sizes, μ = -.06, BF10 = 17.80), and inhibition tasks (41 effect sizes, μ = -.07, BF10 = 15.40), whereas there was not definitive evidence regarding switching (34 effect sizes, μ = -.06, BF10 = .50, i.e., two times greater evidence for no relationship). This suggests that subjective cognitive function can predict objective performance on at least some executive function tasks. We discuss methodological and theoretical factors that constrain the maximum observable correlation and consider the relative insights that subjective measures versus task performance provide.

Reporting checklists in neuroimaging: promoting transparency, replicability, and reproducibility

Neuroimaging plays a crucial role in understanding brain structure and function, but the lack of transparency, reproducibility, and reliability of findings is a significant obstacle for the field. To address these challenges, there are ongoing efforts to develop reporting checklists for neuroimaging studies to improve the reporting of fundamental aspects of study design and execution. In this review, we first define what we mean by a neuroimaging reporting checklist and then discuss how a reporting checklist can be developed and implemented. We consider the core values that should inform checklist design, including transparency, repeatability, data sharing, diversity, and supporting innovations. We then share experiences with currently available neuroimaging checklists. We review the motivation for creating checklists and whether checklists achieve their intended objectives, before proposing a development cycle for neuroimaging reporting checklists and describing each implementation step. We emphasize the importance of reporting checklists in enhancing the quality of data repositories and consortia, how they can support education and best practices, and how emerging computational methods, like artificial intelligence, can help checklist development and adherence. We also highlight the role that funding agencies and global collaborations can play in supporting the adoption of neuroimaging reporting checklists. We hope this review will encourage better adherence to available checklists and promote the development of new ones, and ultimately increase the quality, transparency, and reproducibility of neuroimaging research.

Revisiting the role of computational neuroimaging in the era of integrative neuroscience

Computational models have become integral to human neuroimaging research, providing both mechanistic insights and predictive tools for human cognition and behavior. However, concerns persist regarding the ecological validity of lab-based neuroimaging studies and whether their spatiotemporal resolution is not sufficient for capturing neural dynamics. This review aims to re-examine the utility of computational neuroimaging, particularly in light of the growing prominence of alternative neuroscientific methods and the growing emphasis on more naturalistic behaviors and paradigms. Specifically, we will explore how computational modeling can both enhance the analysis of high-dimensional imaging datasets and, conversely, how neuroimaging, in conjunction with other data modalities, can inform computational models through the lens of neurobiological plausibility. Collectively, this evidence suggests that neuroimaging remains critical for human neuroscience research, and when enhanced by computational models, imaging can serve an important role in bridging levels of analysis and understanding. We conclude by proposing key directions for future research, emphasizing the development of standardized paradigms and the integrative use of computational modeling across neuroimaging techniques.

Testing the convergent validity, domain generality, and temporal stability of selected measures of people's tendency to explore

Given the ubiquity of exploration in everyday life, researchers from many disciplines have developed methods to measure exploratory behaviour. There are therefore many ways to quantify and measure exploration. However, it remains unclear whether the different measures (i) have convergent validity relative to one another, (ii) capture a domain general tendency, and (iii) capture a tendency that is stable across time. In a sample of 678 participants, we found very little evidence of convergent validity for the behavioural measures (Hypothesis 1); most of the behavioural measures lacked sufficient convergent validity with one another or with the self-reports. In psychometric modelling analyses, we could not identify a good fitting model with an assumed general tendency to explore (Hypothesis 2); the best fitting model suggested that the different behavioural measures capture behaviours that are specific to the tasks. In a subsample of 254 participants who completed the study a second time, we found that the measures had stability across an 1 month timespan (Hypothesis 3). Therefore, although there were stable individual differences in how people approached each task across time, there was no generalizability across tasks, and drawing broad conclusions about exploratory behaviour from studies using these tasks may be problematic. The Stage 1 protocol for this Registered Report was accepted in principle on 2nd December 2022 https://doi.org/10.6084/m9.figshare.21717407.v1 . The protocol, as accepted by the journal, can be found at https://doi.org/10.17605/OSF.IO/64QJU .

The value of error-correcting responses for cognitive assessment in games

Traditional conflict-based cognitive assessment tools are highly behaviorally restrictive, which prevents them from capturing the dynamic nature of human cognition, such as the tendency to make error-correcting responses. The cognitive game Tunnel Runner measures interference control, response inhibition, and response-rule switching in a less restrictive manner than traditional cognitive assessment tools by giving players movement control after an initial response and encouraging error-correcting responses. Nevertheless, error-correcting responses remain unused due to a limited understanding of what they measure and how to use them. To facilitate the use of error-correcting responses to measure and understand human cognition, we developed theoretically-grounded measures of error-correcting responses in Tunnel Runner and assessed whether they reflected the same cognitive functions measured via initial responses. Furthermore, we evaluated the measurement potential of error-correcting responses. We found that initial and error-correcting responses similarly reflected players' response inhibition and interference control, but not their response-rule switching. Furthermore, combining the two response types increased the reliability of interference control and response inhibition measurements. Lastly, error-correcting responses showed the potential to measure response inhibition on their own. Our results pave the way toward understanding and using post-decision change of mind data for cognitive measurement and other research and application contexts.

Local but not global graph theoretic measures of semantic networks generalize across tasks

"Dogs" are connected to "cats" in our minds, and "backyard" to "outdoors." Does the structure of this semantic knowledge differ across people? Network-based approaches are a popular representational scheme for thinking about how relations between different concepts are organized. Recent research uses graph theoretic analyses to examine individual differences in semantic networks for simple concepts and how they relate to other higher-level cognitive processes, such as creativity. However, it remains ambiguous whether individual differences captured via network analyses reflect true differences in measures of the structure of semantic knowledge, or differences in how people strategically approach semantic relatedness tasks. To test this, we examine the reliability of local and global metrics of semantic networks for simple concepts across different semantic relatedness tasks. In four experiments, we find that both weighted and unweighted graph theoretic representations reliably capture individual differences in local measures of semantic networks (e.g., how related pot is to pan versus lion). In contrast, we find that metrics of global structural properties of semantic networks, such as the average clustering coefficient and shortest path length, are less robust across tasks and may not provide reliable individual difference measures of how people represent simple concepts. We discuss the implications of these results and offer recommendations for researchers who seek to apply graph theoretic analyses in the study of individual differences in semantic memory.

On the reliability of value-modulated attentional capture: An online replication and multiverse analysis

Stimuli predicting rewards are more likely to capture attention, even when they are not relevant to our current goals. Individual differences in value-modulated attentional capture (VMAC) have been associated with various psychopathological conditions in the scientific literature. However, the claim that this attentional bias can predict individual differences requires further exploration of the psychometric properties of the most common experimental paradigms. The current study replicated the VMAC effect in a large online sample (N = 182) and investigated the internal consistency, with a design that allowed us to measure the effect during learning (rewarded phase) and after acquisition, once feedback was omitted (unrewarded phase). Through the rewarded phase there was gradual increase of the VMAC effect, which did not decline significantly throughout the unrewarded phase. Furthermore, we conducted a reliability multiverse analysis for 288 different data preprocessing specifications across both phases. Specifications including more blocks in the analysis led to better reliability estimates in both phases, while specifications that removed more outliers also improved reliability, suggesting that specifications with more, but less noisy, trials led to better reliability estimates. Nevertheless, in most instances, especially those considering fewer blocks of trials, reliability estimates fell below the minimum recommended thresholds for research on individual differences. Given the present results, we encourage researchers working on VMAC to take into account reliability when designing studies aimed at capturing individual differences and provide recommendations to improve methodological practices.

Modality matters: Three auditory conflict tasks to measure individual differences in attention control

Early work on selective attention used auditory-based tasks, such as dichotic listening, to shed light on capacity limitations and individual differences in these limitations. Today, there is great interest in individual differences in attentional abilities, but the field has shifted towards visual-modality tasks. Furthermore, most conflict-based tests of attention control lack reliability due to low signal-to-noise ratios and the use of difference scores. Critically, it is unclear to what extent attention control generalizes across sensory modalities, and without reliable auditory-based tests, an answer to this question will remain elusive. To this end, we developed three auditory-based tests of attention control that use an adaptive response deadline (DL) to account for speed-accuracy trade-offs: Auditory Simon DL, Auditory Flanker DL, and Auditory Stroop DL. In a large sample (N = 316), we investigated the psychometric properties of the three auditory conflict tasks, tested whether attention control is better modeled as a unitary factor or modality-specific factors, and estimated the extent to which unique variance in modality-specific factors contributed incrementally to the prediction of dichotic listening and multitasking performance. Our analyses indicated that the auditory conflict tasks have strong psychometric properties and demonstrate convergent validity with visual tests of attention control. Auditory and visual attention control factors were highly correlated (r = .81)-even after controlling for perceptual processing speed (r = .75). Modality-specific attention control factors accounted for unique variance in modality-matched criterion measures, but the majority of the explained variance was modality-general. The results suggest an interplay between modality-general attention control and modality-specific processing.

Capturing individual differences in sentence processing: How reliable is the self-paced reading task?

Advances in research on language processing have originally come from group-level comparisons, but there is now a growing interest in individual differences. To investigate individual differences, tasks that have shown robust group-level differences are often used with the implicit assumption that they will also be reliable when used as an individual differences measure. Here, we examined whether one of the primary tasks used in psycholinguistic research on language processing, the self-paced reading task, can reliably measure individual differences in relative clause processing. We replicated the well-established effects of relative clauses at the group level, with object relative clauses being more difficult to process than subject relative clauses. However, when using difference scores, the reliability of the size of the relative clause effect was close to zero because the self-paced reading times for the different relative clause types were highly correlated within individuals. Nonetheless, we found that the self-paced reading task can be used to reliably capture individual differences in overall reading speed as well as key sentence regions when the two types of relative clause sentences are considered separately. Our results indicate that both the reliability and validity of different sentence regions need to be assessed to determine whether and when self-paced reading can be used to examine individual differences in language processing.

Executive Function and Impulsivity Predict Distinct Genetic Variance in Internalizing Problems, Externalizing Problems, Thought Disorders, and Compulsive Disorders: A Genomic Structural Equation Modeling Study

Individual differences in self-control predict many health and life outcomes. Building on twin literature, we used genomic structural equation modeling to test the hypothesis that genetic influences on executive function and impulsivity predict independent variance in mental health and other outcomes. The impulsivity factor (comprising urgency, lack of premeditation, and other facets) was only modestly genetically correlated with low executive function (rg =.13). Controlling for impulsivity, low executive function was genetically associated with increased internalizing (βg =.15), externalizing (βg =.13), thought disorders (βg =.38), compulsive disorders (βg =.22), and chronotype (βg =.11). Controlling for executive function, impulsivity was positively genetically associated with internalizing (βg =.36), externalizing (βg =.55), body mass index (βg =.26), and insomnia (βg =.35), and negatively genetically associated with compulsive disorders (βg = -.17). Executive function and impulsivity were both genetically correlated with general cognitive ability and educational attainment. This work suggests that executive function and impulsivity are genetically separable and show independent associations with mental health.

Implementation of an online spacing flanker task and evaluation of its test-retest reliability using measures of inhibitory control and the distribution of spatial attention

The flanker task (Eriksen & Eriksen, Perception & Psychophysics, 16(1), 143-149, 1974) has been highly influential and widely used in studies of visual attention. Its simplicity has made it popular to include it in experimental software packages and online platforms. The spacing flanker task (SFT), in which the distance between the target and flankers varies, is useful for studying the distribution of attention across space as well as inhibitory control. Use of the SFT requires that the viewing environment (e.g., stimulus size and viewing distance) be controlled, which is a challenge for online delivery. We implement and evaluate an online version of the SFT that includes two calibration pretests to provide the necessary control. Test-retest and split-half reliability of the online version was compared with a laboratory version on measures of inhibitory control and measures of the distribution of attention across space. Analyses show that the online SFT is comparable to laboratory testing on all measures. Results also identify two measures with good test-retest reliability that hold promise for studying performance in the SFT: the mean flanker effect (ICC = 0.745) and RTs on incongruent trials across distances (ICC = 0.65-0.71).

Individual differences in visual search: A systematic review of the link between visual search performance and traits or abilities

Visual search (VS) comprises a class of tasks that we typically perform several times during a day and requires intentionally scanning (with or without moving the eyes) the environment for a specific target (be it an object or a feature) among distractor stimuli. Experimental research in lab-based or real-world settings has offered insight into its underlying neurocognitive mechanisms from a nomothetic point of view. A lesser-known but rapidly growing body of quasi-experimental and correlational research has explored the link between individual differences and VS performance. This combines different research traditions and covers a wide range of individual differences in studies deploying a vast array of VS tasks. As such, it is a challenge to determine whether any associations highlighted in single studies are robust when considering the wider literature. However, clarifying such relationships systematically and comprehensively would help build more accurate models of VS, and it would highlight promising directions for future research. This systematic review provides an up to date and comprehensive synthesis of the existing literature investigating associations between common indices of performance in VS tasks and measures of individual differences mapped onto four categories of cognitive abilities (short-term working memory, fluid reasoning, visual processing and processing speed) and seven categories of traits (Big Five traits, trait anxiety and autistic traits). Consistent associations for both traits (in particular, conscientiousness, autistic traits and trait anxiety - the latter limited to emotional stimuli) and cognitive abilities (particularly visual processing) were identified. Overall, however, informativeness of future studies would benefit from checking and reporting the reliability of all measurement tools, applying multiplicity correction, using complementary techniques, study preregistration and testing why, rather than only if, a robust relation between certain individual differences and VS performance exists.

A systematic review of visual processing in body dysmorphic disorder (BDD)

To understand the visual preponderance of perceived flaws in appearance in body dysmorphic disorder (BDD), the study of visual processing has been growing. Studies have focused on facial and other basic visual stimuli. The current literature does not provide evidence of consistent behavioural patterns, lacking an overarching body of work describing visual processing in BDD. This systematic review aims to characterise behavioural outcomes of visual processing anomalies and/or deficits in BDD. Articles were collected through online databases MEDLINE and PubMed, and were included if they comprised a clinical BDD group, and were published after 1990. Results indicate that individuals with BDD demonstrate deficits in emotional face processing, a possible overreliance on detail processing, aberrant eye-scanning behaviours, and a tendency to overvalue attractiveness. While findings consistently signal towards visual deficits in BDD, there is lack of clarity as to the type. This inconsistency may be attributed to heterogeneity within BDD samples and differences in experimental design (i.e., stimuli, tasks, conditions). There are difficulties distinguishing between BDD-associated deficits and those associated with OCD or eating disorders. A coherent framework, including sample characterisation and task design will seek to generate clear and consistent behavioural patterns to guide future treatments.

A shared temporal window of integration across cognitive control and reinforcement learning paradigms: A correlational study

Cognitive control refers to the ability to override prepotent response tendencies to achieve goal-directed behavior. On the other hand, reinforcement learning refers to the learning of actions through feedback and reward. Although cognitive control and reinforcement learning are often viewed as opposing forces in driving behavior, recent theories have emphasized possible similarities in their underling processes. With this study, we aimed to investigate whether a similar time window of integration could be observed during the learning of control on the one hand, and the learning rate in reinforcement learning paradigms on the other. To this end, we performed a correlational analysis on a large public dataset (n = 522) including data from two reinforcement learning tasks, i.e., a probabilistic selection task and a probabilistic Wisconsin Card Sorting Task (WCST), and data from a classic conflict task (i.e., the Stroop task). Results showed expected correlations between the time scale of control indices and learning rate in the probabilistic WCST. Moreover, the learning-rate parameters of the two reinforcement learning tasks did not correlate with each other. Together, these findings suggest a reliance on a shared learning mechanism between these two traditionally distinct domains, while at the same time emphasizing that value updating processes can still be very task-specific. We speculate that updating processes in the Stroop and WCST may be more related because both tasks require task-specific updating of stimulus features (e.g., color, word meaning, pattern, shape), as opposed to stimulus identity.

No negative effects of intra-abdominal bio-logger implantation under general anaesthesia on spatial cognition learning in a hibernator the edible dormouse

The effect of hibernation on cognitive capacities of individuals is not fully understood, as studies provide conflicting results. Most studies focus on behavioural observations without taking the physiological state of individuals to account. To mechanistically understand the effect of hibernation on the brain, physiological parameters need to be included. The implantation of bio-loggers can provide insights on i.e. body temperature without further manipulation of the animals. Surgeries and anaesthesia, however, can harm animals' health and cause cognitive dysfunction, potentially biasing data collected through bio-loggers. We investigated the effects of bio-logger implantation surgery on cognitive performance and learning, controlling for animal and study design characteristics. First, juvenile dormice successfully learned to solve a spatial cognition task using a vertical maze. Distance, transitions, velocity, and duration were measured as indicators for performance. After training, bio-loggers were implanted intra-abdominally under general anaesthesia. Animals were re-tested in the maze two weeks after. We found no effect of bio-logger implantation and surgery on performance. This study is the first to show spatial cognition learning in edible dormice and provides a full description of the peri-anaesthetic management and a protocol for bio-logger implantation surgery in dormice. Importantly, measures were taken to mitigate common anaesthetic complications that could lead to post-operative cognitive dysfunction and influence animal behaviour. By pairing physiological measurements through bio-logger implantation with behaviour and cognition measurements, future research will significantly advance the understanding on mechanisms of learning and behaviour.

Psychiatric neuroimaging designs for individualised, cohort, and population studies

Psychiatric neuroimaging faces challenges to rigour and reproducibility that prompt reconsideration of the relative strengths and limitations of study designs. Owing to high resource demands and varying inferential goals, current designs differentially emphasise sample size, measurement breadth, and longitudinal assessments. In this overview and perspective, we provide a guide to the current landscape of psychiatric neuroimaging study designs with respect to this balance of scientific goals and resource constraints. Through a heuristic data cube contrasting key design features, we discuss a resulting trade-off among small sample, precision longitudinal studies (e.g., individualised studies and cohorts) and large sample, minimally longitudinal, population studies. Precision studies support tests of within-person mechanisms, via intervention and tracking of longitudinal course. Population studies support tests of generalisation across multifaceted individual differences. A proposed reciprocal validation model (RVM) aims to recursively leverage these complementary designs in sequence to accumulate evidence, optimise relative strengths, and build towards improved long-term clinical utility.

Test-retest reliability of reinforcement learning parameters

It has recently been suggested that parameter estimates of computational models can be used to understand individual differences at the process level. One area of research in which this approach, called computational phenotyping, has taken hold is computational psychiatry. One requirement for successful computational phenotyping is that behavior and parameters are stable over time. Surprisingly, the test-retest reliability of behavior and model parameters remains unknown for most experimental tasks and models. The present study seeks to close this gap by investigating the test-retest reliability of canonical reinforcement learning models in the context of two often-used learning paradigms: a two-armed bandit and a reversal learning task. We tested independent cohorts for the two tasks (N = 69 and N = 47) via an online testing platform with a between-test interval of five weeks. Whereas reliability was high for personality and cognitive measures (with ICCs ranging from .67 to .93), it was generally poor for the parameter estimates of the reinforcement learning models (with ICCs ranging from .02 to .52 for the bandit task and from .01 to .71 for the reversal learning task). Given that simulations indicated that our procedures could detect high test-retest reliability, this suggests that a significant proportion of the variability must be ascribed to the participants themselves. In support of that hypothesis, we show that mood (stress and happiness) can partly explain within-participant variability. Taken together, these results are critical for current practices in computational phenotyping and suggest that individual variability should be taken into account in the future development of the field.

Semantic Stroop interference is modulated by the availability of executive resources: Insights from delta-plot analyses and cognitive load manipulation

We investigated whether, during visual word recognition, semantic processing is modulated by attentional control mechanisms directed at matching semantic information with task-relevant goals. In previous research, we analyzed the semantic Stroop interference as a function of response latency (delta-plot analyses) and found that this phenomenon mainly occurs in the slowest responses. Here, we investigated whether this pattern is due to reduced ability to proactively maintain the task goal in these slowest trials. In two pairs of experiments, participants completed two semantic Stroop tasks: a classic semantic Stroop task (Experiment 1A and 2A) and a semantic Stroop task combined with an n-back task (Experiment 1B and 2B). The two pairs of experiments only differed in the trial pace, which was slightly faster in Experiments 2A and 2B than in Experiments 1A and 1B. By taxing the executive control system, the n-back task was expected to hinder proactive control. Delta-plot analyses of the semantic Stroop task replicated the enhanced effect in the slowest responses, but only under sufficient time pressure. Combining the semantic Stroop task with the n-back task produced a change in the distributional profile of semantic Stroop interference, which we ascribe to a general difficulty in the use of proactive control. Our findings suggest that semantic Stroop interference is, to some extent, dependent on the available executive resources, while also being sensitive to subtle variations in task conditions.

Human laboratory models of reward in substance use disorder

Human laboratory models in substance use disorder provide a key intermediary step between highly controlled and mechanistically informative non-human preclinical methods and clinical trials conducted in human populations. Much like preclinical models, the variety of human laboratory methods provide insights into specific features of substance use disorder rather than modelling the diverse causes and consequences simultaneously in a single model. This narrative review provides a discussion of popular models of reward used in human laboratory research on substance use disorder with a focus on the specific contributions that each model has towards informing clinical outcomes (forward translation) and analogs within preclinical models (backward translation). Four core areas of human laboratory research are discussed: drug self-administration, subjective effects, behavioral economics, and cognitive and executive function. Discussion of common measures and models used, the features of substance use disorder that these methods are purported to evaluate, unique issues for measure validity and application, and translational links to preclinical models and special considerations for studies wishing to evaluate homology across species is provided.

Psychometrics in experimental psychology: A case for calibration

Psychometrics is historically grounded in the study of individual differences. Consequently, common metrics such as quantitative validity and reliability require between-person variance in a psychological variable to be meaningful. Experimental psychology, in contrast, deals with variance between treatments, and experiments often strive to minimise within-group person variance. In this article, I ask whether and how psychometric evaluation can be performed in experimental psychology. A commonly used strategy is to harness between-person variance in the treatment effect. Using simulated data, I show that this approach can be misleading when between-person variance is low, and in the face of methods variance. I argue that this situation is common in experimental psychology, because low between-person variance is desirable, and because methods variance is no more problematic in experimental settings than any other source of between-person variance. By relating validity and reliability with the corresponding concepts in measurement science outside psychology, I show how experiment-based calibration can serve to compare the psychometric quality of different measurement methods in experimental psychology.

Brain imaging of a gamified cognitive flexibility task in young and older adults

The study aimed to develop and validate a gamified cognitive flexibility task through brain imaging, and to investigate behavioral and brain activation differences between young and older adults during task performance. Thirty-one young adults (aged 18-35) and 31 older adults (aged 60-80) were included in the present study. All participants underwent fMRI scans while completing the gamified cognitive flexibility task. Results showed that young adults outperformed older adults on the task. The left inferior frontal junction (IFJ), a key region of cognitive flexibility, was significantly activated during the task in both older and young adults. Comparatively, the percent signal change in the left IFJ was stronger in older adults than in young adults. Moreover, older adults demonstrated more precise representations during the task in the left IFJ. Additionally, the left inferior parietal lobule (IPL) and superior parietal lobule in older adults and the left middle frontal gyrus (MFG) and inferior frontal gyrus in young adults were also activated during the task. Psychophysiological interaction analyses showed significant functional connectivity between the left IFJ and the left IPL, as well as the right precuneus in older adults. In young adults, significant functional connectivity was found between the left IFJ and the left MFG, as well as the right angular. The current study provides preliminary evidence for the validity of the gamified cognitive flexibility task through brain imaging. The findings suggest that this task could serve as a reliable tool for assessing cognitive flexibility and for exploring age-related differences of cognitive flexibility in both brain and behavior.

Convergence of Age Differences in Risk Preference, Impulsivity, and Self-Control: A Multiverse Analysis

OBJECTIVES

Numerous theories exist regarding age differences in risk preference and related constructs, yet many of them offer conflicting predictions and fail to consider convergence between measurement modalities or constructs. To pave the way for conceptual clarification and theoretical refinement, in this preregistered study we aimed to comprehensively examine age effects on risk preference, impulsivity, and self-control using different measurement modalities, and to assess their convergence.

METHODS

We collected a large battery of self-report, informant report, behavioral, hormone, and neuroimaging measures from a cross-sectional sample of 148 (55% female) healthy human participants between 16 and 81 years (mean age = 46 years, standard deviation [SD] = 19). We used an extended sample of 182 participants (54% female, mean age = 46 years, SD = 19) for robustness checks concerning the results from self-reports, informant reports, and behavioral measures. For our main analysis, we performed specification curve analyses to visualize and estimate the convergence between the different modalities and constructs.

RESULTS

Our multiverse analysis approach revealed convergent results for risk preference, impulsivity, and self-control from self- and informant reports, suggesting a negative effect of age. For behavioral, hormonal, and neuroimaging outcomes, age effects were mostly absent.

DISCUSSION

Our findings call for conceptual clarification and improved operationalization to capture the putative mechanisms underlying age-related differences in risk preference and related constructs.

Negative affective priming: Reliability and associations with depression symptoms in three samples

The negative affective priming (NAP) task is a behavioral measure of inhibition of emotional stimuli. Previous studies using the NAP task have found that individuals with depression show reduced inhibition of negative stimuli, suggesting that inhibition biases may play a role in the etiology and maintenance of depression. However, the psychometric properties of the NAP task have not been evaluated or reported. In the present study, we report data on the association between NAP task performance and depression symptoms in three independent samples, and we evaluate the internal consistency and test-retest reliability of the NAP effect indices. The NAP effect for both negative and positive target words had poor internal consistency in all three samples, as well as poor 2-week (Study 2) and 6-month (Study 3) test-retest reliability. The internal consistency and test-retest reliability of response times (RT) for the individual trial types were moderate to high, as were the intercorrelations between trial types. This pattern of results indicates that overall RT is reliable but variance in RTs for the different trial types in the NAP task is indistinguishable from variance in overall RT. Depression symptom severity was not associated with the NAP effect for negative or positive target words in any of the samples, which could be due to the poor reliability of the NAP effect. Based on these findings, we do not recommend that researchers use the NAP task as a measure of individual differences in the inhibition of emotional stimuli.

Test-retest reliability of the attention network test from the perspective of intrinsic network organization

The attention network test (ANT), developed based on the triple-network taxonomy by Posner and colleagues, has been widely used to examine the efficacy of alerting, orienting and executive control in clinical and developmental neuroscience studies. Recent research suggests the imperfect reliability of the behavioural ANT and its variants. However, the classical ANT fMRI task's test-retest reliability has received little attention. Moreover, it remains ambiguous whether the attention-related intrinsic network components, especially the dorsal attention, ventral attention and frontoparietal network, manifest acceptable reliability. The present study approaches these issues by utilizing an openly available ANT fMRI dataset for participants with Parkinson's disease and healthy elderly. The reproducibility of group-level activations across sessions and participant groups and the test-retest reliability at the individual level were examined at the voxel, region and network levels. The intrinsic network was defined using the Yeo-Schaefer atlas. Our results reveal three critical facets: (1) the overlapping of the group-level contrast map between sessions and between participant groups was unsatisfactory; (2) the reliability of alerting, orienting and executive, defined as a contrast between conditions, was worse than estimates of specific conditions. (3) Dorsal attention, ventral attention, visual and somatomotor networks showed acceptable reliability for the congruent and incongruent conditions. Our results suggest that specific condition estimates might be used instead of the contrast map for individual or group-difference studies.

The Stroop legacy: A cautionary tale on methodological issues and a proposed spatial solution

The Stroop task is a seminal paradigm in experimental psychology, so much that various variants of the classical color-word version have been proposed. Here we offer a methodological review of them to emphasize the importance of designing methodologically rigorous Stroop tasks. This is not an end by itself, but it is fundamental to achieve adequate measurement validity, which is currently hindered by methodological heterogeneity and limitations. Among the several Stroop task variants in the literature, our methodological overview shows that the spatial Stroop task is not only a potentially methodologically adequate variant, which can thus assure measuring the Stroop effect with the required validity, but it might even allow researchers to overcome some of the methodological limitations of the classical paradigm due to its use of verbal stimuli. We thus focused on the spatial Stroop tasks in the literature to verify whether they really exploit such inherent potentiality. However, we show that this was generally not the case because only a few of them (1) are purely spatial, (2) ensure both all the three types of conflicts/facilitations (at the stimulus, response, and task levels) and the dimensional overlaps considered fundamental for yielding a complete Stroop effect according to the multiple loci account and Kornblum's theory, respectively, and (3) controlled for low-level binding and priming effects that could bias the estimated Stroop effect. Based on these methodological considerations, we present some examples of spatial Stroop tasks that, in our view, satisfy such requirements and, thus, ensure producing complete Stroop effects.

Intergroup evaluative bias in facial representations of immigrants and citizens in the United States

We used a reverse-correlation image-classification paradigm to visualize facial representations of immigrants and citizens in the United States. Visualizations of immigrants' faces were judged by independent raters as less trustworthy and less competent and were more likely to be categorized as a non-White race/ethnicity than were visualizations of citizens' faces. Additionally, image generators' personal characteristics (e.g., implicit and explicit evaluations of immigrants, nativity status) did not reliably track with independent judges' ratings of image generators' representations of immigrants. These findings suggest that anti-immigrant sentiment and racial/ethnic assumptions characterize facial representations of immigrants in the United States, even among people who harbor positivity toward immigrants.

Reliability Theory for Measurements with Variable Test Length, Illustrated with ERN and Pe Collected in the Flanker Task

In psychophysiology, an interesting question is how to estimate the reliability of event-related potentials collected by means of the Eriksen Flanker Task or similar tests. A special problem presents itself if the data represent neurological reactions that are associated with some responses (in case of the Flanker Task, responding incorrectly on a trial) but not others (like when providing a correct response), inherently resulting in unequal numbers of observations per subject. The general trend in reliability research here is to use generalizability theory and Bayesian estimation. We show that a new approach based on classical test theory and frequentist estimation can do the job as well and in a simpler way, and even provides additional insight to matters that were unsolved in the generalizability method approach. One of our contributions is the definition of a single, overall reliability coefficient for an entire group of subjects with unequal numbers of observations. Both methods have slightly different objectives. We argue in favor of the classical approach but without rejecting the generalizability approach.

Impact of analytic decisions on test-retest reliability of individual and group estimates in functional magnetic resonance imaging: a multiverse analysis using the monetary incentive delay task

Empirical studies reporting low test-retest reliability of individual blood oxygen-level dependent (BOLD) signal estimates in functional magnetic resonance imaging (fMRI) data have resurrected interest among cognitive neuroscientists in methods that may improve reliability in fMRI. Over the last decade, several individual studies have reported that modeling decisions, such as smoothing, motion correction and contrast selection, may improve estimates of test-retest reliability of BOLD signal estimates. However, it remains an empirical question whether certain analytic decisions consistently improve individual and group level reliability estimates in an fMRI task across multiple large, independent samples. This study used three independent samples (Ns: 60, 81, 119) that collected the same task (Monetary Incentive Delay task) across two runs and two sessions to evaluate the effects of analytic decisions on the individual (intraclass correlation coefficient [ICC(3,1)]) and group (Jaccard/Spearman rho) reliability estimates of BOLD activity of task fMRI data. The analytic decisions in this study vary across four categories: smoothing kernel (five options), motion correction (four options), task parameterizing (three options) and task contrasts (four options), totaling 240 different pipeline permutations. Across all 240 pipelines, the median ICC estimates are consistently low, with a maximum median ICC estimate of .43 - .55 across the three samples. The analytic decisions with the greatest impact on the median ICC and group similarity estimates are the Implicit Baseline contrast, Cue Model parameterization and a larger smoothing kernel. Using an Implicit Baseline in a contrast condition meaningfully increased group similarity and ICC estimates as compared to using the Neutral cue. This effect was largest for the Cue Model parameterization; however, improvements in reliability came at the cost of interpretability. This study illustrates that estimates of reliability in the MID task are consistently low and variable at small samples, and a higher test-retest reliability may not always improve interpretability of the estimated BOLD signal.

A measure of reliability convergence to select and optimize cognitive tasks for individual differences research

Surging interest in individual differences has faced setbacks in light of recent replication crises in psychology, for example in brain-wide association studies exploring brain-behavior correlations. A crucial component of replicability for individual differences studies, which is often assumed but not directly tested, is the reliability of the measures we use. Here, we evaluate the reliability of different cognitive tasks on a dataset with over 250 participants, who each completed a multi-day task battery. We show how reliability improves as a function of number of trials, and describe the convergence of the reliability curves for the different tasks, allowing us to score tasks according to their suitability for studies of individual differences. We further show the effect on reliability of measuring over multiple time points, with tasks assessing different cognitive domains being differentially affected. Data collected over more than one session may be required to achieve trait-like stability.

A proof-of-concept study testing the factor structure of the Stop Signal Task: overlap with substance use and mental health symptoms

Background: Research utilizing experimental tasks usually does not report estimates of internal reliability of measurement. However, modern measurement theories conceptualize reliability as sample dependent indicating that reliability should be empirically demonstrated in the samples used to make inferences.Objectives: Test whether confirmatory factor analytic (CFA) estimates of reliability can be applied to a commonly used task measuring response inhibition (the Stop Signal Task) to predict substance use (alcohol and cannabis) and mental health symptoms.Methods: Thirty-seven participants between the ages of 18-20 (72% female; 16% Asian, 3% Native American, 11% Black or African American, 59% White; 32% Latino/a/x) were recruited via social media advertisement and attended a laboratory visit. The Stop Signal Reaction Time (SSRT) was calculated as the outcome for three experimental blocks and used as indicators in a CFA.Results: CFA suggests the task yields reliable scores; factor loadings were statistically significant (p < .05) and substantial (standardized loadings ranged from .74 to .94). However, reliability increased across experimental blocks and error was non-trivial (ranging from 50% to 12% of the variance). The inhibition factor predicted higher maximum number of drinks consumed (β = .37, p < .05), higher frequency of cannabis use (β = .39, p < .05), and more cannabis use occasions within using days (β = .40, p < .05), as well as facets of mental health (anxious/depression, attention, and anxiety problems; all p's < .05).Conclusion: Results support the utility of CFA to test for reliability of measurement, with the ability to inhibit dominant responses serving as a transdiagnostic correlate of substance use and mental health problems.

Testing the social motivation theory of autism: the role of co-occurring anxiety

BACKGROUND

The Social Motivation Theory proposes that social reward processing differences underlie autism. However, low social motivation has also been linked to higher anxiety. Given the co-occurrence between autism and anxiety, it is possible that anxiety drives the association between social motivation and autistic characteristics. This study tests the mechanisms underlying the association between social motivation and autistic traits.

METHODS

Participants were 165 adolescents (71 male), aged 10-16 years, from the Mapping profiles of cognition, motivation and attention in childhood (C-MAPS) study, enriched for autistic traits (70 participants with an autism diagnosis, 37 male). Participants completed a battery of online experimental tasks, including a Choose-a-Movie social motivation task and social cognition measures (theory of mind; emotion recognition), alongside parent-reported child anxiety and autistic traits.

RESULTS

Higher social motivation was significantly associated with lower autistic traits (β = -.26, p < .001). Controlling for social cognition did not change the association between social motivation and autistic traits. Controlling for anxiety did significantly reduce the strength of the association (unstandardized coefficient change: p = .003), although social motivation remained associated with autistic traits (β = -.16, p = .004). Post hoc analyses demonstrated differential sex-effects: The association between social motivation and autistic traits was significant only in the females (β = -.38, p < .001), as was the attenuation by anxiety (unstandardized coefficient change: p < .001).

CONCLUSIONS

The association between social motivation and autistic traits could be partially attributed to co-occurring anxiety. Sex-specific effects found in females may be due to environmental factors such as increased social demands in adolescent female relationships. Results are consistent with self-report by autistic individuals who do not identify as having reduced social motivation.

Don't SNARC me now! Intraindividual variability of cognitive phenomena - Insights from the Ironman paradigm

Two implicit generalizations are often made from group-level studies in cognitive experimental psychology and their common statistical analysis in the general linear model: (1) Group-level phenomena are assumed to be present in every participant with variations between participants being often treated as random error in data analyses; (2) phenomena are assumed to be stable over time. In this preregistered study, we investigated the validity of these generalizations in the commonly used parity judgment task. In the proposed Ironman paradigm, the intraindividual presence and stability of three popular numerical cognition effects were tested in 10 participants on 30 days: the SNARC (Spatial-Numerical Association of Response Codes, i.e., faster left-/right-sided responses to small/large magnitude numbers, respectively; Dehaene, Bossini, & Giraux, 1993), MARC (Linguistic Markedness of Response Codes; i.e., faster left-/right-sided responses to odd/even numbers, respectively; Nuerk, Iversen, & Willmes, 2004), and Odd (i.e., faster responses to even numbers; Hines, 1990) effects. We replicated the group-level effects; however, they were reliably present in only four to five (SNARC), six (MARC) or five (Odd) of 10 participants. Fluctuations seemed unsystematic, although the SNARC effect decreased over time along with reaction times. No correlation between the SNARC and MARC effects and sleep duration, tiredness, daytime, and consumption of stimulants were detected in most participants. These results challenge the frequent generalizations from group-level phenomena to individual participants and from single sessions to typical behavior. The innovative Ironman paradigm combined with bootstrap analyses permits unique insights into the intraindividual presence and stability of cognitive phenomena.

Extended ambulatory assessment of executive function: within-person reliability of working memory and inhibitory control tasks

INTRODUCTION

Ambulatory assessment of executive function - particularly in the form working memory (WM) - is increasingly common. Few studies to date, however, have also incorporated ambulatory measures of inhibitory control. Critically, the extended within-person reliability of ambulatory tasks tapping each of these constructs has been largely overlooked.

METHOD

Participants (N = 283, Mage = 23.74 years, SD = 9.04) received notifications every 3 days (for 4 weeks) to undertake ambulatory assessment versions of the n-Back and Stop-Signal Tasks (SST) via the smartphone application CheckCog. Within-person reliability of these measures was explored.

RESULTS

Compliance ranged from 66% (for eight sessions) to 89% (for four sessions). Our results reveal significant changes in performance within the first two sessions for both the n-Back and SST, with performance remaining largely consistent across the remaining (two to eight) sessions. In terms of test-retest reliability, the ICC (C, 1) values ranged from .29 to .68 on the n-Back (with overall accuracy being .51) and .31-.73 on the SST (with stop-signal reaction time being .53).

CONCLUSION

The results of the current study contribute to the literature by demonstrating the reliability of brief measures of executive function - in the form of inhibitory control and WM - delivered using smartphones in participants' natural environments. Based on our findings, the CheckCog app reliability tracks baseline systematic changes in WM and response inhibition across multiple time points and for an extended period in healthy individuals.

Reliable, rapid, and remote measurement of metacognitive bias

Metacognitive biases have been repeatedly associated with transdiagnostic psychiatric dimensions of 'anxious-depression' and 'compulsivity and intrusive thought', cross-sectionally. To progress our understanding of the underlying neurocognitive mechanisms, new methods are required to measure metacognition remotely, within individuals over time. We developed a gamified smartphone task designed to measure visuo-perceptual metacognitive (confidence) bias and investigated its psychometric properties across two studies (N = 3410 unpaid citizen scientists, N = 52 paid participants). We assessed convergent validity, split-half and test-retest reliability, and identified the minimum number of trials required to capture its clinical correlates. Convergent validity of metacognitive bias was moderate (r(50) = 0.64, p < 0.001) and it demonstrated excellent split-half reliability (r(50) = 0.91, p < 0.001). Anxious-depression was associated with decreased confidence (β =  - 0.23, SE = 0.02, p < 0.001), while compulsivity and intrusive thought was associated with greater confidence (β = 0.07, SE = 0.02, p < 0.001). The associations between metacognitive biases and transdiagnostic psychiatry dimensions are evident in as few as 40 trials. Metacognitive biases in decision-making are stable within and across sessions, exhibiting very high test-retest reliability for the 100-trial (ICC = 0.86, N = 110) and 40-trial (ICC = 0.86, N = 120) versions of Meta Mind. Hybrid 'self-report cognition' tasks may be one way to bridge the recently discussed reliability gap in computational psychiatry.

Different deployments of attentional breadth selectively predict UFOV task performance in older adults

The Useful Field of View task (UFOV) is a strong and reliable predictor of crash risk in older drivers. However, while the functional domain of attention is clearly implicated in UFOV performance, the potential role of one specific attentional process remains unclear: attentional breadth (the spatial extent of the attended region around the point of visual fixation). The goal of the present study was to systematically test the role of two distinct aspects of attentional breadth, maintaining a specific breadth of attention and resizing the attended region, in UFOV performance. To this end, 135 older adults completed the UFOV and modified Navon tasks to measure their efficiency in maintaining, contracting, and expanding the breadth of attention. We then examined individual-difference associations between these aspects of attentional breadth deployment and UFOV performance. We found that performance on UFOV Subtask 2 was associated with efficient contraction of attentional breadth (i.e., resizing the attended region to a smaller area), while Subtask 3 performance was associated with the efficiency of expanding attentional breadth (i.e., resizing the attended region to a larger area). The selectivity of these relationships appears to implicate these specific deployments of attentional breadth in how people complete the task, as it suggests that these relationships are not simply attributable to shared variance in a broader domain of cognitive functioning. The implications of these results for our understanding of UFOV, as well as future research directions that test the relative contributions of different cognitive processes in predicting task performance, are discussed.

Reliability of task-based fMRI in the dorsal horn of the human spinal cord

The application of functional magnetic resonance imaging (fMRI) to the human spinal cord is still a relatively small field of research and faces many challenges. Here we aimed to probe the limitations of task-based spinal fMRI at 3T by investigating the reliability of spinal cord blood oxygen level dependent (BOLD) responses to repeated nociceptive stimulation across two consecutive days in 40 healthy volunteers. We assessed the test-retest reliability of subjective ratings, autonomic responses, and spinal cord BOLD responses to short heat pain stimuli (1s duration) using the intraclass correlation coefficient (ICC). At the group level, we observed robust autonomic responses as well as spatially specific spinal cord BOLD responses at the expected location, but no spatial overlap in BOLD response patterns across days. While autonomic indicators of pain processing showed good-to-excellent reliability, both β-estimates and z-scores of task-related BOLD responses showed poor reliability across days in the target region (gray matter of the ipsilateral dorsal horn). When taking into account the sensitivity of gradient-echo echo planar imaging (GE-EPI) to draining vein signals by including the venous plexus in the analysis, we observed BOLD responses with fair reliability across days. Taken together, these results demonstrate that heat pain stimuli as short as one second are able to evoke a robust and spatially specific BOLD response, which is however strongly variable within participants across time, resulting in low reliability in the dorsal horn gray matter. Further improvements in data acquisition and analysis techniques are thus necessary before event-related spinal cord fMRI as used here can be reliably employed in longitudinal designs or clinical settings.

Neurocomputational Mechanisms Underlying Differential Reinforcement Learning From Wins and Losses in Obesity With and Without Binge Eating

BACKGROUND

Binge-eating disorder (BED) is thought of as a disorder of cognitive control, but evidence regarding its neurocognitive mechanisms is inconclusive. Key limitations of previous research include a lack of consistent separation between effects of BED and obesity and a disregard for self-report evidence suggesting that neurocognitive alterations may emerge primarily in loss- or harm-avoidance contexts.

METHODS

To address these gaps, in this longitudinal study we investigated behavioral flexibility and its underlying neurocomputational processes in reward-seeking and loss-avoidance contexts. Obese participants with BED, obese participants without BED, and healthy normal-weight participants (n = 96) performed a probabilistic reversal learning task during functional imaging, with different blocks focused on obtaining wins or avoiding losses. They were reinvited for a 6-month follow-up assessment.

RESULTS

Analyses informed by computational models of reinforcement learning showed that unlike obese participants with BED, obese participants without BED performed worse in the win than in the loss condition. Computationally, this was explained by differential learning sensitivities in the win versus loss conditions in the groups. In the brain, this was echoed in differential neural learning signals in the ventromedial prefrontal cortex per condition. The differences were subtle but scaled with BED symptoms, such that more severe BED symptoms were associated with increasing bias toward improved learning from wins versus losses. Across conditions, obese participants with BED switched more between choice options than healthy normal-weight participants. This was reflected in diminished representation of choice certainty in the ventromedial prefrontal cortex.

CONCLUSIONS

Our study highlights the importance of distinguishing between obesity with and without BED to identify unique neurocomputational alterations underlying different styles of maladaptive eating behavior.

Impulsivity is a stable, measurable, and predictive psychological trait

Impulsivity is a personality construct frequently employed to explain and predict important human behaviors. Major inconsistencies in its definition and measurement, however, have led some researchers to call for an outright rejection of impulsivity as a psychological construct. We address this highly unsatisfactory state with a large-scale, preregistered study (N = 1,676) in which each participant completed 48 measures of impulsivity derived from 10 self-report scales and 10 behavioral tasks and reported frequencies of seven impulsivity-related behaviors (e.g., impulsive buying and social media usage); a subsample (N = 196) then completed a retest session 3 mo later. We found that correlations between self-report measures were substantially higher than those between behavioral tasks and between self-report measures and behavioral tasks. Bifactor analysis of these measures exacted one general factor of impulsivity I, akin to the general intelligence factor g, and six specific factors. Factor I was related mainly to self-report measures, had high test-retest reliability, and could predict impulsivity-related behaviors better than existing measures. We further developed a scale named the adjustable impulsivity scale (AIMS) to measure I. AIMS possesses excellent psychometric properties that are largely retained in shorter versions and could predict impulsivity-related behaviors equally well as I. These findings collectively support impulsivity as a stable, measurable, and predictive trait, indicating that it may be too early to reject it as a valid and useful psychological construct. The bifactorial structure of impulsivity and AIMS, meanwhile, significantly advance the conceptualization and measurement of construct impulsivity.

On the (un)reliability of common behavioral and electrophysiological measures from the stop signal task: Measures of inhibition lack stability over time

Response inhibition, the intentional stopping of planned or initiated actions, is often considered a key facet of control, impulsivity, and self-regulation. The stop signal task is argued to be the purest inhibition task we have, and it is thus central to much work investigating the role of inhibition in areas like development and psychopathology. Most of this work quantifies stopping behavior by calculating the stop signal reaction time as a measure of individual stopping latency. Individual difference studies aiming to investigate why and how stopping latencies differ between people often do this under the assumption that the stop signal reaction time indexes a stable, dispositional trait. However, empirical support for this assumption is lacking, as common measures of inhibition and control tend to show low test-retest reliability and thus appear unstable over time. The reasons for this could be methodological, where low stability is driven by measurement noise, or substantive, where low stability is driven by a larger influence of state-like and situational factors. To investigate this, we characterized the split-half and test-retest reliability of a range of common behavioral and electrophysiological measures derived from the stop signal task. Across three independent studies, different measurement modalities, and a systematic review of the literature, we found a pattern of low temporal stability for inhibition measures and higher stability for measures of manifest behavior and non-inhibitory processing. This pattern could not be explained by measurement noise and low internal consistency. Consequently, response inhibition appears to have mostly state-like and situational determinants, and there is little support for the validity of conceptualizing common inhibition measures as reflecting stable traits.

A novel biomarker of interleukin 6 activity and clinical and cognitive outcomes in depression

BACKGROUND

Inflammatory cytokines like interleukin-6 (IL-6) are implicated in depression, but most studies have hitherto focused on circulating levels of IL-6 rather than its activity. IL-6 trans-signalling is thought to be responsible for most of the pathogenic effects of IL-6 and is implicated in autoimmune diseases like rheumatoid arthritis. We tested the association between a multi-protein-derived measure of IL-6 trans-signalling and clinical and cognitive outcomes in patients with depression. We hypothesised that this novel measure of IL-6 activity/bioavailability would be associated with clinical and cognitive measures previously reported to be associated with inflammation in depression.

METHODS

Using data from 86 patients with International Classification of Diseases-10 diagnosis of depression, we calculated a ratio score representing IL-6 activity/bioavailability using serum IL-6, soluble IL-6 receptor (sIL-6R), and soluble glycoprotein 130 levels. We tested the relationship of this novel biomarker with 12 cytokines using correlation analyses and with cognitive and clinical measures using multivariable linear regression, following z-transformation of all immune exposures.

RESULTS

The novel measure of IL-6 activity/bioavailability was correlated with IL-6 (r=0.42, P=0.03), C-reactive protein (CRP) (r=0.42, P=0.03), sIL-6R (r=0.91, P<0.01), and tumour necrosis factor alpha (r=0.43, P=0.03). The IL-6 activity/bioavailability measure was associated with higher somatic symptoms of depression (β=1.09; 95% CI 0.30, 1.88; PFDR=0.01), fatigue (β=4.34; 95% CI 1.26, 7.42; PFDR=0.03), depression severity (β=3.06; 95% CI 0.71, 5.40; P=0.02), poorer quality of life (β=-0.07; 95% CI -0.13, -0.01; PFDR=0.045), and decreased psychomotor speed (β=-5.46; 95% CI -9.09, -1.84; PFDR=0.01),. There was little evidence of associations with reaction time, anhedonia, anxiety, emotional perception and recall, executive function, and sustained attention (Ps>0.05). The effect estimates for the associations of the novel measure with depression outcomes were comparable to those for individual immune proteins (i.e., IL-6, CRP, sIL-6R).

CONCLUSION

A novel multi-protein-derived measure of IL-6 activity/bioavailability shows robust associations with various inflammation-related clinical and cognitive outcomes in depression and performs well in comparison to single inflammatory proteins. We need replication of these findings in other samples, experiments for mechanistic validity of this novel biomarker, and clinical studies to assess its usefulness as a marker of illness risk and prognosis.

Covert and overt automatic imitation are correlated

Most theoretical accounts of imitation assume that covert and overt measures of automatic imitation tap into the same underlying construct. Despite this widespread assumption, it is not well supported by empirical evidence. In fact, the only study investigating the relation between covert and overt automatic imitation failed to find a correlation between them (Genschow et al., 2017, PLOS ONE, 12[9], Article e0183784). However, because overt and covert imitation were measured using two very different tasks, and because the measure of overt imitation was found to be unreliable, it is still not clear whether a correlation between both measures exists. Here, we address this question by reanalyzing the results of a previous virtual reality study in which automatic imitation was indexed with an overt and covert measure of gaze following, both obtained within one and the same task (Cracco et al., 2022, IScience, Article 104891). The results show that, in this situation, both types of imitation do correlate. As such, our results provide support for the idea that overt and covert measures of automatic imitation measure the same underlying construct.