Measurement error and the replication crisis

Estimating the Reliability and Stability of Cognitive Processes Contributing to Responses on the Implicit Association Test

Implicit measures were initially assumed to assess stable individual differences, but other perspectives posit that they reflect context-dependent processes. This pre-registered research investigates whether the processes contributing to responses on the race Implicit Association Test are temporally stable and reliably measured using multinomial processing tree modeling. We applied two models-the Quad model and the Process Dissociation Procedure-to six datasets (N = 2,036), each collected over two occasions, examined the within-measurement reliability and between-measurement stability of model parameters, and meta-analyzed the results. Parameters reflecting accuracy-oriented processes demonstrate adequate stability and reliability, which suggests these processes are relatively stable within individuals. Parameters reflecting evaluative associations demonstrate poor stability but modest reliability, which suggests that associations are either context-dependent or stable but noisily measured. These findings suggest that processes contributing to racial bias on implicit measures differ in temporal stability, which has practical implications for predicting behavior using the Implicit Association Test.

A registered report of a two-site study of variations of the flanker task: ERN experimental effects and data quality

Error-related negativity is a widely used measure of error monitoring, and many projects are independently moving ERN recorded during a flanker task toward standardization, optimization, and eventual clinical application. However, each project uses a different version of the flanker task and tacitly assumes ERN is functionally equivalent across each version. The routine neglect of a rigorous test of this assumption undermines efforts to integrate ERN findings across tasks, optimize and standardize ERN assessment, and widely apply ERN in clinical trials. The purpose of this registered report was to determine whether ERN shows similar experimental effects (correct vs. error trials) and data quality (intraindividual variability) during three commonly used versions of a flanker task. ERN was recorded from 172 participants during three versions of a flanker task across two study sites. ERN scores showed numerical differences between tasks, raising questions about the comparability of ERN findings across studies and tasks. Although ERN scores from all three versions of the flanker task yielded high data quality and internal consistency, one version did outperform the other two in terms of the size of experimental effects and the data quality. Exploratory analyses of the error positivity (Pe) provided tentative support for the other two versions of the task over the paradigm that appeared optimal for ERN. The present study provides a roadmap for how to statistically compare psychometric characteristics of ERP scores across paradigms and gives preliminary recommendations for flanker tasks to use for ERN- and Pe-focused studies.

Power to Detect What? Considerations for Planning and Evaluating Sample Size

ACADEMIC ABSTRACT

In the wake of the replication crisis, social and personality psychologists have increased attention to power analysis and the adequacy of sample sizes. In this article, we analyze current controversies in this area, including choosing effect sizes, why and whether power analyses should be conducted on already-collected data, how to mitigate the negative effects of sample size criteria on specific kinds of research, and which power criterion to use. For novel research questions, we advocate that researchers base sample sizes on effects that are likely to be cost-effective for other people to implement (in applied settings) or to study (in basic research settings), given the limitations of interest-based minimums or field-wide effect sizes. We discuss two alternatives to power analysis, precision analysis and sequential analysis, and end with recommendations for improving the practices of researchers, reviewers, and journal editors in social-personality psychology.

PUBLIC ABSTRACT

Recently, social-personality psychology has been criticized for basing some of its conclusions on studies with low numbers of participants. As a result, power analysis, a mathematical way to ensure that a study has enough participants to reliably "detect" a given size of psychological effect, has become popular. This article describes power analysis and discusses some controversies about it, including how researchers should derive assumptions about effect size, and how the requirements of power analysis can be applied without harming research on hard-to-reach and marginalized communities. For novel research questions, we advocate that researchers base sample sizes on effects that are likely to be cost-effective for other people to implement (in applied settings) or to study (in basic research settings). We discuss two alternatives to power analysis, precision analysis and sequential analysis, and end with recommendations for improving the practices of researchers, reviewers, and journal editors in social-personality psychology.

Comparing the DSM-5 Dimensional Trait and Triarchic Model Conceptions of Psychopathy: An External Validity Analysis

In the DSM-5 Alternative Model of Personality Disorders (AMPD), psychopathy is marked by the presence of attention seeking, low anxiousness, and lack of social withdrawal, along with traits from the domains of Antagonism and Disinhibition. The triarchic model of psychopathy (TriPM) posits three biobehaviorally based traits underlying it: disinhibition, meanness, and boldness. The current study directly compared relations for measures of the two models with the broad dimensions of externalizing, internalizing, and positive adjustment. Participants (1,678 adults) were surveyed regarding maladaptive personality traits, clinical symptoms, and positive adjustment features. The TriPM model explained more variance than the AMPD in substance use, positive adjustment, and empathy, whereas the AMPD model explained more variance in internalizing symptoms. In addition, AMPD Antagonism and the Psychopathy Specifier diverged from TriPM Meanness and Boldness in their associations with some specific outcomes. Overall, our study provides evidence for complementarity of the two models in characterizing the multifaceted nature of psychopathy.

Practical Implications of Sum Scores Being Psychometrics' Greatest Accomplishment

This paper reflects on some practical implications of the excellent treatment of sum scoring and classical test theory (CTT) by Sijtsma et al. (Psychometrika 89(1):84-117, 2024). I have no major disagreements about the content they present and found it to be an informative clarification of the properties and possible extensions of CTT. In this paper, I focus on whether sum scores-despite their mathematical justification-are positioned to improve psychometric practice in empirical studies in psychology, education, and adjacent areas. First, I summarize recent reviews of psychometric practice in empirical studies, subsequent calls for greater psychometric transparency and validity, and how sum scores may or may not be positioned to adhere to such calls. Second, I consider limitations of sum scores for prediction, especially in the presence of common features like ordinal or Likert response scales, multidimensional constructs, and moderated or heterogeneous associations. Third, I review previous research outlining potential limitations of using sum scores as outcomes in subsequent analyses where rank ordering is not always sufficient to successfully characterize group differences or change over time. Fourth, I cover potential challenges for providing validity evidence for whether sum scores represent a single construct, particularly if one wishes to maintain minimal CTT assumptions. I conclude with thoughts about whether sum scores-even if mathematically justified-are positioned to improve psychometric practice in empirical studies.

Open instrumentation, like open data, is key to reproducible science. Yet, without incentives it won't thrive

There is far more to open science than simply not shutting away your work. For example, a significant time investment must be made to collate, curate, explain and document precisely how an experiment can be reproduced or data can be reused. This time investment is currently poorly rewarded in our current model of open science where a bare minimum of openness is mandated, but further work is not recognized. As the open science movement looks beyond open access publications and open data towards ongoing detailed work such as open source software and open source hardware, it needs to consider how to properly encourage the extra work that is needed to properly document these projects. Without detailed documentation, the work cannot be replicated, reused and continually improved. If the work cannot be replicated or reused, is it really even open? This article is part of the Theo Murphy meeting issue 'Open, reproducible hardware for microscopy'.

Role of artificial intelligence in brain tumour imaging

Artificial intelligence (AI) is a rapidly evolving field with many neuro-oncology applications. In this review, we discuss how AI can assist in brain tumour imaging, focusing on machine learning (ML) and deep learning (DL) techniques. We describe how AI can help in lesion detection, differential diagnosis, anatomic segmentation, molecular marker identification, prognostication, and pseudo-progression evaluation. We also cover AI applications in non-glioma brain tumours, such as brain metastasis, posterior fossa, and pituitary tumours. We highlight the challenges and limitations of AI implementation in radiology, such as data quality, standardization, and integration. Based on the findings in the aforementioned areas, we conclude that AI can potentially improve the diagnosis and treatment of brain tumours and provide a path towards personalized medicine and better patient outcomes.

High-Intensity Aphasia Intervention Is Minimally Fatiguing in Chronic Aphasia: An Analysis of Participant Self-Ratings From a Large Randomized Controlled Trial

BACKGROUND

High-intensity therapy is recommended in current treatment guidelines for chronic poststroke aphasia. Yet, little is known about fatigue levels induced by treatment, which could interfere with rehabilitation outcomes. We analyzed fatigue experienced by people with chronic aphasia (>6 months) during high-dose interventions at 2 intensities.

METHODS

A retrospective observational analysis was conducted on self-rated fatigue levels of people with chronic aphasia (N=173) collected during a previously published large randomized controlled trial of 2 treatments: constraint-induced aphasia therapy plus and multi-modality aphasia therapy. Interventions were administered at a higher intensity (30 hours over 2 weeks) or lower intensity (30 hours over 5 weeks). Participants rated their fatigue on an 11-point scale before and after each day of therapy. Data were analyzed using Bayesian ordinal multilevel models. Specifically, we considered changes in self-rated participant fatigue across a therapy day and over the intervention period.

RESULTS

Data from 144 participants was analyzed. Participants were English speakers from Australia or New Zealand (mean age, 62 [range, 18-88] years) with 102 men and 42 women. Most had mild (n=115) or moderate (n=52) poststroke aphasia. Median ratings of the level of fatigue by people with aphasia were low (1 on a 0-10-point scale) at the beginning of the day. Ratings increased slightly (+1.0) each day after intervention, with marginally lower increases in the lower intensity schedule. There was no evidence of accumulating fatigue over the 2- or 5-week interventions.

CONCLUSIONS

Findings suggest that intensive intervention was not associated with large increases in fatigue for people with chronic aphasia enrolled in the COMPARE trial (Constraint-Induced or Multimodality Personalised Aphasia Rehabilitation). Fatigue did not change across the course of the intervention. This study provides evidence that intensive treatment was minimally fatiguing for stroke survivors with chronic aphasia, suggesting that fatigue is not a barrier to high-intensity treatment.

Lifetime familiarity cue effects for autobiographical memory

Recollecting an autobiographical memory requires a cue to initiate processes related to accessing and then elaborating on a past personal experience. Prior work has shown that the familiarity of a cue can influence the autobiographical memory retrieval process. Extending this work, we tested how familiarity accrued from cumulative lifetime exposures associated with the cue-as well as associated semantic knowledge-can affect how we access and remember autobiographical memories. In Experiment 1, we measured reaction times to access and report memories in response to cue words. In Experiment 2 we examined the details with which participants described memories in response to cue words. For both experiments, participants provided estimates of lifetime exposure and semantic knowledge for each cue. In Experiment 1, we found a cue's lifetime exposure, independent of semantic knowledge, led to quicker memory access. In Experiment 2, we found the lifetime exposure and semantic knowledge of a cue interactively affected the specificity of a described autobiographical memory. These results provide new evidence that the amount of lifetime exposure associated with a cue, both independently and interactively with semantic knowledge, influences how autobiographical memories are accessed and described.Clinical trial This was not a clinical trial.Trial Registration Number (TRN) N/A.

A proteomic signature of healthspan

The focus of aging research has shifted from increasing lifespan to enhancing healthspan to reduce the time spent living with disability. Despite significant efforts to develop biomarkers of aging, few studies have focused on biomarkers of healthspan. We developed a proteomics-based signature of healthspan (healthspan proteomic score (HPS)) using data from the UK Biobank Pharma Proteomics Project (53,018 individuals and 2920 proteins). A lower HPS was associated with higher mortality risk and several age-related conditions, such as COPD, diabetes, heart failure, cancer, myocardial infarction, dementia, and stroke. HPS showed superior predictive accuracy for these outcomes compared to chronological age and biological age measures. Proteins associated with HPS were enriched in hallmark pathways such as immune response, inflammation, cellular signaling, and metabolic regulation. Our findings demonstrate the validity of HPS, making it a valuable tool for assessing healthspan and as a potential surrogate marker in geroscience-guided studies.

Does adjustment for non-differential misclassification of dichotomous exposure induce positive bias if there is no true association?

This article is a response to an off-the-record discussion that I had at an international meeting of epidemiologists more than decade ago. It centered on a concern, perhaps widely spread, that adjustment for exposure misclassification can induce a false positive result. I trace the possible history of this supposition and test it in a simulated case-control study under the assumption of non-differential misclassification of binary exposure, in which a Bayesian adjustment is applied. Probabilistic bias analysis is also briefly considered. The main conclusion is that adjustment for the presumed non-differential exposure misclassification of dichotomous does not "induce" positive associations, especially if the focus of the interpretation of the result is taken away from the point estimate. The misconception about positive bias induced by adjustment for exposure misclassification, if more clearly explained during the training of epidemiologists, may promote appropriate (and wider) use of the adjustment techniques. The simple message that can be derived from this paper is: "Exposure misclassification as a tractable problem that deserves much more attention than just a typical qualitative throw-away discussion".

Brief online negative affect focused functional imagery training (FIT) improves four-week drinking outcomes in hazardous student drinkers: A pilot randomised controlled trial replication in South Africa

Background

Previous study has shown that functional imagery training (FIT) to utilise positive mental imagery in response to negative affect could improve alcohol-related outcomes. The current study aimed to replicate whether this negative affect focused FIT would improve alcohol-related outcomes in hazardous student drinkers in South Africa at four-week follow-up.

Methods

50 hazardous student drinkers who reported drinking to cope with negative affect were randomised into two groups. The active group (n = 25) was trained online over two weeks to respond to personalised negative drinking triggers by retrieving a personalised adaptive strategy they might use to mitigate negative affect, whereas the control group (n = 25) received standard risk information about binge drinking. Outcome measures including alcohol consumption, drinking motives, anxiety and depression, self-efficacy and use of protective behavioural strategies were obtained at baseline and four-week follow-up.

Results

FIT effects were revealed by three significant group-by-timepoint interactions in a per-protocol analysis: there was a significant decrease in depressive symptoms, drinking to cope and drinking for social reasons from baseline to follow-up in the active group, but not the control group. No effects were observed on alcohol consumption, self-efficacy, protective behaviour strategies and anxiety.

Conclusions

Preliminary evidence supports that online negative affect focused FIT can improve depression as well as coping and social drinking motives in South African hazardous student drinkers who drank to cope, at four-week follow-up, suggesting that the principles of this FIT approach might be adapted and incorporated into a clinical intervention to test for efficacy in mitigating substance use problems.

Ninety-Day Stroke Recurrence in Minor Stroke: Systematic Review and Meta-Analysis of Trials and Observational Studies

BACKGROUND

Risk of recurrence after minor ischemic stroke is usually reported with transient ischemic attack. No previous meta-analysis has focused on minor ischemic stroke alone. The objective was to evaluate the pooled proportion of 90-day stroke recurrence for minor ischemic stroke, defined as a National Institutes of Health Stroke Scale severity score of ≤5.

METHODS AND RESULTS

Published papers found on PubMed from 2000 to January 12, 2021, reference lists of relevant articles, and experts in the field were involved in identifying relevant studies. Randomized controlled trials and observational studies describing minor stroke cohort with reported 90-day stroke recurrence were selected by 2 independent reviewers. Altogether 14 of 432 (3.2%) studies met inclusion criteria. Multilevel random-effects meta-analysis was performed. A total of 6 randomized controlled trials and 8 observational studies totaling 45 462 patients were included. The pooled 90-day stroke recurrence was 8.6% (95% CI, 6.5-10.7), reducing by 0.60% (95% CI, 0.09-1.1; P=0.02) with each subsequent year of publication. Recurrence was lowest in dual antiplatelet trial arms (6.3%, 95% CI, 4.5-8.0) when compared with non-dual antiplatelet trial arms (7.2%, 95% CI, 4.7-9.6) and observational studies 10.6% (95% CI, 7.0-14.2). Age, hypertension, diabetes, ischemic heart disease, or known atrial fibrillation had no significant association with outcome. Defining minor stroke with a lower National Institutes of Health Stroke Scale threshold made no difference - score ≤3: 8.6% (95% CI, 6.0-11.1), score ≤4: 8.4% (95% CI, 6.1-10.6), as did excluding studies with n<500%-7.3% (95% CI, 5.5-9.0).

CONCLUSIONS

The risk of recurrence after minor ischemic stroke is declining over time but remains important.

Computational Approaches and Observer Variation in the 3D Musculoskeletal Modeling of the Heads of Anolis

High-resolution imaging, 3D modeling, and quantitative analyses are equipping evolutionary biologists with new approaches to understanding the variation and evolution of the musculoskeletal system. However, challenges with interpreting DiceCT data and higher order use of modeled muscles have not yet been fully explored, and the error in and accuracy of some digital methods remain unclear. West Indian Anolis lizards are a model clade for exploring patterns in functional adaptation, ecomorphology, and sexual size dimorphism in vertebrates. These lizards possess numerous jaw muscles with potentially different anatomies that sculpt the adductor chamber of the skull. Here we test approaches to quantifying the musculoskeletal shape of the heads of two species of Anolis: A. pulchellus and A. sagrei. We employ comparative approaches such as DiceCT segmentation of jaw muscles, 3D surface attachment mapping, and 3D landmarking with the aim of exploring muscle volumes, 3D muscle fiber architecture, and sexual dimorphism of the skull. We then compare sources of measurement error in these 3D analyses while also presenting new 3D musculoskeletal data from the Anolis feeding apparatus. These findings demonstrate the accessibility and repeatability of these emerging techniques as well as provide details regarding the musculoskeletal anatomy of the heads of A. pulchellus and A. sagrei which show potential for further research of comparative biomechanics and evolution in the clade.

Contamination fear and attention bias variability early in the COVID-19 pandemic

The onset of the COVID-19 pandemic resulted in a dramatic increase in the salience and importance of information relating to both the risk of infection, and factors that could mitigate against such risk. This is likely to have contributed to elevated contamination fear concerns in the general population. Biased attention for contamination-related information has been proposed as a potential mechanism underlying contamination fear, though evidence regarding the presence of such biased attention has been inconsistent. A possible reason for this is that contamination fear may be characterised by variability in attention bias that has not yet been examined. The current study examined the potential association between attention bias variability for both contamination-related and mitigation-related stimuli, and contamination fear during the early stages of the COVID-19 pandemic. A final sample of 315 participants completed measures of attention bias and contamination fear. The measure of average attention bias for contamination-related stimuli and mitigation-related stimuli was not associated with contamination fear (r = 0.055 and r = 0.051, p > 0.10), though both attention bias variability measures did show a small but statistically significant relationship with contamination fear (r = 0.133, p < 0.05; r = 0.147, p < 0.01). These attention bias variability measures also accounted for significant additional variance in contamination fear above the average attention bias measure (and controlling for response time variability). These findings provide initial evidence for the association between attention bias variability and contamination fear, underscoring a potential target for cognitive bias interventions for clinical contamination fear.

Reliability of the serial reaction time task: If at first you don't succeed, try, try, try again

Procedural memory is involved in the acquisition and control of skills and habits that underlie rule and procedural learning, including the acquisition of grammar and phonology. The serial reaction time task (SRTT), commonly used to assess procedural learning, has been shown to have poor stability (test-retest reliability). We investigated factors that may affect the stability of the SRTT in adults. Experiment 1 examined whether the similarity of sequences learned in two sessions would impact stability: test-retest correlations were low regardless of sequence similarity (r < .31). Experiment 2 added a third session to examine whether individual differences in learning would stabilise with further training. There was a small (but nonsignificant) improvement in stability for later sessions (Sessions 1 and 2: r = .42; Sessions 2 and 3: r = .60). Stability of procedural learning on the SRTT remained suboptimal in all conditions, posing a serious obstacle to the use of this task as a sensitive predictor of individual differences and ultimately theoretical advance.

The psychometric upgrade psychophysiology needs

Although biological measurements are constrained by the same fundamental psychometric principles as self-report measurements, these essential principles are often neglected in most fields of neuroscience, including psychophysiology. Potential reasons for this neglect could include a lack of understanding of appropriate measurement theory or a lack of accessible software for psychometric analysis. Generalizability theory is a flexible and multifaceted measurement theory that is well suited to handling the nuances of psychophysiological data, such as the often unbalanced number of trials and intraindividual variability of scores of event-related brain potential (ERP) data. The ERP Reliability Analysis Toolbox (ERA Toolbox) was designed for psychophysiologists and is tractable software that can support the routine evaluation of psychometrics using generalizability theory. Psychometrics can guide task refinement, data-processing decisions, and selection of candidate biomarkers for clinical trials. The present review provides an extensive treatment of additional psychometric characteristics relevant to studies of psychophysiology, including validity and validation, standardization, dimensionality, and measurement invariance. Although the review focuses on ERPs, the discussion applies broadly to psychophysiological measures and beyond. The tools needed to rigorously assess psychometric reliability and validate psychophysiological measures are now readily available. With the profound implications that psychophysiological research can have on understanding brain-behavior relationships and the identification of biomarkers, there is simply too much at stake to ignore the crucial processes of evaluating psychometric reliability and validity.

Beyond single paradigms, pipelines, and outcomes: Embracing multiverse analyses in psychophysiology

Psychophysiological research is an inherently complex undertaking due to the nature of the data, and its analysis is characterized by many decision points that shape the final dataset and a study's findings. These decisions create a "multiverse" of possible outcomes, and each decision from study conceptualization to statistical analysis can lead to different results and interpretations. This review describes the concept of multiverse analyses, a methodological approach designed to understand the impact of different decisions on the robustness of a study's findings and interpretation. The emphasis is on transparently showcasing different reasonable approaches for constructing a final dataset and on highlighting the influence of various decision points, from experimental design to data processing and outcome selection. For example, the choice of an experimental task can significantly impact event-related brain potential (ERP) scores or skin conductance responses (SCRs), and different tasks might elicit unique variances in each measure. This review underscores the importance of transparently embracing the flexibility inherent in psychophysiological research and the potential consequences of not understanding the fragility or robustness of experimental findings. By navigating the intricate terrain of the psychophysiological multiverse, this review serves as an introduction, helping researchers to make informed decisions, improve the collective understanding of psychophysiological findings, and push the boundaries of the field.

Assessing Discriminant Validity through Structural Equation Modeling: The Case of Eating Compulsivity

Food addiction (FA) and disordered eating behaviors related to obesity are gaining attention in clinical and research fields. The modified Yale Food Addiction Scale 2.0 (mYFAS2.0) is the gold standard questionnaire to measure FA, while another tool is the Measure of Eating Compulsivity 10 (MEC10). Discriminant validity is present when two measures of similar but distinct constructs show a correlation that is low enough for the factors to be regarded as distinct. However, the discriminant validity of these measures has never been tested. Through a cross-sectional study design, 717 inpatients (females: 56.20%, age: 53.681 ± 12.74) with severe obesity completed the MEC10, Binge Eating Scale (BES), and mYFAS2.0. A structural equation model (SEM) was fitted, freely estimating latent correlations with 95% confidence intervals (95% CI). The results confirmed the scales' excellent psychometric properties. Importantly, latent factor correlations between MEC10 and mYFAS2.0 (est = 0.783, 95% CI [0.76, 0.80]) supported their discriminant validity. In contrast, the latent correlation of MEC10 and BES (est = 0.86, 95% CI [0.84, 0.87]) exceeded the recommended thresholds, indicating the absence of discriminant validity and suggesting a potential overlap, consistent with previous evidence. In conclusion, MEC10 demonstrates excellent psychometric properties but is more a measure of BED and not FA.

The impact of austerity on children: Uncovering effect heterogeneity by political, economic, and family factors in low- and middle-income countries

Which children are most vulnerable when their government imposes austerity? Research tends to focus on either the political-economic level or the family level. Using a sample of nearly two million children in 67 countries, this study synthesizes theories from family sociology and political science to examine the heterogeneous effects on child poverty of economic shocks following the implementation of an International Monetary Fund (IMF) program. To discover effect heterogeneity, we apply machine learning to policy evaluation. We find that children's average probability of falling into poverty increases by 14 percentage points. We find substantial effect heterogeneity, with family wealth and governments' education spending as the two most important moderators. In contrast to studies that emphasize the vulnerability of low-income families, we find that middle-class children face an equally high risk of poverty. Our results show that synthesizing family and political factors yield deeper knowledge of how economic shocks affect children.

Associations between early trajectories of amygdala development and later school-age anxiety in two longitudinal samples

Amygdala function is implicated in the pathogenesis of autism spectrum disorder (ASD) and anxiety. We investigated associations between early trajectories of amygdala growth and anxiety and ASD outcomes at school age in two longitudinal studies: high- and low-familial likelihood for ASD, Infant Brain Imaging Study (IBIS, n = 257) and typically developing (TD) community sample, Early Brain Development Study (EBDS, n = 158). Infants underwent MRI scanning at up to 3 timepoints from neonate to 24 months. Anxiety was assessed at 6-12 years. Linear multilevel modeling tested whether amygdala volume growth was associated with anxiety symptoms at school age. In the IBIS sample, children with higher anxiety showed accelerated amygdala growth from 6 to 24 months. ASD diagnosis and ASD familial likelihood were not significant predictors. In the EBDS sample, amygdala growth from birth to 24 months was associated with anxiety. More anxious children had smaller amygdala volume and slower rates of amygdala growth. We explore reasons for the contrasting results between high-familial likelihood for ASD and TD samples, grounding results in the broader literature of variable associations between early amygdala volume and later anxiety. Results have the potential to identify mechanisms linking early amygdala growth to later anxiety in certain groups.

Region of Attainable Redaction, an extension of Ellipse of Insignificance analysis for gauging impacts of data redaction in dichotomous outcome trials

In biomedical science, it is a reality that many published results do not withstand deeper investigation, and there is growing concern over a replicability crisis in science. Recently, Ellipse of Insignificance (EOI) analysis was introduced as a tool to allow researchers to gauge the robustness of reported results in dichotomous outcome design trials, giving precise deterministic values for the degree of miscoding between events and non-events tolerable simultaneously in both control and experimental arms (Grimes, 2022). While this is useful for situations where potential miscoding might transpire, it does not account for situations where apparently significant findings might result from accidental or deliberate data redaction in either the control or experimental arms of an experiment, or from missing data or systematic redaction. To address these scenarios, we introduce Region of Attainable Redaction (ROAR), a tool that extends EOI analysis to account for situations of potential data redaction. This produces a bounded cubic curve rather than an ellipse, and we outline how this can be used to identify potential redaction through an approach analogous to EOI. Applications are illustrated, and source code, including a web-based implementation that performs EOI and ROAR analysis in tandem for dichotomous outcome trials is provided.

Adapting Open Science and Pre-registration to Longitudinal Research

Open science practices, such as pre-registration and data sharing, increase transparency and may improve the replicability of developmental science. However, developmental science has lagged behind other fields in implementing open science practices. This lag may arise from unique challenges and considerations of longitudinal research. In this paper, preliminary guidelines are provided for adapting open science practices to longitudinal research to facilitate researchers' use of these practices. The guidelines propose a serial and modular approach to registration that includes an initial pre-registration of the methods and focal hypotheses of the longitudinal study, along with subsequent pre- or co-registered questions, hypotheses, and analysis plans associated with specific papers. Researchers are encouraged to share their research materials and relevant data with associated papers, and to report sufficient information for replicability. In addition, there should be careful consideration about requirements regarding the timing of data sharing, to avoid disincentivizing longitudinal research.

Better Accuracy for Better Science . . . Through Random Conclusions

Conducting research with human subjects can be difficult because of limited sample sizes and small empirical effects. We demonstrate that this problem can yield patterns of results that are practically indistinguishable from flipping a coin to determine the direction of treatment effects. We use this idea of random conclusions to establish a baseline for interpreting effect-size estimates, in turn producing more stringent thresholds for hypothesis testing and for statistical-power calculations. An examination of recent meta-analyses in psychology, neuroscience, and medicine confirms that, even if all considered effects are real, results involving small effects are indeed indistinguishable from random conclusions.

Registered replication report of the construct validity of the error-related negativity (ERN): A multi-site study of task-specific ERN correlations with internalizing and externalizing symptoms

Intact cognitive control is critical for goal-directed behavior and is widely studied using the error-related negativity (ERN). A common assumption in such studies is that ERNs recorded during different experimental paradigms reflect the same construct or functionally equivalent processes and that ERN is functionally distinct from other error-monitoring event-related brain potentials (ERPs; error positivity [Pe]), other neurophysiological indices of cognitive control (N2), and even other theoretically unrelated indices (visual N1). The present registered report represents a replication-plus-extension study of the psychometric validity of cognitive control ERPs and evaluated the convergent and divergent validity of ERN, Pe, N2, and visual N1 recorded during flanker, Stroop, and Go/no-go tasks. Data from 182 participants were collected from two study sites, and ERP psychometric reliability and validity were evaluated. Findings supported replication of convergent and divergent validity of ERN, Pe, and ΔPe (error minus correct)-these ERPs correlated more with themselves across tasks than with other ERPs measured during the same task. Convergent validity of ΔERN across tasks was not replicated, despite high internal consistency. ERN strongly correlated with N2 at levels similar or higher than those in support of convergent validity for other ERPs, and the present study failed to provide evidence of divergent validity for ERN and Pe from N2 or N1. ERN and ΔERN were unrelated to internalizing or externalizing symptoms. Findings underscore the importance of considering the psychometric validity of ERPs, as it provides a foundation for interpreting and comparing ERPs across tasks and studies.

Democrats are better than Republicans at discerning true and false news but do not have better metacognitive awareness

Insight into one's own cognitive abilities is one important aspect of metacognition. Whether this insight varies between groups when discerning true and false information has yet to be examined. We investigated whether demographics like political partisanship and age were associated with discernment ability, metacognitive efficiency, and response bias for true and false news. Participants rated the veracity of true and false news headlines and provided confidence ratings for each judgment. We found that Democrats and older adults were better at discerning true and false news than Republicans and younger adults. However, all demographic groups maintained good insight into their discernment ability. Although Republicans were less accurate than Democrats, they slightly outperformed Democrats in metacognitive efficiency when a politically equated item set was used. These results suggest that even when individuals mistake misinformation to be true, they are aware that they might be wrong.

Effects of home-cage elevation on behavioral tests in mice

BACKGROUND

Research reproducibility is a common problem in preclinical behavioral science. Mice are an important animal model for studying human behavioral disorders. Experimenters, processing methods, and rearing environments are the main causes of data variability in behavioral neuroscience. It is likely that mice adapt their behavior according to the environment outside the breeding cage. We speculated that mice housed on elevated shelves and mice housed on low shelves might have differently altered anxiety-like behavior toward heights.

PURPOSE

The purpose of this study was to investigate potential behavioral changes in mice raised at different heights for 3 weeks. Changes in behavior were examined using various experimental tests.

RESULTS

Mice housed on elevated shelves showed reduced anxiety-like behavior in a light/dark traffic test compared with mice housed on low shelves. There were no significant differences between the two groups in terms of activity, exploratory behavior, muscle strength, or depression-like behavior.

CONCLUSIONS

Our results indicate that different cage heights and corresponding light exposure may alter the anxiety-like behavior of mice in response to brightness. Researchers need to carefully control the cage height and light intensity experienced by the mice to produce reproducible test results.

Influence of respondents' Differentiation of subjective response on water knowledge stock test scale: Evaluation based on two-parameter-multidimensional IRT model

Insufficient awareness of water issues is a crucial bottleneck restricting the sustainable utilization of water resources. To accurately measure citizens' water knowledge stock and overcome the differences between scales and respondents' characteristic levels on test results, the research focuses on developing and evaluating water knowledge stock test scales. The mechanism for identifying indicators is designed based on the grounded theory, and as a result, the water knowledge stock test indicator system is derived. The data was collected by the form of survey questionnaire developed with the test indicator system. A two-parameter multidimensional item response theoretical model is constructed based on item parameter estimation, data model fitting, and item information function. The survey data and optimization model are used to optimize the water knowledge stock test scale and verify the fitting degree with the characteristics of the respondents. The test information function and standard error function indicate that the scale is most informative for individuals with characteristic levels ranging from -2 to 3, resulting in a highly reliable test effect. The research has established a measurement indicator system, methodology, and presented results that serve as a foundation for measuring the stock of water knowledge.

Problematic smartphone usage, objective smartphone engagement, and executive functions: A latent variable analysis

The negative consequences of smartphone usage have seen frequent discourse in popular media. While existing studies seek to resolve these debates in relation to executive functions, findings are still limited and mixed. This is partly due to the lack of conceptual clarity about smartphone usage, the use of self-reported measures, and problems related to task impurity. Addressing these limitations, the current study utilizes a latent variable approach to examine various types of smartphone usage, including objectively measured data-logged screen time and screen-checking, and nine executive function tasks in 260 young adults through a multi-session study. Our structural equation models showed no evidence that self-reported normative smartphone usage, objective screen time, and objective screen-checking are associated with deficits in latent factors of inhibitory control, task-switching, and working memory capacity. Only self-reported problematic smartphone usage was associated with deficits in latent factor task-switching. These findings shed light on the boundary conditions of the link between smartphone usage and executive functions and suggest that smartphone usage in moderation may not have inherent harms on cognitive functions.

The fraternal birth-order effect as a statistical artefact: convergent evidence from probability calculus, simulated data, and multiverse meta-analysis

The fraternal-birth order effect (FBOE) is a research claim which states that each older brother increases the odds of homosexual orientation in men via an immunoreactivity process known as the maternal immune hypothesis. Importantly, older sisters supposedly either do not affect these odds, or affect them to a lesser extent. Consequently, the fraternal birth-order effect predicts that the association between the number of older brothers and homosexual orientation in men is greater in magnitude than any association between the number of older sisters and homosexual orientation. This difference in magnitude represents the main theoretical estimand of the FBOE. In addition, no comparable effects should be observable among homosexual vs heterosexual women. Here, we triangulate the empirical foundations of the FBOE from three distinct, informative perspectives, complementing each other: first, drawing on basic probability calculus, we deduce mathematically that the body of statistical evidence used to make inferences about the main theoretical estimand of the FBOE rests on incorrect statistical reasoning. In particular, we show that throughout the literature researchers ascribe to the false assumptions that effects of family size should be adjusted for and that this could be achieved through the use of ratio variables. Second, using a data-simulation approach, we demonstrate that by using currently recommended statistical practices, researchers are bound to frequently draw incorrect conclusions. And third, we re-examine the empirical evidence of the fraternal birth-order effect in men and women by using a novel specification-curve and multiverse approach to meta-analysis (64 male and 17 female samples, N = 2,778,998). When analyzed correctly, the specific association between the number of older brothers and homosexual orientation is small, heterogenous in magnitude, and apparently not specific to men. In addition, existing research evidence seems to be exaggerated by small-study effects.

Replicable brain-phenotype associations require large-scale neuroimaging data

Numerous neuroimaging studies have investigated the neural basis of interindividual differences but the replicability of brain-phenotype associations remains largely unknown. We used the UK Biobank neuroimaging dataset (N = 37,447) to examine associations with six variables related to physical and mental health: age, body mass index, intelligence, memory, neuroticism and alcohol consumption, and assessed the improvement of replicability for brain-phenotype associations with increasing sampling sizes. Age may require only 300 individuals to provide highly replicable associations but other phenotypes required 1,500 to 3,900 individuals. The required sample size showed a negative power law relation with the estimated effect size. When only comparing the upper and lower quarters, the minimally required sample sizes for imaging decreased by 15-75%. Our findings demonstrate that large-scale neuroimaging data are required for replicable brain-phenotype associations, that this can be mitigated by preselection of individuals and that small-scale studies may have reported false positive findings.

Motoric cognitive risk syndrome trajectories and incident dementia over 10 years

Background

Motoric Cognitive Risk (MCR) syndrome is a high-risk state for adverse health outcomes in older adults characterised by measured slow gait speed and self-reported cognitive complaints. The recent addition to the Lothian Birth Cohort 1936 of robust dementia outcomes enabled us to assess the prognostic value of MCR for dementia and explore the various trajectories of participants diagnosed with MCR.

Methods

We classified 680 community-dwelling participants free from dementia into non-MCR or MCR groups at mean [SD] age 76.3 [0.8] years. We used Cox and competing risk regression methods, adjusted for potential confounders, to evaluate the risk of developing all-cause incident dementia over 10 years of follow-up. Secondarily, we followed the trajectories for individuals with and without MCR at baseline and categorised them into subgroups based on whether MCR was still present at the next research wave, three years later.

Results

The presence of MCR increased the risk of incident dementia (adjusted HR 2.34, 95%CI 1.14-4.78, p = 0.020), as did fewer years of education and higher depression symptoms. However, MCR has a heterogenous progression trajectory. The MCR progression subgroups each have different prognostic values for incident dementia.

Conclusion

MCR showed similar prognostic ability for dementia in a Scottish cohort as for other populations. MCR could identify a target group for early interventions of modifiable risk factors to prevent incident dementia. This study illustrates the heterogeneous nature of MCR progression. Exploring the underlying reasons will be important work in future work.

Reproductive phenology is a repeatable, heritable trait linked to the timing of other life-history events in a migratory marine predator

Population-level shifts in reproductive phenology in response to environmental change are common, but whether individual-level responses are modified by demographic and genetic factors remains less well understood. We used mixed models to quantify how reproductive timing varied across 1772 female southern elephant seals (Mirounga leonina) breeding at Marion Island in the Southern Ocean (1989-2019), and to identify the factors that correlate with phenological shifts within and between individuals. We found strong support for covariation in the timing of breeding arrival dates and the timing of the preceding moult. Breeding arrival dates were more repeatable at the individual level, as compared with the population level, even after accounting for individual traits (wean date as a pup, age and breeding experience) associated with phenological variability. Mother-daughter similarities in breeding phenology were also evident, indicating that additive genetic effects may contribute to between-individual variation in breeding phenology. Over 30 years, elephant seal phenology did not change towards earlier or later dates, and we found no correlation between annual fluctuations in phenology and indices of environmental variation. Our results show how maternal genetic (or non-genetic) effects, individual traits and linkages between cyclical life-history events can drive within- and between-individual variation in reproductive phenology.

The replication crisis has led to positive structural, procedural, and community changes

The emergence of large-scale replication projects yielding successful rates substantially lower than expected caused the behavioural, cognitive, and social sciences to experience a so-called 'replication crisis'. In this Perspective, we reframe this 'crisis' through the lens of a credibility revolution, focusing on positive structural, procedural and community-driven changes. Second, we outline a path to expand ongoing advances and improvements. The credibility revolution has been an impetus to several substantive changes which will have a positive, long-term impact on our research environment.

Feasibility, Acceptability, and Preliminary Efficacy of a Positive Affect Skills Intervention for Adults With Fibromyalgia

Background and Objectives

To examine the feasibility, acceptability, and preliminary efficacy of a positive affect skills intervention for middle-aged and older adults with fibromyalgia syndrome (FMS).

Research Design and Methods

Ninety-five participants with FMS aged 50 and older (94% female) were randomized to 1 of 2 conditions: (a) Lessons in Affect Regulation to Keep Stress and Pain UndeR control (LARKSPUR; n = 49) or (b) emotion reporting/control (n = 46). LARKSPUR included 5 weeks of skill training that targeted 8 skills to help foster positive affect, including (a) noticing positive events, (b) savoring positive events, (c) identifying personal strengths, (d) behavioral activation to set and work toward attainable goals, (e) mindfulness, (f) positive reappraisal, (g) gratitude, and (h) acts of kindness. Outcome data were collected via online surveys at baseline, postintervention, and 1-month follow-up.

Results

Completion rates (88%) and satisfaction ratings (10-point scale) were high (LARKSPUR: M = 9.14, standard deviation (SD) = 1.49; control: M = 8.59, SD = 1.97). Improvements were greater in LARKSPUR participants compared with control participants on measures of positive affect (Cohen's d = 0.19 [0.15, 0.24]), negative affect (Cohen's d = -0.07 [-0.11, -0.02]), and pain catastrophizing (Cohen's d = -0.14 [-0.23, -0.05]). Improvements in positive affect (Cohen's d = 0.17 [0.13, 0.22]) and negative affect (Cohen's d = -0.11 [-0.15, -0.06]) were maintained at 1-month follow-up. Dose-response analyses indicated that intervention engagement significantly predicted pre-to-post and post-to-follow-up reductions in pain catastrophizing.

Discussion and Implications

The current preliminary findings add to existing literature and highlight the specific potential of internet-delivered positive affect skills programs for adults with FMS.

Clinical Trial Registration

NCT04869345.

Automated machine learning (AutoML) can predict 90-day mortality after gastrectomy for cancer

Early postoperative mortality risk prediction is crucial for clinical management of gastric cancer. This study aims to predict 90-day mortality in gastric cancer patients undergoing gastrectomy using automated machine learning (AutoML), optimize models for preoperative prediction, and identify factors influential in prediction. National Cancer Database was used to identify stage I-III gastric cancer patients undergoing gastrectomy between 2004 and 2016. 26 features were used to train predictive models using H2O.ai AutoML. Performance on validation cohort was measured. In 39,108 patients, 90-day mortality rate was 8.8%. The highest performing model was an ensemble (AUC = 0.77); older age, nodal ratio, and length of inpatient stay (LOS) following surgery were most influential for prediction. Removing the latter two parameters decreased model performance (AUC 0.71). For optimizing models for preoperative use, models were developed to first predict node ratio or LOS, and these predicted values were inputted for 90-day mortality prediction (AUC of 0.73-0.74). AutoML performed well in predicting 90-day mortality in a larger cohort of gastric cancer patients that underwent gastrectomy. These models can be implemented preoperatively to inform prognostication and patient selection for surgery. Our study supports broader evaluation and application of AutoML to guide surgical oncologic care.

The reliability of the serial reaction time task: meta-analysis of test-retest correlations

The Serial Reaction Time task, one of the most widely used tasks to index procedural memory, has been increasingly employed in individual differences research examining the role of procedural memory in language and other cognitive abilities. Yet, despite consistently producing robust procedural learning effects at the group level (i.e. faster responses to sequenced/probable trials versus random/improbable trials), these effects have recently been found to have poor reliability. In this meta-analysis (N = 7), comprising 719 participants (M = 20.81, s.d. = 7.13), we confirm this 'reliability paradox'. The overall retest reliability of the robust procedural learning effect elicited by the SRTT was found to be well below acceptable psychometric standards (r < 0.40). However, split-half reliability within a session is better, with an overall estimate of 0.66. There were no significant effects of sampling (participants' age), methodology (e.g. number of trials, sequence type) and analytical decisions (whether all trials were included when computing the procedural learning scores; using different indexes of procedural learning). Thus, despite producing robust effects at the group-level, until we have a better understanding of the factors that improve the reliability of this task using the SRTT for individual differences research should be done with caution.

Problem Solving and Emotion Coping Styles for Social Anxiety: A Meta-analysis of Chinese Mainland Students

Studies of how positive and negative coping styles affect social anxiety show mixed results. Hence, our two meta-analyses determined the overall effect sizes of problem solving-focused coping (PSC) styles and emotion-focused coping (EFC) styles on social anxiety in mainland China (PSC: k = 49 studies, N = 34,669; EFC: k = 52, N = 36,531). PSC was negatively linked to social anxiety (- .198), and EFC was positively linked to social anxiety (.223). In years with more national income, PSC's and EFC's effect sizes were larger. PSC's effect sizes were smaller among rural students (vs. urban students), larger among older students (university, high school, middle school), and larger in cross-sectional (vs. longitudinal) studies. When using SAD (vs. others) social anxiety measures, PSC effect sizes were larger, but EFC effect sizes were smaller. EFC effect sizes were larger in studies with convenience (vs. representative) samples. Gender, single child status, and coping style measurement showed no moderation effects. These findings suggest that using problem solving-focused coping styles rather than emotion-focused may reduce social anxiety, so future experimental studies can test this idea more rigorously.

Charting infants' motor development at home using a wearable system: validation and comparison to physical growth charts

BACKGROUND

Early neurodevelopmental care and research are in urgent need of practical methods for quantitative assessment of early motor development. Here, performance of a wearable system in early motor assessment was validated and compared to developmental tracking of physical growth charts.

METHODS

Altogether 1358 h of spontaneous movement during 226 recording sessions in 116 infants (age 4-19 months) were analysed using a multisensor wearable system. A deep learning-based automatic pipeline quantified categories of infants' postures and movements at a time scale of seconds. Results from an archived cohort (dataset 1, N = 55 infants) recorded under partial supervision were compared to a validation cohort (dataset 2, N = 61) recorded at infants' homes by the parents. Aggregated recording-level measures including developmental age prediction (DAP) were used for comparison between cohorts. The motor growth was also compared with respective DAP estimates based on physical growth data (length, weight, and head circumference) obtained from a large cohort (N = 17,838 infants; age 4-18 months).

FINDINGS

Age-specific distributions of posture and movement categories were highly similar between infant cohorts. The DAP scores correlated tightly with age, explaining 97-99% (94-99% CI 95) of the variance at the group average level, and 80-82% (72-88%) of the variance in the individual recordings. Both the average motor and the physical growth measures showed a very strong fit to their respective developmental models (R² = 0.99). However, single measurements showed more modality-dependent variation that was lowest for motor (σ = 1.4 [1.3-1.5 CI 95] months), length (σ = 1.5 months), and combined physical (σ = 1.5 months) measurements, and it was clearly higher for the weight (σ = 1.9 months) and head circumference (σ = 1.9 months) measurements. Longitudinal tracking showed clear individual trajectories, and its accuracy was comparable between motor and physical measures with longer measurement intervals.

INTERPRETATION

A quantified, transparent and explainable assessment of infants' motor performance is possible with a fully automated analysis pipeline, and the results replicate across independent cohorts from out-of-hospital recordings. A holistic assessment of motor development provides an accuracy that is comparable with the conventional physical growth measures. A quantitative measure of infants' motor development may directly support individual diagnostics and care, as well as facilitate clinical research as an outcome measure in early intervention trials.

FUNDING

This work was supported by the Finnish Academy (314602, 335788, 335872, 332017, 343498), Finnish Pediatric Foundation (Lastentautiensäätiö), Aivosäätiö, Sigrid Jusélius Foundation, and HUS Children's Hospital/HUS diagnostic center research funds.

Intermittent theta burst stimulation to the left dorsolateral prefrontal cortex improves working memory of subjects with methamphetamine use disorder

BACKGROUND

Repetitive transcranial magnetic stimulation has been employed to treat drug dependence, reduce drug use and improve cognition. The aim of the study was to analyze the effectiveness of intermittent theta-burst stimulation (iTBS) on cognition in individuals with methamphetamine use disorder (MUD).

METHODS

This was a secondary analysis of 40 MUD subjects receiving left dorsolateral prefrontal cortex (L-DLPFC) iTBS or sham iTBS for 20 times over 10 days (twice-daily). Changes in working memory (WM) accuracy, reaction time, and sensitivity index were analyzed before and after active and sham rTMS treatment. Resting-state EEG was also acquired to identify potential biological changes that may relate to any cognitive improvement.

RESULTS

The results showed that iTBS increased WM accuracy and discrimination ability, and improved reaction time relative to sham iTBS. iTBS also reduced resting-state delta power over the left prefrontal region. This reduction in resting-state delta power correlated with the changes in WM.

CONCLUSIONS

Prefrontal iTBS may enhance WM performance in MUD subjects. iTBS induced resting EEG changes raising the possibility that such findings may represent a biological target of iTBS treatment response.

Evaluation of inter-rater and test-retest reliability for near-infrared spectroscopy reactive hyperemia measures

BACKGROUND

Near-infrared spectroscopy (NIRS) is a non-invasive tool used to measure blood flow in peripheral tissues. More information on inter-rater agreement and test-retest reliability of NIRS-based reperfusion assessments is needed.

PURPOSE

To assess inter-rater agreement for NIRS based data analysis, and evaluate the measurement's reliability across days.

METHODS

On three separate days (average days between visits 1 and 3: 19.4 ± 6.9 days), participants' (N = 15 males, 22 ± 2 yr.) post-occlusion reactive hyperemia (PORH) was measured in the left gastrocnemius muscle using Continuous-Wave NIRS (CW-NIRS). A blood pressure cuff was placed proximal to the knee and inflated to occlude lower leg blood flow for 5 min. The following CW-NIRS parameters were selected: (1) percent saturation in HbO₂ (StO₂%) at baseline; (2) the O₂Hb range used to normalize the NIRS signal; (3) the time for the O₂Hb signal to reach 50 % peak post-occlusion hyperemia (T_1/2), and (4) the post peak hyperemic O₂Hb recovery slope (O₂REC-SLP). Absolute agreement between the two analysts was calculated using two-way random effects Intraclass Correlation Coefficients (ICC_2,1). Consistency between analysts and across days was calculated using two-way mixed models (ICC_3,1). Mean and 95 % confidence intervals (CI) of ICCs are reported. Coefficient of variation (CV) and standard error of the measurement (SEM) are reported.

RESULTS

The ICC_2,1 data indicated "adequate" to "excellent" absolute agreement between the two NIRS analysts. ICC_2,3 data indicated "adequate" to "good" reliability across visits. The CV and SEM for rater 1 and rater 2 across visit were StO₂ (CV: 3.79 % ± 2.71 % and 4.50 % ± 2.37 %; SEM: 3.42 and 3.82), O₂Hb range (CV: 10.50 ± 5.93 and 12.79 ± 12.41; SEM: 3.26 and 4.71), T_1/2 (CV: 11.15 % ± 5.52 % and 10.96 % ± 4.50; SEM: 1.22 and 1.11), and O₂REC-SLP (CV: 19.49 % ± 9.99 % and 18.45 % ± 9.48 %; SEM: 0.04 and 0.04).

CONCLUSION

It is concluded that NIRS parameters assessed show adequate reliability between analysts and across three visits. It is recommended, when feasible and because of the absence of 100 % reliability, that investigators employ more than one rater for scoring at least a portion of the data across each trial in a study's control condition in order to have the ability to estimate the magnitude of error attributable to imperfect reliability.

The logical structure of experiments lays the foundation for a theory of reproducibility

The scientific reform movement has proposed openness as a potential remedy to the putative reproducibility or replication crisis. However, the conceptual relationship among openness, replication experiments and results reproducibility has been obscure. We analyse the logical structure of experiments, define the mathematical notion of idealized experiment and use this notion to advance a theory of reproducibility. Idealized experiments clearly delineate the concepts of replication and results reproducibility, and capture key differences with precision, allowing us to study the relationship among them. We show how results reproducibility varies as a function of the elements of an idealized experiment, the true data-generating mechanism, and the closeness of the replication experiment to an original experiment. We clarify how openness of experiments is related to designing informative replication experiments and to obtaining reproducible results. With formal backing and evidence, we argue that the current 'crisis' reflects inadequate attention to a theoretical understanding of results reproducibility.

Watching with Argus eyes: Characterization of emotional and physiological responding in adults exposed to childhood maltreatment and/or recent adversity

Exposure to adverse experiences is a well-established major risk factor for affective psychopathology. The vulnerability of deleterious sequelae is assumed in maladaptive processes of the defensive system, particularly in emotional processing. More specifically, childhood maltreatment has been suggested to be associated with the recruitment of specific and distinct defensive response profiles. To date, it remains unclear whether these are specific or generalizable to recent adversity in adulthood. This pre-registered study aimed to investigate the impact of exposure to childhood and recent adversity on emotional processing in 685 healthy adults with the "Affective Startle Modulation" Paradigm (ASM). First, we replicated higher trait anxiety and depression levels in individuals exposed to both types of adversity. Second, we observed increased general skin conductance reactivity in individuals exposed to recent adversity. Third, individuals exposed to childhood maltreatment showed reduced, while individuals exposed to recent adversity showed increased discrimination between pictures of negative and neutral valence, compared with non-exposed individuals in SCR. No association between exposure to adversity and fear potentiated startle was observed. Furthermore, explorative analyses revealed moderate dimensional and categorical agreement between two childhood maltreatment questionnaires and provide insight into potential adversity-type specific effects. Our results support experience-dependent plasticity in sympathetic nervous system reactivity and suggest distinct response profiles in affective modulation in individuals exposed to early versus recent adversity. We emphasize the need to further explore distinct adversity profiles to further our understanding on specific psychophysiological profiles and their potential implication for prevention and intervention.

A cognitive process modeling framework for the ABCD study stop-signal task

The Adolescent Brain Cognitive Development (ABCD) Study is a longitudinal neuroimaging study of unprecedented scale that is in the process of following over 11,000 youth from middle childhood though age 20. However, a design feature of the study's stop-signal task violates "context independence", an assumption critical to current non-parametric methods for estimating stop-signal reaction time (SSRT), a key measure of inhibitory ability in the study. This has led some experts to call for the task to be changed and for previously collected data to be used with caution. We present a cognitive process modeling framework, the RDEX-ABCD model, that provides a parsimonious explanation for the impact of this design feature on "go" stimulus processing and successfully accounts for key behavioral trends in the ABCD data. Simulation studies using this model suggest that failing to account for the context independence violations in the ABCD design can lead to erroneous inferences in several realistic scenarios. However, we demonstrate that RDEX-ABCD effectively addresses these violations and can be used to accurately measure SSRT along with an array of additional mechanistic parameters of interest (e.g., attention to the stop signal, cognitive efficiency), advancing investigators' ability to draw valid and nuanced inferences from ABCD data. AVAILABILITY OF DATA AND MATERIALS: Data from the ABCD Study are available through the NIH Data Archive (NDA): nda.nih.gov/abcd. Code for all analyses featured in this study is openly available on the Open Science Framework (OSF): osf.io/2h8a7/.

Do we really measure what we think we are measuring?

Tests used in the empirical sciences are often (implicitly) assumed to be representative of a given research question in the sense that similar tests should lead to similar results. Here, we show that this assumption is not always valid. We illustrate our argument with the example of resting-state electroencephalogram (EEG). We used multiple analysis methods, contrary to typical EEG studies where one analysis method is used. We found, first, that many EEG features correlated significantly with cognitive tasks. However, these EEG features correlated weakly with each other. Similarly, in a second analysis, we found that many EEG features were significantly different in older compared to younger participants. When we compared these EEG features pairwise, we did not find strong correlations. In addition, EEG features predicted cognitive tasks poorly as shown by cross-validated regression analysis. We discuss several explanations of these results.

Open science, the replication crisis, and environmental public health

Concerns about a crisis of mass irreplicability across scientific fields ("the replication crisis") have stimulated a movement for open science, encouraging or even requiring researchers to publish their raw data and analysis code. Recently, a rule at the US Environmental Protection Agency (US EPA) would have imposed a strong open data requirement. The rule prompted significant public discussion about whether open science practices are appropriate for fields of environmental public health. The aims of this paper are to assess (1) whether the replication crisis extends to fields of environmental public health; and (2) in general whether open science requirements can address the replication crisis. There is little empirical evidence for or against mass irreplicability in environmental public health specifically. Without such evidence, strong claims about whether the replication crisis extends to environmental public health - or not - seem premature. By distinguishing three concepts - reproducibility, replicability, and robustness - it is clear that open data initiatives can promote reproducibility and robustness but do little to promote replicability. I conclude by reviewing some of the other benefits of open science, and offer some suggestions for funding streams to mitigate the costs of adoption of open science practices in environmental public health.

The role of left insula mediating impaired error processing in response inhibition in adult heavy drinkers

Identification of neurobiological mechanisms underlying development of alcohol use disorder is critical to ensuring the appropriate early-phase treatment and prevention of the disorder. To this aim, we tried to elucidate the disturbance of neural functions in heavy drinking, which can lead to alcohol use disorder. Because response inhibition is affected by alcohol use disorder, we examined neural activation and task performance for response inhibition using the Go/No-Go task in an fMRI paradigm in adult non-dependent heavy and light drinkers. We examined the neural activation for error processing and inhibitory control, components of response inhibition. We then investigated the mediating effect of the relevant neural substrate on the relationship between the level of alcohol drinking and task performance using mediation analysis. We found that heavy drinking significantly decreased activation in the left insula during error processing and increased the mean commission error rate for No-Go trials compared with light drinking. Mediation analysis demonstrated full mediation of the left insula activation during error processing for the relationship between drinking level and commission error rate. Our results suggested that left insula activation may be a neural marker pivotal for potential conversion to alcohol use disorder in individuals with high clinical risk such as heavy drinking.

Mediating effect of pubertal stages on the family environment and neurodevelopment: An open-data replication and multiverse analysis of an ABCD Study®

Increasing evidence demonstrates that environmental factors meaningfully impact the development of the brain (Hyde et al., 2020; McEwen and Akil, 2020). Recent work from the Adolescent Brain Cognitive Development (ABCD) Study® suggests that puberty may indirectly account for some association between the family environment and brain structure and function (Thijssen et al., 2020). However, a limited number of large studies have evaluated what, how, and why environmental factors impact neurodevelopment. When these topics are investigated, there is typically inconsistent operationalization of variables between studies which may be measuring different aspects of the environment and thus different associations in the analytic models. Multiverse analyses (Steegen et al., 2016) are an efficacious technique for investigating the effect of different operationalizations of the same construct on underlying interpretations. While one of the assets of Thijssen et al. (2020) was its large sample from the ABCD data, the authors used an early release that contained 38% of the full ABCD sample. Then, the analyses used several 'researcher degrees of freedom' (Gelman and Loken, 2014) to operationalize key independent, mediating and dependent variables, including but not limited to, the use of a latent factor of preadolescents' environment comprised of different subfactors, such as parental monitoring and child-reported family conflict. While latent factors can improve reliability of constructs, the nuances of each subfactor and measure that comprise the environment may be lost, making the latent factors difficult to interpret in the context of individual differences. This study extends the work of Thijssen et al. (2020) by evaluating the extent to which the analytic choices in their study affected their conclusions. In Aim 1, using the same variables and models, we replicate findings from the original study using the full sample in Release 3.0. Then, in Aim 2, using a multiverse analysis we extend findings by considering nine alternative operationalizations of family environment, three of puberty, and five of brain measures (total of 135 models) to evaluate the impact on conclusions from Aim 1. In these results, 90% of the directions of effects and 60% of the p-values (e.g. p > .05 and p < .05) across effects were comparable between the two studies. However, raters agreed that only 60% of the effects had replicated. Across the multiverse analyses, there was a degree of variability in beta estimates across the environmental variables, and lack of consensus between parent reported and child reported pubertal development for the indirect effects. This study demonstrates the challenge in defining which effects replicate, the nuance across environmental variables in the ABCD data, and the lack of consensus across parent and child reported puberty scales in youth.