A discipline-wide investigation of the replicability of Psychology papers over the past two decades

Estimating the replicability of highly cited clinical research (2004-2018)

INTRODUCTION

Previous studies about the replicability of clinical research based on the published literature have suggested that highly cited articles are often contradicted or found to have inflated effects. Nevertheless, there are no recent updates of such efforts, and this situation may have changed over time.

METHODS

We searched the Web of Science database for articles studying medical interventions with more than 2000 citations, published between 2004 and 2018 in high-impact medical journals. We then searched for replications of these studies in PubMed using the PICO (Population, Intervention, Comparator and Outcome) framework. Replication success was evaluated by the presence of a statistically significant effect in the same direction and by overlap of the replication's effect size confidence interval (CIs) with that of the original study. Evidence of effect size inflation and potential predictors of replicability were also analyzed.

RESULTS

A total of 89 eligible studies, of which 24 had valid replications (17 meta-analyses and 7 primary studies) were found. Of these, 21 (88%) had effect sizes with overlapping CIs. Of 15 highly cited studies with a statistically significant difference in the primary outcome, 13 (87%) had a significant effect in the replication as well. When both criteria were considered together, the replicability rate in our sample was of 20 out of 24 (83%). There was no evidence of systematic inflation in these highly cited studies, with a mean effect size ratio of 1.03 [95% CI (0.88, 1.21)] between initial and subsequent effects. Due to the small number of contradicted results, our analysis had low statistical power to detect predictors of replicability.

CONCLUSION

Although most studies did not have eligible replications, the replicability rate of highly cited clinical studies in our sample was higher than in previous estimates, with little evidence of systematic effect size inflation. This estimate is based on a very select sample of studies and may not be generalizable to clinical research in general.

The Alternative Model of Personality Disorders: Assessment, Convergent and Discriminant Validity, and a Look to the Future

The Alternative Model of Personality Disorders (AMPD) is a dimensional, empirically based diagnostic system developed to overcome the serious limitations of traditional categories. We review the mounting evidence on its convergent and discriminant validity, with an incursion into the less-studied ICD-11 system. In the literature, the AMPD's Pathological Trait Model (Criterion B) shows excellent convergence with normal personality traits, and it could be useful as an organizing framework for mental disorders. In contrast, Personality Functioning (Criterion A) cannot be distinguished from personality traits, lacks both discriminant and incremental validity, and has a shaky theoretical background. We offer some suggestions with a view to the future. These include removing Criterion A, using the real-life consequences of traits as indicators of severity, delving into the dynamic mechanisms underlying traits, and furthering the integration of currently disengaged psychological paradigms that can shape a sounder clinical science.

A metascience investigation of inclusive, open, and reproducible science practices in research posters at the 2021 SRCD biennial meeting

Over the past decade, there has been a growing appreciation of metascience issues in psychological science. Using data collected from 2615 posters presented at the 2021 biennial meeting of the Society for Research in Child Development, this article examines the use of transparent research practices to increase rigor and reproducibility as well as generalizability through greater inclusivity of diverse samples. Research presented through poster presentations was heavily skewed toward quantitative studies featuring American researchers using Western hemisphere samples. Sharing of data/materials, preregistrations, and replications were uncommon. During a time when governments are increasingly requiring more open practices and access, this research provides an important baseline by which developmental science can benchmark progress toward the goals of greater inclusivity and openness.

Artificial intelligence: help or hindrance in solving the reproducibility crisis?

Science is in the midst of a reproducibility crisis and the integration of artificial intelligence into scientific research has only compounded the problem; yet could the technology hold the solution to its own problems?[Formula: see text].

Promotional language and the adoption of innovative ideas in science

How are the merits of innovative ideas communicated in science? Here, we conduct semantic analyses of grant application success with a focus on scientific promotional language, which may help to convey an innovative idea's originality and significance. Our analysis attempts to surmount the limitations of prior grant studies by examining the full text of tens of thousands of both funded and unfunded grants from three leading public and private funding agencies: the NIH, the NSF, and the Novo Nordisk Foundation, one of the world's largest private science funding foundations. We find a robust association between promotional language and the support and adoption of innovative ideas by funders and other scientists. First, a grant proposal's percentage of promotional language is associated with up to a doubling of the grant's probability of being funded. Second, a grant's promotional language reflects its intrinsic innovativeness. Third, the percentage of promotional language is predictive of the expected citation and productivity impact of publications that are supported by funded grants. Finally, a computer-assisted experiment that manipulates the promotional language in our data demonstrates how promotional language can communicate the merit of ideas through cognitive activation. With the incidence of promotional language in science steeply rising, and the pivotal role of grants in converting promising and aspirational ideas into solutions, our analysis provides empirical evidence that promotional language is associated with effectively communicating the merits of innovative scientific ideas.

Positive and negative facial valence perception are modulated differently by eccentricity in the parafovea: Replication from KDEF to NimStim

While perceiving the emotional state of others may be crucial for our behavior even when this information is present outside of central vision, emotion perception studies typically focus on central visual field. We have recently investigated emotional valence (pleasantness) perception across the parafovea (≤ 4°) and found that for briefly presented (200 ms) emotional face images (from the established KDEF image-set), positive (happy) valence was the least affected by eccentricity (distance from the central visual field) and negative (fearful) valence the most. Furthermore, we found that performance at 2° predicted performance at 4°. Here we tested (n = 37) whether these effects replicate with face stimuli of different identities from a different well-established image-set (NimStim). All our prior findings replicated and eccentricity-based modulation magnitude was smaller with NimStim (~ 16.6% accuracy reduction at 4°) than with KDEF stimuli (~ 27.3% reduction). Our current investigations support our earlier findings that for briefly presented parafoveal stimuli, positive and negative valence perception are differently affected by eccentricity and may be dissociated. Furthermore, our results highlight the importance of investigating emotions beyond central vision and demonstrate commonalities and differences across different image sets in the parafovea, emphasizing the contribution of replication studies to substantiate our knowledge about perceptual mechanisms.

Why moral psychology needs personality psychology

People vary in how they perceive, think about, and respond to moral issues. Clearly, we cannot fully understand the psychology of morality without accounting for individual differences in moral functioning. But decades of neglect of and explicit skepticism toward such individual differences has resulted in a lack of integration between moral psychology and personality psychology-the study of psychological differences between people. In recent years, these barriers to progress have started to break down. This special issue aims to celebrate and further increase the visibility of the personality psychology of morality. Here, we introduce the articles in this special issue by highlighting some important contributions a personality-based perspective has to offer moral psychology-particularly in comparison to the currently prominent social psychological approach. We show that personality psychology is well-placed to (a) contribute toward a rigorous empirical foundation for moral psychology, (b) tackle the conceptualization and assessment of stable moral tendencies, (c) assess the predictive validity of moral traits in relation to consequential outcomes, (d) uncover the mechanisms underlying individual differences in moral judgments and behavior, and (e) provide insights into moral development. For these reasons, we believe that moral psychology needs personality psychology to reach its full scholarly potential.

Exploring the understanding of reproducibility among stakeholders within academia and their expectations for a web-based education tool: A qualitative study

Although reproducibility is central to the scientific method, its understanding within the research community remains insufficient. We aimed to explore the perceptions of research reproducibility among stakeholders within academia, learn about possible barriers and facilitators to reproducibility-related practices, and gather their suggestions for the Croatian Reproducibility Network website. We conducted four focus groups with researchers, teachers, editors, research managers, and policymakers from Croatia (n = 23). The participants observed a lack of consensus on the core definitions of reproducibility, both generally and between disciplines. They noted that incentivization and recognition of reproducibility-related practices from publishers and institutions, alongside comprehensive education adapted to the researchers' career stage, could help with implementing reproducibility. Education was considered essential to these efforts, as it could help create a research culture based on good reproducibility-related practices and behavior rather than one driven by mandates or career advancement. This was particularly found to be relevant for growing reproducibility efforts globally. Regarding the Croatian Reproducibility Network website, the participants suggested we adapt the content to users from different disciplines or career stages and offer guidance and tools for reproducibility through which we should present core reproducibility concepts. Our findings could inform other initiatives focused on improving research reproducibility.

Artificial intelligence and illusions of understanding in scientific research

Scientists are enthusiastically imagining ways in which artificial intelligence (AI) tools might improve research. Why are AI tools so attractive and what are the risks of implementing them across the research pipeline? Here we develop a taxonomy of scientists' visions for AI, observing that their appeal comes from promises to improve productivity and objectivity by overcoming human shortcomings. But proposed AI solutions can also exploit our cognitive limitations, making us vulnerable to illusions of understanding in which we believe we understand more about the world than we actually do. Such illusions obscure the scientific community's ability to see the formation of scientific monocultures, in which some types of methods, questions and viewpoints come to dominate alternative approaches, making science less innovative and more vulnerable to errors. The proliferation of AI tools in science risks introducing a phase of scientific enquiry in which we produce more but understand less. By analysing the appeal of these tools, we provide a framework for advancing discussions of responsible knowledge production in the age of AI.

Childhood adversity severity modulates the associations between adaptive psychological changes and amygdala volumetric changes in response to behavioral interventions

Background

Adverse Childhood Experience (ACE) has been shown to have detrimental impact on amygdala structure. Prior research found that adaptive psychological changes after Mindfulness-Based Interventions (MBI) were associated with amygdala volumetric changes. The present study aims to further investigate whether such effects also occur among ACE survivors and whether the effects are unique to MBI.

Methods

A total of 64 young adult childhood adversity survivors were randomized to an eight-week MBI or Stress Management Education (SME) as an active control condition. Anatomical MRI and questionnaires on mindfulness, stress and psychological health were collected at baseline and post-intervention. Due to subject dropout, the final sample included 39 subjects (MBI:20, SME:19).

Results

Both groups showed increased mindfulness levels, reduced stress, and improved psychological symptoms (depression, anxiety, and somatization), with no significant group by time interaction effect. There was no significant group difference on amygdala volumetric changes. Within the MBI group, childhood maltreatment severity was a significant mediator between changes of mindfulness levels and right amygdala volumetric changes. Across pooled sample of both groups, childhood maltreatment was a significant moderator for the effect of trait anxiety level changes on left amygdala volumetric changes.

Limitations

Modest sample size, relatively low retention rates, suboptimal monitoring of home practice.

Conclusions

MBI did not demonstrate overall better clinical effects than SME. Psychological-change-dependent amygdala volumetric change was not specific to MBI. Childhood maltreatment severity modulated the relationships between adaptive psychological changes and amygdala volumetric changes.

Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records

BACKGROUND

Recent advances in machine learning combined with the growing availability of digitized health records offer new opportunities for improving early diagnosis of depression. An emerging body of research shows that Electronic Health Records can be used to accurately predict cases of depression on the basis of individual's primary care records. The successes of these studies are undeniable, but there is a growing concern that their results may not be replicable, which could cast doubt on their clinical usefulness.

METHODS

To address this issue in the present paper, we set out to reproduce and replicate the work by Nichols et al. (2018), who trained predictive models of depression among young adults using Electronic Healthcare Records. Our contribution consists of three parts. First, we attempt to replicate the methodology used by the original authors, acquiring a more up-to-date set of primary health care records to the same specification and reproducing their data processing and analysis. Second, we test models presented in the original paper on our own data, thus providing out-of-sample prediction of the predictive models. Third, we extend past work by considering several novel machine-learning approaches in an attempt to improve the predictive accuracy achieved in the original work.

RESULTS

In summary, our results demonstrate that the work of Nichols et al. is largely reproducible and replicable. This was the case both for the replication of the original model and the out-of-sample replication applying NRCBM coefficients to our new EHRs data. Although alternative predictive models did not improve model performance over standard logistic regression, our results indicate that stepwise variable selection is not stable even in the case of large data sets.

CONCLUSION

We discuss the challenges associated with the research on mental health and Electronic Health Records, including the need to produce interpretable and robust models. We demonstrated some potential issues associated with the reliance on EHRs, including changes in the regulations and guidelines (such as the QOF guidelines in the UK) and reliance on visits to GP as a predictor of specific disorders.

Eleven years of student replication projects provide evidence on the correlates of replicability in psychology

Cumulative scientific progress requires empirical results that are robust enough to support theory construction and extension. Yet in psychology, some prominent findings have failed to replicate, and large-scale studies suggest replicability issues are widespread. The identification of predictors of replication success is limited by the difficulty of conducting large samples of independent replication experiments, however: most investigations reanalyse the same set of 170   replications . We introduce a new dataset of 176 replications from students in a graduate-level methods course. Replication results were judged to be successful in 49% of replications; of the 136 where effect sizes could be numerically compared, 46% had point estimates within the prediction interval of the original outcome (versus the expected 95%). Larger original effect sizes and within-participants designs were especially related to replication success. Our results indicate that, consistent with prior reports, the robustness of the psychology literature is low enough to limit cumulative progress by student investigators.

Data, measurement and empirical methods in the science of science

The advent of large-scale datasets that trace the workings of science has encouraged researchers from many different disciplinary backgrounds to turn scientific methods into science itself, cultivating a rapidly expanding 'science of science'. This Review considers this growing, multidisciplinary literature through the lens of data, measurement and empirical methods. We discuss the purposes, strengths and limitations of major empirical approaches, seeking to increase understanding of the field's diverse methodologies and expand researchers' toolkits. Overall, new empirical developments provide enormous capacity to test traditional beliefs and conceptual frameworks about science, discover factors associated with scientific productivity, predict scientific outcomes and design policies that facilitate scientific progress.

Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory

Terror Management Theory (TMT) postulates that humans, in response to awareness of their death, developed complex defenses to remove the salience and discomfort stemming from those thoughts. In a standard paradigm to test this theory, an individual is presented with a death-related prime (Mortality Salience; MS), such as writing the details of their own death, or something neutral, such as watching television. After a distractor task (for delay), participants complete the dependent variable, such as rating how much they like or agree with a pro- or anti-national essay and its author. Individuals in the MS condition typically exhibit greater worldview defense than control conditions by rating the pro-national essay more positively and the anti-national essay more negatively. We completed five separate studies across five unique samples with the goal of replicating and extending this well-established pattern to provide further understanding of the phenomena that underlie the effects of MS. However, despite using standard procedures, we were unable to replicate basic patterns of the dependent variable in the MS conditions. We also pooled all responses into two meta-analyses, one examining all dependent variables and one focusing on the anti-national essay; yet the effect sizes in these analyses did not significantly differ from zero. We discuss the methodological and theoretical implications of these (unintended) failures to replicate. It is not clear if these null findings were due to methodological limitations, restraints of online/crowd-sourced recruitment, or ever-evolving sociocultural factors.