Contextual sensitivity in scientific reproducibility

Methodological concerns underlying a lack of evidence for cultural heterogeneity in the replication of psychological effects

The multi-site replication study, Many Labs 2, concluded that sample location and setting did not substantially affect the replicability of findings. Here, we examine theoretical and methodological considerations for a subset of the analyses, namely exploratory tests of heterogeneity in the replicability of studies between "WEIRD and less-WEIRD cultures". We conducted a review of literature citing the study, a re-examination of the existing cultural variability, a power stimulation for detecting cultural heterogeneity, and re-analyses of the original exploratory tests. Findings indicate little cultural variability and low power to detect cultural heterogeneity effects in the Many Labs 2 data, yet the literature review indicates the study is cited regarding the moderating role of culture. Our reanalysis of the data found that using different operationalizations of culture slightly increased effect sizes but did not substantially alter the conclusions of Many Labs 2. Future studies of cultural heterogeneity can be improved with theoretical consideration of which effects and which cultures are likely to show variation as well as a priori methodological planning for appropriate operationalizations of culture and sufficient power to detect effects.

Our brains sense the future through a new quantum-like implicit learning mechanism

BACKGROUND

Imagine if our brains could unconsciously predict future events. This study explores this concept, presenting evidence for an inherent 'foreseeing' ability, termed anomalous cognition (AC). We introduce a new experimentally verifiable approach to explain anomalous information anticipation (AIA), a type of AC, based on an innovative, quantum-like model of implicit learning, grounded in Nonlocal Plasticity Theory (NPT).

METHODS

Our research involved 203 participants using methods such as continuous flash suppression, random dot motion, and advanced 3D EEG neuroimaging, along with IBM quantum random event generators for precise measurements across 144 trials. These trials tested contingencies between undetectable sensory stimuli and dot movements, focusing on participants' prediction abilities. The design conditions were strictly experimental, violating fundamental classical learning principles, particularly reflex conditioning. If these principles were immutable, their violation would prevent any systematic behavioral changes, resulting in random responses. This violation was implemented through two quantum physics concepts: the mathematical principle of nonlocality and entanglement.

RESULTS

Despite the sensory stimulus being inaccessible, our results showed a significant prediction between the contingencies and an increase in AIA accuracy, with explained variances between 25 % and 48 %. EEG findings supported this, showing a positive link between brain activity in specific regions and AIA success. Electrochemical activations were detected in the posterior occipital cortex, the intraparietal sulcus, and the medial temporal gyri. AIA hits exceeded the threshold value corresponding to one standard deviation above the expected mean, showing moderate effect sizes in the experimental group (Cohen's d = 0.461). Analyzing the learning curve using the derivation technique, we identified the acceleration point of the wave function, indicating systematic implicit learning. This result showed that from repetition 63 onwards, AIA hits increased significantly.

CONCLUSIONS

The results suggest that, despite violating fundamental classical learning principles, cognitive processes produced changes in participants' responses susceptible to neuromodulation, considering quantum physics principles of nonlocality and entanglement (both present in NPT). We discuss (a) why the priming effect does not explain the significant results; (b) the potential discovery of a new form of quantum-like implicit learning, which could scientifically resolve phenomena associated with anomalous cognitions (e.g., AIA); and (c) future research directions, including potential applications and clinical impact.

Challenging the N-Heuristic: Effect size, not sample size, predicts the replicability of psychological science

Large sample size (N) is seen as a key criterion in judging the replicability of psychological research, a phenomenon we refer to as the N-Heuristic. This heuristic has led to the incentivization of fast, online, non-behavioral studies-to the potential detriment of psychological science. While large N should in principle increase statistical power and thus the replicability of effects, in practice it may not. Large-N studies may have other attributes that undercut their power or validity. Consolidating data from all systematic, large-scale attempts at replication (N = 307 original-replication study pairs), we find that the original study's sample size did not predict its likelihood of being replicated (rs = -0.02, p = 0.741), even with study design and research area controlled. By contrast, effect size emerged as a substantial predictor (rs = 0.21, p < 0.001), which held regardless of the study's sample size. N may be a poor predictor of replicability because studies with larger N investigated smaller effects (rs = -0.49, p < 0.001). Contrary to these results, a survey of 215 professional psychologists, presenting them with a comprehensive list of methodological criteria, found sample size to be rated as the most important criterion in judging a study's replicability. Our findings strike a cautionary note with respect to the prioritization of large N in judging the replicability of psychological science.

High replicability of newly discovered social-behavioural findings is achievable

Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigour-enhancing practices: confirmatory tests, large sample sizes, preregistration and methodological transparency. In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing (P < 0.05) in 86% of attempts, slightly exceeding the maximum expected replicability based on observed effect sizes and sample sizes. When one lab attempted to replicate an effect discovered by another lab, the effect size in the replications was 97% that in the original study. This high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries.

Designing optimal behavioral experiments using machine learning

Computational models are powerful tools for understanding human cognition and behavior. They let us express our theories clearly and precisely and offer predictions that can be subtle and often counter-intuitive. However, this same richness and ability to surprise means our scientific intuitions and traditional tools are ill-suited to designing experiments to test and compare these models. To avoid these pitfalls and realize the full potential of computational modeling, we require tools to design experiments that provide clear answers about what models explain human behavior and the auxiliary assumptions those models must make. Bayesian optimal experimental design (BOED) formalizes the search for optimal experimental designs by identifying experiments that are expected to yield informative data. In this work, we provide a tutorial on leveraging recent advances in BOED and machine learning to find optimal experiments for any kind of model that we can simulate data from, and show how by-products of this procedure allow for quick and straightforward evaluation of models and their parameters against real experimental data. As a case study, we consider theories of how people balance exploration and exploitation in multi-armed bandit decision-making tasks. We validate the presented approach using simulations and a real-world experiment. As compared to experimental designs commonly used in the literature, we show that our optimal designs more efficiently determine which of a set of models best account for individual human behavior, and more efficiently characterize behavior given a preferred model. At the same time, formalizing a scientific question such that it can be adequately addressed with BOED can be challenging and we discuss several potential caveats and pitfalls that practitioners should be aware of. We provide code to replicate all analyses as well as tutorial notebooks and pointers to adapt the methodology to different experimental settings.

A synthesis of evidence for policy from behavioural science during COVID-19

Scientific evidence regularly guides policy decisions1, with behavioural science increasingly part of this process2. In April 2020, an influential paper3 proposed 19 policy recommendations ('claims') detailing how evidence from behavioural science could contribute to efforts to reduce impacts and end the COVID-19 pandemic. Here we assess 747 pandemic-related research articles that empirically investigated those claims. We report the scale of evidence and whether evidence supports them to indicate applicability for policymaking. Two independent teams, involving 72 reviewers, found evidence for 18 of 19 claims, with both teams finding evidence supporting 16 (89%) of those 18 claims. The strongest evidence supported claims that anticipated culture, polarization and misinformation would be associated with policy effectiveness. Claims suggesting trusted leaders and positive social norms increased adherence to behavioural interventions also had strong empirical support, as did appealing to social consensus or bipartisan agreement. Targeted language in messaging yielded mixed effects and there were no effects for highlighting individual benefits or protecting others. No available evidence existed to assess any distinct differences in effects between using the terms 'physical distancing' and 'social distancing'. Analysis of 463 papers containing data showed generally large samples; 418 involved human participants with a mean of 16,848 (median of 1,699). That statistical power underscored improved suitability of behavioural science research for informing policy decisions. Furthermore, by implementing a standardized approach to evidence selection and synthesis, we amplify broader implications for advancing scientific evidence in policy formulation and prioritization.

A research agenda for understanding how social inequality is linked to brain structure and function

Consistent evidence documents powerful effects of social inequality on health, well-being and academic achievement. Yet research on whether social inequality may also be linked to brain structure and function has, until recently, been rare. Here we describe three methodological approaches that can be used to study this question-single site, single study; multi-site, single study; and spatial meta-analysis. We review empirical work that, using these approaches, has observed associations between neural outcomes and structural measures of social inequality-including structural stigma, community-level prejudice, gender inequality, neighbourhood disadvantage and the generosity of the social safety net for low-income families. We evaluate the relative strengths and limitations of these approaches, discuss ethical considerations and outline directions for future research. In doing so, we advocate for a paradigm shift in cognitive neuroscience that explicitly incorporates upstream structural and contextual factors, which we argue holds promise for uncovering the neural correlates of social inequality.

Eleven years of student replication projects provide evidence on the correlates of replicability in psychology

Cumulative scientific progress requires empirical results that are robust enough to support theory construction and extension. Yet in psychology, some prominent findings have failed to replicate, and large-scale studies suggest replicability issues are widespread. The identification of predictors of replication success is limited by the difficulty of conducting large samples of independent replication experiments, however: most investigations reanalyse the same set of 170   replications . We introduce a new dataset of 176 replications from students in a graduate-level methods course. Replication results were judged to be successful in 49% of replications; of the 136 where effect sizes could be numerically compared, 46% had point estimates within the prediction interval of the original outcome (versus the expected 95%). Larger original effect sizes and within-participants designs were especially related to replication success. Our results indicate that, consistent with prior reports, the robustness of the psychology literature is low enough to limit cumulative progress by student investigators.

When Patients' Assessments of Treatment for Subjective Symptoms are At Odds with Authoritative Assessments in the Biomedical Literature: The Case of Patients Treated with Acupuncture for Low Back Pain in a Traditional Chinese Medicine Clinic in China (and Story of a Bewildered Chicken)

Assessments of treatments for 'subjective symptoms' are problematic and potentially contentious. These are symptoms without ascertainable pathophysiology, also referred to as 'medically unexplained.' Treatments of them may be assessed from different perspectives, and an assessment as seen from one perspective may be discrepant with an assessment as seen from another perspective. The observational study described in the paper represents one perspective. Patients in the study were treated with acupuncture for the subjective symptom of low back pain in a Traditional Chinese Medicine Clinic in China; acupuncture is a form of TCM, which is indigenous to China. Shortly after treatment began and six months afterward, patients reported 'clinically important improvements' on a variety of standard, cross-culturally validated outcome measures, including pain intensity, physical functioning, and emotional functioning; this assessment is consistent with numerous assessments of acupuncture in China and other Southeastern Asian countries. On the other hand, clinical trials represent another perspective from which to assess acupuncture. Although assessments based on trials are not uniform, several assessments based on them, including those published in journals such as Lancet and New England Journal of Medicine, conclude that acupuncture is not effective. These trials, however, were mainly situated in the U.S., western European countries, and other WEIRD (Western, Educated, Industrialized, Rich, Democratic) countries. This is notable, because the population of China, the indigenous context of acupuncture, is greater than populations of WEIRD countries combined. Patients' expectations of a treatment such as acupuncture and their prior familiarity with it vary among contexts, and patients' experiences of treatment outcomes may vary accordingly. In short, although clinical trials constitute a test of truth in biomedicine and inform authoritative assessments in WEIRD countries, they do not necessarily represent the truth from the perspective of those experiencing outcomes of especially indigenous treatments for subjective symptoms in non-WEIRD contexts.

Subjective socioeconomic status and income inequality are associated with self-reported morality across 67 countries

Individuals can experience a lack of economic resources compared to others, which we refer to as subjective experiences of economic scarcity. While such experiences have been shown to shift cognitive focus, attention, and decision-making, their association with human morality remains debated. We conduct a comprehensive investigation of the relationship between subjective experiences of economic scarcity, as indexed by low subjective socioeconomic status at the individual level, and income inequality at the national level, and various self-reported measures linked to morality. In a pre-registered study, we analyze data from a large, cross-national survey (N = 50,396 across 67 countries) allowing us to address limitations related to cross-cultural generalizability and measurement validity in prior research. Our findings demonstrate that low subjective socioeconomic status at the individual level, and income inequality at the national level, are associated with higher levels of moral identity, higher morality-as-cooperation, a larger moral circle, and increased prosocial intentions. These results appear robust to several advanced control analyses. Finally, exploratory analyses indicate that observed income inequality at the national level is not a statistically significant moderator of the associations between subjective socioeconomic status and the included measures of morality. These findings have theoretical and practical implications for understanding human morality under experiences of resource scarcity.

Research Into Evidence-Based Psychological Interventions Needs a Stronger Focus on Replicability

Background

It is a precondition for evidence-based practice that research is replicable in a wide variety of clinical settings. Current standards for identifying evidence-based psychological interventions and making recommendations for clinical practice in clinical guidelines include criteria that are relevant for replicability, but a better understanding as well refined definitions of replicability are needed enabling empirical research on this topic. Recent advances on this issue were made in the wider field of psychology and in other disciplines, which offers the opportunity to define and potentially increase replicability also in research on psychological interventions.

Method

This article proposes a research strategy for assessing, understanding, and improving replicability in research on psychological interventions.

Results/Conclusion

First, we establish a replication taxonomy ranging from direct to conceptual replication adapted to the field of research on clinical interventions, propose study characteristics that increase the trustworthiness of results, and define statistical criteria for successful replication with respect to the quantitative outcomes of the original and replication studies. Second, we propose how to establish such standards for future research, i.e., in order to design future replication studies for psychological interventions as well as to apply them when investigating which factors are causing the (non-)replicability of findings in the current literature.

A manifesto for applying behavioural science

Recent years have seen a rapid increase in the use of behavioural science to address the priorities of public and private sector actors. There is now a vibrant ecosystem of practitioners, teams and academics building on each other's findings across the globe. Their focus on robust evaluation means we know that this work has had an impact on important issues such as antimicrobial resistance, educational attainment and climate change. However, several critiques have also emerged; taken together, they suggest that applied behavioural science needs to evolve further over its next decade. This manifesto for the future of applied behavioural science looks at the challenges facing the field and sets out ten proposals to address them. Meeting these challenges will mean that behavioural science is better equipped to help to build policies, products and services on stronger empirical foundations-and thereby address the world's crucial challenges.

Manipulating Belief in Free Will and Its Downstream Consequences: A Meta-Analysis

Ever since some scientists and popular media put forward the idea that free will is an illusion, the question has risen what would happen if people stopped believing in free will. Psychological research has investigated this question by testing the consequences of experimentally weakening people's free will beliefs. The results of these investigations have been mixed, with successful experiments and unsuccessful replications. This raises two fundamental questions: Can free will beliefs be manipulated, and do such manipulations have downstream consequences? In a meta-analysis including 145 experiments (95 unpublished), we show that exposing individuals to anti-free will manipulations decreases belief in free will and increases belief in determinism. However, we could not find evidence for downstream consequences. Our findings have important theoretical implications for research on free will beliefs and contribute to the discussion of whether reducing people's belief in free will has societal consequences.

Physical proximity as pleasure or pain? A critical review of employee–customer proximity in sales and services settings

This paper presents a critical review of published findings pertaining to the physical proximity between employees and customers in various sales and service settings. Following an overview of this stream of research, reflections are then offered on how the concepts of personal space and physical proximity may have changed in terms of their financial and well-being-related effects as a function of the COVID-19 pandemic. Due to the risk of infection in interpersonal interactions, and despite the affiliative aspects associated with physical proximity, recent recipes for success—as advocated by academics—may eventually have a negative impact on multiple crucial metrics in a post-pandemic world, such that employees’ physical proximity to customers may soon come with a wide array of costly consequences. The article concludes with a set of future research directions.

Siblings, shopping, and sustainability: Birth-order differences in green consumption

Several studies have examined the role of birth order in shaping human personality, but fewer have tested this variable in relation to other pressing issues. We conducted a birth-order study on green consumption, which enabled us to detect a small-to-moderate effect size equivalent to r = 0.15 or d = 0.30 with sufficient statistical power (N = 335). To capture green consumption, participants indicated their tendency to express the value of environmental protection through purchases and consumption behaviors. Firstborns (vs. laterborns) consistently expressed lower concerns linked to environmental protection in their purchase patterns. While the effect size of this finding was small-to-moderate by conventional standards and in direct contrast to the findings from a recent article on the same topic, these results could still be informative to address challenges associated with climate change considering the number of individuals with siblings in the world and the ease with which birth-order data can be collected.

Debunking Misinformation About a Causal Link Between Vaccines and Autism: Two Preregistered Tests of Dual-Process Versus Single-Process Predictions (With Conflicting Results)

Dual-process and single-process theories lead to conflicting predictions about whether debunking messages negating a state of affairs should change responses on implicit measures in a manner intended by the message. Two preregistered studies (N1 = 550; N2 = 880) tested these predictions using official health information from the U.S. Centers for Disease Control and Prevention debunking the idea that vaccines would cause autism. Consistent with predictions derived from dual-process learning theories, Experiment 1 found that debunking-via-negation increased responses linking vaccines to autism on implicit measures, although it effectively reduced self-reported judgments linking vaccines to autism on explicit measures. Using the same measures and materials, Experiment 2 found that debunking-via-negation effectively reduced responses linking vaccines to autism on both implicit and explicit measures, consistent with predictions derived from single-process propositional theories. Potential reasons for the conflicting outcomes are discussed, including their implications for the debate between dual-process and single-process theories.

Sharing the Recipe: Reproducibility and Replicability in Research Across Disciplines

The open and transparent documentation of scientific processes has been established as a core antecedent of free knowledge. This also holds for generating robust insights in the scope of research projects. To convince academic peers and the public, the research process must be understandable and retraceable (reproducible), and repeatable (replicable) by others, precluding the inclusion of fluke findings into the canon of insights. In this contribution, we outline what reproducibility and replicability (R&R) could mean in the scope of different disciplines and traditions of research and which significance R&R has for generating insights in these fields. We draw on projects conducted in the scope of the Wikimedia "Open Science Fellows Program" (Fellowship Freies Wissen), an interdisciplinary, long-running funding scheme for projects contributing to open research practices. We identify twelve implemented projects from different disciplines which primarily focused on R&R, and multiple additional projects also touching on R&R. From these projects, we identify patterns and synthesize them into a roadmap of how research projects can achieve R&R across different disciplines. We further outline the ground covered by these projects and propose ways forward.

Psychology: Where history, culture, and biology meet

This article argues that the same epistemological assumptions cannot be confidently applied in the transition from the biological to the social arenas of psychology, as a consequence of the sociocultural instability resulting from human linguistic and technological flair. To illustrate this contention, reference is made to historicist theses within critical and sociocultural psychology, the work of Ian Hacking and Norbert Elias, the centrality of language and technology to sociocultural instability, and the illustrative issues raised by cultural neuroscience and replication studies.

Kinds of Replication: Examining the Meanings of "Conceptual Replication" and "Direct Replication"

Although psychology's recent crisis has been attributed to various scientific practices, it has come to be called a "replication crisis," prompting extensive appraisals of this putatively crucial scientific practice. These have yielded disagreements over what kind of replication is to be preferred and what phenomena are being explored, yet the proposals are all grounded in a conventional philosophy of science. This article proposes another avenue that invites moving beyond a discovery metaphor of science to rethink research as enabling realities and to consider how empirical findings enact or perform a reality. An enactment perspective appreciates multiple, dynamic realities and science as producing different entities, enactments that ever encounter differences, uncertainties, and precariousness. The axioms of an enactment perspective are described and employed to more fully understand the two kinds of replication that predominate in the crisis disputes. Although the enactment perspective described here is a relatively recent development in philosophy of science and science studies, some of its core axioms are not new to psychology, and the article concludes by revisiting psychologists' previous calls to apprehend the dynamism of psychological reality to appreciate how scientific practices actively and unavoidably participate in performativity of reality.

Introduction to Bayesian Mindsponge Framework analytics: An innovative method for social and psychological research

The paper introduces Bayesian Mindsponge Framework (BMF) analytics, a new analytical tool for investigating socio, psychological, and behavioral phenomena. The strengths of this method derive from the combination of the mindsponge mechanism's conceptual formulation power and Bayesian analysis's inferential advantages. The BMF-based research procedure includes six main steps, in which the mindsponge-based conceptualization and model construction is the key step that makes the method unique. Therefore, we elaborate on the fundamental components and functions of the mindsponge mechanism and summarize them into five memorable principles so that other researchers can capitalize directly. An exemplary analysis was performed using a dataset of 3071 Vietnamese entrepreneurs' decisiveness and perceptions of the likelihood of success/continuity to validate the method.•The paper provides five strong points of BMF analytics, originating from the good match between the mindsponge mechanism and Bayesian inference.•The paper also provides a step-by-step procedure for conducting BMF-based research.•The mindsponge mechanism's basic components and functions are elaborated and summarized into five core principles that can be applied directly for research conceptualization and model construction.

Investigating Measurement Invariance of the Psychological Entitlement Scale – Grandiose-Based and Vulnerable-Based

Abstract. Entitlement has attracted interest across various social science disciplines due to its broad connection to selfish decision-making outcomes and mental health. Although unidimensional entitlement scales have been widely used, these scales conflate vulnerable- and grandiose-based entitlement forms. The Psychological Entitlement Scale – Grandiose-Based and Vulnerable-Based (PES-G/V) was recently devised to measure these entitlement forms. Prior work has supported the structure and construct validity of the PES-G/V, but no research has addressed the measurement invariance (MI) of the PES-G/V. Hence, we examined MI in relation to gender, two popular sampling frames in psychology studies (US MTurk participants and US college participants), and age. Results supported scalar MI across levels of each of the grouping variables. In sum, the structural properties of the PES-G/V seemed robust to the group distinctions.

Examining the generalizability of research findings from archival data

The extent to which results from complex datasets generalize across contexts is critically important to numerous scientific fields as well as to practitioners who rely on such analyses to guide important strategic decisions. Our initiative systematically investigated whether findings from the field of strategic management would emerge in new time periods and new geographies. Original findings that were statistically reliable in the first place were typically obtained again in novel tests, suggesting surprisingly little sensitivity to context. For some social scientific areas of inquiry, results from a specific time and place can be a meaningful guide as to what will be observed more generally.

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

The Reproducibility Movement in Psychology: Does Researcher Gender Affect How People Perceive Scientists With a Failed Replication?

The reproducibility movement in psychology has resulted in numerous highly publicized instances of replication failures. The goal of the present work was to investigate people’s reactions to a psychology replication failure vs. success, and to test whether a failure elicits harsher reactions when the researcher is a woman vs. a man. We examined these questions in a pre-registered experiment with a working adult sample, a conceptual replication of that experiment with a student sample, and an analysis of data compiled and posted by a psychology researcher on their public weblog with the stated goal to improve research replicability by rank-ordering psychology researchers by their “estimated false discovery risk.” Participants in the experiments were randomly assigned to read a news article describing a successful vs. failed replication attempt of original work from a male vs. female psychological scientist, and then completed measures of researcher competence, likability, integrity, perceptions of the research, and behavioral intentions for future interactions with the researcher. In both working adult and student samples, analyses consistently yielded large main effects of replication outcome, but no interaction with researcher gender. Likewise, the coding of weblog data posted in July 2021 indicated that 66.3% of the researchers scrutinized were men and 33.8% were women, and their rank-ordering was not correlated with researcher gender. The lack of support for our pre-registered gender-replication hypothesis is, at first glance, encouraging for women researchers’ careers; however, the substantial effect sizes we observed for replication outcome underscore the tremendous negative impact the reproducibility movement can have on psychologists’ careers. We discuss the implications of such negative perceptions and the possible downstream consequences for women in the field that are essential for future study.

Testing the reproducibility and robustness of the cancer biology literature by robot

Scientific results should not just be ‘repeatable’ (replicable in the same laboratory under identical conditions), but also ‘reproducible’ (replicable in other laboratories under similar conditions). Results should also, if possible, be ‘robust’ (replicable under a wide range of conditions). The reproducibility and robustness of only a small fraction of published biomedical results has been tested; furthermore, when reproducibility is tested, it is often not found. This situation is termed ‘the reproducibility crisis', and it is one the most important issues facing biomedicine. This crisis would be solved if it were possible to automate reproducibility testing. Here, we describe the semi-automated testing for reproducibility and robustness of simple statements (propositions) about cancer cell biology automatically extracted from the literature. From 12 260 papers, we automatically extracted statements predicted to describe experimental results regarding a change of gene expression in response to drug treatment in breast cancer, from these we selected 74 statements of high biomedical interest. To test the reproducibility of these statements, two different teams used the laboratory automation system Eve and two breast cancer cell lines (MCF7 and MDA-MB-231). Statistically significant evidence for repeatability was found for 43 statements, and significant evidence for reproducibility/robustness in 22 statements. In two cases, the automation made serendipitous discoveries. The reproduced/robust knowledge provides significant insight into cancer. We conclude that semi-automated reproducibility testing is currently achievable, that it could be scaled up to generate a substantive source of reliable knowledge and that automation has the potential to mitigate the reproducibility crisis.

Crisis Ahead? Why Human-Robot Interaction User Studies May Have Replicability Problems and Directions for Improvement

There is a confidence crisis in many scientific disciplines, in particular disciplines researching human behavior, as many effects of original experiments have not been replicated successfully in large-scale replication studies. While human-robot interaction (HRI) is an interdisciplinary research field, the study of human behavior, cognition and emotion in HRI plays also a vital part. Are HRI user studies facing the same problems as other fields and if so, what can be done to overcome them? In this article, we first give a short overview of the replicability crisis in behavioral sciences and its causes. In a second step, we estimate the replicability of HRI user studies mainly 1) by structural comparison of HRI research processes and practices with those of other disciplines with replicability issues, 2) by systematically reviewing meta-analyses of HRI user studies to identify parameters that are known to affect replicability, and 3) by summarizing first replication studies in HRI as direct evidence. Our findings suggest that HRI user studies often exhibit the same problems that caused the replicability crisis in many behavioral sciences, such as small sample sizes, lack of theory, or missing information in reported data. In order to improve the stability of future HRI research, we propose some statistical, methodological and social reforms. This article aims to provide a basis for further discussion and a potential outline for improvements in the field.

Troll and divide: the language of online polarization

The affective animosity between the political left and right has grown steadily in many countries over the past few years, posing a threat to democratic practices and public health. There is a rising concern over the role that "bad actors" or trolls may play in the polarization of online networks. In this research, we examined the processes by which trolls may sow intergroup conflict through polarized rhetoric. We developed a dictionary to assess online polarization by measuring language associated with communications that display partisan bias in their diffusion. We validated the polarized language dictionary in 4 different contexts and across multiple time periods. The polarization dictionary made out-of-set predictions, generalized to both new political contexts (#BlackLivesMatter) and a different social media platform (Reddit), and predicted partisan differences in public opinion polls about COVID-19. Then we analyzed tweets from a known Russian troll source (N = 383,510) and found that their use of polarized language has increased over time. We also compared troll tweets from 3 countries (N = 79,833) and found that they all utilize more polarized language than regular Americans (N = 1,507,300) and trolls have increased their use of polarized rhetoric over time. We also find that polarized language is associated with greater engagement, but this association only holds for politically engaged users (both trolls and regular users). This research clarifies how trolls leverage polarized language and provides an open-source, simple tool for exploration of polarized communications on social media.

Caution, Preprint! Brief Explanations Allow Nonscientists to Differentiate Between Preprints and Peer-Reviewed Journal Articles

A growing number of psychological research findings are initially published as preprints. Preprints are not peer reviewed and thus did not undergo the established scientific quality-control process. Many researchers hence worry that these preprints reach nonscientists, such as practitioners, journalists, and policymakers, who might be unable to differentiate them from the peer-reviewed literature. Across five studies in Germany and the United States, we investigated whether this concern is warranted and whether this problem can be solved by providing nonscientists with a brief explanation of preprints and the peer-review process. Studies 1 and 2 showed that without an explanation, nonscientists perceive research findings published as preprints as equally credible as findings published as peer-reviewed articles. However, an explanation of the peer-review process reduces the credibility of preprints (Studies 3 and 4). In Study 5, we developed and tested a shortened version of this explanation, which we recommend adding to preprints. This explanation again allowed nonscientists to differentiate between preprints and the peer-reviewed literature. In sum, our research demonstrates that even a short explanation of the concept of preprints and their lack of peer review allows nonscientists who evaluate scientific findings to adjust their credibility perception accordingly. This would allow harvesting the benefits of preprints, such as faster and more accessible science communication, while reducing concerns about public overconfidence in the presented findings.

Interventions for Rural Caregiver Health: An Integrative Review

Few health behavior interventions exist for rural caregivers of persons with dementia (PWD) in the United States. Of interventions that have been studied, little is known about health outcomes that are included in interventions and which interventions are most effective on health outcomes. An integrative literature review of intervention studies for rural caregiver health throughout the United States was undertaken with an emphasis on concepts related to health promotion, self-management, self-care, and/or self-efficacy. Findings from this review indicated multicomponent interventions that included self-care strategies were associated with improvements of caregiver psychosocial health outcomes of depression, burden, self-efficacy, social support, and self-rated health. Sleep problems and endurance potential were the only physical health outcomes measured. To prevent adverse physical health outcomes, interventions with attention to rural context are needed that emphasize health promotion with a focus on health behaviors and health outcome measurement in rural caregivers of PWD. [Research in Gerontological Nursing, xx(x), xx-xx.].

The role of generalizability in moral and political psychology

The aim of the social and behavioral sciences is to understand human behavior across a wide array of contexts. Our theories often make sweeping claims about human nature, assuming that our ancestors or offspring will be prone to the same biases and preferences. Yet we gloss over the fact that our research is often based in a single temporal context with a limited set of stimuli. Political and moral psychology are domains in which the context and stimuli are likely to matter a great deal (Van Bavel, Mende-Siedlecki, Brady, & Reinero, 2016). In response to Yarkoni (see BBS issue), we delve into topics related to political and moral psychology that likely depend on features of the research. These topics include understanding differences between liberals and conservatives, when people are willing to sacrifice someone to save others, the behavior of political leaders, and the dynamics of intergroup conflict.

The puzzling relationship between multi-laboratory replications and meta-analyses of the published literature

What is the best way to estimate the size of important effects? Should we aggregate across disparate findings using statistical meta-analysis, or instead run large, multi-laboratory replications (MLR)? A recent paper by Kvarven, Strømland and Johannesson (Kvarven et al. 2020 Nat. Hum. Behav. 4, 423-434. (doi:10.1038/s41562-019-0787-z)) compared effect size estimates derived from these two different methods for 15 different psychological phenomena. The authors reported that, for the same phenomenon, the meta-analytic estimate tended to be about three times larger than the MLR estimate. These results are a specific example of a broader question: What is the relationship between meta-analysis and MLR estimates? Kvarven et al. suggested that their results undermine the value of meta-analysis. By contrast, we argue that both meta-analysis and MLR are informative, and that the discrepancy between the two estimates that they observed is in fact still largely unexplained. Informed by re-analyses of Kvarven et al.'s data and by other empirical evidence, we discuss possible sources of this discrepancy and argue that understanding the relationship between estimates obtained from these two methods is an important puzzle for future meta-scientific research.

Toward a cumulative science of vocal markers of autism: A cross-linguistic meta-analysis-based investigation of acoustic markers in American and Danish autistic children

Acoustic atypicalities in speech production are argued to be potential markers of clinical features in autism spectrum disorder (ASD). A recent meta-analysis highlighted shortcomings in the field, in particular small sample sizes and study heterogeneity. We showcase a cumulative (i.e., explicitly building on previous studies both conceptually and statistically) yet self-correcting (i.e., critically assessing the impact of cumulative statistical techniques) approach to prosody in ASD to overcome these issues. We relied on the recommendations contained in the meta-analysis to build and analyze a cross-linguistic corpus of multiple speech productions in 77 autistic and 72 neurotypical children and adolescents (>1000 recordings in Danish and US English). We used meta-analytically informed and skeptical priors, with informed priors leading to more generalizable inference. We replicated findings of a minimal cross-linguistically reliable distinctive acoustic profile for ASD (higher pitch and longer pauses) with moderate effect sizes. We identified novel reliable differences between the two groups for normalized amplitude quotient, maxima dispersion quotient, and creakiness. However, the differences were small, and there is likely no one acoustic profile characterizing all autistic individuals. We identified reliable relations of acoustic features with individual differences (age, gender), and clinical features (speech rate and ADOS sub-scores). Besides cumulatively building our understanding of acoustic atypicalities in ASD, the study shows how to use systematic reviews and meta-analyses to guide the design and analysis of follow-up studies. We indicate future directions: larger and more diverse cross-linguistic datasets, focus on heterogeneity, self-critical cumulative approaches, and open science. LAY SUMMARY: Autistic individuals are reported to speak in distinctive ways. Distinctive vocal production can affect social interactions and social development and could represent a noninvasive way to support the assessment of autism spectrum disorder (ASD). We systematically checked whether acoustic atypicalities highlighted in previous articles could be actually found across multiple recordings and two languages. We find a minimal acoustic profile of ASD: higher pitch, longer pauses, increased hoarseness and creakiness of the voice. However, there is much individual variability (by age, sex, language, and clinical characteristics). This suggests that the search for one common "autistic voice" might be naive and more fine-grained approaches are needed.

Psychologists update their beliefs about effect sizes after replication studies

Self-correction-a key feature distinguishing science from pseudoscience-requires that scientists update their beliefs in light of new evidence. However, people are often reluctant to change their beliefs. We examined belief updating in action by tracking research psychologists' beliefs in psychological effects before and after the completion of four large-scale replication projects. We found that psychologists did update their beliefs; they updated as much as they predicted they would, but not as much as our Bayesian model suggests they should if they trust the results. We found no evidence that psychologists became more critical of replications when it would have preserved their pre-existing beliefs. We also found no evidence that personal investment or lack of expertise discouraged belief updating, but people higher on intellectual humility updated their beliefs slightly more. Overall, our results suggest that replication studies can contribute to self-correction within psychology, but psychologists may underweight their evidentiary value.

Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks

Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.

Replicability, Robustness, and Reproducibility in Psychological Science

Replication-an important, uncommon, and misunderstood practice-is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Optimizing Research Output: How Can Psychological Research Methods Be Improved?

Recent evidence suggests that research practices in psychology and many other disciplines are far less effective than previously assumed, which has led to what has been called a "crisis of confidence" in psychological research (e.g., Pashler & Wagenmakers 2012). In response to the perceived crisis, standard research practices have come under intense scrutiny, and various changes have been suggested to improve them. The burgeoning field of metascience seeks to use standard quantitative data-gathering and modeling techniques to understand the reasons for inefficiency, to assess the likely effects of suggested changes, and ultimately to tell psychologists how to do better science. We review the pros and cons of suggested changes, highlighting the many complex research trade-offs that must be addressed to identify better methods. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

How to True Psychology’s Objects

Psychology’s current crisis attends most visibly to perceived problems with statistical models, methods, publication practices, and career incentives. Rarely is close attention given to the objects of inquiry—to ontological matters—yet the crisis-related literature does features statements about the nature of psychology’s objects. Close analysis of the ontological claims reveals discrepant understandings: some researchers assume objects to be stable and singular while others posit them to be dynamic and complex. Nevertheless, both views presume the objects under scrutiny to be real. The analysis also finds each of these ontological claims to be associated not only with particular method prescriptions but also with distinct notions of the scientific self. Though both take the scientific self to be objective, one figures the scientist as not always a rational actor and, therefore, requiring some behavior regulation, while the other sees the scientist as largely capable of self-governing sustained through painstakingly acquired expertise and self-control. The fate of these prevalent assemblages of object, method, and scientific self remains to be determined, yet as conditions of possibility they portend quite different futures. Following description of the assemblages, the article ventures a futuristic portrayal of the scientific practices they each might engender.

Objectivity for the research worker

In the last decade, many problematic cases of scientific conduct have been diagnosed; some of which involve outright fraud (e.g., Stapel, 2012) others are more subtle (e.g., supposed evidence of extrasensory perception; Bem, 2011). These and similar problems can be interpreted as caused by lack of scientific objectivity. The current philosophical theories of objectivity do not provide scientists with conceptualizations that can be effectively put into practice in remedying these issues. We propose a novel way of thinking about objectivity for individual scientists; a negative and dynamic approach.We provide a philosophical conceptualization of objectivity that is informed by empirical research. In particular, it is our intention to take the first steps in providing an empirically and methodologically informed inventory of factors that impair the scientific practice. The inventory will be compiled into a negative conceptualization (i.e., what is not objective), which could in principle be used by individual scientists to assess (deviations from) objectivity of scientific practice. We propose a preliminary outline of a usable and testable instrument for indicating the objectivity of scientific practice.

On Methods and Marshmallows: A Roadmap for Science That Is Openly Feminist and Radically Open

In this commentary, we argue that feminist science and open science can benefit from each other’s wisdom and critiques in service of creating systems that produce the highest quality science with the maximum potential for improving the lives of women. To do this, we offer a constructive analysis, focusing on common methods used in open science, including open materials and data, preregistration, and large sample sizes, and illuminate potential benefits and costs from a feminist science perspective. We also offer some solutions and deeper questions both for individual researchers and the feminist psychology and open science communities. By broadening our focus from a myopic prioritization of certain methodological and analytic approaches in open science, we hope to give a balanced perspective of science that emerges from each movement’s strengths and is openly feminist and radically open.

Bringing an Intersectional Lens to “Open” Science: An Analysis of Representation in the Reproducibility Project

Feminist psychologists have called for researchers to consider the social and historical context and the multidimensionality of participants in research studies. The Reproducibility Project documents the degree to which the findings from mainstream psychological studies are reproduced. Drawing on intersectionality theory, we question the value of reproducing findings while ignoring who is represented, intersecting social and group identities, sociohistorical context, and the power and privilege that likely influence participants’ responses in psychology experiments. To critically examine the Reproducibility Project in psychology, we analyzed the 100 replication reports produced between 2011 and 2014 (Open Science Collaboration, 2015). We developed an intersectional analytic framework to investigate (a) representation, (b) whether demographic and identity factors were considered through a multidimensional or intersectional lens, (c) explanations of non-replication, and (d) whether socio-cultural context was considered. Results show that reports predominantly include WEIRD samples (people from Western, educated, industrialized, rich, and democratic countries). Context and identity were rarely considered, even when study design relied on these factors, and intersectional identities and structures (considering power, structural issues, discrimination, and historical context) were absent from nearly all reports. Online slides for instructors who want to use this article for teaching are available on PWQ's website at http://journals.sagepub.com/doi/suppl/10.1177/03616843211035678

Portuguese Version of Parker's (1998) Role Breadth Self-Efficacy Scale: Examining Its Factor Structure, Reliability and Validity to Predict Innovative Work Behaviour

Drawing upon previous theorizing and empirical research efforts supporting the importance of role breadth self-efficacy (RBSE) as a key motivational antecedent of proactive forms of behaviour at work, including individual innovation, this paper reports the process of building and validating a Portuguese version of Parker's RBSE scale. Data from two studies with independent samples of employees (N = 103, N = 148) pertaining to an information technology company, showed that this version (RBSE-P) has, as expected, a unidimensional structure with a high level of factor structure similarity in both samples, as well as appropriate levels of internal consistency. Furthermore, evidence from the second sample (N = 148) revealed that, in line with preliminary research, this construct shows validity to predict the distinctive proactive behaviour of individual innovation, evaluated by the employee's corresponding direct supervisors. These results support the use of this scale for assessing RBSE in Portuguese-speaking work settings and the meaningful role of this construct for predicting this criterion, which is acknowledged as an important driver of organizational success and competitiveness.

The effect of testosterone on economic risk-taking: A multi-study, multi-method investigation

Testosterone has been suggested to influence individuals' economic decision making, yet the effects of testosterone on economic behavior are not well-understood and existing research is equivocal. In response, in three studies, we examined the extent to which testosterone affected or was associated with several different facets of economic decision making. Study 1 was a double-blind, placebo-controlled, within-subjects study examining loss aversion and risk-taking (N = 26), whereas Study 2 was a larger double-blind, placebo-controlled, between-subjects study examining loss aversion and risk-taking behavior (N = 117). As a methodological compliment, Study 3 was a larger correlational design (N = 213) with a highly accurate measure of endogenous testosterone examining a wider range of economic behaviors and trait-like preferences. Broadly, the results of all three studies suggest no consistent relationship between testosterone and financial behavior or preferences. Although there were significant effects in specific cases, these findings did not replicate in other studies or would not remain significant when controlling for family-wise error rate. We consider potential contextual moderators that may determine under what circumstances testosterone affects economic decision making.

Behavioural science is unlikely to change the world without a heterogeneity revolution

In the past decade, behavioural science has gained influence in policymaking but suffered a crisis of confidence in the replicability of its findings. Here, we describe a nascent heterogeneity revolution that we believe these twin historical trends have triggered. This revolution will be defined by the recognition that most treatment effects are heterogeneous, so the variation in effect estimates across studies that defines the replication crisis is to be expected as long as heterogeneous effects are studied without a systematic approach to sampling and moderation. When studied systematically, heterogeneity can be leveraged to build more complete theories of causal mechanism that could inform nuanced and dependable guidance to policymakers. We recommend investment in shared research infrastructure to make it feasible to study behavioural interventions in heterogeneous and generalizable samples, and suggest low-cost steps researchers can take immediately to avoid being misled by heterogeneity and begin to learn from it instead.