Replication, falsification, and the crisis of confidence in social psychology

A model of the interrelationship between research ethics and research integrity

Purpose: The purpose of this article is to explore the interrelationship between research ethics and research integrity with a focus on the primary forms of research misconduct, including plagiarism, fabrication, and falsification. It also details the main factors for their occurrence, and the possible ways for mitigating their use among scholars.Methods: The method employed a detailed examination of the main ethical dilemmas, as delineated in literature, as well as the factors leading to these ethical breaches and the strategies to mitigate them. Further, the teaching experiences of the primary author are reflected in the development of the model.Results: The results of this article are represented in a model illustrating the interrelationship between research ethics and research integrity. Further, a significant aspect of our article is the identification of novel forms of research misconduct concerning the use of irrelevant or forced citations or references.Conclusion: In conclusion, the article highlights the substantial positive effects that adherence to research ethics and integrity have on the academic well-being of scholars.

Using theoretical constraints and the TASI taxonomy to delineate predictably replicable findings

The focus of the present article is not on failures to replicate but on the more optimistically framed and more fruitful question: What stable findings can be reproduced reliably and can be trusted by decision makers, managers, health agents, or politicians? We propagate the working hypothesis that a twofold key to stable and replicable findings lies in the existence of theoretical constraints and, no less important, in researchers' sensitivity to metatheoretical, auxiliary assumptions. We introduce a hierarchy of four levels of theoretical constraints-a priori principles, psychophysical, empirical, and modelling constraints-combined with the TASI taxonomy of theoretical, auxiliary, statistical, and inferential assumptions Trafimow, Journal for the Theory of Social Behaviour, 52, 37-48, (2022). Although theoretical constraints clearly facilitate stable and replicable research findings, TASI reminds us of various reasons why even perfectly valid hypotheses need not always be borne out. The presented framework should help researchers to operationalize conditions under which theoretical constraints render empirical findings most predictable.

Prioritising and incentivising productivity within indicator-based approaches to Research Impact Assessment: a commentary

Research Impact Assessment (RIA) represents one of a suite of policies intended to improve the impact generated from investment in health and medical research (HMR). Positivist indicator-based approaches to RIA are widely implemented but increasingly criticised as theoretically problematic, unfair, and burdensome. This commentary proposes there are useful outcomes that emerge from the process of applying an indicator-based RIA framework, separate from those encapsulated in the metrics themselves. The aim for this commentary is to demonstrate how the act of conducting an indicator-based approach to RIA can serve to optimise the productive gains from the investment in HMR. Prior research found that the issues regarding RIA are less about the choice of indicators/metrics, and more about the discussions prompted and activities incentivised by the process. This insight provides an opportunity to utilise indicator-based methods to purposely optimise the research impact. An indicator-based RIA framework specifically designed to optimise research impacts should: focus on researchers and the research process, rather than institution-level measures; utilise a project level unit of analysis that provides control to researchers and supports collaboration and accountability; provide for prospective implementation of RIA and the prospective orientation of research; establish a line of sight to the ultimate anticipated beneficiaries and impacts; Include process metrics/indicators to acknowledge interim steps on the pathway to final impacts; integrate 'next' users and prioritise the utilisation of research outputs as a critical measure; Integrate and align the incentives for researchers/research projects arising from RIA, with those existing within the prevailing research system; integrate with existing peer-review processes; and, adopt a system-wide approach where incremental improvements in the probability of translation from individual research projects, yields higher impact across the whole funding portfolio.Optimisation of the impacts from HMR investment represents the primary purpose of Research Impact policy. The process of conducting an indicator-based approach to RIA, which engages the researcher during the inception and planning phase, can directly contribute to this goal through improvements in the probability that an individual project will generate interim impacts. The research project funding process represents a promising forum to integrate this approach within the existing research system.

Assessing the Evidential Value of Mental Fatigue and Exercise Research

It has often been reported that mental exertion, presumably leading to mental fatigue, can negatively affect exercise performance; however, recent findings have questioned the strength of the effect. To further complicate this issue, an overlooked problem might be the presence of publication bias in studies using underpowered designs, which is known to inflate false positive report probability and effect size estimates. Altogether, the presence of bias is likely to reduce the evidential value of the published literature on this topic, although it is unknown to what extent. The purpose of the current work was to assess the evidential value of studies published to date on the effect of mental exertion on exercise performance by assessing the presence of publication bias and the observed statistical power achieved by these studies. A traditional meta-analysis revealed a Cohen's dz effect size of - 0.54, 95% CI [- 0.68, - 0.40], p < .001. However, when we applied methods for estimating and correcting for publication bias (based on funnel plot asymmetry and observed p-values), we found that the bias-corrected effect size became negligible with most of publication-bias methods and decreased to - 0.36 in the more optimistic of all the scenarios. A robust Bayesian meta-analysis found strong evidence in favor of publication bias, BFpb > 1000, and inconclusive evidence in favor of the effect, adjusted dz = 0.01, 95% CrI [- 0.46, 0.37], BF10 = 0.90. Furthermore, the median observed statistical power assuming the unadjusted meta-analytic effect size (i.e., - 0.54) as the true effect size was 39% (min = 19%, max = 96%), indicating that, on average, these studies only had a 39% chance of observing a significant result if the true effect was Cohen's dz = - 0.54. If the more optimistic adjusted effect size (- 0.36) was assumed as the true effect, the median statistical power was just 20%. We conclude that the current literature is a useful case study for illustrating the dangers of conducting underpowered studies to detect the effect size of interest.

The role of free will beliefs in social behavior: Priority areas for future research

Recent research has examined the consequences that holding views about free will has on social behavior. Specifically, through manipulating people's belief in free will, researchers have tested the psychological and behavioral consequences of free will belief change. However, findings of such manipulations have been shown to be relatively small and inconsistent. The purpose of this paper is to outline four key areas for researchers in this area to consider. We believe considering these areas will give a more nuanced understanding of the role of free will beliefs.

The trustworthiness of the cumulative knowledge in industrial/organizational psychology: The current state of affairs and a path forward

The goal of industrial/organizational (IO) psychology, is to build and organize trustworthy knowledge about people-related phenomena in the workplace. Unfortunately, as with other scientific disciplines, our discipline may be experiencing a "crisis of confidence" stemming from the lack of reproducibility and replicability of many of our field's research findings, which would suggest that much of our research may be untrustworthy. If a scientific discipline's research is deemed untrustworthy, it can have dire consequences, including the withdraw of funding for future research. In this focal article, we review the current state of reproducibility and replicability in IO psychology and related fields. As part of this review, we discuss factors that make it less likely that research findings will be trustworthy, including the prevalence of scientific misconduct, questionable research practices (QRPs), and errors. We then identify some root causes of these issues and provide several potential remedies. In particular, we highlight the need for improved research methods and statistics training as well as a re-alignment of the incentive structure in academia. To accomplish this, we advocate for changes in the reward structure, improvements to the peer review process, and the implementation of open science practices. Overall, addressing the current "crisis of confidence" in IO psychology requires individual researchers, academic institutions, and publishers to embrace system-wide change.

Evaluating the Evidence for Enclothed Cognition: Z-Curve and Meta-Analyses

Enclothed cognition refers to the systematic influence that clothes can have on the wearer's feelings, thoughts, and behaviors through their symbolic meaning. It has attracted considerable academic and nonacademic interest, with the 2012 article that coined the phrase cited more than 600 times and covered in more than 160 news outlets. However, a recent high-powered replication failed to replicate one of the original effects. To determine whether the larger body of research on enclothed cognition possesses evidential value and replicable effects, we performed z-curve and meta-analyses using 105 effects from 40 studies across 24 articles (N = 3,789). Underscoring the marked improvement of psychological research practices in the mid-2010s, our results raise concerns about the replicability of early enclothed cognition studies but affirm the evidential value for effects published after 2015. These later studies support the core principle of enclothed cognition-what we wear influences how we think, feel, and act.

Follow-up on the U.S. Central Intelligence Agency's (CIA) remote viewing experiments☆

OBJECTIVES

Since 1972, the U.S. Central Intelligence Agency (CIA) commissioned several research programs on remote viewing (RV) that were progressively declassified from 1995 to 2003. The main objectives of this research were to statistically replicate the original findings and address the question: What are the underlying cognitive mechanisms involved in RV? The research focused on emotional intelligence (EI) theory and intuitive information processing as possible hypothetical mechanisms.

METHODS

We used a quasi-experimental design with new statistical control techniques based on structural equation modeling, analysis of invariance, and forced-choice experiments to accurately objectify results. We measured emotional intelligence with the Mayer-Salovey-Caruso Emotional Intelligence Test. A total of 347 participants who were nonbelievers in psychic experiences completed an RV experiment using targets based on location coordinates. A total of 287 participants reported beliefs in psychic experiences and completed another RV experiment using targets based on images of places. Moreover, we divided the total sample into further subsamples for the purpose of replicating the findings and also used different thresholds on standard deviations to test for variation in effect sizes. The hit rates on the psi-RV task were contrasted with the estimated chance.

RESULTS

The results of our first group analysis were nonsignificant, but the analysis applied to the second group produced significant RV-related effects corresponding to the positive influence of EI (i.e., hits in the RV experiments were 19.5% predicted from EI) with small to moderate effect sizes (between 0. 457 and 0.853).

CONCLUSIONS

These findings have profound implications for a new hypothesis of anomalous cognitions relative to RV protocols. Emotions perceived during RV sessions may play an important role in the production of anomalous cognitions. We propose the Production-Identification-Comprehension (PIC) emotional model as a function of behavior that could enhance VR test success.

Theory of brain complexity and marital behaviors: The application of complexity science and neuroscience to explain the complexities of marital behaviors

The extant theories on the quality and stability of marital relationships have some difficulties in explaining some of the complexities of marital behaviors. The present article is an initial attempt to explain the complexities of marital behaviors based on the science of complexity and neuroscience. This article proposes a new theoretical framework relying on this simple argument that marital behaviors, as one of the most complex human behaviors, are the product of one's brain's complex adaptive system (CAS). Hence, to understand the complexities of marital behaviors, a movement toward familiarity with the brain's CAS involved in marital behaviors needs to be started. The article presents the theory of brain complexity and marital behaviors (BCM) and outlines its assumptions, concepts, and propositions. Then, BCM is compared with the extant theories on happy and stable marriage, and finally, it was concluded by discussing the testability and the potential application of the theory. This article might inspire interdisciplinary studies of marital relationships, complex systems, and neuroscience and may have considerable practical implications.

The Bangor Gambling Task: Computerized replication and reappraisal of an emotion-based decision task

Emotion-based decision making (EBDM) is the capacity to make decisions based on prior emotional consequences of actions. Several neuropsychological tasks, using different gambling paradigms and with different levels of complexity, have been designed to assess EBDM. The Bangor Gambling Task (BGT) was created as a brief and simple card gambling-task to assess EBDM. BGT contains a single-card deck and requires participants to decide whether to gamble or not, which can result in wins or losses. Unknown to the participant, the winning probabilities decrease throughout the task (from 0.75 in the first block to 0.25 in the fifth block), requiring participants to reduce their gambling probability to avoid long-term losses. A few studies have offered evidence regarding the BGT convergent validity. However, there are no computerized versions of BGT available, thus slowing the process of gathering information to explore the EBDM mechanisms behind the task, its validity, and clinical usefulness. In this article, we present a computerized version of the BGT using the Matlab environment and make all our code available. We explore BGT's replicability and analyze its probabilistic structure, providing trial-level and block-level analyses. Eighty-one participants performed the computerized version, which followed the same structure as the original version. It took participants 8.5 ± 3.3 minutes to complete the task, which is faster than the paper version. Replicating previous studies, participants diminished their gambling probability throughout the task, learning to inhibit the initially rewarded gambling behavior. This change in gambling probability could be considered a proxy for EBDM. Our analyses suggest that the last blocks are especially sensitive to capturing deficits in EBDM, and we propose some modifications to BGT's original version to enhance the initial exploratory and learning phase. Our results show that the BGT constitutes a quick and simple task to evaluate EBDM capacities.

Promoting Liberal Education through Introductory Psychology: The Perspective-Based Approach

Introductory psychology is typically presented to undergraduates as a set of loosely related topics reflecting the organization of most textbooks. The empirically based evidence presented in the topical format is likely to be limited by progressive knowledge obsolesce and replicability challenges impacting contemporary science. We suggest that psychology instructors consider an alternative course format organized by theories or perspectives. A perspective-centered introductory psychology provides a consistent and pluralistic view of the field and embeds research findings and methodology within psychology’s major explanatory theories. Current trends in undergraduate higher education emphasize career readiness. Development of critical thinking and applying psychological principles to real-world situations, including students’ own lives, may be better achieved through a perspective-based approach. Given that the typical introductory psychology student is a non-major taking the course to fulfil liberal arts requirements, a perspective approach offers several advantages. A focus on understanding human behavior from multiple vantage points is a skill valued by prospective employers of new bachelor’s degree recipients. With its emphasis on divergent and dialectical reasoning, a perspectivecenteredintroductory psychology course can become the centerpiece of the general education curriculum. In addition to promoting post-formal reasoning, a perspective-centered introductory psychology illustrates the diverse epistemologies that have shaped our field.

Proposal of a Selection Protocol for Replication of Studies in Sports and Exercise Science

INTRODUCTION

To improve the rigor of science, experimental evidence for scientific claims ideally needs to be replicated repeatedly with comparable analyses and new data to increase the collective confidence in the veracity of those claims. Large replication projects in psychology and cancer biology have evaluated the replicability of their fields but no collaborative effort has been undertaken in sports and exercise science. We propose to undertake such an effort here. As this is the first large replication project in this field, there is no agreed-upon protocol for selecting studies to replicate. Criticism of previous selection protocols include claims they were non-randomised and non-representative. Any selection protocol in sports and exercise science must be representative to provide an accurate estimate of replicability of the field. Our aim is to produce a protocol for selecting studies to replicate for inclusion in a large replication project in sports and exercise science.

METHODS

The proposed selection protocol uses multiple inclusion and exclusion criteria for replication study selection, including: the year of publication and citation rankings, research disciplines, study types, the research question and key dependent variable, study methods and feasibility. Studies selected for replication will be stratified into pools based on instrumentation and expertise required, and will then be allocated to volunteer laboratories for replication. Replication outcomes will be assessed using a multiple inferential strategy and descriptive information will be reported regarding the final number of included and excluded studies, and original author responses to requests for raw data.

Psychology: Where history, culture, and biology meet

This article argues that the same epistemological assumptions cannot be confidently applied in the transition from the biological to the social arenas of psychology, as a consequence of the sociocultural instability resulting from human linguistic and technological flair. To illustrate this contention, reference is made to historicist theses within critical and sociocultural psychology, the work of Ian Hacking and Norbert Elias, the centrality of language and technology to sociocultural instability, and the illustrative issues raised by cultural neuroscience and replication studies.

Putting down the revolt: Enactivism as a philosophy of nature

Enactivists frequently argue their account heralds a revolution in cognitive science: enactivism will unseat cognitivism as the dominant paradigm. We examine the lines of reasoning enactivists employ in stirring revolt, but show that none of these prove compelling reasons for cognitivism to be replaced by enactivism. First, we examine the hard sell of enactivism: enactivism reveals a critical explanatory gap at the heart of cognitivism. We show that enactivism does not meet the requirements to incite a paradigm shift in the Kuhnian sense—there is no internal crisis in cognitivism. Nor does it provide inherently better explanations of cognition as some have claimed. Second, we consider the soft sell of enactivism: enactivism provides a more attractive, parsimonious, or clear-eyed lens on cognition. This move proves to boil down to a misunderstanding of how theories are selected in science. Instead we lend support to a broader and more desirable way to conceive of enactivism, the recent proposal that enactivism is a philosophy of nature. We explain how a philosophy of nature does more than support a single research paradigm by integrating scientific questions into a cohesive picture.

Warm-Up and Handgrip Strength in Physically Inactive Chilean Older Females According to Baseline Nutritional Status

This study aims to analyze the effect of different types of warm-ups on handgrip strength (HGS) in physically inactive older females. Secondarily, it aims to compare HGS according to their baseline nutritional status. A randomized crossover trial study was conducted with 44 physically inactive older females distributed into normal weight (n = 12, BMI = 23.9 ± 3.2 kg/m²), overweight (n =16, BMI = 27 ± 4.7 kg/m²) and obese (n = 16, BMI = 31.6 ± 5.3 kg/m²), who participated in three warm-up conditions (static stretching condition, SSC; elastic band condition, EBC; and therapeutic compression ball condition, TCBC) and one control condition (CC, no warm-up). All participants performed the four randomized conditions with recovery within 72 h. A significant decrease (p < 0.05) in HGS for the dominant and non-dominant hands was observed when comparing SSC vs. CC. In contrast, comparing the warm-up conditions according to the baseline nutritional status, statistically significant differences (p < 0.05) were only reported in the obese group in the dominant and non-dominant hand in favor of CC concerning SSC. In conclusion, warm-up with static flexibility led to a decrease in HGS in physically inactive older females. Only the obese group exhibited this result when analyzed by nutritional status.

Social Influence and Autobiographical Recall: Shared Reality and Epistemic Trust Shape Perceptions of Autobiographical Events

Two experiments studied the evaluative adaptation process at the outset of a communication event to examine how autobiographical memory could be shaped by audience attitude, shared reality, and epistemic trust. Experiment 1 found that audience attitude influenced communicator perceptions of their own autobiographical memories and attitudes toward the memory topic. These effects were more pronounced when communicators experienced a shared reality with their audience. Experiment 2 found that epistemic trust in the audience increased shared reality with the audience, which in turn led to greater audience-congruent memory bias in which communicators had memory perceptions and attitudes that were evaluatively consistent with the attitudes of their audience. This project underscores the prevalence of social influence processes in autobiographical recall and identifies how processes that occur at the initial stages of interpersonal communication (i.e., perceived audience attitude, trust, and shared reality) can influence how individuals construe their own life events.

Science Forum: How failure to falsify in high-volume science contributes to the replication crisis

The number of scientific papers published every year continues to increase, but scientific knowledge is not progressing at the same rate. Here we argue that a greater emphasis on falsification – the direct testing of strong hypotheses – would lead to faster progress by allowing well-specified hypotheses to be eliminated. We describe an example from neuroscience where there has been little work to directly test two prominent but incompatible hypotheses related to traumatic brain injury. Based on this example, we discuss how building strong hypotheses and then setting out to falsify them can bring greater precision to the clinical neurosciences, and argue that this approach could be beneficial to all areas of science.

The Role of Mating-relevant Factors in the Perpetration of Digital Dating Abuse

Previous research has explored offline intimate partner violence from an evolutionary perspective, primarily focusing on the role of individual differences in perpetration and victimization. However, a current form of intimate partner violence is digital dating abuse, which involves abuse toward a romantic partner, occuring online through the use of electronic communication technology. This form of abuse differs from offline abuse, in that physical proximity is not required. Although research has focused on the effects digital dating abuse has on victims, little research has focused on the perpetration of digital dating abuse. This is important as research focused on perpetration can inform a wide range of initiatives geared toward understanding the factors which drive this behavior. Recent research has focused on evolutionary mating-relevant factors that drive the perpetration of digital dating abuse. Here, we extended and replicated previous work by reporting two studies (study 1, n = 114; study 2, n = 162) which explored the roles of mate value discrepancy, intrasexual competition, and relationship-contingent self-esteem in the perpetration of digital dating abuse. We found that mate value discrepancy (studies 1 and 2) and intrasexual competition (study 2) positively predicted the perpetration of digital dating abuse. To our knowledge, this article is the first to provide support that those who report high intrasexual competition, engage in greater levels of digital dating abuse, thus furthering theoretical advancements in this field by showing digital dating abuse is a mate retention tactic. Our findings further our understanding of online behavior in romantic relationships through an evolutionary psychological lens.

Before/after Bayes: A comparison of frequentist and Bayesian mixed-effects models in applied psychological research

Bayesian methods are becoming increasingly used in applied psychological research. Previous researchers have thoroughly written about much of the details already, including the philosophy underlying Bayesian methods, computational issues associated with Bayesian model estimation, Bayesian model development and summary, and the role of Bayesian methods in the so-called replication crisis. In this paper, we seek to provide case studies comparing the use of frequentist methods to the use of Bayesian methods in applied psychological research. These case studies are intended to 'illustrate by example' the ways that Bayesian modelling differs from frequentist modelling and the differing conclusions that one may arrive at using the two methods. The intended audience is applied psychological researchers who have been trained in the traditional frequentist framework, who are familiar with mixed-effects models and who are curious about how statistical results might look in a Bayesian context. Along with our case studies, we provide general opinions and guidance on the use of Bayesian methods in applied psychological research.

The unappreciated relevance of auxiliary assumptions for evaluating theory-based interventions in health psychology

The use of theory in health behavior change interventions has been recently questioned with mixed results found for theory-based intervention effectiveness. But theory testing in intervention depends on not only theoretical assumptions, but on auxiliary assumptions too. Specifically, auxiliary assumptions are required to traverse the distance from nonobservational terms in theories and observational terms at the level of the empirical hypotheses in interventions. We believe intervention failures are often due to flaws in auxiliary assumptions rather than assumptions at the theoretical level. We use the theory of planned behavior to illustrate how the consideration of these auxiliary assumptions is important to appraise the effectiveness of interventions informed by theory. We hope that bringing attention to the importance of auxiliary assumptions provides a more nuanced and accurate appraisal of theory utility.

Kinds of Replication: Examining the Meanings of "Conceptual Replication" and "Direct Replication"

Although psychology’s recent crisis has been attributed to various scientific practices, it has come to be called a “replication crisis,” prompting extensive appraisals of this putatively crucial scientific practice. These have yielded disagreements over what kind of replication is to be preferred and what phenomena are being explored, yet the proposals are all grounded in a conventional philosophy of science. This article proposes another avenue that invites moving beyond a discovery metaphor of science to rethink research as enabling realities and to consider how empirical findings enact or perform a reality. An enactment perspective appreciates multiple, dynamic realities and science as producing different entities, enactments that ever encounter differences, uncertainties, and precariousness. The axioms of an enactment perspective are described and employed to more fully understand the two kinds of replication that predominate in the crisis disputes. Although the enactment perspective described here is a relatively recent development in philosophy of science and science studies, some of its core axioms are not new to psychology, and the article concludes by revisiting psychologists’ previous calls to apprehend the dynamism of psychological reality to appreciate how scientific practices actively and unavoidably participate in performativity of reality.

Exploratory Structural Equation Modeling: Practical Guidelines and Tutorial With a Convenient Online Tool for Mplus

Critics of positive psychology have questioned the validity of positive psychological assessment measures (PPAMs), which negatively affects the credibility and public perception of the discipline. Psychometric evaluations of PPAMs have shown that various instruments produce inconsistent factor structures between groups/contexts/times frames, that their predictive validity is questionable, and that popular PPAMs are culturally biased. Further, it would seem positive psychological researchers prioritize date-model-fit over measurement quality. To address these analytical challenges, more innovative and robust approaches toward the validation and evaluation of PPAMs are required to enhance the discipline's credibility and to advance positive psychological science. Exploratory Structural Equation Modeling (ESEM) has recently emerged as a promising alternative to overcome some of these challenges by incorporating the best elements from exploratory- and confirmatory factor analyses. ESEM is still a relatively novel approach, and estimating these models in statistical software packages can be complex and tedious. Therefore, the purpose of this paper is to provide novice researchers with a practical tutorial on how to estimate ESEM with a convenient online tool for Mplus. Specifically, we aim to demonstrate the use of ESEM through an illustrative example by using a popular positive psychological instrument: the Mental Health Continuum-SF. By using the MHC-SF as an example, we aim to provide (a) a brief overview of ESEM (and different ESEM models/approaches), (b) guidelines for novice researchers on how to estimate, compare, report, and interpret ESEM, and (c) a step-by-step tutorial on how to run ESEM analyses in Mplus with the De Beer and Van Zy ESEM syntax generator. The results of this study highlight the value of ESEM, over and above that of traditional confirmatory factor analytical approaches. The results also have practical implications for measuring mental health with the MHC-SF, illustrating that a bifactor ESEM Model fits the data significantly better than any other theoretical model.

Developing an open science 'mindset'

BACKGROUND

Identification of widespread biases present in reported research findings in many scientific disciplines, including psychology, such as failures to replicate and the likely extensive application of questionable research practices, has raised serious concerns over the reliability and trustworthiness of scientific research. This has led to the development of, and advocacy for, 'open science' practices, including data, materials, analysis, and output sharing, pre-registration of study predictions and analysis plans, and increased access to published research findings. Implementation of such practices has been enthusiastic in some quarters, but literacy in, and adoption of, these practices has lagged behind among many researchers in the scientific community.

ADVANCES

In the current article I propose that researchers adopt an open science 'mindset', a comprehensive approach to open science predicated on researchers' operating under the basic assumption that, wherever possible, open science practices will be a central component of all steps of their research projects. The primary, defining feature of the mindset is a commitment to open science principles in all research projects from inception to dissemination. Other features of the mindset include the assumption that all components of research projects (e.g. pre-registered hypotheses, protocols, materials, analysis plans, data, and output) will be accessible broadly; pro-active selection of open fora to disseminate research components and findings; open and transparent dissemination of reports of the research findings in advance of, and after, formal publication; and active promotion of open science practices through education, modeling, and advocacy.

CONCLUSION

The open science mindset is a 'farm to fork' approach to open science aimed at promoting comprehensive quality in application of open science, and widening participation in open science practices so that they become the norm in research in health psychology and behavioral medicine going forward.

Psychology Exceptionalism and the Multiple Discovery of the Replication Crisis

This article outlines what we call the “narrative of psychology exceptionalism” in commentaries on the replication crisis: many thoughtful commentaries link the current crisis to the specificity of psychology’s history, methods, and subject matter, but explorations of the similarities between psychology and other fields are comparatively thin. Historical analyses of the replication crisis in psychology further contribute to this exceptionalism by creating a genealogy of events and personalities that shares little in common with other fields. We aim to rebalance this narrative by examining the emergence and evolution of replication discussions in psychology alongside their emergence and evolution in biomedicine. Through a mixed-methods analysis of commentaries on replication in psychology and the biomedical sciences, we find that these conversations have, from the early years of the crisis, shared a common core that centers on concerns about the effectiveness of traditional peer review, the need for greater transparency in methods and data, and the perverse incentive structure of academia. Drawing on Robert Merton’s framework for analyzing multiple discovery in science, we argue that the nearly simultaneous emergence of this narrative across fields suggests that there are shared historical, cultural, or institutional factors driving disillusionment with established scientific practices.

The Effect of Replications on Citation Patterns: Evidence From a Large-Scale Reproducibility Project

Replication of existing research is often referred to as one of the cornerstones of modern science. In this study, I tested whether the publication of independent replication attempts affects the citation patterns of the original studies. Investigating 95 replications conducted in the context of the Reproducibility Project: Psychology, I found little evidence for an adjustment of citation patterns in response to the publication of these independent replication attempts. This finding was robust to the choice of replication criterion, various model specifications, and the composition of the contrast group. I further present some suggestive evidence that shifts in the underlying composition of supporting and disputing citations have likely been small. I conclude with a review of the evidence in favor of the remaining explanations and discuss the potential consequences of these findings for the workings of the scientific process.

Experimental philosophical bioethics and normative inference

This paper explores an emerging sub-field of both empirical bioethics and experimental philosophy, which has been called "experimental philosophical bioethics" (bioxphi). As an empirical discipline, bioxphi adopts the methods of experimental moral psychology and cognitive science; it does so to make sense of the eliciting factors and underlying cognitive processes that shape people's moral judgments, particularly about real-world matters of bioethical concern. Yet, as a normative discipline situated within the broader field of bioethics, it also aims to contribute to substantive ethical questions about what should be done in a given context. What are some of the ways in which this aim has been pursued? In this paper, we employ a case study approach to examine and critically evaluate four strategies from the recent literature by which scholars in bioxphi have leveraged empirical data in the service of normative arguments.

Context Dependency as a Predictor of Replicability

We scrutinize the argument that unsuccessful replications—and heterogeneous effect sizes more generally—may reflect an underappreciated influence of context characteristics. Notably, while some of these context characteristics may be conceptually irrelevant (as they merely affect psychometric properties of the measured/manipulated variables), others are conceptually relevant as they qualify a theory. Here, we present a conceptual and analytical framework that allows researchers to empirically estimate the extent to which effect size heterogeneity is due to conceptually relevant versus irrelevant context characteristics. According to this framework, contextual characteristics are conceptually relevant when the observed heterogeneity of effect sizes cannot be attributed to psychometric properties. As an illustrative example, we demonstrate that the observed heterogeneity of the “moral typecasting” effect, which had been included in the ManyLabs 2 replication project, is more likely attributable to conceptually relevant rather than irrelevant context characteristics, which suggests that the psychological theory behind this effect may need to be specified. In general, we argue that context dependency should be taken more seriously and treated more carefully by replication research.

Programas de intervención y replicabilidad: consideraciones sobre su evaluación en psicología

En este trabajo se hace un repaso acerca de qué es la evaluación de programas de intervención, así como el procedimiento y finalidad que se ha buscado brindarle desde la psicología; y el procedimiento utilizado para llevar a cabo la replicación de estos estudios, especialmente en psicología, para así poder dar contexto a la llamada “crisis de confianza” en psicología. Esto con la finalidad de proponer que la replicabilidad de programas de intervención no podría lograrse de forma satisfactoriamente sin antes llevar a cabo un proceso de evaluación del estudio original. Para esto, se proponen tres parámetros, Factor de Bayes, Tamaño del efecto e Intervalos de Confianza; que han mostrado su utilidad en la literatura de habla inglesa, pero, sin embargo, no se ha difundido de manera extensa en aquella de habla hispana. Se propone lograr una mayor difusión de la estadística Bayesiana para solventar las problemáticas ocasionadas por la estadística frecuentista de prueba de significancia de hipótesis nula, puesto que una de las finalidades del teorema de Bayes es la acumulación y actualización continua del conocimiento obtenido de replicaciones, sin dejar de lado la evaluación de los grados de certeza de los resultados obtenidos. Así, este trabajo está dirigido para profesionistas nuevos y veteranos en el ámbito de la evaluación, que empiezan a adentrarse en la estadística como una forma de evaluar programas de investigación e intervención, por lo que se ha buscado explicar los parámetros propuestos de la forma más clara y concisa posible.

Programas de intervención y replicabilidad: consideraciones sobre su evaluación en psicología

En este trabajo se hace un repaso acerca de qué es la evaluación de programas de intervención, así como el procedimiento y finalidad que se ha buscado brindarle desde la psicología; y el procedimiento utilizado para llevar a cabo la replicación de estos estudios, especialmente en psicología, para así poder dar contexto a la llamada “crisis de confianza” en psicología. Esto con la finalidad de proponer que la replicabilidad de programas de intervención no podría lograrse de forma satisfactoriamente sin antes llevar a cabo un proceso de evaluación del estudio original. Para esto, se proponen tres parámetros, Factor de Bayes, Tamaño del efecto e Intervalos de Confianza; que han mostrado su utilidad en la literatura de habla inglesa, pero, sin embargo, no se ha difundido de manera extensa en aquella de habla hispana. Se propone lograr una mayor difusión de la estadística Bayesiana para solventar las problemáticas ocasionadas por la estadística frecuentista de prueba de significancia de hipótesis nula, puesto que una de las finalidades del teorema de Bayes es la acumulación y actualización continua del conocimiento obtenido de replicaciones, sin dejar de lado la evaluación de los grados de certeza de los resultados obtenidos. Así, este trabajo está dirigido para profesionistas nuevos y veteranos en el ámbito de la evaluación, que empiezan a adentrarse en la estadística como una forma de evaluar programas de investigación e intervención, por lo que se ha buscado explicar los parámetros propuestos de la forma más clara y concisa posible.

Manipulating Environmental Commitment: A Replication and Extension

Environmental commitment, the subjective experience of dependence on the natural environment, is marked by a long-term orientation and psychological attachment towards the natural environment. The current research replicates and extends previous research on temporarily increasing environmental commitment (Davis et al., 2009). We employed Davis et al.'s manipulation in two experimental studies (one laboratory, one online): we asked participants to spend time writing either about ways in which they are interdependent with the natural environment (high commitment manipulation) or unconnected with the environment (low commitment manipulation). In both studies we replicated the key finding that reflecting on one's interdependence with the environment increases commitment. We extended the previous research by finding evidence that this commitment effect was mediated by satisfaction with one's relationship to the environment. We did not replicate the original findings that the interdependence manipulation influences environmental behavioral intentions.

An Experimental Investigation of Variations in Judgments of Hypothetical Males and Females Initiating Mixed-Gender Threesomes: An Application of Sexual Script Theory

Although many young adults are interested in mixed-gender threesomes (MGTs), little research has assessed attitudes toward them. Yet, MGTs offer a rare context to investigate how consensually nonmonogamous sexual encounters and involvement with same-sex others influence attitudes. Thus, by adopting sexual script theory as a framework, the current study compared three dimensions of character judgments (cognitive abilities, morality, partner quality) and assumptions about the sexual history of hypothetical males and females who initiated a MGT (two females and one male; two males and one female) or mixed-sex dyadic sexual activity with a casual or committed partner. To do so, a between-subject design was adopted in which 690 U.S. adults (405 women, 285 men) evaluated a hypothetical initiator described in one of 12 vignettes. On average, participants made neutral judgments about the initiator, yet those initiating dyadic sexual behavior were judged more favorably and as having a less extensive sexual history than MGT initiators. Male initiators were judged more favorably than female initiators, particularly by men. Those initiating in the context of a committed relationship were judged as more moral and as higher-quality partners than those initiating within a casual relationship; female (but not male) initiators in the committed context were judged as having a less extensive sexual history than female initiators in the casual context. These results confirm the presence of mononormativity biases and the sexual double standard and have implications for educators and practitioners related to stigma reduction and the promotion of inclusive sexual education.

Heterogeneity of Research Results: A New Perspective From Which to Assess and Promote Progress in Psychological Science

Heterogeneity emerges when multiple close or conceptual replications on the same subject produce results that vary more than expected from the sampling error. Here we argue that unexplained heterogeneity reflects a lack of coherence between the concepts applied and data observed and therefore a lack of understanding of the subject matter. Typical levels of heterogeneity thus offer a useful but neglected perspective on the levels of understanding achieved in psychological science. Focusing on continuous outcome variables, we surveyed heterogeneity in 150 meta-analyses from cognitive, organizational, and social psychology and 57 multiple close replications. Heterogeneity proved to be very high in meta-analyses, with powerful moderators being conspicuously absent. Population effects in the average meta-analysis vary from small to very large for reasons that are typically not understood. In contrast, heterogeneity was moderate in close replications. A newly identified relationship between heterogeneity and effect size allowed us to make predictions about expected heterogeneity levels. We discuss important implications for the formulation and evaluation of theories in psychology. On the basis of insights from the history and philosophy of science, we argue that the reduction of heterogeneity is important for progress in psychology and its practical applications, and we suggest changes to our collective research practice toward this end.

A Critical Review of Cranial Electrotherapy Stimulation for Neuromodulation in Clinical and Non-clinical Samples

Cranial electrotherapy stimulation (CES) is a neuromodulation tool used for treating several clinical disorders, including insomnia, anxiety, and depression. More recently, a limited number of studies have examined CES for altering affect, physiology, and behavior in healthy, non-clinical samples. The physiological, neurochemical, and metabolic mechanisms underlying CES effects are currently unknown. Computational modeling suggests that electrical current administered with CES at the earlobes can reach cortical and subcortical regions at very low intensities associated with subthreshold neuromodulatory effects, and studies using electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) show some effects on alpha band EEG activity, and modulation of the default mode network during CES administration. One theory suggests that CES modulates brain stem (e.g., medulla), limbic (e.g., thalamus, amygdala), and cortical (e.g., prefrontal cortex) regions and increases relative parasympathetic to sympathetic drive in the autonomic nervous system. There is no direct evidence supporting this theory, but one of its assumptions is that CES may induce its effects by stimulating afferent projections of the vagus nerve, which provides parasympathetic signals to the cardiorespiratory and digestive systems. In our critical review of studies using CES in clinical and non-clinical populations, we found severe methodological concerns, including potential conflicts of interest, risk of methodological and analytic biases, issues with sham credibility, lack of blinding, and a severe heterogeneity of CES parameters selected and employed across scientists, laboratories, institutions, and studies. These limitations make it difficult to derive consistent or compelling insights from the extant literature, tempering enthusiasm for CES and its potential to alter nervous system activity or behavior in meaningful or reliable ways. The lack of compelling evidence also motivates well-designed and relatively high-powered experiments to assess how CES might modulate the physiological, affective, and cognitive responses to stress. Establishing reliable empirical links between CES administration and human performance is critical for supporting its prospective use during occupational training, operations, or recovery, ensuring reliability and robustness of effects, characterizing if, when, and in whom such effects might arise, and ensuring that any benefits of CES outweigh the risks of adverse events.

The Impact of Complexity on Methods and Findings in Psychological Science

The study of human behavior is severely hampered by logistical problems, ethical and legal constraints, and funding shortfalls. However, the biggest difficulty of conducting social and behavioral research is the extraordinary complexity of the study phenomena. In this article, we review the impact of complexity on research design, hypothesis testing, measurement, data analyses, reproducibility, and the communication of findings in psychological science. The systematic investigation of the world often requires different approaches because of the variability in complexity. Confirmatory testing, multi-factorial designs, survey methods, large samples, and modeling are frequently needed to study complex social and behavioral topics. Complexity impedes the measurement of general constructs, the reproducibility of results and scientific reporting, and the general rigor of research. Many of the benchmarks established by classic work in physical science are not attainable in studies of more complex phenomena. Consequently, the standards used to evaluate scientific research should be tethered to the complexity of the study topic.

A systematic review of screen-time literature to inform educational policy and practice during COVID-19

There is an urgent need for consensus around the matter of screen time (ST) during the COVID-19 pandemic. Some governments announced restrictions for online schooling time per day to protect students from perceived risks of prolonged screen-use, but critics and an emerging body of research question such regulations. Our review of 52 empirical studies found (a) an overwhelming majority of literature shows effect sizes too small to be of practical or clinical significance, and (b) findings more specifically on educational ST are inconclusive and critically underrepresented. These facts, along with the undeniable benefits of online learning in the absence of brick-and-mortar schooling and the ominous forecasts of learning loss caused by prolonged school closure, inform our recommendations for a more moderate policy and practical stance on restrictions - one that is focused on responsibly leveraging the educational and social benefits of ST in a world still recovering from the COVID-19 pandemic.

Reducing versus Embracing Variation as Strategies for Reproducibility: The Microbiome of Laboratory Mice

Irreproducibility is a well-recognized problem in biomedical animal experimentation. Phenotypic variation in animal models is one of the many challenging causes of irreproducibility. How to deal with phenotypic variation in experimental designs is a topic of debate. Both reducing and embracing variation are highlighted as strategies for reproducibility. In this theoretical review, we use variation in mouse microbiome composition as an example to analyze this ongoing discussion, drawing on both animal research and philosophy of science. We provide a conceptual explanation of reproducibility and analyze how the microbiome affects mouse phenotypes to demonstrate that the role of the microbiome in irreproducibility can be understood in two ways: (i) the microbiome can act as a confounding factor, and (ii) the result may not be generalizable to mice harboring a different microbiome composition. We elucidate that reducing variation minimizes confounding, whereas embracing variation ensures generalizability. These contrasting strategies make dealing with variation in experimental designs extremely complex. Here, we conclude that the most effective strategy depends on the specific research aim and question. The field of biomedical animal experimentation is too broad to identify a single optimal strategy. Thus, dealing with variation should be considered on a case-by-case basis, and awareness amongst researchers is essential.