Replication, falsification, and the crisis of confidence in social psychology

A Demonstration of the Collaborative Replication and Education Project: Replication Attempts of the Red-Romance Effect

The present article reports the results of a meta-analysis of nine student replication projects of Elliot et al.’s (2010) findings from Experiment 3, that women were more attracted to photographs of men with red borders (total n = 640). The eight student projects were part of the Collaborative Replication and Education Project (CREP; https://osf.io/wfc6u/), a research crowdsourcing project for undergraduate students. All replications were reviewed by experts to ensure high quality data, and were pre-registered prior to data collection. Results of this meta-analysis showed no effect of red on attractiveness ratings for either perceived attractiveness (mean ratings difference = –0.07, 95% CI [–0.31, 0.16]) or sexual attractiveness (mean ratings difference = –0.06, 95% CI [–0.36, 0.24]); this null result held with and without Elliot et al.’s (2010) data included in analyses. Exploratory analyses examining whether being in a relationship moderated the effect of color on attractiveness ratings also produced null results.

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Testing the effect of cooperative/competitive priming on the Prisoner's Dilemma. A replication study

The replicability crisis in psychology demands direct replications to test the reliability of relevant phenomena. Prime-to-behavior effects have been an area under intense scrutiny given its surprising results. However, intuitive unsurprising effects have been mostly neglected, while they may lack robustness as well. In the present study, we focused on an intuitive prime-to-behavior effect in which Kay and Ross (2003) used a 2x2 design to test cooperation/competition priming crossed with an explicit/non-explicit construal of a Prisoner’s Dilemma (PD). They found a stronger assimilation effect of priming when the situational construal anteceded the decision, but we could not reproduce their findings in the present close replication, despite counting on higher power. Even with limitations due to the unavailability of original materials, this replication presents evidence that questions the existence of the original finding, and highlights the need for further replications to get a deeper understanding of the hypothesized effect. The complete project is available at: https://osf.io/dhfns/.

Psychedelic Moral Enhancement

The moral enhancement (or bioenhancement) debate seems stuck in a dilemma. On the one hand, the more radical proposals, while certainly novel and interesting, seem unlikely to be feasible in practice, or if technically feasible then most likely imprudent. But on the other hand, the more sensible proposals – sensible in the sense of being both practically achievable and more plausibly ethically justifiable – can be rather hard to distinguish from both traditional forms of moral enhancement, such as non-drug-mediated social or moral education, and non-moral forms of bioenhancement, such as smart-drug style cognitive enhancement. In this essay, I argue that bioethicists have paid insufficient attention to an alternative form of moral bioenhancement – or at least a likely candidate – that falls somewhere between these two extremes, namely the (appropriately qualified) use of certain psychedelic drugs.

Unpacking decision difficulty: Testing action dynamics in Intertemporal, gamble, and consumer choices

The study examined the two-factor structure of decision difficulty proposed by Cheng and González-Vallejo (2017) in new domains, and the role of numeracy in relation to these factors. Using the measurement methodology of 'mouse' (cursor) movements, participants' temporal and spatial measures were recorded when making decisions in the domains of intertemporal, gamble, and consumer choices. Task manipulations designed to affect difficulty included the sign of the payoffs (gains vs. losses), the similarity of the attribute values being compared, and attribute importance. A psychometric analysis of the measures revealed three orthogonal components, two of which, conflict and wavering, described decision difficulty. The conflict component was most affected by changes in the sign of the payoffs of intertemporal and gamble choices, with greater means observed in the loss than in the gain context. By contrast, the wavering component was most affected by changes of the similarity between the options' attributes, with greater means when the options were more similar. The study also found that choosing the long-term advantageous options in an intertemporal choice task; choosing the riskier gain and safer loss in a gamble choice task; and choosing the more expensive/better-quality hotel in a consumer choice task demonstrated greater conflict and/or wavering. The study further found that numeracy, or the degree to which individuals are able to use and interpret numbers, was negatively related to the conflict component. Taken together, the study demonstrated that decision difficulty varied with contextual changes, and action-dynamic measures reflected different facets of decision difficulty.

Family reciprocity of older Singaporeans

Reciprocity is a powerful motivation in social life. We study what older people give to their family for help received. Data are from the Panel on Health and Aging of Singaporean Elderly, Wave 2 (2011; persons aged 62+; N = 3103). Giving and receiving help are with family members other than spouse in the same household, in the past year. Types of help given and received are money, food/clothes/other material goods, housework/cooking, babysitting grandchildren, emotional support/advice, help for personal care, and help for going out. Multivariate models predict each type of giving help, with independent variables about the older person's resources, needs, and help received. Reciprocity is demonstrated by positive relationships between receiving and giving help. Results show two kinds of reciprocity: "nontangibles for tangibles" and "same for same." First, older people give their time and effort in return for money and material goods. This aligns with contemporary Singapore circumstances, in that older people tend to have ample time but limited financial resources, while family members (often midlife children) have the reverse. Second, same-for-same exchanges, such as housework both given and received, are shared tasks in families or normative behaviors in Singapore society. The results replicate and extend prior ones for Singapore. We discuss prospects for change in frequency and shape of family reciprocity as the state continues to modernize.

The Relationship Between Immorality and Cleansing

Abstract. In a widely publicized set of studies, participants who were primed to consider unethical events preferred cleansing products more than did those primed with ethical events ( Zhong & Liljenquist, 2006 ). This tendency to respond to moral threat with physical cleansing is known as the Macbeth Effect. Several subsequent efforts, however, did not replicate this relationship. The present manuscript reports the results of a meta-analysis of 15 studies testing this relationship. The weighted mean effect size was small across all studies (g = 0.17, 95% CI [0.04, 0.31]), and nonsignificant across studies conducted in independent laboratories (g = 0.07, 95% CI [−0.04, 0.19]). We conclude that there is little evidence for an overall Macbeth Effect; however, there may be a Macbeth Effect under certain conditions.

A Unified Framework to Quantify the Credibility of Scientific Findings

Societies invest in scientific studies to better understand the world and attempt to harness such improved understanding to address pressing societal problems. Published research, however, can be useful for theory or application only if it is credible. In science, a credible finding is one that has repeatedly survived risky falsification attempts. However, state-of-the-art meta-analytic approaches cannot determine the credibility of an effect because they do not account for the extent to which each included study has survived such attempted falsification. To overcome this problem, we outline a unified framework for estimating the credibility of published research by examining four fundamental falsifiability-related dimensions: (a) transparency of the methods and data, (b) reproducibility of the results when the same data-processing and analytic decisions are reapplied, (c) robustness of the results to different data-processing and analytic decisions, and (d) replicability of the effect. This framework includes a standardized workflow in which the degree to which a finding has survived scrutiny is quantified along these four facets of credibility. The framework is demonstrated by applying it to published replications in the psychology literature. Finally, we outline a Web implementation of the framework and conclude by encouraging the community of researchers to contribute to the development and crowdsourcing of this platform.

Perceived Statistical Knowledge Level and Self-Reported Statistical Practice Among Academic Psychologists

Introduction: Publications arguing against the null hypothesis significance testing (NHST) procedure and in favor of good statistical practices have increased. The most frequently mentioned alternatives to NHST are effect size statistics (ES), confidence intervals (CIs), and meta-analyses. A recent survey conducted in Spain found that academic psychologists have poor knowledge about effect size statistics, confidence intervals, and graphic displays for meta-analyses, which might lead to a misinterpretation of the results. In addition, it also found that, although the use of ES is becoming generalized, the same thing is not true for CIs. Finally, academics with greater knowledge about ES statistics presented a profile closer to good statistical practice and research design. Our main purpose was to analyze the extension of these results to a different geographical area through a replication study.

Methods: For this purpose, we elaborated an on-line survey that included the same items as the original research, and we asked academic psychologists to indicate their level of knowledge about ES, their CIs, and meta-analyses, and how they use them. The sample consisted of 159 Italian academic psychologists (54.09% women, mean age of 47.65 years). The mean number of years in the position of professor was 12.90 (SD = 10.21).

Results: As in the original research, the results showed that, although the use of effect size estimates is becoming generalized, an under-reporting of CIs for ES persists. The most frequent ES statistics mentioned were Cohen's d and R²/η², which can have outliers or show non-normality or violate statistical assumptions. In addition, academics showed poor knowledge about meta-analytic displays (e.g., forest plot and funnel plot) and quality checklists for studies. Finally, academics with higher-level knowledge about ES statistics seem to have a profile closer to good statistical practices.

Conclusions: Changing statistical practice is not easy.This change requires statistical training programs for academics, both graduate and undergraduate.

The Creative Cycle and the Growth of Psychological Science

Scientific progress relies on the dialectics of loosening and tightening processes. Although most gripping accomplishments in psychological science testify to the critical role of creative and innovative theorizing (loosening), the ongoing debate on the quality of psychological science is focused almost totally on a restrictive sense of tightening, revolving around statistical hypothesis testing (tightening). A discussion of the imbalance between the conspicuous neglect of theory and the overstated importance of the sacred cow of significance testing raises a skeptical question: Do we seriously believe that stricter compliance rules, exact p values, effect size calculations, new statistics, and monitoring of research practices will foster the growth of excellent science? A more effective strategy would be to start a positive debate that focuses on the best exemplars of strong theorizing and fascinating findings, replacing the focus on insufficient science and unwanted practices.

The misery-is-not-miserly effect revisited: Replication despite opportunities for compensatory consumption

Sadness increases how much decision makers pay to acquire goods, even when decision makers are unaware of it. This effect is coined the "misery-is-not-miserly effect". The paper that first established this effect is the second most-cited article appearing in Psychological Science in 2004. In light of its impact, the present study sought to assess whether the misery-is-not-miserly effect would replicate (a) in a novel context and (b) even when another way of alleviating a sense of loss (i.e., compensatory consumption) was available. Results revealed that the effect replicated in the novel context and, despite a prediction otherwise, even when individuals had an opportunity to engage in compensatory consumption. Moreover, a meta-analysis of the original effect and that observed in the present study yielded a small-to-medium effect (Cohen's d = 0.43). As such, the present study lends evidentiary support to the misery-is-not-miserly effect and provides impetus for future research exploring the impact of sadness on consumer decision-making, specifically, and of emotion on decision processes, more generally.

The misery-is-not-miserly effect revisited: Replication despite opportunities for compensatory consumption

Sadness increases how much decision makers pay to acquire goods, even when decision makers are unaware of it. This effect is coined the “misery-is-not-miserly effect”. The paper that first established this effect is the second most-cited article appearing in Psychological Science in 2004. In light of its impact, the present study sought to assess whether the misery-is-not-miserly effect would replicate (a) in a novel context and (b) even when another way of alleviating a sense of loss (i.e., compensatory consumption) was available. Results revealed that the effect replicated in the novel context and, despite a prediction otherwise, even when individuals had an opportunity to engage in compensatory consumption. Moreover, a meta-analysis of the original effect and that observed in the present study yielded a small-to-medium effect (Cohen’s d = 0.43). As such, the present study lends evidentiary support to the misery-is-not-miserly effect and provides impetus for future research exploring the impact of sadness on consumer decision-making, specifically, and of emotion on decision processes, more generally.

Further Validation of a Laboratory Analog Sexual Aggression Task: Associations With Novel Risk Factors for Sexual Violence

Official crime statistics and self-reports of sexual aggression perpetration are limited by various factors (e.g., lack of reporting, social desirability bias), as well as an inability to use these measures in experimental studies of sexual aggression. To address these issues, Nagayama Hall et al. (1994) developed a laboratory analog measure of sexual aggression, which has received empirical support as a valid measure of sexual aggression proclivity. Here, we seek to replicate these findings and further validate the paradigm by examining sexually aggressive responses in relation to a range of recently emerging predictors of sexual aggression (e.g., sexual objectification, sexual narcissism) as well as participants' perceptions of a female confederate serving as the target of sexual aggression. A sample of 49 undergraduate men completed questionnaires and participated in the sexual aggression analog task. Results of logistic regression analyses supported both criterion and construct validity of the analog task; men who chose the sexually explicit video were more likely to report prior sexual aggression, greater sexual objectification of women, higher sexual narcissism, greater hostile sexism, and higher impersonal dating. These men were also more likely to express dehumanizing beliefs about the female confederate, consider her less intelligent, and believe she was more distressed by the video. These results replicate and extend prior research supporting this paradigm as a valid laboratory-based measure of sexual aggression proclivity that can be used in tandem with validated self-report measures.

Modeling Linguistic Variables With Regression Models: Addressing Non-Gaussian Distributions, Non-independent Observations, and Non-linear Predictors With Random Effects and Generalized Additive Models for Location, Scale, and Shape

As statistical approaches are getting increasingly used in linguistics, attention must be paid to the choice of methods and algorithms used. This is especially true since they require assumptions to be satisfied to provide valid results, and because scientific articles still often fall short of reporting whether such assumptions are met. Progress is being, however, made in various directions, one of them being the introduction of techniques able to model data that cannot be properly analyzed with simpler linear regression models. We report recent advances in statistical modeling in linguistics. We first describe linear mixed-effects regression models (LMM), which address grouping of observations, and generalized linear mixed-effects models (GLMM), which offer a family of distributions for the dependent variable. Generalized additive models (GAM) are then introduced, which allow modeling non-linear parametric or non-parametric relationships between the dependent variable and the predictors. We then highlight the possibilities offered by generalized additive models for location, scale, and shape (GAMLSS). We explain how they make it possible to go beyond common distributions, such as Gaussian or Poisson, and offer the appropriate inferential framework to account for 'difficult' variables such as count data with strong overdispersion. We also demonstrate how they offer interesting perspectives on data when not only the mean of the dependent variable is modeled, but also its variance, skewness, and kurtosis. As an illustration, the case of phonemic inventory size is analyzed throughout the article. For over 1,500 languages, we consider as predictors the number of speakers, the distance from Africa, an estimation of the intensity of language contact, and linguistic relationships. We discuss the use of random effects to account for genealogical relationships, the choice of appropriate distributions to model count data, and non-linear relationships. Relying on GAMLSS, we assess a range of candidate distributions, including the Sichel, Delaporte, Box-Cox Green and Cole, and Box-Cox t distributions. We find that the Box-Cox t distribution, with appropriate modeling of its parameters, best fits the conditional distribution of phonemic inventory size. We finally discuss the specificities of phoneme counts, weak effects, and how GAMLSS should be considered for other linguistic variables.

A Research Note on the Influence of Relationship Length and Sex on Preferences for Altruistic and Cooperative Mates

Previous literature suggests that altruism may have evolved as a sexually selectable trait. Recent research suggests that women seek altruistic traits for long-term, not short-term relationships, as altruism can serve as an honest signal of one’s character. We tested this hypothesis by asking 102 participants to complete a modified version of Buss’s Mate Preferences Questionnaire. We found that women placed higher importance on altruism in a mate compared to men, and this preference was greater when seeking a long-term mate, compared to a short-term mate. We also found that although women placed greater importance on cooperativeness in a mate compared to men, this preference was not influenced by whether they were seeking a short-term or a long-term mate. We successfully replicate previous literature exploring the role of altruism in mate choice.

Does it actually feel right? A replication attempt of the rounded price effect

How does the roundedness of prices affect product evaluations? The 'rounded price effect' postulates that depending on the context, rounded or non-rounded prices increase the purchase likelihood of consumers. The study presented here is a replication attempt of these findings and the proposed mediation of the effect through a sense of 'feeling right' when evaluating the product. p-Curve analysis and the R-Index are used to assess the robustness of the originally reported statistics since original data were not available. A pre-registered replication of study 5 from the original article was conducted in a sample of N=588 participants. For both the original product and one alternative product neither an interaction between price roundedness and context, nor a mediation through 'a sense of feeling right' was found. Our results suggest that the effect is either smaller than originally reported or contingent on other, not investigated factors. Further studies might investigate contingencies in larger samples.

Implications of Debunking the "Critical Positivity Ratio" for Humanistic Psychology: Introduction to Special Issue

An extraordinary claim was made by one of the leading researchers within positive psychology, namely, there is a universal–invariant ratio between positive to negative emotions that serves as a unique tipping point between flourishing and languishing in individuals, marriages, organizations, and other human systems across all cultures and times. Known as the “critical positivity ratio,” this finding was supposedly derived from the famous Lorenz equation in physics by using the mathematics of nonlinear dynamic systems, and was defined precisely as “2.9013.” This exact number was widely touted as a great discovery by many leaders of positive psychology, had tremendous impact in various applied areas of psychology, and, more broadly, and was extensively cited in both the scientific literature and in the global popular media. However, this finding has been demonstrated to be bogus. Since its advent as a relatively new subdiscipline, positive psychology has claimed superiority to its precursor, the subdiscipline of humanistic psychology, in terms of supposedly both using more rigorous science and avoiding popularizing nonsense. The debunking of the critical positivity ratio demonstrates that positive psychology did not live up to these claims, and this has important implications, which are discussed in terms of “romantic scientism” and “voodoo science.” In addition, articles in the special issue on the “Implications of Debunking the ‘Critical Positivity Ratio’ for Humanistic Psychology” are introduced, as they also delve into these concerns.

Are all "research fields" equal? Rethinking practice for the use of data from crowdsourcing market places

New technologies like large-scale social media sites (e.g., Facebook and Twitter) and crowdsourcing services (e.g., Amazon Mechanical Turk, Crowdflower, Clickworker) are impacting social science research and providing many new and interesting avenues for research. The use of these new technologies for research has not been without challenges, and a recently published psychological study on Facebook has led to a widespread discussion of the ethics of conducting large-scale experiments online. Surprisingly little has been said about the ethics of conducting research using commercial crowdsourcing marketplaces. In this article, I focus on the question of which ethical questions are raised by data collection with crowdsourcing tools. I briefly draw on the implications of Internet research more generally, and then focus on the specific challenges that research with crowdsourcing tools faces. I identify fair pay and the related issue of respect for autonomy, as well as problems with the power dynamic between researcher and participant, which has implications for withdrawal without prejudice, as the major ethical challenges of crowdsourced data. Furthermore, I wish to draw attention to how we can develop a "best practice" for researchers using crowdsourcing tools.

Modeling stimulus variation in three common implicit attitude tasks

We explored the consequences of ignoring the sampling variation due to stimuli in the domain of implicit attitudes. A large literature in psycholinguistics has examined the statistical treatment of random stimulus materials, but the recommendations from this literature have not been applied to the social psychological literature on implicit attitudes. This is partly because of inherent complications in applying crossed random-effect models to some of the most common implicit attitude tasks, and partly because no work to date has demonstrated that random stimulus variation is in fact consequential in implicit attitude measurement. We addressed this problem by laying out statistically appropriate and practically feasible crossed random-effect models for three of the most commonly used implicit attitude measures-the Implicit Association Test, affect misattribution procedure, and evaluative priming task-and then applying these models to large datasets (average N = 3,206) that assess participants' implicit attitudes toward race, politics, and self-esteem. We showed that the test statistics from the traditional analyses are substantially (about 60 %) inflated relative to the more-appropriate analyses that incorporate stimulus variation. Because all three tasks used the same stimulus words and faces, we could also meaningfully compare the relative contributions of stimulus variation across the tasks. In an appendix, we give syntax in R, SAS, and SPSS for fitting the recommended crossed random-effects models to data from all three tasks, as well as instructions on how to structure the data file.

Strengthening the Practice of Exercise and Sport-Science Research

Exercise and sport sciences continue to grow as a collective set of disciplines investigating a broad array of basic and applied research questions. Despite the progress, there is room for improvement. A number of problems pertaining to reliability and validity of research practices hinder advancement and the potential impact of the field. These problems include inadequate validation of surrogate outcomes, too few longitudinal and replication studies, limited reporting of null or trivial results, and insufficient scientific transparency. The purpose of this review is to discuss these problems as they pertain to exercise and sport sciences based on their treatment in other disciplines, namely psychology and medicine, and to propose a number of solutions and recommendations.

Wo sind die wissenschaftlichen Standards für hochwertige Replikationsforschung?

Zusammenfassung. Es gibt einen breiten Konsens, dass Replikation ein wichtiges Instrument ist, um valide Befunde und solide Forschung zu erkennen. Wenn sie aber wissenschaftlich bedeutsam ist, dann muss auch die Replikationsforschung an strengen methodischen Regeln und an klar artikulierten wissenschaftlichen Zielen gemessen werden. Eine kritische Beschäftigung mit der aktuellen Replikationsforschung – etwa im jüngst veröffentlichten Bericht der Open Science Collaboration – zeigt jedoch, dass eine strenge und forschungslogisch begründete Methodologie für Replikationsstudien bislang weder angewandt noch entwickelt wurde. Infolgedessen bleibt die Validität der Schlüsse, die aus Replikationsstudien gezogen werden dürfen, oftmals unklar. Dieses grundlegende Problem wird hier unter vier Gesichtspunkten diskutiert: Unklarheit des Gegenstandes der Replikation (Replicandum), Vernachlässigung einschlägiger methodischer Probleme (Regressivität; Reliabilität der Veränderungsmessung), einseitige Vermeidung von angeblich kostenträchtigen „Falsch-Positiven“ ohne Versuch einer systematischen Kosten-Nutzen-Messung sowie das vernachlässigte Ziel, Replikationsforschung so zu implementieren, dass sie echte Erkenntnisfortschritte bringt und als exzellente Forschung anerkannt werden kann.

"Maybe I Will Just Send a Quick Text…" - An Examination of Drivers' Distractions, Causes, and Potential Interventions

Many people use cars all over the world. This is, however, not done without risk, as traffic accidents are one of the most common causes of death for adolescents worldwide. The number of deaths has steadily decreased, both worldwide and in Norway. Many of these accidents involve passenger cars and distracted driving. While there are many campaigns to improve safety in traffic, little research has looked at distractions. A recent report has investigated the occurrence of and damage caused by distraction, and one article has looked at what predicts baseline differences in levels of distracted driving. However, no one has tested an intervention to decrease distracted behavior in traffic. Motivational variables suggested by the Theory of Planned Behavior, personality traits, and demographic variables show utility in similar contexts and are all tested in this project. Data from two samples were collected to investigate the nature of distractions in traffic, what factors predict baseline levels of distractions, and to test an intervention to reduce distractions. Both samples feature randomly assigned intervention and control groups. The first sample (n = 1100 total; n = 208 was licensed to drive) consisted of high school students from all over Norway as a part of a larger attitudinal campaign, while the second sample (n = 414) was more general. The second tested a digital version of implementation intentions designed as volitional help sheets. The results from both samples suggest that there are some robust differences between people in how much they are distracted in everyday life, while some variables need further research. The second study failed to uncover any effects of the intervention. Reasons for this are discussed, along with points on the efficacy of digital interventions, the design of the volitional help sheets, and the design of the study in general. Notwithstanding the ineffectual interventions, this study contains novel information about baseline differences in distractive behavior that may further impact future behavior change interventions and guide future research.

On Nomological Validity and Auxiliary Assumptions: The Importance of Simultaneously Testing Effects in Social Cognitive Theories Applied to Health Behavior and Some Guidelines

Tests of social cognitive theories provide informative data on the factors that relate to health behavior, and the processes and mechanisms involved. In the present article, we contend that tests of social cognitive theories should adhere to the principles of nomological validity, defined as the degree to which predictions in a formal theoretical network are confirmed. We highlight the importance of nomological validity tests to ensure theory predictions can be disconfirmed through observation. We argue that researchers should be explicit on the conditions that lead to theory disconfirmation, and identify any auxiliary assumptions on which theory effects may be conditional. We contend that few researchers formally test the nomological validity of theories, or outline conditions that lead to model rejection and the auxiliary assumptions that may explain findings that run counter to hypotheses, raising potential for ‘falsification evasion.’ We present a brief analysis of studies (k = 122) testing four key social cognitive theories in health behavior to illustrate deficiencies in reporting theory tests and evaluations of nomological validity. Our analysis revealed that few articles report explicit statements suggesting that their findings support or reject the hypotheses of the theories tested, even when findings point to rejection. We illustrate the importance of explicit a priori specification of fundamental theory hypotheses and associated auxiliary assumptions, and identification of the conditions which would lead to rejection of theory predictions. We also demonstrate the value of confirmatory analytic techniques, meta-analytic structural equation modeling, and Bayesian analyses in providing robust converging evidence for nomological validity. We provide a set of guidelines for researchers on how to adopt and apply the nomological validity approach to testing health behavior models.

Hypothesis Testing, p Values, Confidence Intervals, Measures of Effect Size, and Bayesian Methods in Light of Modern Robust Techniques

The article provides perspectives on p values, null hypothesis testing, and alternative techniques in light of modern robust statistical methods. Null hypothesis testing and p values can provide useful information provided they are interpreted in a sound manner, which includes taking into account insights and advances that have occurred during the past 50 years. There are, of course, limitations to what null hypothesis testing and p values reveal about data. But modern advances make it clear that there are serious limitations and concerns associated with conventional confidence intervals, standard Bayesian methods, and commonly used measures of effect size. Many of these concerns can be addressed using modern robust methods.

Reproducibility in Psychological Science: When Do Psychological Phenomena Exist?

Scientific evidence has recently been used to assert that certain psychological phenomena do not exist. Such claims, however, cannot be made because (1) scientific method itself is seriously limited (i.e., it can never prove a negative); (2) non-existence of phenomena would require a complete absence of both logical (theoretical) and empirical support; even if empirical support is weak, logical and theoretical support can be strong; (3) statistical data are only one piece of evidence and cannot be used to reduce psychological phenomena to statistical phenomena; and (4) psychological phenomena vary across time, situations and persons. The human mind is unreproducible from one situation to another. Psychological phenomena are not particles that can decisively be tested and discovered. Therefore, a declaration that a phenomenon is not real is not only theoretically and empirically unjustified but runs counter to the propositional and provisional nature of scientific knowledge. There are only "temporary winners" and no "final truths" in scientific knowledge. Psychology is a science of subtleties in human affect, cognition and behavior. Its phenomena fluctuate with conditions and may sometimes be difficult to detect and reproduce empirically. When strictly applied, reproducibility is an overstated and even questionable concept in psychological science. Furthermore, statistical measures (e.g., effect size) are poor indicators of the theoretical importance and relevance of phenomena (cf. "deliberate practice" vs. "talent" in expert performance), not to mention whether phenomena are real or unreal. To better understand psychological phenomena, their theoretical and empirical properties should be examined via multiple parameters and criteria. Ten such parameters are suggested.

We need more replication research - A case for test-retest reliability

Following debates in psychology on the importance of replication research, we have also started to see pleas for a more prominent role for replication research in medical education. To enable replication research, it is of paramount importance to carefully study the reliability of the instruments we use. Cronbach's alpha has been the most widely used estimator of reliability in the field of medical education, notably as some kind of quality label of test or questionnaire scores based on multiple items or of the reliability of assessment across exam stations. However, as this narrative review outlines, Cronbach's alpha or alternative reliability statistics may complement but not replace psychometric methods such as factor analysis. Moreover, multiple-item measurements should be preferred above single-item measurements, and when using single-item measurements, coefficients as Cronbach's alpha should not be interpreted as indicators of the reliability of a single item when that item is administered after fundamentally different activities, such as learning tasks that differ in content. Finally, if we want to follow up on recent pleas for more replication research, we have to start studying the test-retest reliability of the instruments we use.

The effects of arousal reappraisal on stress responses, performance and attention

BACKGROUND AND OBJECTIVES

This study examined the effects of arousal reappraisal on cardiovascular responses, demand and resource evaluations, self-confidence, performance and attention under pressurized conditions. A recent study by Moore et al. [2015. Reappraising threat: How to optimize performance under pressure. Journal of Sport and Exercise Psychology, 37(3), 339-343. doi: 10.1123/jsep.2014-0186 ] suggested that arousal reappraisal is beneficial to the promotion of challenge states and leads to improvements in single-trial performance. This study aimed to further the work of Moore and colleagues (2015) by examining the effects of arousal reappraisal on cardiovascular responses, demand and resource evaluations, self-confidence, performance and attention in a multi-trial pressurized performance situation.

DESIGN AND METHODS

Participants were randomly assigned to either an arousal reappraisal intervention or control condition, and completed a pressurized dart throwing task. The intervention encouraged participants to view their physiological arousal as facilitative rather than debilitative to performance. Measures of cardiovascular reactivity, demand and resource evaluations, self-confidence, task performance and attention were recorded.

RESULTS

The reappraisal group displayed more favorable cardiovascular reactivity and reported higher resource evaluations and higher self-confidence than the control group but no task performance or attention effects were detected.

CONCLUSION

These findings demonstrate the strength of arousal reappraisal in promoting adaptive stress responses, perceptions of resources and self-confidence.

Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices

Recent research in psychology has highlighted a number of replication problems in the discipline, with publication bias - the preference for publishing original and positive results, and a resistance to publishing negative results and replications- identified as one reason for replication failure. However, little empirical research exists to demonstrate that journals explicitly refuse to publish replications. We reviewed the instructions to authors and the published aims of 1151 psychology journals and examined whether they indicated that replications were permitted and accepted. We also examined whether journal practices differed across branches of the discipline, and whether editorial practices differed between low and high impact journals. Thirty three journals (3%) stated in their aims or instructions to authors that they accepted replications. There was no difference between high and low impact journals. The implications of these findings for psychology are discussed.

German and English Bodies: No Evidence for Cross-Linguistic Differences in Preferred Orthographic Grain Size

Previous studies have found that words and nonwords with many body neighbours (i.e., words with the same orthographic body, e.g., cat, brat, at) are read faster than items with fewer body neighbours. This body-N effect has been explored in the context of cross-linguistic differences in reading where it has been reported that the size of the effect differs as a function of orthographic depth: readers of English, a deep orthography, show stronger facilitation than readers of German, a shallow orthography. Such findings support the psycholinguistic grain size theory, which proposes that readers of English rely on large orthographic units to reduce ambiguity of print-to-speech correspondences in their orthography. Here we re-examine the evidence for this pattern and find that there is no reliable evidence for such a cross-linguistic difference. Re-analysis of a key study (Ziegler et al., 2001), analysis of data from the English Lexicon Project (Balota et al., 2007), and a large-scale analysis of nine new experiments all support this conclusion. Using Bayesian analysis techniques, we find little evidence of the body-N effect in most tasks and conditions. Where we do find evidence for a body-N effect (lexical decision for nonwords), we find evidence against an interaction with language.

Science: the slow march of accumulating evidence

Recent crises over the credibility of research in psychology and the biomedical sciences have highlighted the need for researchers to view and treat replication research as essential to the accumulation of knowledge. In this article, the authors make the case for the utility of replication in medical education research. Specifically, the authors contend that because research in medical education often adopts theories from other disciplines, replication is necessary to gauge the applicability of those theories to the specific medical education context. This article introduces readers to the two major types of replication - direct and conceptual - and provides a primer on conceptual replication. In particular, the article presents key elements of conceptual replication and considers how it can be used to strengthen approaches to knowledge generation, theory testing, and theory development in medical education research.

On the quality of auxiliary assumptions

In a recent contribution to this journal, David Trafimow and Joshua Uhalt (2015) argue against the tradeoff between predictive power and explanatory breadth. In their view, it is the quality of the auxiliary assumptions that allows one to make testable predictions; hence it is possible that theories of considerable explanatory breadth, when combined with the right quality auxiliary assumptions, lead to testable predictions. Unfortunately, they leave the notion “quality” unspecified. In this review, I consider three possible properties that might render auxiliary assumptions capable of yielding testable predictions, namely observability, detail, and precision. All three proposals are rejected. I end with an appeal to further reflect on the role of auxiliary assumptions in deriving testable predictions.

Theory-Based Interventions Combining Mental Simulation and Planning Techniques to Improve Physical Activity: Null Results from Two Randomized Controlled Trials

Interventions to assist individuals in initiating and maintaining regular participation in physical activity are not always effective. Psychological and behavioral theories advocate the importance of both motivation and volition in interventions to change health behavior. Interventions adopting self-regulation strategies that foster motivational and volitional components may, therefore, have utility in promoting regular physical activity participation. We tested the efficacy of an intervention adopting motivational (mental simulation) and volitional (implementation intentions) components to promote a regular physical activity in two studies. Study 1 adopted a cluster randomized design in which participants (n = 92) were allocated to one of three conditions: mental simulation plus implementation intention, implementation intention only, or control. Study 2 adopted a 2 (mental simulation vs. no mental simulation) × 2 (implementation intention vs. no implementation intention) randomized controlled design in which fitness center attendees (n = 184) were randomly allocated one of four conditions: mental simulation only, implementation intention only, combined, or control. Physical activity behavior was measured by self-report (Study 1) or fitness center attendance (Study 2) at 4- (Studies 1 and 2) and 19- (Study 2 only) week follow-up periods. Findings revealed no statistically significant main or interactive effects of the mental simulation and implementation intention conditions on physical activity outcomes in either study. Findings are in contrast to previous research which has found pervasive effects for both intervention strategies. Findings are discussed in light of study limitations including the relatively small sample sizes, particularly for Study 1, deviations in the operationalization of the intervention components from previous research and the lack of a prompt for a goal intention. Future research should focus on ensuring uniformity in the format of the intervention components, test the effects of each component alone and in combination using standardized measures across multiple samples, and systematically explore effects of candidate moderators.

Does Language Matter? Exploring Chinese-Korean Differences in Holistic Perception

Cross-cultural research suggests that East Asians display a holistic attentional bias by paying attention to the entire field and to relationships between objects, whereas Westerners pay attention primarily to salient objects, displaying an analytic attentional bias. The assumption of a universal pan-Asian holistic attentional bias has recently been challenged in experimental research involving Japanese and Chinese participants, which suggests that linguistic factors may contribute to the formation of East Asians' holistic attentional patterns. The present experimental research explores differences in attention and information processing styles between Korean and Chinese speakers, who have been assumed to display the same attentional bias due to cultural commonalities. We hypothesize that the specific structure of the Korean language predisposes speakers to pay more attention to ground information than to figure information, thus leading to a stronger holistic attentional bias compared to Chinese speakers. Findings of the present research comparing different groups of English, Chinese, and Korean speakers provide further evidence for differences in East Asians' holistic attentional bias, which may be due to the influence of language. Furthermore, we also extend prior theorizing by discussing the potential impact of other cultural factors. In line with critical voices calling for more research investigating differences between cultures that are assumed to be culturally similar, we highlight important avenues for future studies exploring the language-culture relationship.

Die Vertrauenskrise empirischer Forschung in der Psychologie

Zusammenfassung. Ziel des vorliegenden Artikels ist es darzustellen, warum aktuell eine Vertrauenskrise der psychologischen Forschung postuliert wird. Wir beschreiben die Anzeichen der Vertrauenskrise bevor wir uns ihren mutmaßlichen Ursachen zuwenden: Researcher Degrees of Freedom, kleine Stichprobengrößen, Analyse- und Publikationspraktiken. Zusammen genommen haben diese Faktoren Anlass gegeben, die Reproduzierbarkeit psychologischer Forschung anzuzweifeln. Anschließend präsentieren wir ausgewählte Vorschläge, wie die Reproduzierbarkeit psychologischer Forschung erhöht werden kann. Mit dem vorliegenden Artikel möchten wir zu einer Forschungskultur innerhalb der Sportpsychologie beitragen, die geprägt ist von Reproduzierbarkeit, Transparenz und Vertrauen.

Misconceptions of the p-value among Chilean and Italian Academic Psychologists

Common misconceptions of p-values are based on certain beliefs and attributions about the significance of the results. Thus, they affect the professionals' decisions and jeopardize the quality of interventions and the accumulation of valid scientific knowledge. We conducted a survey on 164 academic psychologists (134 Italian, 30 Chilean) questioned on this topic. Our findings are consistent with previous research and suggest that some participants do not know how to correctly interpret p-values. The inverse probability fallacy presents the greatest comprehension problems, followed by the replication fallacy. These results highlight the importance of the statistical re-education of researchers. Recommendations for improving statistical cognition are proposed.

The experimenters' regress reconsidered: Replication, tacit knowledge, and the dynamics of knowledge generation

This paper revisits the debate between Harry Collins and Allan Franklin, concerning the experimenters' regress. Focusing my attention on a case study from recent psychology (regarding experimental evidence for the existence of a Mozart Effect), I argue that Franklin is right to highlight the role of epistemological strategies in scientific practice, but that his account does not sufficiently appreciate Collins's point about the importance of tacit knowledge in experimental practice. In turn, Collins rightly highlights the epistemic uncertainty (and skepticism) surrounding much experimental research. However, I will argue that his analysis of tacit knowledge fails to elucidate the reasons why scientists often are (and should be) skeptical of other researchers' experimental results. I will present an analysis of tacit knowledge in experimental research that not only answers to this desideratum, but also shows how such skepticism can in fact be a vital enabling factor for the dynamic processes of experimental knowledge generation.