Registered Replication Report: Schooler and Engstler-Schooler (1990)

Heterogeneity in effect size estimates

A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.

Phenotypic mismatch between suspects and fillers but not phenotypic bias increases eyewitness identifications of Black suspects

Introduction

Despite converging evidence that people more closely associate the construct of criminality with Black people who exhibit a more African facial phenotype than Black people who express a more European phenotype, eyewitness researchers have largely ignored phenotypic bias as a potential contributor to the racial disparities in the criminal legal system. If this form of phenotypic bias extends to eyewitness identification tasks, eyewitnesses may be more likely to identify Black suspects with an African rather than European phenotype, regardless of their guilt status. Further, in cases where the witness's description of the perpetrator does not contain phenotypic information, phenotypic mismatch between the suspect and the other lineup members may bias identification decisions toward or against the suspect. If witnesses can use elements of the lineup construction to guide their identification decisions rather than relying on their recognition memory, then the lineup should be deemed unfair due to suspect bias. The current study also investigated lineup presentation method as a procedural safeguard, predicting that that when lineups were presented simultaneously, there would be a significant two-way interaction of phenotypic bias and lineup composition, with a larger simple main effect of phenotypic bias when lineups were suspect-biased (i.e., the fillers were a phenotypic mismatch to the suspect) than when all lineup members shared the same phenotype. We expected that this interaction would be significantly smaller or non-significant for sequential lineups.

Methods

Participants watched a mock crime video that contained a Black culprit with either a more African phenotype or a less African phenotype before attempting identifications from a photo array that contained a suspect whose phenotype always matched the culprit viewed in the video, but varied in culprit-presence, phenotypic match of the suspect and fillers, and presentation method.

Results

Participants did not identify Black suspects with Afrocentric features more often than Black suspects with Eurocentric features. However, witnesses made more identifications of suspects when the fillers did not match the suspect's phenotype compared to when all lineup members possessed similar phenotypic features.

Discussion

In sum, phenotypic bias did not influence our participant-witnesses' identification decisions, nor interact with lineup composition and lineup presentation type to affect identifications of suspects, suggesting that phenotypic bias may be less influential in match-to-memory tasks than other types of legal decision-making (e.g., determining guilt and sentencing). However, the suggestiveness created by failing to match fillers' phenotypes to the suspect's phenotype can be avoided with proper attention to fair lineup construction.

No verbal overshadowing in aphantasia: The role of visual imagery for the verbal overshadowing effect

The verbal overshadowing effect refers to the phenomenon that the verbal description of a past complex stimulus impairs its subsequent recognition. Theoretical explanations range from interference between different mental representations to the activation of different processing orientations or a provoked shift in the recognition criterion. In our study, 61 participants with aphantasia (= lack of mental imagery) and 70 controls participated in a verbal overshadowing paradigm. The verbal overshadowing effect did not occur in people with aphantasia, although the effect was replicated in controls. We speculate that this is either due to the lack of visual representations in people with aphantasia that verbal descriptions could interfere with, or to the absence of a shift in processing orientation during verbalisation. To rule out criterion-based explanations, further research is needed to distinguish between discriminability and response bias in people with aphantasia. Finally, data indicated that the verbal overshadowing effect may even be reversed in individuals with aphantasia, partly due to a lower memory performance in the no verbalisation condition. Effects of further variables are discussed, such as mental strategies, memory confidence, and difficulty, quantity and quality of verbalisation.

Verbal interference paradigms: A systematic review investigating the role of language in cognition

This paper presents a systematic review of the empirical literature that uses dual-task interference methods for investigating the on-line involvement of language in various cognitive tasks. In these studies, participants perform some primary task X putatively recruiting linguistic resources while also engaging in a secondary, concurrent task. If performance on the primary task decreases under interference, there is evidence for language involvement in the primary task. We assessed studies (N = 101) reporting at least one experiment with verbal interference and at least one control task (either primary or secondary). We excluded papers with an explicitly clinical, neurological, or developmental focus. The primary tasks identified include categorization, memory, mental arithmetic, motor control, reasoning (verbal and visuospatial), task switching, theory of mind, visual change, and visuospatial integration and wayfinding. Overall, the present review found that covert language is likely to play a facilitative role in memory and categorization when items to be remembered or categorized have readily available labels, when inner speech can act as a form of behavioral self-cuing (inhibitory control, task set reminders, verbal strategy), and when inner speech is plausibly useful as "workspace," for example, for mental arithmetic. There is less evidence for the role of covert language in cross-modal integration, reasoning relying on a high degree of visual detail or items low on nameability, and theory of mind. We discuss potential pitfalls and suggestions for streamlining and improving the methodology.

Lineup identification in young and older witnesses: does describing the criminal help or hinder?

The world population is getting older and, as a result, the number of older victims of crime is expected to increase. It is therefore essential to understand how ageing affects eyewitness identification, so procedures can be developed that enable victims of crime of all ages to provide evidence as accurately and reliably as possible. In criminal investigations, witnesses often provide a description of the perpetrator of the crime before later making an identification. While describing the perpetrator prior to making a lineup identification can have a detrimental effect on identification in younger adults, referred to as verbal overshadowing, it is unclear whether older adults are affected in the same way. Our study compared lineup identification of a group of young adults and a group of older adults using the procedure that has consistently revealed verbal overshadowing in young adults. Participants watched a video of a mock crime. Following a 20-min filled delay, they either described the perpetrator or completed a control task. Immediately afterwards, they identified the perpetrator from a lineup, or indicated that the perpetrator was not present, and rated their confidence. We found that describing the perpetrator decreased subsequent correct identification of the perpetrator in both young and older adults. This effect of verbal overshadowing was not explained by a change in discrimination but was consistent with participants adopting a more conservative criterion. Confidence and response time were both found to predict identification accuracy for young and older groups, particularly in the control condition.

Without more theory, psychology will be a headless rider

We argue that Yarkoni's proposed solutions to the generalizability crisis are half-measures because he does not recognize that the crisis arises from investigators' underappreciation of the roles of theory in experimental research. Rather than embracing qualitative analysis, the research community should make an effort to develop better theories and work toward consistently incorporating theoretical results into experimental practice.

Replicability, Robustness, and Reproducibility in Psychological Science

Replication-an important, uncommon, and misunderstood practice-is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Four dimensions characterize attributions from faces using a representative set of English trait words

People readily (but often inaccurately) attribute traits to others based on faces. While the details of attributions depend on the language available to describe social traits, psychological theories argue that two or three dimensions (such as valence and dominance) summarize social trait attributions from faces. However, prior work has used only a small number of trait words (12 to 18), limiting conclusions to date. In two large-scale, preregistered studies we ask participants to rate 100 faces (obtained from existing face stimuli sets), using a list of 100 English trait words that we derived using deep neural network analysis of words that have been used by other participants in prior studies to describe faces. In study 1 we find that these attributions are best described by four psychological dimensions, which we interpret as “warmth”, “competence”, “femininity”, and “youth”. In study 2 we partially reproduce these four dimensions using the same stimuli among additional participant raters from multiple regions around the world, in both aggregated and individual-level data. These results provide a comprehensive characterization of trait attributions from faces, although we note our conclusions are limited by the scope of our study (in particular we note only white faces and English trait words were included).

People form impressions about others from seeing their faces, and use many words to describe those impressions. Here, using ratings of 100 representatively sampled white adult faces on a large set of traits, the authors show that trait impressions from faces can be summarized by four psychological dimensions.

Never forget a face: Verbalization facilitates recollection as evidenced by flexible responding to contrasting recognition memory tests

Verbal facilitation occurs when describing a face improves its subsequent recognition; but there are several theoretical explanations debated in the literature. The results of the present studies support a relatively unrestricted, parsimonious theory that verbal facilitation occurs because describing a face supports recollection of several different facets of the face-viewing experience. This recollection is then demonstrated by flexibly responding to two competing types of recognition task demands. Participants studied a list of faces and, following each face, performed a nonverbalization task (Experiment 1) or described its features or traits (Experiment 2). Two subsequent recognition tests included intact faces, new faces, and conjunctions (each of which recombined features of two studied faces). Inclusion test instructions emphasized featural information: respond "yes" to both intact and conjunction faces (both of which contained studied features), but "no" to new faces. Exclusion test instructions emphasized configural information: respond "yes" only to intact faces (which were the only test items that matched studied configurations), and "no" to both conjunctions and new faces. Both yes/no responses and confidence ratings supported our hypothesis that verbalization improved discrimination between (a) conjunctions and new faces in the inclusion test, and (b) intact faces and conjunctions in the exclusion test. Additional secondary responses about face type elucidated that verbalization at study improves the ability to recollect either featural or configural information, depending on which type of response the recognition test required. We discuss these findings about practical applications of improved face memory in real-world contexts.

Les pratiques de recherche ouvertes en psychologie

Cet article vise à offrir une vision d’ensemble des récentes évolutions des pratiques de recherche en psychologie. Un rappel des différents symptômes de la crise de la réplicabilité (et de confiance) ayant affecté la psychologie sera suivi par une discussion approfondie et nuancée des facteurs responsables de cette situation. Il s’agira ensuite, en s’appuyant sur des illustrations et des ressources, de démontrer le rôle crucial des pratiques de recherche ouvertes comme moyen de résoudre ces difficultés. La connaissance et l’adoption de ces pratiques de recherche popularisées par le mouvement de la science ouverte sont indispensables afin de contribuer, via la transparence et l’ouverture, à l’effort collectif d’amélioration de la fiabilité et de la réplicabilité des résultats en psychologie.

The generalizability crisis

Most theories and hypotheses in psychology are verbal in nature, yet their evaluation overwhelmingly relies on inferential statistical procedures. The validity of the move from qualitative to quantitative analysis depends on the verbal and statistical expressions of a hypothesis being closely aligned - that is, that the two must refer to roughly the same set of hypothetical observations. Here, I argue that many applications of statistical inference in psychology fail to meet this basic condition. Focusing on the most widely used class of model in psychology - the linear mixed model - I explore the consequences of failing to statistically operationalize verbal hypotheses in a way that respects researchers' actual generalization intentions. I demonstrate that although the "random effect" formalism is used pervasively in psychology to model intersubject variability, few researchers accord the same treatment to other variables they clearly intend to generalize over (e.g., stimuli, tasks, or research sites). The under-specification of random effects imposes far stronger constraints on the generalizability of results than most researchers appreciate. Ignoring these constraints can dramatically inflate false-positive rates, and often leads researchers to draw sweeping verbal generalizations that lack a meaningful connection to the statistical quantities they are putatively based on. I argue that failure to take the alignment between verbal and statistical expressions seriously lies at the heart of many of psychology's ongoing problems (e.g., the replication crisis), and conclude with a discussion of several potential avenues for improvement.

A Validity-Based Framework for Understanding Replication in Psychology

In recent years, psychology has wrestled with the broader implications of disappointing rates of replication of previously demonstrated effects. This article proposes that many aspects of this pattern of results can be understood within the classic framework of four proposed forms of validity: statistical conclusion validity, internal validity, construct validity, and external validity. The article explains the conceptual logic for how differences in each type of validity across an original study and a subsequent replication attempt can lead to replication “failure.” Existing themes in the replication literature related to each type of validity are also highlighted. Furthermore, empirical evidence is considered for the role of each type of validity in non-replication. The article concludes with a discussion of broader implications of this classic validity framework for improving replication rates in psychological research.

Cross-codal integration of bridging-event information in narrative understanding

Visual narratives communicate event sequences by using different code systems such as pictures and texts. Thus, comprehenders must integrate information from different codalities. This study addressed such cross-codal integration processes by investigating how the codality of bridging-event information (i.e., pictures, text) affects the understanding of visual narrative events. In Experiment 1, bridging-event information was either present (as picture or text) or absent (i.e., not shown). The viewing times for the subsequent picture depicting the end state of the action were comparable within the absent and the text conditions. Further, the viewing times for the end-state picture were significantly longer in the text condition as compared to the pictorial condition. In Experiment 2, we tested whether replacing bridging-event information with a blank panel increases viewing times in a way similar to the text condition. Bridging event information was either present (as picture) or absent (not shown vs. blank panel). The results replicated Experiment 1. Additionally, the viewing times for the end-state pictures were longest in the blank condition. In Experiment 3, we investigated the costs related to integrating information from different codalities by directly comparing the text and picture conditions with the blank condition. The results showed that the distortion caused by the blank panel is larger than the distortion caused by cross-codal integration processes. Summarizing, we conclude that cross-codal information processing during narrative comprehension is possible but associated with additional mental effort. We discuss the results with regard to theories of narrative understanding.

No relationship between researcher impact and replication effect: an analysis of five studies with 100 replications

What explanation is there when teams of researchers are unable to successfully replicate already established 'canonical' findings? One suggestion that has been put forward, but left largely untested, is that those researchers who fail to replicate prior studies are of low 'expertise and diligence' and lack the skill necessary to successfully replicate the conditions of the original experiment. Here we examine the replication success of 100 scientists of differing 'expertise and diligence' who attempted to replicate five different studies. Using a bibliometric tool (h-index) as our indicator of researcher 'expertise and diligence', we examine whether this was predictive of replication success. Although there was substantial variability in replication success and in the h-factor of the investigators, we find no relationship between these variables. The present results provide no evidence for the hypothesis that systematic replications fail because of low 'expertise and diligence' among replicators.

Comparing meta-analyses and preregistered multiple-laboratory replication projects

Many researchers rely on meta-analysis to summarize research evidence. However, there is a concern that publication bias and selective reporting may lead to biased meta-analytic effect sizes. We compare the results of meta-analyses to large-scale preregistered replications in psychology carried out at multiple laboratories. The multiple-laboratory replications provide precisely estimated effect sizes that do not suffer from publication bias or selective reporting. We searched the literature and identified 15 meta-analyses on the same topics as multiple-laboratory replications. We find that meta-analytic effect sizes are significantly different from replication effect sizes for 12 out of the 15 meta-replication pairs. These differences are systematic and, on average, meta-analytic effect sizes are almost three times as large as replication effect sizes. We also implement three methods of correcting meta-analysis for bias, but these methods do not substantively improve the meta-analytic results.

Replicator degrees of freedom allow publication of misleading failures to replicate

We show that commonly exercised flexibility at the experimental design and data analysis stages of replication testing makes it easy to publish false-negative replication results while maintaining the impression of methodological rigor. These findings have important implications for how the many ostensible nonreplications already in the literature should be interpreted and for how future replication tests should be conducted.

In recent years, the field of psychology has begun to conduct replication tests on a large scale. Here, we show that “replicator degrees of freedom” make it far too easy to obtain and publish false-negative replication results, even while appearing to adhere to strict methodological standards. Specifically, using data from an ongoing debate, we show that commonly exercised flexibility at the experimental design and data analysis stages of replication testing can make it appear that a finding was not replicated when, in fact, it was. The debate that we focus on is representative, on key dimensions, of a large number of other replication tests in psychology that have been published in recent years, suggesting that the lessons of this analysis may be far reaching. The problems with current practice in replication science that we uncover here are particularly worrisome because they are not adequately addressed by the field’s standard remedies, including preregistration. Implications for how the field could develop more effective methodological standards for replication are discussed.

Screen Recordings as a Tool to Document Computer Assisted Data Collection Procedures

Recently in psychological science and many related fields, a surprisingly large amount of experiments could not be replicated by independent researchers. A non-replication could indicate that a previous finding might have been a false positive statistical result and the effect does not exist. However, it could also mean that a specific detail of the experimental procedure is essential for the effect to emerge, which might not have been included in the replication attempt. Therefore any replication attempt that does not replicate the original effect is most informative when the original procedure is closely adhered to. One proposed solution to facilitate the empirical reproducibility of the experimental procedures in psychology is to upload the experimental script and materials to a public repository. However, we believe that merely providing the materials of an experimental procedure is not sufficient, as many software solutions are not freely available, software solutions might change, and it is time consuming to set up the procedure. We argue that there is a simple solution to these problems when an experiment is conducted using computers: recording an example procedure with a screen capture software and providing the video in an online repository. We therefore provide a brief tutorial on screen recordings using an open source screen recording software. With this information, individual researchers should be able to record their experimental procedures and we hope to facilitate the use of screen recordings in computer assisted data collection procedures.

Effects of Auditory Distraction on Face Memory

Effects of auditory distraction by task-irrelevant background speech on the immediate serial recall of verbal material are well established. Less is known about the influence of background speech on memory for visual configural information. A recent study demonstrated that face learning is disrupted by joyful music relative to soothing violin music and quiet. This pattern is parallel to findings in the serial-recall paradigm showing that auditory distraction is primarily caused by auditory changes. Here we connect these two streams of research by testing whether face learning is impaired by irrelevant speech. Participants learned faces either in quiet or while ignoring auditory changing-state sequences (sentential speech) or steady-state sequences (word repetitions). Face recognition was impaired by irrelevant speech relative to quiet. Furthermore, changing-state speech disrupted performance more than steady-state speech. The results were replicated in a second study using reversed speech, suggesting that the disruptive potential of the background speech does not depend on its semantic content. These findings thus demonstrate robust effects of auditory distraction on face learning. Theoretical explanations and applied implications are discussed.

Scientific Utopia III: Crowdsourcing Science

Most scientific research is conducted by small teams of investigators who together formulate hypotheses, collect data, conduct analyses, and report novel findings. These teams operate independently as vertically integrated silos. Here we argue that scientific research that is horizontally distributed can provide substantial complementary value, aiming to maximize available resources, promote inclusiveness and transparency, and increase rigor and reliability. This alternative approach enables researchers to tackle ambitious projects that would not be possible under the standard model. Crowdsourced scientific initiatives vary in the degree of communication between project members from largely independent work curated by a coordination team to crowd collaboration on shared activities. The potential benefits and challenges of large-scale collaboration span the entire research process: ideation, study design, data collection, data analysis, reporting, and peer review. Complementing traditional small science with crowdsourced approaches can accelerate the progress of science and improve the quality of scientific research.

Excess Success in “Ray of hope: Hopelessness Increases Preferences for Brighter Lighting”

Dong, Huang, and Zhong (2015) report five successful experiments linking brightness perception with the feeling of hopelessness. They argue that a gloomy future is psychologically represented as darkness, not just metaphorically but as an actual perceptual bias. Based on multiple results, they conclude that people who feel hopeless perceive their environment as darker and therefore prefer brighter lighting than controls. Reversely, dim lighting caused participants to feel more hopeless. However, the experiments succeed at a rate much higher than predicted by the magnitude of the reported effects. Based on the reported statistics, the estimated probability of all five experiments being fully successful, if replicated with the same sample sizes, is less than 0.016. This low rate suggests that the original findings are (perhaps unintentionally) the result of questionable research practices or publication bias. Readers should therefore be skeptical about the original results and conclusions. Finally, we discuss how to design future studies to investigate the relationship between hopelessness and brightness.

Should social scientists be distanced from or engaged with the people they study?

This commentary focuses on two important contrasts in the behavioral sciences: (i) default versus nondefault study populations, where default samples have been used disproportionately (for psychology, the default is undergraduates at major research universities), and (ii) the adoption of a distant versus close (engaged) attitude toward study samples. Previous research has shown a strong correlation between these contrasts, where default samples and distant perspectives are the norm. Distancing is sometimes seen as necessary for objectivity, and an engaged orientation is sometimes criticized as biased, advocacy research, especially if the researcher shares a social group membership with the study population (e.g., a black male researcher studying black male students). The lack of diversity in study samples has been paralleled by a lack of diversity in the researchers themselves. The salience of default samples and distancing in prior research creates potential (and presumed) risk factors for engaged research with nondefault samples. However, a distant perspective poses risks as well, and particularly so for research with nondefault populations. We suggest that engaged research can usefully encourage attention to the study context and taking the perspective of study samples, both of which are good research practices. More broadly, we argue that social and educational sciences need skepticism, interestedness, and engagement, not distancing. Fostering an engaged perspective in research may also foster a more diverse population of social scientists.

Participant Nonnaiveté and the reproducibility of cognitive psychology

Many argue that there is a reproducibility crisis in psychology. We investigated nine well-known effects from the cognitive psychology literature-three each from the domains of perception/action, memory, and language, respectively-and found that they are highly reproducible. Not only can they be reproduced in online environments, but they also can be reproduced with nonnaïve participants with no reduction of effect size. Apparently, some cognitive tasks are so constraining that they encapsulate behavior from external influences, such as testing situation and prior recent experience with the experiment to yield highly robust effects.

The Psychological Science Accelerator: Advancing Psychology through a Distributed Collaborative Network

Concerns have been growing about the veracity of psychological research. Many findings in psychological science are based on studies with insufficient statistical power and nonrepresentative samples, or may otherwise be limited to specific, ungeneralizable settings or populations. Crowdsourced research, a type of large-scale collaboration in which one or more research projects are conducted across multiple lab sites, offers a pragmatic solution to these and other current methodological challenges. The Psychological Science Accelerator (PSA) is a distributed network of laboratories designed to enable and support crowdsourced research projects. These projects can focus on novel research questions, or attempt to replicate prior research, in large, diverse samples. The PSA's mission is to accelerate the accumulation of reliable and generalizable evidence in psychological science. Here, we describe the background, structure, principles, procedures, benefits, and challenges of the PSA. In contrast to other crowdsourced research networks, the PSA is ongoing (as opposed to time-limited), efficient (in terms of re-using structures and principles for different projects), decentralized, diverse (in terms of participants and researchers), and inclusive (of proposals, contributions, and other relevant input from anyone inside or outside of the network). The PSA and other approaches to crowdsourced psychological science will advance our understanding of mental processes and behaviors by enabling rigorous research and systematically examining its generalizability.

Replication and Reproducibility in Cross-Cultural Psychology

Replication is the scientific gold standard that enables the confirmation of research findings. Concerns related to publication bias, flexibility in data analysis, and high-profile cases of academic misconduct have led to recent calls for more replication and systematic accumulation of scientific knowledge in psychological science. This renewed emphasis on replication may pose specific challenges to cross-cultural research due to inherent practical difficulties in emulating an original study in other cultural groups. The purpose of the present article is to discuss how the core concepts of this replication debate apply to cross-cultural psychology. Distinct to replications in cross-cultural research are examinations of bias and equivalence in manipulations and procedures, and that targeted research populations may differ in meaningful ways. We identify issues in current psychological research (analytic flexibility, low power) and possible solutions (preregistration, power analysis), and discuss ways to implement best practices in cross-cultural replication attempts.

Hypothesis-Testing Demands Trustworthy Data-A Simulation Approach to Inferential Statistics Advocating the Research Program Strategy

In psychology as elsewhere, the main statistical inference strategy to establish empirical effects is null-hypothesis significance testing (NHST). The recent failure to replicate allegedly well-established NHST-results, however, implies that such results lack sufficient statistical power, and thus feature unacceptably high error-rates. Using data-simulation to estimate the error-rates of NHST-results, we advocate the research program strategy (RPS) as a superior methodology. RPS integrates Frequentist with Bayesian inference elements, and leads from a preliminary discovery against a (random) H₀-hypothesis to a statistical H₁-verification. Not only do RPS-results feature significantly lower error-rates than NHST-results, RPS also addresses key-deficits of a “pure” Frequentist and a standard Bayesian approach. In particular, RPS aggregates underpowered results safely. RPS therefore provides a tool to regain the trust the discipline had lost during the ongoing replicability-crisis.

Belastbare und effiziente Wissenschaft

Zusammenfassung. Die Replikationskrise innerhalb der Psychologie hat eine Diskussion über gängige Praktiken im Forschungsprozess und die beteiligten Institutionen ausgelöst. Wir stellen Maßnahmen vor, die dazu beitragen können, Forschung effizienter und Forschungsergebnisse belastbarer zu machen, indem sie eine strategische Ausrichtung und stärkere Verzahnung von Forschungsprozessen in Gang setzen. Die notwendigen Veränderungen setzen dabei auf der Ebene des Individuums sowie der Institutionen an und betreffen die Inhaltsbereiche Theorie, Empirie und Evidenzakkumulation. Über die bekannten eher spezifischen Maßnahmen zur Verbesserung der Reproduzierbarkeit hinaus, zielen unsere Vorschläge darauf ab, die Voraussetzungen für effizientes wissenschaftliches Arbeiten zu verbessern, indem u. a. (i) die exakte Spezifikation und die kritische Testung und Revision von Theorien wieder stärker in den Mittelpunkt der Forschung gerückt werden, (ii) eine Kultur der Transparenz, akzeptierten Fehlbarkeit und Offenheit für Empirie- aber insbesondere auch Theorierevision etabliert wird, (iii) diese Kultur und die damit verbundene Methodik als integrale Bestandteile durchgängig in der Lehre verankert werden und (iv) Evidenz und Theorien in fortlaufend dezentral aktualisierten und miteinander verknüpften Theorie- und Empirie-Datenbanken zusammengeführt werden.

Collections2: Using “Crowdsourcing” within Psychological Research

“Crowdsourcing” is a methodological approach in which several researchers coordinate their resources to achieve research goals that would otherwise be difficult to attain individually. This article introduces a Nexus—a collection of empirical and theoretical articles that will be published in Collabra: Psychology—that is intended to encourage more crowdsourced research in psychological science by providing a specific outlet for such projects and by assisting researchers in developing and executing their projects. We describe how individuals can propose and lead a crowdsourced research project, how individuals can contribute to other ongoing projects, and other ways to contribute to this Nexus. Ultimately, we hope this Nexus will contain a set of highly-informative articles that demonstrate the flexibility and range of the types of research questions that can be addressed with crowdsourced research methods.

Making replication mainstream

Many philosophers of science and methodologists have argued that the ability to repeat studies and obtain similar results is an essential component of science. A finding is elevated from single observation to scientific evidence when the procedures that were used to obtain it can be reproduced and the finding itself can be replicated. Recent replication attempts show that some high profile results – most notably in psychology, but in many other disciplines as well – cannot be replicated consistently. These replication attempts have generated a considerable amount of controversy, and the issue of whether direct replications have value has, in particular, proven to be contentious. However, much of this discussion has occurred in published commentaries and social media outlets, resulting in a fragmented discourse. To address the need for an integrative summary, we review various types of replication studies and then discuss the most commonly voiced concerns about direct replication. We provide detailed responses to these concerns and consider different statistical ways to evaluate replications. We conclude there are no theoretical or statistical obstacles to making direct replication a routine aspect of psychological science.

The interplay of language and visual perception in working memory

How do perception and language interact to form the representations that guide our thoughts and actions over the short-term? Here, we provide a first examination of this question by investigating the role of verbal labels in a continuous visual working memory (WM) task. Across four experiments, participants retained in memory the continuous color of a set of dots which were presented sequentially (Experiments 1-3) or simultaneously (Experiment 4). At test, they reproduced the colors of all dots using a color wheel. During stimulus presentation participants were required to either label the colors (color labeling) or to repeat "bababa" aloud (articulatory suppression), hence prompting or preventing verbal labeling, respectively. We tested four competing hypotheses of the labeling effect: (1) labeling generates a verbal representation that overshadows the visual representation; (2) labeling yields a verbal representation in addition to the visual one; (3) the labels function as a retrieval cue, adding distinctiveness to items in memory; and (4) labels activate visual categorical representations in long-term memory. Collectively, our experiments show that labeling does not overshadow the visual input; it augments it. Mixture modeling showed that labeling increased the quantity and quality of information in WM. Our findings are consistent with the hypothesis that labeling activates visual long-term categorical representations which help in reducing the noise in the internal representations of the visual stimuli in WM.

Addressing the "Replication Crisis": Using Original Studies to Design Replication Studies with Appropriate Statistical Power

Psychology is undergoing a replication crisis. The discussion surrounding this crisis has centered on mistrust of previous findings. Researchers planning replication studies often use the original study sample effect size as the basis for sample size planning. However, this strategy ignores uncertainty and publication bias in estimated effect sizes, resulting in overly optimistic calculations. A psychologist who intends to obtain power of .80 in the replication study, and performs calculations accordingly, may have an actual power lower than .80. We performed simulations to reveal the magnitude of the difference between actual and intended power based on common sample size planning strategies and assessed the performance of methods that aim to correct for effect size uncertainty and/or bias. Our results imply that even if original studies reflect actual phenomena and were conducted in the absence of questionable research practices, popular approaches to designing replication studies may result in a low success rate, especially if the original study is underpowered. Methods correcting for bias and/or uncertainty generally had higher actual power, but were not a panacea for an underpowered original study. Thus, it becomes imperative that 1) original studies are adequately powered and 2) replication studies are designed with methods that are more likely to yield the intended level of power.

A Multilab Preregistered Replication of the Ego-Depletion Effect

Good self-control has been linked to adaptive outcomes such as better health, cohesive personal relationships, success in the workplace and at school, and less susceptibility to crime and addictions. In contrast, self-control failure is linked to maladaptive outcomes. Understanding the mechanisms by which self-control predicts behavior may assist in promoting better regulation and outcomes. A popular approach to understanding self-control is the strength or resource depletion model. Self-control is conceptualized as a limited resource that becomes depleted after a period of exertion resulting in self-control failure. The model has typically been tested using a sequential-task experimental paradigm, in which people completing an initial self-control task have reduced self-control capacity and poorer performance on a subsequent task, a state known as ego depletion Although a meta-analysis of ego-depletion experiments found a medium-sized effect, subsequent meta-analyses have questioned the size and existence of the effect and identified instances of possible bias. The analyses served as a catalyst for the current Registered Replication Report of the ego-depletion effect. Multiple laboratories (k = 23, total N = 2,141) conducted replications of a standardized ego-depletion protocol based on a sequential-task paradigm by Sripada et al. Meta-analysis of the studies revealed that the size of the ego-depletion effect was small with 95% confidence intervals (CIs) that encompassed zero (d = 0.04, 95% CI [-0.07, 0.15]. We discuss implications of the findings for the ego-depletion effect and the resource depletion model of self-control.

Implied motion language can influence visual spatial memory

How do language and vision interact? Specifically, what impact can language have on visual processing, especially related to spatial memory? What are typically considered errors in visual processing, such as remembering the location of an object to be farther along its motion trajectory than it actually is, can be explained as perceptual achievements that are driven by our ability to anticipate future events. In two experiments, we tested whether the prior presentation of motion language influences visual spatial memory in ways that afford greater perceptual prediction. Experiment 1 showed that motion language influenced judgments for the spatial memory of an object beyond the known effects of implied motion present in the image itself. Experiment 2 replicated this finding. Our findings support a theory of perception as prediction.

On the applied implications of the "verbal overshadowing effect"

Schooler and Engstler-Schooler (1990) found that participants who wrote out a description of the perpetrator's face after watching a simulated crime video were subsequently less likely to identify that perpetrator from a photo lineup compared to participants in a control condition (i.e., the correct ID rate was reduced). The first registered replication report in Perspectives on Psychological Science confirmed this verbal overshadowing effect (Alogna et al., 2014). Does this result indicate a reduced ability to recognize the person who was verbally described, or does it instead reflect more conservative responding? The answer depends on the still unknown likelihood of identifying an innocent suspect from a lineup (the false ID rate). Assuming the reduced correct ID rate does reflect memory impairment, should the legal system be advised to give less weight to a suspect identification if the witness previously provided a verbal description of the perpetrator? Intuitively, the answer is "yes," but without knowing the false ID rate, it is unclear if a suspect identification following a verbal description should be given less weight or more weight. This is true even if the correct and false ID rates show that verbal descriptions impair memory. In our view, psychologists should withhold giving advice to the legal system about the effect of verbal descriptions on suspect identifications until the issue is investigated by including lineups that contain an innocent suspect.

A Bayesian bird's eye view of 'Replications of important results in social psychology'

We applied three Bayesian methods to reanalyse the preregistered contributions to the Social Psychology special issue 'Replications of Important Results in Social Psychology' (Nosek & Lakens. 2014 Registered reports: a method to increase the credibility of published results. Soc. Psychol.45, 137-141. (doi:10.1027/1864-9335/a000192)). First, individual-experiment Bayesian parameter estimation revealed that for directed effect size measures, only three out of 44 central 95% credible intervals did not overlap with zero and fell in the expected direction. For undirected effect size measures, only four out of 59 credible intervals contained values greater than [Formula: see text] (10% of variance explained) and only 19 intervals contained values larger than [Formula: see text]. Second, a Bayesian random-effects meta-analysis for all 38 t-tests showed that only one out of the 38 hierarchically estimated credible intervals did not overlap with zero and fell in the expected direction. Third, a Bayes factor hypothesis test was used to quantify the evidence for the null hypothesis against a default one-sided alternative. Only seven out of 60 Bayes factors indicated non-anecdotal support in favour of the alternative hypothesis ([Formula: see text]), whereas 51 Bayes factors indicated at least some support for the null hypothesis. We hope that future analyses of replication success will embrace a more inclusive statistical approach by adopting a wider range of complementary techniques.