1
|
Field SM, Wagenmakers EJ, Kiers HAL, Hoekstra R, Ernst AF, van Ravenzwaaij D. The effect of preregistration on trust in empirical research findings: results of a registered report. R Soc Open Sci 2020; 7:181351. [PMID: 32431853 PMCID: PMC7211853 DOI: 10.1098/rsos.181351] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/15/2018] [Accepted: 02/19/2020] [Indexed: 05/19/2023]
Abstract
The crisis of confidence has undermined the trust that researchers place in the findings of their peers. In order to increase trust in research, initiatives such as preregistration have been suggested, which aim to prevent various questionable research practices. As it stands, however, no empirical evidence exists that preregistration does increase perceptions of trust. The picture may be complicated by a researcher's familiarity with the author of the study, regardless of the preregistration status of the research. This registered report presents an empirical assessment of the extent to which preregistration increases the trust of 209 active academics in the reported outcomes, and how familiarity with another researcher influences that trust. Contrary to our expectations, we report ambiguous Bayes factors and conclude that we do not have strong evidence towards answering our research questions. Our findings are presented along with evidence that our manipulations were ineffective for many participants, leading to the exclusion of 68% of complete datasets, and an underpowered design as a consequence. We discuss other limitations and confounds which may explain why the findings of the study deviate from a previously conducted pilot study. We reflect on the benefits of using the registered report submission format in light of our results. The OSF page for this registered report and its pilot can be found here: http://dx.doi.org/10.17605/OSF.IO/B3K75.
Collapse
Affiliation(s)
- Sarahanne M. Field
- Department of Psychometrics and Statistics, Rijksuniversiteit Groningen, Groningen, The Netherlands
| | - E.-J. Wagenmakers
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | - Henk A. L. Kiers
- Department of Psychometrics and Statistics, Rijksuniversiteit Groningen, Groningen, The Netherlands
| | - Rink Hoekstra
- Department of Psychometrics and Statistics, Rijksuniversiteit Groningen, Groningen, The Netherlands
| | - Anja F. Ernst
- Department of Psychometrics and Statistics, Rijksuniversiteit Groningen, Groningen, The Netherlands
| | - Don van Ravenzwaaij
- Department of Psychometrics and Statistics, Rijksuniversiteit Groningen, Groningen, The Netherlands
| |
Collapse
|
2
|
Abstract
Bayesian inference for rank-order problems is frustrated by the absence of an explicit likelihood function. This hurdle can be overcome by assuming a latent normal representation that is consistent with the ordinal information in the data: the observed ranks are conceptualized as an impoverished reflection of an underlying continuous scale, and inference concerns the parameters that govern the latent representation. We apply this generic data-augmentation method to obtain Bayes factors for three popular rank-based tests: the rank sum test, the signed rank test, and Spearman's ρs.
Collapse
Affiliation(s)
- J. van Doorn
- Department of Psychological Methods, University of Amsterdam, Amsterdam, the Netherlands
| | - A. Ly
- Department of Psychological Methods, University of Amsterdam, Amsterdam, the Netherlands
- Centrum voor Wiskunde & Informatica, Amsterdam, the Netherlands
| | - M. Marsman
- Department of Psychological Methods, University of Amsterdam, Amsterdam, the Netherlands
| | - E.-J. Wagenmakers
- Department of Psychological Methods, University of Amsterdam, Amsterdam, the Netherlands
| |
Collapse
|
3
|
Silberzahn R, Uhlmann EL, Martin DP, Anselmi P, Aust F, Awtrey E, Bahník Š, Bai F, Bannard C, Bonnier E, Carlsson R, Cheung F, Christensen G, Clay R, Craig MA, Dalla Rosa A, Dam L, Evans MH, Flores Cervantes I, Fong N, Gamez-Djokic M, Glenz A, Gordon-McKeon S, Heaton TJ, Hederos K, Heene M, Hofelich Mohr AJ, Högden F, Hui K, Johannesson M, Kalodimos J, Kaszubowski E, Kennedy DM, Lei R, Lindsay TA, Liverani S, Madan CR, Molden D, Molleman E, Morey RD, Mulder LB, Nijstad BR, Pope NG, Pope B, Prenoveau JM, Rink F, Robusto E, Roderique H, Sandberg A, Schlüter E, Schönbrodt FD, Sherman MF, Sommer SA, Sotak K, Spain S, Spörlein C, Stafford T, Stefanutti L, Tauber S, Ullrich J, Vianello M, Wagenmakers EJ, Witkowiak M, Yoon S, Nosek BA. Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. Advances in Methods and Practices in Psychological Science 2018. [DOI: 10.1177/2515245917747646] [Citation(s) in RCA: 267] [Impact Index Per Article: 44.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 ( Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.
Collapse
Affiliation(s)
- R. Silberzahn
- Organisational Behaviour, University of Sussex Business School
| | | | - D. P. Martin
- Department of Psychology, University of Virginia
| | - P. Anselmi
- Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua
| | - F. Aust
- Department of Psychology, University of Cologne
| | - E. Awtrey
- Department of Management, University of Cincinnati
| | - Š. Bahník
- Department of Management, Faculty of Business Administration, University of Economics, Prague
| | - F. Bai
- Department of Management and Marketing, Hong Kong Polytechnic University
| | - C. Bannard
- Department of Psychology, University of Liverpool
| | - E. Bonnier
- Department of Economics, Stockholm School of Economics
| | - R. Carlsson
- Department of Psychology, Linnaeus University
| | - F. Cheung
- School of Public Health, University of Hong Kong
| | - G. Christensen
- Berkeley Institute for Data Science, University of California, Berkeley
| | - R. Clay
- Department of Psychology, College of Staten Island, City University of New York
| | - M. A. Craig
- Department of Psychology, New York University
| | - A. Dalla Rosa
- Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua
| | - L. Dam
- Faculty of Economics and Business, University of Groningen
| | - M. H. Evans
- Division of Neuroscience and Experimental Psychology, University of Manchester
| | | | - N. Fong
- Department of Marketing and Supply Chain Management, Temple University
| | - M. Gamez-Djokic
- Department of Management and Organizations, Kellogg School of Management, Northwestern University
| | - A. Glenz
- Department of Psychology, University of Zurich
| | | | - T. J. Heaton
- School of Mathematics and Statistics, University of Sheffield
| | - K. Hederos
- Swedish Institute for Social Research (SOFI), Stockholm University
| | - M. Heene
- Department of Psychology, Ludwig-Maximilians-Universität München
| | | | - F. Högden
- Department of Psychology, University of Cologne
| | - K. Hui
- School of Management, Xiamen University
| | | | | | - E. Kaszubowski
- Department of Psychology, Federal University of Santa Catarina
| | - D. M. Kennedy
- School of Business, University of Washington Bothell
| | - R. Lei
- Department of Psychology, New York University
| | | | - S. Liverani
- School of Mathematical Sciences, Queen Mary University of London
| | - C. R. Madan
- School of Psychology, University of Nottingham
| | - D. Molden
- Department of Psychology, Northwestern University
| | - E. Molleman
- Faculty of Economics and Business, University of Groningen
| | | | - L. B. Mulder
- Faculty of Economics and Business, University of Groningen
| | - B. R. Nijstad
- Faculty of Economics and Business, University of Groningen
| | - N. G. Pope
- Department of Economics, University of Maryland
| | - B. Pope
- Department of Economics, Brigham Young University
| | | | - F. Rink
- Faculty of Economics and Business, University of Groningen
| | - E. Robusto
- Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua
| | - H. Roderique
- Rotman School of Management, University of Toronto
| | - A. Sandberg
- Swedish Institute for Social Research (SOFI), Stockholm University
| | - E. Schlüter
- Department of Social Sciences and Cultural Studies, Institute of Sociology, Justus Liebig University, Giessen
| | - F. D. Schönbrodt
- Department of Psychology, Ludwig-Maximilians-Universität München
| | - M. F. Sherman
- Department of Psychology, Loyola University Maryland
| | | | - K. Sotak
- Department of Marketing and Management, SUNY Oswego
| | - S. Spain
- John Molson School of Business, Concordia University
| | - C. Spörlein
- Lehrstuhl für Soziologie, insb. Sozialstrukturanalyse, Otto-Friedrich-Universität Bamberg
| | - T. Stafford
- Department of Psychology, University of Sheffield
| | - L. Stefanutti
- Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua
| | - S. Tauber
- Faculty of Economics and Business, University of Groningen
| | - J. Ullrich
- Department of Psychology, University of Zurich
| | - M. Vianello
- Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua
| | | | | | - S. Yoon
- Department of Marketing and Supply Chain Management, Temple University
| | - B. A. Nosek
- Department of Psychology, University of Virginia
- Center for Open Science, Charlottesville, Virginia
| |
Collapse
|
4
|
Wagenmakers EJ, Beek T, Dijkhoff L, Gronau QF, Acosta A, Adams RB, Albohn DN, Allard ES, Benning SD, Blouin-Hudon EM, Bulnes LC, Caldwell TL, Calin-Jageman RJ, Capaldi CA, Carfagno NS, Chasten KT, Cleeremans A, Connell L, DeCicco JM, Dijkstra K, Fischer AH, Foroni F, Hess U, Holmes KJ, Jones JLH, Klein O, Koch C, Korb S, Lewinski P, Liao JD, Lund S, Lupianez J, Lynott D, Nance CN, Oosterwijk S, Ozdoğru AA, Pacheco-Unguetti AP, Pearson B, Powis C, Riding S, Roberts TA, Rumiati RI, Senden M, Shea-Shumsky NB, Sobocko K, Soto JA, Steiner TG, Talarico JM, van Allen ZM, Vandekerckhove M, Wainwright B, Wayand JF, Zeelenberg R, Zetzer EE, Zwaan RA. Registered Replication Report. Perspect Psychol Sci 2016; 11:917-928. [DOI: 10.1177/1745691616674458] [Citation(s) in RCA: 201] [Impact Index Per Article: 25.1] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
According to the facial feedback hypothesis, people’s affective responses can be influenced by their own facial expression (e.g., smiling, pouting), even when their expression did not result from their emotional experiences. For example, Strack, Martin, and Stepper (1988) instructed participants to rate the funniness of cartoons using a pen that they held in their mouth. In line with the facial feedback hypothesis, when participants held the pen with their teeth (inducing a “smile”), they rated the cartoons as funnier than when they held the pen with their lips (inducing a “pout”). This seminal study of the facial feedback hypothesis has not been replicated directly. This Registered Replication Report describes the results of 17 independent direct replications of Study 1 from Strack et al. (1988), all of which followed the same vetted protocol. A meta-analysis of these studies examined the difference in funniness ratings between the “smile” and “pout” conditions. The original Strack et al. (1988) study reported a rating difference of 0.82 units on a 10-point Likert scale. Our meta-analysis revealed a rating difference of 0.03 units with a 95% confidence interval ranging from −0.11 to 0.16.
Collapse
|
5
|
Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ, Buck S, Chambers CD, Chin G, Christensen G, Contestabile M, Dafoe A, Eich E, Freese J, Glennerster R, Goroff D, Green DP, Hesse B, Humphreys M, Ishiyama J, Karlan D, Kraut A, Lupia A, Mabry P, Madon TA, Malhotra N, Mayo-Wilson E, McNutt M, Miguel E, Paluck EL, Simonsohn U, Soderberg C, Spellman BA, Turitto J, VandenBos G, Vazire S, Wagenmakers EJ, Wilson R, Yarkoni T. SCIENTIFIC STANDARDS. Promoting an open research culture. Science 2015; 348:1422-5. [PMID: 26113702 DOI: 10.1126/science.aab2374] [Citation(s) in RCA: 954] [Impact Index Per Article: 106.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/25/2023]
Affiliation(s)
- B A Nosek
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials.
| | - G Alter
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - G C Banks
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - D Borsboom
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - S D Bowman
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - S J Breckler
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - S Buck
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - C D Chambers
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - G Chin
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - G Christensen
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - M Contestabile
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - A Dafoe
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - E Eich
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - J Freese
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - R Glennerster
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - D Goroff
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - D P Green
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - B Hesse
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - M Humphreys
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - J Ishiyama
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - D Karlan
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - A Kraut
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - A Lupia
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - P Mabry
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - T A Madon
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - N Malhotra
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - E Mayo-Wilson
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - M McNutt
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - E Miguel
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - E Levy Paluck
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - U Simonsohn
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - C Soderberg
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - B A Spellman
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - J Turitto
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - G VandenBos
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - S Vazire
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - E J Wagenmakers
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - R Wilson
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| | - T Yarkoni
- Affiliations for the authors, all of whom are members of the TOP Guidelines Committee, are given in the supplementary materials
| |
Collapse
|
6
|
Wagenmakers EJ, Zeelenberg R, Raaijmakers JG. Testing the counter model for perceptual identification: effects of repetition priming and word frequency. Psychon Bull Rev 2000; 7:662-7. [PMID: 11206207 DOI: 10.3758/bf03213004] [Citation(s) in RCA: 29] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
The counter model for perceptual identification (Ratcliff & McKoon, 1997) differs from alternative views of word recognition in two important ways. First, it assumes that prior study of a word does not result in increased sensitivity but, rather, in bias. Second, the effects of word frequency and prior study are explained by different mechanisms. In the present experiment, study status and word frequency of target and foil were varied independently. Using a forced-choice task, we replicated the bias effect. However, we also found several interactions between frequency and prior study that are in direct conflict with the counter model. Most important, prior study of both alternatives resulted in an attenuation of the frequency effect and an increase in performance for low-frequency targets, but not for high-frequency targets. These findings suggest that the effects of frequency and prior study are not mediated by completely independent mechanisms.
Collapse
|
7
|
Wagenmakers EJ, Zeelenberg R, Schooler LJ, Raaijmakers JG. A criterion-shift model for enhanced discriminability in perceptual identification: a note on the counter model. Psychon Bull Rev 2000; 7:718-26. [PMID: 11206215 DOI: 10.3758/bf03213012] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
The original version of the counter model for perceptual identification (Ratcliff & McKoon, 1997) assumed that word frequency and prior study act solely to bias the identification process (i.e., subjects have a tendency to prefer high-frequency and studied low-frequency words, irrespective of the presented word). In a recent study, using a two-alternative forced-choice paradigm, we showed an enhanced discriminability effect for high-frequency and studied low-frequency words (Wagenmakers, Zeelenberg, & Raaijmakers, 2000). These results have led to a fundamental modification of the counter model: Prior study and high frequency not only result in bias, but presumably also result in a higher rate of feature extraction (i.e., better perception). We demonstrate that a criterion-shift model, assuming limited perceptual information extracted from the flash as well as a reduced distance to an identification threshold for high-frequency and studied low-frequency words, can also account for enhanced discriminability.
Collapse
|