|
1
|
Protzko J, Krosnick J, Nelson L, Nosek BA, Axt J, Berent M, Buttrick N, DeBell M, Ebersole CR, Lundmark S, MacInnis B, O'Donnell M, Perfecto H, Pustejovsky JE, Roeder SS, Walleczek J, Schooler JW. High replicability of newly discovered social-behavioural findings is achievable. Nat Hum Behav 2024; 8:311-319. [PMID: 37945809 PMCID: PMC10896719 DOI: 10.1038/s41562-023-01749-9] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2023] [Accepted: 10/05/2023] [Indexed: 11/12/2023]
Abstract
Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigour-enhancing practices: confirmatory tests, large sample sizes, preregistration and methodological transparency. In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing (P < 0.05) in 86% of attempts, slightly exceeding the maximum expected replicability based on observed effect sizes and sample sizes. When one lab attempted to replicate an effect discovered by another lab, the effect size in the replications was 97% that in the original study. This high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries.
Collapse
Affiliation(s)
- John Protzko
- Department of Psychological & Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA.
- Department of Psychological Science, Central Connecticut State University, New Britain, CT, USA.
| | - Jon Krosnick
- Institute for Research in the Social Sciences, Stanford University, Stanford, CA, USA
| | - Leif Nelson
- Haas School of Business, University of California, Berkeley, Berkeley, CA, USA
| | - Brian A Nosek
- Center for Open Science, Charlottesville, VA, USA
- Department of Psychology, University of Virginia, Charlottesville, VA, USA
| | - Jordan Axt
- Department of Psychology, McGill University, Montreal, Quebec, Canada
| | | | - Nicholas Buttrick
- Department of Psychology, University of Wisconsin-Madison, Madison, WI, USA
| | - Matthew DeBell
- Institute for Research in the Social Sciences, Stanford University, Stanford, CA, USA
| | - Charles R Ebersole
- Department of Psychology, University of Virginia, Charlottesville, VA, USA
| | | | - Bo MacInnis
- Institute for Research in the Social Sciences, Stanford University, Stanford, CA, USA
| | - Michael O'Donnell
- McDonough School of Business, Georgetown University, Washington, DC, USA
| | - Hannah Perfecto
- Olin School of Business, Washington University in St. Louis, St. Louis, MO, USA
| | - James E Pustejovsky
- Educational Psychology Department, University of Wisconsin-Madison, Madison, WI, USA
| | - Scott S Roeder
- Darla Moore School of Business, University of South Carolina, Columbia, SC, USA
| | | | - Jonathan W Schooler
- Department of Psychological & Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA
| |
Collapse
|
|
2
|
Gonzalez MF, Liu W, Shirase L, Tomczak DL, Lobbe CE, Justenhoven R, Martin NR. Allying with AI? Reactions toward human-based, AI/ML-based, and augmented hiring processes. COMPUTERS IN HUMAN BEHAVIOR 2022. [DOI: 10.1016/j.chb.2022.107179] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/03/2022]
|
|
3
|
Mülberger A. Early Experimental Psychology: How did Replication Work Before P-Hacking? REVIEW OF GENERAL PSYCHOLOGY 2022. [DOI: 10.1177/10892680211066468] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Affiliation(s)
- Annette Mülberger
- Theory & History of Psychology, University of Groningen Faculty of Behavioural and Social Sciences, Groningen, Netherlands
| |
Collapse
|
|
4
|
Dou D, Shek DTL, Zhu X, Zhao L. Dimensionality of the Chinese CES-D: Is It Stable across Gender, Time, and Samples? INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2021; 18:ijerph182211818. [PMID: 34831573 PMCID: PMC8625664 DOI: 10.3390/ijerph182211818] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 09/21/2021] [Revised: 11/08/2021] [Accepted: 11/09/2021] [Indexed: 01/04/2023]
Abstract
Depression is a common mental illness among Chinese adolescents. Although the Epidemiological Studies Depression Scale (CES-D) has been widely used in diverse populations, the reported factor structures are inconsistent, and its longitudinal invariance is under-researched. This study examined the psychometric properties and factorial invariance across gender and time of the CES-D among Chinese adolescents. Adolescents aged above 11 years from five schools in Chengdu responded to a questionnaire at Wave 1 (n = 5690). Among them, 4981 participants completed the same questionnaire after six months (Wave 2). The matched sample was composed of 4922 students (51.5% were girls; mean age = 13.15 years) at Wave 1. We used exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) to examine the factor structure and performed multi-group CFA to test the factorial invariance across gender and time. A three-factor solution was identified, including "positive affect", "somatic complaints", and "depressed affect". Results of multi-group CFA comparisons supported the factorial invariance of the resultant three-factor solution. Using a new sample of Chinese adolescents in Southwestern China, the present study reproduced earlier findings on adolescents in other areas in China. This study has implications for depression assessment and research in Chinese adolescents.
Collapse
Affiliation(s)
- Diya Dou
- Department of Applied Social Sciences, The Hong Kong Polytechnic University, Hong Kong, China; (D.D.); (X.Z.)
| | - Daniel T. L. Shek
- Department of Applied Social Sciences, The Hong Kong Polytechnic University, Hong Kong, China; (D.D.); (X.Z.)
- Correspondence:
| | - Xiaoqin Zhu
- Department of Applied Social Sciences, The Hong Kong Polytechnic University, Hong Kong, China; (D.D.); (X.Z.)
| | - Li Zhao
- West China School of Public Health/West China Fourth Hospital, Sichuan University, Chengdu 610041, China;
| |
Collapse
|