1
|
Buchberger ES, Ngo CT, Peikert A, Brandmaier AM, Werkle-Bergner M. Estimating statistical power for structural equation models in developmental cognitive science: A tutorial in R : Power simulation for SEMs. Behav Res Methods 2024; 56:1-18. [PMID: 38807007 PMCID: PMC11362481 DOI: 10.3758/s13428-024-02396-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/06/2024] [Indexed: 05/30/2024]
Abstract
Determining the compositional structure and dimensionality of psychological constructs lies at the heart of many research questions in developmental science. Structural equation modeling (SEM) provides a versatile framework for formalizing and estimating the relationships among multiple latent constructs. While the flexibility of SEM can accommodate many complex assumptions on the underlying structure of psychological constructs, it makes a priori estimation of statistical power and required sample size challenging. This difficulty is magnified when comparing non-nested SEMs, which prevents the use of traditional likelihood-ratio tests. Sample size estimates for SEM model fit comparisons typically rely on generic rules of thumb. Such heuristics can be misleading because statistical power in SEM depends on a variety of model properties. Here, we demonstrate a Monte Carlo simulation approach for estimating a priori statistical power for model selection when comparing non-nested models in an SEM framework. We provide a step-by-step guide to this approach based on an example from our memory development research in children.
Collapse
Affiliation(s)
- Elisa S Buchberger
- Center for Lifespan Psychology, Max Planck Institute for Human Development, Lentzeallee 94, 14195, Berlin, Germany.
| | - Chi T Ngo
- Center for Lifespan Psychology, Max Planck Institute for Human Development, Lentzeallee 94, 14195, Berlin, Germany
| | - Aaron Peikert
- Center for Lifespan Psychology, Max Planck Institute for Human Development, Lentzeallee 94, 14195, Berlin, Germany
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, Berlin, Germany
- Department of Imaging Neuroscience, University College London, London, UK
| | - Andreas M Brandmaier
- Center for Lifespan Psychology, Max Planck Institute for Human Development, Lentzeallee 94, 14195, Berlin, Germany
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, Berlin, Germany
- Department of Psychology, MSB Medical School Berlin, Berlin, Germany
| | - Markus Werkle-Bergner
- Center for Lifespan Psychology, Max Planck Institute for Human Development, Lentzeallee 94, 14195, Berlin, Germany
| |
Collapse
|
2
|
Bloxham A, Horton CL. Enhancing and advancing the understanding and study of dreaming and memory consolidation: Reflections, challenges, theoretical clarity, and methodological considerations. Conscious Cogn 2024; 123:103719. [PMID: 38941924 DOI: 10.1016/j.concog.2024.103719] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/06/2023] [Revised: 04/24/2024] [Accepted: 06/12/2024] [Indexed: 06/30/2024]
Abstract
Empirical investigations that search for a link between dreaming and sleep-dependent memory consolidation have focused on testing for an association between dreaming of what was learned, and improved memory performance for learned material. Empirical support for this is mixed, perhaps owing to the inherent challenges presented by the nature of dreams, and methodological inconsistencies. The purpose of this paper is to address critically prevalent assumptions and practices, with the aim of clarifying and enhancing research on this topic, chiefly by providing a theoretical synthesis of existing models and evidence. Also, it recommends the method of Targeted Memory Reactivation (TMR) as a means for investigating if dream content can be linked to specific cued activations. Other recommendations to enhance research practice and enquiry on this subject are also provided, focusing on the HOW and WHY we search for memory sources in dreams, and what purpose (if any) they might serve.
Collapse
Affiliation(s)
- Anthony Bloxham
- Nottingham Trent University, Nottingham, NG1 4FQ, United Kingdom.
| | | |
Collapse
|
3
|
Preobrazenski N, McCaig A, Turner A, Kushner M, Pacitti L, Mendolia P, MacDonald B, Storoschuk K, Bouck T, Zaza Y, Lu S, Gurd BJ. Risk of bias in exercise science: A systematic review of 340 studies. iScience 2024; 27:109010. [PMID: 38405604 PMCID: PMC10884506 DOI: 10.1016/j.isci.2024.109010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2023] [Revised: 12/19/2023] [Accepted: 01/22/2024] [Indexed: 02/27/2024] Open
Abstract
Risk of bias can contribute to irreproducible science and mislead decision making. Analyses of smaller subsections of the exercise science literature suggest many exercise science studies have unclear or high risk of bias. The current review (osf.io/jznv8) assesses whether this unclear or high risk of bias is more widespread in the exercise science literature and whether this bias has decreased since the publication of the 1996 Consolidated Standards of Reporting Trials (CONSORT) guidelines. We report significant reductions in selection, performance, detection, and reporting biases in 2020 compared with 1995 in the 340 of 5,451 studies assessed using the Cochrane Risk of Bias tool. Despite these improvements, most 2020 studies still had unclear or high risks of bias. These results underscore the need for methodological vigilance, adherence to reporting standards, and education on experimental bias. Factors contributing to these improvements, such advancements in education and journal requirements, remain uncertain.
Collapse
Affiliation(s)
| | - Abby McCaig
- School of Kinesiology and Health Studies, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Anna Turner
- School of Kinesiology and Health Studies, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Maddy Kushner
- School of Kinesiology and Health Studies, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Lauren Pacitti
- School of Kinesiology and Health Studies, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Peter Mendolia
- School of Kinesiology and Health Studies, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Ben MacDonald
- School of Kinesiology and Health Studies, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Kristi Storoschuk
- School of Kinesiology and Health Studies, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Tori Bouck
- School of Kinesiology and Health Studies, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Youssef Zaza
- School of Kinesiology and Health Studies, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Stephanie Lu
- School of Kinesiology and Health Studies, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Brendon J. Gurd
- School of Kinesiology and Health Studies, Queen’s University, Kingston, ON K7L 3N6, Canada
| |
Collapse
|
4
|
Harkin B, Davies LE, Yates A. Contamination-Focussed Vignettes as an Analogue of Infectious Pandemics: An Experimental Validation using the State Disgust and Anxiety Responses in OCD. Psychol Rep 2024:332941241238208. [PMID: 38462961 DOI: 10.1177/00332941241238208] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/12/2024]
Abstract
Despite infectious pandemics proving particularly detrimental to those with Obsessive-Compulsive Disorder (OCD), the investigation of analogous experimental paradigms is lacking. To address this gap, we conducted two studies employing vignettes that depicted contamination-related situations commonly experienced during a pandemic (e.g., Coughing into hands and failing to use hand sanitizer). We manipulated the salience of these vignettes across three levels: high contamination, low contamination, and a neutral control condition. Our examination of state anxiety and disgust responses in all participants revealed the successful manipulation of the vignettes' impact. Specifically, individuals with more severe OCD symptoms reported significantly higher levels of state disgust and anxiety for both high and low contamination vignettes, in contrast to the group with lower symptom severity. No significant differences were observed in the neutral vignette condition between the high- and low-scoring groups. Interestingly, for those with higher OCD symptoms, high salience contamination-focused vignettes resulted in similarly elevated state disgust and anxiety, regardless of whether the vignettes were situated in public (Study 1) or domestic (Study 2) settings. This suggests that the heightened sensitivity to contamination-related scenarios observed in individuals with OCD symptoms in the present study is not confined to a specific context. These findings support the use of contamination-focused vignettes as analogues for studying infectious pandemics and provide valuable insights into OCD models, interventions, and future research.
Collapse
Affiliation(s)
- Ben Harkin
- Department of Psychology, Manchester Metropolitan University, Manchester, UK
| | - Lucy E Davies
- Department of Psychology, Manchester Metropolitan University, Manchester, UK
| | - Alan Yates
- Department of Psychology, Manchester Metropolitan University, Manchester, UK
| |
Collapse
|
5
|
Li W, Germine LT, Mehr SA, Srinivasan M, Hartshorne J. Developmental psychologists should adopt citizen science to improve generalization and reproducibility. INFANT AND CHILD DEVELOPMENT 2024; 33:e2348. [PMID: 38515737 PMCID: PMC10957098 DOI: 10.1002/icd.2348] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2021] [Accepted: 05/17/2022] [Indexed: 11/08/2022]
Abstract
Widespread failures of replication and generalization are, ironically, a scientific triumph, in that they confirm the fundamental metascientific theory that underlies our field. Generalizable and replicable findings require testing large numbers of subjects from a wide range of demographics with a large, randomly-sampled stimulus set, and using a variety of experimental parameters. Because few studies accomplish any of this, meta-scientists predict that findings will frequently fail to replicate or generalize. We argue that to be more robust and replicable, developmental psychology needs to find a mechanism for collecting data at greater scale and from more diverse populations. Luckily, this mechanism already exists: Citizen science, in which large numbers of uncompensated volunteers provide data. While best-known for its contributions to astronomy and ecology, citizen science has also produced major findings in neuroscience and psychology, and increasingly in developmental psychology. We provide examples, address practical challenges, discuss limitations, and compare to other methods of obtaining large datasets. Ultimately, we argue that the range of studies where it makes sense *not* to use citizen science is steadily dwindling.
Collapse
Affiliation(s)
- Wei Li
- Department of Psychology and Neuroscience, Boston College, Chestnut Hill, MA, USA
| | - Laura Thi Germine
- McLean Hospital, Belmont, MA, USA
- Department of Psychiatry, Harvard Medical School, Cambridge, MA
| | - Samuel A. Mehr
- Data Science Initiative, Harvard University, Cambridge, MA
- School of Psychology, Victoria University of Wellington, Wellington, New Zealand
| | | | - Joshua Hartshorne
- Department of Psychology and Neuroscience, Boston College, Chestnut Hill, MA, USA
| |
Collapse
|
6
|
Delfin C. Improving the stability of bivariate correlations using informative Bayesian priors: a Monte Carlo simulation study. Front Psychol 2023; 14:1253452. [PMID: 37744589 PMCID: PMC10517051 DOI: 10.3389/fpsyg.2023.1253452] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2023] [Accepted: 08/24/2023] [Indexed: 09/26/2023] Open
Abstract
Objective Much of psychological research has suffered from small sample sizes and low statistical power, resulting in unstable parameter estimates. The Bayesian approach offers a promising solution by incorporating prior knowledge into statistical models, which may lead to improved stability compared to a frequentist approach. Methods Simulated data from four populations with known bivariate correlations (ρ = 0.1, 0.2, 0.3, 0.4) was used to estimate the sample correlation as samples were sequentially added from the population, from n = 10 to n = 500. The impact of three different, subjectively defined prior distributions (weakly, moderately, and highly informative) was investigated and compared to a frequentist model. Results The results show that bivariate correlation estimates are unstable, and that the risk of obtaining an estimate that is exaggerated or in the wrong direction is relatively high, for sample sizes for below 100, and considerably so for sample sizes below 50. However, this instability can be constrained by informative Bayesian priors. Conclusion Informative Bayesian priors have the potential to significantly reduce sample size requirements and help ensure that obtained estimates are in line with realistic expectations. The combined stabilizing and regularizing effect of a weakly informative prior is particularly useful when conducting research with small samples. The impact of more informative Bayesian priors depends on one's threshold for probability and whether one's goal is to obtain an estimate merely in the correct direction, or to obtain a high precision estimate whose associated interval falls within a narrow range. Implications for sample size requirements and directions for future research are discussed.
Collapse
Affiliation(s)
- Carl Delfin
- Lund Clinical Research on Externalizing and Developmental Psychopathology (LU-CRED), Child and Adolescent Psychiatry, Department of Clinical Sciences Lund, Lund University, Lund, Sweden
- Centre for Ethics, Law and Mental Health (CELAM), Department of Psychiatry and Neurochemistry, Institute of Neuroscience and Physiology, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
| |
Collapse
|
7
|
Tseng H, Damian MF. Exploring synchrony effects in performance on tasks involving cognitive inhibition: An online study of young adults. Chronobiol Int 2023; 40:1209-1223. [PMID: 37700626 DOI: 10.1080/07420528.2023.2256843] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 06/26/2023] [Accepted: 09/02/2023] [Indexed: 09/14/2023]
Abstract
Inhibition is one of the core components of cognitive control. In experimental tasks which measure cognitive inhibition, performance may vary according to an interplay of individuals' chronotype and the time of day of testing ("synchrony effect", or the beneficial impact on cognitive performance of aligning testing with the time of day preferred by an individual's chronotype). Some prior studies have reported a synchrony effect specifically emerging in activities which require cognitive inhibition, but not in general processing speed, but existing findings are inconsistent. If genuine, synchrony effects should be taken into account when comparing groups of participants. Here we explored whether synchrony effects emerge in a sample of young adults. In a multi-part online study, we captured various components of inhibition (response suppression; inhibitory control; switching) plus a general measure of processing speed across various times of the day. Individuals' chronotype was included as a predictor of performance. Critically, we found no evidence of a synchrony effect (an association between chronotype and component of interest where the directionality is dependent on time of testing) in our study.
Collapse
Affiliation(s)
- Hayley Tseng
- School of Psychological Science, University of Bristol, Bristol, UK
| | - Markus F Damian
- School of Psychological Science, University of Bristol, Bristol, UK
| |
Collapse
|
8
|
Vankov II. The hazards of dealing with response time outliers. Front Psychol 2023; 14:1220281. [PMID: 37691812 PMCID: PMC10484222 DOI: 10.3389/fpsyg.2023.1220281] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2023] [Accepted: 08/07/2023] [Indexed: 09/12/2023] Open
Abstract
The presence of outliers in response times can affect statistical analyses and lead to incorrect interpretation of the outcome of a study. Therefore, it is a widely accepted practice to try to minimize the effect of outliers by preprocessing the raw data. There exist numerous methods for handling outliers and researchers are free to choose among them. In this article, we use computer simulations to show that serious problems arise from this flexibility. Choosing between alternative ways for handling outliers can result in the inflation of p-values and the distortion of confidence intervals and measures of effect size. Using Bayesian parameter estimation and probability distributions with heavier tails eliminates the need to deal with response times outliers, but at the expense of opening another source of flexibility.
Collapse
Affiliation(s)
- Ivan I. Vankov
- Institute of Neurobiology, Bulgarian Academy of Sciences, Sofia, Sofia City, Bulgaria
| |
Collapse
|
9
|
Langenberg B, Janczyk M, Koob V, Kliegl R, Mayer A. A tutorial on using the paired t test for power calculations in repeated measures ANOVA with interactions. Behav Res Methods 2023; 55:2467-2484. [PMID: 36002625 PMCID: PMC10439102 DOI: 10.3758/s13428-022-01902-8] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/03/2022] [Indexed: 11/08/2022]
Abstract
The a priori calculation of statistical power has become common practice in behavioral and social sciences to calculate the necessary sample size for detecting an expected effect size with a certain probability (i.e., power). In multi-factorial repeated measures ANOVA, these calculations can sometimes be cumbersome, especially for higher-order interactions. For designs that only involve factors with two levels each, the paired t test can be used for power calculations, but some pitfalls need to be avoided. In this tutorial, we provide practical advice on how to express main and interaction effects in repeated measures ANOVA as single difference variables. In particular, we demonstrate how to calculate the effect size Cohen's d of this difference variable either based on means, variances, and covariances of conditions or by transforming [Formula: see text] or [Formula: see text] from the ANOVA framework into d. With the effect size correctly specified, we then show how to use the t test for sample size considerations by means of an empirical example. The relevant R code is provided in an online repository for all example calculations covered in this article.
Collapse
Affiliation(s)
| | | | | | | | - Axel Mayer
- Bielefeld University, Bielefeld, Germany
| |
Collapse
|
10
|
Schauer JM. On the Accuracy of Replication Failure Rates. MULTIVARIATE BEHAVIORAL RESEARCH 2023; 58:598-615. [PMID: 37339430 DOI: 10.1080/00273171.2022.2066500] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/22/2023]
Abstract
A prominent approach to studying the replication crisis has been to conduct replications of several different scientific findings as part of the same research effort. The reported proportion of findings that these programs determined failed to replicate have become important statistics in the replication crisis. However, these "failure rates" are based on decisions about whether individual studies replicated, which are themselves subject to statistical uncertainty. In this article, we examine how that uncertainty impacts the accuracy of reported failure rates and find that the reported failure rates can be substantially biased and highly variable. Indeed, very high or very low failure rates could arise from chance alone.
Collapse
|
11
|
Ellis RJ. Questionable Research Practices, Low Statistical Power, and Other Obstacles to Replicability: Why Preclinical Neuroscience Research Would Benefit from Registered Reports. eNeuro 2022; 9:ENEURO.0017-22.2022. [PMID: 35922130 PMCID: PMC9351632 DOI: 10.1523/eneuro.0017-22.2022] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2022] [Revised: 05/22/2022] [Accepted: 05/31/2022] [Indexed: 02/03/2023] Open
Abstract
Replicability, the degree to which a previous scientific finding can be repeated in a distinct set of data, has been considered an integral component of institutionalized scientific practice since its inception several hundred years ago. In the past decade, large-scale replication studies have demonstrated that replicability is far from favorable, across multiple scientific fields. Here, I evaluate this literature and describe contributing factors including the prevalence of questionable research practices (QRPs), misunderstanding of p-values, and low statistical power. I subsequently discuss how these issues manifest specifically in preclinical neuroscience research. I conclude that these problems are multifaceted and difficult to solve, relying on the actions of early and late career researchers, funding sources, academic publishers, and others. I assert that any viable solution to the problem of substandard replicability must include changing academic incentives, with adoption of registered reports being the most immediately impactful and pragmatic strategy. For animal research in particular, comprehensive reporting guidelines that document potential sources of sensitivity for experimental outcomes is an essential addition.
Collapse
Affiliation(s)
- Randall J Ellis
- Friedman Brain Institute, Department of Neuroscience, Addiction Institute of Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY 10029
| |
Collapse
|
12
|
How Cross-Discipline Understanding and Communication Can Improve Research on Multiracial Populations. SOCIAL SCIENCES-BASEL 2022. [DOI: 10.3390/socsci11030090] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
Abstract
One of the strengths of Critical Mixed Race Studies is that it represents research methodologies and frameworks from multiple disciplines across the social sciences and humanities. However, if these disciplines are not in dialogue with each other, that benefit may be lost. Here, we use psychological and sociological research on Multiracial populations as examples to argue how strict disciplinarity and methodological trends may limit scientific production. We propose that reading and citing work across disciplines, expanding methodological training, and rejecting hegemonic “white logic” assumptions about what is “publishable” can enhance Multiracial research. First, the ability to cite effectively across disciplines will shorten the time it takes for new theories to be developed that focus on empirically underrepresented populations. Secondly, increasing understanding of both quantitative and qualitative methods will allow more effective reading between disciplines while also creating opportunities to engage with both causality and the richness of experiences that comprise being Multiracial. Finally, these changes would then situate scholars to be more effective reviewers, thereby enhancing the peer-reviewed publication process to one that routinely rejects color evasive racist practices that privilege work on majority populations.
Collapse
|
13
|
Stanley TD, Doucouliagos H, Ioannidis JPA. Retrospective median power, false positive meta-analysis and large-scale replication. Res Synth Methods 2021; 13:88-108. [PMID: 34628722 DOI: 10.1002/jrsm.1529] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2020] [Revised: 08/25/2021] [Accepted: 09/23/2021] [Indexed: 11/08/2022]
Abstract
Recent, high-profile, large-scale, preregistered failures to replicate uncover that many highly-regarded experiments are "false positives"; that is, statistically significant results of underlying null effects. Large surveys of research reveal that statistical power is often low and inadequate. When the research record includes selective reporting, publication bias and/or questionable research practices, conventional meta-analyses are also likely to be falsely positive. At the core of research credibility lies the relation of statistical power to the rate of false positives. This study finds that high (>50%-60%) median retrospective power (MRP) is associated with credible meta-analysis and large-scale, preregistered, multi-lab "successful" replications; that is, with replications that corroborate the effect in question. When median retrospective power is low (<50%), positive meta-analysis findings should be interpreted with great caution or discounted altogether.
Collapse
Affiliation(s)
- T D Stanley
- Deakin Lab for the Meta-Analysis of Research (DeLMAR), School of Business and Law, Deakin University, Burwood, Victoria, Australia
| | | | - John P A Ioannidis
- Department of Medicine, METRICS, Stanford University, Stanford, California, USA.,METRIC B, Berlin Institute of Health, Berlin, Germany
| |
Collapse
|
14
|
Haucke M, Hoekstra R, van Ravenzwaaij D. When numbers fail: do researchers agree on operationalization of published research? ROYAL SOCIETY OPEN SCIENCE 2021; 8:191354. [PMID: 34527263 PMCID: PMC8424321 DOI: 10.1098/rsos.191354] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/05/2019] [Accepted: 08/04/2021] [Indexed: 06/13/2023]
Abstract
Current discussions on improving the reproducibility of science often revolve around statistical innovations. However, equally important for improving methodological rigour is a valid operationalization of phenomena. Operationalization is the process of translating theoretical constructs into measurable laboratory quantities. Thus, the validity of operationalization is central for the quality of empirical studies. But do differences in the validity of operationalization affect the way scientists evaluate scientific literature? To investigate this, we manipulated the strength of operationalization of three published studies and sent them to researchers via email. In the first task, researchers were presented with a summary of the Method and Result section from one of the studies and were asked to guess the hypothesis that was investigated via a multiple-choice questionnaire. In a second task, researchers were asked to rate the perceived quality of the study. Our results show that (1) researchers are better at inferring the underlying research question from empirical results if the operationalization is more valid, but (2) the different validity is only to some extent reflected in a judgement of the study's quality. These results combined give partial corroboration to the notion that researchers' evaluations of research results are not affected by operationalization validity.
Collapse
Affiliation(s)
- Matthias Haucke
- Clinical Psychology and Psychotherapy, Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
- Department of Psychometrics, University of Groningen, Groningen, The Netherlands
| | - Rink Hoekstra
- Department of Pedagogical and Educational Sciences, University of Groningen, Groningen, The Netherlands
| | - Don van Ravenzwaaij
- Department of Psychometrics, University of Groningen, Groningen, The Netherlands
| |
Collapse
|
15
|
Stanley TD, Doucouliagos H, Ioannidis JPA, Carter EC. Detecting publication selection bias through excess statistical significance. Res Synth Methods 2021; 12:776-795. [PMID: 34196473 DOI: 10.1002/jrsm.1512] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2021] [Revised: 05/08/2021] [Accepted: 06/09/2021] [Indexed: 11/11/2022]
Abstract
We introduce and evaluate three tests for publication selection bias based on excess statistical significance (ESS). The proposed tests incorporate heterogeneity explicitly in the formulas for expected and ESS. We calculate the expected proportion of statistically significant findings in the absence of selective reporting or publication bias based on each study's SE and meta-analysis estimates of the mean and variance of the true-effect distribution. A simple proportion of statistical significance test (PSST) compares the expected to the observed proportion of statistically significant findings. Alternatively, we propose a direct test of excess statistical significance (TESS). We also combine these two tests of excess statistical significance (TESSPSST). Simulations show that these ESS tests often outperform the conventional Egger test for publication selection bias and the three-parameter selection model (3PSM).
Collapse
Affiliation(s)
- T D Stanley
- School of Business and Law, Deakin University, Burwood, Victoria, Australia
| | | | - John P A Ioannidis
- Epidemiology and Population Health, and (by courtesy) of Biomedical Data Science, and of Statistics, and Co-Director, METRICS, Stanford University, Stanford, California, USA.,METRIC B, Berlin Institute of Health, Berlin, Germany
| | - Evan C Carter
- Human Research and Engineering Directorate, United States Army Research Laboratory, Aberdeen Proving Ground, Aberdeen, Maryland, USA
| |
Collapse
|
16
|
Schauer JM, Fitzgerald KG, Peko-Spicer S, Whalen MCR, Zejnullahi R, Hedges LV. An evaluation of statistical methods for aggregate patterns of replication failure. Ann Appl Stat 2021. [DOI: 10.1214/20-aoas1387] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
17
|
Smalheiser NR, Graetz EE, Yu Z, Wang J. Effect size, sample size and power of forced swim test assays in mice: Guidelines for investigators to optimize reproducibility. PLoS One 2021; 16:e0243668. [PMID: 33626103 PMCID: PMC7904226 DOI: 10.1371/journal.pone.0243668] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2020] [Accepted: 02/10/2021] [Indexed: 11/19/2022] Open
Abstract
A recent flood of publications has documented serious problems in scientific reproducibility, power, and reporting of biomedical articles, yet scientists persist in their usual practices. Why? We examined a popular and important preclinical assay, the Forced Swim Test (FST) in mice used to test putative antidepressants. Whether the mice were assayed in a naïve state vs. in a model of depression or stress, and whether the mice were given test agents vs. known antidepressants regarded as positive controls, the mean effect sizes seen in the experiments were indeed extremely large (1.5-2.5 in Cohen's d units); most of the experiments utilized 7-10 animals per group which did have adequate power to reliably detect effects of this magnitude. We propose that this may at least partially explain why investigators using the FST do not perceive intuitively that their experimental designs fall short-even though proper prospective design would require ~21-26 animals per group to detect, at a minimum, large effects (0.8 in Cohen's d units) when the true effect of a test agent is unknown. Our data provide explicit parameters and guidance for investigators seeking to carry out prospective power estimation for the FST. More generally, altering the real-life behavior of scientists in planning their experiments may require developing educational tools that allow them to actively visualize the inter-relationships among effect size, sample size, statistical power, and replicability in a direct and intuitive manner.
Collapse
Affiliation(s)
- Neil R. Smalheiser
- Department of Psychiatry, University of Illinois School of Medicine, Chicago, Illinois, United States of America
| | - Elena E. Graetz
- Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago, Chicago, Illinois, United States of America
| | - Zhou Yu
- Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago, Chicago, Illinois, United States of America
| | - Jing Wang
- Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago, Chicago, Illinois, United States of America
| |
Collapse
|
18
|
Neuroanatomical correlates of extraversion: a test-retest study implicating gray matter volume in the caudate nucleus. Neuroreport 2019; 30:953-959. [PMID: 31469723 DOI: 10.1097/wnr.0000000000001306] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/09/2023]
Abstract
Extraversion is a propensity for optimism, positive emotion, and sociability. Many studies have explored the brain correlates of extraversion, and the results have mainly emphasized the role of reward-related brain regions. However, it is unclear whether imaging studies of the brain's reward system can show good test-retest reliability. We aim to investigate the neuroanatomical correlates of extraversion and to examine the test-retest reliability of the results. Voxel-based morphometry derived from MRI and the Revised NEO Personality Inventory were performed in a sample of 382 subjects (dataset 1), and multiple regression was used to analyze the relation between regional gray matter volume (GMV) and extraversion scores. We found that bilateral GMV in the caudate region was positively associated with extraversion. Two years later, 133 subjects from dataset 1 were re-examined with MRI and the Revised NEO Personality Inventory. A conjunction analysis showed that the body of left caudate nucleus was consistently associated with extraversion. The bilateral GMV of the caudate nucleus, which may be related to sensitivity to rewards, may be a critical brain structure underlying extraverted behavior. This study is the first to use two different time points to assess the consistency of the association of brain structures with extraversion.
Collapse
|
19
|
Vasilev MR, Yates M, Slattery TJ. Do Readers Integrate Phonological Codes Across Saccades? A Bayesian Meta-Analysis and a Survey of the Unpublished Literature. J Cogn 2019; 2:43. [PMID: 31750415 PMCID: PMC6838770 DOI: 10.5334/joc.87] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2019] [Accepted: 10/08/2019] [Indexed: 11/28/2022] Open
Abstract
It is commonly accepted that phonological codes can be activated parafoveally during reading and later used to aid foveal word recognition- a finding known as the phonological preview benefit. However, a closer look at the literature shows that this effect may be less consistent than what is sometimes believed. To determine the extent to which phonology is processed parafoveally, a Bayesian meta-analysis of 27 experiments and a survey of the unpublished literature were conducted. While the results were generally consistent with the phonological preview benefit (>90% probability of a true effect in gaze durations), the size of the effect was small. Readers of alphabetical orthographies obtained a modest benefit of only 4 ms in gaze durations. Interestingly, Chinese readers showed a larger effect (6-14 ms in size). There was no difference in the magnitude of the phonological preview benefit between homophone and pseudo-homophone previews, thus suggesting that the modest processing advantage is indeed related to the activation of phonological codes from the parafoveal word. Simulations revealed that the results are relatively robust to missing studies, although the effects may be 19-22% smaller if all missing studies found a null effect. The results suggest that while phonology can be processed parafoveally, this happens only to a limited extent. Because phonological priming effects in single-word recognition are small (10-13 ms; Rastle & Brysbaert, 2006) and there is a loss of visual acuity in the parafovea, it is argued that large phonological preview benefit effects may be unlikely.
Collapse
Affiliation(s)
| | - Mark Yates
- University of South Alabama, Department of Psychology, US
| | | |
Collapse
|
20
|
Brysbaert M. How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables. J Cogn 2019; 2:16. [PMID: 31517234 PMCID: PMC6640316 DOI: 10.5334/joc.72] [Citation(s) in RCA: 287] [Impact Index Per Article: 57.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2018] [Accepted: 05/22/2019] [Indexed: 12/24/2022] Open
Abstract
Given that an effect size of d = .4 is a good first estimate of the smallest effect size of interest in psychological research, we already need over 50 participants for a simple comparison of two within-participants conditions if we want to run a study with 80% power. This is more than current practice. In addition, as soon as a between-groups variable or an interaction is involved, numbers of 100, 200, and even more participants are needed. As long as we do not accept these facts, we will keep on running underpowered studies with unclear results. Addressing the issue requires a change in the way research is evaluated by supervisors, examiners, reviewers, and editors. The present paper describes reference numbers needed for the designs most often used by psychologists, including single-variable between-groups and repeated-measures designs with two and three levels, two-factor designs involving two repeated-measures variables or one between-groups variable and one repeated-measures variable (split-plot design). The numbers are given for the traditional, frequentist analysis with p < .05 and Bayesian analysis with BF > 10. These numbers provide researchers with a standard to determine (and justify) the sample size of an upcoming study. The article also describes how researchers can improve the power of their study by including multiple observations per condition per participant.
Collapse
Affiliation(s)
- Marc Brysbaert
- Department of Experimental Psychology, Ghent University, BE
| |
Collapse
|
21
|
Should Psychology Journals Adopt Specialized Statistical Review? ADVANCES IN METHODS AND PRACTICES IN PSYCHOLOGICAL SCIENCE 2019. [DOI: 10.1177/2515245919858428] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Readers of peer-reviewed research may assume that the reported statistical analyses supporting scientific claims have been closely scrutinized and surpass a high-quality threshold. However, widespread misunderstanding and misuse of statistical concepts and methods suggests that suboptimal or erroneous statistical practice is routinely overlooked during peer review in psychology. Here, we explore whether psychology journals could ameliorate some of the field’s statistical ailments by adopting specialized statistical review: a focused technical assessment, performed by statistical experts, that addresses the analysis and presentation of quantitative information and supplements regular peer review. We discuss evidence from a recent survey of journal editors suggesting that specialized statistical review may be unusual in psychology journals and is regarded by many editors as unnecessary. We contrast these views with those in the biomedical domain, where statistical review has been considered a partial preventive measure against the improper use of statistics since the late 1970s. We suggest that the current “credibility revolution” presents an opportune occasion for psychology journals to consider adopting specialized statistical review.
Collapse
|
22
|
Calin-Jageman RJ, Cumming G. Estimation for Better Inference in Neuroscience. eNeuro 2019; 6:ENEURO.0205-19.2019. [PMID: 31453316 PMCID: PMC6709209 DOI: 10.1523/eneuro.0205-19.2019] [Citation(s) in RCA: 82] [Impact Index Per Article: 16.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2019] [Revised: 06/03/2019] [Accepted: 06/08/2019] [Indexed: 12/31/2022] Open
Abstract
The estimation approach to inference emphasizes reporting effect sizes with expressions of uncertainty (interval estimates). In this perspective we explain the estimation approach and describe how it can help nudge neuroscientists toward a more productive research cycle by fostering better planning, more thoughtful interpretation, and more balanced evaluation of evidence.
Collapse
Affiliation(s)
| | - Geoff Cumming
- La Trobe University, Melbourne, Victoria 3086, Australia
| |
Collapse
|
23
|
Affiliation(s)
- Paul E Jenkins
- School of Psychology and Clinical Language Sciences,University of Reading,RG6 6AL,UK
| |
Collapse
|
24
|
Hartshorne JK, Skorb L, Dietz SL, Garcia CR, Iozzo GL, Lamirato KE, Ledoux JR, Mu J, Murdock KN, Ravid J, Savery AA, Spizzirro JE, Trimm KA, van Horne KD, Vidal J. The Meta-Science of Adult Statistical Word Segmentation: Part 1. COLLABRA: PSYCHOLOGY 2019. [DOI: 10.1525/collabra.181] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022] Open
Abstract
We report the first set of results in a multi-year project to assess the robustness – and the factors promoting robustness – of the adult statistical word segmentation literature. This includes eight total experiments replicating six different experiments. The purpose of these replications is to assess the reproducibility of reported experiments, examine the replicability of their results, and provide more accurate effect size estimates. Reproducibility was mixed, with several papers either lacking crucial details or containing errors in the description of method, making it difficult to ascertain what was done. Replicability was also mixed: although in every instance we confirmed above-chance statistical word segmentation, many theoretically important moderations of that learning failed to replicate. Moreover, learning success was generally much lower than in the original studies. In the General Discussion, we consider whether these differences are due to differences in subject populations, low power in the original studies, or some combination of these and other factors. We also consider whether these findings are likely to generalize to the broader statistical word segmentation literature.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | - Jesse Mu
- Department of Psychology, Boston College, US
| | | | - Jon Ravid
- Department of Psychology, Boston College, US
| | | | | | | | | | | |
Collapse
|
25
|
Giner-Sorolla R. From crisis of evidence to a “crisis” of relevance? Incentive-based answers for social psychology’s perennial relevance worries. EUROPEAN REVIEW OF SOCIAL PSYCHOLOGY 2018. [DOI: 10.1080/10463283.2018.1542902] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/27/2022]
|
26
|
Trafimow D, Amrhein V, Areshenkoff CN, Barrera-Causil CJ, Beh EJ, Bilgiç YK, Bono R, Bradley MT, Briggs WM, Cepeda-Freyre HA, Chaigneau SE, Ciocca DR, Correa JC, Cousineau D, de Boer MR, Dhar SS, Dolgov I, Gómez-Benito J, Grendar M, Grice JW, Guerrero-Gimenez ME, Gutiérrez A, Huedo-Medina TB, Jaffe K, Janyan A, Karimnezhad A, Korner-Nievergelt F, Kosugi K, Lachmair M, Ledesma RD, Limongi R, Liuzza MT, Lombardo R, Marks MJ, Meinlschmidt G, Nalborczyk L, Nguyen HT, Ospina R, Perezgonzalez JD, Pfister R, Rahona JJ, Rodríguez-Medina DA, Romão X, Ruiz-Fernández S, Suarez I, Tegethoff M, Tejo M, van de Schoot R, Vankov II, Velasco-Forero S, Wang T, Yamada Y, Zoppino FCM, Marmolejo-Ramos F. Manipulating the Alpha Level Cannot Cure Significance Testing. Front Psychol 2018; 9:699. [PMID: 29867666 PMCID: PMC5962803 DOI: 10.3389/fpsyg.2018.00699] [Citation(s) in RCA: 51] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2018] [Accepted: 04/23/2018] [Indexed: 11/30/2022] Open
Abstract
We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p = 0.05 to p = 0.005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of 0.05, 0.01, 0.005, or anything else, is not acceptable.
Collapse
Affiliation(s)
- David Trafimow
- Department of Psychology, New Mexico State University, Las Cruces, NM, United States
| | - Valentin Amrhein
- Zoological Institute, University of Basel, Basel, Switzerland.,Swiss Ornithological Institute, Sempach, Switzerland
| | | | - Carlos J Barrera-Causil
- Faculty of Applied and Exact Sciences, Metropolitan Technological Institute, Medellín, Colombia
| | - Eric J Beh
- School of Mathematical and Physical Sciences, University of Newcastle, Callaghan, NSW, Australia
| | - Yusuf K Bilgiç
- Department of Mathematics, State University of New York at Geneseo, Geneseo, NY, United States
| | - Roser Bono
- Quantitative Psychology Unit, Faculty of Psychology, University of Barcelona, Barcelona, Spain.,Institut de Neurociències, University of Barcelona, Barcelona, Spain
| | - Michael T Bradley
- Department of Psychology, Faculty of Arts, University of New Brunswick, Saint John, NB, Canada
| | | | | | - Sergio E Chaigneau
- Center for Social and Cognitive Neuroscience, School of Psychology, Universidad Adolfo Ibáñez, Santiago, Chile
| | - Daniel R Ciocca
- Oncology Laboratory, Instituto de Medicina y Biologia Experimental de Cuyo, CCT CONICET Mendoza, Mendoza, Argentina
| | - Juan C Correa
- School of Statistics, Faculty of Sciences, National University of Colombia, Medellín, Colombia
| | - Denis Cousineau
- School of Psychology, University of Ottawa, Ottawa, ON, Canada
| | - Michiel R de Boer
- Department of Health Sciences, Vrije Universiteit Amsterdam and Amsterdam Public Health Research Institute, Amsterdam, Netherlands
| | - Subhra S Dhar
- Department of Mathematics and Statistics, Indian Institute of Technology, Kanpur, India
| | - Igor Dolgov
- Department of Psychology, New Mexico State University, Las Cruces, NM, United States
| | - Juana Gómez-Benito
- Quantitative Psychology Unit, Faculty of Psychology, University of Barcelona, Barcelona, Spain.,Institut de Neurociències, University of Barcelona, Barcelona, Spain
| | - Marian Grendar
- Biomedical Center Martin, Jessenius Faculty of Medicine, Comenius University, Martin, Slovakia.,Institute of Measurement Science, Slovak Academy of Sciences, Bratislava, Slovakia
| | - James W Grice
- Department of Psychology, Oklahoma State University, Stillwater, OK, United States
| | - Martin E Guerrero-Gimenez
- Oncology Laboratory, Instituto de Medicina y Biologia Experimental de Cuyo, CCT CONICET Mendoza, Mendoza, Argentina
| | - Andrés Gutiérrez
- Faculty of Statistics, Saint Thomas University, Bogotá, Colombia
| | - Tania B Huedo-Medina
- Department of Allied Health Sciences, College of Health, Agriculture, and Natural Resources, University of Connecticut, Storrs, CT, United States
| | - Klaus Jaffe
- Departamento de Biología de Organismos, Universidad Simón Bolívar, Caracas, Venezuela
| | - Armina Janyan
- Department of Cognitive Science and Psychology, New Bulgarian University, Sofia, Bulgaria.,National Research Tomsk State University, Tomsk, Russia
| | - Ali Karimnezhad
- Department of Biochemistry, Microbiology, and Immunology, University of Ottawa, Ottawa, ON, Canada
| | | | - Koji Kosugi
- School of Human Sciences, Senshu University, Kawasaki, Japan
| | - Martin Lachmair
- Multimodal Interaction Lab, Leibniz-Institut für Wissensmedien, Tübingen, Germany
| | - Rubén D Ledesma
- Consejo Nacional de Investigaciones Científicas y Técnicas, Mar del Plata, Argentina.,Facultad de Psicología, Universidad Nacional de Mar del Plata, Mar del Plata, Argentina
| | - Roberto Limongi
- Pontificia Universidad Católica de Valparaíso, Valparaíso, Chile.,Vicerrectoría de Investigación y Desarrollo, Universidad Tecnológica de Chile INACAP, Santiago, Chile
| | - Marco T Liuzza
- Department of Medical and Surgical Sciences, "Magna Graecia" University of Catanzaro, Catanzaro, Italy
| | - Rosaria Lombardo
- Economics Department, University of Campania "Luigi Vanvitelli", Capua, Italy
| | - Michael J Marks
- Department of Psychology, New Mexico State University, Las Cruces, NM, United States
| | - Gunther Meinlschmidt
- Department of Psychosomatic Medicine, University Hospital Basel and University of Basel, Basel, Switzerland.,Division of Clinical Psychology and Cognitive Behavioral Therapy, International Psychoanalytic University, Berlin, Germany.,Division of Clinical Psychology and Epidemiology, Department of Psychology, University of Basel, Basel, Switzerland
| | - Ladislas Nalborczyk
- Université Grenoble Alpes, Centre National de la Recherche Scientifique, LPNC, Grenoble, France.,Department of Experimental Clinical and Health Psychology, Ghent University, Ghent, Belgium
| | - Hung T Nguyen
- Department of Mathematical Sciences, New Mexico State University, Las Cruces, NM, United States
| | - Raydonal Ospina
- Computational Statistics Laboratory (CAST), Department of Statistics, Universidade Federal de Pernambuco, Recife, Brazil
| | | | - Roland Pfister
- Department of Psychology III, University of Würzburg, Würzburg, Germany
| | - Juan J Rahona
- Multimodal Interaction Lab, Leibniz-Institut für Wissensmedien, Tübingen, Germany
| | | | - Xavier Romão
- CONSTRUCT-LESE, Faculty of Engineering, University of Porto, Porto, Portugal
| | - Susana Ruiz-Fernández
- Multimodal Interaction Lab, Leibniz-Institut für Wissensmedien, Tübingen, Germany.,FOM Hochschule für Oekonomie und Management, Essen, Germany.,LEAD Graduate School & Research Network, University of Tübingen, Tübingen, Germany
| | - Isabel Suarez
- Department of Psychology, Universidad del Norte, Barranquilla, Colombia
| | - Marion Tegethoff
- Division of Clinical Psychology and Psychiatry, Department of Psychology, University of Basel, Basel, Switzerland
| | - Mauricio Tejo
- Facultad de Ciencias Naturales y Exactas, Universidad de Playa Ancha, Valparaíso, Chile
| | - Rens van de Schoot
- Department of Methods and Statistics, Faculty of Social and Behavioural Sciences, Utrecht University, Utrecht, Netherlands.,North-West University, Optentia Research Focus Area, Vanderbijlpark, South Africa
| | - Ivan I Vankov
- Department of Cognitive Science and Psychology, New Bulgarian University, Sofia, Bulgaria
| | | | - Tonghui Wang
- Department of Mathematical Sciences, New Mexico State University, Las Cruces, NM, United States
| | - Yuki Yamada
- Faculty of Arts and Science, Kyushu University, Fukuoka, Japan
| | - Felipe C M Zoppino
- Oncology Laboratory, Instituto de Medicina y Biologia Experimental de Cuyo, CCT CONICET Mendoza, Mendoza, Argentina
| | | |
Collapse
|
27
|
Jamieson KH. Crisis or self-correction: Rethinking media narratives about the well-being of science. Proc Natl Acad Sci U S A 2018; 115:2620-2627. [PMID: 29531076 PMCID: PMC5856501 DOI: 10.1073/pnas.1708276114] [Citation(s) in RCA: 26] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/05/2023] Open
Abstract
After documenting the existence and exploring some implications of three alternative news narratives about science and its challenges, this essay outlines ways in which those who communicate science can more accurately convey its investigatory process, self-correcting norms, and remedial actions, without in the process legitimizing an unwarranted "science is broken/in crisis" narrative. The three storylines are: (i) quest discovery, which features scientists producing knowledge through an honorable journey; (ii) counterfeit quest discovery, which centers on an individual or group of scientists producing a spurious finding through a dishonorable one; and (iii) a systemic problem structure, which suggests that some of the practices that protect science are broken, or worse, that science is no longer self-correcting or in crisis.
Collapse
Affiliation(s)
- Kathleen Hall Jamieson
- Annenberg School for Communication, University of Pennsylvania, Philadelphia, PA 19104;
- Annenberg Public Policy Center, University of Pennsylvania, Philadelphia, PA 19104
| |
Collapse
|
28
|
Beyond ANOVA: An Introduction to Structural Equation Models for Experimental Designs. ORGANIZATIONAL RESEARCH METHODS 2018. [DOI: 10.1177/1094428118754988] [Citation(s) in RCA: 42] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
|
29
|
Abstract
In psychology, attempts to replicate published findings are less successful than expected. For properly powered studies replication rate should be around 80%, whereas in practice less than 40% of the studies selected from different areas of psychology can be replicated. Researchers in cognitive psychology are hindered in estimating the power of their studies, because the designs they use present a sample of stimulus materials to a sample of participants, a situation not covered by most power formulas. To remedy the situation, we review the literature related to the topic and introduce recent software packages, which we apply to the data of two masked priming studies with high power. We checked how we could estimate the power of each study and how much they could be reduced to remain powerful enough. On the basis of this analysis, we recommend that a properly powered reaction time experiment with repeated measures has at least 1,600 word observations per condition (e.g., 40 participants, 40 stimuli). This is considerably more than current practice. We also show that researchers must include the number of observations in meta-analyses because the effect sizes currently reported depend on the number of stimuli presented to the participants. Our analyses can easily be applied to new datasets gathered.
Collapse
|
30
|
Amrhein V, Korner-Nievergelt F, Roth T. The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research. PeerJ 2017; 5:e3544. [PMID: 28698825 PMCID: PMC5502092 DOI: 10.7717/peerj.3544] [Citation(s) in RCA: 144] [Impact Index Per Article: 20.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2017] [Accepted: 06/14/2017] [Indexed: 11/25/2022] Open
Abstract
The widespread use of 'statistical significance' as a license for making a claim of a scientific finding leads to considerable distortion of the scientific process (according to the American Statistical Association). We review why degrading p-values into 'significant' and 'nonsignificant' contributes to making studies irreproducible, or to making them seem irreproducible. A major problem is that we tend to take small p-values at face value, but mistrust results with larger p-values. In either case, p-values tell little about reliability of research, because they are hardly replicable even if an alternative hypothesis is true. Also significance (p ≤ 0.05) is hardly replicable: at a good statistical power of 80%, two studies will be 'conflicting', meaning that one is significant and the other is not, in one third of the cases if there is a true effect. A replication can therefore not be interpreted as having failed only because it is nonsignificant. Many apparent replication failures may thus reflect faulty judgment based on significance thresholds rather than a crisis of unreplicable research. Reliable conclusions on replicability and practical importance of a finding can only be drawn using cumulative evidence from multiple independent studies. However, applying significance thresholds makes cumulative knowledge unreliable. One reason is that with anything but ideal statistical power, significant effect sizes will be biased upwards. Interpreting inflated significant results while ignoring nonsignificant results will thus lead to wrong conclusions. But current incentives to hunt for significance lead to selective reporting and to publication bias against nonsignificant findings. Data dredging, p-hacking, and publication bias should be addressed by removing fixed significance thresholds. Consistent with the recommendations of the late Ronald Fisher, p-values should be interpreted as graded measures of the strength of evidence against the null hypothesis. Also larger p-values offer some evidence against the null hypothesis, and they cannot be interpreted as supporting the null hypothesis, falsely concluding that 'there is no effect'. Information on possible true effect sizes that are compatible with the data must be obtained from the point estimate, e.g., from a sample average, and from the interval estimate, such as a confidence interval. We review how confusion about interpretation of larger p-values can be traced back to historical disputes among the founders of modern statistics. We further discuss potential arguments against removing significance thresholds, for example that decision rules should rather be more stringent, that sample sizes could decrease, or that p-values should better be completely abandoned. We conclude that whatever method of statistical inference we use, dichotomous threshold thinking must give way to non-automated informed judgment.
Collapse
Affiliation(s)
- Valentin Amrhein
- Zoological Institute, University of Basel, Basel, Switzerland
- Research Station Petite Camargue Alsacienne, Saint-Louis, France
- Swiss Ornithological Institute, Sempach, Switzerland
| | | | - Tobias Roth
- Zoological Institute, University of Basel, Basel, Switzerland
- Research Station Petite Camargue Alsacienne, Saint-Louis, France
| |
Collapse
|
31
|
Oakes LM. Sample size, statistical power, and false conclusions in infant looking-time research. INFANCY 2017; 22:436-469. [PMID: 28966558 PMCID: PMC5618719 DOI: 10.1111/infa.12186] [Citation(s) in RCA: 103] [Impact Index Per Article: 14.7] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2016] [Accepted: 03/01/2017] [Indexed: 11/27/2022]
Abstract
Infant research is hard. It is difficult, expensive, and time consuming to identify, recruit and test infants. As a result, ours is a field of small sample sizes. Many studies using infant looking time as a measure have samples of 8 to 12 infants per cell, and studies with more than 24 infants per cell are uncommon. This paper examines the effect of such sample sizes on statistical power and the conclusions drawn from infant looking time research. An examination of the state of the current literature suggests that most published looking time studies have low power, which leads in the long run to an increase in both false positive and false negative results. Three data sets with large samples (>30 infants) were used to simulate experiments with smaller sample sizes; 1000 random subsamples of 8, 12, 16, 20, and 24 infants from the overall samples were selected, making it possible to examine the systematic effect of sample size on the results. This approach revealed that despite clear results with the original large samples, the results with smaller subsamples were highly variable, yielding both false positive and false negative outcomes. Finally, a number of emerging possible solutions are discussed.
Collapse
|
32
|
Jones A, Hardman CA, Lawrence N, Field M. Cognitive training as a potential treatment for overweight and obesity: A critical review of the evidence. Appetite 2017; 124:50-67. [PMID: 28546010 DOI: 10.1016/j.appet.2017.05.032] [Citation(s) in RCA: 81] [Impact Index Per Article: 11.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2017] [Revised: 05/08/2017] [Accepted: 05/17/2017] [Indexed: 02/07/2023]
Abstract
The aim of this review is to critically evaluate the effectiveness and candidate mechanisms of action of psychological interventions which aim to either (a) improve the capacity for self-regulatory, reflective processes or (b) reduce the impact of automatic appetitive processes, in an attempt to influence food intake and associated weight-gain. Our aim was to examine three important issues regarding each type of intervention: i) whether the intervention influenced behaviour in the laboratory, ii) whether the intervention influenced behaviour and/or body mass index in the real world, and iii) whether the proposed mechanism of action was supported by evidence. We systematically searched three commonly used databases and identified 32 articles which were relevant to at least one of these issues. The majority of studies attempted to manipulate food intake in the laboratory using associative learning paradigms, in normal-weight female participants. Most of the laboratory studies demonstrated the predicted effects of interventions on behaviour in the laboratory, but studies that attempted to translate these interventions outside of the laboratory yielded more mixed findings. The hypothesised mechanisms of action received inconsistent support across studies. We identified several limitations which may complicate interpretation of findings in this area, including heterogeneity of study methods, small sample sizes, and absence of adequate control groups. We provide recommendations for future studies that aim to develop and evaluate these promising interventions for the reduction of overweight and obesity.
Collapse
Affiliation(s)
- Andrew Jones
- Department of Psychological Sciences, University of Liverpool, UK; UK Centre for Tobacco and Alcohol Studies, University of Liverpool, UK.
| | | | | | - Matt Field
- Department of Psychological Sciences, University of Liverpool, UK; UK Centre for Tobacco and Alcohol Studies, University of Liverpool, UK
| |
Collapse
|
33
|
Giofrè D, Cumming G, Fresc L, Boedker I, Tressoldi P. The influence of journal submission guidelines on authors' reporting of statistics and use of open research practices. PLoS One 2017; 12:e0175583. [PMID: 28414751 PMCID: PMC5393581 DOI: 10.1371/journal.pone.0175583] [Citation(s) in RCA: 46] [Impact Index Per Article: 6.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/24/2016] [Accepted: 03/28/2017] [Indexed: 11/19/2022] Open
Abstract
From January 2014, Psychological Science introduced new submission guidelines that encouraged the use of effect sizes, estimation, and meta-analysis (the "new statistics"), required extra detail of methods, and offered badges for use of open science practices. We investigated the use of these practices in empirical articles published by Psychological Science and, for comparison, by the Journal of Experimental Psychology: General, during the period of January 2013 to December 2015. The use of null hypothesis significance testing (NHST) was extremely high at all times and in both journals. In Psychological Science, the use of confidence intervals increased markedly overall, from 28% of articles in 2013 to 70% in 2015, as did the availability of open data (3 to 39%) and open materials (7 to 31%). The other journal showed smaller or much smaller changes. Our findings suggest that journal-specific submission guidelines may encourage desirable changes in authors' practices.
Collapse
Affiliation(s)
- David Giofrè
- School of Natural Sciences and Psychology, Liverpool John Moores University, Liverpool, United Kingdom
| | - Geoff Cumming
- School of Psychology and Public Health, La Trobe University, Melbourne, Victoria, Australia
| | - Luca Fresc
- Department of General Psychology, University of Padova, Padova, Italy
| | - Ingrid Boedker
- Institute of Psychology, Health and Society, University of Liverpool, Liverpool, United Kingdom
| | | |
Collapse
|
34
|
Dumas-Mallet E, Button KS, Boraud T, Gonon F, Munafò MR. Low statistical power in biomedical science: a review of three human research domains. ROYAL SOCIETY OPEN SCIENCE 2017; 4:160254. [PMID: 28386409 PMCID: PMC5367316 DOI: 10.1098/rsos.160254] [Citation(s) in RCA: 119] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/12/2016] [Accepted: 01/04/2017] [Indexed: 05/23/2023]
Abstract
Studies with low statistical power increase the likelihood that a statistically significant finding represents a false positive result. We conducted a review of meta-analyses of studies investigating the association of biological, environmental or cognitive parameters with neurological, psychiatric and somatic diseases, excluding treatment studies, in order to estimate the average statistical power across these domains. Taking the effect size indicated by a meta-analysis as the best estimate of the likely true effect size, and assuming a threshold for declaring statistical significance of 5%, we found that approximately 50% of studies have statistical power in the 0-10% or 11-20% range, well below the minimum of 80% that is often considered conventional. Studies with low statistical power appear to be common in the biomedical sciences, at least in the specific subject areas captured by our search strategy. However, we also observe evidence that this depends in part on research methodology, with candidate gene studies showing very low average power and studies using cognitive/behavioural measures showing high average power. This warrants further investigation.
Collapse
Affiliation(s)
- Estelle Dumas-Mallet
- Institute of Neurodegenerative Diseases, CNRS UMR-5293, University of Bordeaux, Bordeaux, France
- Centre Emile Durkheim, CNRS UMR-5116, University of Bordeaux, Bordeaux, France
| | | | - Thomas Boraud
- Institute of Neurodegenerative Diseases, CNRS UMR-5293, University of Bordeaux, Bordeaux, France
| | - Francois Gonon
- Institute of Neurodegenerative Diseases, CNRS UMR-5293, University of Bordeaux, Bordeaux, France
| | - Marcus R. Munafò
- MRC Integrative Epidemiology Unit at the University of Bristol, Bristol, UK
- UK Centre for Tobacco and Alcohol Studies, School of Experimental Psychology, University of Bristol, Bristol, UK
| |
Collapse
|
35
|
Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, Simonsohn U, Wagenmakers EJ, Ware JJ, Ioannidis JPA. A manifesto for reproducible science. Nat Hum Behav 2017; 1:0021. [PMID: 33954258 PMCID: PMC7610724 DOI: 10.1038/s41562-016-0021] [Citation(s) in RCA: 1172] [Impact Index Per Article: 167.4] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/11/2022]
Abstract
Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.
Collapse
Affiliation(s)
- Marcus R. Munafò
- MRC Integrative Epidemiology Unit, University of Bristol, Bristol, BS8 2BN UK
- UK Centre for Tobacco and Alcohol Studies, School of Experimental Psychology, University of Bristol, 12a Priory Road, Bristol, BS8 1TU UK
| | - Brian A. Nosek
- Department of Psychology, University of Virginia, Charlottesville, 22904 Virginia USA
- Center for Open Science, Charlottesville, 22903 Virginia USA
| | - Dorothy V. M. Bishop
- Department of Experimental Psychology, University of Oxford, 9 South Parks Road, Oxford, OX1 3UD UK
| | | | - Christopher D. Chambers
- Cardiff University Brain Research Imaging Centre, School of Psychology, Cardiff University, Cardiff, CF24 4HQ UK
| | - Nathalie Percie du Sert
- National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs), London, NW1 2BE UK
| | - Uri Simonsohn
- The Wharton School, University of Pennsylvania, Philadelphia, 19104 Pennsylvania USA
| | - Eric-Jan Wagenmakers
- Department of Psychology, University of Amsterdam, Amsterdam, 1018 WT Netherlands
| | | | - John P. A. Ioannidis
- Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, 94304 California USA
- Department of Medicine and Department of Health Research and Policy, Stanford Prevention Research Center, Stanford University School of Medicine, Stanford, 94305 California USA
- Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, 94305 California USA
| |
Collapse
|
36
|
Button KS, Bal L, Clark A, Shipley T. Preventing the ends from justifying the means: withholding results to address publication bias in peer-review. BMC Psychol 2016; 4:59. [PMID: 27903302 PMCID: PMC5131510 DOI: 10.1186/s40359-016-0167-7] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2016] [Accepted: 11/15/2016] [Indexed: 02/06/2023] Open
Abstract
The evidence that many of the findings in the published literature may be unreliable is compelling. There is an excess of positive results, often from studies with small sample sizes, or other methodological limitations, and the conspicuous absence of null findings from studies of a similar quality. This distorts the evidence base, leading to false conclusions and undermining scientific progress. Central to this problem is a peer-review system where the decisions of authors, reviewers, and editors are more influenced by impressive results than they are by the validity of the study design. To address this, BMC Psychology is launching a pilot to trial a new 'results-free' peer-review process, whereby editors and reviewers are blinded to the study's results, initially assessing manuscripts on the scientific merits of the rationale and methods alone. The aim is to improve the reliability and quality of published research, by focusing editorial decisions on the rigour of the methods, and preventing impressive ends justifying poor means.
Collapse
|
37
|
Smaldino PE, McElreath R. The natural selection of bad science. ROYAL SOCIETY OPEN SCIENCE 2016; 3:160384. [PMID: 27703703 PMCID: PMC5043322 DOI: 10.1098/rsos.160384] [Citation(s) in RCA: 287] [Impact Index Per Article: 35.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/01/2016] [Accepted: 08/17/2016] [Indexed: 05/21/2023]
Abstract
Poor research design and data analysis encourage false-positive findings. Such poor methods persist despite perennial calls for improvement, suggesting that they result from something more than just misunderstanding. The persistence of poor methods results partly from incentives that favour them, leading to the natural selection of bad science. This dynamic requires no conscious strategizing-no deliberate cheating nor loafing-by scientists, only that publication is a principal factor for career advancement. Some normative methods of analysis have almost certainly been selected to further publication instead of discovery. In order to improve the culture of science, a shift must be made away from correcting misunderstandings and towards rewarding understanding. We support this argument with empirical evidence and computational modelling. We first present a 60-year meta-analysis of statistical power in the behavioural sciences and show that power has not improved despite repeated demonstrations of the necessity of increasing power. To demonstrate the logical consequences of structural incentives, we then present a dynamic model of scientific communities in which competing laboratories investigate novel or previously published hypotheses using culturally transmitted research methods. As in the real world, successful labs produce more 'progeny,' such that their methods are more often copied and their students are more likely to start labs of their own. Selection for high output leads to poorer methods and increasingly high false discovery rates. We additionally show that replication slows but does not stop the process of methodological deterioration. Improving the quality of research requires change at the institutional level.
Collapse
Affiliation(s)
- Paul E. Smaldino
- Cognitive and Information Sciences, University of California, Merced, CA 95343, USA
| | - Richard McElreath
- Department of Human Behavior, Ecology, and Culture, Max Planck Institute for Evolutionary Anthropology, Leipzig, Germany
| |
Collapse
|
38
|
Human LJ, Thorson KR, Mendes WB. Interactive Effects Between Extraversion and Oxytocin Administration. SOCIAL PSYCHOLOGICAL AND PERSONALITY SCIENCE 2016. [DOI: 10.1177/1948550616644964] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/19/2023]
Abstract
Intranasal administration of the neuropeptide oxytocin (OT) appears to have positive social consequences, but these effects are often highly context- and person-specific. The present research examined whether the core personality trait of extraversion may be one important person-specific factor that plays a role in these associations. Across two double-blind randomized placebo-controlled studies (total ns: Study 1 = 121; Study 2 = 112), we observed significant interactions between OT administration and extraversion predicting prosocial outcomes. For individuals low in extraversion, OT administration relative to placebo led to greater perceived social connection and prosocial tendencies (Study 1) and more positive behavioral responses to help and greater trust of an interaction partner (Study 2). In contrast, OT administration was not beneficial for individuals high in extraversion. Overall, these findings contribute to growing evidence that OT administration has complex, person-specific effects on social behavior, indicating that extraversion plays an important role in these associations.
Collapse
Affiliation(s)
- Lauren J. Human
- Department of Psychology, McGill University, Montreal, Quebec, Canada
| | | | - Wendy Berry Mendes
- Department of Psychiatry, University of California, San Francisco, CA, USA
| |
Collapse
|
39
|
Van Bavel JJ, Mende-Siedlecki P, Brady WJ, Reinero DA. Contextual sensitivity in scientific reproducibility. Proc Natl Acad Sci U S A 2016; 113:6454-9. [PMID: 27217556 PMCID: PMC4988618 DOI: 10.1073/pnas.1521897113] [Citation(s) in RCA: 168] [Impact Index Per Article: 21.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022] Open
Abstract
In recent years, scientists have paid increasing attention to reproducibility. For example, the Reproducibility Project, a large-scale replication attempt of 100 studies published in top psychology journals found that only 39% could be unambiguously reproduced. There is a growing consensus among scientists that the lack of reproducibility in psychology and other fields stems from various methodological factors, including low statistical power, researcher's degrees of freedom, and an emphasis on publishing surprising positive results. However, there is a contentious debate about the extent to which failures to reproduce certain results might also reflect contextual differences (often termed "hidden moderators") between the original research and the replication attempt. Although psychologists have found extensive evidence that contextual factors alter behavior, some have argued that context is unlikely to influence the results of direct replications precisely because these studies use the same methods as those used in the original research. To help resolve this debate, we recoded the 100 original studies from the Reproducibility Project on the extent to which the research topic of each study was contextually sensitive. Results suggested that the contextual sensitivity of the research topic was associated with replication success, even after statistically adjusting for several methodological characteristics (e.g., statistical power, effect size). The association between contextual sensitivity and replication success did not differ across psychological subdisciplines. These results suggest that researchers, replicators, and consumers should be mindful of contextual factors that might influence a psychological process. We offer several guidelines for dealing with contextual sensitivity in reproducibility.
Collapse
Affiliation(s)
- Jay J Van Bavel
- Department of Psychology, New York University, New York, NY 10003
| | | | - William J Brady
- Department of Psychology, New York University, New York, NY 10003
| | - Diego A Reinero
- Department of Psychology, New York University, New York, NY 10003
| |
Collapse
|
40
|
Bayarri MJ, Benjamin DJ, Berger JO, Sellke TM. Rejection odds and rejection ratios: A proposal for statistical practice in testing hypotheses. JOURNAL OF MATHEMATICAL PSYCHOLOGY 2016; 72:90-103. [PMID: 30713353 PMCID: PMC6358203 DOI: 10.1016/j.jmp.2015.12.007] [Citation(s) in RCA: 45] [Impact Index Per Article: 5.6] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/05/2023]
Abstract
Much of science is (rightly or wrongly) driven by hypothesis testing. Even in situations where the hypothesis testing paradigm is correct, the common practice of basing inferences solely on p-values has been under intense criticism for over 50 years. We propose, as an alternative, the use of the odds of a correct rejection of the null hypothesis to incorrect rejection. Both pre-experimental versions (involving the power and Type I error) and post-experimental versions (depending on the actual data) are considered. Implementations are provided that range from depending only on the p-value to consideration of full Bayesian analysis. A surprise is that all implementations - even the full Bayesian analysis - have complete frequentist justification. Versions of our proposal can be implemented that require only minor modifications to existing practices yet overcome some of their most severe shortcomings.
Collapse
|
41
|
|
42
|
Tressoldi PE, Giofré D. The pervasive avoidance of prospective statistical power: major consequences and practical solutions. Front Psychol 2015; 6:726. [PMID: 26074863 PMCID: PMC4446541 DOI: 10.3389/fpsyg.2015.00726] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2015] [Accepted: 05/15/2015] [Indexed: 11/13/2022] Open
Affiliation(s)
| | - David Giofré
- Dipartimento di Psicologia Generale, Università di Padova Padova, Italy
| |
Collapse
|
43
|
Francis G. Excess success for three related papers on racial bias. Front Psychol 2015; 6:512. [PMID: 25983701 PMCID: PMC4416456 DOI: 10.3389/fpsyg.2015.00512] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2014] [Accepted: 04/10/2015] [Indexed: 11/16/2022] Open
Abstract
Three related articles reported that racial bias altered perceptual experience and influenced decision-making. These findings have been applied to training programs for law enforcement, and elsewhere, to mitigate racial bias. However, a statistical analysis of each of the three articles finds that the reported experimental results should be rare, even if the theoretical ideas were correct. The analysis estimates that the probability of the reported experimental success for the articles is 0.003, 0.048, and 0.070, respectively. These low probabilities suggest that similar future work is unlikely to produce as successful outcomes and indicates that readers should be skeptical about the validity of the reported findings and their theoretical implications. The reported findings should not be used to guide policies related to racial bias, and new experimental work is needed to judge the merit of the theoretical ideas.
Collapse
Affiliation(s)
- Gregory Francis
- Department of Psychological Sciences, Purdue University, West Lafayette, IN USA ; Brain Mind Institute, École Polytechnique Fédérale de Lausanne, Lausanne Switzerland
| |
Collapse
|
44
|
Gervais WM, Jewell JA, Najle MB, Ng BKL. A Powerful Nudge? Presenting Calculable Consequences of Underpowered Research Shifts Incentives Toward Adequately Powered Designs. SOCIAL PSYCHOLOGICAL AND PERSONALITY SCIENCE 2015. [DOI: 10.1177/1948550615584199] [Citation(s) in RCA: 26] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
If psychologists have recognized the pitfalls of underpowered research for decades, why does it persist? Incentives, perhaps: underpowered research benefits researchers individually (increased productivity), but harms science collectively (inflated Type I error rates and effect size estimates but low replication rates). Yet, researchers can selectively reward power at various scientific bottlenecks (e.g., peer review, hiring, funding, and promotion). We designed a stylized thought experiment to evaluate the degree to which researchers consider power and productivity in hiring decisions. Accomplished psychologists chose between a low sample size candidate and a high sample size candidate who were otherwise identical. We manipulated the degree to which participants received information about (1) productivity, (2) sample size, and (3) directly calculable Type I error and replication rates. Participants were intolerant of the negative consequences of low-power research, yet merely indifferent regarding the practices that logically produce those consequences, unless those consequences were made quite explicit.
Collapse
|
45
|
Savalei V, Dunn E. Is the call to abandon p-values the red herring of the replicability crisis? Front Psychol 2015; 6:245. [PMID: 25798124 PMCID: PMC4351564 DOI: 10.3389/fpsyg.2015.00245] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2014] [Accepted: 02/17/2015] [Indexed: 12/02/2022] Open
Affiliation(s)
- Victoria Savalei
- Department of Psychology, University of British Columbia Vancouver, BC, Canada
| | - Elizabeth Dunn
- Department of Psychology, University of British Columbia Vancouver, BC, Canada
| |
Collapse
|
46
|
Ware JJ, Munafò MR. Significance chasing in research practice: causes, consequences and possible solutions. Addiction 2015; 110:4-8. [PMID: 25040652 PMCID: PMC4962906 DOI: 10.1111/add.12673] [Citation(s) in RCA: 33] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/23/2013] [Revised: 11/11/2013] [Accepted: 06/18/2014] [Indexed: 12/17/2022]
Abstract
BACKGROUND AND AIMS The low reproducibility of findings within the scientific literature is a growing concern. This may be due to many findings being false positives which, in turn, can misdirect research effort and waste money. METHODS We review factors that may contribute to poor study reproducibility and an excess of 'significant' findings within the published literature. Specifically, we consider the influence of current incentive structures and the impact of these on research practices. RESULTS The prevalence of false positives within the literature may be attributable to a number of questionable research practices, ranging from the relatively innocent and minor (e.g. unplanned post-hoc tests) to the calculated and serious (e.g. fabrication of data). These practices may be driven by current incentive structures (e.g. pressure to publish), alongside the preferential emphasis placed by journals on novelty over veracity. There are a number of potential solutions to poor reproducibility, such as new publishing formats that emphasize the research question and study design, rather than the results obtained. This has the potential to minimize significance chasing and non-publication of null findings. CONCLUSIONS Significance chasing, questionable research practices and poor study reproducibility are the unfortunate consequence of a 'publish or perish' culture and a preference among journals for novel findings. It is likely that top-down change implemented by those with the ability to modify current incentive structure (e.g. funders and journals) will be required to address problems of poor reproducibility.
Collapse
Affiliation(s)
- Jennifer J. Ware
- School of Social and Community Medicine, University of Bristol,
Bristol, UK
- MRC Integrative Epidemiology Unit (IEU) at the University of
Bristol, Bristol, UK
- UK Centre for Tobacco and Alcohol Studies, University of Bristol,
Bristol, UK
| | - Marcus R. Munafò
- School of Experimental Psychology, University of Bristol, Bristol,
UK
- MRC Integrative Epidemiology Unit (IEU) at the University of
Bristol, Bristol, UK
- UK Centre for Tobacco and Alcohol Studies, University of Bristol,
Bristol, UK
| |
Collapse
|