1
|
Brisco E, Kulinskaya E, Koricheva J. Assessment of temporal instability in the applied ecology and conservation evidence base. Res Synth Methods 2024; 15:398-412. [PMID: 38111354 DOI: 10.1002/jrsm.1691] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2023] [Revised: 09/18/2023] [Accepted: 11/22/2023] [Indexed: 12/20/2023]
Abstract
Outcomes of meta-analyses are increasingly used to inform evidence-based decision making in various research fields. However, a number of recent studies have reported rapid temporal changes in magnitude and significance of the reported effects which could make policy-relevant recommendations from meta-analyses to quickly go out of date. We assessed the extent and patterns of temporal trends in magnitude and statistical significance of the cumulative effects in meta-analyses in applied ecology and conservation published between 2004 and 2018. Of the 121 meta-analyses analysed, 93% showed a temporal trend in cumulative effect magnitude or significance with 27% of the datasets exhibiting temporal trends in both. The most common trend was the early study effect when at least one of the first 5 years effect size estimates exhibited more than 50% magnitude difference to the subsequent estimate. The observed temporal trends persisted in majority of datasets once moderators were accounted for. Only 5 datasets showed significant changes in sample size over time which could potentially explain the observed temporal change in the cumulative effects. Year of publication of meta-analysis had no significant effect on presence of temporal trends in cumulative effects. Our results show that temporal changes in magnitude and statistical significance in applied ecology are widespread and represent a serious potential threat to use of meta-analyses for decision-making in conservation and environmental management. We recommend use of cumulative meta-analyses and call for more studies exploring the causes of the temporal effects.
Collapse
Affiliation(s)
- Elizabeth Brisco
- Department of Biological Sciences, Royal Holloway University of London, Egham, UK
| | - Elena Kulinskaya
- School of Computing Sciences, University of East Anglia, Norwich Research Park, Norwich, UK
| | - Julia Koricheva
- Department of Biological Sciences, Royal Holloway University of London, Egham, UK
| |
Collapse
|
2
|
Koevoet D, Postma A. Is there selective retroactive memory enhancement in humans?: a meta-analysis. Psychon Bull Rev 2024; 31:531-540. [PMID: 37749381 PMCID: PMC11061014 DOI: 10.3758/s13423-023-02372-5] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/19/2023] [Indexed: 09/27/2023]
Abstract
Memory is an adaptive and flexible system that preferentially stores motivationally relevant information. However, in some cases information that is initially irrelevant can become relevant at a later time. The question arises whether and to what extent the memory system can retroactively boost memories of the initially irrelevant information. Experimental studies in animals and humans have provided evidence for such retroactive memory boosting. Additionally, these studies suggest that retroactive memory enhancement (RME) can be selective to the semantic meaning of the material. Nonetheless, recent experimental work could not replicate these findings, posing the question whether the selective RME effect is reliable. To synthesize the available evidence, we conducted meta-analyses of 14 experiments. Although the classical meta-analytic procedure suggested a small selective RME effect, Cohen's dz = 0.16, when accounting for small-study bias using robust Bayesian meta-analysis the null hypothesis was supported, Cohen's dz = 0.02, BF01 = 3.03. Furthermore strong evidence was found for a bias due to small-study effects, BF10 = 11.39. Together, this calls the reliability of a selective RME effect into question.
Collapse
Affiliation(s)
- Damian Koevoet
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, The Netherlands.
| | - Albert Postma
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
3
|
van Assen MALM, van den Akker OR, Augusteijn HEM, Bakker M, Nuijten MB, Olsson-Collentine A, Stoevenbelt AH, Wicherts JM, van Aert RCM. The Meta-Plot. ZEITSCHRIFT FUR PSYCHOLOGIE-JOURNAL OF PSYCHOLOGY 2023. [DOI: 10.1027/2151-2604/a000513] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/24/2023]
Abstract
Abstract. The meta-plot is a descriptive visual tool for meta-analysis that provides information on the primary studies in the meta-analysis and the results of the meta-analysis. More precisely, the meta-plot portrays (1) the precision and statistical power of the primary studies in the meta-analysis, (2) the estimate and confidence interval of a random-effects meta-analysis, (3) the results of a cumulative random-effects meta-analysis yielding a robustness check of the meta-analytic effect size with respect to primary studies’ precision, and (4) evidence of publication bias. After explaining the underlying logic and theory, the meta-plot is applied to two cherry-picked meta-analyses that appear to be biased and to 10 randomly selected meta-analyses from the psychological literature. We recommend accompanying any meta-analysis of common effect size measures with the meta-plot.
Collapse
Affiliation(s)
- Marcel A. L. M. van Assen
- Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, the Netherlands
- Department of Sociology, Faculty of Social and Behavioural Sciences, Utrecht University, the Netherlands
| | - Olmo R. van den Akker
- Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, the Netherlands
| | - Hilde E. M. Augusteijn
- Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, the Netherlands
| | - Marjan Bakker
- Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, the Netherlands
| | - Michèle B. Nuijten
- Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, the Netherlands
| | - Anton Olsson-Collentine
- Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, the Netherlands
| | - Andrea H. Stoevenbelt
- Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, the Netherlands
| | - Jelte M. Wicherts
- Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, the Netherlands
| | - Robbie C. M. van Aert
- Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, the Netherlands
| |
Collapse
|
4
|
Fraley RC, Chong JY, Baacke KA, Greco AJ, Guan H, Vazire S. Journal N-Pact Factors From 2011 to 2019: Evaluating the Quality of Social/Personality Journals With Respect to Sample Size and Statistical Power. ADVANCES IN METHODS AND PRACTICES IN PSYCHOLOGICAL SCIENCE 2022. [DOI: 10.1177/25152459221120217] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Scholars and institutions commonly use impact factors to evaluate the quality of empirical research. However, a number of findings published in journals with high impact factors have failed to replicate, suggesting that impact alone may not be an accurate indicator of quality. Fraley and Vazire proposed an alternative index, the N-pact factor, which indexes the median sample size of published studies, providing a narrow but relevant indicator of research quality. In the present research, we expand on the original report by examining the N-pact factor of social/personality-psychology journals between 2011 and 2019, incorporating additional journals and accounting for study design (i.e., between persons, repeated measures, and mixed). There was substantial variation in the sample sizes used in studies published in different journals. Journals that emphasized personality processes and individual differences had larger N-pact factors than journals that emphasized social-psychological processes. Moreover, N-pact factors were largely independent of traditional markers of impact. Although the majority of journals in 2011 published studies that were not well powered to detect an effect of ρ = .20, this situation had improved considerably by 2019. In 2019, eight of the nine journals we sampled published studies that were, on average, powered at 80% or higher to detect such an effect. After decades of unheeded warnings from methodologists about the dangers of small-sample designs, the field of social/personality psychology has begun to use larger samples. We hope the N-pact factor will be supplemented by other indices that can be used as alternatives to improve further the evaluation of research.
Collapse
Affiliation(s)
- R. Chris Fraley
- Department of Psychology, University of Illinois at Urbana-Champaign, Champaign, Illinois
| | - Jia Y. Chong
- Department of Psychology, University of Illinois at Urbana-Champaign, Champaign, Illinois
| | - Kyle A. Baacke
- Department of Psychology, University of Illinois at Urbana-Champaign, Champaign, Illinois
| | - Anthony J. Greco
- Department of Psychology, University of Illinois at Urbana-Champaign, Champaign, Illinois
| | - Hanxiong Guan
- Department of Psychology, University of Illinois at Urbana-Champaign, Champaign, Illinois
| | - Simine Vazire
- Melbourne School of Psychological Sciences, University of Melbourne, Melbourne, Australia
| |
Collapse
|
5
|
Wake GE, Fitie GW, Tizazu MA. A Systematic Review and Meta-Analysis on Post-Abortion Contraceptive Utilization and Associated Factors in Ethiopia. Front Public Health 2022; 10:883710. [PMID: 35669760 PMCID: PMC9163437 DOI: 10.3389/fpubh.2022.883710] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2022] [Accepted: 04/22/2022] [Indexed: 11/13/2022] Open
Abstract
Background Post-abortion contraceptive utilization is the initiation and use of family planning methods at the time of management of abortion or before fertility returns. A significant discrepancy was reported regarding the prevalence and its associated factors of post-abortion contraceptive utilization in Ethiopia. So, this systematic review and meta-analysis aimed to estimate the pooled prevalence of post-abortion contraceptive utilization and its associated factors in Ethiopia. Methods Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guideline was used. The databases such as PubMed, Google Scholar, Science Direct, Cochrane library, Scopus, CINAHL, Web of Science, and additional searches by using direct Google search, libraries, and preprint were searched. Joanna Briggs Institute Meta-Analysis of Statistics Assessment and Review Instrument (JBI-MAStARI) was used for critical appraisal. Results A total of 17 studies with 13,075 individuals were included. Of these, 14 studies with 5,719 individuals were used to estimate the prevalence. The pooled prevalence of post-abortion contraceptive utilization in Ethiopia was 63.64% (95% CI: 57.75–69.53). The subgroup analysis reported the highest prevalence of post-abortion contraceptive utilization in a study conducted in Addis Ababa (77.40%), a study published in 2015–2021 (66.15%), and among studies with a sample size >400 (66.84%). The pooled odds ratio (OR) of post-abortion contraceptive utilization for the mothers who had post-abortion family planning counseling was 4.15 (95% CI = 1.30, 13.2), and history of family planning utilization was 4.28 (95% CI = 2.66, 6.89). Conclusions The pooled prevalence of utilization of post-abortion contraceptives in this meta-analysis remains low. Post-abortion family planning counseling and the history of the utilization of modern family planning methods were significantly associated with the practice of post-abortion contraceptives. The Ministry of Health should encourage post-abortion family planning utilization, making more efforts on post-abortion contraceptive counseling. Health facilities should work hard to strengthen the family planning counseling service, especially focusing on those who had no previous family planning utilization.
Collapse
|
6
|
Wicht CA, De Pretto M, Mouthon M, Spierer L. Neural correlates of expectations-induced effects of caffeine intake on executive functions. Cortex 2022; 150:61-84. [DOI: 10.1016/j.cortex.2022.02.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2021] [Revised: 01/25/2022] [Accepted: 02/18/2022] [Indexed: 11/29/2022]
|
7
|
Wake GE, Mittiku YM. Prevalence of exclusive breastfeeding practice and its association with maternal employment in Ethiopia: a systematic review and meta-analysis. Int Breastfeed J 2021; 16:86. [PMID: 34717673 PMCID: PMC8557507 DOI: 10.1186/s13006-021-00432-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/03/2020] [Accepted: 10/19/2021] [Indexed: 11/10/2022] Open
Abstract
Background Exclusive breastfeeding is defined as the practice of providing only breast milk for an infant for the first 6 months of life without the addition of any other food or water, except for vitamins, mineral supplements, and medicines. Findings are inconsistent regarding the prevalence of exclusive breastfeeding in Ethiopia. Full-time maternal employment is an important factor contributing to the low rates of practice of exclusive breastfeeding. Empowering women to exclusively breastfeed, by enacting 6 months’ mandatory paid maternity leave can increase the rate of exclusive breastfeeding in the first 6 months of life up to 50%. The purpose of this review was to estimate the pooled prevalence of exclusive breastfeeding and its association with full-time maternal employment in the first 6 months of life for infants in the context of Ethiopia. Methods The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline was used in this systematic review and meta-analysis. All observational studies reporting the prevalence of exclusive breastfeeding and its association with maternal employment in Ethiopia were considered. The search was conducted from 6 November 2020 to 31 December 2020 and all papers published in the English language from 1 January 2015 to 31 December 2020 were included in this review. Results Forty-five studies were included in the final analysis after reviewing 751 studies in this meta-analysis yielding the pooled prevalence of EBF 60.42% (95% CI 55.81, 65.02) at 6 months in Ethiopia. Those full-time employed mothers in the first 6 months were 57% less likely to practice exclusive breastfeeding in comparison to mothers not in paid employment in Ethiopia (OR 0.43; 95% CI 0.31, 0.61). Conclusions Full-time maternal employment was negatively associated with the practice of exclusive breastfeeding in comparison to unemployed mothers. The prevalence of exclusive breastfeeding in Ethiopia is low in comparison to the global recommendation. The Ethiopian government should implement policies that empower women. The governmental and non-governmental organizations should create a conducive environment for mothers to practice exclusive breastfeeding in the workplace. Supplementary Information The online version contains supplementary material available at 10.1186/s13006-021-00432-x.
Collapse
Affiliation(s)
- Getu Engida Wake
- College of Health Science, Debre Berhan University, Debre Berhan, Ethiopia.
| | | |
Collapse
|
8
|
Nosek BA, Hardwicke TE, Moshontz H, Allard A, Corker KS, Dreber A, Fidler F, Hilgard J, Struhl MK, Nuijten MB, Rohrer JM, Romero F, Scheel AM, Scherer LD, Schönbrodt FD, Vazire S. Replicability, Robustness, and Reproducibility in Psychological Science. Annu Rev Psychol 2021; 73:719-748. [PMID: 34665669 DOI: 10.1146/annurev-psych-020821-114157] [Citation(s) in RCA: 118] [Impact Index Per Article: 39.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Replication-an important, uncommon, and misunderstood practice-is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
Collapse
Affiliation(s)
- Brian A Nosek
- Department of Psychology, University of Virginia, Charlottesville, Virginia 22904, USA; .,Center for Open Science, Charlottesville, Virginia 22903, USA
| | - Tom E Hardwicke
- Department of Psychology, University of Amsterdam, 1012 ZA Amsterdam, The Netherlands
| | - Hannah Moshontz
- Addiction Research Center, University of Wisconsin-Madison, Madison, Wisconsin 53706, USA
| | - Aurélien Allard
- Department of Psychology, University of California, Davis, California 95616, USA
| | - Katherine S Corker
- Psychology Department, Grand Valley State University, Allendale, Michigan 49401, USA
| | - Anna Dreber
- Department of Economics, Stockholm School of Economics, 113 83 Stockholm, Sweden
| | - Fiona Fidler
- School of Biosciences, University of Melbourne, Parkville VIC 3010, Australia
| | - Joe Hilgard
- Department of Psychology, Illinois State University, Normal, Illinois 61790, USA
| | | | - Michèle B Nuijten
- Meta-Research Center, Tilburg University, 5037 AB Tilburg, The Netherlands
| | - Julia M Rohrer
- Department of Psychology, Leipzig University, 04109 Leipzig, Germany
| | - Felipe Romero
- Department of Theoretical Philosophy, University of Groningen, 9712 CP, The Netherlands
| | - Anne M Scheel
- Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands
| | - Laura D Scherer
- University of Colorado Anschutz Medical Campus, Aurora, Colorado 80045, USA
| | - Felix D Schönbrodt
- Department of Psychology, Ludwig Maximilian University of Munich, 80539 Munich, Germany
| | - Simine Vazire
- School of Psychological Sciences, University of Melbourne, Parkville VIC 3052, Australia
| |
Collapse
|
9
|
A meta-review of transparency and reproducibility-related reporting practices in published meta-analyses on clinical psychological interventions (2000-2020). Behav Res Methods 2021; 54:334-349. [PMID: 34173943 PMCID: PMC8863703 DOI: 10.3758/s13428-021-01644-z] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/02/2021] [Indexed: 12/12/2022]
Abstract
Meta-analysis is a powerful and important tool to synthesize the literature about a research topic. Like other kinds of research, meta-analyses must be reproducible to be compliant with the principles of the scientific method. Furthermore, reproducible meta-analyses can be easily updated with new data and reanalysed applying new and more refined analysis techniques. We attempted to empirically assess the prevalence of transparency and reproducibility-related reporting practices in published meta-analyses from clinical psychology by examining a random sample of 100 meta-analyses. Our purpose was to identify the key points that could be improved, with the aim of providing some recommendations for carrying out reproducible meta-analyses. We conducted a meta-review of meta-analyses of psychological interventions published between 2000 and 2020. We searched PubMed, PsycInfo and Web of Science databases. A structured coding form to assess transparency indicators was created based on previous studies and existing meta-analysis guidelines. We found major issues concerning: completely reproducible search procedures report, specification of the exact method to compute effect sizes, choice of weighting factors and estimators, lack of availability of the raw statistics used to compute the effect size and of interoperability of available data, and practically total absence of analysis script code sharing. Based on our findings, we conclude with recommendations intended to improve the transparency, openness, and reproducibility-related reporting practices of meta-analyses in clinical psychology and related areas.
Collapse
|
10
|
Aschwanden D, Strickhouser JE, Luchetti M, Stephan Y, Sutin AR, Terracciano A. Is personality associated with dementia risk? A meta-analytic investigation. Ageing Res Rev 2021; 67:101269. [PMID: 33561581 PMCID: PMC8005464 DOI: 10.1016/j.arr.2021.101269] [Citation(s) in RCA: 62] [Impact Index Per Article: 20.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2020] [Revised: 09/08/2020] [Accepted: 02/02/2021] [Indexed: 02/06/2023]
Abstract
This study provides a quantitative synthesis of the prospective associations between personality traits (neuroticism, extraversion, openness, agreeableness, conscientiousness) and the risk of incident Alzheimer's disease and related dementias. We conducted five separate meta-analyses with 8-12 samples (N = 30,036 to 33,054) that were identified through a systematic literature search following the MOOSE guidelines. Higher neuroticism (HR = 1.24, 95% CI [1.17, 1.31]) and lower conscientiousness (HR = 0.77, 95% CI [0.73, 0.81]) were associated with increased dementia risk, even after accounting for covariates such as depressive symptoms. Lower extraversion (HR = 0.92, 95% CI [0.86, 0.97]), openness (HR = 0.91, 95% CI [0.86, 0.96]), and agreeableness (HR = 0.90, 95% CI [0.83, 0.98]) were also associated with increased risk, but these associations were less robust and not significant in fully adjusted models. No evidence of publication bias was found. The strength of associations was unrelated to publication year (i.e., no evidence of winner's curse). Meta-regressions indicated consistent effects for neuroticism, openness, and conscientiousness across methods to assess dementia, dementia type, follow-up length, sample age, minority, country, and personality measures. The association of extraversion and agreeableness varied by country. Our findings indicate robust associations of neuroticism and conscientiousness with dementia risk.
Collapse
|
11
|
Rominger C, Graßmann TM, Weber B, Schwerdtfeger AR. Does contingent biofeedback improve cardiac interoception? A preregistered replication of Meyerholz, Irzinger, Withöft, Gerlach, and Pohl (2019) using the heartbeat discrimination task in a randomised control trial. PLoS One 2021; 16:e0248246. [PMID: 33725020 PMCID: PMC7963047 DOI: 10.1371/journal.pone.0248246] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2020] [Accepted: 02/19/2021] [Indexed: 12/23/2022] Open
Abstract
Meyerholz, Irzinger, Withöft, Gerlach, and Pohl (2019) reported on a comparably large effect (d = 1.21) of a contingent biofeedback procedure on cardiac accuracy as assessed by the heartbeat tracking task. However, this task has recently been criticized as a measure of interoceptive accuracy. We aimed to replicate this finding by using the well-validated heartbeat discrimination task and to compare the biofeedback with a deep breathing and a control condition (viewing a film clip). The trial was preregistered at open science framework (https://osf.io/9fxn6). Overall, 93 participants were randomized to one of the three conditions and the heartbeat discrimination task was presented prior and after the 20-minutes training sessions. The study had a power of .86 to detect a medium-sized effect in the biofeedback group and a power of .96 to detect a medium-sized interaction of intervention group and time. A general tendency for improvement in heartbeat detection accuracy was found across intervention groups (d = 0.19, p = .08); however, groups did not differ significantly. In particular, there was no significant interaction of intervention group and time (f = .00, p = .98) and no reliable effect for the biofeedback group (d = 0.15, p = .42). One limitation is that a different, but well-validated task was used to quantify interoceptive accuracy. This study suggests that biofeedback might not improve interoceptive accuracy in the cardiac domain, but effects seem to depend on the specific task applied.
Collapse
Affiliation(s)
- Christian Rominger
- Institute of Psychology, University of Graz, Graz, Austria
- Otto Loewi Research Center, Section of Physiology, Medical University of Graz, Graz, Austria
- * E-mail:
| | | | - Bernhard Weber
- Institute of Psychology, University of Graz, Graz, Austria
| | | |
Collapse
|
12
|
McClain MB, Callan GL, Harris B, Floyd RG, Haverkamp CR, Golson ME, Longhurst DN, Benallie KJ. Methods for addressing publication bias in school psychology journals: A descriptive review of meta-analyses from 1980 to 2019. J Sch Psychol 2021; 84:74-94. [DOI: 10.1016/j.jsp.2020.11.002] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2019] [Revised: 08/20/2020] [Accepted: 11/17/2020] [Indexed: 11/24/2022]
|
13
|
Abstract
Red color supposedly affects cognitive functioning in achievement situations and impairs test performance. Although this has been shown for different cognitive domains in different populations and cultural contexts, recent studies including close replications failed to corroborate this effect. Reported here is a random-effects meta-analysis of 67 effect sizes (38 samples) that compared test performance after viewing red or a control color. For anagram tests and knowledge tests no significant difference between color conditions was found (Cohen's d of -0.06 and -0.04); for reasoning tests the pooled effect of d = -0.34, 95% CI [-0.61, -0.06] indicated significantly lower scores in the red condition. The cumulative meta-analysis revealed substantially larger effects in initial studies as compared to subsequent research. After correcting for publication bias no evidential value for an effect of red color on intellectual performance was available. The review casts doubt on the existence of a robust color-priming effect in achievement situations.
Collapse
Affiliation(s)
- Timo Gnambs
- Leibniz Institute for Educational Trajectories, Wilhelmsplatz 3, 96047, Bamberg, Germany.
- Johannes Kepler University Linz, Linz, Austria.
| |
Collapse
|
14
|
Nuijten MB, van Assen MALM, Augusteijn HEM, Crompvoets EAV, Wicherts JM. Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis. J Intell 2020; 8:E36. [PMID: 33023250 PMCID: PMC7720125 DOI: 10.3390/jintelligence8040036] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2020] [Revised: 09/14/2020] [Accepted: 09/24/2020] [Indexed: 11/23/2022] Open
Abstract
In this meta-study, we analyzed 2442 effect sizes from 131 meta-analyses in intelligence research, published from 1984 to 2014, to estimate the average effect size, median power, and evidence for bias. We found that the average effect size in intelligence research was a Pearson's correlation of 0.26, and the median sample size was 60. Furthermore, across primary studies, we found a median power of 11.9% to detect a small effect, 54.5% to detect a medium effect, and 93.9% to detect a large effect. We documented differences in average effect size and median estimated power between different types of intelligence studies (correlational studies, studies of group differences, experiments, toxicology, and behavior genetics). On average, across all meta-analyses (but not in every meta-analysis), we found evidence for small-study effects, potentially indicating publication bias and overestimated effects. We found no differences in small-study effects between different study types. We also found no convincing evidence for the decline effect, US effect, or citation bias across meta-analyses. We concluded that intelligence research does show signs of low power and publication bias, but that these problems seem less severe than in many other scientific fields.
Collapse
Affiliation(s)
- Michèle B. Nuijten
- Department of Methodology & Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Warandelaan 2, 5037 AB Tilburg, The Netherlands; (M.A.L.M.v.A.); (H.E.M.A.); (E.A.V.C.); (J.M.W.)
| | - Marcel A. L. M. van Assen
- Department of Methodology & Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Warandelaan 2, 5037 AB Tilburg, The Netherlands; (M.A.L.M.v.A.); (H.E.M.A.); (E.A.V.C.); (J.M.W.)
- Section Sociology, Faculty of Social and Behavioral Sciences, Utrecht University, Heidelberglaan 1, 3584 CS Utrecht, The Netherlands
| | - Hilde E. M. Augusteijn
- Department of Methodology & Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Warandelaan 2, 5037 AB Tilburg, The Netherlands; (M.A.L.M.v.A.); (H.E.M.A.); (E.A.V.C.); (J.M.W.)
| | - Elise A. V. Crompvoets
- Department of Methodology & Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Warandelaan 2, 5037 AB Tilburg, The Netherlands; (M.A.L.M.v.A.); (H.E.M.A.); (E.A.V.C.); (J.M.W.)
| | - Jelte M. Wicherts
- Department of Methodology & Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Warandelaan 2, 5037 AB Tilburg, The Netherlands; (M.A.L.M.v.A.); (H.E.M.A.); (E.A.V.C.); (J.M.W.)
| |
Collapse
|
15
|
Chen X, Latham GP, Piccolo RF, Itzchakov G. Advancing Primed Goal Research in Organizational Behavior. APPLIED PSYCHOLOGY-AN INTERNATIONAL REVIEW-PSYCHOLOGIE APPLIQUEE-REVUE INTERNATIONALE 2020. [DOI: 10.1111/apps.12282] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Affiliation(s)
- Xiao Chen
- University of Prince Edward Island Canada
| | | | | | | |
Collapse
|
16
|
Bakker M, Veldkamp CLS, van den Akker OR, van Assen MALM, Crompvoets E, Ong HH, Wicherts JM. Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size. PLoS One 2020; 15:e0236079. [PMID: 32735597 PMCID: PMC7394423 DOI: 10.1371/journal.pone.0236079] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2019] [Accepted: 06/29/2020] [Indexed: 12/02/2022] Open
Abstract
In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of pre-registrations created according to the guidelines for the Center for Open Science Preregistration Challenge (PCRs) and a sample of institutional review board (IRB) proposals from Tilburg School of Behavior and Social Sciences, which both include a recommendation to do a formal power analysis, and (ii) a sample of pre-registrations created according to the guidelines for Open Science Framework Standard Pre-Data Collection Registrations (SPRs) in which no guidance on sample size planning is given. We found that PCRs and IRBs (72%) more often included sample size decisions based on power analyses than the SPRs (45%). However, this did not result in larger planned sample sizes. The determined sample size of the PCRs and IRB proposals (Md = 90.50) was not higher than the determined sample size of the SPRs (Md = 126.00; W = 3389.5, p = 0.936). Typically, power analyses in the registrations were conducted with G*power, assuming a medium effect size, α = .05 and a power of .80. Only 20% of the power analyses contained enough information to fully reproduce the results and only 62% of these power analyses pertained to the main hypothesis test in the pre-registration. Therefore, we see ample room for improvements in the quality of the registrations and we offer several recommendations to do so.
Collapse
Affiliation(s)
- Marjan Bakker
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | | | - Olmo R. van den Akker
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | - Marcel A. L. M. van Assen
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
- Department of Sociology, Utrecht University, Utrecht, The Netherlands
| | - Elise Crompvoets
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
- Cito National Institute for Educational Measurement, Arnhem, The Netherlands
| | - How Hwee Ong
- Department of Social Psychology, Tilburg University, Tilburg, The Netherlands
| | - Jelte M. Wicherts
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| |
Collapse
|
17
|
Boayue NM, Csifcsák G, Aslaksen P, Turi Z, Antal A, Groot J, Hawkins GE, Forstmann B, Opitz A, Thielscher A, Mittner M. Increasing propensity to mind‐wander by transcranial direct current stimulation? A registered report. Eur J Neurosci 2020; 51:755-780. [DOI: 10.1111/ejn.14347] [Citation(s) in RCA: 20] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2018] [Revised: 11/26/2018] [Accepted: 11/30/2018] [Indexed: 12/20/2022]
Affiliation(s)
| | - Gábor Csifcsák
- Department of PsychologyUniversity of Tromsø Tromsø Norway
| | - Per Aslaksen
- Department of PsychologyUniversity of Tromsø Tromsø Norway
| | - Zsolt Turi
- Department of Clinical NeurophysiologyUniversity Medical Center Göttingen Göttingen Germany
| | - Andrea Antal
- Department of Clinical NeurophysiologyUniversity Medical Center Göttingen Göttingen Germany
| | - Josephine Groot
- Department of PsychologyUniversity of Tromsø Tromsø Norway
- Integrative Model‐based Cognitive Neuroscience Research UnitUniversity of Amsterdam Amsterdam The Netherlands
| | - Guy E. Hawkins
- School of PsychologyUniversity of Newcastle Newcastle New South Wales Australia
| | - Birte Forstmann
- Integrative Model‐based Cognitive Neuroscience Research UnitUniversity of Amsterdam Amsterdam The Netherlands
| | - Alexander Opitz
- Department of Biomedical EngineeringUniversity of Minnesota Minneapolis MN
| | - Axel Thielscher
- Danish Research Centre for Magnetic ResonanceCentre for Functional and Diagnostic Imaging and ResearchCopenhagen University Hospital Hvidovre Denmark
- Department of Electrical EngineeringTechnical University of Denmark Lyngby Denmark
| | | |
Collapse
|
18
|
Appel M, Marker C, Gnambs T. Are Social Media Ruining Our Lives? A Review of Meta-Analytic Evidence. REVIEW OF GENERAL PSYCHOLOGY 2019. [DOI: 10.1177/1089268019880891] [Citation(s) in RCA: 57] [Impact Index Per Article: 11.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
Abstract
A growing number of studies have examined the psychological corollaries of using social networking sites (SNSs) such as Facebook, Instagram, or Twitter (often called social media). The interdisciplinary research area and conflicting evidence from primary studies complicate the assessment of current scholarly knowledge in this field of high public attention. We review meta-analytic evidence on three hotly debated topics regarding the effects of SNSs: well-being, academic achievement, and narcissism. Meta-analyses from different laboratories draw a rather equivocal picture. They show small associations in the r = .10 range between the intensity of SNS use and loneliness, self-esteem, life satisfaction, or self-reported depression, and somewhat stronger links to a thin body ideal and higher social capital. There is no indication for potential devastating effects of social media on school achievement; social media use and school grades are unrelated for adolescents. The meta-analyses revealed small to moderate associations between narcissism and SNS use. In sum, meta-analytic evidence is not in support of dramatic claims relating social media use to mischief.
Collapse
Affiliation(s)
| | | | - Timo Gnambs
- Johannes Kepler University Linz, Linz, Austria
- Leibniz Institute for Educational Trajectories, Bamberg, Germany
| |
Collapse
|
19
|
Koricheva J, Kulinskaya E. Temporal Instability of Evidence Base: A Threat to Policy Making? Trends Ecol Evol 2019; 34:895-902. [DOI: 10.1016/j.tree.2019.05.006] [Citation(s) in RCA: 35] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2019] [Revised: 05/12/2019] [Accepted: 05/16/2019] [Indexed: 11/26/2022]
|
20
|
Köbis NC, Verschuere B, Bereby-Meyer Y, Rand D, Shalvi S. Intuitive Honesty Versus Dishonesty: Meta-Analytic Evidence. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 2019; 14:778-796. [PMID: 31291557 DOI: 10.1177/1745691619851778] [Citation(s) in RCA: 41] [Impact Index Per Article: 8.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Is self-serving lying intuitive? Or does honesty come naturally? Many experiments have manipulated reliance on intuition in behavioral-dishonesty tasks, with mixed results. We present two meta-analyses (with evidential value) testing whether an intuitive mind-set affects the proportion of liars (k = 73; n = 12,711) and the magnitude of lying (k = 50; n = 6,473). The results indicate that when dishonesty harms abstract others, promoting intuition causes more people to lie, log odds ratio = 0.38, p = .0004, and people to lie more, Hedges's g = 0.26, p < .0001. However, when dishonesty inflicts harm on concrete others, promoting intuition has no significant effect on dishonesty (p > .63). We propose one potential explanation: The intuitive appeal of prosociality may cancel out the intuitive selfish appeal of dishonesty, suggesting that the social consequences of lying could be a promising key to the riddle of intuition's role in honesty. We discuss limitations such as the relatively unbalanced distribution of studies using concrete versus abstract victims and the overall large interstudy heterogeneity.
Collapse
Affiliation(s)
- Nils C Köbis
- 1 Center for Research in Experimental Economics and Political Decision Making, University of Amsterdam
| | | | | | - David Rand
- 4 Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| | - Shaul Shalvi
- 1 Center for Research in Experimental Economics and Political Decision Making, University of Amsterdam
| |
Collapse
|
21
|
Nuijten MB. Practical tools and strategies for researchers to increase replicability. Dev Med Child Neurol 2019; 61:535-539. [PMID: 30294813 DOI: 10.1111/dmcn.14054] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 08/21/2018] [Indexed: 11/30/2022]
Abstract
A growing body of evidence indicates that the effects reported in many scientific fields may be overestimated or even false. This problem has gained a lot of attention in the field of psychology, where researchers have even started to speak of a 'replication crisis'. Fortunately, a number of measures to rectify this problem have already been proposed and implemented, some inspired by practices in other scientific fields. In this review, I briefly examine this issue in the field of psychology and suggest some practical tools and strategies that researchers can implement to increase replicability and the overall quality of their scientific research. WHAT THIS PAPER ADDS: Researchers can implement many practical tools and strategies to improve replicability of their findings. Strategies include improving statistical inference, pre-registration, multisite collaboration, and sharing data. Different scientific fields could benefit from looking at each other's best practices.
Collapse
Affiliation(s)
- Michèle B Nuijten
- Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg, the Netherlands
| |
Collapse
|
22
|
van Aert RCM, Wicherts JM, van Assen MALM. Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis. PLoS One 2019; 14:e0215052. [PMID: 30978228 PMCID: PMC6461282 DOI: 10.1371/journal.pone.0215052] [Citation(s) in RCA: 112] [Impact Index Per Article: 22.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2018] [Accepted: 03/26/2019] [Indexed: 01/08/2023] Open
Abstract
Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias exists, how strongly it affects different scientific literatures is currently less well-known. We examined evidence of publication bias in a large-scale data set of primary studies that were included in 83 meta-analyses published in Psychological Bulletin (representing meta-analyses from psychology) and 499 systematic reviews from the Cochrane Database of Systematic Reviews (CDSR; representing meta-analyses from medicine). Publication bias was assessed on all homogeneous subsets (3.8% of all subsets of meta-analyses published in Psychological Bulletin) of primary studies included in meta-analyses, because publication bias methods do not have good statistical properties if the true effect size is heterogeneous. Publication bias tests did not reveal evidence for bias in the homogeneous subsets. Overestimation was minimal but statistically significant, providing evidence of publication bias that appeared to be similar in both fields. However, a Monte-Carlo simulation study revealed that the creation of homogeneous subsets resulted in challenging conditions for publication bias methods since the number of effect sizes in a subset was rather small (median number of effect sizes equaled 6). Our findings are in line with, in its most extreme case, publication bias ranging from no bias until only 5% statistically nonsignificant effect sizes being published. These and other findings, in combination with the small percentages of statistically significant primary effect sizes (28.9% and 18.9% for subsets published in Psychological Bulletin and CDSR), led to the conclusion that evidence for publication bias in the studied homogeneous subsets is weak, but suggestive of mild publication bias in both psychology and medicine.
Collapse
Affiliation(s)
- Robbie C. M. van Aert
- Department of Methodology and Statistics, Tilburg University, Tilburg, the Netherlands
| | - Jelte M. Wicherts
- Department of Methodology and Statistics, Tilburg University, Tilburg, the Netherlands
| | - Marcel A. L. M. van Assen
- Department of Methodology and Statistics, Tilburg University, Tilburg, the Netherlands
- Department of Sociology, Utrecht University, Utrecht, the Netherlands
| |
Collapse
|
23
|
Examining reproducibility in psychology: A hybrid method for combining a statistically significant original study and a replication. Behav Res Methods 2019; 50:1515-1539. [PMID: 28936638 PMCID: PMC6096648 DOI: 10.3758/s13428-017-0967-6] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
The unrealistically high rate of positive results within psychology has increased the attention to replication research. However, researchers who conduct a replication and want to statistically combine the results of their replication with a statistically significant original study encounter problems when using traditional meta-analysis techniques. The original study's effect size is most probably overestimated because it is statistically significant, and this bias is not taken into consideration in traditional meta-analysis. We have developed a hybrid method that does take the statistical significance of an original study into account and enables (a) accurate effect size estimation, (b) estimation of a confidence interval, and (c) testing of the null hypothesis of no effect. We analytically approximate the performance of the hybrid method and describe its statistical properties. By applying the hybrid method to data from the Reproducibility Project: Psychology (Open Science Collaboration, 2015), we demonstrate that the conclusions based on the hybrid method are often in line with those of the replication, suggesting that many published psychological studies have smaller effect sizes than those reported in the original study, and that some effects may even be absent. We offer hands-on guidelines for how to statistically combine an original study and replication, and have developed a Web-based application ( https://rvanaert.shinyapps.io/hybrid ) for applying the hybrid method.
Collapse
|
24
|
Abstract
Scientists, for the most part, want to get it right. However, the social structures that govern their work undermine that aim, and this leads to nonreplicable findings in many fields. Because the social structure of science is a decentralized system, it is difficult to intervene. In this article, I discuss how we might do so, focusing on self-corrective-labor schemes (i.e., ways of distributing replication efforts within the scientific community). First, I argue that we need to implement a scheme that makes replication work outcome independent, systematic, and sustainable. Second, I use these three criteria to evaluate extant proposals, which place the responsibility for replication on original researchers, consumers of their research, students, or many labs. Third, on the basis of a philosophical analysis of the reward system of science and the benefits of the division of cognitive labor, I propose a scheme that satisfies the criteria better: the professional scheme. This scheme has two main components. First, the scientific community is organized into two groups: discovery researchers, who produce new findings, and confirmation researchers, whose primary function is to do confirmation work (i.e., replication, reproduction, meta-analysis). Second, a distinct reward system is established for confirmation researchers so that their career advancement is separated from whether they obtain positive experimental results.
Collapse
|
25
|
Pennington CR, Litchfield D, McLatchie N, Heim D. Stereotype threat may not impact women's inhibitory control or mathematical performance: Providing support for the null hypothesis. EUROPEAN JOURNAL OF SOCIAL PSYCHOLOGY 2018. [DOI: 10.1002/ejsp.2540] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Affiliation(s)
| | | | - Neil McLatchie
- Department of Psychology Lancaster University Lancaster UK
| | - Derek Heim
- Department of Psychology Edge Hill University Ormskirk UK
| |
Collapse
|
26
|
Lamberink HJ, Otte WM, Sinke MR, Lakens D, Glasziou PP, Tijdink JK, Vinkers CH. Statistical power of clinical trials increased while effect size remained stable: an empirical analysis of 136,212 clinical trials between 1975 and 2014. J Clin Epidemiol 2018; 102:123-128. [DOI: 10.1016/j.jclinepi.2018.06.014] [Citation(s) in RCA: 20] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2018] [Revised: 06/14/2018] [Accepted: 06/28/2018] [Indexed: 10/28/2022]
|
27
|
Milfont TL, Klein RA. Replication and Reproducibility in Cross-Cultural Psychology. JOURNAL OF CROSS-CULTURAL PSYCHOLOGY 2018. [DOI: 10.1177/0022022117744892] [Citation(s) in RCA: 25] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Replication is the scientific gold standard that enables the confirmation of research findings. Concerns related to publication bias, flexibility in data analysis, and high-profile cases of academic misconduct have led to recent calls for more replication and systematic accumulation of scientific knowledge in psychological science. This renewed emphasis on replication may pose specific challenges to cross-cultural research due to inherent practical difficulties in emulating an original study in other cultural groups. The purpose of the present article is to discuss how the core concepts of this replication debate apply to cross-cultural psychology. Distinct to replications in cross-cultural research are examinations of bias and equivalence in manipulations and procedures, and that targeted research populations may differ in meaningful ways. We identify issues in current psychological research (analytic flexibility, low power) and possible solutions (preregistration, power analysis), and discuss ways to implement best practices in cross-cultural replication attempts.
Collapse
Affiliation(s)
- Taciano L. Milfont
- Centre for Applied Cross-Cultural Research, Victoria University of Wellington, New Zealand
| | | |
Collapse
|
28
|
Alinaghi N, Reed WR. Meta-analysis and publication bias: How well does the FAT-PET-PEESE procedure work? Res Synth Methods 2018. [DOI: 10.1002/jrsm.1298] [Citation(s) in RCA: 24] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Affiliation(s)
- Nazila Alinaghi
- Department of Economics and Finance; University of Canterbury; Christchurch New Zealand
| | - W. Robert Reed
- Department of Economics and Finance; University of Canterbury; Christchurch New Zealand
| |
Collapse
|
29
|
Crutzen R, Peters GJY. Targeting Next Generations to Change the Common Practice of Underpowered Research. Front Psychol 2017; 8:1184. [PMID: 28751873 PMCID: PMC5507958 DOI: 10.3389/fpsyg.2017.01184] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2017] [Accepted: 06/28/2017] [Indexed: 12/04/2022] Open
Affiliation(s)
- Rik Crutzen
- Department of Health Promotion, Care and Public Health Research Institute (CAPHRI), Maastricht UniversityMaastricht, Netherlands
| | - Gjalt-Jorn Y. Peters
- Faculty of Psychology and Education Science, Open University of the NetherlandsHeerlen, Netherlands
- Department of Work and Social Psychology, Faculty of Psychology and Neuroscience, Maastricht UniversityMaastricht, Netherlands
| |
Collapse
|
30
|
Szucs D, Ioannidis JPA. Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biol 2017; 15:e2000797. [PMID: 28253258 PMCID: PMC5333800 DOI: 10.1371/journal.pbio.2000797] [Citation(s) in RCA: 346] [Impact Index Per Article: 49.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2016] [Accepted: 02/06/2017] [Indexed: 11/19/2022] Open
Abstract
We have empirically assessed the distribution of published effect sizes and estimated power by analyzing 26,841 statistical records from 3,801 cognitive neuroscience and psychology papers published recently. The reported median effect size was D = 0.93 (interquartile range: 0.64-1.46) for nominally statistically significant results and D = 0.24 (0.11-0.42) for nonsignificant results. Median power to detect small, medium, and large effects was 0.12, 0.44, and 0.73, reflecting no improvement through the past half-century. This is so because sample sizes have remained small. Assuming similar true effect sizes in both disciplines, power was lower in cognitive neuroscience than in psychology. Journal impact factors negatively correlated with power. Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature. In light of our findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience.
Collapse
Affiliation(s)
- Denes Szucs
- Department of Psychology, University of Cambridge, Cambridge, United Kingdom
| | - John P. A. Ioannidis
- Meta-Research Innovation Center at Stanford (METRICS) and Department of Medicine, Department of Health Research and Policy, and Department of Statistics, Stanford University, Stanford, California, United States of America
| |
Collapse
|
31
|
van Witteloostuijn A. What happened to Popperian falsification? Publishing neutral and negative findings. CROSS CULTURAL & STRATEGIC MANAGEMENT 2016. [DOI: 10.1108/ccsm-03-2016-0084] [Citation(s) in RCA: 46] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
|
32
|
|
33
|
Nilsonne G, Tamm S, Månsson KNT, Åkerstedt T, Lekander M. Leukocyte telomere length and hippocampus volume: a meta-analysis. F1000Res 2015; 4:1073. [PMID: 26674112 PMCID: PMC4670011 DOI: 10.12688/f1000research.7198.1] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 10/08/2015] [Indexed: 12/28/2022] Open
Abstract
Leukocyte telomere length has been shown to correlate to hippocampus volume, but effect estimates differ in magnitude and are not uniformly positive. This study aimed primarily to investigate the relationship between leukocyte telomere length and hippocampus gray matter volume by meta-analysis and secondarily to investigate possible effect moderators. Five studies were included with a total of 2107 participants, of which 1960 were contributed by one single influential study. A random-effects meta-analysis estimated the effect to r = 0.12 [95% CI -0.13, 0.37] in the presence of heterogeneity and a subjectively estimated moderate to high risk of bias. There was no evidence that apolipoprotein E (APOE) genotype was an effect moderator, nor that the ratio of leukocyte telomerase activity to telomere length was a better predictor than leukocyte telomere length for hippocampus volume. This meta-analysis, while not proving a positive relationship, also is not able to disprove the earlier finding of a positive correlation in the one large study included in analyses. We propose that a relationship between leukocyte telomere length and hippocamus volume may be mediated by transmigrating monocytes which differentiate into microglia in the brain parenchyma.
Collapse
Affiliation(s)
- Gustav Nilsonne
- Stress Research Institute, Stockholm University, Stockholm, Sweden
- Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden
| | - Sandra Tamm
- Stress Research Institute, Stockholm University, Stockholm, Sweden
- Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden
| | - Kristoffer N. T. Månsson
- Department of Behavioural Sciences and Learning, Linköping University, Linköping, Sweden
- PRIMA Psychiatry, Stockholm, Sweden
| | - Torbjörn Åkerstedt
- Stress Research Institute, Stockholm University, Stockholm, Sweden
- Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden
| | - Mats Lekander
- Stress Research Institute, Stockholm University, Stockholm, Sweden
- Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden
| |
Collapse
|