1
|
Schneider JW, Allum N, Andersen JP, Petersen MB, Madsen EB, Mejlgaard N, Zachariae R. Is something rotten in the state of Denmark? Cross-national evidence for widespread involvement but not systematic use of questionable research practices across all fields of research. PLoS One 2024; 19:e0304342. [PMID: 39133711 PMCID: PMC11318862 DOI: 10.1371/journal.pone.0304342] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Accepted: 05/10/2024] [Indexed: 08/15/2024] Open
Abstract
Questionable research practices (QRP) are believed to be widespread, but empirical assessments are generally restricted to a few types of practices. Furthermore, conceptual confusion is rife with use and prevalence of QRPs often being confused as the same quantity. We present the hitherto most comprehensive study examining QRPs across scholarly fields and knowledge production modes. We survey perception, use, prevalence and predictors of QRPs among 3,402 researchers in Denmark and 1,307 in the UK, USA, Croatia and Austria. Results reveal remarkably similar response patterns among Danish and international respondents (τ = 0.85). Self-reported use indicates whether respondents have used a QRP in recent publications. 9 out of 10 respondents admitted using at least one QRP. Median use is three out of nine QRP items. Self-reported prevalence reflects the frequency of use. On average, prevalence rates were roughly three times lower compared to self-reported use. Findings indicated that the perceived social acceptability of QRPs influenced self-report patterns. Results suggest that most researchers use different types of QRPs within a restricted time period. The prevalence estimates, however, do not suggest outright systematic use of specific QRPs. Perceived pressure was the strongest systemic predictor for prevalence. Conversely, more local attention to research cultures and academic age was negatively related to prevalence. Finally, the personality traits conscientiousness and, to a lesser degree, agreeableness were also inversely associated with self-reported prevalence. Findings suggest that explanations for engagement with QRPs are not only attributable to systemic factors, as hitherto suggested, but a complicated mixture of experience, systemic and individual factors, and motivated reasoning.
Collapse
Affiliation(s)
- Jesper W. Schneider
- Danish Centre for Studies in Research and Research Policy, Aarhus University, Aarhus, Denmark
| | - Nick Allum
- Department of Sociology, University of Essex, Essex, United Kingdom
| | - Jens Peter Andersen
- Danish Centre for Studies in Research and Research Policy, Aarhus University, Aarhus, Denmark
| | | | - Emil B. Madsen
- Danish Centre for Studies in Research and Research Policy, Aarhus University, Aarhus, Denmark
| | - Niels Mejlgaard
- Danish Centre for Studies in Research and Research Policy, Aarhus University, Aarhus, Denmark
| | - Robert Zachariae
- Unit for Psychooncology and Health Psychology (EPoS), Department of Oncology, Aarhus University Hospital, Aarhus, Denmark
- Department Psychology and Behavioral Science, Aarhus University, Aarhus, Denmark
| |
Collapse
|
2
|
Coles NA, Tenney ER, Chin JM, Friedrich JC, O'Dea RE, Holcombe AO. Team scientists should normalize disagreement. Science 2024; 384:1076-1077. [PMID: 38843342 DOI: 10.1126/science.ado7070] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/15/2024]
Affiliation(s)
- Nicholas A Coles
- Center for the Study of Language and Information, Stanford University, Stanford, CA, USA
| | - Elizabeth R Tenney
- David Eccles School of Business, University of Utah, Salt Lake City, UT, USA
| | - Jason M Chin
- College of Law, Australian National University, Canberra, ACT, Australia
| | - Jack C Friedrich
- Department of Psychology, Saint Louis University, St. Louis, MO, USA
| | - Rose E O'Dea
- School of Agriculture, Food and Ecosystem Sciences, University of Melbourne, Melbourne, VIC, Australia
| | - Alex O Holcombe
- School of Psychology, University of Sydney, Sydney, NSW, Australia
| |
Collapse
|
3
|
Kimmel K, Avolio ML, Ferraro PJ. Empirical evidence of widespread exaggeration bias and selective reporting in ecology. Nat Ecol Evol 2023; 7:1525-1536. [PMID: 37537387 DOI: 10.1038/s41559-023-02144-3] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2022] [Accepted: 06/28/2023] [Indexed: 08/05/2023]
Abstract
In many scientific disciplines, common research practices have led to unreliable and exaggerated evidence about scientific phenomena. Here we describe some of these practices and quantify their pervasiveness in recent ecology publications in five popular journals. In an analysis of over 350 studies published between 2018 and 2020, we detect empirical evidence of exaggeration bias and selective reporting of statistically significant results. This evidence implies that the published effect sizes in ecology journals exaggerate the importance of the ecological relationships that they aim to quantify. An exaggerated evidence base hinders the ability of empirical ecology to reliably contribute to science, policy, and management. To increase the credibility of ecology research, we describe a set of actions that ecologists should take, including changes to scientific norms about what high-quality ecology looks like and expectations about what high-quality studies can deliver.
Collapse
Affiliation(s)
- Kaitlin Kimmel
- Mad Agriculture, Boulder, CO, USA
- Department of Earth and Planetary Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Meghan L Avolio
- Department of Earth and Planetary Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Paul J Ferraro
- Carey Business School, Johns Hopkins University, Baltimore, MD, USA.
- Department of Environmental Health and Engineering, a joint department of the Bloomberg School of Public Health and the Whiting School of Engineering, Johns Hopkins University, Baltimore, MD, USA.
| |
Collapse
|
4
|
Wulff JN, Sajons GB, Pogrebna G, Lonati S, Bastardoz N, Banks GC, Antonakis J. Common methodological mistakes. THE LEADERSHIP QUARTERLY 2023. [DOI: 10.1016/j.leaqua.2023.101677] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/23/2023]
|
5
|
Naaman K, Grant S, Kianersi S, Supplee L, Henschel B, Mayo-Wilson E. Exploring enablers and barriers to implementing the Transparency and Openness Promotion Guidelines: a theory-based survey of journal editors. ROYAL SOCIETY OPEN SCIENCE 2023; 10:221093. [PMID: 36756061 PMCID: PMC9890101 DOI: 10.1098/rsos.221093] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/24/2022] [Accepted: 01/09/2023] [Indexed: 05/27/2023]
Abstract
The Transparency and Openness Promotion (TOP) Guidelines provide a framework to help journals develop open science policies. Theories of behaviour change can guide understanding of why journals do (not) implement open science policies and the development of interventions to improve these policies. In this study, we used the Theoretical Domains Framework to survey 88 journal editors on their capability, opportunity and motivation to implement TOP. Likert-scale questions assessed editor support for TOP, and enablers and barriers to implementing TOP. A qualitative question asked editors to provide reflections on their ratings. Most participating editors supported adopting TOP at their journal (71%) and perceived other editors in their discipline to support adopting TOP (57%). Most editors (93%) agreed their roles include maintaining policies that reflect current best practices. However, most editors (74%) did not see implementing TOP as a high priority compared with other editorial responsibilities. Qualitative responses expressed structural barriers to implementing TOP (e.g. lack of time, resources and authority to implement changes) and varying support for TOP depending on study type, open science standard, and level of implementation. We discuss how these findings could inform the development of theoretically guided interventions to increase open science policies, procedures and practices.
Collapse
Affiliation(s)
- Kevin Naaman
- School of Public Health, Indiana University-Bloomington, Bloomington, IN, USA
- School of Education, Indiana University-Bloomington, Bloomington, IN, USA
| | - Sean Grant
- HEDCO Institute for Evidence-Based Educational Practice, University of Oregon, Eugene, OR, USA
- Richard M. Fairbanks School of Public Health, Indiana University-Indianapolis, Indianapolis, IN, USA
| | - Sina Kianersi
- School of Public Health, Indiana University-Bloomington, Bloomington, IN, USA
- Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA
| | | | - Beate Henschel
- School of Public Health, Indiana University-Bloomington, Bloomington, IN, USA
| | - Evan Mayo-Wilson
- School of Public Health, Indiana University-Bloomington, Bloomington, IN, USA
- Gillings School of Global Public Health, University of North Carolina-Chapel Hill, Chapel Hill, NC, USA
| |
Collapse
|
6
|
Greene D, Palmer MJ, Relman DA. Motivating Proactive Biorisk Management. Health Secur 2023; 21:46-60. [PMID: 36633603 PMCID: PMC9940813 DOI: 10.1089/hs.2022.0101] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2022] [Revised: 10/29/2022] [Accepted: 11/03/2022] [Indexed: 01/13/2023] Open
Abstract
Scholars and practitioners of biosafety and biosecurity (collectively, biorisk management or BRM) have argued that life scientists should play a more proactive role in monitoring their work for potential risks, mitigating harm, and seeking help as necessary. However, most efforts to promote proactive BRM have focused on training life scientists in technical skills and have largely ignored the extent to which life scientists wish to use them (ie, their motivation). In this article, we argue that efforts to promote proactive BRM would benefit from a greater focus on life scientists' motivation. We review relevant literature on life scientists' motivation to practice BRM, offer examples of successful interventions from adjacent fields, and outline ideas for possible interventions to promote proactive BRM, along with strategies for iterative development, testing, and scaling.
Collapse
Affiliation(s)
- Daniel Greene
- Daniel Greene, PhD, is a Senior Analyst, Biosafety, Biosecurity & Emerging Technologies, Gryphon Scientific, Takoma Park, MD. Daniel Greene is also a Postdoctoral Fellow, Center for International Security and Cooperation; Stanford University, Stanford, CA
| | - Megan J. Palmer
- Megan J. Palmer, PhD, is Executive Director, Bio Policy & Leadership Initiatives, and an Adjunct Professor, Department of Bioengineering; Stanford University, Stanford, CA
| | - David A. Relman
- David A. Relman, MD, is a Senior Fellow, Center for International Security and Cooperation; Stanford University, Stanford, CA. David A. Relman is also the Thomas C. and Joan M. Merigan Professor, Departments of Medicine, and of Microbiology and Immunology, Stanford University School of Medicine, Stanford, CA
| |
Collapse
|
7
|
Adoption of Transparency and Openness Promotion (TOP) Guidelines across Journals. PUBLICATIONS 2022. [DOI: 10.3390/publications10040046] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022] Open
Abstract
Journal policies continuously evolve to enable knowledge sharing and support reproducible science. However, that change happens within a certain framework. Eight modular standards with three levels of increasing stringency make Transparency and Openness Promotion (TOP) guidelines which can be used to evaluate to what extent and with which stringency journals promote open science. Guidelines define standards for data citation, transparency of data, material, code and design and analysis, replication, plan and study pre-registration, and two effective interventions: “Registered reports” and “Open science badges”, and levels of adoption summed up across standards define journal’s TOP Factor. In this paper, we analysed the status of adoption of TOP guidelines across two thousand journals reported in the TOP Factor metrics. We show that the majority of the journals’ policies align with at least one of the TOP’s standards, most likely “Data citation” (70%) followed by “Data transparency” (19%). Two-thirds of adoptions of TOP standard are of the stringency Level 1 (less stringent), whereas only 9% is of the stringency Level 3. Adoption of TOP standards differs across science disciplines and multidisciplinary journals (N = 1505) and journals from social sciences (N = 1077) show the greatest number of adoptions. Improvement of the measures that journals take to implement open science practices could be done: (1) discipline-specific, (2) journals that have not yet adopted TOP guidelines could do so, (3) the stringency of adoptions could be increased.
Collapse
|
8
|
Rahal RM, Hamann H, Brohmer H, Pethig F. Sharing the Recipe: Reproducibility and Replicability in Research Across Disciplines. RESEARCH IDEAS AND OUTCOMES 2022. [DOI: 10.3897/rio.8.e89980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022] Open
Abstract
The open and transparent documentation of scientific processes has been established as a core antecedent of free knowledge. This also holds for generating robust insights in the scope of research projects. To convince academic peers and the public, the research process must be understandable and retraceable (reproducible), and repeatable (replicable) by others, precluding the inclusion of fluke findings into the canon of insights. In this contribution, we outline what reproducibility and replicability (R&R) could mean in the scope of different disciplines and traditions of research and which significance R&R has for generating insights in these fields. We draw on projects conducted in the scope of the Wikimedia "Open Science Fellows Program" (Fellowship Freies Wissen), an interdisciplinary, long-running funding scheme for projects contributing to open research practices. We identify twelve implemented projects from different disciplines which primarily focused on R&R, and multiple additional projects also touching on R&R. From these projects, we identify patterns and synthesize them into a roadmap of how research projects can achieve R&R across different disciplines. We further outline the ground covered by these projects and propose ways forward.
Collapse
|
9
|
Mayo-Wilson E, Grant S, Supplee LH. Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations. PREVENTION SCIENCE : THE OFFICIAL JOURNAL OF THE SOCIETY FOR PREVENTION RESEARCH 2022. [PMID: 34357509 DOI: 10.1007/s11121-021-01284] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/02/2023]
Abstract
Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are "evidence-based," clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as "evidence-based." In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions-an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed "TOP Guidelines for Clearinghouses" includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving "evidence-based" designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.
Collapse
Affiliation(s)
- Evan Mayo-Wilson
- Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, 1025 E. 7th Street, Bloomington, IN, 47405, USA. .,Department of Social & Behavioral Sciences, Fairbanks School of Public Health, Indiana University Richard M, Indianapolis, IN, USA.
| | - Sean Grant
- Department of Social & Behavioral Sciences, Fairbanks School of Public Health, Indiana University Richard M, Indianapolis, IN, USA
| | | |
Collapse
|
10
|
Turoman N, Hautekiet C, Jeanneret S, Valentini B, Langerock N. Open and reproducible practices in developmental psychology research: The workflow of the
WomCogDev
lab as an example. INFANT AND CHILD DEVELOPMENT 2022. [DOI: 10.1002/icd.2333] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Affiliation(s)
- Nora Turoman
- Faculty of Psychology and Educational Sciences University of Geneva Geneva Switzerland
| | - Caro Hautekiet
- Faculty of Psychology and Educational Sciences University of Geneva Geneva Switzerland
| | - Stéphanie Jeanneret
- Faculty of Psychology and Educational Sciences University of Geneva Geneva Switzerland
| | - Beatrice Valentini
- Faculty of Psychology and Educational Sciences University of Geneva Geneva Switzerland
| | - Naomi Langerock
- Faculty of Psychology and Educational Sciences University of Geneva Geneva Switzerland
| |
Collapse
|
11
|
Gopalakrishna G, Wicherts JM, Vink G, Stoop I, van den Akker OR, ter Riet G, Bouter LM. Prevalence of responsible research practices among academics in The Netherlands. F1000Res 2022; 11:471. [PMID: 36128558 PMCID: PMC9478502 DOI: 10.12688/f1000research.110664.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 04/20/2022] [Indexed: 09/05/2024] Open
Abstract
Background: Traditionally, research integrity studies have focused on research misbehaviors and their explanations. Over time, attention has shifted towards preventing questionable research practices and promoting responsible ones. However, data on the prevalence of responsible research practices, especially open methods, open codes and open data and their underlying associative factors, remains scarce. Methods: We conducted a web-based anonymized questionnaire, targeting all academic researchers working at or affiliated to a university or university medical center in The Netherlands, to investigate the prevalence and potential explanatory factors of 11 responsible research practices. Results: A total of 6,813 academics completed the survey, the results of which show that prevalence of responsible practices differs substantially across disciplines and ranks, with 99 percent avoiding plagiarism in their work but less than 50 percent pre-registering a research protocol. Arts and humanities scholars as well as PhD candidates and junior researchers engaged less often in responsible research practices. Publication pressure negatively affected responsible practices, while mentoring, scientific norms subscription and funding pressure stimulated them. Conclusions: Understanding the prevalence of responsible research practices across disciplines and ranks, as well as their associated explanatory factors, can help to systematically address disciplinary- and academic rank-specific obstacles, and thereby facilitate responsible conduct of research.
Collapse
Affiliation(s)
- Gowri Gopalakrishna
- Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, The Netherlands
| | - Jelte M. Wicherts
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | - Gerko Vink
- Department of Methodology and Statistics, Utrecht University, Utrecht, The Netherlands
| | - Ineke Stoop
- The Netherlands Institute for Social Research, Den Haag, The Netherlands
| | - Olmo R. van den Akker
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | - Gerben ter Riet
- Center of Expertise Urban Vitality, Faculty of Health, Amsterdam University of Applied Science, Amsterdam, The Netherlands
| | - Lex M. Bouter
- Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, The Netherlands
- Department of Philosophy, Faculty of Humanities, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| |
Collapse
|
12
|
Gopalakrishna G, Wicherts JM, Vink G, Stoop I, van den Akker OR, ter Riet G, Bouter LM. Prevalence of responsible research practices among academics in The Netherlands. F1000Res 2022; 11:471. [PMID: 36128558 PMCID: PMC9478502 DOI: 10.12688/f1000research.110664.2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 07/28/2022] [Indexed: 11/20/2022] Open
Abstract
Background: Traditionally, research integrity studies have focused on research misbehaviors and their explanations. Over time, attention has shifted towards preventing questionable research practices and promoting responsible ones. However, data on the prevalence of responsible research practices, especially open methods, open codes and open data and their underlying associative factors, remains scarce. Methods: We conducted a web-based anonymized questionnaire, targeting all academic researchers working at or affiliated to a university or university medical center in The Netherlands, to investigate the prevalence and potential explanatory factors of 11 responsible research practices. Results: A total of 6,813 academics completed the survey, the results of which show that prevalence of responsible practices differs substantially across disciplines and ranks, with 99 percent avoiding plagiarism in their work but less than 50 percent pre-registering a research protocol. Arts and humanities scholars as well as PhD candidates and junior researchers engaged less often in responsible research practices. Publication pressure negatively affected responsible practices, while mentoring, scientific norms subscription and funding pressure stimulated them. Conclusions: Understanding the prevalence of responsible research practices across disciplines and ranks, as well as their associated explanatory factors, can help to systematically address disciplinary- and academic rank-specific obstacles, and thereby facilitate responsible conduct of research.
Collapse
Affiliation(s)
- Gowri Gopalakrishna
- Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, The Netherlands
| | - Jelte M. Wicherts
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | - Gerko Vink
- Department of Methodology and Statistics, Utrecht University, Utrecht, The Netherlands
| | - Ineke Stoop
- The Netherlands Institute for Social Research, Den Haag, The Netherlands
| | - Olmo R. van den Akker
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | - Gerben ter Riet
- Center of Expertise Urban Vitality, Faculty of Health, Amsterdam University of Applied Science, Amsterdam, The Netherlands
| | - Lex M. Bouter
- Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, The Netherlands
- Department of Philosophy, Faculty of Humanities, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| |
Collapse
|
13
|
Gopalakrishna G, ter Riet G, Vink G, Stoop I, Wicherts JM, Bouter LM. Prevalence of questionable research practices, research misconduct and their potential explanatory factors: A survey among academic researchers in The Netherlands. PLoS One 2022; 17:e0263023. [PMID: 35171921 PMCID: PMC8849616 DOI: 10.1371/journal.pone.0263023] [Citation(s) in RCA: 59] [Impact Index Per Article: 29.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2021] [Accepted: 01/10/2022] [Indexed: 11/19/2022] Open
Abstract
Prevalence of research misconduct, questionable research practices (QRPs) and their associations with a range of explanatory factors has not been studied sufficiently among academic researchers. The National Survey on Research Integrity targeted all disciplinary fields and academic ranks in the Netherlands. It included questions about engagement in fabrication, falsification and 11 QRPs over the previous three years, and 12 explanatory factor scales. We ensured strict identity protection and used the randomized response method for questions on research misconduct. 6,813 respondents completed the survey. Prevalence of fabrication was 4.3% (95% CI: 2.9, 5.7) and of falsification 4.2% (95% CI: 2.8, 5.6). Prevalence of QRPs ranged from 0.6% (95% CI: 0.5, 0.9) to 17.5% (95% CI: 16.4, 18.7) with 51.3% (95% CI: 50.1, 52.5) of respondents engaging frequently in at least one QRP. Being a PhD candidate or junior researcher increased the odds of frequently engaging in at least one QRP, as did being male. Scientific norm subscription (odds ratio (OR) 0.79; 95% CI: 0.63, 1.00) and perceived likelihood of detection by reviewers (OR 0.62, 95% CI: 0.44, 0.88) were associated with engaging in less research misconduct. Publication pressure was associated with more often engaging in one or more QRPs frequently (OR 1.22, 95% CI: 1.14, 1.30). We found higher prevalence of misconduct than earlier surveys. Our results suggest that greater emphasis on scientific norm subscription, strengthening reviewers in their role as gatekeepers of research quality and curbing the "publish or perish" incentive system promotes research integrity.
Collapse
Affiliation(s)
- Gowri Gopalakrishna
- Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, The Netherlands
- * E-mail:
| | - Gerben ter Riet
- Faculty of Health, Center of Expertise Urban Vitality Amsterdam University of Applied Science, Amsterdam, The Netherlands
| | - Gerko Vink
- Department of Methodology & Statistics, Utrecht University, Utrecht, The Netherlands
| | - Ineke Stoop
- The Netherlands Institute for Social Research, Den Haag, The Netherlands
| | - Jelte M. Wicherts
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | - Lex M. Bouter
- Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, The Netherlands
- Department of Philosophy, Faculty of Humanities, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| |
Collapse
|
14
|
Grant S, Wendt KE, Leadbeater BJ, Supplee LH, Mayo-Wilson E, Gardner F, Bradshaw CP. Transparent, Open, and Reproducible Prevention Science. PREVENTION SCIENCE : THE OFFICIAL JOURNAL OF THE SOCIETY FOR PREVENTION RESEARCH 2022; 23:701-722. [PMID: 35175501 PMCID: PMC9283153 DOI: 10.1007/s11121-022-01336-w] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/05/2022] [Indexed: 01/20/2023]
Abstract
The field of prevention science aims to understand societal problems, identify effective interventions, and translate scientific evidence into policy and practice. There is growing interest among prevention scientists in the potential for transparency, openness, and reproducibility to facilitate this mission by providing opportunities to align scientific practice with scientific ideals, accelerate scientific discovery, and broaden access to scientific knowledge. The overarching goal of this manuscript is to serve as a primer introducing and providing an overview of open science for prevention researchers. In this paper, we discuss factors motivating interest in transparency and reproducibility, research practices associated with open science, and stakeholders engaged in and impacted by open science reform efforts. In addition, we discuss how and why different types of prevention research could incorporate open science practices, as well as ways that prevention science tools and methods could be leveraged to advance the wider open science movement. To promote further discussion, we conclude with potential reservations and challenges for the field of prevention science to address as it transitions to greater transparency, openness, and reproducibility. Throughout, we identify activities that aim to strengthen the reliability and efficiency of prevention science, facilitate access to its products and outputs, and promote collaborative and inclusive participation in research activities. By embracing principles of transparency, openness, and reproducibility, prevention science can better achieve its mission to advance evidence-based solutions to promote individual and collective well-being.
Collapse
Affiliation(s)
- Sean Grant
- Department of Social & Behavioral Sciences, Fairbanks School of Public Health, Indiana University Richard M, 1050 Wishard Blvd, Indianapolis, IN, 46202, USA.
| | - Kathleen E Wendt
- Department of Human Development and Family Studies, Colorado State University, Fort Collins, CO, USA
| | | | | | - Evan Mayo-Wilson
- Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, IN, USA
| | - Frances Gardner
- Department of Social Policy and Intervention, University of Oxford, Oxford, UK
| | - Catherine P Bradshaw
- School of Education & Human Development, University of Virginia, Charlottesville, VA, USA
| |
Collapse
|
15
|
Miguel Ibaviosa C, Chin JM. Beyond CSI: Calibrating public beliefs about the reliability of forensic science through openness and transparency. Sci Justice 2022; 62:272-283. [DOI: 10.1016/j.scijus.2022.02.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/15/2021] [Revised: 01/15/2022] [Accepted: 02/13/2022] [Indexed: 11/26/2022]
|
16
|
Errington TM, Denis A, Perfito N, Iorns E, Nosek BA. Challenges for assessing replicability in preclinical cancer biology. eLife 2021. [DOI: 10.10.7554/elife.67995] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022] Open
Abstract
We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication – it is hard to assess whether reported findings are credible.
Collapse
|
17
|
Errington TM, Denis A, Perfito N, Iorns E, Nosek BA. Challenges for assessing replicability in preclinical cancer biology. eLife 2021; 10:67995. [PMID: 34874008 PMCID: PMC8651289 DOI: 10.7554/elife.67995] [Citation(s) in RCA: 88] [Impact Index Per Article: 29.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2021] [Accepted: 07/20/2021] [Indexed: 02/07/2023] Open
Abstract
We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication - it is hard to assess whether reported findings are credible.
Collapse
Affiliation(s)
| | | | | | | | - Brian A Nosek
- Center for Open Science, Charlottesville, United States.,University of Virginia, Charlottesville, United States
| |
Collapse
|
18
|
Hoekstra R, Vazire S. Aspiring to greater intellectual humility in science. Nat Hum Behav 2021; 5:1602-1607. [PMID: 34711978 DOI: 10.1038/s41562-021-01203-8] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2020] [Accepted: 08/18/2021] [Indexed: 11/09/2022]
Abstract
The replication crisis in the social, behavioural and life sciences has spurred a reform movement aimed at increasing the credibility of scientific studies. Many of these credibility-enhancing reforms focus, appropriately, on specific research and publication practices. A less often mentioned aspect of credibility is the need for intellectual humility or being transparent about and owning the limitations of our work. Although intellectual humility is presented as a widely accepted scientific norm, we argue that current research practice does not incentivize intellectual humility. We provide a set of recommendations on how to increase intellectual humility in research articles and highlight the central role peer reviewers can play in incentivizing authors to foreground the flaws and uncertainty in their work, thus enabling full and transparent evaluation of the validity of research.
Collapse
Affiliation(s)
- Rink Hoekstra
- University of Groningen, Groningen, The Netherlands.
| | - Simine Vazire
- University of California, Davis, Davis, CA, USA.,University of Melbourne, Melbourne, Victoria, Australia
| |
Collapse
|
19
|
Cashin AG, Bagg MK, Richards GC, Toomey E, McAuley JH, Lee H. Limited engagement with transparent and open science standards in the policies of pain journals: a cross-sectional evaluation. BMJ Evid Based Med 2021; 26:313-319. [PMID: 31980469 DOI: 10.1136/bmjebm-2019-111296] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 01/06/2020] [Indexed: 11/04/2022]
Abstract
Scientific progress requires transparency and openness. The ability to critique, replicate and implement scientific findings depends on the transparency of the study design and methods, and the open availability of study materials, data and code. Journals are key stakeholders in supporting transparency and openness. This study aimed to evaluate 10 highest ranked pain journals' authorship policies with respect to their support for transparent and open research practices. Two independent authors evaluated the journal policies (as at 27 May 2019) using three tools: the self-developed Transparency and Openness Evaluation Tool, the Centre for Open Science (COS) Transparency Factor and the International Committee of Medical Journal Editors (ICMJE) requirements for disclosure of conflicts of interest. We found that the journal policies had an overall low level of engagement with research transparency and openness standards. The median COS Transparency Factor score was 3.5 (IQR 2.8) of 29 possible points, and only 7 of 10 journals' stated requirements for disclosure of conflicts of interest aligned fully with the ICMJE recommendations. Improved transparency and openness of pain research has the potential to benefit all that are involved in generating and using research findings. Journal policies that endorse and facilitate transparent and open research practices will ultimately improve the evidence base that informs the care provided for people with pain.
Collapse
Affiliation(s)
- Aidan G Cashin
- Prince of Wales Clinical School, University of New South Wales Faculty of Medicine, Randwick, New South Wales, Australia
- Centre for Pain IMPACT, Neuroscience Research Australia, Randwick, New South Wales, Australia
| | - Matthew K Bagg
- Prince of Wales Clinical School, University of New South Wales Faculty of Medicine, Randwick, New South Wales, Australia
- Centre for Pain IMPACT, Neuroscience Research Australia, Randwick, New South Wales, Australia
| | - Georgia C Richards
- Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, Oxfordshire, UK
| | - Elaine Toomey
- Health Behaviour Change Research Group, School of Psychology, National University of Ireland Galway, Galway, Ireland
| | - James H McAuley
- Centre for Pain IMPACT, Neuroscience Research Australia, Randwick, New South Wales, Australia
- School of Medical Sciences, Faculty of Medicine, University of New South Wales, Randwick, New South Wales, Australia
| | - Hopin Lee
- Centre for Statistics in Medicine & Rehabilitation Research in Oxford, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences (NDORMS), University of Oxford, Oxford, Oxfordshire, UK
- School of Medicine and Public Health, The University of Newcastle, Callaghan, New South Wales, Australia
| |
Collapse
|
20
|
Seven steps toward more transparency in statistical practice. Nat Hum Behav 2021; 5:1473-1480. [PMID: 34764461 DOI: 10.1038/s41562-021-01211-8] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2021] [Accepted: 09/06/2021] [Indexed: 01/03/2023]
Abstract
We argue that statistical practice in the social and behavioural sciences benefits from transparency, a fair acknowledgement of uncertainty and openness to alternative interpretations. Here, to promote such a practice, we recommend seven concrete statistical procedures: (1) visualizing data; (2) quantifying inferential uncertainty; (3) assessing data preprocessing choices; (4) reporting multiple models; (5) involving multiple analysts; (6) interpreting results modestly; and (7) sharing data and code. We discuss their benefits and limitations, and provide guidelines for adoption. Each of the seven procedures finds inspiration in Merton's ethos of science as reflected in the norms of communalism, universalism, disinterestedness and organized scepticism. We believe that these ethical considerations-as well as their statistical consequences-establish common ground among data analysts, despite continuing disagreements about the foundations of statistical inference.
Collapse
|
21
|
Nosek BA, Hardwicke TE, Moshontz H, Allard A, Corker KS, Dreber A, Fidler F, Hilgard J, Struhl MK, Nuijten MB, Rohrer JM, Romero F, Scheel AM, Scherer LD, Schönbrodt FD, Vazire S. Replicability, Robustness, and Reproducibility in Psychological Science. Annu Rev Psychol 2021; 73:719-748. [PMID: 34665669 DOI: 10.1146/annurev-psych-020821-114157] [Citation(s) in RCA: 118] [Impact Index Per Article: 39.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Replication-an important, uncommon, and misunderstood practice-is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
Collapse
Affiliation(s)
- Brian A Nosek
- Department of Psychology, University of Virginia, Charlottesville, Virginia 22904, USA; .,Center for Open Science, Charlottesville, Virginia 22903, USA
| | - Tom E Hardwicke
- Department of Psychology, University of Amsterdam, 1012 ZA Amsterdam, The Netherlands
| | - Hannah Moshontz
- Addiction Research Center, University of Wisconsin-Madison, Madison, Wisconsin 53706, USA
| | - Aurélien Allard
- Department of Psychology, University of California, Davis, California 95616, USA
| | - Katherine S Corker
- Psychology Department, Grand Valley State University, Allendale, Michigan 49401, USA
| | - Anna Dreber
- Department of Economics, Stockholm School of Economics, 113 83 Stockholm, Sweden
| | - Fiona Fidler
- School of Biosciences, University of Melbourne, Parkville VIC 3010, Australia
| | - Joe Hilgard
- Department of Psychology, Illinois State University, Normal, Illinois 61790, USA
| | | | - Michèle B Nuijten
- Meta-Research Center, Tilburg University, 5037 AB Tilburg, The Netherlands
| | - Julia M Rohrer
- Department of Psychology, Leipzig University, 04109 Leipzig, Germany
| | - Felipe Romero
- Department of Theoretical Philosophy, University of Groningen, 9712 CP, The Netherlands
| | - Anne M Scheel
- Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands
| | - Laura D Scherer
- University of Colorado Anschutz Medical Campus, Aurora, Colorado 80045, USA
| | - Felix D Schönbrodt
- Department of Psychology, Ludwig Maximilian University of Munich, 80539 Munich, Germany
| | - Simine Vazire
- School of Psychological Sciences, University of Melbourne, Parkville VIC 3052, Australia
| |
Collapse
|
22
|
Abstract
It is often said that science is self-correcting, but the replication crisis suggests that self-correction mechanisms have fallen short. How can we know whether a particular scientific field has effective self-correction mechanisms, that is, whether its findings are credible? The usual processes that supposedly provide mechanisms for scientific self-correction, such as journal-based peer review and institutional committees, have been inadequate. We describe more verifiable indicators of a field’s commitment to self-correction. These fall under the broad headings of 1) transparency, which is already the subject of many reform efforts and 2) critical appraisal, which has received less attention and which we focus on here. Only by obtaining Observable Self-Correction Indicators (OSCIs) can we begin to evaluate the claim that “science is self-correcting.” We expect that the veracity of this claim varies across fields and subfields, and suggest that some fields, such as psychology and biomedicine, fall far short of an appropriate level of transparency and, especially, critical appraisal. Fields without robust, verifiable mechanisms for transparency and critical appraisal cannot reasonably be said to be self-correcting, and thus do not warrant the credibility often imputed to science as a whole.
Collapse
Affiliation(s)
- Simine Vazire
- Melbourne School of Psychological Sciences, University of Melbourne, Melbourne, VIC, Australia
| | - Alex O. Holcombe
- School of Psychology, The University of Sydney, Sydney, NSW, Australia
| |
Collapse
|
23
|
Mayo-Wilson E, Grant S, Supplee LH. Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations. PREVENTION SCIENCE : THE OFFICIAL JOURNAL OF THE SOCIETY FOR PREVENTION RESEARCH 2021; 23:774-786. [PMID: 34357509 PMCID: PMC9283145 DOI: 10.1007/s11121-021-01284-x] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/14/2021] [Indexed: 12/22/2022]
Abstract
Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.
Collapse
Affiliation(s)
- Evan Mayo-Wilson
- Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, 1025 E. 7th Street, Bloomington, IN, 47405, USA.
- Department of Social & Behavioral Sciences, Fairbanks School of Public Health, Indiana University Richard M, Indianapolis, IN, USA.
| | - Sean Grant
- Department of Social & Behavioral Sciences, Fairbanks School of Public Health, Indiana University Richard M, Indianapolis, IN, USA
| | | |
Collapse
|
24
|
Ravn T, Sørensen MP. Exploring the Gray Area: Similarities and Differences in Questionable Research Practices (QRPs) Across Main Areas of Research. SCIENCE AND ENGINEERING ETHICS 2021; 27:40. [PMID: 34136962 DOI: 10.1007/s11948-021-00310-z] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/09/2020] [Accepted: 05/03/2021] [Indexed: 06/12/2023]
Abstract
This paper explores the gray area of questionable research practices (QRPs) between responsible conduct of research and severe research misconduct in the form of fabrication, falsification, and plagiarism (Steneck in SEE 12(1): 53-57, 2006). Up until now, we have had very little knowledge of disciplinary similarities and differences in QRPs. The paper is the first systematic account of variances and similarities. It reports on the findings of a comprehensive study comprising 22 focus groups on practices and perceptions of QRPs across main areas of research. The paper supports the relevance of the idea of epistemic cultures (Knorr Cetina in: Epistemic cultures: how the sciences make knowledge, Harvard University Press, Cambridge, 1999), also when it comes to QRPs. It shows which QRPs researchers from different areas of research (humanities, social sciences, medical sciences, natural sciences, and technical sciences) report as the most severe and prevalent within their fields. Furthermore, it shows where in the research process these self-reported QRPs can be found. This is done by using a five-phase analytical model of the research process (idea generation, research design, data collection, data analysis, scientific publication and reporting). The paper shows that QRPs are closely connected to the distinct research practices within the different areas of research. Many QRPs can therefore only be found within one area of research, and QRPs that cut across main areas often cover relatively different practices. In a few cases, QRPs in one area are considered good research practice in another.
Collapse
Affiliation(s)
- Tine Ravn
- Department of Political Science, The Danish Centre for Studies in Research and Research Policy, Aarhus University, Bartholins Allé 7, 8000, Aarhus C, Denmark.
| | - Mads P Sørensen
- Department of Political Science, The Danish Centre for Studies in Research and Research Policy, Aarhus University, Bartholins Allé 7, 8000, Aarhus C, Denmark
| |
Collapse
|
25
|
Fostering Interdisciplinary Data Cultures through Early Career Development: The RDA/US Data Share Fellowship. DATA SCIENCE JOURNAL 2021. [DOI: 10.5334/dsj-2021-002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022] Open
|
26
|
Valdez D, Vorland CJ, Brown AW, Mayo-Wilson E, Otten J, Ball R, Grant S, Levy R, Svetina Valdivia D, Allison DB. Improving open and rigorous science: ten key future research opportunities related to rigor, reproducibility, and transparency in scientific research. F1000Res 2020; 9:1235. [PMID: 33628434 PMCID: PMC7898357 DOI: 10.12688/f1000research.26594.1] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 10/02/2020] [Indexed: 11/20/2022] Open
Abstract
Background: As part of a coordinated effort to expand research activity around rigor, reproducibility, and transparency (RRT) across scientific disciplines, a team of investigators at the Indiana University School of Public Health-Bloomington hosted a workshop in October 2019 with international leaders to discuss key opportunities for RRT research. Objective: The workshop aimed to identify research priorities and opportunities related to RRT. Design: Over two-days, workshop attendees gave presentations and participated in three working groups: (1) Improving Education & Training in RRT, (2) Reducing Statistical Errors and Increasing Analytic Transparency, and (3) Looking Outward: Increasing Truthfulness and Accuracy of Research Communications. Following small-group discussions, the working groups presented their findings, and participants discussed the research opportunities identified. The investigators compiled a list of research priorities, which were circulated to all participants for feedback. Results: Participants identified the following priority research questions: (1) Can RRT-focused statistics and mathematical modeling courses improve statistics practice?; (2) Can specialized training in scientific writing improve transparency?; (3) Does modality (e.g. face to face, online) affect the efficacy RRT-related education?; (4) How can automated programs help identify errors more efficiently?; (5) What is the prevalence and impact of errors in scientific publications (e.g., analytic inconsistencies, statistical errors, and other objective errors)?; (6) Do error prevention workflows reduce errors?; (7) How do we encourage post-publication error correction?; (8) How does 'spin' in research communication affect stakeholder understanding and use of research evidence?; (9) Do tools to aid writing research reports increase comprehensiveness and clarity of research reports?; and (10) Is it possible to inculcate scientific values and norms related to truthful, rigorous, accurate, and comprehensive scientific reporting? Conclusion: Participants identified important and relatively unexplored questions related to improving RRT. This list may be useful to the scientific community and investigators seeking to advance meta-science (i.e. research on research).
Collapse
Affiliation(s)
- Danny Valdez
- Indiana University School of Public Health, Bloomington, IN, 47403, USA
| | - Colby J. Vorland
- Indiana University School of Public Health, Bloomington, IN, 47403, USA
| | - Andrew W. Brown
- Indiana University School of Public Health, Bloomington, IN, 47403, USA
| | - Evan Mayo-Wilson
- Indiana University School of Public Health, Bloomington, IN, 47403, USA
| | - Justin Otten
- Indiana University School of Public Health, Bloomington, IN, 47403, USA
| | - Richard Ball
- Project TIER, Haverford College, Haverford, Pennsylvania, 19041, USA
| | - Sean Grant
- Indiana University Purdue University Indianapolis Fairbanks School of Public Health, Indianapolis, IN, 46223, USA
| | - Rachel Levy
- Rachel Levy, Mathematical Association of America, 1529 18th St. NW, Washington, DC, 20036, USA
| | | | - David B. Allison
- Indiana University School of Public Health, Bloomington, IN, 47403, USA
| |
Collapse
|
27
|
Soderberg CK, Errington TM, Nosek BA. Credibility of preprints: an interdisciplinary survey of researchers. ROYAL SOCIETY OPEN SCIENCE 2020; 7:201520. [PMID: 33204484 DOI: 10.31222/osf.io/kabux] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/25/2020] [Accepted: 10/05/2020] [Indexed: 05/28/2023]
Abstract
Preprints increase accessibility and can speed scholarly communication if researchers view them as credible enough to read and use. Preprint services do not provide the heuristic cues of a journal's reputation, selection, and peer-review processes that, regardless of their flaws, are often used as a guide for deciding what to read. We conducted a survey of 3759 researchers across a wide range of disciplines to determine the importance of different cues for assessing the credibility of individual preprints and preprint services. We found that cues related to information about open science content and independent verification of author claims were rated as highly important for judging preprint credibility, and peer views and author information were rated as less important. As of early 2020, very few preprint services display any of the most important cues. By adding such cues, services may be able to help researchers better assess the credibility of preprints, enabling scholars to more confidently use preprints, thereby accelerating scientific communication and discovery.
Collapse
Affiliation(s)
| | | | - Brian A Nosek
- Center for Open Science, Charlottesville VA, USA
- Department of Psychology, University of Virginia, Charlottesville VA, USA
| |
Collapse
|
28
|
Soderberg CK, Errington TM, Nosek BA. Credibility of preprints: an interdisciplinary survey of researchers. ROYAL SOCIETY OPEN SCIENCE 2020; 7:201520. [PMID: 33204484 PMCID: PMC7657885 DOI: 10.1098/rsos.201520] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/25/2020] [Accepted: 10/05/2020] [Indexed: 06/11/2023]
Abstract
Preprints increase accessibility and can speed scholarly communication if researchers view them as credible enough to read and use. Preprint services do not provide the heuristic cues of a journal's reputation, selection, and peer-review processes that, regardless of their flaws, are often used as a guide for deciding what to read. We conducted a survey of 3759 researchers across a wide range of disciplines to determine the importance of different cues for assessing the credibility of individual preprints and preprint services. We found that cues related to information about open science content and independent verification of author claims were rated as highly important for judging preprint credibility, and peer views and author information were rated as less important. As of early 2020, very few preprint services display any of the most important cues. By adding such cues, services may be able to help researchers better assess the credibility of preprints, enabling scholars to more confidently use preprints, thereby accelerating scientific communication and discovery.
Collapse
Affiliation(s)
| | | | - Brian A. Nosek
- Center for Open Science, Charlottesville VA, USA
- Department of Psychology, University of Virginia, Charlottesville VA, USA
| |
Collapse
|
29
|
Smith E, Williams-Jones B, Master Z, Larivière V, Sugimoto CR, Paul-Hus A, Shi M, Resnik DB. Misconduct and Misbehavior Related to Authorship Disagreements in Collaborative Science. SCIENCE AND ENGINEERING ETHICS 2020; 26:1967-1993. [PMID: 31161378 PMCID: PMC6888995 DOI: 10.1007/s11948-019-00112-4] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/11/2018] [Accepted: 05/20/2019] [Indexed: 05/30/2023]
Abstract
Scientific authorship serves to identify and acknowledge individuals who "contribute significantly" to published research. However, specific authorship norms and practices often differ within and across disciplines, labs, and cultures. As a consequence, authorship disagreements are commonplace in team research. This study aims to better understand the prevalence of authorship disagreements, those factors that may lead to disagreements, as well as the extent and nature of resulting misbehavior. Methods include an international online survey of researchers who had published from 2011 to 2015 (8364 respondents). Of the 6673 who completed the main questions pertaining to authorship disagreement and misbehavior, nearly half (46.6%) reported disagreements regarding authorship naming; and discipline, rank, and gender had significant effects on disagreement rates. Paradoxically, researchers in multidisciplinary teams that typically reflect a range of norms and values, were less likely to have faced disagreements regarding authorship. Respondents reported having witnessed a wide range of misbehavior including: instances of hostility (24.6%), undermining of a colleague's work during meetings/talks (16.4%), cutting corners on research (8.3%), sabotaging a colleague's research (6.4%), or producing fraudulent work to be more competitive (3.3%). These findings suggest that authorship disputes may contribute to an unhealthy competitive dynamic that can undermine researchers' wellbeing, team cohesion, and scientific integrity.
Collapse
Affiliation(s)
- Elise Smith
- National Institute of Environmental Health Sciences, National Institutes of Health, Research Triangle Park, NC, 27709, USA.
| | - Bryn Williams-Jones
- Bioethics Program, School of Public Health, University of Montreal, Montreal, QC, H3C 3J7, Canada
| | - Zubin Master
- Biomedical Ethics Research Program and Center for Regenerative Medicine, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA
| | - Vincent Larivière
- School of Library and Information Science, University of Montreal, Montreal, QC, H3C 3J7, Canada
| | - Cassidy R Sugimoto
- School of Informatics, Computing and Engineering, Indiana University Bloomington, Bloomington, IN, 47408, USA
| | - Adèle Paul-Hus
- School of Library and Information Science, University of Montreal, Montreal, QC, H3C 3J7, Canada
| | - Min Shi
- National Institute of Environmental Health Sciences, National Institutes of Health, Research Triangle Park, NC, 27709, USA
| | - David B Resnik
- National Institute of Environmental Health Sciences, National Institutes of Health, Research Triangle Park, NC, 27709, USA
| |
Collapse
|
30
|
Actionable recommendations for narrowing the science-practice gap in open science. ORGANIZATIONAL BEHAVIOR AND HUMAN DECISION PROCESSES 2020. [DOI: 10.1016/j.obhdp.2020.02.007] [Citation(s) in RCA: 30] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/15/2023]
|
31
|
Hofmann B, Holm S. Research integrity: environment, experience, or ethos? RESEARCH ETHICS REVIEW 2019. [DOI: 10.1177/1747016119880844] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Background: Research integrity has gained attention in the general public as well as in the research community. We wanted to investigate knowledge, attitudes, and practices amongst researchers that have recently finished their PhD and compare this to their responses during their PhD fellowship. In particular, we wanted to investigate whether their attitudes are related to their experiences of their immediate research environment. Material and method: Researchers ( n = 86) awarded the PhD degree at the Faculty of Medicine at the University of Oslo in 2016 were invited to answer a questionnaire about knowledge, attitudes, and actions related to scientific dishonesty. Seventy-two responded (83.7%). The results were compared with results among first-year doctoral students who responded to the same questionnaire during 2010–2017. Results: Overall, 13% of PhDs reported that they knew of people in their immediate research environment who had committed serious forms of scientific dishonesty. A small percentage of PhDs (1.4%) indicated that they themselves had committed such acts. About 3% of the candidates had experienced pressure to commit serious forms of dishonesty and nearly a third of respondents had experienced unethical pressure with respect to authorship during the course of their fellowship. Thirteen percent reported that they had experienced unethical pressure in relation to other forms of dishonesty and 11% had experienced the consequences of some form of scientific dishonesty. Eighteen percent of the respondents believed that one or more actions, which in the literature were perceived as scientific misconduct, were not wrong. We find a connection between attitudes and the perceived research integrity of their research environment. The results also show a difference between PhD students and graduated PhDs in terms of scientific dishonesty. In some areas, the PhDs’ norms are stricter, such as for the use of statistical analysis methods, while there is little change in others, such as in misconduct in order to expedite publications. Conclusion: Many PhDs knew about serious forms of scientific misconduct from the research environment in which they are trained, and some also report misconduct themselves. Some experienced pressure to serious forms of misconduct and a large proportion of the respondents had experienced unethical pressure with respect to authorship during their fellowship. Attitudes change during the PhD studies, but ambiguously. Scientific misconduct seems to be an environmental issue as much as a matter of personal integrity.
Collapse
Affiliation(s)
- Bjørn Hofmann
- NTNU Gjøvik, Norway
- Centre for Medical Ethics, University of Oslo, Norway
| | - Søren Holm
- University of Manchester, UK
- Centre for Medical Ethics, University of Oslo, Norway
| |
Collapse
|
32
|
Weeks KL, Henstridge DC, Salim A, Shaw JE, Marwick TH, McMullen JR. CORP: Practical tools for improving experimental design and reporting of laboratory studies of cardiovascular physiology and metabolism. Am J Physiol Heart Circ Physiol 2019; 317:H627-H639. [PMID: 31347916 DOI: 10.1152/ajpheart.00327.2019] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
Abstract
The exercise consisted of: 1) a short survey to acquire baseline data on current practices regarding the conduct of animal studies, 2) a series of presentations for promoting awareness and providing advice and practical tools for improving experimental design, and 3) a follow-up survey 12 mo later to assess whether practices had changed. The surveys were compulsory for responsible investigators (n = 16; paired data presented). Other investigators named on animal ethics applications were encouraged to participate (2017, total of 36 investigators; 2018, 37 investigators). The major findings to come from the exercise included 1) a willingness of investigators to make changes when provided with knowledge/tools and solutions that were relatively simple to implement (e.g., proportion of responsible investigators showing improved practices using a structured method for randomization was 0.44, 95% CI (0.19; 0.70), P = 0.003, and deidentifying drugs/interventions was 0.40, 95% CI (0.12; 0.68), P = 0.010); 2) resistance to change if this involved more personnel and time (e.g., as required for allocation concealment); and 3) evidence that changes to long-term practices ("habits") require time and follow-up. Improved practices could be verified based on changes in reporting within publications or documented evidence provided during laboratory visits. In summary, this exercise resulted in changed attitudes, practices, and reporting, but continued follow-up, monitoring, and incentives are required. Efforts to improve experimental rigor will reduce bias and will lead to findings with the greatest translational potential.NEW & NOTEWORTHY The goal of this exercise was to encourage preclinical researchers to improve the quality of their cardiac and metabolic animal studies by 1) increasing awareness of concerns, which can arise from suboptimal experimental designs; 2) providing knowledge, tools, and templates to overcome bias; and 3) conducting two short surveys over 12 mo to monitor change. Improved practices were identified for the uptake of structured methods for randomization, and de-identifying interventions/drugs.Listen to this article's corresponding podcast at https://ajpheart.podbean.com/e/experimental-design-survey-training-practical-tools/.
Collapse
Affiliation(s)
- Kate L Weeks
- Baker Heart and Diabetes Institute, Melbourne, Australia.,Department of Diabetes, Central Clinical School, Monash University, Clayton, Victoria, Australia
| | | | - Agus Salim
- Baker Heart and Diabetes Institute, Melbourne, Australia.,Department of Mathematics and Statistics, La Trobe University Victoria, Australia
| | | | | | - Julie R McMullen
- Baker Heart and Diabetes Institute, Melbourne, Australia.,Department of Diabetes, Central Clinical School, Monash University, Clayton, Victoria, Australia
| |
Collapse
|
33
|
Antes AL, Kuykendall A, DuBois JM. The lab management practices of "Research Exemplars" that foster research rigor and regulatory compliance: A qualitative study of successful principal investigators. PLoS One 2019; 14:e0214595. [PMID: 31017929 PMCID: PMC6481787 DOI: 10.1371/journal.pone.0214595] [Citation(s) in RCA: 23] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2018] [Accepted: 03/17/2019] [Indexed: 11/30/2022] Open
Abstract
INTRODUCTION Conducting rigorous scientific inquiry within the bounds of research regulation and acceptable practice requires a principal investigator to lead and manage research processes and personnel. This study explores the practices used by investigators nominated as exemplars of research excellence and integrity to produce rigorous, reproducible research and comply with research regulations. METHODS Using a qualitative research design, we interviewed 52 principal investigators working in the United States at top research universities and the National Institutes of Health Intramural Research Program. We solicited nominations of researchers meeting two criteria: (1) they are federally-funded researchers doing high-quality, high-impact research, and (2) have reputations for professionalism and integrity. Each investigator received an initial nomination addressing both criteria and at least one additional endorsement corroborating criteria 2. A panel of researchers and our research team reviewed the nominations to select finalists who were invited to participate. The cohort of "Research Exemplars" includes highly accomplished researchers in diverse scientific disciplines. The semi-structured interview questions asked them to describe the routine practices they employ to foster rigor and regulatory compliance. We used inductive thematic analysis to identify common practices. RESULTS The exemplars identified a core set of 8 practices and provided strategies for employing them. The practices included holding regular team meetings, encouraging shared ownership, providing supervision, ensuring adequate training, fostering positive attitudes about compliance, scrutinizing data and findings, and following standard operating procedures. Above all, the use of these practices aim to create a psychologically safe work environment in which lab members openly collaborate to scrutinize their work and share in accountability for rigorous, compliant research. CONCLUSIONS Researchers typically receive limited systematic training in how to lead and manage their research teams. Training and education for principal investigators should include essential leadership and management practices and strategies that support doing high-quality research with integrity.
Collapse
Affiliation(s)
- Alison L. Antes
- Division of General Medical Sciences, Washington University School of Medicine, St. Louis, Missouri, United States of America
| | - Ashley Kuykendall
- Division of General Medical Sciences, Washington University School of Medicine, St. Louis, Missouri, United States of America
| | - James M. DuBois
- Division of General Medical Sciences, Washington University School of Medicine, St. Louis, Missouri, United States of America
| |
Collapse
|
34
|
Krishna A, Peter SM. Questionable research practices in student final theses - Prevalence, attitudes, and the role of the supervisor's perceived attitudes. PLoS One 2018; 13:e0203470. [PMID: 30161249 PMCID: PMC6117074 DOI: 10.1371/journal.pone.0203470] [Citation(s) in RCA: 15] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2018] [Accepted: 08/21/2018] [Indexed: 11/18/2022] Open
Abstract
Although questionable research practices (QRPs) and p-hacking have received attention in recent years, little research has focused on their prevalence and acceptance in students. Students are the researchers of the future and will represent the field in the future. Therefore, they should not be learning to use and accept QRPs, which would reduce their ability to produce and evaluate meaningful research. 207 psychology students and fresh graduates provided self-report data on the prevalence and predictors of QRPs. Attitudes towards QRPs, belief that significant results constitute better science or lead to better grades, motivation, and stress levels were predictors. Furthermore, we assessed perceived supervisor attitudes towards QRPs as an important predictive factor. The results were in line with estimates of QRP prevalence from academia. The best predictor of QRP use was students' QRP attitudes. Perceived supervisor attitudes exerted both a direct and indirect effect via student attitudes. Motivation to write a good thesis was a protective factor, whereas stress had no effect. Students in this sample did not subscribe to beliefs that significant results were better for science or their grades. Such beliefs further did not impact QRP attitudes or use in this sample. Finally, students engaged in more QRPs pertaining to reporting and analysis than those pertaining to study design. We conclude that supervisors have an important function in shaping students' attitudes towards QRPs and can improve their research practices by motivating them well. Furthermore, this research provides some impetus towards identifying predictors of QRP use in academia.
Collapse
Affiliation(s)
- Anand Krishna
- Department of Motivational and Emotional Psychology, Julius-Maximilians-Universität, Würzburg, Germany
| | - Sebastian M. Peter
- Department of Social Psychology, Julius-Maximilians-Universität, Würzburg, Germany
| |
Collapse
|
35
|
Grand JA, Rogelberg SG, Banks GC, Landis RS, Tonidandel S. From Outcome to Process Focus: Fostering a More Robust Psychological Science Through Registered Reports and Results-Blind Reviewing. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 2018; 13:448-456. [DOI: 10.1177/1745691618767883] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
A variety of alternative mechanisms, strategies, and “ways of doing” have been proposed for improving the rigor and robustness of published research in the psychological sciences in recent years. In this article, we describe two existing but underused publication models—registered reporting (RR) and results-blind reviewing (RBR)—that we believe would contribute in important ways to improving both the conduct and evaluation of psychological research. We first outline the procedures and distinguishing features of both publication pathways and note their value for promoting positive changes to current scientific practices. We posit that a significant value of RR and RBR is their potential to promote a greater focus on the research process (i.e., how and why research is conducted) relative to research outcomes (i.e., what was observed or concluded from research). We conclude by discussing what we perceive to be five common beliefs about RR and RBR practices and attempt to provide a balanced perspective of the realities likely to be experienced with these systems.
Collapse
Affiliation(s)
| | - Steven G. Rogelberg
- Department of Psychological Science, University of North Carolina at Charlotte
| | - George C. Banks
- Department of Management, University of North Carolina at Charlotte
| | | | | |
Collapse
|
36
|
Holm S, Hofmann B. Associations between attitudes towards scientific misconduct and self-reported behavior. Account Res 2018; 25:290-300. [PMID: 29874932 DOI: 10.1080/08989621.2018.1485493] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/14/2022]
Abstract
We investigate the relationship between doctoral students' attitudes towards scientific misconduct and their self-reported behavior. 203 questionnaires were distributed to doctoral candidates at the Faculty of Medicine, University of Oslo 2016/2017. The response rate was 74%. The results show a correlation between attitudes towards misconduct and self-reported problematic behaviors among doctoral students in biomedicine. The four most common reported misbehaviors are adding author(s) who did not qualify for authorship (17.9%), collecting more data after seeing that the results were almost statistically significant (11.8%), turning a blind eye to colleagues' use of flawed data or questionable interpretation of data (11.2%), and reporting an unexpected finding as having been hypothesized from the start (10.5%). We find correlations between scientific misbehavior and the location of undergraduate studies and whether the respondents have had science ethics lectures previously. The study provides evidence for the concurrent validity of the two instruments used to measure attitudes and behavior, i.e. the Kalichman scale and the Research Misbehavior Severity Score (RMSS). Although the direction of causality between attitudes and misbehavior cannot be determined in this study the correlation between the two indicates that it can be important to engender the right attitudes in early career researchers.
Collapse
Affiliation(s)
- Søren Holm
- a Centre for Social Ethics and Policy, School of Law , University of Manchester , Manchester , UK.,b Centre of Medical Ethics , University of Oslo , Oslo , Norway
| | - Bjørn Hofmann
- b Centre of Medical Ethics , University of Oslo , Oslo , Norway.,c Institute for the Health Sciences , Norwegian University of Science and Technology , Gjøvik , Norway
| |
Collapse
|
37
|
Abstract
Research ethics provide important and necessary standards related to the conduct and dissemination of research. To better understand the current state of research ethics discourse in social work, a systematic literature search was undertaken and numbers of publications per year were compared between STEM, social science, and social work disciplines. While many professions have embraced the need for discipline-specific research ethics subfield development, social work has remained absent. Low publication numbers, compared to other disciplines, were noted for the years (2006-2016) included in the study. Social work published 16 (1%) of the 1409 articles included in the study, contributing 3 (>1%) for each of the disciplines highest producing years (2011 and 2013). Comparatively, psychology produced 75 (5%) articles, psychiatry produced 64 (5%) articles, and nursing added 50 (4%) articles. The STEM disciplines contributed 956 (68%) articles between 2006 and 2016, while social science produced 453 (32%) articles. Examination of the results is provided in an extended discussion of several misconceptions about research ethics that may be found in the social work profession. Implications and future directions are provided, focusing on the need for increased engagement, education, research, and support for a new subfield of social work research ethics.
Collapse
Affiliation(s)
- Aidan Ferguson
- a College of Social Work , Florida State University , Tallahassee , USA
| | - James J Clark
- a College of Social Work , Florida State University , Tallahassee , USA
| |
Collapse
|
38
|
Antes AL, English T, Baldwin KA, DuBois JM. The Role of Culture and Acculturation in Researchers' Perceptions of Rules in Science. SCIENCE AND ENGINEERING ETHICS 2018; 24:361-391. [PMID: 28321685 PMCID: PMC5607071 DOI: 10.1007/s11948-017-9876-4] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/07/2016] [Accepted: 01/16/2017] [Indexed: 05/23/2023]
Abstract
Successfully navigating the norms of a society is a complex task that involves recognizing diverse kinds of rules as well as the relative weight attached to them. In the United States (U.S.), different kinds of rules-federal statutes and regulations, scientific norms, and professional ideals-guide the work of researchers. Penalties for violating these different kinds of rules and norms can range from the displeasure of peers to criminal sanctions. We proposed that it would be more difficult for researchers working in the U.S. who were born in other nations to distinguish the seriousness of violating rules across diverse domains. We administered a new measure, the evaluating rules in science task (ERST), to National Institutes of Health-funded investigators (101 born in the U.S. and 102 born outside of the U.S.). The ERST assessed perceptions of the seriousness of violating research regulations, norms, and ideals, and allowed us to calculate the degree to which researchers distinguished between the seriousness of each rule category. The ERST also assessed researchers' predictions of the seriousness that research integrity officers (RIOs) would assign to the rules. We compared researchers' predictions to the seriousness ratings of 112 RIOs working at U.S. research-intensive universities. U.S.-born researchers were significantly better at distinguishing between the seriousness of violating federal research regulations and violating ideals of science, and they were more accurate in their predictions of the views of RIOs. Acculturation to the U.S. moderated the effects of nationality on accuracy. We discuss the implications of these findings in terms of future research and education.
Collapse
Affiliation(s)
- Alison L Antes
- Division of General Medical Sciences, Washington University School of Medicine, 4523 Clayton Avenue, Campus Box 8005, St. Louis, MO, 63110, USA
| | - Tammy English
- Department of Psychological and Brain Sciences, Washington University in St. Louis, One Brookings Drive, Campus Box 1125, St. Louis, MO, 63130, USA
| | - Kari A Baldwin
- Division of General Medical Sciences, Washington University School of Medicine, 4523 Clayton Avenue, Campus Box 8005, St. Louis, MO, 63110, USA
| | - James M DuBois
- Division of General Medical Sciences, Washington University School of Medicine, 4523 Clayton Avenue, Campus Box 8005, St. Louis, MO, 63110, USA.
| |
Collapse
|
39
|
Abstract
Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes-a process called preregistration. Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting. Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.
Collapse
Affiliation(s)
- Brian A Nosek
- Center for Open Science, Charlottesville, VA 22903;
- Department of Psychology, University of Virginia, Charlottesville, VA 22904
| | - Charles R Ebersole
- Department of Psychology, University of Virginia, Charlottesville, VA 22904
| | | | | |
Collapse
|
40
|
Houtkoop BL, Chambers C, Macleod M, Bishop DVM, Nichols TE, Wagenmakers EJ. Data Sharing in Psychology: A Survey on Barriers and Preconditions. ADVANCES IN METHODS AND PRACTICES IN PSYCHOLOGICAL SCIENCE 2018. [DOI: 10.1177/2515245917751886] [Citation(s) in RCA: 80] [Impact Index Per Article: 13.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Despite its potential to accelerate academic progress in psychological science, public data sharing remains relatively uncommon. In order to discover the perceived barriers to public data sharing and possible means for lowering them, we conducted a survey, which elicited responses from 600 authors of articles in psychology. The results confirmed that data are shared only infrequently. Perceived barriers included respondents’ belief that sharing is not a common practice in their fields, their preference to share data only upon request, their perception that sharing requires extra work, and their lack of training in sharing data. Our survey suggests that strong encouragement from institutions, journals, and funders will be particularly effective in overcoming these barriers, in combination with educational materials that demonstrate where and how data can be shared effectively.
Collapse
Affiliation(s)
| | - Chris Chambers
- Brain Research Imaging Centre (CUBRIC), School of Psychology, Cardiff University
| | | | | | - Thomas E. Nichols
- Oxford Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford
- Wellcome Centre for Integrative Neuroimaging, FMRIB, Nuffield Department of Clinical Neurosciences, University of Oxford
- Department of Statistics, University of Warwick
| | | |
Collapse
|
41
|
|
42
|
Schönbrodt FD, Maier M, Heene M, Bühner M. Forschungstransparenz als hohes wissenschaftliches Gut stärken. PSYCHOLOGISCHE RUNDSCHAU 2018. [DOI: 10.1026/0033-3042/a000386] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Zusammenfassung. Groß angelegte Replikationsprojekte der letzten Jahre legen ein aus unserer Sicht beunruhigendes Ausmaß an nicht-replizierbaren Befunden in der wissenschaftlichen Literatur nahe, sowohl in der Psychologie als auch in anderen Disziplinen. Basierend auf einer Analyse einiger Ursachen dieser Situation argumentieren wir, dass der Wandel hin zu einer offenen Wissenschaft („Open Science“) eine Konsequenz aus der Glaubwürdigkeitskrise sein muss. Wir plädieren für konkrete und machbare Änderungen in den Arbeitseinheiten und Instituten vor Ort, und zeigen exemplarisch, welche Schritte am Department Psychologie der Ludwig-Maximilians-Universität München umgesetzt wurden. Diese Schritte betreffen Anreizstrukturen, die Forschungskultur, die Lehre und die Verzahnung mit der Ethikkommission. Sie haben das Ziel, eine reproduzierbarere und glaubwürdigere Forschung zu unterstützen, ohne unnötige bürokratische Belastungen zu erzeugen.
Collapse
|
43
|
Wicherts JM. The Weak Spots in Contemporary Science (and How to Fix Them). Animals (Basel) 2017; 7:E90. [PMID: 29186879 PMCID: PMC5742784 DOI: 10.3390/ani7120090] [Citation(s) in RCA: 19] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2017] [Revised: 11/19/2017] [Accepted: 11/23/2017] [Indexed: 01/17/2023] Open
Abstract
In this review, the author discusses several of the weak spots in contemporary science, including scientific misconduct, the problems of post hoc hypothesizing (HARKing), outcome switching, theoretical bloopers in formulating research questions and hypotheses, selective reading of the literature, selective citing of previous results, improper blinding and other design failures, p-hacking or researchers' tendency to analyze data in many different ways to find positive (typically significant) results, errors and biases in the reporting of results, and publication bias. The author presents some empirical results highlighting problems that lower the trustworthiness of reported results in scientific literatures, including that of animal welfare studies. Some of the underlying causes of these biases are discussed based on the notion that researchers are only human and hence are not immune to confirmation bias, hindsight bias, and minor ethical transgressions. The author discusses solutions in the form of enhanced transparency, sharing of data and materials, (post-publication) peer review, pre-registration, registered reports, improved training, reporting guidelines, replication, dealing with publication bias, alternative inferential techniques, power, and other statistical tools.
Collapse
Affiliation(s)
- Jelte M Wicherts
- Department of Methodology and Statistics, Tilburg University, Warandelaan 2, 5037 AB, Tilburg, The Netherlands.
| |
Collapse
|
44
|
Agnoli F, Wicherts JM, Veldkamp CLS, Albiero P, Cubelli R. Questionable research practices among italian research psychologists. PLoS One 2017; 12:e0172792. [PMID: 28296929 PMCID: PMC5351839 DOI: 10.1371/journal.pone.0172792] [Citation(s) in RCA: 52] [Impact Index Per Article: 7.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/18/2016] [Accepted: 02/09/2017] [Indexed: 11/18/2022] Open
Abstract
A survey in the United States revealed that an alarmingly large percentage of university psychologists admitted having used questionable research practices that can contaminate the research literature with false positive and biased findings. We conducted a replication of this study among Italian research psychologists to investigate whether these findings generalize to other countries. All the original materials were translated into Italian, and members of the Italian Association of Psychology were invited to participate via an online survey. The percentages of Italian psychologists who admitted to having used ten questionable research practices were similar to the results obtained in the United States although there were small but significant differences in self-admission rates for some QRPs. Nearly all researchers (88%) admitted using at least one of the practices, and researchers generally considered a practice possibly defensible if they admitted using it, but Italian researchers were much less likely than US researchers to consider a practice defensible. Participants’ estimates of the percentage of researchers who have used these practices were greater than the self-admission rates, and participants estimated that researchers would be unlikely to admit it. In written responses, participants argued that some of these practices are not questionable and they have used some practices because reviewers and journals demand it. The similarity of results obtained in the United States, this study, and a related study conducted in Germany suggest that adoption of these practices is an international phenomenon and is likely due to systemic features of the international research and publication processes.
Collapse
Affiliation(s)
- Franca Agnoli
- Department of Developmental Psychology and Socialization, University of Padova, Padova, Italy
- * E-mail:
| | - Jelte M. Wicherts
- Department of Methodology and Statistics, Tilburg University, Tilburg, Netherlands
| | | | - Paolo Albiero
- Department of Developmental Psychology and Socialization, University of Padova, Padova, Italy
| | - Roberto Cubelli
- Department of Psychology and Cognitive Science, University of Trento, Trento, Italy
| |
Collapse
|
45
|
Penders B. Machiavelli in the Lab. Account Res 2017. [DOI: 10.1080/08989621.2016.1225499] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
46
|
Abstract
Do lay people and scientists themselves recognize that scientists are human and therefore prone to human fallibilities such as error, bias, and even dishonesty? In a series of three experimental studies and one correlational study (total N = 3,278) we found that the “storybook image of the scientist” is pervasive: American lay people and scientists from over 60 countries attributed considerably more objectivity, rationality, open-mindedness, intelligence, integrity, and communality to scientists than to other highly-educated people. Moreover, scientists perceived even larger differences than lay people did. Some groups of scientists also differentiated between different categories of scientists: established scientists attributed higher levels of the scientific traits to established scientists than to early-career scientists and Ph.D. students, and higher levels to Ph.D. students than to early-career scientists. Female scientists attributed considerably higher levels of the scientific traits to female scientists than to male scientists. A strong belief in the storybook image and the (human) tendency to attribute higher levels of desirable traits to people in one’s own group than to people in other groups may decrease scientists’ willingness to adopt recently proposed practices to reduce error, bias and dishonesty in science.
Collapse
Affiliation(s)
- Coosje L S Veldkamp
- a Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences , Tilburg University , Tilburg , The Netherlands
| | - Chris H J Hartgerink
- a Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences , Tilburg University , Tilburg , The Netherlands
| | - Marcel A L M van Assen
- a Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences , Tilburg University , Tilburg , The Netherlands.,b Department of Sociology, Faculty of Social and Behavioral Sciences , Utrecht University , Utrecht , The Netherlands
| | - Jelte M Wicherts
- a Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences , Tilburg University , Tilburg , The Netherlands
| |
Collapse
|
47
|
Forstmeier W, Wagenmakers E, Parker TH. Detecting and avoiding likely false‐positive findings – a practical guide. Biol Rev Camb Philos Soc 2016; 92:1941-1968. [DOI: 10.1111/brv.12315] [Citation(s) in RCA: 212] [Impact Index Per Article: 26.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2016] [Revised: 10/17/2016] [Accepted: 10/19/2016] [Indexed: 12/17/2022]
Affiliation(s)
- Wolfgang Forstmeier
- Department of Behavioural Ecology and Evolutionary Genetics Max Planck Institute for Ornithology 82319 Seewiesen Germany
| | - Eric‐Jan Wagenmakers
- Department of Psychology University of Amsterdam PO Box 15906 1001 NK Amsterdam The Netherlands
| | | |
Collapse
|
48
|
Giorgini V, Mecca JT, Gibson C, Medeiros K, Mumford MD, Connelly S, Devenport LD. Researcher perceptions of ethical guidelines and codes of conduct. Account Res 2016; 22:123-38. [PMID: 25635845 DOI: 10.1080/08989621.2014.955607] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
Abstract
Ethical codes of conduct exist in almost every profession. Field-specific codes of conduct have been around for decades, each articulating specific ethical and professional guidelines. However, there has been little empirical research on researchers' perceptions of these codes of conduct. In the present study, we interviewed faculty members in six research disciplines and identified five themes bearing on the circumstances under which they use ethical guidelines and the underlying reasons for not adhering to such guidelines. We then identify problems with the manner in which codes of conduct in academia are constructed and offer solutions for overcoming these problems.
Collapse
Affiliation(s)
- Vincent Giorgini
- a Department of Psychology, University of Oklahoma , Norman , Oklahoma , USA
| | | | | | | | | | | | | |
Collapse
|
49
|
Hofmann B, Holm S. [Research misconduct: Knowledge, actions and attitudes of PhD candidates]. TIDSSKRIFT FOR DEN NORSKE LEGEFORENING 2016; 136:1442-7. [PMID: 27686203 DOI: 10.4045/tidsskr.16.0149] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022] Open
Abstract
BACKGROUND Increasing attention is being paid to research misconduct in academic journals and in the media, but we know relatively little about its extent or attitudes to research misconduct, or how these are changing. This study therefore aims to investigate PhD candidates' knowledge, own actions and attitudes to specific forms of research misconduct.MATERIAL AND METHOD In autumn 2015, an anonymous questionnaire survey was distributed to all participants in the introductory course for PhD candidates at the Faculty of Medicine at the University of Oslo.RESULTS Altogether 77 PhD candidates (79 %) responded to the questionnaire. A total of 62 % conducted clinical research and 25 % conducted basic research. Around one in four had heard about serious forms of research misconduct in the previous year, and around 4 % were aware of various forms of serious research misconduct in their own department in the previous year. Compared to earlier studies, an increasing number (16 %) responded that they had been subjected to unethical pressure with regard to inclusion or order of authors. Approximately two-thirds were uncertain of whether their department had written policies for academic conduct. One-third of PhD candidates did not disassociate themselves from actions that are generally viewed as scientific misconduct. One-tenth thought it acceptable to falsify or fabricate data in order to expedite publication, one-fifth did not object to taking the credit for others' ideas, and almost half did not believe it was wrong to attempt a number of methods of analysis until one arrived at a significant answer.INTERPRETATION PhD candidates at the Faculty of Medicine were aware of research misconduct, both generally and from their own department. They themselves reported some type of scientific misconduct, and a large majority were uncertain of their department's guidelines. Some of the candidates also accepted several forms of research misconduct.
Collapse
Affiliation(s)
- Bjørn Hofmann
- Norges teknisk-naturvitenskapelige universitet Gjøvik og Senter for medisinsk etikk Universitetet i Oslo
| | - Søren Holm
- University of Manchester og Senter for medisinsk etikk Universitetet i Oslo
| |
Collapse
|
50
|
Abstract
Abstract
This article discusses the responsible conduct of research, questionable research practices, and research misconduct. Responsible conduct of research is often defined in terms of a set of abstract, normative principles, professional standards, and ethics in doing research. In order to accommodate the normative principles of scientific research, the professional standards, and a researcher’s moral principles, transparent research practices can serve as a framework for responsible conduct of research. We suggest a “prune-and-add” project structure to enhance transparency and, by extension, responsible conduct of research. Questionable research practices are defined as practices that are detrimental to the research process. The prevalence of questionable research practices remains largely unknown, and reproducibility of findings has been shown to be problematic. Questionable practices are discouraged by transparent practices because practices that arise from them will become more apparent to scientific peers. Most effective might be preregistrations of research design, hypotheses, and analyses, which reduce particularism of results by providing an a priori research scheme. Research misconduct has been defined as fabrication, falsification, and plagiarism (FFP), which is clearly the worst type of research practice. Despite it being clearly wrong, it can be approached from a scientific and legal perspective. The legal perspective sees research misconduct as a form of white-collar crime. The scientific perspective seeks to answer the following question: “Were results invalidated because of the misconduct?” We review how misconduct is typically detected, how its detection can be improved, and how prevalent it might be. Institutions could facilitate detection of data fabrication and falsification by implementing data auditing. Nonetheless, the effect of misconduct is pervasive: many retracted articles are still cited after the retraction has been issued.
Main points
Researchers systematically evaluate their own conduct as more responsible than colleagues, but not as responsible as they would like.
Transparent practices, facilitated by the Open Science Framework, help embody scientific norms that promote responsible conduct.
Questionable research practices harm the research process and work counter to the generally accepted scientific norms, but are hard to detect.
Research misconduct requires active scrutiny of the research community because editors and peer-reviewers do not pay adequate attention to detecting this. Tips are given on how to improve your detection of potential problems.
Collapse
|