Normative dissonance in science: results from a national survey of u.s. Scientists

Is something rotten in the state of Denmark? Cross-national evidence for widespread involvement but not systematic use of questionable research practices across all fields of research

Questionable research practices (QRP) are believed to be widespread, but empirical assessments are generally restricted to a few types of practices. Furthermore, conceptual confusion is rife with use and prevalence of QRPs often being confused as the same quantity. We present the hitherto most comprehensive study examining QRPs across scholarly fields and knowledge production modes. We survey perception, use, prevalence and predictors of QRPs among 3,402 researchers in Denmark and 1,307 in the UK, USA, Croatia and Austria. Results reveal remarkably similar response patterns among Danish and international respondents (τ = 0.85). Self-reported use indicates whether respondents have used a QRP in recent publications. 9 out of 10 respondents admitted using at least one QRP. Median use is three out of nine QRP items. Self-reported prevalence reflects the frequency of use. On average, prevalence rates were roughly three times lower compared to self-reported use. Findings indicated that the perceived social acceptability of QRPs influenced self-report patterns. Results suggest that most researchers use different types of QRPs within a restricted time period. The prevalence estimates, however, do not suggest outright systematic use of specific QRPs. Perceived pressure was the strongest systemic predictor for prevalence. Conversely, more local attention to research cultures and academic age was negatively related to prevalence. Finally, the personality traits conscientiousness and, to a lesser degree, agreeableness were also inversely associated with self-reported prevalence. Findings suggest that explanations for engagement with QRPs are not only attributable to systemic factors, as hitherto suggested, but a complicated mixture of experience, systemic and individual factors, and motivated reasoning.

Empirical evidence of widespread exaggeration bias and selective reporting in ecology

In many scientific disciplines, common research practices have led to unreliable and exaggerated evidence about scientific phenomena. Here we describe some of these practices and quantify their pervasiveness in recent ecology publications in five popular journals. In an analysis of over 350 studies published between 2018 and 2020, we detect empirical evidence of exaggeration bias and selective reporting of statistically significant results. This evidence implies that the published effect sizes in ecology journals exaggerate the importance of the ecological relationships that they aim to quantify. An exaggerated evidence base hinders the ability of empirical ecology to reliably contribute to science, policy, and management. To increase the credibility of ecology research, we describe a set of actions that ecologists should take, including changes to scientific norms about what high-quality ecology looks like and expectations about what high-quality studies can deliver.

Exploring enablers and barriers to implementing the Transparency and Openness Promotion Guidelines: a theory-based survey of journal editors

The Transparency and Openness Promotion (TOP) Guidelines provide a framework to help journals develop open science policies. Theories of behaviour change can guide understanding of why journals do (not) implement open science policies and the development of interventions to improve these policies. In this study, we used the Theoretical Domains Framework to survey 88 journal editors on their capability, opportunity and motivation to implement TOP. Likert-scale questions assessed editor support for TOP, and enablers and barriers to implementing TOP. A qualitative question asked editors to provide reflections on their ratings. Most participating editors supported adopting TOP at their journal (71%) and perceived other editors in their discipline to support adopting TOP (57%). Most editors (93%) agreed their roles include maintaining policies that reflect current best practices. However, most editors (74%) did not see implementing TOP as a high priority compared with other editorial responsibilities. Qualitative responses expressed structural barriers to implementing TOP (e.g. lack of time, resources and authority to implement changes) and varying support for TOP depending on study type, open science standard, and level of implementation. We discuss how these findings could inform the development of theoretically guided interventions to increase open science policies, procedures and practices.

Motivating Proactive Biorisk Management

Scholars and practitioners of biosafety and biosecurity (collectively, biorisk management or BRM) have argued that life scientists should play a more proactive role in monitoring their work for potential risks, mitigating harm, and seeking help as necessary. However, most efforts to promote proactive BRM have focused on training life scientists in technical skills and have largely ignored the extent to which life scientists wish to use them (ie, their motivation). In this article, we argue that efforts to promote proactive BRM would benefit from a greater focus on life scientists' motivation. We review relevant literature on life scientists' motivation to practice BRM, offer examples of successful interventions from adjacent fields, and outline ideas for possible interventions to promote proactive BRM, along with strategies for iterative development, testing, and scaling.

Adoption of Transparency and Openness Promotion (TOP) Guidelines across Journals

Journal policies continuously evolve to enable knowledge sharing and support reproducible science. However, that change happens within a certain framework. Eight modular standards with three levels of increasing stringency make Transparency and Openness Promotion (TOP) guidelines which can be used to evaluate to what extent and with which stringency journals promote open science. Guidelines define standards for data citation, transparency of data, material, code and design and analysis, replication, plan and study pre-registration, and two effective interventions: “Registered reports” and “Open science badges”, and levels of adoption summed up across standards define journal’s TOP Factor. In this paper, we analysed the status of adoption of TOP guidelines across two thousand journals reported in the TOP Factor metrics. We show that the majority of the journals’ policies align with at least one of the TOP’s standards, most likely “Data citation” (70%) followed by “Data transparency” (19%). Two-thirds of adoptions of TOP standard are of the stringency Level 1 (less stringent), whereas only 9% is of the stringency Level 3. Adoption of TOP standards differs across science disciplines and multidisciplinary journals (N = 1505) and journals from social sciences (N = 1077) show the greatest number of adoptions. Improvement of the measures that journals take to implement open science practices could be done: (1) discipline-specific, (2) journals that have not yet adopted TOP guidelines could do so, (3) the stringency of adoptions could be increased.

Sharing the Recipe: Reproducibility and Replicability in Research Across Disciplines

The open and transparent documentation of scientific processes has been established as a core antecedent of free knowledge. This also holds for generating robust insights in the scope of research projects. To convince academic peers and the public, the research process must be understandable and retraceable (reproducible), and repeatable (replicable) by others, precluding the inclusion of fluke findings into the canon of insights. In this contribution, we outline what reproducibility and replicability (R&R) could mean in the scope of different disciplines and traditions of research and which significance R&R has for generating insights in these fields. We draw on projects conducted in the scope of the Wikimedia "Open Science Fellows Program" (Fellowship Freies Wissen), an interdisciplinary, long-running funding scheme for projects contributing to open research practices. We identify twelve implemented projects from different disciplines which primarily focused on R&R, and multiple additional projects also touching on R&R. From these projects, we identify patterns and synthesize them into a roadmap of how research projects can achieve R&R across different disciplines. We further outline the ground covered by these projects and propose ways forward.

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are "evidence-based," clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as "evidence-based." In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions-an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed "TOP Guidelines for Clearinghouses" includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving "evidence-based" designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

Prevalence of responsible research practices among academics in The Netherlands

Background: Traditionally, research integrity studies have focused on research misbehaviors and their explanations. Over time, attention has shifted towards preventing questionable research practices and promoting responsible ones. However, data on the prevalence of responsible research practices, especially open methods, open codes and open data and their underlying associative factors, remains scarce. Methods: We conducted a web-based anonymized questionnaire, targeting all academic researchers working at or affiliated to a university or university medical center in The Netherlands, to investigate the prevalence and potential explanatory factors of 11 responsible research practices. Results: A total of 6,813 academics completed the survey, the results of which show that prevalence of responsible practices differs substantially across disciplines and ranks, with 99 percent avoiding plagiarism in their work but less than 50 percent pre-registering a research protocol. Arts and humanities scholars as well as PhD candidates and junior researchers engaged less often in responsible research practices. Publication pressure negatively affected responsible practices, while mentoring, scientific norms subscription and funding pressure stimulated them. Conclusions: Understanding the prevalence of responsible research practices across disciplines and ranks, as well as their associated explanatory factors, can help to systematically address disciplinary- and academic rank-specific obstacles, and thereby facilitate responsible conduct of research.

Prevalence of responsible research practices among academics in The Netherlands

Background: Traditionally, research integrity studies have focused on research misbehaviors and their explanations. Over time, attention has shifted towards preventing questionable research practices and promoting responsible ones. However, data on the prevalence of responsible research practices, especially open methods, open codes and open data and their underlying associative factors, remains scarce. Methods: We conducted a web-based anonymized questionnaire, targeting all academic researchers working at or affiliated to a university or university medical center in The Netherlands, to investigate the prevalence and potential explanatory factors of 11 responsible research practices. Results: A total of 6,813 academics completed the survey, the results of which show that prevalence of responsible practices differs substantially across disciplines and ranks, with 99 percent avoiding plagiarism in their work but less than 50 percent pre-registering a research protocol. Arts and humanities scholars as well as PhD candidates and junior researchers engaged less often in responsible research practices. Publication pressure negatively affected responsible practices, while mentoring, scientific norms subscription and funding pressure stimulated them. Conclusions: Understanding the prevalence of responsible research practices across disciplines and ranks, as well as their associated explanatory factors, can help to systematically address disciplinary- and academic rank-specific obstacles, and thereby facilitate responsible conduct of research.

Prevalence of questionable research practices, research misconduct and their potential explanatory factors: A survey among academic researchers in The Netherlands

Prevalence of research misconduct, questionable research practices (QRPs) and their associations with a range of explanatory factors has not been studied sufficiently among academic researchers. The National Survey on Research Integrity targeted all disciplinary fields and academic ranks in the Netherlands. It included questions about engagement in fabrication, falsification and 11 QRPs over the previous three years, and 12 explanatory factor scales. We ensured strict identity protection and used the randomized response method for questions on research misconduct. 6,813 respondents completed the survey. Prevalence of fabrication was 4.3% (95% CI: 2.9, 5.7) and of falsification 4.2% (95% CI: 2.8, 5.6). Prevalence of QRPs ranged from 0.6% (95% CI: 0.5, 0.9) to 17.5% (95% CI: 16.4, 18.7) with 51.3% (95% CI: 50.1, 52.5) of respondents engaging frequently in at least one QRP. Being a PhD candidate or junior researcher increased the odds of frequently engaging in at least one QRP, as did being male. Scientific norm subscription (odds ratio (OR) 0.79; 95% CI: 0.63, 1.00) and perceived likelihood of detection by reviewers (OR 0.62, 95% CI: 0.44, 0.88) were associated with engaging in less research misconduct. Publication pressure was associated with more often engaging in one or more QRPs frequently (OR 1.22, 95% CI: 1.14, 1.30). We found higher prevalence of misconduct than earlier surveys. Our results suggest that greater emphasis on scientific norm subscription, strengthening reviewers in their role as gatekeepers of research quality and curbing the "publish or perish" incentive system promotes research integrity.

Transparent, Open, and Reproducible Prevention Science

The field of prevention science aims to understand societal problems, identify effective interventions, and translate scientific evidence into policy and practice. There is growing interest among prevention scientists in the potential for transparency, openness, and reproducibility to facilitate this mission by providing opportunities to align scientific practice with scientific ideals, accelerate scientific discovery, and broaden access to scientific knowledge. The overarching goal of this manuscript is to serve as a primer introducing and providing an overview of open science for prevention researchers. In this paper, we discuss factors motivating interest in transparency and reproducibility, research practices associated with open science, and stakeholders engaged in and impacted by open science reform efforts. In addition, we discuss how and why different types of prevention research could incorporate open science practices, as well as ways that prevention science tools and methods could be leveraged to advance the wider open science movement. To promote further discussion, we conclude with potential reservations and challenges for the field of prevention science to address as it transitions to greater transparency, openness, and reproducibility. Throughout, we identify activities that aim to strengthen the reliability and efficiency of prevention science, facilitate access to its products and outputs, and promote collaborative and inclusive participation in research activities. By embracing principles of transparency, openness, and reproducibility, prevention science can better achieve its mission to advance evidence-based solutions to promote individual and collective well-being.

Challenges for assessing replicability in preclinical cancer biology

We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication – it is hard to assess whether reported findings are credible.

Challenges for assessing replicability in preclinical cancer biology

We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication - it is hard to assess whether reported findings are credible.

Aspiring to greater intellectual humility in science

The replication crisis in the social, behavioural and life sciences has spurred a reform movement aimed at increasing the credibility of scientific studies. Many of these credibility-enhancing reforms focus, appropriately, on specific research and publication practices. A less often mentioned aspect of credibility is the need for intellectual humility or being transparent about and owning the limitations of our work. Although intellectual humility is presented as a widely accepted scientific norm, we argue that current research practice does not incentivize intellectual humility. We provide a set of recommendations on how to increase intellectual humility in research articles and highlight the central role peer reviewers can play in incentivizing authors to foreground the flaws and uncertainty in their work, thus enabling full and transparent evaluation of the validity of research.

Limited engagement with transparent and open science standards in the policies of pain journals: a cross-sectional evaluation

Scientific progress requires transparency and openness. The ability to critique, replicate and implement scientific findings depends on the transparency of the study design and methods, and the open availability of study materials, data and code. Journals are key stakeholders in supporting transparency and openness. This study aimed to evaluate 10 highest ranked pain journals' authorship policies with respect to their support for transparent and open research practices. Two independent authors evaluated the journal policies (as at 27 May 2019) using three tools: the self-developed Transparency and Openness Evaluation Tool, the Centre for Open Science (COS) Transparency Factor and the International Committee of Medical Journal Editors (ICMJE) requirements for disclosure of conflicts of interest. We found that the journal policies had an overall low level of engagement with research transparency and openness standards. The median COS Transparency Factor score was 3.5 (IQR 2.8) of 29 possible points, and only 7 of 10 journals' stated requirements for disclosure of conflicts of interest aligned fully with the ICMJE recommendations. Improved transparency and openness of pain research has the potential to benefit all that are involved in generating and using research findings. Journal policies that endorse and facilitate transparent and open research practices will ultimately improve the evidence base that informs the care provided for people with pain.

Seven steps toward more transparency in statistical practice

We argue that statistical practice in the social and behavioural sciences benefits from transparency, a fair acknowledgement of uncertainty and openness to alternative interpretations. Here, to promote such a practice, we recommend seven concrete statistical procedures: (1) visualizing data; (2) quantifying inferential uncertainty; (3) assessing data preprocessing choices; (4) reporting multiple models; (5) involving multiple analysts; (6) interpreting results modestly; and (7) sharing data and code. We discuss their benefits and limitations, and provide guidelines for adoption. Each of the seven procedures finds inspiration in Merton's ethos of science as reflected in the norms of communalism, universalism, disinterestedness and organized scepticism. We believe that these ethical considerations-as well as their statistical consequences-establish common ground among data analysts, despite continuing disagreements about the foundations of statistical inference.

Replicability, Robustness, and Reproducibility in Psychological Science

Replication-an important, uncommon, and misunderstood practice-is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Where are the Self-Correcting Mechanisms in Science?

It is often said that science is self-correcting, but the replication crisis suggests that self-correction mechanisms have fallen short. How can we know whether a particular scientific field has effective self-correction mechanisms, that is, whether its findings are credible? The usual processes that supposedly provide mechanisms for scientific self-correction, such as journal-based peer review and institutional committees, have been inadequate. We describe more verifiable indicators of a field’s commitment to self-correction. These fall under the broad headings of 1) transparency, which is already the subject of many reform efforts and 2) critical appraisal, which has received less attention and which we focus on here. Only by obtaining Observable Self-Correction Indicators (OSCIs) can we begin to evaluate the claim that “science is self-correcting.” We expect that the veracity of this claim varies across fields and subfields, and suggest that some fields, such as psychology and biomedicine, fall far short of an appropriate level of transparency and, especially, critical appraisal. Fields without robust, verifiable mechanisms for transparency and critical appraisal cannot reasonably be said to be self-correcting, and thus do not warrant the credibility often imputed to science as a whole.

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

Exploring the Gray Area: Similarities and Differences in Questionable Research Practices (QRPs) Across Main Areas of Research

This paper explores the gray area of questionable research practices (QRPs) between responsible conduct of research and severe research misconduct in the form of fabrication, falsification, and plagiarism (Steneck in SEE 12(1): 53-57, 2006). Up until now, we have had very little knowledge of disciplinary similarities and differences in QRPs. The paper is the first systematic account of variances and similarities. It reports on the findings of a comprehensive study comprising 22 focus groups on practices and perceptions of QRPs across main areas of research. The paper supports the relevance of the idea of epistemic cultures (Knorr Cetina in: Epistemic cultures: how the sciences make knowledge, Harvard University Press, Cambridge, 1999), also when it comes to QRPs. It shows which QRPs researchers from different areas of research (humanities, social sciences, medical sciences, natural sciences, and technical sciences) report as the most severe and prevalent within their fields. Furthermore, it shows where in the research process these self-reported QRPs can be found. This is done by using a five-phase analytical model of the research process (idea generation, research design, data collection, data analysis, scientific publication and reporting). The paper shows that QRPs are closely connected to the distinct research practices within the different areas of research. Many QRPs can therefore only be found within one area of research, and QRPs that cut across main areas often cover relatively different practices. In a few cases, QRPs in one area are considered good research practice in another.

Improving open and rigorous science: ten key future research opportunities related to rigor, reproducibility, and transparency in scientific research

Background: As part of a coordinated effort to expand research activity around rigor, reproducibility, and transparency (RRT) across scientific disciplines, a team of investigators at the Indiana University School of Public Health-Bloomington hosted a workshop in October 2019 with international leaders to discuss key opportunities for RRT research. Objective: The workshop aimed to identify research priorities and opportunities related to RRT. Design: Over two-days, workshop attendees gave presentations and participated in three working groups: (1) Improving Education & Training in RRT, (2) Reducing Statistical Errors and Increasing Analytic Transparency, and (3) Looking Outward: Increasing Truthfulness and Accuracy of Research Communications. Following small-group discussions, the working groups presented their findings, and participants discussed the research opportunities identified. The investigators compiled a list of research priorities, which were circulated to all participants for feedback. Results: Participants identified the following priority research questions: (1) Can RRT-focused statistics and mathematical modeling courses improve statistics practice?; (2) Can specialized training in scientific writing improve transparency?; (3) Does modality (e.g. face to face, online) affect the efficacy RRT-related education?; (4) How can automated programs help identify errors more efficiently?; (5) What is the prevalence and impact of errors in scientific publications (e.g., analytic inconsistencies, statistical errors, and other objective errors)?; (6) Do error prevention workflows reduce errors?; (7) How do we encourage post-publication error correction?; (8) How does 'spin' in research communication affect stakeholder understanding and use of research evidence?; (9) Do tools to aid writing research reports increase comprehensiveness and clarity of research reports?; and (10) Is it possible to inculcate scientific values and norms related to truthful, rigorous, accurate, and comprehensive scientific reporting? Conclusion: Participants identified important and relatively unexplored questions related to improving RRT. This list may be useful to the scientific community and investigators seeking to advance meta-science (i.e. research on research).

Credibility of preprints: an interdisciplinary survey of researchers

Preprints increase accessibility and can speed scholarly communication if researchers view them as credible enough to read and use. Preprint services do not provide the heuristic cues of a journal's reputation, selection, and peer-review processes that, regardless of their flaws, are often used as a guide for deciding what to read. We conducted a survey of 3759 researchers across a wide range of disciplines to determine the importance of different cues for assessing the credibility of individual preprints and preprint services. We found that cues related to information about open science content and independent verification of author claims were rated as highly important for judging preprint credibility, and peer views and author information were rated as less important. As of early 2020, very few preprint services display any of the most important cues. By adding such cues, services may be able to help researchers better assess the credibility of preprints, enabling scholars to more confidently use preprints, thereby accelerating scientific communication and discovery.

Credibility of preprints: an interdisciplinary survey of researchers

Preprints increase accessibility and can speed scholarly communication if researchers view them as credible enough to read and use. Preprint services do not provide the heuristic cues of a journal's reputation, selection, and peer-review processes that, regardless of their flaws, are often used as a guide for deciding what to read. We conducted a survey of 3759 researchers across a wide range of disciplines to determine the importance of different cues for assessing the credibility of individual preprints and preprint services. We found that cues related to information about open science content and independent verification of author claims were rated as highly important for judging preprint credibility, and peer views and author information were rated as less important. As of early 2020, very few preprint services display any of the most important cues. By adding such cues, services may be able to help researchers better assess the credibility of preprints, enabling scholars to more confidently use preprints, thereby accelerating scientific communication and discovery.

Misconduct and Misbehavior Related to Authorship Disagreements in Collaborative Science

Scientific authorship serves to identify and acknowledge individuals who "contribute significantly" to published research. However, specific authorship norms and practices often differ within and across disciplines, labs, and cultures. As a consequence, authorship disagreements are commonplace in team research. This study aims to better understand the prevalence of authorship disagreements, those factors that may lead to disagreements, as well as the extent and nature of resulting misbehavior. Methods include an international online survey of researchers who had published from 2011 to 2015 (8364 respondents). Of the 6673 who completed the main questions pertaining to authorship disagreement and misbehavior, nearly half (46.6%) reported disagreements regarding authorship naming; and discipline, rank, and gender had significant effects on disagreement rates. Paradoxically, researchers in multidisciplinary teams that typically reflect a range of norms and values, were less likely to have faced disagreements regarding authorship. Respondents reported having witnessed a wide range of misbehavior including: instances of hostility (24.6%), undermining of a colleague's work during meetings/talks (16.4%), cutting corners on research (8.3%), sabotaging a colleague's research (6.4%), or producing fraudulent work to be more competitive (3.3%). These findings suggest that authorship disputes may contribute to an unhealthy competitive dynamic that can undermine researchers' wellbeing, team cohesion, and scientific integrity.

Research integrity: environment, experience, or ethos?

Background:

Research integrity has gained attention in the general public as well as in the research community. We wanted to investigate knowledge, attitudes, and practices amongst researchers that have recently finished their PhD and compare this to their responses during their PhD fellowship. In particular, we wanted to investigate whether their attitudes are related to their experiences of their immediate research environment.

Material and method:

Researchers ( n = 86) awarded the PhD degree at the Faculty of Medicine at the University of Oslo in 2016 were invited to answer a questionnaire about knowledge, attitudes, and actions related to scientific dishonesty. Seventy-two responded (83.7%). The results were compared with results among first-year doctoral students who responded to the same questionnaire during 2010–2017.

Results:

Overall, 13% of PhDs reported that they knew of people in their immediate research environment who had committed serious forms of scientific dishonesty. A small percentage of PhDs (1.4%) indicated that they themselves had committed such acts. About 3% of the candidates had experienced pressure to commit serious forms of dishonesty and nearly a third of respondents had experienced unethical pressure with respect to authorship during the course of their fellowship. Thirteen percent reported that they had experienced unethical pressure in relation to other forms of dishonesty and 11% had experienced the consequences of some form of scientific dishonesty. Eighteen percent of the respondents believed that one or more actions, which in the literature were perceived as scientific misconduct, were not wrong. We find a connection between attitudes and the perceived research integrity of their research environment. The results also show a difference between PhD students and graduated PhDs in terms of scientific dishonesty. In some areas, the PhDs’ norms are stricter, such as for the use of statistical analysis methods, while there is little change in others, such as in misconduct in order to expedite publications.

Conclusion:

Many PhDs knew about serious forms of scientific misconduct from the research environment in which they are trained, and some also report misconduct themselves. Some experienced pressure to serious forms of misconduct and a large proportion of the respondents had experienced unethical pressure with respect to authorship during their fellowship. Attitudes change during the PhD studies, but ambiguously. Scientific misconduct seems to be an environmental issue as much as a matter of personal integrity.

CORP: Practical tools for improving experimental design and reporting of laboratory studies of cardiovascular physiology and metabolism

The exercise consisted of: 1) a short survey to acquire baseline data on current practices regarding the conduct of animal studies, 2) a series of presentations for promoting awareness and providing advice and practical tools for improving experimental design, and 3) a follow-up survey 12 mo later to assess whether practices had changed. The surveys were compulsory for responsible investigators (n = 16; paired data presented). Other investigators named on animal ethics applications were encouraged to participate (2017, total of 36 investigators; 2018, 37 investigators). The major findings to come from the exercise included 1) a willingness of investigators to make changes when provided with knowledge/tools and solutions that were relatively simple to implement (e.g., proportion of responsible investigators showing improved practices using a structured method for randomization was 0.44, 95% CI (0.19; 0.70), P = 0.003, and deidentifying drugs/interventions was 0.40, 95% CI (0.12; 0.68), P = 0.010); 2) resistance to change if this involved more personnel and time (e.g., as required for allocation concealment); and 3) evidence that changes to long-term practices ("habits") require time and follow-up. Improved practices could be verified based on changes in reporting within publications or documented evidence provided during laboratory visits. In summary, this exercise resulted in changed attitudes, practices, and reporting, but continued follow-up, monitoring, and incentives are required. Efforts to improve experimental rigor will reduce bias and will lead to findings with the greatest translational potential.NEW & NOTEWORTHY The goal of this exercise was to encourage preclinical researchers to improve the quality of their cardiac and metabolic animal studies by 1) increasing awareness of concerns, which can arise from suboptimal experimental designs; 2) providing knowledge, tools, and templates to overcome bias; and 3) conducting two short surveys over 12 mo to monitor change. Improved practices were identified for the uptake of structured methods for randomization, and de-identifying interventions/drugs.Listen to this article's corresponding podcast at https://ajpheart.podbean.com/e/experimental-design-survey-training-practical-tools/.

The lab management practices of "Research Exemplars" that foster research rigor and regulatory compliance: A qualitative study of successful principal investigators

INTRODUCTION

Conducting rigorous scientific inquiry within the bounds of research regulation and acceptable practice requires a principal investigator to lead and manage research processes and personnel. This study explores the practices used by investigators nominated as exemplars of research excellence and integrity to produce rigorous, reproducible research and comply with research regulations.

METHODS

Using a qualitative research design, we interviewed 52 principal investigators working in the United States at top research universities and the National Institutes of Health Intramural Research Program. We solicited nominations of researchers meeting two criteria: (1) they are federally-funded researchers doing high-quality, high-impact research, and (2) have reputations for professionalism and integrity. Each investigator received an initial nomination addressing both criteria and at least one additional endorsement corroborating criteria 2. A panel of researchers and our research team reviewed the nominations to select finalists who were invited to participate. The cohort of "Research Exemplars" includes highly accomplished researchers in diverse scientific disciplines. The semi-structured interview questions asked them to describe the routine practices they employ to foster rigor and regulatory compliance. We used inductive thematic analysis to identify common practices.

RESULTS

The exemplars identified a core set of 8 practices and provided strategies for employing them. The practices included holding regular team meetings, encouraging shared ownership, providing supervision, ensuring adequate training, fostering positive attitudes about compliance, scrutinizing data and findings, and following standard operating procedures. Above all, the use of these practices aim to create a psychologically safe work environment in which lab members openly collaborate to scrutinize their work and share in accountability for rigorous, compliant research.

CONCLUSIONS

Researchers typically receive limited systematic training in how to lead and manage their research teams. Training and education for principal investigators should include essential leadership and management practices and strategies that support doing high-quality research with integrity.

Questionable research practices in student final theses - Prevalence, attitudes, and the role of the supervisor's perceived attitudes

Although questionable research practices (QRPs) and p-hacking have received attention in recent years, little research has focused on their prevalence and acceptance in students. Students are the researchers of the future and will represent the field in the future. Therefore, they should not be learning to use and accept QRPs, which would reduce their ability to produce and evaluate meaningful research. 207 psychology students and fresh graduates provided self-report data on the prevalence and predictors of QRPs. Attitudes towards QRPs, belief that significant results constitute better science or lead to better grades, motivation, and stress levels were predictors. Furthermore, we assessed perceived supervisor attitudes towards QRPs as an important predictive factor. The results were in line with estimates of QRP prevalence from academia. The best predictor of QRP use was students' QRP attitudes. Perceived supervisor attitudes exerted both a direct and indirect effect via student attitudes. Motivation to write a good thesis was a protective factor, whereas stress had no effect. Students in this sample did not subscribe to beliefs that significant results were better for science or their grades. Such beliefs further did not impact QRP attitudes or use in this sample. Finally, students engaged in more QRPs pertaining to reporting and analysis than those pertaining to study design. We conclude that supervisors have an important function in shaping students' attitudes towards QRPs and can improve their research practices by motivating them well. Furthermore, this research provides some impetus towards identifying predictors of QRP use in academia.

From Outcome to Process Focus: Fostering a More Robust Psychological Science Through Registered Reports and Results-Blind Reviewing

A variety of alternative mechanisms, strategies, and “ways of doing” have been proposed for improving the rigor and robustness of published research in the psychological sciences in recent years. In this article, we describe two existing but underused publication models—registered reporting (RR) and results-blind reviewing (RBR)—that we believe would contribute in important ways to improving both the conduct and evaluation of psychological research. We first outline the procedures and distinguishing features of both publication pathways and note their value for promoting positive changes to current scientific practices. We posit that a significant value of RR and RBR is their potential to promote a greater focus on the research process (i.e., how and why research is conducted) relative to research outcomes (i.e., what was observed or concluded from research). We conclude by discussing what we perceive to be five common beliefs about RR and RBR practices and attempt to provide a balanced perspective of the realities likely to be experienced with these systems.

Associations between attitudes towards scientific misconduct and self-reported behavior

We investigate the relationship between doctoral students' attitudes towards scientific misconduct and their self-reported behavior. 203 questionnaires were distributed to doctoral candidates at the Faculty of Medicine, University of Oslo 2016/2017. The response rate was 74%. The results show a correlation between attitudes towards misconduct and self-reported problematic behaviors among doctoral students in biomedicine. The four most common reported misbehaviors are adding author(s) who did not qualify for authorship (17.9%), collecting more data after seeing that the results were almost statistically significant (11.8%), turning a blind eye to colleagues' use of flawed data or questionable interpretation of data (11.2%), and reporting an unexpected finding as having been hypothesized from the start (10.5%). We find correlations between scientific misbehavior and the location of undergraduate studies and whether the respondents have had science ethics lectures previously. The study provides evidence for the concurrent validity of the two instruments used to measure attitudes and behavior, i.e. the Kalichman scale and the Research Misbehavior Severity Score (RMSS). Although the direction of causality between attitudes and misbehavior cannot be determined in this study the correlation between the two indicates that it can be important to engender the right attitudes in early career researchers.

The status of research ethics in social work

Research ethics provide important and necessary standards related to the conduct and dissemination of research. To better understand the current state of research ethics discourse in social work, a systematic literature search was undertaken and numbers of publications per year were compared between STEM, social science, and social work disciplines. While many professions have embraced the need for discipline-specific research ethics subfield development, social work has remained absent. Low publication numbers, compared to other disciplines, were noted for the years (2006-2016) included in the study. Social work published 16 (1%) of the 1409 articles included in the study, contributing 3 (>1%) for each of the disciplines highest producing years (2011 and 2013). Comparatively, psychology produced 75 (5%) articles, psychiatry produced 64 (5%) articles, and nursing added 50 (4%) articles. The STEM disciplines contributed 956 (68%) articles between 2006 and 2016, while social science produced 453 (32%) articles. Examination of the results is provided in an extended discussion of several misconceptions about research ethics that may be found in the social work profession. Implications and future directions are provided, focusing on the need for increased engagement, education, research, and support for a new subfield of social work research ethics.

The Role of Culture and Acculturation in Researchers' Perceptions of Rules in Science

Successfully navigating the norms of a society is a complex task that involves recognizing diverse kinds of rules as well as the relative weight attached to them. In the United States (U.S.), different kinds of rules-federal statutes and regulations, scientific norms, and professional ideals-guide the work of researchers. Penalties for violating these different kinds of rules and norms can range from the displeasure of peers to criminal sanctions. We proposed that it would be more difficult for researchers working in the U.S. who were born in other nations to distinguish the seriousness of violating rules across diverse domains. We administered a new measure, the evaluating rules in science task (ERST), to National Institutes of Health-funded investigators (101 born in the U.S. and 102 born outside of the U.S.). The ERST assessed perceptions of the seriousness of violating research regulations, norms, and ideals, and allowed us to calculate the degree to which researchers distinguished between the seriousness of each rule category. The ERST also assessed researchers' predictions of the seriousness that research integrity officers (RIOs) would assign to the rules. We compared researchers' predictions to the seriousness ratings of 112 RIOs working at U.S. research-intensive universities. U.S.-born researchers were significantly better at distinguishing between the seriousness of violating federal research regulations and violating ideals of science, and they were more accurate in their predictions of the views of RIOs. Acculturation to the U.S. moderated the effects of nationality on accuracy. We discuss the implications of these findings in terms of future research and education.

The preregistration revolution

Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes-a process called preregistration. Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting. Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.

Data Sharing in Psychology: A Survey on Barriers and Preconditions

Despite its potential to accelerate academic progress in psychological science, public data sharing remains relatively uncommon. In order to discover the perceived barriers to public data sharing and possible means for lowering them, we conducted a survey, which elicited responses from 600 authors of articles in psychology. The results confirmed that data are shared only infrequently. Perceived barriers included respondents’ belief that sharing is not a common practice in their fields, their preference to share data only upon request, their perception that sharing requires extra work, and their lack of training in sharing data. Our survey suggests that strong encouragement from institutions, journals, and funders will be particularly effective in overcoming these barriers, in combination with educational materials that demonstrate where and how data can be shared effectively.

Forschungstransparenz als hohes wissenschaftliches Gut stärken

Zusammenfassung. Groß angelegte Replikationsprojekte der letzten Jahre legen ein aus unserer Sicht beunruhigendes Ausmaß an nicht-replizierbaren Befunden in der wissenschaftlichen Literatur nahe, sowohl in der Psychologie als auch in anderen Disziplinen. Basierend auf einer Analyse einiger Ursachen dieser Situation argumentieren wir, dass der Wandel hin zu einer offenen Wissenschaft („Open Science“) eine Konsequenz aus der Glaubwürdigkeitskrise sein muss. Wir plädieren für konkrete und machbare Änderungen in den Arbeitseinheiten und Instituten vor Ort, und zeigen exemplarisch, welche Schritte am Department Psychologie der Ludwig-Maximilians-Universität München umgesetzt wurden. Diese Schritte betreffen Anreizstrukturen, die Forschungskultur, die Lehre und die Verzahnung mit der Ethikkommission. Sie haben das Ziel, eine reproduzierbarere und glaubwürdigere Forschung zu unterstützen, ohne unnötige bürokratische Belastungen zu erzeugen.

The Weak Spots in Contemporary Science (and How to Fix Them)

In this review, the author discusses several of the weak spots in contemporary science, including scientific misconduct, the problems of post hoc hypothesizing (HARKing), outcome switching, theoretical bloopers in formulating research questions and hypotheses, selective reading of the literature, selective citing of previous results, improper blinding and other design failures, p-hacking or researchers' tendency to analyze data in many different ways to find positive (typically significant) results, errors and biases in the reporting of results, and publication bias. The author presents some empirical results highlighting problems that lower the trustworthiness of reported results in scientific literatures, including that of animal welfare studies. Some of the underlying causes of these biases are discussed based on the notion that researchers are only human and hence are not immune to confirmation bias, hindsight bias, and minor ethical transgressions. The author discusses solutions in the form of enhanced transparency, sharing of data and materials, (post-publication) peer review, pre-registration, registered reports, improved training, reporting guidelines, replication, dealing with publication bias, alternative inferential techniques, power, and other statistical tools.

Questionable research practices among italian research psychologists

A survey in the United States revealed that an alarmingly large percentage of university psychologists admitted having used questionable research practices that can contaminate the research literature with false positive and biased findings. We conducted a replication of this study among Italian research psychologists to investigate whether these findings generalize to other countries. All the original materials were translated into Italian, and members of the Italian Association of Psychology were invited to participate via an online survey. The percentages of Italian psychologists who admitted to having used ten questionable research practices were similar to the results obtained in the United States although there were small but significant differences in self-admission rates for some QRPs. Nearly all researchers (88%) admitted using at least one of the practices, and researchers generally considered a practice possibly defensible if they admitted using it, but Italian researchers were much less likely than US researchers to consider a practice defensible. Participants’ estimates of the percentage of researchers who have used these practices were greater than the self-admission rates, and participants estimated that researchers would be unlikely to admit it. In written responses, participants argued that some of these practices are not questionable and they have used some practices because reviewers and journals demand it. The similarity of results obtained in the United States, this study, and a related study conducted in Germany suggest that adoption of these practices is an international phenomenon and is likely due to systemic features of the international research and publication processes.

Who Believes in the Storybook Image of the Scientist?

Do lay people and scientists themselves recognize that scientists are human and therefore prone to human fallibilities such as error, bias, and even dishonesty? In a series of three experimental studies and one correlational study (total N = 3,278) we found that the “storybook image of the scientist” is pervasive: American lay people and scientists from over 60 countries attributed considerably more objectivity, rationality, open-mindedness, intelligence, integrity, and communality to scientists than to other highly-educated people. Moreover, scientists perceived even larger differences than lay people did. Some groups of scientists also differentiated between different categories of scientists: established scientists attributed higher levels of the scientific traits to established scientists than to early-career scientists and Ph.D. students, and higher levels to Ph.D. students than to early-career scientists. Female scientists attributed considerably higher levels of the scientific traits to female scientists than to male scientists. A strong belief in the storybook image and the (human) tendency to attribute higher levels of desirable traits to people in one’s own group than to people in other groups may decrease scientists’ willingness to adopt recently proposed practices to reduce error, bias and dishonesty in science.

Researcher perceptions of ethical guidelines and codes of conduct

Ethical codes of conduct exist in almost every profession. Field-specific codes of conduct have been around for decades, each articulating specific ethical and professional guidelines. However, there has been little empirical research on researchers' perceptions of these codes of conduct. In the present study, we interviewed faculty members in six research disciplines and identified five themes bearing on the circumstances under which they use ethical guidelines and the underlying reasons for not adhering to such guidelines. We then identify problems with the manner in which codes of conduct in academia are constructed and offer solutions for overcoming these problems.

[Research misconduct: Knowledge, actions and attitudes of PhD candidates]

BACKGROUND Increasing attention is being paid to research misconduct in academic journals and in the media, but we know relatively little about its extent or attitudes to research misconduct, or how these are changing. This study therefore aims to investigate PhD candidates' knowledge, own actions and attitudes to specific forms of research misconduct.MATERIAL AND METHOD In autumn 2015, an anonymous questionnaire survey was distributed to all participants in the introductory course for PhD candidates at the Faculty of Medicine at the University of Oslo.RESULTS Altogether 77 PhD candidates (79 %) responded to the questionnaire. A total of 62 % conducted clinical research and 25 % conducted basic research. Around one in four had heard about serious forms of research misconduct in the previous year, and around 4 % were aware of various forms of serious research misconduct in their own department in the previous year. Compared to earlier studies, an increasing number (16 %) responded that they had been subjected to unethical pressure with regard to inclusion or order of authors. Approximately two-thirds were uncertain of whether their department had written policies for academic conduct. One-third of PhD candidates did not disassociate themselves from actions that are generally viewed as scientific misconduct. One-tenth thought it acceptable to falsify or fabricate data in order to expedite publication, one-fifth did not object to taking the credit for others' ideas, and almost half did not believe it was wrong to attempt a number of methods of analysis until one arrived at a significant answer.INTERPRETATION PhD candidates at the Faculty of Medicine were aware of research misconduct, both generally and from their own department. They themselves reported some type of scientific misconduct, and a large majority were uncertain of their department's guidelines. Some of the candidates also accepted several forms of research misconduct.

Research practices and assessment of research misconduct

This article discusses the responsible conduct of research, questionable research practices, and research misconduct. Responsible conduct of research is often defined in terms of a set of abstract, normative principles, professional standards, and ethics in doing research. In order to accommodate the normative principles of scientific research, the professional standards, and a researcher’s moral principles, transparent research practices can serve as a framework for responsible conduct of research. We suggest a “prune-and-add” project structure to enhance transparency and, by extension, responsible conduct of research. Questionable research practices are defined as practices that are detrimental to the research process. The prevalence of questionable research practices remains largely unknown, and reproducibility of findings has been shown to be problematic. Questionable practices are discouraged by transparent practices because practices that arise from them will become more apparent to scientific peers. Most effective might be preregistrations of research design, hypotheses, and analyses, which reduce particularism of results by providing an a priori research scheme. Research misconduct has been defined as fabrication, falsification, and plagiarism (FFP), which is clearly the worst type of research practice. Despite it being clearly wrong, it can be approached from a scientific and legal perspective. The legal perspective sees research misconduct as a form of white-collar crime. The scientific perspective seeks to answer the following question: “Were results invalidated because of the misconduct?” We review how misconduct is typically detected, how its detection can be improved, and how prevalent it might be. Institutions could facilitate detection of data fabrication and falsification by implementing data auditing. Nonetheless, the effect of misconduct is pervasive: many retracted articles are still cited after the retraction has been issued.

Main points

Researchers systematically evaluate their own conduct as more responsible than colleagues, but not as responsible as they would like.

Transparent practices, facilitated by the Open Science Framework, help embody scientific norms that promote responsible conduct.

Questionable research practices harm the research process and work counter to the generally accepted scientific norms, but are hard to detect.

Research misconduct requires active scrutiny of the research community because editors and peer-reviewers do not pay adequate attention to detecting this. Tips are given on how to improve your detection of potential problems.