Opinion: Is science really facing a reproducibility crisis, and do we need it to?

Improving open and rigorous science: ten key future research opportunities related to rigor, reproducibility, and transparency in scientific research

Background: As part of a coordinated effort to expand research activity around rigor, reproducibility, and transparency (RRT) across scientific disciplines, a team of investigators at the Indiana University School of Public Health-Bloomington hosted a workshop in October 2019 with international leaders to discuss key opportunities for RRT research. Objective: The workshop aimed to identify research priorities and opportunities related to RRT. Design: Over two-days, workshop attendees gave presentations and participated in three working groups: (1) Improving Education & Training in RRT, (2) Reducing Statistical Errors and Increasing Analytic Transparency, and (3) Looking Outward: Increasing Truthfulness and Accuracy of Research Communications. Following small-group discussions, the working groups presented their findings, and participants discussed the research opportunities identified. The investigators compiled a list of research priorities, which were circulated to all participants for feedback. Results: Participants identified the following priority research questions: (1) Can RRT-focused statistics and mathematical modeling courses improve statistics practice?; (2) Can specialized training in scientific writing improve transparency?; (3) Does modality (e.g. face to face, online) affect the efficacy RRT-related education?; (4) How can automated programs help identify errors more efficiently?; (5) What is the prevalence and impact of errors in scientific publications (e.g., analytic inconsistencies, statistical errors, and other objective errors)?; (6) Do error prevention workflows reduce errors?; (7) How do we encourage post-publication error correction?; (8) How does 'spin' in research communication affect stakeholder understanding and use of research evidence?; (9) Do tools to aid writing research reports increase comprehensiveness and clarity of research reports?; and (10) Is it possible to inculcate scientific values and norms related to truthful, rigorous, accurate, and comprehensive scientific reporting? Conclusion: Participants identified important and relatively unexplored questions related to improving RRT. This list may be useful to the scientific community and investigators seeking to advance meta-science (i.e. research on research).

The polymorphism of Hydra microsatellite sequences provides strain-specific signatures

Hydra are freshwater polyps widely studied for their amazing regenerative capacity, adult stem cell populations, low senescence and value as ecotoxicological marker. Many wild-type strains of H. vulgaris have been collected worldwide and maintained effectively under laboratory conditions by asexual reproduction, while stable transgenic lines have been continuously produced since 2006. Efforts are now needed to ensure the genetic characterization of all these strains, which despite similar morphologies, show significant variability in their response to gene expression silencing procedures, pharmacological treatments or environmental conditions. Here, we established a rapid and reliable procedure at the single polyp level to produce via PCR amplification of three distinct microsatellite sequences molecular signatures that distinguish between Hydra strains and species. The TG-rich region of an uncharacterized gene (ms-c25145) helps to distinguish between Eurasian H. vulgaris-Pallas strains (Hm-105, Basel1, Basel2 and reg-16), between Eurasian and North American H. vulgaris strains (H. carnea, AEP), and between the H. vulgaris and H. oligactis species. The AT-rich microsatellite sequences located in the AIP gene (Aryl Hydrocarbon Receptor Interaction Protein, ms-AIP) also differ between Eurasian and North American H. vulgaris strains. Finally, the AT-rich microsatellite located in the Myb-Like cyclin D-binding transcription factor1 gene (ms-DMTF1) gene helps to distinguish certain transgenic AEP lines. This study shows that the analysis of microsatellite sequences, which is capable of tracing genomic variations between closely related lineages of Hydra, provides a sensitive and robust tool for characterizing the Hydra strains.

Is “the time ripe” for quantitative research on misconduct in science?

Misconduct in science is a timely and substantively important problem in the social study of science. But in the absence of comprehensive and reliable data needed for analysis, formidable obstacles stand in the way of its being studied quantitively. Accessible databases, including government data, are flawed, while undertaking new data collection presents its own problems. First, little is known about biases in official government reports. Second, official reports exclude classes of malfeasance other than fabrication, falsification, and plagiarism of evidence (FFP). Third, while drawing on official data is expedient, available official information is thin; it tells little about miscreants and fails to identify potential causes of their actions and the environments in which misconduct occurred. Fourth, it also fails the test of permitting estimates to be made of populations at risk, making it impossible to calculate incidence. A healthy dose of skepticism is in order in evaluating both the findings of current quantitative studies and of proposals for remediation.

Find the Needle in the Haystack, Then Find It Again: Replication and Validation in the 'Omics Era

Advancements in high-throughput technologies have made it feasible to study thousands of biological pathways simultaneously for a holistic assessment of health and disease risk via ‘omics platforms. A major challenge in ‘omics research revolves around the reproducibility of findings—a feat that hinges upon balancing false-positive associations with generalizability. Given the foundational role of reproducibility in scientific inference, replication and validation of ‘omics findings are cornerstones of this effort. In this narrative review, we define key terms relevant to replication and validation, present issues surrounding each concept with historical and contemporary examples from genomics (the most well-established and upstream ‘omics), discuss special issues and unique considerations for replication and validation in metabolomics (an emerging field and most downstream ‘omics for which best practices remain yet to be established), and make suggestions for future research leveraging multiple ‘omics datasets.

Responsible Science and Research Animal Use

This issue of the ILAR Journal focuses on the topic of responsible science as it relates to animal research. We start with the concept of the scientist as a responsible citizen and then move through multiple phases of research including careful experimental planning, reporting, and incorporation of laboratory animal science. The work of the Institutional Animal Care and Use Committee (IACUC) or animal ethical/oversight body in reviewing both animal use and contributing to scientific excellence is explored. Additional topics include protection of animal handlers from multiple experimental hazards, use of agricultural animals and wildlife studies, regulatory ambiguities, and harmonization of animal research. Rounding out the issue is a discussion of how animal care and use programs can enhance animal welfare while mitigating regulatory burden, and our responsibility to clearly communicate the ethical use of animals in advancing biomedical research. A deeper understanding of these topics can assist scientists in simultaneously advancing their research and animal welfare.

Workshop on reproducibility in research

In recent years, concern about research reproducibility has increased dramatically for scientists, funders of research, and the general public. With a view to explicitly address what is often called a reproducibility crisis and putting the focus on research being done by individual trainees, a two-hour workshop was developed and introduced into six courses at UC San Diego. Participation in the workshop resulted in a statistically significant increase in the number of different types of strategies identified by the trainees for fostering reproducibility. The findings are consistent with having increased awareness of strategies to promote reproducibility.

COPO: a metadata platform for brokering FAIR data in the life sciences

Scientific innovation is increasingly reliant on data and computational resources. Much of today’s life science research involves generating, processing, and reusing heterogeneous datasets that are growing exponentially in size. Demand for technical experts (data scientists and bioinformaticians) to process these data is at an all-time high, but these are not typically trained in good data management practices. That said, we have come a long way in the last decade, with funders, publishers, and researchers themselves making the case for open, interoperable data as a key component of an open science philosophy. In response, recognition of the FAIR Principles (that data should be Findable, Accessible, Interoperable and Reusable) has become commonplace. However, both technical and cultural challenges for the implementation of these principles still exist when storing, managing, analysing and disseminating both legacy and new data. COPO is a computational system that attempts to address some of these challenges by enabling scientists to describe their research objects (raw or processed data, publications, samples, images, etc.) using community-sanctioned metadata sets and vocabularies, and then use public or institutional repositories to share them with the wider scientific community. COPO encourages data generators to adhere to appropriate metadata standards when publishing research objects, using semantic terms to add meaning to them and specify relationships between them. This allows data consumers, be they people or machines, to find, aggregate, and analyse data which would otherwise be private or invisible, building upon existing standards to push the state of the art in scientific data dissemination whilst minimising the burden of data publication and sharing.

Professionalism in Science: Competence, Autonomy, and Service

Some of the most significant policy responses to cases of fraudulent and questionable conduct by scientists have been to strengthen professionalism among scientists, whether by codes of conduct, integrity boards, or mandatory research integrity training programs. Yet there has been little systematic discussion about what professionalism in scientific research should mean. In this paper I draw on the sociology of the professions and on data comparing codes of conduct in science to those in the professions, in order to examine what precisely the model of professionalism implies for scientific research. I argue that professionalism, more than any other single organizational logic, is appropriate for scientific research, and that codes of conduct for scientists should strengthen statements concerning scientific autonomy and competence, as well as the scientific service ideal.

Atypical Electrophysiological Indices of Eyes-Open and Eyes-Closed Resting-State in Children and Adolescents with ADHD and Autism

Investigating electrophysiological measures during resting-state might be useful to investigate brain functioning and responsivity in individuals under diagnostic assessment for attention deficit hyperactivity disorder (ADHD) and autism. EEG was recorded in 43 children with or without ADHD and autism, during a 4-min-long resting-state session which included an eyes-closed and an eyes-open condition. We calculated and analyzed occipital absolute and relative spectral power in the alpha frequency band (8-12 Hz), and alpha reactivity, conceptualized as the difference in alpha power between eyes-closed and eyes-open conditions. Alpha power was increased during eyes-closed compared to eyes-open resting-state. While absolute alpha power was reduced in children with autism, relative alpha power was reduced in children with ADHD, especially during the eyes-closed condition. Reduced relative alpha reactivity was mainly associated with lower IQ and not with ADHD or autism. Atypical brain functioning during resting-state seems differently associated with ADHD and autism, however further studies replicating these results are needed; we therefore suggest involving research groups worldwide by creating a shared and publicly available repository of resting-state EEG data collected in people with different psychological, psychiatric, or neurodevelopmental conditions, including ADHD and autism.

A simple model suggesting economically rational sample-size choice drives irreproducibility

Several systematic studies have suggested that a large fraction of published research is not reproducible. One probable reason for low reproducibility is insufficient sample size, resulting in low power and low positive predictive value. It has been suggested that insufficient sample-size choice is driven by a combination of scientific competition and 'positive publication bias'. Here we formalize this intuition in a simple model, in which scientists choose economically rational sample sizes, balancing the cost of experimentation with income from publication. Specifically, assuming that a scientist's income derives only from 'positive' findings (positive publication bias) and that individual samples cost a fixed amount, allows to leverage basic statistical formulas into an economic optimality prediction. We find that if effects have i) low base probability, ii) small effect size or iii) low grant income per publication, then the rational (economically optimal) sample size is small. Furthermore, for plausible distributions of these parameters we find a robust emergence of a bimodal distribution of obtained statistical power and low overall reproducibility rates, both matching empirical findings. Finally, we explore conditional equivalence testing as a means to align economic incentives with adequate sample sizes. Overall, the model describes a simple mechanism explaining both the prevalence and the persistence of small sample sizes, and is well suited for empirical validation. It proposes economic rationality, or economic pressures, as a principal driver of irreproducibility and suggests strategies to change this.

A Strong Test of Atomically Detailed Models of Molecular Adsorption in Zeolites Using Multilaboratory Experimental Data for CO2 Adsorption in Ammonium ZSM-5

A recent international interlaboratory study led by the U.S. National Institute of Standards (NIST) reported CO₂ adsorption isotherms measured independently by 11 groups on reference material RM 8852, an ammonium ZSM-5 zeolite. Good reproducibility and high reliability of this experimental data provide a strong test for the ability of atomically detailed models to predict adsorption of CO₂ in zeolites. We developed force fields for CO₂ in ammonium zeolites based on first-principles calculations and also independently performed experiments with RM 8852 by microcalorimetry. At low pressures good agreement was obtained between predictions and experiments. At high pressures, however, deviations were observed. We show that the charge-balancing cations in the experimental material are the predominant source of the discrepancy between simulation and experiment at high pressures; the experimental sample treatment causes deammoniation. In addition, accounting for a small amount of noncrystalline mesoporosity in the zeolite brings predictions into much better agreement with experiments.

Perspectives for early-career researchers on plagiarism and scientific integrity

Lapses in scientific integrity, such as plagiarism, persist in the scientific realm. To be successful and contributory, early-career researchers (ECRs), including graduate students, need to be able to effectively navigate the literature, peer-review process, and scientific research with integrity. Here we discuss different aspects of scientific integrity related to ECRs. Our discussion centres on the concepts of plagiarism and intellectual property, predatory journals, aspects of peer review, transparency in publishing, and false advanced accreditations. Negative elements within these topics may be especially damaging to ECRs, who may be less familiar with the research landscape. We highlight the need for ECRs to approach scientific investigation cautiously and thoughtfully to promote integrity through critical thinking.

Does repeat synthesis in materials chemistry obey a power law?

Finding examples where experimental measurements have been repeated is a powerful strategy for assessing reproducibility of scientific data. Here, we collect quantitative data to assess how often synthesis of a newly reported material is repeated in the scientific literature. We present a simple power-law model for the frequency of repeat syntheses and assess the validity of this model using a specific class of materials, metal-organic frameworks (MOFs). Our data suggest that a power law describes the frequency of repeat synthesis of many MOFs but that a small number of "supermaterials" exist that have been replicated many times more than a power law would predict. Our results also hint that there are many repeat syntheses that have been performed but not reported in the literature, which suggests simple steps that could be taken to greatly increase the number of reports of replicate experiments in materials chemistry.

Effect Declines Are Systematic, Strong, and Ubiquitous: A Meta-Meta-Analysis of the Decline Effect in Intelligence Research

Empirical sciences in general and psychological science in particular are plagued by replicability problems and biased published effect sizes. Although dissemination bias-related phenomena such as publication bias, time-lag bias, or visibility bias are well-known and have been intensively studied, another variant of effect distorting mechanisms, so-called decline effects, have not. Conceptually, decline effects are rooted in low initial (exploratory) study power due to strategic researcher behavior and can be expected to yield overproportional effect declines. Although decline effects have been documented in individual meta-analytic investigations, systematic evidence for decline effects in the psychological literature remains to date unavailable. Therefore, we present in this meta-meta-analysis a systematic investigation of the decline effect in intelligence research. In all, data from 22 meta-analyses comprising 36 meta-analytical and 1,391 primary effect sizes (N = 697,000+) that have been published in the journal Intelligence were included in our analyses. Two different analytic approaches showed consistent evidence for a higher prevalence of cross-temporal effect declines compared to effect increases, yielding a ratio of about 2:1. Moreover, effect declines were considerably stronger when referenced to the initial primary study within a meta-analysis, yielding about twice the magnitude of effect increases. Effect misestimations were more substantial when initial studies had smaller sample sizes and reported larger effects, thus indicating suboptimal initial study power as the main driver of effect misestimations in initial studies. Post hoc study power comparisons of initial versus subsequent studies were consistent with this interpretation, showing substantially lower initial study power of declining, than of increasing effects. Our findings add another facet to the ever accumulating evidence about non-trivial effect misestimations in the scientific literature. We therefore stress the necessity for more rigorous protocols when it comes to designing and conducting primary research as well as reporting findings in exploratory and replication studies. Increasing transparency in scientific processes such as data sharing, (exploratory) study preregistration, but also self- (or independent) replication preceding the publication of exploratory findings may be suitable approaches to strengthen the credibility of empirical research in general and psychological science in particular.

Burning the straw man: What exactly is psychological science?

Problemification: Efendic and Van Zyl (2019) argue for following open access-based principles in IO psychology following the recent crises in psychological research. Among others, these refer to the failure to replicate empirical studies which cast doubt on the trustworthiness of what we believe to be psychological knowledge. However, saving knowledge is not the issue at stake: focusing on transparency and compliance to standards might solve some problems but not all.Implications: The crisis focuses our attention on what science is and particularly science in psychology and its related disciplines. Both the scientist–practitioner model of training psychologists and the quantitative–qualitative methods polarity reveal the influence of the received or positivistic view of science as characterised by quantification and measurement. Postmodern resistance to positivism feeds these polarities and conceals the true nature of psychological science.Purpose: This article argues for a realist conception of science that sustains a variety of methods, from interpretative and constructionist approaches to measurement. However, in this view, measurement is not a defining characteristic of science, but a way to find things out and the latter supports a critical process.Recommendations: Revising our understanding of science, thus moving beyond the received view to a realist one, is crucial to manage misconceptions about what counts as knowledge and as appropriate measures when our discipline is in the crossfire. Thus, Efendic and Van Zyl’s (2019) proposals make sense and can be taken on board where measurement as one of the ways to find things out is appropriate. However, realism supports a broader enterprise that can be called scientific because it involves a critical movement of claim and counter-claim while executing its taxonomical and explanatory tasks. Thus, the psychosocial researcher, when analysing discourse, for example, can also be regarded as a scientist.

MiSet RFC Standards: Defining a Universal Minimum Set of Standards Required for Reproducibility and Rigor in Research Flow Cytometry Experiments

Poor adherence to best practices, insufficient training, and pressure to produce data quickly may lead to publications of suboptimal biomedical research flow cytometry data, which contributes to the body of irreproducible research findings. In addition, documentation of compliance with best flow cytometry practices for submission, visualization, and publication of flow cytometry data is currently endorsed by very few scientific journals, which is particularly concerning as numerous peer-reviewed flow cytometry publications emphasize instrumentation, experimental design, and data analysis as important sources of variability. Guidelines and resources for adequate reporting, annotation and deposition of flow cytometry experiments are provided by MIFlowCyt and the FlowRepository database, and comprehensive expert recommendations covering principles and techniques of field-specific flow cytometry applications have been published. To facilitate the integration of quality-defining parameters into manuscript and grant submission and publication requirements across biomedical fields that rely on the use of flow-cytometry-based techniques, a single comprehensive yet easily and universally applicable document is needed. To produce such a list of gold-standard parameters that assess whether a research flow cytometry experiment has been planned, conducted, interpreted, and reported at the highest standard, a new initiative defining the minimum set of standards a robust and rigorous research flow experiment must fulfill (MiSet RFC Standards) was proposed at CYTO 2019. MiSet RFC Standards will integrate and simplify existing resources to provide a universal benchmark a flow cytometry experiment can easily be measured against. The goal of MiSET RFC Standards is its integration into peer-review and publication procedures through partnership with stakeholders, journals and publishers in biomedical and translational research. This article introduces the aims and anticipated timeline and discusses strategies for interdisciplinary consensus and implementation. A single-resource broadly applicable guideline will harmonize standards across different fields of biomedical research and lead to publication of more robust research findings. © 2019 International Society for Advancement of Cytometry.

Incorporating Laboratory Animal Science into Responsible Biomedical Research

Biomedical research has made great strides in the past century leading to rapid advances in human life expectancy, all derived from improved understanding, prevention, and treatment of many diseases and conditions. Research involving laboratory animals has played a significant role in this medical progress. However, there continues to be controversy surrounding the use of animals in research, and animal models have been questioned regarding their relevance to human conditions. While research fraud and questionable research practices could potentially contribute to this problem, we argue that a relative ignorance of laboratory animal science has contributed to the “uncontrolled vivarium experiment” that runs parallel to the more controlled scientific experiment. Several variables are discussed, including husbandry, animal environment, social housing, and more, that can contribute to this uncontrolled experiment, and that can simultaneously decrease quality of life for rodent test subjects when ignored. An argument is put forward that laboratory animal veterinarians and scientists can and should play an important role in better controlling such variables. Similarly, the laboratory animal veterinarian and scientist should play an important role in responsible science by addressing complex interdisciplinary challenges.

Data-Driven Materials Science: Status, Challenges, and Perspectives

Data-driven science is heralded as a new paradigm in materials science. In this field, data is the new resource, and knowledge is extracted from materials datasets that are too big or complex for traditional human reasoning-typically with the intent to discover new or improved materials or materials phenomena. Multiple factors, including the open science movement, national funding, and progress in information technology, have fueled its development. Such related tools as materials databases, machine learning, and high-throughput methods are now established as parts of the materials research toolset. However, there are a variety of challenges that impede progress in data-driven materials science: data veracity, integration of experimental and computational data, data longevity, standardization, and the gap between industrial interests and academic efforts. In this perspective article, the historical development and current state of data-driven materials science, building from the early evolution of open science to the rapid expansion of materials data infrastructures are discussed. Key successes and challenges so far are also reviewed, providing a perspective on the future development of the field.

Poly(I:C) source, molecular weight and endotoxin contamination affect dam and prenatal outcomes, implications for models of maternal immune activation

The viral mimetic polyinosinic:polycytidylic acid (poly(I:C)) is increasingly used to induce maternal immune activation (mIA) to model neurodevelopmental disorders (NDDs). Robust and reproducible phenotypes across studies are essential for the generation of models that will enhance our understanding of NDDs and enable the development of improved therapeutic strategies. However, differences in mIA-induced phenotypes using poly(I:C) have been widely observed, and this has prompted the reporting of useful and much needed methodological guidelines. Here, we perform a detailed investigation of molecular weight and endotoxin variations in poly(I:C) procured from two of the most commonly used suppliers, Sigma and InvivoGen. We demonstrate that endotoxin contamination and molecular weight differences in poly(I:C) composition lead to considerable variability in maternal IL-6 response in rats treated on gestational day (GD)15 and impact on fetal outcomes. Specifically, both endotoxin contamination and molecular weight predicted reductions in litter size on GD21. Further, molecular weight predicted a reduction in placental weight at GD21. While fetal body weight at GD21 was not affected by poly(I:C) treatment, male fetal brain weight was significantly reduced by poly(I:C), dependent on supplier. Our data are in agreement with recent reports of the importance of poly(I:C) molecular weight, and extend this work to demonstrate a key role of endotoxin on relevant phenotypic outcomes. We recommend that the source and batch numbers of poly(I:C) used should always be stated and that molecular weight variability and endotoxin contamination should be minimised for more robust mIA modelling.

Engineered nanomaterials: From their properties and applications, to their toxicity towards marine bivalves in a changing environment

As a consequence of their unique characteristics, the use of Engineered Nanomaterials (ENMs) is rapidly increasing in industrial, agricultural products, as well as in environmental technology. However, this fast expansion and use make likely their release into the environment with particular concerns for the aquatic ecosystems, which tend to be the ultimate sink for this type of contaminants. Considering the settling behaviour of particulates, benthic organisms are more likely to be exposed to these compounds. In this way, the present review aims to summarise the most recent data available from the literature on ENMs behaviour and fate in aquatic ecosystems, focusing on their ecotoxicological impacts towards marine and estuarine bivalves. The selection of ENMs presented here was based on the OECD's Working Party on Manufactured Nanomaterials (WPMN), which involves the safety testing and risk assessment of ENMs. Physical-chemical characteristics and properties, applications, environmental relevant concentrations and behaviour in aquatic environment, as well as their toxic impacts towards marine bivalves are discussed. Moreover, it is also identified the impacts derived from the simultaneous exposure of marine organisms to ENMs and climate changes as an ecologically relevant scenario.

No Replication, No Trust? How Low Replicability Influences Trust in Psychology

In the current psychological debate, low replicability of psychological findings is a central topic. While the discussion about the replication crisis has a huge impact on psychological research, we know less about how it impacts public trust in psychology. In this article, we examine whether low replicability damages public trust and how this damage can be repaired. Studies 1–3 provide correlational and experimental evidence that low replicability reduces public trust in psychology. Additionally, Studies 3–5 evaluate the effectiveness of commonly used trust-repair strategies such as information about increased transparency (Study 3), explanations for low replicability (Study 4), or recovered replicability (Study 5). We found no evidence that these strategies significantly repair trust. However, it remains possible that they have small but potentially meaningful effects, which could be detected with larger samples. Overall, our studies highlight the importance of replicability for public trust in psychology.

Open Up - the Mission Statement of the Control of Impulsive Action (Ctrl-ImpAct) Lab on Open Science

The present paper is the mission statement of the Control of Impulsive Action (Ctrl-ImpAct) Lab regarding Open Science. As early-career researchers (ECRs) in the lab, we first state our personal motivation to conduct research based on the principles of Open Science. We then describe how we incorporate four specific Open Science practices (i.e., Open Methodology, Open Data, Open Source, and Open Access) into our scientific workflow. In more detail, we explain how Open Science practices are embedded into the so-called 'co-pilot' system in our lab. The 'co-pilot' researcher is involved in all tasks of the 'pilot' researcher, that is designing a study, double-checking experimental and data analysis scripts, as well as writing the manuscript. The lab has set up this co-pilot system to increase transparency, reduce potential errors that could occur during the entire workflow, and to intensify collaborations between lab members. Finally, we discuss potential solutions for general problems that could arise when practicing Open Science.

Rinse and Repeat: Understanding the Value of Replication across Different Ways of Knowing

The increasing pursuit of replicable research and actual replication of research is a political project that articulates a very specific technology of accountability for science. This project was initiated in response to concerns about the openness and trustworthiness of science. Though applicable and valuable in many fields, here we argue that this value cannot be extended everywhere, since the epistemic content of fields, as well as their accountability infrastructures, differ. Furthermore, we argue that there are limits to replicability across all fields; but in some fields, including parts of the humanities, these limits severely undermine the value of replication to account for the value of research.

Ten Hot Topics around Scholarly Publishing

The changing world of scholarly communication and the emerging new wave of ‘Open Science’ or ‘Open Research’ has brought to light a number of controversial and hotly debated topics. Evidence-based rational debate is regularly drowned out by misinformed or exaggerated rhetoric, which does not benefit the evolving system of scholarly communication. This article aims to provide a baseline evidence framework for ten of the most contested topics, in order to help frame and move forward discussions, practices, and policies. We address issues around preprints and scooping, the practice of copyright transfer, the function of peer review, predatory publishers, and the legitimacy of ‘global’ databases. These arguments and data will be a powerful tool against misinformation across wider academic research, policy and practice, and will inform changes within the rapidly evolving scholarly publishing system.

Developing Reflection and Collaboration in Translational Medicine Toward Patients and Unmet Medical Needs

This perspective article aims to highlight the importance of values-driven personal reflection and collaboration for effective translational medicine training. We frame the dilemma in translational medicine and provide an approach for solution emphasizing collaboration and co-creation for innovative change in translational medicine. We cite the science in transition literature suggesting why personal reflection and a collaborative approach is important. We identify the problem with publication pressures and the bibliometric mindset. We focus on motivation to seek and find results that really matter for patients and individuals to maintain health in the real world. We review how the international EUREKA Institute for Translational Medicine (established in 2007) works with students, to harness their core values and develop personal growth skills to improve their leadership effectiveness, to work toward collaborative gain and potentially more meaningful results for patients and medical needs. We describe how the EUREKA Institute's unique setting, curriculum and hidden curriculum aspects effectively train program participants. The article highlights creating an immersive safe space, personal reflection, connection, structured brainstorming, group problem solving, collaboration and co-creation to facilitate innovation in translational medicine. The article relates program features to their theoretical underpinnings such as Theory U, Mediation Theory and Strategic Innovation Theory. The six authors from different global regions, ages, career stages, translational medicine contexts and years of attendance at the EUREKA Institute provide their reflections on training impact. Lessons learned and recommendations for research and application are discussed.

Understanding of researcher behavior is required to improve data reliability

BACKGROUND

A lack of data reproducibility ("reproducibility crisis") has been extensively debated across many academic disciplines.

RESULTS

Although a reproducibility crisis is widely perceived, conclusive data on the scale of the problem and the underlying reasons are largely lacking. The debate is primarily focused on methodological issues. However, examples such as the use of misidentified cell lines illustrate that the availability of reliable methods does not guarantee good practice. Moreover, research is often characterized by a lack of established methods. Despite the crucial importance of researcher conduct, research and conclusive data on the determinants of researcher behavior are widely missing.

CONCLUSION

Meta-research that establishes an understanding of the factors that determine researcher behavior is urgently needed. This knowledge can then be used to implement and iteratively improve measures that incentivize researchers to apply the highest standards, resulting in high-quality data.

Where is the "where" in the brain? A meta-analysis of neuroimaging studies on spatial cognition

Spatial representations are processed in the service of several different cognitive functions. The present study capitalizes on the Activation Likelihood Estimation (ALE) method of meta-analysis to identify: (a) the shared neural activations among spatial functions to reveal the "core" network of spatial processing; (b) the specific neural activations associated with each of these functions. Following PRISMA guidelines, a total of 133 fMRI and PET studies were included in the meta-analysis. The overall analysis showed that the core network of spatial processing comprises regions that are symmetrically distributed on both hemispheres and that include dorsal frontoparietal regions, presupplementary motor area, anterior insula, and frontal operculum. The specific analyses revealed the brain regions that are selectively recruited for each spatial function, such as the right temporoparietal junction for shift of spatial attention, the right parahippocampal gyrus, and the retrosplenial cortex for navigation and spatial long-term memory. The findings are integrated within a systematic review of the neuroimaging literature and a new neurocognitive model of spatial cognition is proposed.

Ten simple rules towards healthier research labs

The negative effects of extremely competitive academic and research environments on the performance and health of researchers are well known and common worldwide. The prevalence of these effects, particularly among early career researchers, calls for a more humane and people-centered way of working within research labs. Although there is growing concern about the urgent need for a better life-work balance when doing science, there are not many examples about how this could be achieved in practice. In this article, I introduce 10 simple rules to make the working environment of research labs more nurturing, collaborative, and people-centered. These rules are directed towards existing and future principal investigators (PIs) but will be of interest to anyone working in a research lab and/or dealing with how to improve working conditions for scientists.

Changing times for science and the public: Science journalists' roles for the responsible communication of science

The increasing number of corrections in the scientific record and the debate about reproducibility affect journalists’ reporting about science and thereby public opinion on scientists and research.

A theory and methodology to quantify knowledge

This article proposes quantitative answers to meta-scientific questions including 'how much knowledge is attained by a research field?', 'how rapidly is a field making progress?', 'what is the expected reproducibility of a result?', 'how much knowledge is lost from scientific bias and misconduct?', 'what do we mean by soft science?', and 'what demarcates a pseudoscience?'. Knowledge is suggested to be a system-specific property measured by K, a quantity determined by how much of the information contained in an explanandum is compressed by an explanans, which is composed of an information 'input' and a 'theory/methodology' conditioning factor. This approach is justified on three grounds: (i) K is derived from postulating that information is finite and knowledge is information compression; (ii) K is compatible and convertible to ordinary measures of effect size and algorithmic complexity; (iii) K is physically interpretable as a measure of entropic efficiency. Moreover, the K function has useful properties that support its potential as a measure of knowledge. Examples given to illustrate the possible uses of K include: the knowledge value of proving Fermat's last theorem; the accuracy of measurements of the mass of the electron; the half life of predictions of solar eclipses; the usefulness of evolutionary models of reproductive skew; the significance of gender differences in personality; the sources of irreproducibility in psychology; the impact of scientific misconduct and questionable research practices; the knowledge value of astrology. Furthermore, measures derived from K may complement ordinary meta-analysis and may give rise to a universal classification of sciences and pseudosciences. Simple and memorable mathematical formulae that summarize the theory's key results may find practical uses in meta-research, philosophy and research policy.

Does Chemical Engineering Research Have a Reproducibility Problem?

Concerns have been raised in multiple scientific fields in recent years about the reproducibility of published results. Systematic efforts to examine this issue have been undertaken in biomedicine and psychology, but less is known about this important issue in the materials-oriented research that underpins much of modern chemical engineering. Here, we relate a dramatic historical episode from our own institution to illustrate the implications of performing reproducible research and describe two case studies based on literature analysis to provide concrete information on the reproducibility of modern materials-oriented research. The two case studies deal with the properties of metal-organic frameworks (MOFs), a class of materials that have generated tens of thousands of papers. We do not claim that research on MOFs is less (or more) reproducible than other subfields; rather, we argue that the characteristics of this subfield are common to many areas of materials-oriented research. We conclude with specific recommendations for action by individual researchers, journal editors, publishers, and research communities.

The Space Physics Environment Data Analysis System (SPEDAS)

With the advent of the Heliophysics/Geospace System Observatory (H/GSO), a complement of multi-spacecraft missions and ground-based observatories to study the space environment, data retrieval, analysis, and visualization of space physics data can be daunting. The Space Physics Environment Data Analysis System (SPEDAS), a grass-roots software development platform (www.spedas.org), is now officially supported by NASA Heliophysics as part of its data environment infrastructure. It serves more than a dozen space missions and ground observatories and can integrate the full complement of past and upcoming space physics missions with minimal resources, following clear, simple, and well-proven guidelines. Free, modular and configurable to the needs of individual missions, it works in both command-line (ideal for experienced users) and Graphical User Interface (GUI) mode (reducing the learning curve for first-time users). Both options have "crib-sheets," user-command sequences in ASCII format that can facilitate record-and-repeat actions, especially for complex operations and plotting. Crib-sheets enhance scientific interactions, as users can move rapidly and accurately from exchanges of technical information on data processing to efficient discussions regarding data interpretation and science. SPEDAS can readily query and ingest all International Solar Terrestrial Physics (ISTP)-compatible products from the Space Physics Data Facility (SPDF), enabling access to a vast collection of historic and current mission data. The planned incorporation of Heliophysics Application Programmer's Interface (HAPI) standards will facilitate data ingestion from distributed datasets that adhere to these standards. Although SPEDAS is currently Interactive Data Language (IDL)-based (and interfaces to Java-based tools such as Autoplot), efforts are under-way to expand it further to work with python (first as an interface tool and potentially even receiving an under-the-hood replacement). We review the SPEDAS development history, goals, and current implementation. We explain its "modes of use" with examples geared for users and outline its technical implementation and requirements with software developers in mind. We also describe SPEDAS personnel and software management, interfaces with other organizations, resources and support structure available to the community, and future development plans.

ELECTRONIC SUPPLEMENTARY MATERIAL

The online version of this article (10.1007/s11214-018-0576-4) contains supplementary material, which is available to authorized users.

Preconditioning against renal ischaemia reperfusion injury: the failure to translate to the clinic

Acute kidney injury (AKI) as a result of ischaemia-reperfusion represents a major healthcare burden worldwide. Mortality rates from AKI in hospitalized patients are extremely high and have changed little despite decades of research and medical advances. In 1986, Murry et al. demonstrated for the first time the phenomenon of ischaemic preconditioning to protect against ischaemia-reperfusion injury (IRI). This seminal finding paved the way for a broad body of research, which attempted to understand and ultimately harness this phenomenon for human application. The ability of preconditioning to limit renal IRI has now been demonstrated in multiple different animal models. However, more than 30 years later, a safe and consistent method of protecting human organs, including the kidneys, against IRI is still not available. This review highlights agents which, despite strong preclinical data, have recently failed to reduce AKI in human trials. The multiple reasons which may have contributed to the failure to translate some of the promising findings to clinical therapies are discussed. Agents which hold promise in the clinic because of their recent efficacy in preclinical large animal models are also reviewed.

Ensuring Scientific Publishing Credibility in Translational Biomedical Optics

Editor-in-Chief Brian Pogue writes about publishing credibility in the field of translational biomedical optics.

Workshop Report: Zebrafish and Other Fish Models-Description of Extrinsic Environmental Factors for Rigorous Experiments and Reproducible Results

Current standards for husbandry and maintenance of zebrafish and other aquatic species in the laboratory are diverse, and are subject to laboratory performance, engineering, and practice standards (the Guide), institutional interpretation, national animal welfare laws, and cultural differences. Consequently, it is difficult, and probably not advantageous, to establish a single standard in view of the hardy nature of zebrafish and the diversity of research requirements it is used to address. Based on their natural habitat, zebrafish can thrive in a variety of environmental conditions, which is a specific advantage for working with this laboratory organism. However, it also makes reporting and reproducibility difficult, because variations in the husbandry and environmental conditions, including the environmental conditions before and during experiments, are often underreported in the scientific literature. This lack of consistency presents a potential problem for research reproducibility. To begin addressing this emerging scientific gap, the National Institutes of Health's (NIH) Office of Research Infrastructure Programs (ORIP), Division of Construction and Instruments (DCI), hosted a workshop in late 2017, entitled "Zebrafish and Other Aquatic Models: Reporting of Environmental Husbandry Conditions for Rigorous Experiments and Reproducible Results," that was attended by ∼60 participants. The objectives of the workshop were to bring together a diverse group of stakeholders-researchers, facility managers, veterinarians, journal editors, commercial vendors, and others to (1) review current husbandry and environmental management practices for the care of zebrafish and other aquatic organisms in the laboratory and to (2) propose a process for the development of a minimal set of environmental parameters that should be reported in publications to ensure rigor and robustness of experiments and reproducible outcomes. The participants also discussed how these recommendations, as an initial step, might be collected, disseminated, implemented, and improved upon after future iteration.