Enhancing reproducibility: Failures from Reproducibility Initiatives underline core challenges

Decatastrophizing research irreproducibility

The reported inability to replicate research findings from the published literature precipitated extensive efforts to identify and correct perceived deficiencies in the execution and reporting of biomedical research. Despite these efforts, quantification of the magnitude of irreproducible research or the effectiveness of associated remediation initiatives, across diverse biomedical disciplines, has made little progress over the last decade. The idea that science is self-correcting has been further challenged in recent years by the proliferation of unverified or fraudulent scientific content generated by predatory journals, paper mills, pre-print server postings, and the inappropriate use of artificial intelligence technologies. The degree to which the field of pharmacology has been negatively impacted by these evolving pressures is unknown. Regardless of these ambiguities, pharmacology societies and their associated journals have championed best practices to enhance the experimental rigor and reporting of pharmacological research. The value of transparent and independent validation of raw data generation and its analysis in basic and clinical research is exemplified by the discovery, development, and approval of Highly Effective Modulator Therapy (HEMT) for Cystic Fibrosis (CF) patients. This provides a didactic counterpoint to concerns regarding the current state of biomedical research. Key features of this important therapeutic advance include objective construction of basic and translational research hypotheses, associated experimental designs, and validation of experimental effect sizes with quantitative alignment to meaningful clinical endpoints with input from the FDA, which enhanced scientific rigor and transparency with real world deliverables for patients in need.

Data platforms for open life sciences-A systematic analysis of management instruments

Open data platforms are interfaces between data demand of and supply from their users. Yet, data platform providers frequently struggle to aggregate data to suit their users' needs and to establish a high intensity of data exchange in a collaborative environment. Here, using open life science data platforms as an example for a diverse data structure, we systematically categorize these platforms based on their technology intermediation and the range of domains they cover to derive general and specific success factors for their management instruments. Our qualitative content analysis is based on 39 in-depth interviews with experts employed by data platforms and external stakeholders. We thus complement peer initiatives which focus solely on data quality, by additionally highlighting the data platforms' role to enable data utilization for innovative output. Based on our analysis, we propose a clearly structured and detailed guideline for seven management instruments. This guideline helps to establish and operationalize data platforms and to best exploit the data provided. Our findings support further exploitation of the open innovation potential in the life sciences and beyond.

Predictive validity in drug discovery: what it is, why it matters and how to improve it

Successful drug discovery is like finding oases of safety and efficacy in chemical and biological deserts. Screens in disease models, and other decision tools used in drug research and development (R&D), point towards oases when they score therapeutic candidates in a way that correlates with clinical utility in humans. Otherwise, they probably lead in the wrong direction. This line of thought can be quantified by using decision theory, in which 'predictive validity' is the correlation coefficient between the output of a decision tool and clinical utility across therapeutic candidates. Analyses based on this approach reveal that the detectability of good candidates is extremely sensitive to predictive validity, because the deserts are big and oases small. Both history and decision theory suggest that predictive validity is under-managed in drug R&D, not least because it is so hard to measure before projects succeed or fail later in the process. This article explains the influence of predictive validity on R&D productivity and discusses methods to evaluate and improve it, with the aim of supporting the application of more effective decision tools and catalysing investment in their creation.

The Gut Microbiome and Xenobiotics: Identifying Knowledge Gaps

There is an increasing awareness that the gut microbiome plays a critical role in human health and disease, but mechanistic insights are often lacking. In June 2018, the Health and Environmental Sciences Institute (HESI) held a workshop, "The Gut Microbiome: Markers of Human Health, Drug Efficacy and Xenobiotic Toxicity" (https://hesiglobal.org/event/the-gut-microbiome-workshop) to identify data gaps in determining how gut microbiome alterations may affect human health. Speakers and stakeholders from academia, government, and industry addressed multiple topics including the current science on the gut microbiome, endogenous and exogenous metabolites, biomarkers, and model systems. The workshop presentations and breakout group discussions formed the basis for identifying data gaps and research needs. Two critical issues that emerged were defining the microbial composition and function related to health and developing standards for models, methods and analysis in order to increase the ability to compare and replicate studies. A series of key recommendations were formulated to focus efforts to further understand host-microbiome interactions and the consequences of exposure to xenobiotics as well as identifying biomarkers of microbiome-associated disease and toxicity.

Aedes aegypti (Aag2)-derived clonal mosquito cell lines reveal the effects of pre-existing persistent infection with the insect-specific bunyavirus Phasi Charoen-like virus on arbovirus replication

Background

Aedes aegypti is a vector mosquito of major public health importance, transmitting arthropod-borne viruses (arboviruses) such as chikungunya, dengue, yellow fever and Zika viruses. Wild mosquito populations are persistently infected at high prevalence with insect-specific viruses that do not replicate in vertebrate hosts. In experimental settings, acute infections with insect-specific viruses have been shown to modulate arbovirus infection and transmission in Ae. aegypti and other vector mosquitoes. However, the impact of persistent insect-specific virus infections, which arboviruses encounter more commonly in nature, has not been investigated extensively. Cell lines are useful models for studying virus-host interactions, however the available Ae. aegypti cell lines are poorly defined and heterogenous cultures.

Methodology/Principle findings

We generated single cell-derived clonal cell lines from the commonly used Ae. aegypti cell line Aag2. Two of the fourteen Aag2-derived clonal cell lines generated harboured markedly and consistently reduced levels of the insect-specific bunyavirus Phasi Charoen-like virus (PCLV) known to persistently infect Aag2 cells. In contrast to studies with acute insect-specific virus infections in cell culture and in vivo, we found that pre-existing persistent PCLV infection had no major impact on the replication of the flaviviruses dengue virus and Zika virus, the alphavirus Sindbis virus, or the rhabdovirus vesicular stomatitis virus. We also performed a detailed characterisation of the morphology, transfection efficiency and immune status of our Aag2-derived clonal cell lines, and have made a clone that we term Aag2-AF5 available to the research community as a well-defined cell culture model for arbovirus-vector interaction studies.

Conclusions/Significance

Our findings highlight the need for further in vivo studies that more closely recapitulate natural arbovirus transmission settings in which arboviruses encounter mosquitoes harbouring persistent rather than acute insect-specific virus infections. Furthermore, we provide the well-characterised Aag2-derived clonal cell line as a valuable resource to the arbovirus research community.

Mosquito-borne viruses usually only infect humans through the bite of a mosquito that carries the virus. Viruses transmitted by the ‘yellow fever mosquito’ Aedes aegypti, including dengue virus, Zika virus, yellow fever virus and chikungunya virus, are causing an ever-increasing number of human disease cases globally. Mosquito-borne viruses have to infect and replicate inside the mosquito before they are transmitted to humans, and the presence of other infectious agents can change the efficiency of virus transmission. Mosquitoes are known to be infected with ‘insect-specific viruses’ that only infect mosquitoes and cannot cause human disease. We have shown here that in laboratory cell cultures derived from the Aedes aegypti mosquito, pre-existing infection with an insect-specific virus called Phasi Charoen-like virus does not affect the infection and growth of the mosquito-borne viruses dengue virus, Zika virus, Sindbis virus or vesicular stomatitis virus. Our research provides important new insights into whether and how insect-specific viruses may affect mosquito-borne virus replication. Ultimately, this information could inform ongoing research into whether insect-specific viruses could be used to prevent the transmission of mosquito-borne viruses to reduce global disease burdens.

Aedes aegypti (Aag2)-derived clonal mosquito cell lines reveal the effects of pre-existing persistent infection with the insect-specific bunyavirus Phasi Charoen-like virus on arbovirus replication

BACKGROUND

Aedes aegypti is a vector mosquito of major public health importance, transmitting arthropod-borne viruses (arboviruses) such as chikungunya, dengue, yellow fever and Zika viruses. Wild mosquito populations are persistently infected at high prevalence with insect-specific viruses that do not replicate in vertebrate hosts. In experimental settings, acute infections with insect-specific viruses have been shown to modulate arbovirus infection and transmission in Ae. aegypti and other vector mosquitoes. However, the impact of persistent insect-specific virus infections, which arboviruses encounter more commonly in nature, has not been investigated extensively. Cell lines are useful models for studying virus-host interactions, however the available Ae. aegypti cell lines are poorly defined and heterogenous cultures.

METHODOLOGY/PRINCIPLE FINDINGS

We generated single cell-derived clonal cell lines from the commonly used Ae. aegypti cell line Aag2. Two of the fourteen Aag2-derived clonal cell lines generated harboured markedly and consistently reduced levels of the insect-specific bunyavirus Phasi Charoen-like virus (PCLV) known to persistently infect Aag2 cells. In contrast to studies with acute insect-specific virus infections in cell culture and in vivo, we found that pre-existing persistent PCLV infection had no major impact on the replication of the flaviviruses dengue virus and Zika virus, the alphavirus Sindbis virus, or the rhabdovirus vesicular stomatitis virus. We also performed a detailed characterisation of the morphology, transfection efficiency and immune status of our Aag2-derived clonal cell lines, and have made a clone that we term Aag2-AF5 available to the research community as a well-defined cell culture model for arbovirus-vector interaction studies.

CONCLUSIONS/SIGNIFICANCE

Our findings highlight the need for further in vivo studies that more closely recapitulate natural arbovirus transmission settings in which arboviruses encounter mosquitoes harbouring persistent rather than acute insect-specific virus infections. Furthermore, we provide the well-characterised Aag2-derived clonal cell line as a valuable resource to the arbovirus research community.

Experts' Perspectives on Key Ethical Issues Associated With HIV Phylogenetics as Applied in HIV Transmission Dynamics Research

The use of phylogenetics in HIV molecular epidemiology has considerably increased our ability to understand the origin, spread, and characteristics of HIV epidemics. Despite its potential to advance knowledge on HIV transmission dynamics, the ethical issues associated with HIV molecular epidemiology have received minimal attention. In-depth interviews were conducted with scientists from diverse backgrounds to explore their perspectives on ethical issues associated with phylogenetic analysis of HIV genetic data as applied to HIV transmission dynamics studies. The Emanuel framework was used as the analytical framework. Favorable risk-benefit ratio and informed consent were the most invoked ethical principles and fair participant selection the least. Fear of loss of privacy and disclosure of HIV transmission were invariably cited as key ethical concerns. As HIV sequence data become increasingly available, comprehensive guidelines should be developed to guide its access, sharing and use, cognizant of the potential harms that may result.

Ensuring reproducibility and ethics in animal experiments reporting in Korea using the ARRIVE guideline

The aim of this study is to evaluate the reporting quality of animal experiments in Korea using the Animals in Research: Reporting In Vivo Experiments (ARRIVE) guideline developed in 2010 to overcome the reproducibility problem and to encourage compliance with replacement, refinement and reduction of animals in research (3R's principle). We reviewed 50 papers published by a Korean research group from 2013 to 2016 and scored the conformity with the 20-items ARRIVE guideline. The median conformity score was 39.50%. For more precise evaluation, the 20 items were subdivided into 57 sub-items. Among the sub-items, status of experimental animals, housing and husbandry were described under the average level. Microenvironment sub-items, such as enrichment, bedding material, cage type, number of companions, scored under 10%. Although statistical methods used for the studies were given in most publications (84%), sample size calculation and statistical assumption were rarely described. Most publications mentioned the IACUC approval, but only 8% mentioned welfare-related assessments and interventions, and only 4% mentioned any implications of experimental methods or findings for 3R. We may recommend the revision of the present IACUC proposal to collect more detailed information and improving educational program for animal researchers according to the ARRIVE guideline.

Opinion: Is science really facing a reproducibility crisis, and do we need it to?

Efforts to improve the reproducibility and integrity of science are typically justified by a narrative of crisis, according to which most published results are unreliable due to growing problems with research and publication practices. This article provides an overview of recent evidence suggesting that this narrative is mistaken, and argues that a narrative of epochal changes and empowerment of scientists would be more accurate, inspiring, and compelling.

Ten Basic Rules of Antibody Validation

The quality of research antibodies is an issue for decades. Although several papers have been published to improve the situation, their impact seems to be limited. This publication makes the effort to simplify the description of validation criteria in a way that the occasional antibody user is able to assess the validation level of an immunochemical reagent. A simple, 1-page checklist is supplied for the practical application of these criteria.

While it is not deliberate, much of today's biomedical research contains logical and technical flaws, showing a need for corrective action

Biomedical research has advanced swiftly in recent decades, largely due to progress in biotechnology. However, this rapid spread of new, and not always-fully understood, technology has also created a lot of false or irreproducible data and artifacts, which sometimes have led to erroneous conclusions. When describing various scientific issues, scientists have developed a habit of saying "on one hand… but on the other hand…", because discrepant data and conclusions have become omnipresent. One reason for this problematic situation is that we are not always thoughtful enough in study design, and sometimes lack enough philosophical contemplation. Another major reason is that we are too rushed in introducing new technology into our research without assimilating technical details. In this essay, we provide examples in different research realms to justify our points. To help readers test their own weaknesses, we raise questions on technical details of RNA reverse transcription, polymerase chain reactions, western blotting and immunohistochemical staining, as these methods are basic and are the base for other modern biotechnologies. Hopefully, after contemplation and reflection on these questions, readers will agree that we indeed know too little about these basic techniques, especially about the artifacts they may create, and thus many conclusions drawn from the studies using those ever-more-sophisticated techniques may be even more problematic.

The Academic-Industrial Complexity: Failure to Launch

The pharmaceutical industry has long known that ∼80% of the results of academic laboratories cannot be reproduced when repeated in industry laboratories. Yet academic investigators are typically unaware of this problem, which severely impedes the drug development process. This academic-industrial complication is not one of deception, but rather a complex issue related to how scientific research is carried out and translated in strikingly different enterprises. This Opinion describes the reasons for inconsistencies between academic and industrial laboratories and what can be done to repair this failure of translation.