Experimental replications in animal trials

The Elephant in the Room: Recognition and Documentation of Personnel Practices That Confound Reproducibility

The ability to apply findings from animal studies efficiently and effectively is predicated on an understanding of biology and pathobiology, how that biology relates to the human systems being modeled, and how the studies are conducted and reported. This overview discusses various factors in research within the animal environment (referred to as extrinsic factors) that the NIH now expects to be documented to foster replicability in science and expand interpretations of study outcomes. Specifically, an important extrinsic factor in research with animals is that of individual personnel who perform handling practices, participate in research interactions, and share an overall presence in the housing facility with animals, all of which can confound reproducibility efforts in biomedical science. An improved understanding of the influences and behaviors of animal research personnel on animal responses is critical with regard to research results and the interpretation of data collected from animal models of biomedical disease.

Antibiotics promote intestinal growth of carbapenem-resistant Enterobacteriaceae by enriching nutrients and depleting microbial metabolites

The intestine is the primary colonisation site for carbapenem-resistant Enterobacteriaceae (CRE) and serves as a reservoir of CRE that cause invasive infections (e.g. bloodstream infections). Broad-spectrum antibiotics disrupt colonisation resistance mediated by the gut microbiota, promoting the expansion of CRE within the intestine. Here, we show that antibiotic-induced reduction of gut microbial populations leads to an enrichment of nutrients and depletion of inhibitory metabolites, which enhances CRE growth. Antibiotics decrease the abundance of gut commensals (including Bifidobacteriaceae and Bacteroidales) in ex vivo cultures of human faecal microbiota; this is accompanied by depletion of microbial metabolites and enrichment of nutrients. We measure the nutrient utilisation abilities, nutrient preferences, and metabolite inhibition susceptibilities of several CRE strains. We find that CRE can use the nutrients (enriched after antibiotic treatment) as carbon and nitrogen sources for growth. These nutrients also increase in faeces from antibiotic-treated mice and decrease following intestinal colonisation with carbapenem-resistant Escherichia coli. Furthermore, certain microbial metabolites (depleted upon antibiotic treatment) inhibit CRE growth. Our results show that killing gut commensals with antibiotics facilitates CRE colonisation by enriching nutrients and depleting inhibitory microbial metabolites.

Prenatal heroin exposure alters brain morphology and connectivity in adolescent mice

The United States is experiencing a dramatic increase in maternal opioid misuse and, consequently, the number of individuals exposed to opioids in utero. Prenatal opioid exposure has both acute and long-lasting effects on health and wellbeing. Effects on the brain, often identified at school age, manifest as cognitive impairment, attention deficit, and reduced scholastic achievement. The neurobiological basis for these effects is poorly understood. Here, we examine how in utero exposure to heroin affects brain development into early adolescence in a mouse model. Pregnant C57BL/6J mice received escalating doses of heroin twice daily on gestational days 4-18. The brains of offspring were assessed on postnatal day 28 using 9.4 T diffusion MRI of postmortem specimens at 36 μm resolution. Whole-brain volumes and the volumes of 166 bilateral regions were compared between heroin-exposed and control offspring. We identified a reduction in whole-brain volume in heroin-exposed offspring and heroin-associated volume changes in 29 regions after standardizing for whole-brain volume. Regions with bilaterally reduced standardized volumes in heroin-exposed offspring relative to controls include the ectorhinal and insular cortices. Regions with bilaterally increased standardized volumes in heroin-exposed offspring relative to controls include the periaqueductal gray, septal region, striatum, and hypothalamus. Leveraging microscopic resolution diffusion tensor imaging and precise regional parcellation, we generated whole-brain structural MRI diffusion connectomes. Using a dimension reduction approach with multivariate analysis of variance to assess group differences in the connectome, we found that in utero heroin exposure altered structure-based connectivity of the left septal region and the region that acts as a hub for limbic regulatory actions. Consistent with clinical evidence, our findings suggest that prenatal opioid exposure may have effects on brain morphology, connectivity, and, consequently, function that persist into adolescence. This work expands our understanding of the risks associated with opioid misuse during pregnancy and identifies biomarkers that may facilitate diagnosis and treatment.

Establishing Institutional Scores With the Rigor and Transparency Index: Large-scale Analysis of Scientific Reporting Quality

BACKGROUND

Improving rigor and transparency measures should lead to improvements in reproducibility across the scientific literature; however, the assessment of measures of transparency tends to be very difficult if performed manually.

OBJECTIVE

This study addresses the enhancement of the Rigor and Transparency Index (RTI, version 2.0), which attempts to automatically assess the rigor and transparency of journals, institutions, and countries using manuscripts scored on criteria found in reproducibility guidelines (eg, Materials Design, Analysis, and Reporting checklist criteria).

METHODS

The RTI tracks 27 entity types using natural language processing techniques such as Bidirectional Long Short-term Memory Conditional Random Field-based models and regular expressions; this allowed us to assess over 2 million papers accessed through PubMed Central.

RESULTS

Between 1997 and 2020 (where data were readily available in our data set), rigor and transparency measures showed general improvement (RTI 2.29 to 4.13), suggesting that authors are taking the need for improved reporting seriously. The top-scoring journals in 2020 were the Journal of Neurochemistry (6.23), British Journal of Pharmacology (6.07), and Nature Neuroscience (5.93). We extracted the institution and country of origin from the author affiliations to expand our analysis beyond journals. Among institutions publishing >1000 papers in 2020 (in the PubMed Central open access set), Capital Medical University (4.75), Yonsei University (4.58), and University of Copenhagen (4.53) were the top performers in terms of RTI. In country-level performance, we found that Ethiopia and Norway consistently topped the RTI charts of countries with 100 or more papers per year. In addition, we tested our assumption that the RTI may serve as a reliable proxy for scientific replicability (ie, a high RTI represents papers containing sufficient information for replication efforts). Using work by the Reproducibility Project: Cancer Biology, we determined that replication papers (RTI 7.61, SD 0.78) scored significantly higher (P<.001) than the original papers (RTI 3.39, SD 1.12), which according to the project required additional information from authors to begin replication efforts.

CONCLUSIONS

These results align with our view that RTI may serve as a reliable proxy for scientific replicability. Unfortunately, RTI measures for journals, institutions, and countries fall short of the replicated paper average. If we consider the RTI of these replication studies as a target for future manuscripts, more work will be needed to ensure that the average manuscript contains sufficient information for replication attempts.

Between two stools: preclinical research, reproducibility, and statistical design of experiments

Translation of animal-based preclinical research is hampered by poor validity and reproducibility issues. Unfortunately, preclinical research has ‘fallen between the stools’ of competing study design traditions. Preclinical studies are often characterised by small sample sizes, large variability, and ‘problem’ data. Although Fisher-type designs with randomisation and blocking are appropriate and have been vigorously promoted, structured statistically-based designs are almost unknown. Traditional analysis methods are commonly misapplied, and basic terminology and principles of inference testing misinterpreted. Problems are compounded by the lack of adequate statistical training for researchers, and failure of statistical educators to account for the unique demands of preclinical research. The solution is a return to the basics: statistical education tailored to non-statistician investigators, with clear communication of statistical concepts, and curricula that address design and data issues specific to preclinical research. Statistics curricula should focus on statistics as process: data sampling and study design before analysis and inference. Properly-designed and analysed experiments are a matter of ethics as much as procedure. Shifting the focus of statistical education from rote hypothesis testing to sound methodology will reduce the numbers of animals wasted in noninformative experiments and increase overall scientific quality and value of published research.

The "completely randomised" and the "randomised block" are the only experimental designs suitable for widespread use in pre-clinical research

Too many pre-clinical experiments are giving results which cannot be reproduced. This may be because the experiments are incorrectly designed. In “Completely randomized” (CR) and “Randomised block” (RB) experimental designs, both the assignment of treatments to experimental subjects and the order in which the experiment is done, are randomly determined. These designs have been used successfully in agricultural and industrial research and in clinical trials for nearly a century without excessive levels of irreproducibility. They must also be used in pre-clinical research if the excessive level of irreproducibility is to be eliminated. A survey of 100 papers involving mice and rats was used to determine whether scientists had used the CR or RB designs. The papers were assigned to three categories “Design acceptable”, “Randomised to treatment groups”, so of doubtful validity, or “Room for improvement”. Only 32 ± 4.7% of the papers fell into the first group, although none of them actually named either the CR or RB design. If the current high level of irreproducibility is to be eliminated, it is essential that scientists engaged in pre-clinical research use “Completely randomised” (CR), “Randomised block” (RB), or one of the more specialised named experimental designs described in textbooks on the subject.