Enhancing Scientific Foundations to Ensure Reproducibility: A New Paradigm

Computational reproducibility of Jupyter notebooks from biomedical publications

BACKGROUND

Jupyter notebooks facilitate the bundling of executable code with its documentation and output in one interactive environment, and they represent a popular mechanism to document and share computational workflows, including for research publications. The reproducibility of computational aspects of research is a key component of scientific reproducibility but has not yet been assessed at scale for Jupyter notebooks associated with biomedical publications.

APPROACH

We address computational reproducibility at 2 levels: (i) using fully automated workflows, we analyzed the computational reproducibility of Jupyter notebooks associated with publications indexed in the biomedical literature repository PubMed Central. We identified such notebooks by mining the article's full text, trying to locate them on GitHub, and attempting to rerun them in an environment as close to the original as possible. We documented reproduction success and exceptions and explored relationships between notebook reproducibility and variables related to the notebooks or publications. (ii) This study represents a reproducibility attempt in and of itself, using essentially the same methodology twice on PubMed Central over the course of 2 years, during which the corpus of Jupyter notebooks from articles indexed in PubMed Central has grown in a highly dynamic fashion.

RESULTS

Out of 27,271 Jupyter notebooks from 2,660 GitHub repositories associated with 3,467 publications, 22,578 notebooks were written in Python, including 15,817 that had their dependencies declared in standard requirement files and that we attempted to rerun automatically. For 10,388 of these, all declared dependencies could be installed successfully, and we reran them to assess reproducibility. Of these, 1,203 notebooks ran through without any errors, including 879 that produced results identical to those reported in the original notebook and 324 for which our results differed from the originally reported ones. Running the other notebooks resulted in exceptions.

CONCLUSIONS

We zoom in on common problems and practices, highlight trends, and discuss potential improvements to Jupyter-related workflows associated with biomedical publications.

Evidence for Endogenous Opioid Dependence Related to Latent Sensitization in a Rat Model of Chronic Inflammatory Pain

Studies performed in a mouse model of chronic inflammatory pain induced by intraplantar injection of complete Freund's adjuvant (CFA) have shown that constitutive activation of the endogenous opioid signaling, besides serving as a mechanism of endogenous analgesia that tonically represses pain sensitization, also generates a state of endogenous opioid dependence. Since species-related differences concerning pain biology and addictive behaviors occur between mice and rats, the present study explored whether the coexistence of endogenous opioid analgesia and endogenous opioid dependence also characterizes a homologous rat model. To this aim, CFA-injured Wistar rats were treated with either 3 mg/kg or 10 mg/kg of the opioid receptor inverse agonist naltrexone (NTX) during the pain remission phase and monitored for 60 min for possible withdrawal behaviors. At 3 mg/kg, NTX, besides inducing the reinstatement of mechanical allodynia, also caused a distinct appearance of ptosis, with slight but nonsignificant changes to the occurrence of teeth chatters and rearing. On the other hand, 10 mg/kg of NTX failed to unmask pain sensitization and induced significantly lower levels of ptosis than 3 mg/kg. Such an NTX-related response pattern observed in the rat CFA model seems to differ substantially from the pattern previously described in the mouse CFA model. This supports the knowledge that mice and rats are not identical in terms of pharmacological response and stresses the importance of choosing the appropriate species for preclinical pain research purposes depending on the scientific question being asked.

Scientific and Regulatory Policy Committee Best Practices: Recommended ("Best") Practices for Informed (Non-blinded) Versus Masked (Blinded) Microscopic Evaluation in Animal Toxicity Studies

This article describes the Society of Toxicologic Pathology's (STP) five recommended ("best") practices for appropriate use of informed (non-blinded) versus masked (blinded) microscopic evaluation in animal toxicity studies intended for regulatory review. (1) Informed microscopic evaluation is the default approach for animal toxicity studies. (2) Masked microscopic evaluation has merit for confirming preliminary diagnoses for target organs and/or defining thresholds ("no observed adverse effect level" and similar values) identified during an initial informed evaluation, addressing focused hypotheses, or satisfying guidance or requests from regulatory agencies. (3) If used as the approach for an animal toxicity study to investigate a specific research question, masking of the initial microscopic evaluation should be limited to withholding only information about the group (control or test article-treated) and dose equivalents. (4) The decision regarding whether or not to perform a masked microscopic evaluation is best made by a toxicologic pathologist with relevant experience. (5) Pathology peer review, performed to verify the microscopic diagnoses and interpretations by the study pathologist, should use an informed evaluation approach. The STP maintains that implementing these five best practices has and will continue to consistently deliver robust microscopic data with high sensitivity for animal toxicity studies intended for regulatory review. Consequently, when conducting animal toxicity studies, the advantages of informed microscopic evaluation for maximizing sensitivity outweigh the perceived advantages of minimizing bias through masked microscopic examination.

HYDROCORTISONE, ASCORBIC ACID, AND THIAMINE THERAPY DECREASE RENAL OXIDATIVE STRESS AND ACUTE KIDNEY INJURY IN MURINE SEPSIS

ABSTRACT

Background: Acute kidney injury (AKI) occurs frequently in septic patients and correlates with increased mortality. Because clinical studies investigating hydrocortisone, ascorbic acid, and thiamine (HAT) have demonstrated discordant results, studies were performed using mortality stratification for therapy to identify candidates for therapy and determine mechanisms of organ injury. Methods: Sepsis was induced using the cecal ligation and puncture (CLP) model of sepsis with fluid and antibiotic support. Heart rate (HR) measurements obtained 6 hours after CLP stratified mice into live predicted (P-Live) or die predicted (P-Die). Stratified mice were then randomized for treatment with HAT or vehicle given 7 hours after CLP. Physiologic measurements were taken again at 24 hours, and mice were killed to collect blood and organs. Results: The following five groups were created: (1) P-Live vehicle, (2) P-Live HAT, (3) P-Die vehicle, (4) P-Die HAT, and (5) naive mice. Comparisons were made to test the hypotheses that (1) P-Die vehicle mice will have significant deterioration compared with P-Live mice targeting the kidney and (2) HAT will correct these deleterious changes in P-Die mice. Compared with P-Live, P-Die mice had a significant decline in all measured physiologic parameters (HR, cardiac output, breath rate, and temperature), which were corrected with HAT therapy (P < 0.05 for all parameters). The P-Die mice had declines in the ascorbic acid within the blood, peritoneal lavage, and kidney homogenate compared with P-Live mice indicating consumption, and the decline was corrected with HAT. Elevated IL-6, KC, Macrophage Inflammatory Protein-2, and IL-1RA were found in P-Die mice and decreased with HAT. Markers of endothelial cell injury (glypican 1 and glypican 4) were elevated in the P-Die mice, and these values were decreased with HAT therapy. Low oxygen levels with subsequent oxidative stress (OS) in the kidney were visualized in histologic sections using hypoxyprobe and also with carbonyl proteins and 8-iso-prostaglandin F2α in kidney homogenates. The P-Die mice had significant elevations of renal OSs, which was ameliorated with HAT. Kidney injury was evident in the P-Die mice compared with P-Live mice with elevations in blood urea nitrogen and cystatin C, which were significantly reduced with HAT. There was no evidence of global hypoxia or organ injury because hepatic parameters remained normal. Conclusions: Our data show that in CLP-induced sepsis, P-Die mice have increased inflammation, OS, and kidney injury. Hydrocortisone, ascorbic acid, and thiamine therapy decreased renal OS and injury in the P-Die group when given after the onset of sepsis-induced physiologic changes.

Contextualising sex and gender research to improve women's health: An early- and mid-career researcher perspective

The field of sex and gender research in health and medicine is growing, and many early- and mid-career researchers (EMCRs) are developing skills in this area. As EMCRs specialising in sex and gender research, we aim to better understand sex- and gender-based determinants of human health, challenge long-standing and pervasive gender biases, and contribute to improving the evidence base upon which clinical guidelines and policy interventions are developed. To effectively achieve these goals, we believe that EMCRs would benefit from understanding the challenges of working in this space and participate in driving change in three key areas. First, in creating greater links between the goals of sex and gender research and addressing systemic bias against women and gender minorities, to effectively translate knowledge about sex and gender differences into improved health outcomes. Second, in expanding the reach of sex and gender research to address women's health in an intersectional way and ensure that it also benefits the health of men, transgender and gender-diverse people and those who are intersex. Third, in working with others in the scientific community to improve methods for sex and gender research, including updating data collection practises, ensuring appropriate statistical analyses and shifting scientific culture to recognise the importance of null findings. By improving focus on these three areas, we see greater potential to translate this research to improve women's health and reduce health inequities for all.

Evidence of misuse of nonparametric tests in the presence of heteroscedasticity within obesity research

BACKGROUND

Classic nonparametric tests (cNPTs), like Kruskal-Wallis or Mann-Whitney U, are sometimes used to detect differences in central tendency ( i.e., means or medians). However, when the tests' assumptions are violated, such as in the presence of unequal variance and other forms of heteroscedasticity, they are no longer valid for testing differences in central tendency. Yet, sometimes researchers erroneously use cNPTs to account for heteroscedasticity.

OBJECTIVE

To document the appropriateness of cNPT use in obesity literature, characterize studies that use cNPTs, and evaluate the citation and public sharing patterns of these articles.

METHODS

We reviewed obesity studies published in 2017 to determine whether the authors used cNPTs: (1) to correct for heteroscedasticity (invalid); (2) when heteroscedasticity was clearly not present (correct); or (3) when it was unclear whether heteroscedasticity was present (unclear). Open science R packages were used to transparently search literature and extract data on how often papers with errors have been cited in academic literature, read in Mendeley, and disseminated in the media.

RESULTS

We identified nine studies that used a cNPT in the presence of heteroscedasticity (some because of the mistaken rationale that the test corrected for heteroscedasticity), 25 articles that did not explicitly state whether heteroscedasticity was present when a cNPT was used, and only four articles that appropriately reported that heteroscedasticity was not present when a cNPT was used. Errors were found in observational and interventional studies, in human and rodent studies, and only when studies were unregistered. Studies with errors have been cited 113 times, read in Mendeley 123 times, and disseminated in the media 41 times, by the public, scientists, science communicators, and doctors.

CONCLUSIONS

Examples of inappropriate use of cNPTs exist in the obesity literature, and those articles perpetuate the errors via various audiences and dissemination platforms.

Persistent confusion in nutrition and obesity research about the validity of classic nonparametric tests in the presence of heteroscedasticity: evidence of the problem and valid alternatives

The use of classic nonparametric tests (cNPTs), such as the Kruskal-Wallis and Mann-Whitney U tests, in the presence of unequal variance for between-group comparisons of means and medians may lead to marked increases in the rate of falsely rejecting null hypotheses and decreases in statistical power. Yet, this practice remains prevalent in the scientific literature, including nutrition and obesity literature. Some nutrition and obesity studies use a cNPT in the presence of unequal variance (i.e., heteroscedasticity), sometimes because of the mistaken rationale that the test corrects for heteroscedasticity. Herein, we discuss misconceptions of using cNPTs in the presence of heteroscedasticity. We then discuss assumptions, purposes, and limitations of 3 common tests used to test for mean differences between multiple groups, including 2 parametric tests: Fisher's ANOVA and Welch's ANOVA; and 1 cNPT: the Kruskal-Wallis test. To document the impact of heteroscedasticity on the validity of these tests under conditions similar to those used in nutrition and obesity research, we conducted simple simulations and assessed type I error rates (i.e., false positives, defined as incorrectly rejecting the null hypothesis). We demonstrate that type I error rates for Fisher's ANOVA, which does not account for heteroscedasticity, and Kruskal-Wallis, which tests for differences in distributions rather than means, deviated from the expected significance level. Greater deviation from the expected type I error rate was observed as the heterogeneity increased, especially in the presence of an imbalanced sample size. We provide brief tutorial guidance for authors, editors, and reviewers to identify appropriate statistical tests when test assumptions are violated, with a particular focus on cNPTs.

Plasma polymer surface modified expanded polytetrafluoroethylene promotes epithelial monolayer formation in vitro and can be transplanted into the dystrophic rat subretinal space

The aim of this study was to evaluate whether the surface modification of expanded polytetrafluoroethylene (ePTFE) using an n-heptylamine (HA) plasma polymer would allow for functional epithelial monolayer formation suitable for subretinal transplant into a non-dystrophic rat model. Freshly isolated iris pigment epithelial (IPE) cells from two rat strains (Long Evans [LE] and Dark Agouti [DA]) were seeded onto HA, fibronectin-coated n-heptylamine modified (F-HA) and unmodified ePFTE and fibronectin-coated tissue culture (F-TCPS) substrates. Both F-HA ePTFE and F-TCPS substrates enabled functional monolayer formation with both strains of rat. Without fibronectin coating, only LE IPE formed a monolayer on HA-treated ePTFE. Functional assessment of both IPE strains on F-HA ePTFE demonstrated uptake of POS that increased significantly with time that was greater than control F-TCPS. Surgical optimization using Healon GV and mixtures of Healon GV: phosphate buffered saline (PBS) to induce retinal detachment demonstrated that only Healon GV:PBS allowed F-HA ePTFE substrates to be successfully transplanted into the subretinal space of Royal College of Surgeons rats, where they remained flat beneath the neural retina for up to 4 weeks. No apparent substrate-induced inflammatory response was observed by fundus microscopy or immunohistochemical analysis, indicating the potential of this substrate for future clinical applications.

Premise for Standardized Sepsis Models

Sepsis morbidity and mortality exacts a toll on patients and contributes significantly to healthcare costs. Preclinical models of sepsis have been used to study disease pathogenesis and test new therapies, but divergent outcomes have been observed with the same treatment even when using the same sepsis model. Other disorders such as diabetes, cancer, malaria, obesity, and cardiovascular diseases have used standardized, preclinical models that allow laboratories to compare results. Standardized models accelerate the pace of research and such models have been used to test new therapies or changes in treatment guidelines. The National Institutes of Health mandated that investigators increase data reproducibility and the rigor of scientific experiments and has also issued research funding announcements about the development and refinement of standardized models. Our premise is that refinement and standardization of preclinical sepsis models may accelerate the development and testing of potential therapeutics for human sepsis, as has been the case with preclinical models for other disorders. As a first step toward creating standardized models, we suggest standardizing the technical standards of the widely used cecal ligation and puncture model and creating a list of appropriate organ injury and immune dysfunction parameters. Standardized sepsis models could enhance reproducibility and allow comparison of results between laboratories and may accelerate our understanding of the pathogenesis of sepsis.

Part II: Minimum Quality Threshold in Preclinical Sepsis Studies (MQTiPSS) for Types of Infections and Organ Dysfunction Endpoints

Although the clinical definitions of sepsis and recommended treatments are regularly updated, a systematic review has not been done for preclinical models. To address this deficit, a Wiggers-Bernard Conference on preclinical sepsis modeling reviewed the 260 most highly cited papers between 2003 and 2012 using sepsis models to create a series of recommendations. This Part II report provides recommendations for the types of infections and documentation of organ injury in preclinical sepsis models. Concerning the types of infections, the review showed that the cecal ligation and puncture model was used for 44% of the studies while 40% injected endotoxin. Recommendation #8 (numbered sequentially from Part I): endotoxin injection should not be considered as a model of sepsis; live bacteria or fungal strains derived from clinical isolates are more appropriate. Recommendation #9: microorganisms should replicate those typically found in human sepsis. Sepsis-3 states that sepsis is life-threatening organ dysfunction caused by a dysregulated host response to infection, but the review of the papers showed limited attempts to document organ dysfunction. Recommendation #10: organ dysfunction definitions should be used in preclinical models. Recommendation #11: not all activities in an organ/system need to be abnormal to verify organ dysfunction. Recommendation #12: organ dysfunction should be measured in an objective manner using reproducible scoring systems. Recommendation #13: not all experiments must measure all parameters of organ dysfunction, but investigators should attempt to fully capture as much information as possible. These recommendations are proposed as "best practices" for animal models of sepsis.

Quantity and Reporting Quality of Kidney Research

BACKGROUND

In 2004, researchers reported that the number of nephrology clinical trials was low and that the reporting quality of such trials was suboptimal. Furthermore, the number or quality of preclinical kidney-related studies has not been systematically evaluated.

METHODS

We performed a systematic review of randomized clinical trials published in 1966-2017 (listed in the Cochrane Library) and preclinical studies published in 1945-2017 (listed in PubMed). For reporting quality analysis, we evaluated the final main paper of 118 clinical trial reports and 135 preclinical studies published in leading journals in 1996, 2006, and 2016 on the basis of criteria from the widely used CONSORT and ARRIVE guidelines.

RESULTS

The annual number of reports of clinical kidney-related trials more than doubled between 2004 and 2014 along with reports in other medical disciplines. Hypertension remains the dominant focus of study, but ongoing trials also center on CKD, ESRD, and AKI. The reporting quality analysis revealed improvements, but deficits in reporting of clinical trial design, mode of randomization, and intention-to-treat analysis remain. Annual numbers of kidney-related preclinical studies remained low between 1945 and 2017 compared with other disciplines. Reporting quality analysis of preclinical studies revealed substantial reporting deficits across all leading journals, with little improvement over the last 20 years, especially for group size calculations, defining primary versus secondary outcomes, and blinded analysis.

CONCLUSIONS

Nephrology studies keep increasing in number but still lag behind other medical disciplines, and the quality of data reporting in kidney research can be further improved.

Mild Traumatic Brain Injury in Mice Beneficially Alters Lung NK1R and Structural Protein Expression to Enhance Survival after Pseudomonas aeruginosa Infection

Mild traumatic brain injury (mTBI) in a murine model increases survival to a bacterial pulmonary challenge compared with blunt tail trauma (TT). We hypothesize substance P and its receptor, the neurokinin 1 receptor (NK1R; official name TACR1), play a role in the increased survival of mTBI mice. Mice were subjected to mTBI or TT, and 48 hours after trauma, the levels of NK1R mRNA and protein were significantly up-regulated in mTBI lungs. Examination of the lung 48 hours after injury by microarray showed significant differences in the expression of 433 gene sets between groups, most notably genes related to intercellular proteins. Despite down-regulated gene expression of connective proteins, the presence of an intact pulmonary vasculature was supported by normal histology and bronchoalveolar lavage protein levels. To determine whether these mTBI-induced lung changes benefited in vivo responses, two chemotactic stimuli (a CXCL1 chemokine and a live Pseudomonas aeruginosa infection) were administered 48 hours after trauma. For both stimuli, mTBI mice recruited more neutrophils to the lung 4 hours after instillation (CXCL1: mTBI = 6.3 ± 1.3 versus TT = 3.3 ± 0.7 neutrophils/mL; Pseudomonas aeruginosa: mTBI = 9.4 ± 1.4 versus TT = 5.3 ± 1.1 neutrophils/mL). This study demonstrates that the downstream consequences of mTBI on lung NK1R levels and connective protein expression enhance neutrophil recruitment to a stimulus that may contribute to increased survival.