A reaction norm perspective on reproducibility

Understanding and tackling the reproducibility crisis - Why we need to study scientists' trust in data

In the life sciences, there is an ongoing discussion about a perceived 'reproducibility crisis'. However, it remains unclear to which extent the perceived lack of reproducibility is the consequence of issues that can be tackled and to which extent it may be the consequence of unrealistic expectations of the technical level of reproducibility. Large-scale, multi-institutional experimental replication studies are very cost- and time-intensive. This Perspective suggests an alternative, complementary approach: meta-research using sociological and philosophical methodologies to examine researcher trust in data. An improved understanding of the criteria used by researchers to judge data reliability will provide crucial, initial evidence on the actual scale of the reproducibility crisis and on measures to tackle it.

Using mice from different breeding sites fails to improve replicability of results from single-laboratory studies

Theoretical and empirical evidence indicates that low external validity due to rigorous standardization of study populations is a cause of poor replicability in animal research. Here we report a multi-laboratory study aimed at investigating whether heterogenization of study populations by using animals from different breeding sites increases the replicability of results from single-laboratory studies. We used male C57BL/6J mice from six different breeding sites to test a standardized against a heterogenized (HET) study design in six independent replicate test laboratories. For the standardized design, each laboratory ordered mice from a single breeding site (each laboratory from a different one), while for the HET design, each laboratory ordered proportionate numbers of mice from the five remaining breeding sites. To test our hypothesis, we assessed 14 outcome variables, including body weight, behavioral measures obtained from a single session on an elevated plus maze, and clinical blood parameters. Both breeding site and test laboratory affected variation in outcome variables, but the effect of test laboratory was more pronounced for most outcome variables. Moreover, heterogenization of study populations by breeding site (HET) did not reduce variation in outcome variables between test laboratories, which was most likely due to the fact that breeding site had only little effect on variation in outcome variables, thereby limiting the scope for HET to reduce between-lab variation. We conclude that heterogenization of study populations by breeding site has limited capacity for improving the replicability of results from single-laboratory animal studies.

Mapping strategies towards improved external validity in preclinical translational research

INTRODUCTION

Translation is about successfully bringing findings from preclinical contexts into the clinic. This transfer is challenging as clinical trials frequently fail despite positive preclinical results. Limited robustness of preclinical research has been marked as one of the drivers of such failures. One suggested solution is to improve the external validity of in vitro and in vivo experiments via a suite of complementary strategies.

AREAS COVERED

In this review, the authors summarize the literature available on different strategies to improve external validity in in vivo, in vitro, or ex vivo experiments; systematic heterogenization; generalizability tests; and multi-batch and multicenter experiments. Articles that tested or discussed sources of variability in systematically heterogenized experiments were identified, and the most prevalent sources of variability are reviewed further. Special considerations in sample size planning, analysis options, and practical feasibility associated with each strategy are also reviewed.

EXPERT OPINION

The strategies reviewed differentially influence variation in experiments. Different research projects, with their unique goals, can leverage the strengths and limitations of each strategy. Applying a combination of these approaches in confirmatory stages of preclinical research putatively increases the chances of success in clinical studies.

Narrowing the translational research gap by aligning replication concepts in basic and clinical neuroscience

Replicability and reproducibility are widely considered to be cornerstones of valid scientific research. Yet, the elements of replication in fundamental neuroscience studies do not fully overlap with the process of replication in clinical neuroscience involving patients. Here we discuss how better aligning the concept of replication across this translational spectrum might enhance the rate at which basic findings in the organization and function of the nervous system are leveraged to develop new treatments for psychiatric and neurological disorders.

The importance of reaction norms in dietary restriction and ageing research

Ageing research has progressed rapidly through our ability to modulate the ageing process. Pharmacological and dietary treatments can increase lifespan and have been instrumental in our understanding of the mechanisms of ageing. Recently, several studies have reported genetic variance in response to these anti-ageing interventions, questioning their universal application and making a case for personalised medicine in our field. As an extension of these findings the response to dietary restriction was found to not be repeatable when the same genetic mouse lines were retested. We show here that this effect is more widespread with the response to dietary restriction also showing low repeatability across genetic lines in the fly (Drosophila melanogaster). We further argue that variation in reaction norms, the relationship between dose and response, can explain such conflicting findings in our field. We simulate genetic variance in reaction norms and show that such variation can: 1) lead to over- or under-estimation of treatment responses, 2) dampen the response measured if a genetically heterogeneous population is studied, and 3) illustrate that genotype-by-dose-by-environment interactions can lead to low repeatability of DR and potentially other anti-ageing interventions. We suggest that putting experimental biology and personalised geroscience in a reaction norm framework will aid progress in ageing research.

Systematic assessment of the replicability and generalizability of preclinical findings: Impact of protocol harmonization across laboratory sites

The influence of protocol standardization between laboratories on their replicability of preclinical results has not been addressed in a systematic way. While standardization is considered good research practice as a means to control for undesired external noise (i.e., highly variable results), some reports suggest that standardized protocols may lead to idiosyncratic results, thus undermining replicability. Through the EQIPD consortium, a multi-lab collaboration between academic and industry partners, we aimed to elucidate parameters that impact the replicability of preclinical animal studies. To this end, 3 experimental protocols were implemented across 7 laboratories. The replicability of results was determined using the distance travelled in an open field after administration of pharmacological compounds known to modulate locomotor activity (MK-801, diazepam, and clozapine) in C57BL/6 mice as a worked example. The goal was to determine whether harmonization of study protocols across laboratories improves the replicability of the results and whether replicability can be further improved by systematic variation (heterogenization) of 2 environmental factors (time of testing and light intensity during testing) within laboratories. Protocols were tested in 3 consecutive stages and differed in the extent of harmonization across laboratories and standardization within laboratories: stage 1, minimally aligned across sites (local protocol); stage 2, fully aligned across sites (harmonized protocol) with and without systematic variation (standardized and heterogenized cohort); and stage 3, fully aligned across sites (standardized protocol) with a different compound. All protocols resulted in consistent treatment effects across laboratories, which were also replicated within laboratories across the different stages. Harmonization of protocols across laboratories reduced between-lab variability substantially compared to each lab using their local protocol. In contrast, the environmental factors chosen to introduce systematic variation within laboratories did not affect the behavioral outcome. Therefore, heterogenization did not reduce between-lab variability further compared to the harmonization of the standardized protocol. Altogether, these findings demonstrate that subtle variations between lab-specific study protocols may introduce variation across independent replicate studies even after protocol harmonization and that systematic heterogenization of environmental factors may not be sufficient to account for such between-lab variation. Differences in replicability of results within and between laboratories highlight the ubiquity of study-specific variation due to between-lab variability, the importance of transparent and fine-grained reporting of methodologies and research protocols, and the importance of independent study replication.

The rearing environment persistently modulates mouse phenotypes from the molecular to the behavioural level

The phenotype of an organism results from its genotype and the influence of the environment throughout development. Even when using animals of the same genotype, independent studies may test animals of different phenotypes, resulting in poor replicability due to genotype-by-environment interactions. Thus, genetically defined strains of mice may respond differently to experimental treatments depending on their rearing environment. However, the extent of such phenotypic plasticity and its implications for the replicability of research findings have remained unknown. Here, we examined the extent to which common environmental differences between animal facilities modulate the phenotype of genetically homogeneous (inbred) mice. We conducted a comprehensive multicentre study, whereby inbred C57BL/6J mice from a single breeding cohort were allocated to and reared in 5 different animal facilities throughout early life and adolescence, before being transported to a single test laboratory. We found persistent effects of the rearing facility on the composition and heterogeneity of the gut microbial community. These effects were paralleled by persistent differences in body weight and in the behavioural phenotype of the mice. Furthermore, we show that environmental variation among animal facilities is strong enough to influence epigenetic patterns in neurons at the level of chromatin organisation. We detected changes in chromatin organisation in the regulatory regions of genes involved in nucleosome assembly, neuronal differentiation, synaptic plasticity, and regulation of behaviour. Our findings demonstrate that common environmental differences between animal facilities may produce facility-specific phenotypes, from the molecular to the behavioural level. Furthermore, they highlight an important limitation of inferences from single-laboratory studies and thus argue that study designs should take environmental background into account to increase the robustness and replicability of findings.

The phenotype of an organism results not only from its genotype but also the influence of its environment throughout development. This study shows that common environmental differences between animal facilities can induce substantial variation in the phenotype of mice, thereby highlighting an important limitation of inferences from single-laboratory studies in animal research.

Mice in translational neuroscience: What R we doing?

Animal models play a pivotal role in translational neuroscience but recurrent problems in data collection, analyses, and interpretation, lack of biomarkers, and a tendency to over-reliance on mice have marred neuroscience progress, leading to one of the highest attrition rates in drug translation. Global initiatives to improve reproducibility and model selection are being implemented. Notwithstanding, mice are still the preferred animal species to model human brain disorders even when the translation has been shown to be limited. Non-human primates are better positioned to provide relevant translational information because of their higher brain complexity and homology to humans. Among others, lack of resources and formal training, strict legislation, and ethical issues may impede broad access to large animals. We propose that instead of increasingly restrictive legislation, more resources for training, education, husbandry, and data sharing are urgently needed. The creation of multidisciplinary teams, in which veterinarians need to play a key role, would be critical to improve translational efficiency. Furthermore, it is not usually acknowledged by researchers and regulators the value of comparative studies in lower species, that are instrumental in toxicology, target identification, and mechanistic studies. Overall, we highlight here the need for a conceptual shift in neuroscience research and policies to reach the patients.

Conventional laboratory housing increases morbidity and mortality in research rodents: results of a meta-analysis

BACKGROUND

Over 120 million mice and rats are used annually in research, conventionally housed in shoebox-sized cages that restrict natural behaviours (e.g. nesting and burrowing). This can reduce physical fitness, impair thermoregulation and reduce welfare (e.g. inducing abnormal stereotypic behaviours). In humans, chronic stress has biological costs, increasing disease risks and potentially shortening life. Using a pre-registered protocol ( https://atrium.lib.uoguelph.ca/xmlui/handle/10214/17955 ), this meta-analysis therefore tested the hypothesis that, compared to rodents in 'enriched' housing that better meets their needs, conventional housing increases stress-related morbidity and all-cause mortality.

RESULTS

Comprehensive searches (via Ovid, CABI, Web of Science, Proquest and SCOPUS on May 24 2020) yielded 10,094 publications. Screening for inclusion criteria (published in English, using mice or rats and providing 'enrichments' in long-term housing) yielded 214 studies (within 165 articles, using 6495 animals: 59.1% mice; 68.2% male; 31.8% isolation-housed), and data on all-cause mortality plus five experimentally induced stress-sensitive diseases: anxiety, cancer, cardiovascular disease, depression and stroke. The Systematic Review Center for Laboratory animal Experimentation (SYRCLE) tool assessed individual studies' risks of bias. Random-effects meta-analyses supported the hypothesis: conventional housing significantly exacerbated disease severity with medium to large effect sizes: cancer (SMD = 0.71, 95% CI = 0.54-0.88); cardiovascular disease (SMD = 0.72, 95% CI = 0.35-1.09); stroke (SMD = 0.87, 95% CI = 0.59-1.15); signs of anxiety (SMD = 0.91, 95% CI = 0.56-1.25); signs of depression (SMD = 1.24, 95% CI = 0.98-1.49). It also increased mortality rates (hazard ratio = 1.48, 95% CI = 1.25-1.74; relative median survival = 0.91, 95% CI = 0.89-0.94). Meta-regressions indicated that such housing effects were ubiquitous across species and sexes, but could not identify the most impactful improvements to conventional housing. Data variability (assessed via coefficient of variation) was also not increased by 'enriched' housing.

CONCLUSIONS

Conventional housing appears sufficiently distressing to compromise rodent health, raising ethical concerns. Results also add to previous work to show that research rodents are typically CRAMPED (cold, rotund, abnormal, male-biased, poorly surviving, enclosed and distressed), raising questions about the validity and generalisability of the data they generate. This research was funded by NSERC, Canada.

Safety of a Novel Listeria monocytogenes-Based Vaccine Vector Expressing NcSAG1 (Neospora caninum Surface Antigen 1)

Listeria monocytogenes (LM) has been proposed as vaccine vector in various cancers and infectious diseases since LM induces a strong immune response. In this study, we developed a novel and safe LM-based vaccine vector platform, by engineering a triple attenuated mutant (Lm3Dx) (ΔactA, ΔinlA, ΔinlB) of the wild-type LM strain JF5203 (CC 1, phylogenetic lineage I). We demonstrated the strong attenuation of Lm3Dx while maintaining its capacity to selectively infect antigen-presenting cells (APCs) in vitro. Furthermore, as proof of concept, we introduced the immunodominant Neospora caninum (Nc) surface antigen NcSAG1 into Lm3Dx. The NcSAG1 protein was expressed by Lm3Dx_SAG1 during cellular infection. To demonstrate safety of Lm3Dx_SAG1 in vivo, we vaccinated BALB/C mice by intramuscular injection. Following vaccination, mice did not suffer any adverse effects and only sporadically shed bacteria at very low levels in the feces (<100 CFU/g). Additionally, bacterial load in internal organs was very low to absent at day 1.5 and 4 following the 1st vaccination and at 2 and 4 weeks after the second boost, independently of the physiological status of the mice. Additionally, vaccination of mice prior and during pregnancy did not interfere with pregnancy outcome. However, Lm3Dx_SAG1 was shed into the milk when inoculated during lactation, although it did not cause any clinical adverse effects in either dams or pups. Also, we have indications that the vector persists more days in the injected muscle of lactating mice. Therefore, impact of physiological status on vector dynamics in the host and mechanisms of milk shedding requires further investigation. In conclusion, we provide strong evidence that Lm3Dx is a safe vaccine vector in non-lactating animals. Additionally, we provide first indications that mice vaccinated with Lm3Dx_SAG1 develop a strong and Th1-biased immune response against the Lm3Dx-expressed neospora antigen. These results encourage to further investigate the efficiency of Lm3Dx_SAG1 to prevent and treat clinical neosporosis.