The study design elements employed by researchers in preclinical animal experiments from two research domains and implications for automation of systematic reviews

Evaluating Study Design Rigor in Preclinical Cardiovascular Research: A Replication Study

Background

Methodological rigor remains a priority in preclinical cardiovascular research to ensure experimental reproducibility and high-quality research. Limited reproducibility diminishes the translation of preclinical discoveries into medical practice. In addition, lack of reproducibility fosters uncertainty in the public's acceptance of reported research results.

Methods

We evaluated the reporting of methodological practices in preclinical cardiovascular research studies published in leading scientific journals by screening articles for the inclusion of the following study design elements (SDEs): considering sex as a biological variable, randomization, blinding, and sample size power estimation. We screened for these SDEs across articles regarding preclinical cardiovascular research studies published between 2011 and 2021. We replicated and extended a study published in 2017 by Ramirez et al. We hypothesized a higher SDE inclusion across preclinical studies over time, that preclinical studies that include human and animal substudies within the same study will exhibit greater SDE inclusion than animal-only preclinical studies, and that a difference exists in SDE usage between large and small animal models.

Results

SDE inclusion was low; with 15.2% of animal-only studies including both sexes as a biological variable, 30.4% including randomization, 32.1% including blinding, and 8.2% including sample size estimation. The incorporation of SDEs did not significantly increase over the ten-year timeframe in the screened articles. Randomization and sample size estimation differed significantly between animal and human substudies (corrected p=1.85e-05 and corrected p=3.81e-07, respectively.).

Conclusions

Evidence of methodological rigor varies depending on the study type and model organisms used. From 2011-2021, SDE reporting within preclinical studies has not increased, suggesting more work is needed to foster the inclusion of rigorous study design elements in cardiovascular research.

Reporting and analysis of repeated measurements in preclinical animals experiments

A common feature of preclinical animal experiments is repeated measurement of the outcome, e.g., body weight measured in mice pups weekly for 20 weeks. Separate time point analysis or repeated measures analysis approaches can be used to analyze such data. Each approach requires assumptions about the underlying data and violations of these assumptions have implications for estimation of precision, and type I and type II error rates. Given the ethical responsibilities to maximize valid results obtained from animals used in research, our objective was to evaluate approaches to reporting repeated measures design used by investigators and to assess how assumptions about variation in the outcome over time impact type I and II error rates and precision of estimates. We assessed the reporting of repeated measures designs of 58 studies in preclinical animal experiments. We used simulation modelling to evaluate three approaches to statistical analysis of repeated measurement data. In particular, we assessed the impact of (a) repeated measure analysis assuming that the outcome had non-constant variation at all time points (heterogeneous variance) (b) repeated measure analysis assuming constant variation in the outcome (homogeneous variance), (c) separate ANOVA at individual time point in repeated measures designs. The evaluation of the three model fitting was based on comparing the p-values distributions, the type I and type II error rates and by implication, the shrinkage or inflation of standard error estimates from 1000 simulated dataset. Of 58 studies with repeated measures design, three provided a rationale for repeated measurement and 23 studies reported using a repeated-measures analysis approach. Of the 35 studies that did not use repeated-measures analysis, fourteen studies used only two time points to calculate weight change which potentially means collected data was not fully utilized. Other studies reported only select time points (n = 12) raising the issue of selective reporting. Simulation studies showed that an incorrect assumption about the variance structure resulted in modified error rates and precision estimates. The reporting of the validity of assumptions for repeated measurement data is very poor. The homogeneous variation assumption, which is often invalid for body weight measurements, should be confirmed prior to conducting the repeated-measures analysis using homogeneous covariance structure and adjusting the analysis using corrections or model specifications if this is not met.