Weaknesses in Experimental Design and Reporting Decrease the Likelihood of Reproducibility and Generalization of Recent Cardiovascular Research

Evaluating Study Design Rigor in Preclinical Cardiovascular Research: A Replication Study

Background

Methodological rigor is a major priority in preclinical cardiovascular research to ensure experimental reproducibility and high quality research. Lack of reproducibility results in diminished translation of preclinical discoveries into medical practice and wastes resources. In addition, lack of reproducibility fosters uncertainty in the public's acceptance of reported research results.

Methods

We evaluate the reporting of rigorous methodological practices in preclinical cardiovascular research studies published in leading scientific journals by screening articles for the inclusion of the following key study design elements (SDEs): considering sex as a biological variable, randomization, blinding, and sample size power estimation. We have specifically chosen to screen for these SDEs across articles pertaining to preclinical cardiovascular research studies published between 2011 and 2021. Our study replicates and extends a study published in 2017 by Ramirez et al. We hypothesized that there would be higher SDE inclusion across preclinical studies over time, that preclinical studies that also include human and animal substudies within the same study will exhibit greater SDE inclusion than animal-only preclinical studies, and that there will be a difference in SDE usage between large and small animal models.

Results

Overall, inclusion of SDEs was low. 15.2% of animal only studies included both sexes as a biological variable, 30.4% included randomization, 32.1% included blinding, and 8.2% included sample size estimation. Incorporation of SDE in preclinical studies did not significantly increase over the ten year time period in the articles we assessed. Although the inclusion of sex as a biological variable increased over the 10 year time frame, that change was not significant (p=0.411, corrected p=8.22). These trends were consistent across journals. Reporting of randomization and sample size estimation differs significantly between animal and human substudies (corrected p=3.690e-06 and corrected p=7.252e-08, respectively.) Large animal studies had a significantly greater percentage of blinding reported when compared to small animal studies (corrected p=0.01.) Additionally, overall, large animal studies tended to have higher SDE usage.

Conclusions

In summary, evidence of methodological rigor varies substantially depending on the study type and model organisms used. Over the time period of 2011-2021, the reporting of SDEs within preclinical cardiovascular studies has not improved and suggests extensive evaluation of other SDEs used in cardiovascular research. Limited incorporation of SDEs within research hinders experimental reproducibility that is critical to future research.

Current Status and Future Opportunities in Modeling Clinical Characteristics of Multiple Sclerosis

Development of effective treatments requires understanding of disease mechanisms. For diseases of the central nervous system (CNS), such as multiple sclerosis (MS), human pathology studies and animal models tend to identify candidate disease mechanisms. However, these studies cannot easily link the identified processes to clinical outcomes, such as MS severity, required for causality assessment of candidate mechanisms. Technological advances now allow the generation of thousands of biomarkers in living human subjects, derived from genes, transcripts, medical images, and proteins or metabolites in biological fluids. These biomarkers can be assembled into computational models of clinical value, provided such models are generalizable. Reproducibility of models increases with the technical rigor of the study design, such as blinding, control implementation, the use of large cohorts that encompass the entire spectrum of disease phenotypes and, most importantly, model validation in independent cohort(s). To facilitate the growth of this important research area, we performed a meta-analysis of publications (n = 302) that model MS clinical outcomes extracting effect sizes, while also scoring the technical quality of the study design using predefined criteria. Finally, we generated a Shiny-App-based website that allows dynamic exploration of the data by selective filtering. On average, the published studies fulfilled only one of the seven criteria of study design rigor. Only 15.2% of the studies used any validation strategy, and only 8% used the gold standard of independent cohort validation. Many studies also used small cohorts, e.g., for magnetic resonance imaging (MRI) and blood biomarker predictors, the median sample size was <100 subjects. We observed inverse relationships between reported effect sizes and the number of study design criteria fulfilled, expanding analogous reports from non-MS fields, that studies that fail to limit bias overestimate effect sizes. In conclusion, the presented meta-analysis represents a useful tool for researchers, reviewers, and funders to improve the design of future modeling studies in MS and to easily compare new studies with the published literature. We expect that this will accelerate research in this important area, leading to the development of robust models with proven clinical value.