Insufficient transparency of statistical reporting in preclinical research: a scoping review

Poor statistical reporting: do we have a reason for concern? A narrative review and recommendations

PURPOSE OF REVIEW

The aim of the review conducted was to present recent articles indicating the need to implement statistical recommendations in the daily work of biomedical journals.

RECENT FINDINGS

The most recent literature shows an unchanged percentage of journals using specialized statistical review over 20 years. The problems of finding statistical reviewers, the impractical way in which biostatistics is taught and the nonimplementation of published statistical recommendations contribute to the fact that a small percentage of accepted manuscripts contain correctly performed analysis. The statistical recommendations published for authors and editorial board members in recent years contain important advice, but more emphasis should be placed on their practical and rigorous implementation. If this is not the case, we will additionally continue to experience low reproducibility of the research.

SUMMARY

There is a low level of statistical reporting these days. Recommendations related to the statistical review of submitted manuscripts should be followed more rigorously.

CSL-Tox: an open-source analytical framework for the comparison of short-term and long-term toxicity end points and assessing the need of chronic studies in drug development

In-vivo toxicity assessment is an important step prior to clinical development and is still the main source of data for overall risk assessment of a new molecular entity (NCE). All in-vivo studies are performed according to regulatory requirements and many efforts have been exerted to minimize these studies in accordance with the (Replacement, Reduction and Refinement) 3Rs principle. Many aspects of in-vivo toxicology packages can be optimized to reduce animal use, including the number of studies performed as well as study durations, which is the main focus of this analysis. We performed a statistical comparison of adverse findings observed in 116 short-term versus 78 long-term in-house or in-house sponsored Contract Research Organizations (CRO) studies, in order to explore the possibility of using only short-term studies as a prediction tool for the longer-term effects. All the data analyzed in this study was manually extracted from the toxicology reports (in PDF formats) to construct the dataset. Annotation of treatment related findings was one of the challenges faced during this work. A specific focus was therefore put on the summary and conclusion sections of the reports since they contain expert assessments on whether the findings were considered adverse or were attributed to other reasons. Our analysis showed a general good concordance between short-term and long-term toxicity findings for large molecules and the majority of small molecules. Less concordance was seen for certain body organs, which can be named as "target organ systems' findings". While this work supports the minimization of long-term studies, a larger-scale effort would be needed to provide more evidence. We therefore present the steps performed in this study as an open-source R workflow for the Comparison of Short-term and Long-term Toxicity studies (CSL-Tox). The dataset used in the work is provided to allow researchers to reproduce such analysis, re-evaluate the statistical tools used and promote large-scale application of this study. Important aspects of animal research reproducibility are highlighted in this work, specifically, the necessity of a reproducible adverse effects reporting system and utilization of the controlled terminologies in-vivo toxicology reports and finally the importance of open-source analytical workflows that can be assessed by other scientists in the field of preclinical toxicology.

Photobiomodulation therapy on skeletal muscles exposed to diabetes mellitus: a systematic review of animal studies

Diabetes-related muscle damage has been overlooked despite its known association with increased morbidity and mortality in DM individuals. PBMT is a recognized alternative to improve skeletal muscle health in other populations, but its effectiveness in DM is still unclear. To address this issue, we reviewed preclinical studies, available in any language and period, in ten sources of information. The methods were previously registered at PROSPERO (CRD42021271041), based on PRISMA recommendations. Studies in murine models of T1DM or T2DM that reported quantitative analyses of skeletal muscles treated with low-level light therapy could be included after a blind selection process. Most of the seven included studies focus on decompensated T1DM rats with acute muscle injury (cryoinjury or contusion). In these five studies, PBMT improved muscle regeneration, by reducing inflammation and stimulating factors pro-angiogenesis and pro-myogenesis. Some positive effects could also be observed in two studies on muscles without acute injury: control of oxidative stress (T1DM) and reduction of myosteatosis (T2DM). Although infrared laser applied locally appears to be a promising approach, optimal parameters are undefined due to the heterogeneity of outcomes and high risk of bias, which prevented a quantitative synthesis. Several aspects of this growing field have yet to be investigated, particularly regarding the DM model (e.g., aged animals, T2DM), intervention (e.g., comparison with LED), and outcomes (e.g., muscle mass, strength, and function). Future research should aim to improve the internal validity by following guidelines for animal studies and enhance the translatability to clinical trials by using animal models that closely mimic patients with DM in rehabilitation settings.

Evaluating Study Design Rigor in Preclinical Cardiovascular Research: A Replication Study

Background

Methodological rigor is a major priority in preclinical cardiovascular research to ensure experimental reproducibility and high quality research. Lack of reproducibility results in diminished translation of preclinical discoveries into medical practice and wastes resources. In addition, lack of reproducibility fosters uncertainty in the public's acceptance of reported research results.

Methods

We evaluate the reporting of rigorous methodological practices in preclinical cardiovascular research studies published in leading scientific journals by screening articles for the inclusion of the following key study design elements (SDEs): considering sex as a biological variable, randomization, blinding, and sample size power estimation. We have specifically chosen to screen for these SDEs across articles pertaining to preclinical cardiovascular research studies published between 2011 and 2021. Our study replicates and extends a study published in 2017 by Ramirez et al. We hypothesized that there would be higher SDE inclusion across preclinical studies over time, that preclinical studies that also include human and animal substudies within the same study will exhibit greater SDE inclusion than animal-only preclinical studies, and that there will be a difference in SDE usage between large and small animal models.

Results

Overall, inclusion of SDEs was low. 15.2% of animal only studies included both sexes as a biological variable, 30.4% included randomization, 32.1% included blinding, and 8.2% included sample size estimation. Incorporation of SDE in preclinical studies did not significantly increase over the ten year time period in the articles we assessed. Although the inclusion of sex as a biological variable increased over the 10 year time frame, that change was not significant (p=0.411, corrected p=8.22). These trends were consistent across journals. Reporting of randomization and sample size estimation differs significantly between animal and human substudies (corrected p=3.690e-06 and corrected p=7.252e-08, respectively.) Large animal studies had a significantly greater percentage of blinding reported when compared to small animal studies (corrected p=0.01.) Additionally, overall, large animal studies tended to have higher SDE usage.

Conclusions

In summary, evidence of methodological rigor varies substantially depending on the study type and model organisms used. Over the time period of 2011-2021, the reporting of SDEs within preclinical cardiovascular studies has not improved and suggests extensive evaluation of other SDEs used in cardiovascular research. Limited incorporation of SDEs within research hinders experimental reproducibility that is critical to future research.

Responsible innovation in stem cell research: using responsibility as a strategy

Responsible innovation has been introduced as an important condition for advancing the field of regenerative medicine. This is reflected in the frequent references to responsible research conduct and responsible innovation in guidelines and recommendations in academic literature. The meaning of responsibility, how responsibility could be fostered and the context in which responsibilities should be enacted, however, remain unclear. The goal of this paper is to clarify the concept of responsibility in stem cell research and to illustrate how this concept could inform strategies to deal effectively with the ethical implications of stem cell research. Responsibility can be dissected into four categories: responsibility-as-accountability, responsibility-as-liability, responsibility-as-an-obligation and responsibility-as-a-virtue. The authors focus on responsible research conduct and responsible innovation in general to move beyond the scope of research integrity and illustrate that different notions of responsibility have different implications for how stem cell research is organized.

How to write statistical analysis section in medical research

Reporting of statistical analysis is essential in any clinical and translational research study. However, medical research studies sometimes report statistical analysis that is either inappropriate or insufficient to attest to the accuracy and validity of findings and conclusions. Published works involving inaccurate statistical analyses and insufficient reporting influence the conduct of future scientific studies, including meta-analyses and medical decisions. Although the biostatistical practice has been improved over the years due to the involvement of statistical reviewers and collaborators in research studies, there remain areas of improvement for transparent reporting of the statistical analysis section in a study. Evidence-based biostatistics practice throughout the research is useful for generating reliable data and translating meaningful data to meaningful interpretation and decisions in medical research. Most existing research reporting guidelines do not provide guidance for reporting methods in the statistical analysis section that helps in evaluating the quality of findings and data interpretation. In this report, we highlight the global and critical steps to be reported in the statistical analysis of grants and research articles. We provide clarity and the importance of understanding study objective types, data generation process, effect size use, evidence-based biostatistical methods use, and development of statistical models through several thematic frameworks. We also provide published examples of adherence or non-adherence to methodological standards related to each step in the statistical analysis and their implications. We believe the suggestions provided in this report can have far-reaching implications for education and strengthening the quality of statistical reporting and biostatistical practice in medical research.

Statistical practice and transparent reporting in the neurosciences: Preclinical motor behavioral experiments

Longitudinal and behavioral preclinical animal studies generate complex data, which may not be well matched to statistical approaches common in this literature. Analyses that do not adequately account for complexity may result in overly optimistic study conclusions, with consequences for reproducibility and translational decision-making. Recent work interrogating methodological shortcomings in animal research has not yet comprehensively investigated statistical shortcomings in the analysis of complex longitudinal and behavioral data. To this end, the current cross-sectional meta-research study rigorously reviewed published mouse or rat controlled experiments for motor rehabilitation in three neurologic conditions to evaluate statistical choices and reporting. Medline via PubMed was queried in February 2020 for English-language articles published January 1, 2017- December 31, 2019. Included were articles that used rat or mouse models of stroke, Parkinson’s disease, or traumatic brain injury, employed a therapeutic controlled experimental design to determine efficacy, and assessed at least one functional behavioral assessment or global evaluation of function. 241 articles from 99 journals were evaluated independently by a team of nine raters. Articles were assessed for statistical handling of non-independence, animal attrition, outliers, ordinal data, and multiplicity. Exploratory analyses evaluated whether transparency or statistical choices differed as a function of journal factors. A majority of articles failed to account for sources of non-independence in the data (74–93%) and/or did not analytically account for mid-treatment animal attrition (78%). Ordinal variables were often treated as continuous (37%), outliers were predominantly not mentioned (83%), and plots often concealed the distribution of the data (51%) Statistical choices and transparency did not differ with regards to journal rank or reporting requirements. Statistical misapplication can result in invalid experimental findings and inadequate reporting obscures errors. Clinician-scientists evaluating preclinical work for translational promise should be mindful of commonplace errors. Interventions are needed to improve statistical decision-making in preclinical behavioral neurosciences research.

Between two stools: preclinical research, reproducibility, and statistical design of experiments

Translation of animal-based preclinical research is hampered by poor validity and reproducibility issues. Unfortunately, preclinical research has ‘fallen between the stools’ of competing study design traditions. Preclinical studies are often characterised by small sample sizes, large variability, and ‘problem’ data. Although Fisher-type designs with randomisation and blocking are appropriate and have been vigorously promoted, structured statistically-based designs are almost unknown. Traditional analysis methods are commonly misapplied, and basic terminology and principles of inference testing misinterpreted. Problems are compounded by the lack of adequate statistical training for researchers, and failure of statistical educators to account for the unique demands of preclinical research. The solution is a return to the basics: statistical education tailored to non-statistician investigators, with clear communication of statistical concepts, and curricula that address design and data issues specific to preclinical research. Statistics curricula should focus on statistics as process: data sampling and study design before analysis and inference. Properly-designed and analysed experiments are a matter of ethics as much as procedure. Shifting the focus of statistical education from rote hypothesis testing to sound methodology will reduce the numbers of animals wasted in noninformative experiments and increase overall scientific quality and value of published research.