Systematic reviews and meta-analyses of preclinical studies: publication bias in laboratory animal experiments

Tissue Engineering of the Urethra: A Systematic Review and Meta-analysis of Preclinical and Clinical Studies

CONTEXT

Urethra repair by tissue engineering has been extensively studied in laboratory animals and patients, but is not routinely used in clinical practice.

OBJECTIVE

To systematically investigate preclinical and clinical evidence of the efficacy of tissue engineering for urethra repair in order to stimulate translation of preclinical studies to the clinic.

EVIDENCE ACQUISITION

A systematic search strategy was applied in PubMed and EMBASE. Studies were independently screened for relevance by two reviewers, resulting in 80 preclinical and 23 clinical studies of which 63 and 13 were selected for meta-analysis to assess side effects, functionality, and study completion. Analyses for preclinical and clinical studies were performed separately. Full circumferential and inlay procedures were assessed independently. Evaluated parameters included seeding of cells and type of biomaterial.

EVIDENCE SYNTHESIS

Meta-analysis revealed that cell seeding significantly reduced the probability of encountering side effects in preclinical studies. Remarkably though, cells were only sparsely used in the clinic (4/23 studies) and showed no significant reduction of side effects. ln 21 out of 23 clinical studies, decellularized templates were used, while in preclinical studies other biomaterials showed promising outcomes as well. No direct comparison to current clinical practice could be made due to the limited number of randomized controlled studies.

CONCLUSIONS

Due to a lack of controlled (pre)clinical studies, the efficacy of tissue engineering for urethra repair could not be determined. Meta-analysis outcome measures were similar to current treatment options described in literature. Surprisingly, it appeared that favorable preclinical results, that is inclusion of cells, were not translated to the clinic. Improved (pre)clinical study designs may enhance clinical translation.

PATIENT SUMMARY

We reviewed all available literature on urethral tissue engineering to assess the efficacy in preclinical and clinical studies. We show that improvements to (pre)clinical study design is required to improve clinical translation of tissue engineering technologies.

Animal Study Registries: Results from a Stakeholder Analysis on Potential Strengths, Weaknesses, Facilitators, and Barriers

Publication bias in animal research, its extent, its predictors, and its potential countermeasures are increasingly discussed. Recent reports and conferences highlight the potential strengths of animal study registries (ASRs) in this regard. Others have warned that prospective registration of animal studies could diminish creativity, add administrative burdens, and complicate intellectual property issues in translational research. A literature review and 21 international key-informant interviews were conducted and thematically analyzed to develop a comprehensive matrix of main- and subcategories for potential ASR-related strengths, weaknesses, facilitators, and barriers (SWFBs). We identified 130 potential SWFBs. All stakeholder groups agreed that ASRs could in various ways improve the quality and refinement of animal studies while allowing their number to be reduced, as well as supporting meta-research on animal studies. However, all stakeholder groups also highlighted the potential for theft of ideas, higher administrative burdens, and reduced creativity and serendipity in animal studies. Much more detailed reasoning was captured in the interviews than is currently found in the literature, providing a comprehensive account of the issues and arguments around ASRs. All stakeholder groups highlighted compelling potential strengths of ASRs. Although substantial weaknesses and implementation barriers were highlighted as well, different governance measures might help to minimize or even eliminate their impact. Such measures might include confidentiality time frames for accessing prospectively registered protocols, harmonized reporting requirements across ASRs, ethics reviews, lab notebooks, and journal submissions. The comprehensive information gathered in this study could help to guide a more evidence-based debate and to design pilot tests for ASRs.

The manifold contributions over the last years on “publication bias” and “reproducibility crisis” in animal research initiated a debate on whether and how prospective animal study registries (ASRs) should be established in analogy to clinical trial registries. All recent debate, however, followed rather broad lines of argumentation and concluded that future decision-making on the issue of ASRs depends strongly on better knowledge about relevant characteristics of ASRs and about conflicting stakeholder interests. More qualitative but systematically developed evidence in this regard is needed. The primary objective of this study, therefore, was to present a systematically derived spectrum of all relevant strengths, weaknesses, facilitators and barriers (SWFBs) for ASRs. A systematic literature review and 21 key-informant interviews with experts from preclinical and clinical research, industry, and regulatory bodies were conducted to fulfill this objective. Our investigations resulted in a comprehensive and structured account of 130 issues and arguments around ASRs. Future debate and decision-making on ASRs might be heavily influenced by arguments and reasoning from individual experts and thus result in “eminence-based” policy making that relies on expert opinion. This study’s comprehensive spectrum of arguments and issues around ASR, developed through systematic and transparent methods, helps to balance the ongoing debate and thus facilitate a more evidence-based policy making.

The use of GRADE approach in systematic reviews of animal studies

OBJECTIVE

The application of Grading of Recommendations Assessment, Development and Evaluation (GRADE) in systematic review (SR) of animal studies can promote the translation from bench to bedside. We aim to explore the use of GRADE in SRs of animal studies.

METHODS

We used a theoretical analysis method to explore the use of GRADE in SR of animal studies and applied in an SR of animal studies. Meanwhile, we presented and discussed our results in two international conferences.

RESULTS

Five downgrade factors were considered as follows in SRs of animal studies: (1) Risk of bias: the SYRCLE tool can be used for assessing the risk of bias of animal studies. (2) Indirectness: we can assess indirectness in SRs of animal studies from the PICO. (3) Inconsistency: similarity of point estimates, extent of overlap of confidence intervals (CIs), and statistical heterogeneity are also suitable to evaluate inconsistency of evidence from animal studies. (4) Imprecision: optimal information size and 95% CIs are also suitable for SRs of animal studies, like those of clinical trials. (5) Publication bias: we need to consider publication bias comprehensively through the qualitative and quantitative methods.

CONCLUSIONS

The methods about the use of GRADE in SR of animal studies are explicit. However, the principle about GRADE in developing the policy based on the evidence from animal studies when there is an emergency of public health.

Effect of Stem Cell Therapy on Bone Mineral Density: A Meta-Analysis of Preclinical Studies in Animal Models of Osteoporosis

Background

Preclinical studies of the therapeutic role of stem cell based therapy in animal models of osteoporosis have largely yielded inconsistent results. We performed a meta-analysis to provide an overview of the currently available evidence.

Methods

Pubmed, Embase and Cochrane Library databases were systematically searched for relevant controlled studies. A random-effect model was used for pooled analysis of the effect of stem cell based therapy on bone mineral density (BMD). Stratified analyses were performed to explore the effect of study characteristics on the outcomes.

Results

Pooled results from 12 preclinical studies (110 animals in stem cell treatment groups, and 106 animals in control groups) indicated that stem cell based treatment was associated with significantly improved BMD (standardized mean difference [SMD] = 1.29, 95% Confidence Interval [CI]: 0.84–1.74, P < 0.001) with moderate heterogeneity (Cochrane’s Q test: P = 0.02, I² = 45%) among the constituent studies. Implantation of bone marrow cells, bone marrow mesenchymal stem cells, adipose-derived stem cells, and human umbilical cord blood-derived CD34+ cells, were all associated with improved BMD as compared to that in the controls (P < 0.05 for all); the only exception being the use of embryonic stem cell transplantation (P > 0.05). Egger’s test detected potential publication bias (P = 0.055); however, ‘trim and fill’ analysis yielded similar results after statistically incorporating the hypothetical studies in the analysis (SMD = 1.24, 95% CI: 0.32–2.16, P < 0.001).

Conclusions

Stem cell transplantation may improve BMD in animal models of osteoporosis. Our meta-analysis indicates a potential therapeutic role of stem cell based therapy for osteoporosis, and serves to augment the rationale for clinical studies.

Method parameters' impact on mortality and variability in mouse stroke experiments: a meta-analysis

Although hundreds of promising substances have been tested in clinical trials, thrombolysis currently remains the only specific pharmacological treatment for ischemic stroke. Poor quality, e.g. low statistical power, in the preclinical studies has been suggested to play an important role in these failures. Therefore, it would be attractive to use animal models optimized to minimize unnecessary mortality and outcome variability, or at least to be able to power studies more exactly by predicting variability and mortality given a certain experimental setup. The possible combinations of methodological parameters are innumerous, and an experimental comparison of them all is therefore not feasible. As an alternative approach, we extracted data from 334 experimental mouse stroke articles and, using a hypothesis-driven meta-analysis, investigated the method parameters’ impact on infarct size variability and mortality. The use of Swiss and C57BL6 mice as well as permanent occlusion of the middle cerebral artery rendered the lowest variability of the infarct size while the emboli methods increased variability. The use of Swiss mice increased mortality. Our study offers guidance for researchers striving to optimize mouse stroke models.

The impact of ginsenosides on cognitive deficits in experimental animal studies of Alzheimer's disease: a systematic review

Background

The efficacy of ginsenoside treatment on cognitive decline in individuals with Alzheimer’s disease (AD) has yet to be investigated. In this protocal, we conducted a systematic review to evaluate the effect of ginsenosides on cognitive deficits in experimental rodent AD models.

Methods

We identified eligible studies by searching seven electronic databases spanning from January 1980 to October 2014. We assessed the study quality, evaluated the efficacy of ginsenoside treatment, and performed a stratified meta-analysis and meta-regression analysis to assess the influence of the study design on ginsenoside efficacy.

Results

Twelve studies fulfilled our inclusion criteria from a total of 283 publications. The overall methodological quality of these studies was poor. The meta-analysis revealed that ginsenosides have a statistically significant positive effect on cognitive performance in experimental AD models. The stratified analysis revealed that ginsenoside Rg1 had the greatest effect on acquisition and retention memory in AD models. The effect size was significantly higher for both acquisition and retention memory in studies that used female animals compared with male animals.

Conclusions

We conclude that ginsenosides might reduce cognitive deficits in AD models. However, additional well-designed and well-reported animal studies are needed to inform further clinical investigations.

The usefulness of systematic reviews of animal experiments for the design of preclinical and clinical studies

The question of how animal studies should be designed, conducted, and analyzed remains underexposed in societal debates on animal experimentation. This is not only a scientific but also a moral question. After all, if animal experiments are not appropriately designed, conducted, and analyzed, the results produced are unlikely to be reliable and the animals have in effect been wasted. In this article, we focus on one particular method to address this moral question, namely systematic reviews of previously performed animal experiments. We discuss how the design, conduct, and analysis of future (animal and human) experiments may be optimized through such systematic reviews. In particular, we illustrate how these reviews can help improve the methodological quality of animal experiments, make the choice of an animal model and the translation of animal data to the clinic more evidence-based, and implement the 3Rs. Moreover, we discuss which measures are being taken and which need to be taken in the future to ensure that systematic reviews will actually contribute to optimizing experimental design and thereby to meeting a necessary condition for making the use of animals in these experiments justified.

Can animal data translate to innovations necessary for a new era of patient-centred and individualised healthcare? Bias in preclinical animal research

Background

The public and healthcare workers have a high expectation of animal research which they perceive as necessary to predict the safety and efficacy of drugs before testing in clinical trials. However, the expectation is not always realised and there is evidence that the research often fails to stand up to scientific scrutiny and its 'predictive value' is either weak or absent.

Discussion

Problems with the use of animals as models of humans arise from a variety of biases and systemic failures including: 1) bias and poor practice in research methodology and data analysis; 2) lack of transparency in scientific assessment and regulation of the research; 3) long-term denial of weaknesses in cross-species translation; 4) profit-driven motives overriding patient interests; 5) lack of accountability of expenditure on animal research; 6) reductionist-materialism in science which tends to dictate scientific inquiry and control the direction of funding in biomedical research.

Summary

Bias in animal research needs to be addressed before medical research and healthcare decision-making can be more evidence-based. Research funding may be misdirected on studying 'disease mechanisms' in animals that cannot be replicated outside tightly controlled laboratory conditions, and without sufficient critical evaluation animal research may divert attention away from avenues of research that hold promise for human health. The potential for harm to patients and trial volunteers from reliance on biased animal data¹ requires measures to improve its conduct, regulation and analysis. This article draws attention to a few of the many forms of bias in animal research that have come to light in the last decade and offers a strategy incorporating ten recommendations stated at the end of each section on bias. The proposals need development through open debate and subsequent rigorous implementation so that reviewers may determine the value of animal research to human health. The 10Rs + are protected by a Creative Commons Attribution 3.0 Unported License and therefore may be 'shared, remixed or built on, even commercially, so long as attributed by giving appropriate credit with a link to the license, and indicate if changes were made.’

Electronic supplementary material

The online version of this article (doi:10.1186/s12910-015-0043-7) contains supplementary material, which is available to authorized users.

Industry sponsorship and publication bias among animal studies evaluating the effects of statins on atherosclerosis and bone outcomes: a meta-analysis

Background

The effect that sponsorship has on publication rates or overall effect estimates in animal studies is unclear, though methodological biases are prevalent in animal studies of statins and there may be differences in efficacy estimates between industry and non-industry sponsored studies. In the present analysis, we evaluated the impact of funding source on publication bias in animal studies estimating the effect of statins on atherosclerosis and bone outcomes.

Methods

We conducted two independent systematic reviews and meta-analyses identifying animal studies evaluating the effect of statins on reducing the risk of atherosclerosis outcomes (n = 49) and increasing the likelihood of beneficial bone outcomes (n = 45). After stratifying the included studies within each systematic review by funding source, three separate analyses were employed to assess publication bias in these meta-analyses—funnel plots, Egger’s Linear Regression, and the Trim and Fill methods.

Results

We found potential evidence of publication bias, primarily in non-industry sponsored studies. In all 3 assessments of publication bias, we found evidence of publication bias in non-industry sponsored studies, while in industry-sponsored studies publication bias was not evident in funnel plots and Egger’s regression tests. We also found that inadequate reporting of sponsorship in animal studies is still exceedingly common.

Conclusions

In meta-analyses assessing the effects of statins on atherosclerosis and bone outcomes in animal studies, we found evidence of publication bias, though small numbers of industry-sponsored studies limit the interpretation of the trim-and-fill results. This publication bias is more prominent in non-industry sponsored studies. Industry and non-industry funded researchers may have different incentives for publication. Industry may have a financial interest to publish all preclinical animal studies to maximize the success of subsequent trials in humans, whereas non-industry funded academics may prefer to publish high impact statistically significant results only. Differences in previously published effect estimates between industry- and non-industry sponsored animal studies may be partially explained by publication bias.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-015-0008-z) contains supplementary material, which is available to authorized users.

Dissemination bias in systematic reviews of animal research: a systematic review

BACKGROUND

Systematic reviews of preclinical studies, in vivo animal experiments in particular, can influence clinical research and thus even clinical care. Dissemination bias, selective dissemination of positive or significant results, is one of the major threats to validity in systematic reviews also in the realm of animal studies. We conducted a systematic review to determine the number of published systematic reviews of animal studies until present, to investigate their methodological features especially with respect to assessment of dissemination bias, and to investigate the citation of preclinical systematic reviews on clinical research.

METHODS

Eligible studies for this systematic review constitute systematic reviews that summarize in vivo animal experiments whose results could be interpreted as applicable to clinical care. We systematically searched Ovid Medline, Embase, ToxNet, and ScienceDirect from 1st January 2009 to 9th January 2013 for eligible systematic reviews without language restrictions. Furthermore we included articles from two previous systematic reviews by Peters et al. and Korevaar et al.

RESULTS

The literature search and screening process resulted in 512 included full text articles. We found an increasing number of published preclinical systematic reviews over time. The methodological quality of preclinical systematic reviews was low. The majority of preclinical systematic reviews did not assess methodological quality of the included studies (71%), nor did they assess heterogeneity (81%) or dissemination bias (87%). Statistics quantifying the importance of clinical research citing systematic reviews of animal studies showed that clinical studies referred to the preclinical research mainly to justify their study or a future study (76%).

DISCUSSION

Preclinical systematic reviews may have an influence on clinical research but their methodological quality frequently remains low. Therefore, systematic reviews of animal research should be critically appraised before translating them to a clinical context.

The Future of Preclinical Animal Models in Pharmaceutical Discovery and Development

Animal models have provided an important tool to help make the decision to take potential therapies from preclinical studies to humans. In the past several years, the strong reliance of the pharmaceutical discovery and development process on the use of animal models has come under increasing scrutiny for ethical and scientific reasons. Several prominent and widely publicized articles have reported limited concordance of animal experiments with subsequent human clinical trials. Recent assessments of the quality of animal studies have suggested that this translational failure may be due in part to shortcomings in the planning, conduct, and reporting of in vivo studies. This article will emphasize methods to assure best practice rigor in animal study methods and reporting. It will introduce the so-called scientific 3Rs of relevance, robustness, and reproducibility to the in vivo study approach and will review important new trends in the animal research and pharmaceutical discovery and development communities.

Systematic reviews of animal studies; missing link in translational research?

BACKGROUND

The methodological quality of animal studies is an important factor hampering the translation of results from animal studies to a clinical setting. Systematic reviews of animal studies may provide a suitable method to assess and thereby improve their methodological quality.

OBJECTIVES

The aims of this study were: 1) to evaluate the risk of bias assessment in animal-based systematic reviews, and 2) to study the internal validity of the primary animal studies included in these systematic reviews.

DATA SOURCES

We systematically searched Pubmed and Embase for SRs of preclinical animal studies published between 2005 and 2012.

RESULTS

A total of 91 systematic reviews met our inclusion criteria. The risk of bias was assessed in 48 (52.7%) of these 91 systematic reviews. Thirty-three (36.3%) SRs provided sufficient information to evaluate the internal validity of the included studies. Of the evaluated primary studies, 24.6% was randomized, 14.6% reported blinding of the investigator/caretaker, 23.9% blinded the outcome assessment, and 23.1% reported drop-outs.

CONCLUSIONS

To improve the translation of animal data to clinical practice, systematic reviews of animal studies are worthwhile, but the internal validity of primary animal studies needs to be improved. Furthermore, risk of bias should be assessed by systematic reviews of animal studies to provide insight into the reliability of the available evidence.

Publication bias in animal research: a systematic review protocol

BACKGROUND

Systematic reviews and meta-analyses of pre-clinical studies, in vivo animal experiments in particular, can influence clinical care. Publication bias is one of the major threats of validity in systematic reviews and meta-analyses. Previous empirical studies suggested that systematic reviews and meta-analyses have become more prevalent until 2010 and found evidence for compromised methodological rigor with a trend towards improvement. We aim to comprehensively summarize and update the evidence base on systematic reviews and meta-analyses of animal studies, their methodological quality and assessment of publication bias in particular.

METHODS/DESIGN

The objectives of this systematic review are as follows: •To investigate the epidemiology of published systematic reviews of animal studies until present. •To examine methodological features of systematic reviews and meta-analyses of animal studies with special attention to the assessment of publication bias. •To investigate the influence of systematic reviews of animal studies on clinical research by examining citations of the systematic reviews by clinical studies. Eligible studies for this systematic review constitute systematic reviews and meta-analyses that summarize in vivo animal experiments with the purpose of reviewing animal evidence to inform human health. We will exclude genome-wide association studies and animal experiments with the main purpose to learn more about fundamental biology, physical functioning or behavior. In addition to the inclusion of systematic reviews and meta-analyses identified by other empirical studies, we will systematically search Ovid Medline, Embase, ToxNet, and ScienceDirect from 2009 to January 2013 for further eligible studies without language restrictions. Two reviewers working independently will assess titles, abstracts, and full texts for eligibility and extract relevant data from included studies. Data reporting will involve a descriptive summary of meta-analyses and systematic reviews.

DISCUSSION

Results are expected to be publicly available later in 2013 and may form the basis for recommendations to improve the quality of systematic reviews and meta-analyses of animal studies and their use with respect to clinical care.

Progress in using systematic reviews of animal studies to improve translational research

Carlijn Hooijmans and colleagues discuss developments that might improve the quality and translation of animal research, focusing on the importance of systematic reviews, the role of an international register of animal studies, and cooperation across the scientific community. Please see later in the article for the Editors' Summary

Publication bias in laboratory animal research: a survey on magnitude, drivers, consequences and potential solutions

Context

Publication bias jeopardizes evidence-based medicine, mainly through biased literature syntheses. Publication bias may also affect laboratory animal research, but evidence is scarce.

Objectives

To assess the opinion of laboratory animal researchers on the magnitude, drivers, consequences and potential solutions for publication bias. And to explore the impact of size of the animals used, seniority of the respondent, working in a for-profit organization and type of research (fundamental, pre-clinical, or both) on those opinions.

Design

Internet-based survey.

Setting

All animal laboratories in The Netherlands.

Participants

Laboratory animal researchers.

Main Outcome Measure(s)

Median (interquartile ranges) strengths of beliefs on 5 and 10-point scales (1: totally unimportant to 5 or 10: extremely important).

Results

Overall, 454 researchers participated. They considered publication bias a problem in animal research (7 (5 to 8)) and thought that about 50% (32–70) of animal experiments are published. Employees (n = 21) of for-profit organizations estimated that 10% (5 to 50) are published. Lack of statistical significance (4 (4 to 5)), technical problems (4 (3 to 4)), supervisors (4 (3 to 5)) and peer reviewers (4 (3 to 5)) were considered important reasons for non-publication (all on 5-point scales). Respondents thought that mandatory publication of study protocols and results, or the reasons why no results were obtained, may increase scientific progress but expected increased bureaucracy. These opinions did not depend on size of the animal used, seniority of the respondent or type of research.

Conclusions

Non-publication of “negative” results appears to be prevalent in laboratory animal research. If statistical significance is indeed a main driver of publication, the collective literature on animal experimentation will be biased. This will impede the performance of valid literature syntheses. Effective, yet efficient systems should be explored to counteract selective reporting of laboratory animal research.