The Potential of Adaptive Design in Animal Studies

Reducing the false discovery rate of preclinical animal research with Bayesian statistical decision criteria

The success of preclinical research hinges on exploratory and confirmatory animal studies. Traditional null hypothesis significance testing is a common approach to eliminate the chaff from a collection of drugs, so that only the most promising treatments are funneled through to clinical research phases. Balancing the number of false discoveries and false omissions is an important aspect to consider during this process. In this paper, we compare several preclinical research pipelines, either based on null hypothesis significance testing or based on Bayesian statistical decision criteria. We build on a recently published large-scale meta-analysis of reported effect sizes in preclinical animal research and elicit a non-informative prior distribution under which both approaches are compared. After correcting for publication bias and shrinkage of effect sizes in replication studies, simulations show that (i) a shift towards statistical approaches which explicitly incorporate the minimum clinically important difference reduces the false discovery rate of frequentist approaches and (ii) a shift towards Bayesian statistical decision criteria can improve the reliability of preclinical animal research by reducing the number of false-positive findings. It is shown that these benefits hold while keeping the number of experimental units low which are required for a confirmatory follow-up study. Results show that Bayesian statistical decision criteria can help in improving the reliability of preclinical animal research and should be considered more frequently in practice.

Medical honey for canine nasal intertrigo: A randomized, blinded, placebo-controlled, adaptive clinical trial to support antimicrobial stewardship in veterinary dermatology

Intertrigo is a skin fold dermatitis often requiring recurrent treatment with topical antiseptics or antibiotics, which can select antimicrobial resistance. To minimize this risk, we tested the effectiveness of medical-grade Manuka honey at treating intertrigo as compared to a placebo hydrogel. We additionally characterized the culturable microbial flora of intertrigo and recorded any adverse effect with either treatment. During this randomized, placebo-controlled, double-blinded, adaptive group-sequential trial, the owners washed the affected sites on their dog with water, dried and applied a thin film of either the honey or the placebo product once daily for 21 days. Cytological and lesional composite scores, owner-assessed pruritus, and microbial cultures were assessed prior to treatment and on Day-22. The fixed effects of time, treatment, and animal-related variables on the pruritus and on each composite score, accounting for random dog effect, were estimated separately with generalized linear mixed models for repeated count outcomes (α = 0.05). The null hypothesis of equal treatment effects was rejected at the first interim analysis. The placebo (n = 16 dogs) outperformed the medical honey (n = 13 dogs) at improving both the cytological score (Treatment×Time = -0.35±0.17; P = 0.04) and clinical score (Treatment×Time = -0.28±0.13; P = 0.04). A microbial burden score higher than 4 increased the severity of the cytological score (dichotomous score: 0.29±0.11; P = 0.01), which in turn increased the severity of the clinical score and pruritus score. For every unit increase in cytological score, the linear predictor of clinical score increased by 0.042±0.019 (P = 0.03), and the one of pruritus score increased by 0.12±0.05 (P = 0.01). However, medical honey outperformed the placebo at alleviating the dog's owner-assessed pruritus after statistically controlling for masking effects (Time = -0.94±0.24; P = 0.002; and Treatment×Time = 0.80±0.36; P = 0.04). Unilateral tests of the least-square mean estimates revealed that honey only significantly improved the pruritus (Hommel-adjusted P = 0.003), while the placebo only improved the cytological and clinical scores (Hommel-adjusted P = 0.01 and 0.002, respectively). Taken together, these results question the value of Manuka honey at treating nasal intertrigo in dogs.

ToxRefDB version 2.0: Improved utility for predictive and retrospective toxicology analyses

The Toxicity Reference Database (ToxRefDB) structures information from over 5000 in vivo toxicity studies, conducted largely to guidelines or specifications from the US Environmental Protection Agency and the National Toxicology Program, into a public resource for training and validation of predictive models. Herein, ToxRefDB version 2.0 (ToxRefDBv2) development is described. Endpoints were annotated (e.g. required, not required) according to guidelines for subacute, subchronic, chronic, developmental, and multigenerational reproductive designs, distinguishing negative responses from untested. Quantitative data were extracted, and dose-response modeling for nearly 28,000 datasets from nearly 400 endpoints using Benchmark Dose (BMD) Modeling Software were generated and stored. Implementation of controlled vocabulary improved data quality; standardization to guideline requirements and cross-referencing with United Medical Language System (UMLS) connects ToxRefDBv2 observations to vocabularies linked to UMLS, including PubMed medical subject headings. ToxRefDBv2 allows for increased connections to other resources and has greatly enhanced quantitative and qualitative utility for predictive toxicology.

Efficient two-stage designs and proper inference for animal studies

Cost-effective yet efficient designs are critical to the success of animal studies. We propose a two-stage design for cost-effectiveness animal studies with continuous outcomes. Given the data from the two-stage design, we derive the exact distribution of the test statistic under null hypothesis to appropriately adjust for the design's adaptiveness. We further generalize the design and inferential procedure to the K-sample case with multiple comparison adjustment. We conduct simulation studies to evaluate the small sample behavior of the proposed design and test procedure. The results indicate that the proposed test procedure controls the type I error rate for the one-sample design and the family-wise error rate for K-sample design very well; whereas the naive approach that ignores the design's adaptiveness due to the interim look severely inflates the type I error rate or family-wise error rate. Compared with the standard one-stage design, the proposed design generally requires a smaller sample size.

Increasing efficiency of preclinical research by group sequential designs

Despite the potential benefits of sequential designs, studies evaluating treatments or experimental manipulations in preclinical experimental biomedicine almost exclusively use classical block designs. Our aim with this article is to bring the existing methodology of group sequential designs to the attention of researchers in the preclinical field and to clearly illustrate its potential utility. Group sequential designs can offer higher efficiency than traditional methods and are increasingly used in clinical trials. Using simulation of data, we demonstrate that group sequential designs have the potential to improve the efficiency of experimental studies, even when sample sizes are very small, as is currently prevalent in preclinical experimental biomedicine. When simulating data with a large effect size of d = 1 and a sample size of n = 18 per group, sequential frequentist analysis consumes in the long run only around 80% of the planned number of experimental units. In larger trials (n = 36 per group), additional stopping rules for futility lead to the saving of resources of up to 30% compared to block designs. We argue that these savings should be invested to increase sample sizes and hence power, since the currently underpowered experiments in preclinical biomedicine are a major threat to the value and predictiveness in this research domain.