Structured approach to the elicitation of expert beliefs for a Bayesian-designed clinical trial: a case study

Opioid-related Mortality in United States Death Certificate Data: A Quantitative Bias Analysis With Expert Elicitation of Bias Parameters

BACKGROUND

Opioid-related mortality is an important public health problem in the United States. Incidence estimates rely on death certificate data generated by health care providers and medical examiners. Opioid overdoses may be underreported when other causes of death appear plausible. We applied physician-elicited death certificate bias parameters to quantitative bias analyses assessing potential age-related differential misclassification in US opioid-related mortality estimates.

METHODS

We obtained cause-of-death data (US, 2017) from the National Center for Health Statistics and calculated crude opioid-related outpatient death counts by age category (25-54, 55-64, 65+). We elicited beliefs from 10 primary care physicians on sensitivity of opioid-related death classification from death certificates. We summarized elicited sensitivity estimates, calculated plausible specificity values, and applied resulting parameters in a probabilistic bias analysis.

RESULTS

Physicians estimated wide sensitivity ranges for classification of opioid-related mortality by death certificates, with lower estimated sensitivities among older age groups. Probabilistic bias analyses adjusting for physician-estimated misclassification indicated 3.1 times more (95% uncertainty interval: 1.2-23.5) opioid-related deaths than the observed death count in the 65+ age group. All age groups had substantial increases in bias-adjusted death counts.

CONCLUSIONS

We developed and implemented a feasible method of eliciting physician expert opinion on bias parameters for sensitivity of a medical record-based death indicator and applied findings in quantitative bias analyses adjusting for differential misclassification. Our findings are consistent with the hypothesis that opioid-related mortality rates may be substantially underestimated, particularly among older adults, due to misclassification in cause-of-death data from death certificates.

Informative g-Priors for Mixed Models

Zellner’s objective g-prior has been widely used in linear regression models due to its simple interpretation and computational tractability in evaluating marginal likelihoods. However, the g-prior further allows portioning the prior variability explained by the linear predictor versus that of pure noise. In this paper, we propose a novel yet remarkably simple g-prior specification when a subject matter expert has information on the marginal distribution of the response yi. The approach is extended for use in mixed models with some surprising but intuitive results. Simulation studies are conducted to compare the model fitting under the proposed g-prior with that under other existing priors.

Conditional power and friends: The why and how of (un)planned, unblinded sample size recalculations in confirmatory trials

Adapting the final sample size of a trial to the evidence accruing during the trial is a natural way to address planning uncertainty. Since the sample size is usually determined by an argument based on the power of the trial, an interim analysis raises the question of how the final sample size should be determined conditional on the accrued information. To this end, we first review and compare common approaches to estimating conditional power, which is often used in heuristic sample size recalculation rules. We then discuss the connection of heuristic sample size recalculation and optimal two-stage designs, demonstrating that the latter is the superior approach in a fully preplanned setting. Hence, unplanned design adaptations should only be conducted as reaction to trial-external new evidence, operational needs to violate the originally chosen design, or post hoc changes in the optimality criterion but not as a reaction to trial-internal data. We are able to show that commonly discussed sample size recalculation rules lead to paradoxical adaptations where an initially planned optimal design is not invariant under the adaptation rule even if the planning assumptions do not change. Finally, we propose two alternative ways of reacting to newly emerging trial-external evidence in ways that are consistent with the originally planned design to avoid such inconsistencies.

A Review of Bayesian Perspectives on Sample Size Derivation for Confirmatory Trials

Sample size derivation is a crucial element of planning any confirmatory trial. The required sample size is typically derived based on constraints on the maximal acceptable Type I error rate and minimal desired power. Power depends on the unknown true effect and tends to be calculated either for the smallest relevant effect or a likely point alternative. The former might be problematic if the minimal relevant effect is close to the null, thus requiring an excessively large sample size, while the latter is dubious since it does not account for the a priori uncertainty about the likely alternative effect. A Bayesian perspective on sample size derivation for a frequentist trial can reconcile arguments about the relative a priori plausibility of alternative effects with ideas based on the relevance of effect sizes. Many suggestions as to how such "hybrid" approaches could be implemented in practice have been put forward. However, key quantities are often defined in subtly different ways in the literature. Starting from the traditional entirely frequentist approach to sample size derivation, we derive consistent definitions for the most commonly used hybrid quantities and highlight connections, before discussing and demonstrating their use in sample size derivation for clinical trials.

Prior Elicitation for Use in Clinical Trial Design and Analysis: A Literature Review

Bayesian inference is increasingly popular in clinical trial design and analysis. The subjective knowledge derived from an expert elicitation procedure may be useful to define a prior probability distribution when no or limited data is available. This work aims to investigate the state-of-the-art Bayesian prior elicitation methods with a focus on clinical trial research. A literature search on the Current Index to Statistics (CIS), PubMed, and Web of Science (WOS) databases, considering “prior elicitation” as a search string, was run on 1 November 2020. Summary statistics and trend of publications over time were reported. Finally, a Latent Dirichlet Allocation (LDA) model was developed to recognise latent topics in the pertinent papers retrieved. A total of 460 documents pertinent to the Bayesian prior elicitation were identified. Of these, 213 (45.4%) were published in the “Probability and Statistics” area. A total of 42 articles pertain to clinical trial and the majority of them (81%) reports parametric techniques as elicitation method. The last decade has seen an increased interest in prior elicitation and the gap between theory and application getting narrower and narrower. Given the promising flexibility of non-parametric approaches to the experts’ elicitation, more efforts are needed to ensure their diffusion also in applied settings.

Better decision making in drug development through adoption of formal prior elicitation

With the continued increase in the use of Bayesian methods in drug development, there is a need for statisticians to have tools to develop robust and defensible informative prior distributions. Whilst relevant empirical data should, where possible, provide the basis for such priors, it is often the case that limitations in data and/or our understanding may preclude direct construction of a data-based prior. Formal expert elicitation methods are a key technique that can be used to determine priors in these situations. Within GlaxoSmithKline, we have adopted a structured approach to prior elicitation on the basis of the SHELF elicitation framework and routinely use this in conjunction with calculation of probability of success (assurance) of the next study(s) to inform internal decision making at key project milestones. The aim of this paper is to share our experiences of embedding the use of prior elicitation within a large pharmaceutical company, highlighting both the benefits and challenges of prior elicitation through a series of case studies. We have found that putting team beliefs into the shape of a quantitative probability distribution provides a firm anchor for all internal decision making, enabling teams to provide investment boards with formally appropriate estimates of the probability of trial success as well as robust plans for interim decision rules where appropriate. As an added benefit, the elicitation process provides transparency about the beliefs and risks of the potential medicine, ultimately enabling better portfolio and company-wide decision making.

Decision-theoretic designs for a series of trials with correlated treatment effects using the Sarmanov multivariate beta-binomial distribution

The motivation for the work in this article is the setting in which a number of treatments are available for evaluation in phase II clinical trials and where it may be infeasible to try them concurrently because the intended population is small. This paper introduces an extension of previous work on decision-theoretic designs for a series of phase II trials. The program encompasses a series of sequential phase II trials with interim decision making and a single two-arm phase III trial. The design is based on a hybrid approach where the final analysis of the phase III data is based on a classical frequentist hypothesis test, whereas the trials are designed using a Bayesian decision-theoretic approach in which the unknown treatment effect is assumed to follow a known prior distribution. In addition, as treatments are intended for the same population it is not unrealistic to consider treatment effects to be correlated. Thus, the prior distribution will reflect this. Data from a randomized trial of severe arthritis of the hip are used to test the application of the design. We show that the design on average requires fewer patients in phase II than when the correlation is ignored. Correspondingly, the time required to recommend an efficacious treatment for phase III is quicker.

Value of information methods to design a clinical trial in a small population to optimise a health economic utility function

Background

Most confirmatory randomised controlled clinical trials (RCTs) are designed with specified power, usually 80% or 90%, for a hypothesis test conducted at a given significance level, usually 2.5% for a one-sided test. Approval of the experimental treatment by regulatory agencies is then based on the result of such a significance test with other information to balance the risk of adverse events against the benefit of the treatment to future patients. In the setting of a rare disease, recruiting sufficient patients to achieve conventional error rates for clinically reasonable effect sizes may be infeasible, suggesting that the decision-making process should reflect the size of the target population.

Methods

We considered the use of a decision-theoretic value of information (VOI) method to obtain the optimal sample size and significance level for confirmatory RCTs in a range of settings. We assume the decision maker represents society. For simplicity we assume the primary endpoint to be normally distributed with unknown mean following some normal prior distribution representing information on the anticipated effectiveness of the therapy available before the trial. The method is illustrated by an application in an RCT in haemophilia A. We explicitly specify the utility in terms of improvement in primary outcome and compare this with the costs of treating patients, both financial and in terms of potential harm, during the trial and in the future.

Results

The optimal sample size for the clinical trial decreases as the size of the population decreases. For non-zero cost of treating future patients, either monetary or in terms of potential harmful effects, stronger evidence is required for approval as the population size increases, though this is not the case if the costs of treating future patients are ignored.

Conclusions

Decision-theoretic VOI methods offer a flexible approach with both type I error rate and power (or equivalently trial sample size) depending on the size of the future population for whom the treatment under investigation is intended. This might be particularly suitable for small populations when there is considerable information about the patient population.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0475-0) contains supplementary material, which is available to authorized users.

Approaches to sample size calculation for clinical trials in rare diseases

We discuss 3 alternative approaches to sample size calculation: traditional sample size calculation based on power to show a statistically significant effect, sample size calculation based on assurance, and sample size based on a decision-theoretic approach. These approaches are compared head-to-head for clinical trial situations in rare diseases. Specifically, we consider 3 case studies of rare diseases (Lyell disease, adult-onset Still disease, and cystic fibrosis) with the aim to plan the sample size for an upcoming clinical trial. We outline in detail the reasonable choice of parameters for these approaches for each of the 3 case studies and calculate sample sizes. We stress that the influence of the input parameters needs to be investigated in all approaches and recommend investigating different sample size approaches before deciding finally on the trial size. Highly influencing for the sample size are choice of treatment effect parameter in all approaches and the parameter for the additional cost of the new treatment in the decision-theoretic approach. These should therefore be discussed extensively.

Reporting Guidelines for the Use of Expert Judgement in Model-Based Economic Evaluations

INTRODUCTION

Expert judgement has a role in model-based economic evaluations (EEs) of healthcare interventions. This study aimed to produce reporting criteria for two types of study design to use expert judgement in model-based EE: (i) an expert elicitation (quantitative) study; and (ii) a Delphi study to collate (qualitative) expert opinion.

METHODS

A two-round online Delphi process identified the degree of consensus for four core definitions (expert; expert parameter values; expert elicitation study; expert opinion) and two sets of reporting criteria in a purposive sample of experts. The initial set of reporting criteria comprised 17 statements for reporting a study to elicit parameter values and/or distributions and 11 statements for reporting a Delphi survey to obtain expert opinion. Fifty experts were invited to become members of the Delphi process panel by e-mail. Data analysis summarised the extent of agreement (using a pre-defined 75 % 'consensus' threshold) on the definitions and suggested reporting criteria. Free-text comments were analysed using thematic analysis.

RESULTS

The final panel comprised 12 experts. Consensus was achieved for the definitions of expert (88 %); expert parameter values (83 %); and expert elicitation study (83 %). The panel recommended criteria to use when reporting an expert elicitation study (16 criteria) and a Delphi study to collate expert opinion (11 criteria).

CONCLUSION

This study has produced guidelines for reporting two types of study design to use expert judgement in model-based EE: (i) an expert elicitation study requiring 16 reporting criteria; and (ii) a Delphi study to collate expert opinion requiring 11 reporting criteria.

Evidence, eminence and extrapolation

A full independent drug development programme to demonstrate efficacy may not be ethical and/or feasible in small populations such as paediatric populations or orphan indications. Different levels of extrapolation from a larger population to smaller target populations are widely used for supporting decisions in this situation. There are guidance documents in drug regulation, where a weakening of the statistical rigour for trials in the target population is mentioned to be an option for dealing with this problem. To this end, we propose clinical trials designs, which make use of prior knowledge on efficacy for inference. We formulate a framework based on prior beliefs in order to investigate when the significance level for the test of the primary endpoint in confirmatory trials can be relaxed (and thus the sample size can be reduced) in the target population while controlling a certain posterior belief in effectiveness after rejection of the null hypothesis in the corresponding confirmatory statistical test. We show that point‐priors may be used in the argumentation because under certain constraints, they have favourable limiting properties among other types of priors. The crucial quantity to be elicited is the prior belief in the possibility of extrapolation from a larger population to the target population. We try to illustrate an existing decision tree for extrapolation to paediatric populations within our framework. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

Changing the Paradigm for the Treatment and Development of New Therapies for FSGS

Focal segmental glomerulosclerosis (FSGS) is a renal pathology finding that represents a constellation of rare kidney diseases, which manifest as proteinuria, edema nephrotic syndrome, hypertension, and increased risk for kidney failure. Therapeutic options for FSGS are reviewed displaying the expected efficacy from 25 to 69% depending on specific therapy, patient characteristics, cost, and common side effects. This variability in treatment response is likely caused, in part, by the heterogeneity in the etiology and active molecular mechanisms of FSGS. Clinical trials in FSGS have been scant in number and slow to recruit, which may stem, in part, from reliance on classic clinical trial design paradigms. Traditional clinical trial designs based on the "learn and confirm" paradigm may not be appropriate for rare diseases, such as FSGS. Future drug development and testing will require novel approaches to trial designs that have the capacity to enrich study populations and adapt the trial in a planned way to gain efficiencies in trial completion timelines. A clinical trial simulation is provided that compares a classical and more modern design to determine the maximum tolerated dose in FSGS.

Integration of PKPD relationships into benefit-risk analysis

AIM

Despite the continuous endeavour to achieve high standards in medical care through effectiveness measures, a quantitative framework for the assessment of the benefit-risk balance of new medicines is lacking prior to regulatory approval. The aim of this short review is to summarise the approaches currently available for benefit-risk assessment. In addition, we propose the use of pharmacokinetic-pharmacodynamic (PKPD) modelling as the pharmacological basis for evidence synthesis and evaluation of novel therapeutic agents.

METHODS

A comprehensive literature search has been performed using MESH terms in PubMed, in which articles describing benefit-risk assessment and modelling and simulation were identified. In parallel, a critical review of multi-criteria decision analysis (MCDA) is presented as a tool for characterising a drug's safety and efficacy profile.

RESULTS

A definition of benefits and risks has been proposed by the European Medicines Agency (EMA), in which qualitative and quantitative elements are included. However, in spite of the value of MCDA as a quantitative method, decisions about benefit-risk balance continue to rely on subjective expert opinion. By contrast, a model-informed approach offers the opportunity for a more comprehensive evaluation of benefit-risk balance before extensive evidence is generated in clinical practice.

CONCLUSIONS

Benefit-risk balance should be an integral part of the risk management plan and as such considered before marketing authorisation. Modelling and simulation can be incorporated into MCDA to support the evidence synthesis as well evidence generation taking into account the underlying correlations between favourable and unfavourable effects. In addition, it represents a valuable tool for the optimization of protocol design in effectiveness trials.

Decision-theoretic designs for small trials and pilot studies: A review

Pilot studies and other small clinical trials are often conducted but serve a variety of purposes and there is little consensus on their design. One paradigm that has been suggested for the design of such studies is Bayesian decision theory. In this article, we review the literature with the aim of summarizing current methodological developments in this area. We find that decision-theoretic methods have been applied to the design of small clinical trials in a number of areas. We divide our discussion of published methods into those for trials conducted in a single stage, those for multi-stage trials in which decisions are made through the course of the trial at a number of interim analyses, and those that attempt to design a series of clinical trials or a drug development programme. In all three cases, a number of methods have been proposed, depending on the decision maker’s perspective being considered and the details of utility functions that are used to construct the optimal design.

Bayesian methods including nonrandomized study data increased the efficiency of postlaunch RCTs

OBJECTIVES

Findings from nonrandomized studies on safety or efficacy of treatment in patient subgroups may trigger postlaunch randomized clinical trials (RCTs). In the analysis of such RCTs, results from nonrandomized studies are typically ignored. This study explores the trade-off between bias and power of Bayesian RCT analysis incorporating information from nonrandomized studies.

STUDY DESIGN AND SETTING

A simulation study was conducted to compare frequentist with Bayesian analyses using noninformative and informative priors in their ability to detect interaction effects. In simulated subgroups, the effect of a hypothetical treatment differed between subgroups (odds ratio 1.00 vs. 2.33). Simulations varied in sample size, proportions of the subgroups, and specification of the priors.

RESULTS

As expected, the results for the informative Bayesian analyses were more biased than those from the noninformative Bayesian analysis or frequentist analysis. However, because of a reduction in posterior variance, informative Bayesian analyses were generally more powerful to detect an effect. In scenarios where the informative priors were in the opposite direction of the RCT data, type 1 error rates could be 100% and power 0%.

CONCLUSION

Bayesian methods incorporating data from nonrandomized studies can meaningfully increase power of interaction tests in postlaunch RCTs.

Implementation of maximin efficient designs in dose-finding studies

This paper considers the maximin approach for designing clinical studies. A maximin efficient design maximizes the smallest efficiency when compared with a standard design, as the parameters vary in a specified subset of the parameter space. To specify this subset of parameters in a real situation, a four-step procedure using elicitation based on expert opinions is proposed. Further, we describe why and how we extend the initially chosen subset of parameters to a much larger set in our procedure. By this procedure, the maximin approach becomes feasible for dose-finding studies. Maximin efficient designs have shown to be numerically difficult to construct. However, a new algorithm, the H-algorithm, considerably simplifies the construction of these designs. We exemplify the maximin efficient approach by considering a sigmoid Emax model describing a dose-response relationship and compare inferential precision with that obtained when using a uniform design. The design obtained is shown to be at least 15% more efficient than the uniform design.

Bayesian methods for the design and interpretation of clinical trials in very rare diseases

This paper considers the design and interpretation of clinical trials comparing treatments for conditions so rare that worldwide recruitment efforts are likely to yield total sample sizes of 50 or fewer, even when patients are recruited over several years. For such studies, the sample size needed to meet a conventional frequentist power requirement is clearly infeasible. Rather, the expectation of any such trial has to be limited to the generation of an improved understanding of treatment options. We propose a Bayesian approach for the conduct of rare-disease trials comparing an experimental treatment with a control where patient responses are classified as a success or failure. A systematic elicitation from clinicians of their beliefs concerning treatment efficacy is used to establish Bayesian priors for unknown model parameters. The process of determining the prior is described, including the possibility of formally considering results from related trials. As sample sizes are small, it is possible to compute all possible posterior distributions of the two success rates. A number of allocation ratios between the two treatment groups can be considered with a view to maximising the prior probability that the trial concludes recommending the new treatment when in fact it is non-inferior to control. Consideration of the extent to which opinion can be changed, even by data from the best feasible design, can help to determine whether such a trial is worthwhile.

Methods to elicit probability distributions from experts: a systematic review of reported practice in health technology assessment

BACKGROUND

Elicitation is a technique that can be used to obtain probability distribution from experts about unknown quantities. We conducted a methodology review of reports where probability distributions had been elicited from experts to be used in model-based health technology assessments.

METHODS

Databases including MEDLINE, EMBASE and the CRD database were searched from inception to April 2013. Reference lists were checked and citation mapping was also used. Studies describing their approach to the elicitation of probability distributions were included. Data was abstracted on pre-defined aspects of the elicitation technique. Reports were critically appraised on their consideration of the validity, reliability and feasibility of the elicitation exercise.

RESULTS

Fourteen articles were included. Across these studies, the most marked features were heterogeneity in elicitation approach and failure to report key aspects of the elicitation method. The most frequently used approaches to elicitation were the histogram technique and the bisection method. Only three papers explicitly considered the validity, reliability and feasibility of the elicitation exercises.

CONCLUSION

Judged by the studies identified in the review, reports of expert elicitation are insufficient in detail and this impacts on the perceived usability of expert-elicited probability distributions. In this context, the wider credibility of elicitation will only be improved by better reporting and greater standardisation of approach. Until then, the advantage of eliciting probability distributions from experts may be lost.

Guidance on the implementation and reporting of a drug safety Bayesian network meta-analysis

The Drug Information Association Bayesian Scientific Working Group (BSWG) was formed in 2011 with a vision to ensure that Bayesian methods are well understood and broadly utilized for design and analysis and throughout the medical product development process, and to improve industrial, regulatory, and economic decision making. The group, composed of individuals from academia, industry, and regulatory, has as its mission to facilitate the appropriate use and contribute to the progress of Bayesian methodology. In this paper, the safety sub-team of the BSWG explores the use of Bayesian methods when applied to drug safety meta-analysis and network meta-analysis. Guidance is presented on the conduct and reporting of such analyses. We also discuss different structural model assumptions and provide discussion on prior specification. The work is illustrated through a case study involving a network meta-analysis related to the cardiovascular safety of non-steroidal anti-inflammatory drugs.