Better decision making in drug development through adoption of formal prior elicitation

Delta Inflation, Optimism Bias, and Uncertainty in Clinical Trials

The phenomenon of delta inflation, in which design treatment effects tend to exceed observed treatment effects, has been documented in several therapeutic areas. Delta inflation has often been attributed to investigators' optimism bias, or an unwarranted belief in the efficacy of new treatments. In contrast, we argue that delta inflation may be a natural consequence of clinical equipoise, that is, genuine uncertainty about the relative benefits of treatments before a trial is initiated. We review alternative methodologies that can offer more direct evidence about investigators' beliefs, including Bayesian priors and forecasting analysis. The available evidence for optimism bias appears to be mixed, and can be assessed only where uncertainty is expressed explicitly at the trial design stage.

Mixture and Non-mixture Cure Models for Health Technology Assessment: What You Need to Know

There is increasing interest in the use of cure modelling to inform health technology assessment (HTA) due to the development of new treatments that appear to offer the potential for cure in some patients. However, cure models are often not included in evidence dossiers submitted to HTA agencies, and they are relatively rarely relied upon to inform decision-making. This is likely due to a lack of understanding of how cure models work, what they assume, and how reliable they are. In this tutorial we explain why and when cure models may be useful for HTA, describe the key characteristics of mixture and non-mixture cure models, and demonstrate their use in a range of scenarios, providing Stata code. We highlight key issues that must be taken into account by analysts when fitting these models and by reviewers and decision-makers when interpreting their predictions. In particular, we note that flexible parametric non-mixture cure models have not been used in HTA, but they offer advantages that make them well suited to an HTA context when a cure assumption is valid but follow-up is limited.

Assurance methods for designing a clinical trial with a delayed treatment effect

An assurance calculation is a Bayesian alternative to a power calculation. One may be performed to aid the planning of a clinical trial, specifically setting the sample size or to support decisions about whether or not to perform a study. Immuno-oncology is a rapidly evolving area in the development of anticancer drugs. A common phenomenon that arises in trials of such drugs is one of delayed treatment effects, that is, there is a delay in the separation of the survival curves. To calculate assurance for a trial in which a delayed treatment effect is likely to be present, uncertainty about key parameters needs to be considered. If uncertainty is not considered, the number of patients recruited may not be enough to ensure we have adequate statistical power to detect a clinically relevant treatment effect and the risk of an unsuccessful trial is increased. We present a new elicitation technique for when a delayed treatment effect is likely and show how to compute assurance using these elicited prior distributions. We provide an example to illustrate how this can be used in practice and develop open-source software to implement our methods. Our methodology has the potential to improve the success rate and efficiency of Phase III trials in immuno-oncology and for other treatments where a delayed treatment effect is expected to occur.

Futility Interim Analysis Based on Probability of Success Using a Surrogate Endpoint

In clinical trials with time-to-event data, the evaluation of treatment efficacy can be a long and complex process, especially when considering long-term primary endpoints. Using surrogate endpoints to correlate the primary endpoint has become a common practice to accelerate decision-making. Moreover, the ethical need to minimize sample size and the practical need to optimize available resources have encouraged the scientific community to develop methodologies that leverage historical data. Relying on the general theory of group sequential design and using a Bayesian framework, the methodology described in this paper exploits a documented historical relationship between a clinical "final" endpoint and a surrogate endpoint to build an informative prior for the primary endpoint, using surrogate data from an early interim analysis of the clinical trial. The predictive probability of success of the trial is then used to define a futility-stopping rule. The methodology demonstrates substantial enhancements in trial operating characteristics when there is a good agreement between current and historical data. Furthermore, incorporating a robust approach that combines the surrogate prior with a vague component mitigates the impact of the minor prior-data conflicts while maintaining acceptable performance even in the presence of significant prior-data conflicts. The proposed methodology was applied to design a Phase III clinical trial in metastatic colorectal cancer, with overall survival as the primary endpoint and progression-free survival as the surrogate endpoint.

Going beyond probability of success: Opportunities for statisticians to influence quantitative decision-making at the portfolio level

The pharmaceutical industry is plagued with long, costly development and high risk. Therefore, a company's effective management and optimisation of a portfolio of projects is critical for success. Project metrics such as the probability of success enable modelling of a company's pipeline accounting for the high uncertainty inherent within the industry. Making portfolio decisions inherently involves managing risk, and statisticians are ideally positioned to champion not only the derivation of metrics for individual projects, but also advocate decision-making at a broader portfolio level. This article aims to examine the existing different portfolio decision-making approaches and to suggest opportunities for statisticians to add value in terms of introducing probabilistic thinking, quantitative decision-making, and increasingly advanced methodologies.

Algorithmic benchmark modulation: A novel method to develop success rates for clinical studies

BACKGROUND

High-quality decision-making in the pharmaceutical industry requires accurate assessments of the Probability of Technical Success of clinical trials. Failure to do so will lead to lost opportunities for both patients and investors. Pharmaceutical companies employ different methodologies to determine Probability of Technical Success values. Some companies use power and assurance calculations; others prefer to use industry benchmarks with or without the overlay of subjective modulations. At AstraZeneca, both assurance calculations and industry benchmarks are used, and both methods are combined with modulations.

METHODS

AstraZeneca has recently implemented a simple algorithm that allows for modulation of a Probability of Technical Success value. The algorithm is based on a set of multiple-choice questions. These questions cover a comprehensive set of issues that have historically been considered by AstraZeneca when subjective modulations to Probability of Technical Success values were made but do so in a much more structured way.

RESULTS

A set of 57 phase 3 Probability of Technical Success assessments suggests that AstraZeneca's historical estimation of Probability of Technical Success has been reasonably accurate. A good correlation between the subjective modulation and the modulation algorithm was found. This latter observation, combined with the finding that historically AstraZeneca has been reasonably accurate in its estimation of Probability of Technical Success, gives confidence in the validity of the novel method.

DISCUSSION

Although it is too early to demonstrate whether the method has improved the accuracy of company's Probability of Technical Success assessments, we present our data and analysis here in the hope that it may assist the pharmaceutical industry in addressing this key challenge. This new methodology, developed for pivotal studies, enables AstraZeneca to develop more consistent Probability of Technical Success assessments with less effort and can be used to adjust benchmarks as well as assurance calculations.

CONCLUSION

The Probability of Technical Success modulation algorithm addresses several concerns generally associated with assurance calculations or benchmark without modulation: selection biases, situations where little relevant prior data are available and the difficulty to model many factors affecting study outcomes. As opposed to using industry benchmarks, the Probability of Technical Success modulation algorithm allows to accommodate project-specific considerations.

Estimating the effect of nintedanib on forced vital capacity in children and adolescents with fibrosing interstitial lung disease using a Bayesian dynamic borrowing approach

BACKGROUND

The rarity of childhood interstitial lung disease (chILD) makes it challenging to conduct powered trials. In the InPedILD trial, among 39 children and adolescents with fibrosing ILD, there was a numerical benefit of nintedanib versus placebo on change in forced vital capacity (FVC) over 24 weeks (difference in mean change in FVC % predicted of 1.21 [95% confidence interval: -3.40, 5.81]). Nintedanib has shown a consistent effect on FVC across populations of adults with different diagnoses of fibrosing ILD.

METHODS

In a Bayesian dynamic borrowing analysis, prespecified before data unblinding, we incorporated data on the effect of nintedanib in adults and the data from the InPedILD trial to estimate the effect of nintedanib on FVC in children and adolescents with fibrosing ILD. The data from adults were represented as a meta-analytic predictive (MAP) prior distribution with mean 1.69 (95% credible interval: 0.49, 3.08). The adult data were weighted according to expert judgment on their relevance to the efficacy of nintedanib in chILD, obtained in a formal elicitation exercise.

RESULTS

Combined data from the MAP prior and InPedILD trial analyzed within the Bayesian framework resulted in a median difference between nintedanib and placebo in change in FVC % predicted at Week 24 of 1.63 (95% credible interval: -0.69, 3.40). The posterior probability for superiority of nintedanib versus placebo was 95.5%, reaching the predefined success criterion of at least 90%.

CONCLUSION

These findings, together with the safety data from the InPedILD trial, support the use of nintedanib in children and adolescents with fibrosing ILDs.

How to select the initial dose for a pediatric study?

In typical clinical development programs, a new drug is first developed for the adult use. Drugs are often approved for adult use or in the process of obtaining approval in adults in the target indication before pediatric development is initiated. In designing the first pediatric clinical trial, one of the challenges is to select the initial dose to be tested. The ICH E11 R1 guidance advises that chronologic age alone may not always be the most appropriate categorical determinant to define developmental subgroups in pediatric studies. In this manuscript, the approaches to utilize available data in adults related to those factors beyond age to inform the starting dose selection in pediatric drug development are discussed. Practical considerations and approaches are provided for informing pediatric starting dose. Additional considerations to use pre-clinical information are provided in the case when adult information is limited or not available.

Trends, challenges, and success factors in pharmaceutical portfolio management: Cognitive biases in decision-making and their mitigating measures

Effective portfolio management is crucial for innovation and sustaining revenue in pharmaceutical companies. This article holistically reviews trends, challenges, and approaches to pharmaceutical portfolio management and focuses, in particular, on cognitive biases in portfolio decision-making. Portfolio managers strongly rely on external innovation and face increasing competitive pressure and portfolio complexity. The ability to address biases and make robust decisions remains a challenge. Portfolio management practitioners most commonly face confirmation bias, champion bias, or misaligned incentives, which they seek to mitigate through expert input, team diversity, and rewarding truth-seeking. Ultimately, highest-quality portfolio management decision-making could be enabled by three factors: high-quality data, structured review processes, and comprehensive mitigating measures against biases in decision-making.

The Role of Expert Opinion in Projecting Long-Term Survival Outcomes Beyond the Horizon of a Clinical Trial

INTRODUCTION

Clinical trials often have short follow-ups, and long-term outcomes such as survival must be extrapolated. Current extrapolation methods often produce a wide range of survival values. To minimize uncertainty in projections, we developed a novel method that incorporates formally elicited expert opinion in a Bayesian analysis and used it to extrapolate survival in the placebo arm of DAPA-CKD, a phase 3 trial of dapagliflozin in patients with chronic kidney disease (NCT03036150).

METHODS

A summary of mortality data from 13 studies that included DAPA-CKD-like populations and training on elicitation were provided to six experts. An elicitation survey was used to gather the experts' 10- and 20-year survival estimates for patients in the placebo arm of DAPA-CKD. These estimates were combined with DAPA-CKD mortality and general population mortality (GPM) data in a Bayesian analysis to extrapolate long-term survival using seven parametric distributions. Results were compared with those from standard frequentist approaches (with and without GPM data) that do not incorporate expert opinion.

RESULTS

The group expert-elicited estimate for 20-year survival was 31% (lower estimate, 10%; upper estimate, 40%). In the Bayesian analysis, the 20-year extrapolated survival across the seven distributions was 14.9-39.1%, a range that was 2.4- and 1.6-fold smaller than those produced by the frequentist methods (0.0-56.9% without and 0.0-39.2% with GPM data).

CONCLUSIONS

Using expert opinion in a Bayesian analysis provided a robust method for extrapolating long-term survival in the placebo arm of DAPA-CKD. The method could be applied to other populations with limited survival data.

Informative g-Priors for Mixed Models

Zellner’s objective g-prior has been widely used in linear regression models due to its simple interpretation and computational tractability in evaluating marginal likelihoods. However, the g-prior further allows portioning the prior variability explained by the linear predictor versus that of pure noise. In this paper, we propose a novel yet remarkably simple g-prior specification when a subject matter expert has information on the marginal distribution of the response yi. The approach is extended for use in mixed models with some surprising but intuitive results. Simulation studies are conducted to compare the model fitting under the proposed g-prior with that under other existing priors.

Generalisations of a Bayesian decision-theoretic randomisation procedure and the impact of delayed responses

The design of sequential experiments and, in particular, randomised controlled trials involves a trade-off between operational characteristics such as statistical power, estimation bias and patient benefit. The family of randomisation procedures referred to as Constrained Randomised Dynamic Programming (CRDP), which is set in the Bayesian decision-theoretic framework, can be used to balance these competing objectives. A generalisation and novel interpretation of CRDP is proposed to highlight its inherent flexibility to adapt to a variety of practicalities and align with individual trial objectives. CRDP, as with most response-adaptive randomisation procedures, hinges on the limiting assumption of patient responses being available before allocation of the next patient. This forms one of the greatest barriers to their implementation in practice which, despite being an important research question, has not received a thorough treatment. Therefore, motivated by the existing gap between the theory of response-adaptive randomisation (which is abundant with proposed methods in the immediate response setting) and clinical practice (in which responses are typically delayed), the performance of CRDP in the presence of fixed and random delays is evaluated. Simulation results show that CRDP continues to offer patient benefit gains over alternative procedures and is relatively robust to delayed responses. To compensate for a fixed delay, a method which adjusts the time horizon used in the optimisation objective is proposed and its performance illustrated.

From innovative thinking to pharmaceutical industry implementation: Some success stories

In industry, successful innovation involves not only developing new statistical methodology, but also ensuring that this methodology is implemented successfully. This includes enabling applied statisticians to understand the method, its benefits and limitations and empowering them to implement the new method. This will include advocacy, influencing in-house and external stakeholders, such that these stakeholders are receptive to the new methodology. In this paper, we describe some industry successes and focus on our colleague, Andy Grieve's role in these.

Quantitative decision making for investment in global health intervention trials: Case study of the NEWBORN study on emollient therapy in preterm infants in Kenya

Background

Partners from an NGO, academia, industry and government applied a tool originating in the private sector – Quantitative Decision Making (QDM) – to rigorously assess whether to invest in testing a global health intervention. The proposed NEWBORN study was designed to assess whether topical emollient therapy with sunflower seed oil in infants with very low birthweight <1500 g in Kenya would result in a significant reduction in neonatal mortality compared to standard of care.

Methods

The QDM process consisted of prior elicitation, modelling of prior distributions, and simulations to assess Probability of Success (PoS) via assurance calculations. Expert opinion was elicited on the probability that emollient therapy with sunflower seed oil will have any measurable benefit on neonatal mortality based on available evidence. The distribution of effect sizes was modelled and trial data simulated using Statistical Analysis System to obtain the overall assurance which represents the PoS for the planned study. A decision-making framework was then applied to characterise the ability of the study to meet pre-selected decision-making endpoints.

Results

There was a 47% chance of a positive outcome (defined as a significant relative reduction in mortality of ≥15%), a 45% chance of a negative outcome (defined as a significant relative reduction in mortality <10%), and an 8% chance of ending in the consider zone (ie, a mortality reduction of 10 to <15%) for infants <1500 g.

Conclusions

QDM is a novel tool from industry which has utility for prioritisation of investments in global health, complementing existing tools [eg, Child Health and Nutrition Research Initiative]. Results from application of QDM to the NEWBORN study suggests that it has a high probability of producing clear results. Findings encourage future formation of public-private partnerships for health.

Optimal unplanned design modification in adaptive two-stage trials

Adaptive planning of clinical trials allows modifying the entire trial design at any time point mid-course. In this paper, we consider the case when a trial-external update of the planning assumptions during the ongoing trial makes an unforeseen design adaptation necessary. We take up the idea to construct adaptive designs with defined features by solving an optimization problem and apply it to the situation of unplanned design reassessment. By using the conditional error principle, we present an approach on how to optimally modify the trial design at an unplanned interim analysis while at the same time strictly protecting the type I error rate. This linking of optimal design planning and the conditional error principle allows sound reactions to unforeseen events that make a design reassessment necessary.

Application of Bayesian methods to accelerate rare disease drug development: scopes and hurdles

Background

Design and analysis of clinical trials for rare and ultra-rare disease pose unique challenges to the practitioners. Meeting conventional power requirements is infeasible for diseases where sample sizes are inherently very small. Moreover, rare disease populations are generally heterogeneous and widely dispersed, which complicates study enrollment and design. Leveraging all available information in rare and ultra-rare disease trials can improve both drug development and informed decision-making processes.

Main text

Bayesian statistics provides a formal framework for combining all relevant information at all stages of the clinical trial, including trial design, execution, and analysis. This manuscript provides an overview of different Bayesian methods applicable to clinical trials in rare disease. We present real or hypothetical case studies that address the key needs of rare disease drug development highlighting several specific Bayesian examples of clinical trials. Advantages and hurdles of these approaches are discussed in detail. In addition, we emphasize the practical and regulatory aspects in the context of real-life applications.

Conclusion

The use of innovative trial designs such as master protocols and complex adaptive designs in conjunction with a Bayesian approach may help to reduce sample size, select the correct treatment and population, and accurately and reliably assess the treatment effect in the rare disease setting.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13023-022-02342-5.

Methods for Extrapolating Survival Analyses for the Economic Evaluation of Advanced Therapy Medicinal Products

There are two significant challenges for analysts conducting economic evaluations of advanced therapy medicinal products (ATMPs): (i) estimating long-term treatment effects in the absence of mature clinical data, and (ii) capturing potentially complex hazard functions. This review identifies and critiques a variety of methods that can be used to overcome these challenges. The narrative review is informed by a rapid literature review of methods used for the extrapolation of survival analyses in the economic evaluation of ATMPs. There are several methods that are more suitable than traditional parametric survival modelling approaches for capturing complex hazard functions, including, cure-mixture models and restricted cubic spline models. In the absence of mature clinical data, analysts may augment clinical trial data with data from other sources to aid extrapolation, however, the relative merits of employing methods for including data from different sources is not well understood. Given the high and potentially irrecoverable costs of making incorrect decisions concerning the reimbursement or commissioning of ATMPs, it is important that economic evaluations are correctly specified, and that both parameter and structural uncertainty associated with survival extrapolations are considered. Value of information analyses allow for this uncertainty to be expressed explicitly, and in monetary terms.

Optimization of adaptive designs with respect to a performance score

Adaptive designs are an increasingly popular method for the adaptation of design aspects in clinical trials, such as the sample size. Scoring different adaptive designs helps to make an appropriate choice among the numerous existing adaptive design methods. Several scores have been proposed to evaluate adaptive designs. Moreover, it is possible to determine optimal two-stage adaptive designs with respect to a customized objective score by solving a constrained optimization problem. In this paper, we use the conditional performance score by Herrmann et al. (2020) as the optimization criterion to derive optimal adaptive two-stage designs. We investigate variations of the original performance score, for example, by assigning different weights to the score components and by incorporating prior assumptions on the effect size. We further investigate a setting where the optimization framework is extended by a global power constraint, and additional optimization of the critical value function next to the stage-two sample size is performed. Those evaluations with respect to the sample size curves and the resulting design's performance can contribute to facilitate the score's usage in practice.

Conditional power and friends: The why and how of (un)planned, unblinded sample size recalculations in confirmatory trials

Adapting the final sample size of a trial to the evidence accruing during the trial is a natural way to address planning uncertainty. Since the sample size is usually determined by an argument based on the power of the trial, an interim analysis raises the question of how the final sample size should be determined conditional on the accrued information. To this end, we first review and compare common approaches to estimating conditional power, which is often used in heuristic sample size recalculation rules. We then discuss the connection of heuristic sample size recalculation and optimal two-stage designs, demonstrating that the latter is the superior approach in a fully preplanned setting. Hence, unplanned design adaptations should only be conducted as reaction to trial-external new evidence, operational needs to violate the originally chosen design, or post hoc changes in the optimality criterion but not as a reaction to trial-internal data. We are able to show that commonly discussed sample size recalculation rules lead to paradoxical adaptations where an initially planned optimal design is not invariant under the adaptation rule even if the planning assumptions do not change. Finally, we propose two alternative ways of reacting to newly emerging trial-external evidence in ways that are consistent with the originally planned design to avoid such inconsistencies.

Improving the assessment of the probability of success in late stage drug development

There are several steps to confirming the safety and efficacy of a new medicine. A sequence of trials, each with its own objectives, is usually required. Quantitative risk metrics can be useful for informing decisions about whether a medicine should transition from one stage of development to the next. To obtain an estimate of the probability of regulatory approval, pharmaceutical companies may start with industry-wide success rates and then apply to these subjective adjustments to reflect program-specific information. However, this approach lacks transparency and fails to make full use of data from previous clinical trials. We describe a quantitative Bayesian approach for calculating the probability of success (PoS) at the end of phase II which incorporates internal clinical data from one or more phase IIb studies, industry-wide success rates, and expert opinion or external data if needed. Using an example, we illustrate how PoS can be calculated accounting for differences between the phase II data and future phase III trials, and discuss how the methods can be extended to accommodate accelerated drug development pathways.

A New Comprehensive Approach to Assess the Probability of Success of Development Programs Before Pivotal Trials

The point at which clinical development programs transition from early phase to pivotal trials is a critical milestone. Substantial uncertainty about the outcome of pivotal trials may remain even after seeing positive early phase data, and companies may need to make difficult prioritization decisions for their portfolio. The probability of success (PoS) of a program, a single number expressed as a percentage reflecting the multitude of risks that may influence the final program outcome, is a key decision-making tool. Despite its importance, companies often rely on crude industry benchmarks that may be "adjusted" by experts based on undocumented criteria and which are typically misaligned with the definition of success used to drive commercial forecasts, leading to overly optimistic expected net present value calculations. We developed a new framework to assess the PoS of a program before pivotal trials begin. Our definition of success encompasses the successful outcome of pivotal trials, regulatory approval and meeting the requirements for market access as outlined in the target product profile. The proposed approach is organized in four steps and uses an innovative Bayesian approach to synthesize all relevant evidence. The new PoS framework is systematic and transparent. It will help organizations to make more informed decisions. In this paper, we outline the rationale and elaborate on the structure of the proposed framework, provide examples, and discuss the benefits and challenges associated with its adoption.

Integrating expert knowledge for dementia risk prediction in individuals with mild cognitive impairment (MCI): a study protocol

INTRODUCTION

To date, there is no broadly accepted dementia risk score for use in individuals with mild cognitive impairment (MCI), partly because there are few large datasets available for model development. When evidence is limited, the knowledge and experience of experts becomes more crucial for risk stratification and providing MCI patients with prognosis. Structured expert elicitation (SEE) includes formal methods to quantify experts' beliefs and help experts to express their beliefs in a quantitative form, reducing biases in the process. This study proposes to (1) assess experts' beliefs about important predictors for 3-year dementia risk in persons with MCI through SEE methodology and (2) to integrate expert knowledge and patient data to derive dementia risk scores in persons with MCI using a Bayesian approach.

METHODS AND ANALYSIS

This study will use a combination of SEE methodology, prospectively collected clinical data, and statistical modelling to derive a dementia risk score in persons with MCI . Clinical expert knowledge will be quantified using SEE methodology that involves the selection and training of the experts, administration of questionnaire for eliciting expert knowledge, discussion meetings and results aggregation. Patient data from the Prospective Registry for Persons with Memory Symptoms of the Cognitive Neurosciences Clinic at the University of Calgary; the Alzheimer's Disease Neuroimaging Initiative; and the National Alzheimer's Coordinating Center's Uniform Data Set will be used for model training and validation. Bayesian Cox models will be used to incorporate patient data and elicited data to predict 3-year dementia risk.

DISCUSSION

This study will develop a robust dementia risk score that incorporates clinician expert knowledge with patient data for accurate risk stratification, prognosis and management of dementia.

Assurance in vaccine efficacy clinical trial design based on immunological responses

The assurance of a future clinical trial is a key quantitative tool for decision-making in drug development. It is derived from prior knowledge (Bayesian approach) about the clinical endpoint of interest, typically from previous clinical trials. In this paper, we examine assurance in the specific context of vaccine development, where early development (Phase 2) is often based on immunological endpoints (e.g., antibody levels), while the confirmatory trial (Phase 3) is based on the clinical endpoint (very large sample sizes and long follow-up). Our proposal is to use the Phase 2 vaccine efficacy predicted by the immunological endpoint (using a model estimated from epidemiological studies) as prior information for the calculation of the assurance.

Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study

BACKGROUND

Many decisions in health care aim to maximise health, requiring judgements about interventions that may have higher health effects but potentially incur additional costs (cost-effectiveness framework). The evidence used to establish cost-effectiveness is typically uncertain and it is important that this uncertainty is characterised. In situations in which evidence is uncertain, the experience of experts is essential. The process by which the beliefs of experts can be formally collected in a quantitative manner is structured expert elicitation. There is heterogeneity in the existing methodology used in health-care decision-making. A number of guidelines are available for structured expert elicitation; however, it is not clear if any of these are appropriate for health-care decision-making.

OBJECTIVES

The overall aim was to establish a protocol for structured expert elicitation to inform health-care decision-making. The objectives are to (1) provide clarity on methods for collecting and using experts' judgements, (2) consider when alternative methodology may be required in particular contexts, (3) establish preferred approaches for elicitation on a range of parameters, (4) determine which elicitation methods allow experts to express uncertainty and (5) determine the usefulness of the reference protocol developed.

METHODS

A mixed-methods approach was used: systemic review, targeted searches, experimental work and narrative synthesis. A review of the existing guidelines for structured expert elicitation was conducted. This identified the approaches used in existing guidelines (the 'choices') and determined if dominant approaches exist. Targeted review searches were conducted for selection of experts, level of elicitation, fitting and aggregation, assessing accuracy of judgements and heuristics and biases. To sift through the available choices, a set of principles that underpin the use of structured expert elicitation in health-care decision-making was defined using evidence generated from the targeted searches, quantities to elicit experimental evidence and consideration of constraints in health-care decision-making. These principles, including fitness for purpose and reflecting individual expert uncertainty, were applied to the set of choices to establish a reference protocol. An applied evaluation of the developed reference protocol was also undertaken.

RESULTS

For many elements of structured expert elicitation, there was a lack of consistency across the existing guidelines. In almost all choices, there was a lack of empirical evidence supporting recommendations, and in some circumstances the principles are unable to provide sufficient justification for discounting particular choices. It is possible to define reference methods for health technology assessment. These include a focus on gathering experts with substantive skills, eliciting observable quantities and individual elicitation of beliefs. Additional considerations are required for decision-makers outside health technology assessment, for example at a local level, or for early technologies. Access to experts may be limited and in some circumstances group discussion may be needed to generate a distribution.

LIMITATIONS

The major limitation of the work conducted here lies not in the methods employed in the current work but in the evidence available from the wider literature relating to how appropriate particular methodological choices are.

CONCLUSIONS

The reference protocol is flexible in many choices. This may be a useful characteristic, as it is possible to apply this reference protocol across different settings. Further applied studies, which use the choices specified in this reference protocol, are required.

FUNDING

This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 37. See the NIHR Journals Library website for further project information. This work was also funded by the Medical Research Council (reference MR/N028511/1).

Selection bias, investment decisions and treatment effect distributions

When making decisions regarding the investment and design for a Phase 3 programme in the development of a new drug, the results from preceding Phase 2 trials are an important source of information. However, only projects in which the Phase 2 results show promising treatment effects will typically be considered for a Phase 3 investment decision. This implies that, for those projects where Phase 3 is pursued, the underlying Phase 2 estimates are subject to selection bias. We will in this article investigate the nature of this selection bias based on a selection of distributions for the treatment effect. We illustrate some properties of Bayesian estimates, providing shrinkage of the Phase 2 estimate to counteract the selection bias. We further give some empirical guidance regarding the choice of prior distribution and comment on the consequences for decision-making in investment and planning for Phase 3 programmes.

Conditional assurance: the answer to the questions that should be asked within drug development

In this paper, we extend the use of assurance for a single study to explore how meeting a study's pre-defined success criteria could update our beliefs about the true treatment effect and impact the assurance of subsequent studies. This concept of conditional assurance, the assurance of a subsequent study conditional on success in an initial study, can be used assess the de-risking potential of the study requiring immediate investment, to ensure it provides value within the overall development plan. If the planned study does not discharge sufficient later phase risk, alternative designs and/or success criteria should be explored. By transparently laying out the different design options and the risks associated, this allows for decision makers to make quantitative investment choices based on their risk tolerance levels and potential return on investment. This paper lays out the derivation of conditional assurance, discusses how changing the design of a planned study will impact the conditional assurance of a future study, as well as presenting a simple illustrative example of how this methodology could be used to transparently compare development plans to aid decision making within an organisation.

A Review of Bayesian Perspectives on Sample Size Derivation for Confirmatory Trials

Sample size derivation is a crucial element of planning any confirmatory trial. The required sample size is typically derived based on constraints on the maximal acceptable Type I error rate and minimal desired power. Power depends on the unknown true effect and tends to be calculated either for the smallest relevant effect or a likely point alternative. The former might be problematic if the minimal relevant effect is close to the null, thus requiring an excessively large sample size, while the latter is dubious since it does not account for the a priori uncertainty about the likely alternative effect. A Bayesian perspective on sample size derivation for a frequentist trial can reconcile arguments about the relative a priori plausibility of alternative effects with ideas based on the relevance of effect sizes. Many suggestions as to how such "hybrid" approaches could be implemented in practice have been put forward. However, key quantities are often defined in subtly different ways in the literature. Starting from the traditional entirely frequentist approach to sample size derivation, we derive consistent definitions for the most commonly used hybrid quantities and highlight connections, before discussing and demonstrating their use in sample size derivation for clinical trials.

Bayesian design and analysis of external pilot trials for complex interventions

External pilot trials of complex interventions are used to help determine if and how a confirmatory trial should be undertaken, providing estimates of parameters such as recruitment, retention, and adherence rates. The decision to progress to the confirmatory trial is typically made by comparing these estimates to pre-specified thresholds known as progression criteria, although the statistical properties of such decision rules are rarely assessed. Such assessment is complicated by several methodological challenges, including the simultaneous evaluation of multiple endpoints, complex multi-level models, small sample sizes, and uncertainty in nuisance parameters. In response to these challenges, we describe a Bayesian approach to the design and analysis of external pilot trials. We show how progression decisions can be made by minimizing the expected value of a loss function, defined over the whole parameter space to allow for preferences and trade-offs between multiple parameters to be articulated and used in the decision-making process. The assessment of preferences is kept feasible by using a piecewise constant parametrization of the loss function, the parameters of which are chosen at the design stage to lead to desirable operating characteristics. We describe a flexible, yet computationally intensive, nested Monte Carlo algorithm for estimating operating characteristics. The method is used to revisit the design of an external pilot trial of a complex intervention designed to increase the physical activity of care home residents.

Optimising the trade-off between type I and II error rates in the Bayesian context

For any decision-making study, there are two sorts of errors that can be made, declaring a positive result when the truth is negative, and declaring a negative result when the truth is positive. Traditionally, the primary analysis of a study is a two-sided hypothesis test, the type I error rate will be set to 5% and the study is designed to give suitably low type II error - typically 10 or 20% - to detect a given effect size. These values are standard, arbitrary and, other than the choice between 10 and 20%, do not reflect the context of the study, such as the relative costs of making type I and II errors and the prior belief the drug will be placebo-like. Several authors have challenged this paradigm, typically for the scenario where the planned analysis is frequentist. When resource is limited, there will always be a trade-off between the type I and II error rates, and this article explores optimising this trade-off for a study with a planned Bayesian statistical analysis. This work provides a scientific basis for a discussion between stakeholders as to what type I and II error rates may be appropriate and some algebraic results for normally distributed data.

Prior Elicitation for Use in Clinical Trial Design and Analysis: A Literature Review

Bayesian inference is increasingly popular in clinical trial design and analysis. The subjective knowledge derived from an expert elicitation procedure may be useful to define a prior probability distribution when no or limited data is available. This work aims to investigate the state-of-the-art Bayesian prior elicitation methods with a focus on clinical trial research. A literature search on the Current Index to Statistics (CIS), PubMed, and Web of Science (WOS) databases, considering “prior elicitation” as a search string, was run on 1 November 2020. Summary statistics and trend of publications over time were reported. Finally, a Latent Dirichlet Allocation (LDA) model was developed to recognise latent topics in the pertinent papers retrieved. A total of 460 documents pertinent to the Bayesian prior elicitation were identified. Of these, 213 (45.4%) were published in the “Probability and Statistics” area. A total of 42 articles pertain to clinical trial and the majority of them (81%) reports parametric techniques as elicitation method. The last decade has seen an increased interest in prior elicitation and the gap between theory and application getting narrower and narrower. Given the promising flexibility of non-parametric approaches to the experts’ elicitation, more efforts are needed to ensure their diffusion also in applied settings.

Optimal designs for phase II/III drug development programs including methods for discounting of phase II results

Background

Go/no-go decisions after phase II and sample size chosen for phase III are usually based on phase II results (e.g., the treatment effect estimate of phase II). Due to the decision rule (only promising phase II results lead to phase III), treatment effect estimates from phase II that initiate a phase III trial commonly overestimate the true treatment effect. Underpowered phase III trials are the consequence. Optimistic findings may then not be reproduced, leading to the failure of potentially expensive drug development programs. For some disease areas these failure rates are described to be quite high: 62.5%.

Methods

We integrate the ideas of multiplicative and additive adjustment of treatment effect estimates after go decisions in a utility-based framework for optimizing drug development programs. The design of a phase II/III program, i.e., the “right amount of adjustment”, the allocation of the resources to phase II and III in terms of sample size, and the rule applied to decide whether to stop or to proceed with phase III influences its success considerably. Given specific drug development program characteristics (e.g., fixed and variable per patient costs for phase II and III, probable gain in case of market launch), optimal designs with respect to the maximal expected utility can be identified by the proposed Bayesian-frequentist approach. The method will be illustrated by application to practical examples characteristic for oncological studies.

Results

In general, our results show that the program set-ups with adjusted treatment effect estimate used for phase III planning are superior to the “naïve” program set-ups with respect to the maximal expected utility. Therefore, we recommend considering an adjusted phase II treatment effect estimate for the phase III sample size calculation. However, there is no one-fits-all design.

Conclusion

Individual drug development planning for a specific program is necessary to find the optimal design. The optimal choice of the design parameters for a specific drug development program at hand can be found by our user friendly R Shiny application and package (both assessable open-source via [1]).

Assurance for clinical trial design with normally distributed outcomes: Eliciting uncertainty about variances

The assurance method is growing in popularity in clinical trial planning. The method involves eliciting a prior distribution for the treatment effect, and then calculating the probability that a proposed trial will produce a "successful" outcome. For normally distributed observations, uncertainty about the variance of the normal distribution also needs to be accounted for, but there is little guidance in the literature on how to elicit a distribution for a variance parameter. We present a simple elicitation method, and illustrate how the elicited distribution is incorporated within an assurance calculation. We also consider multi-stage trials, where a decision to proceed with a larger trial will follow from the outcome of a smaller trial; we illustrate the role of the elicited distribution in assessing the information provided by a proposed smaller trial. Free software is available for implementing our methods.

A Bayesian approach in design and analysis of pediatric cancer clinical trials

It is well recognized that cancer drug development for children and adolescents has many challenges, from biological and societal to economic. Pediatric cancer consists of a diverse group of rare diseases, and the relatively small population of children with multiple, disparate tumor types across various age groups presents a significant challenge for drug development programs as compared to oncology drug development programs for adults. Due to the different types of cancers, limited opportunities exist for extrapolation of efficacy from adult cancer indications to children. Thus, innovative study designs including Bayesian statistical approaches should be considered. A Bayesian approach can be a flexible tool to formally leverage prior knowledge of adult or external controls in pediatric cancer trials. In this article, we provide in a case example of how Bayesian approaches can be used to design, monitor, and analyze pediatric trials. Particularly, Bayesian sequential monitoring can be useful to monitor pediatric trial results as data accumulate. In addition, designing a pediatric trial with both skeptical and enthusiastic priors with Bayesian sequential monitoring can be an efficient mechanism for early trial cessation for both efficacy and futility. The interpretation of efficacy using a Bayesian approach is based on posterior probability and is intuitive and interpretable for patients, parents and prescribers given limited data.

Elicitation of prior probability distributions for a proposed Bayesian randomized clinical trial of whole blood for trauma resuscitation

BACKGROUND

Whole blood trauma resuscitation is conceptually appealing and increasingly used but lacks evidence. A randomized controlled trial is needed but challenging to design. A Bayesian approach might be more efficient and more interpretable than a conventional frequentist design. We report the results on an elicitation meeting to create prior probability distributions to help develop such a trial.

METHODS

In‐person expert elicitation meeting, based on Sheffield Elicitation Framework methodology. We used an interactive graphical tool to elicit the quantities of interest (24‐hour mortality and certainty required). Two rounds were conducted, with an intervening discussion of deidentified responses. Individual responses were aggregated into probability distributions.

RESULTS

Fifteen experts participated. The pooled belief was that the median 24‐hour mortality of trauma patients with hemorrhagic shock treated with component therapy (the current standard of care) was 19% (95% credible interval [CrI], 6%‐45%), and the median 24‐hour mortality of those treated with whole blood, 16% (95% CrI, 5%‐39%). The pooled prior distribution for the relative risk had a median of 0.84 (95% CrI, 0.26‐3.1), indicating that the expert group had a 64% prior belief that whole blood decreases 24‐hour mortality compared to component therapy.

CONCLUSIONS

Experts had moderately strong beliefs that whole blood reduces the 24‐hour mortality of trauma patients with hemorrhagic shock. These data will assist with the design and planning of a Bayesian trial of whole blood resuscitation, which will help to answer a key question in contemporary transfusion practice.

What Do You Think? Using Expert Opinion to Improve Predictions of Response Propensity Under a Bayesian Framework

Responsive survey designs introduce protocol changes to survey operations based on accumulating paradata. Case-level predictions, including response propensity, can be used to tailor data collection features in pursuit of cost or quality goals. Unfortunately, predictions based only on partial data from the current round of data collection can be biased, leading to ineffective tailoring. Bayesian approaches can provide protection against this bias. Prior beliefs, which are generated from data external to the current survey implementation, contribute information that may be lacking from the partial current data. Those priors are then updated with the accumulating paradata. The elicitation of the prior beliefs, then, is an important characteristic of these approaches. While historical data for the same or a similar survey may be the most natural source for generating priors, eliciting prior beliefs from experienced survey managers may be a reasonable choice for new surveys, or when historical data are not available. Here, we fielded a questionnaire to survey managers, asking about expected attempt-level response rates for different subgroups of cases, and developed prior distributions for attempt-level response propensity model coefficients based on the mean and standard error of their responses. Then, using respondent data from a real survey, we compared the predictions of response propensity when the expert knowledge is incorporated into a prior to those based on a standard method that considers accumulating paradata only, as well as a method that incorporates historical survey data.

Defining consensus opinion to develop randomised controlled trials in rare diseases using Bayesian design: An example of a proposed trial of adalimumab versus pamidronate for children with CNO/CRMO

Introduction

Chronic nonbacterial osteomyelitis (CNO) is a rare autoinflammatory bone disorder primarily affecting children and adolescents. It can lead to chronic pain, bony deformities and fractures. The pathophysiology of CNO is incompletely understood. Scientific evidence suggests dysregulated expression of pro- and anti-inflammatory cytokines to be centrally involved. Currently, treatment is largely based on retrospective observational studies and expert opinion. Treatment usually includes nonsteroidal anti-inflammatory drugs and/or glucocorticoids, followed by a range of drugs in unresponsive cases. While randomised clinical trials are lacking, retrospective and prospective non-controlled studies suggest effectiveness of TNF inhibitors and bisphosphonates. The objective of the Bayesian consensus meeting was to quantify prior expert opinion.

Methods

Twelve international CNO experts were randomly chosen to be invited to a Bayesian prior elicitation meeting.

Results

Results showed that a typical new patient treated with pamidronate would have an 84% chance of improvement in their pain score relative to baseline at 26 weeks and an 83% chance on adalimumab. Experts thought there was a 50% chance that a new typical patient would record a pain score of 28mm (pamidronate) to 30mm (adalimumab) or better at 26 weeks. There was a modest trend in prior opinion to indicate an advantage of pamidronate vs adalimumab, with a 68% prior chance that pamidronate is superior to adalimumab by some margin. However, it is clear that there is considerable uncertainty about the precise relative merits of the two treatments.

Conclusions

The rarity of CNO leads to challenges in conducting randomised controlled trials with sufficient power to provide a definitive outcome. We address this using a Bayesian design, and here describe the process and outcome of the elicitation exercise to establish expert prior opinion. This opinion will be tested in the planned prospective CNO study. The process for establishing expert consensus opinion in CNO will be helpful for developing studies in other rare paediatric diseases.

Predictive probability of success using surrogate endpoints

The predictive probability of success of a future clinical trial is a key quantitative tool for decision-making in drug development. It is derived from prior knowledge and available evidence, and the latter typically comes from the accumulated data on the clinical endpoint of interest in previous clinical trials. However, a surrogate endpoint could be used as primary endpoint in early development and, usually, no or limited data are collected on the clinical endpoint of interest. We propose a general, reliable, and broadly applicable methodology to predict the success of a future trial from surrogate endpoints, in a way that makes the best use of all the available evidence. The predictions are based on an informative prior, called surrogate prior, derived from the results of past trials on one or several surrogate endpoints. If available, in a Bayesian framework, this prior could be combined with data from past trials on the clinical endpoint of interest. Two methods are proposed to address a potential discordance between the surrogate prior and the data on the clinical endpoint. We investigate the patterns of behavior of the predictions in a comprehensive simulation study, and we present an application to the development of a drug in Multiple Sclerosis. The proposed methodology is expected to support decision-making in many different situations, since the use of predictive markers is important to accelerate drug developments and to select promising drug candidates, better and earlier.

Practical experiences of adopting assurance as a quantitative framework to support decision making in drug development

All clinical trials are designed for success of their primary objectives. Hence, evaluating the probability of success (PoS) should be a key focus at the design stage both to support funding approval from sponsor governance boards and to inform trial design itself. Use of assurance-that is, expected success probability averaged over a prior probability distribution for the treatment effect-to quantify PoS of a planned study has grown across the industry in recent years, and has now become routine within the authors' company. In this paper, we illustrate some of the benefits of systematically adopting assurance as a quantitative framework to support decision making in drug development through several case-studies where evaluation of assurance has proved impactful in terms of trial design and in supporting governance-board reviews of project proposals. In addition, we describe specific features of how the assurance framework has been implemented within our company, highlighting the critical role that prior elicitation plays in this process, and illustrating how the overall assurance calculation may be decomposed into a sequence of conditional PoS estimates which can provide greater insight into how and when different development options are able to discharge risk.