Predicting subgroup treatment effects for a new study: Motivations, results and learnings from running a data challenge in a pharmaceutical corporation

We present the motivation, experience, and learnings from a data challenge conducted at a large pharmaceutical corporation on the topic of subgroup identification. The data challenge aimed at exploring approaches to subgroup identification for future clinical trials. To mimic a realistic setting, participants had access to 4 Phase III clinical trials to derive a subgroup and predict its treatment effect on a future study not accessible to challenge participants. A total of 30 teams registered for the challenge with around 100 participants, primarily from Biostatistics organization. We outline the motivation for running the challenge, the challenge rules, and logistics. Finally, we present the results of the challenge, the participant feedback as well as the learnings. We also present our view on the implications of the results on exploratory analyses related to treatment effect heterogeneity.

Interim decision making in seamless trial designs: An application in an adaptive dose-finding study in a rare kidney disease

Adaptive seamless trial designs, combining the learning and confirming cycles of drug development in a single trial, have gained popularity in recent years. Adaptations may include dose selection, sample size re-estimation and enrichment of the study population. Despite methodological advances and recognition of the potential efficiency gains such designs offer, their implementation, including how to enable efficient decision making on the adaptations in interim analyzes, remains a key challenge in their adoption. This manuscript uses a case study of an adaptive seamless proof-of-concept (Phase 2a)/dose-finding (Phase 2b) to showcase potential adaptive features that can be implemented in trial designs at earlier development stages and the role of simulations in assessing the design operating characteristics and specifying the decision rules for the adaptations. It further outlines the elements needed to support successful interim analysis decision making on the adaptations while safeguarding study integrity, including the role of different stakeholders, interactive simulation-based tools to facilitate decision making and operational aspects requiring preplanning. The benefits of the adaptive Phase 2a/2b design chosen compared to following the traditional two separate studies (2a and 2b) paradigm are discussed. With careful planning and appreciation of their complexity and components needed for their implementation, seamless adaptive designs have the potential to yield significant savings both in terms of time and resources.

Comparing algorithms for characterizing treatment effect heterogeneity in randomized trials

The identification and estimation of heterogeneous treatment effects in biomedical clinical trials are challenging, because trials are typically planned to assess the treatment effect in the overall trial population. Nevertheless, the identification of how the treatment effect may vary across subgroups is of major importance for drug development. In this work, we review some existing simulation work and perform a simulation study to evaluate recent methods for identifying and estimating the heterogeneous treatments effects using various metrics and scenarios relevant for drug development. Our focus is not only on a comparison of the methods in general, but on how well these methods perform in simulation scenarios that reflect real clinical trials. We provide the R package benchtm that can be used to simulate synthetic biomarker distributions based on real clinical trial data and to create interpretable scenarios to benchmark methods for identification and estimation of treatment effect heterogeneity.

Bayesian inference for a principal stratum estimand on recurrent events truncated by death

Recurrent events are often important endpoints in randomized clinical trials. For example, the number of recurrent disease-related hospitalizations may be considered as a clinically meaningful endpoint in cardiovascular studies. In some settings, the recurrent event process may be terminated by an event such as death, which makes it more challenging to define and estimate a causal treatment effect on recurrent event endpoints. In this paper, we focus on the principal stratum estimand, where the treatment effect of interest on recurrent events is defined among subjects who would be alive regardless of the assigned treatment. For the estimation of the principal stratum effect in randomized clinical trials, we propose a Bayesian approach based on a joint model of the recurrent event and death processes with a frailty term accounting for within-subject correlation. We also present Bayesian posterior predictive check procedures for assessing the model fit. The proposed approaches are demonstrated in the randomized Phase III chronic heart failure trial PARAGON-HF (NCT01920711).

Sensitivity analyses for the principal ignorability assumption using multiple imputation

In the context of clinical trials, there is interest in the treatment effect for subpopulations of patients defined by intercurrent events, namely disease-related events occurring after treatment initiation that affect either the interpretation or the existence of endpoints. With the principal stratum strategy, the ICH E9(R1) guideline introduces a formal framework in drug development for defining treatment effects in such subpopulations. Statistical estimation of the treatment effect can be performed based on the principal ignorability assumption using multiple imputation approaches. Principal ignorability is a conditional independence assumption that cannot be directly verified; therefore, it is crucial to evaluate the robustness of results to deviations from this assumption. As a sensitivity analysis, we propose a joint model that multiply imputes the principal stratum membership and the outcome variable while allowing different levels of violation of the principal ignorability assumption. We illustrate with a simulation study that the joint imputation model-based approaches are superior to naive subpopulation analyses. Motivated by an oncology clinical trial, we implement the sensitivity analysis on a time-to-event outcome to assess the treatment effect in the subpopulation of patients who discontinued due to adverse events using a synthetic dataset. Finally, we explore the potential usage and provide interpretation of such analyses in clinical settings, as well as possible extension of such models in more general cases.

Improving the assessment of the probability of success in late stage drug development

There are several steps to confirming the safety and efficacy of a new medicine. A sequence of trials, each with its own objectives, is usually required. Quantitative risk metrics can be useful for informing decisions about whether a medicine should transition from one stage of development to the next. To obtain an estimate of the probability of regulatory approval, pharmaceutical companies may start with industry-wide success rates and then apply to these subjective adjustments to reflect program-specific information. However, this approach lacks transparency and fails to make full use of data from previous clinical trials. We describe a quantitative Bayesian approach for calculating the probability of success (PoS) at the end of phase II which incorporates internal clinical data from one or more phase IIb studies, industry-wide success rates, and expert opinion or external data if needed. Using an example, we illustrate how PoS can be calculated accounting for differences between the phase II data and future phase III trials, and discuss how the methods can be extended to accommodate accelerated drug development pathways.

A New Comprehensive Approach to Assess the Probability of Success of Development Programs Before Pivotal Trials

The point at which clinical development programs transition from early phase to pivotal trials is a critical milestone. Substantial uncertainty about the outcome of pivotal trials may remain even after seeing positive early phase data, and companies may need to make difficult prioritization decisions for their portfolio. The probability of success (PoS) of a program, a single number expressed as a percentage reflecting the multitude of risks that may influence the final program outcome, is a key decision-making tool. Despite its importance, companies often rely on crude industry benchmarks that may be "adjusted" by experts based on undocumented criteria and which are typically misaligned with the definition of success used to drive commercial forecasts, leading to overly optimistic expected net present value calculations. We developed a new framework to assess the PoS of a program before pivotal trials begin. Our definition of success encompasses the successful outcome of pivotal trials, regulatory approval and meeting the requirements for market access as outlined in the target product profile. The proposed approach is organized in four steps and uses an innovative Bayesian approach to synthesize all relevant evidence. The new PoS framework is systematic and transparent. It will help organizations to make more informed decisions. In this paper, we outline the rationale and elaborate on the structure of the proposed framework, provide examples, and discuss the benefits and challenges associated with its adoption.

A framework to guide dose & regimen strategy for clinical drug development

Optimizing new drug therapies remains a challenge for clinical development, despite the use of ever more sophisticated quantitative methodologies. Although conceptually simple, the idea of finding the right treatment at the right dose for the right patient to ensure an appropriate balance of risks and benefits is challenging and requires a multidisciplinary approach. In this paper, we present a framework developed as a tool for organizing knowledge and facilitating collaboration in development teams.

Connecting Instrumental Variable methods for causal inference to the Estimand Framework

Causal inference methods are gaining increasing prominence in pharmaceutical drug development in light of the recently published addendum on estimands and sensitivity analysis in clinical trials to the E9 guideline of the International Council for Harmonisation. The E9 addendum emphasises the need to account for post-randomization or 'intercurrent' events that can potentially influence the interpretation of a treatment effect estimate at a trial's conclusion. Instrumental Variables (IV) methods have been used extensively in economics, epidemiology, and academic clinical studies for 'causal inference,' but less so in the pharmaceutical industry setting until now. In this tutorial article we review the basic tools for causal inference, including graphical diagrams and potential outcomes, as well as several conceptual frameworks that an IV analysis can sit within. We discuss in detail how to map these approaches to the Treatment Policy, Principal Stratum and Hypothetical 'estimand strategies' introduced in the E9 addendum, and provide details of their implementation using standard regression models. Specific attention is given to discussing the assumptions each estimation strategy relies on in order to be consistent, the extent to which they can be empirically tested and sensitivity analyses in which specific assumptions can be relaxed. We finish by applying the methods described to simulated data closely matching two recent pharmaceutical trials to further motivate and clarify the ideas.

Identifying treatment effect heterogeneity in dose-finding trials using Bayesian hierarchical models

An important task in drug development is to identify patients, which respond better or worse to an experimental treatment. Identifying predictive covariates, which influence the treatment effect and can be used to define subgroups of patients, is a key aspect of this task. Analyses of treatment effect heterogeneity are however known to be challenging, since the number of possible covariates or subgroups is often large, while samples sizes in earlier phases of drug development are often small. In addition, distinguishing predictive covariates from prognostic covariates, which influence the response independent of the given treatment, can often be difficult. While many approaches for these types of problems have been proposed, most of them focus on the two-arm clinical trial setting, where patients are given either the treatment or a control. In this article we consider parallel groups dose-finding trials, in which patients are administered different doses of the same treatment. To investigate treatment effect heterogeneity in this setting we propose a Bayesian hierarchical dose-response model with covariate effects on dose-response parameters. We make use of shrinkage priors to prevent overfitting, which can easily occur, when the number of considered covariates is large and sample sizes are small. We compare several such priors in simulations and also investigate dependent modeling of prognostic and predictive effects to better distinguish these two types of effects. We illustrate the use of our proposed approach using a Phase II dose-finding trial and show how it can be used to identify predictive covariates and subgroups of patients with increased treatment effects.

Principal stratum strategy: Potential role in drug development

A randomized trial allows estimation of the causal effect of an intervention compared to a control in the overall population and in subpopulations defined by baseline characteristics. Often, however, clinical questions also arise regarding the treatment effect in subpopulations of patients, which would experience clinical or disease related events post-randomization. Events that occur after treatment initiation and potentially affect the interpretation or the existence of the measurements are called intercurrent events in the ICH E9(R1) guideline. If the intercurrent event is a consequence of treatment, randomization alone is no longer sufficient to meaningfully estimate the treatment effect. Analyses comparing the subgroups of patients without the intercurrent events for intervention and control will not estimate a causal effect. This is well known, but post-hoc analyses of this kind are commonly performed in drug development. An alternative approach is the principal stratum strategy, which classifies subjects according to their potential occurrence of an intercurrent event on both study arms. We illustrate with examples that questions formulated through principal strata occur naturally in drug development and argue that approaching these questions with the ICH E9(R1) estimand framework has the potential to lead to more transparent assumptions as well as more adequate analyses and conclusions. In addition, we provide an overview of assumptions required for estimation of effects in principal strata. Most of these assumptions are unverifiable and should hence be based on solid scientific understanding. Sensitivity analyses are needed to assess robustness of conclusions.

Exposure-response modelling approaches for determining optimal dosing rules in children

Within paediatric populations, there may be distinct age groups characterised by different exposure-response relationships. Several regulatory guidance documents have suggested general age groupings. However, it is not clear whether these categorisations will be suitable for all new medicines and in all disease areas. We consider two model-based approaches to quantify how exposure-response model parameters vary over a continuum of ages: Bayesian penalised B-splines and model-based recursive partitioning. We propose an approach for deriving an optimal dosing rule given an estimate of how exposure-response model parameters vary with age. Methods are initially developed for a linear exposure-response model. We perform a simulation study to systematically evaluate how well the various approaches estimate linear exposure-response model parameters and the accuracy of recommended dosing rules. Simulation scenarios are motivated by an application to epilepsy drug development. Results suggest that both bootstrapped model-based recursive partitioning and Bayesian penalised B-splines can estimate underlying changes in linear exposure-response model parameters as well as (and in many scenarios, better than) a comparator linear model adjusting for a categorical age covariate with levels following International Conference on Harmonisation E11 groupings. Furthermore, the Bayesian penalised B-splines approach consistently estimates the intercept and slope more accurately than the bootstrapped model-based recursive partitioning. Finally, approaches are extended to estimate Emax exposure-response models and are illustrated with an example motivated by an in vitro study of cyclosporine.

Testing multiple dose combinations in clinical trials

Drug combination trials are often motivated by the fact that individual drugs target the same disease but via different routes. A combination of such drugs may then have an overall better effect than the individual treatments which has to be verified by clinical trials. Several statistical methods have been explored that discuss the problem of comparing a fixed-dose combination therapy to each of its components. But an extension of these approaches to multiple dose combinations can be difficult and is not yet fully investigated. In this paper, we propose two approaches by which one can provide confirmatory assurance with familywise error rate control, that the combination of two drugs at differing doses is more effective than either component doses alone. These approaches involve multiple comparisons in multilevel factorial designs where the type 1 error can be controlled first, by bootstrapping tests, and second, by considering the least favorable null configurations for a family of union intersection tests. The main advantage of the new approaches is that their implementation is simple. The implementation of these new approaches is illustrated with a real data example from a blood pressure reduction trial. Extensive simulations are also conducted to evaluate the new approaches and benchmark them with existing ones. We also present an illustration of the relationship between the different approaches. We observed that the bootstrap provided some power advantages over the other approaches with the disadvantage that there may be some error rate inflation for small sample sizes.

A multiple comparison procedure for dose-finding trials with subpopulations

Identifying subgroups of patients with an enhanced response to a new treatment has become an area of increased interest in the last few years. When there is knowledge about possible subpopulations with an enhanced treatment effect before the start of a trial it might be beneficial to set up a testing strategy, which tests for a significant treatment effect not only in the full population, but also in these prespecified subpopulations. In this paper, we present a parametric multiple testing approach for tests in multiple populations for dose-finding trials. Our approach is based on the MCP-Mod methodology, which uses multiple comparison procedures (MCPs) to test for a dose-response signal, while considering multiple possible candidate dose-response shapes. Our proposed methods allow for heteroscedastic error variances between populations and control the family-wise error rate over tests in multiple populations and for multiple candidate models. We show in simulations that the proposed multipopulation testing approaches can increase the power to detect a significant dose-response signal over the standard single-population MCP-Mod, when the specified subpopulation has an enhanced treatment effect.

A proposal for a new PhD level curriculum on quantitative methods for drug development

This paper provides an overview of "Improving Design, Evaluation and Analysis of early drug development Studies" (IDEAS), a European Commission-funded network bringing together leading academic institutions and small- to large-sized pharmaceutical companies to train a cohort of graduate-level medical statisticians. The network is composed of a diverse mix of public and private sector partners spread across Europe, which will host 14 early-stage researchers for 36 months. IDEAS training activities are composed of a well-rounded mixture of specialist methodological components and generic transferable skills. Particular attention is paid to fostering collaborations between researchers and supervisors, which span academia and the private sector. Within this paper, we review existing medical statistics programmes (MSc and PhD) and highlight the training they provide on skills relevant to drug development. Motivated by this review and our experiences with the IDEAS project, we propose a concept for a joint, harmonised European PhD programme to train statisticians in quantitative methods for drug development.

A Longitudinal Item Response Theory Model to Characterize Cognition Over Time in Elderly Subjects

For drug development in neurodegenerative diseases such as Alzheimer's disease, it is important to understand which cognitive domains carry the most information on the earliest signs of cognitive decline, and which subject characteristics are associated with a faster decline. A longitudinal Item Response Theory (IRT) model was developed for the Basel Study on the Elderly, in which the Consortium to Establish a Registry for Alzheimer's Disease - Neuropsychological Assessment Battery (with additions) and the California Verbal Learning Test were measured on 1,750 elderly subjects for up to 13.9 years. The model jointly captured the multifaceted nature of cognition and its longitudinal trajectory. The word list learning and delayed recall tasks carried the most information. Greater age at baseline, fewer years of education, and positive APOEɛ4 carrier status were associated with a faster cognitive decline. Longitudinal IRT modeling is a powerful approach for progressive diseases with multifaceted endpoints.

Advanced Methods for Dose and Regimen Finding During Drug Development: Summary of the EMA/EFPIA Workshop on Dose Finding (London 4-5 December 2014)

Inadequate dose selection for confirmatory trials is currently still one of the most challenging issues in drug development, as illustrated by high rates of late‐stage attritions in clinical development and postmarketing commitments required by regulatory institutions. In an effort to shift the current paradigm in dose and regimen selection and highlight the availability and usefulness of well‐established and regulatory‐acceptable methods, the European Medicines Agency (EMA) in collaboration with the European Federation of Pharmaceutical Industries Association (EFPIA) hosted a multistakeholder workshop on dose finding (London 4–5 December 2014). Some methodologies that could constitute a toolkit for drug developers and regulators were presented. These methods are described in the present report: they include five advanced methods for data analysis (empirical regression models, pharmacometrics models, quantitative systems pharmacology models, MCP‐Mod, and model averaging) and three methods for study design optimization (Fisher information matrix (FIM)‐based methods, clinical trial simulations, and adaptive studies). Pairwise comparisons were also discussed during the workshop; however, mostly for historical reasons. This paper discusses the added value and limitations of these methods as well as challenges for their implementation. Some applications in different therapeutic areas are also summarized, in line with the discussions at the workshop. There was agreement at the workshop on the fact that selection of dose for phase III is an estimation problem and should not be addressed via hypothesis testing. Dose selection for phase III trials should be informed by well‐designed dose‐finding studies; however, the specific choice of method(s) will depend on several aspects and it is not possible to recommend a generalized decision tree. There are many valuable methods available, the methods are not mutually exclusive, and they should be used in conjunction to ensure a scientifically rigorous understanding of the dosing rationale.

Phase II dose-response trials: A simulation study to compare analysis method performance under design considerations

Phase II trials are intended to provide information about the dose-response relationship and to support the choice of doses for a pivotal phase III trial. Recently, new analysis methods have been proposed to address these objectives, and guidance is needed to select the most appropriate analysis method in specific situations. We set up a simulation study to evaluate multiple performance measures of one traditional and three more recent dose-finding approaches under four design options and illustrate the investigated analysis methods with an example from clinical practice. Our results reveal no general recommendation for a particular analysis method across all design options and performance measures. However, we also demonstrate that the new analysis methods are worth the effort compared to the traditional ANOVA-based approach.

Model selection versus model averaging in dose finding studies

A key objective of Phase II dose finding studies in clinical drug development is to adequately characterize the dose response relationship of a new drug. An important decision is then on the choice of a suitable dose response function to support dose selection for the subsequent Phase III studies. In this paper, we compare different approaches for model selection and model averaging using mathematical properties as well as simulations. We review and illustrate asymptotic properties of model selection criteria and investigate their behavior when changing the sample size but keeping the effect size constant. In a simulation study, we investigate how the various approaches perform in realistically chosen settings. Finally, the different methods are illustrated with a recently conducted Phase II dose finding study in patients with chronic obstructive pulmonary disease. Copyright © 2016 John Wiley & Sons, Ltd.

Multiple comparison procedure and modeling: a versatile tool for evaluating dose-response relationships in veterinary pharmacology - a case study with furosemide

Congestive heart failure (CHF) is a leading cause of mortality with an increasing prevalence in human and canine populations. While furosemide is a loop diuretic prescribed for the majority of CHF patients to reduce fluid retention, it also activates the renin-angiotensin aldosterone system (RAAS) which further contributes to the accelerated progression of heart failure. Our objective was to quantify the effect of furosemide on diuresis, renin activity (RA), and aldosterone (AL) in dogs, using a combined multiple comparisons and model-based approach (MCP-Mod). Twenty-four healthy beagle dogs were allocated to four treatment groups (saline vs. furosemide 1, 2, and 4 mg/kg i.m., q12 h for 5 days). Data from RA and AL values at furosemide trough concentrations, as well as 24-h Diuresis, were analyzed using the MCP-Mod procedure. A combination of E_max models adequately described the dose-response relationships of furosemide for the various endpoints. The dose-response curves of RA and AL were found to be well in agreement, with an apparent shallower slope compared with 24-h Diuresis. The research presented herein constitutes the first application of MCP-Mod in Veterinary Medicine. Our data show that furosemide produces a submaximal effect on diuresis at doses lower than those identified to activate the circulating RAAS.

Viewpoint: model selection uncertainty, pre-specification, and model averaging

Scientific progress in all empirical sciences relies on selecting models and performing inferences from selected models. Standard statistical properties (e.g., repeated sampling coverage probability of confidence intervals) cannot be guaranteed after a model selection. This viewpoint reviews this dilemma, puts the role that pre-specification can play into perspective and illustrates model averaging as a way to relax the problem of model selection uncertainty.

Practical considerations for using functional uniform prior distributions for dose-response estimation in clinical trials

Estimating nonlinear dose-response relationships in the context of pharmaceutical clinical trials is often a challenging problem. The data in these trials are typically variable and sparse, making this a hard inference problem, despite sometimes seemingly large sample sizes. Maximum likelihood estimates often fail to exist in these situations, while for Bayesian methods, prior selection becomes a delicate issue when no carefully elicited prior is available, as the posterior distribution will often be sensitive to the priors chosen. This article provides guidance on the usage of functional uniform prior distributions in these situations. The essential idea of functional uniform priors is to employ a distribution that weights the functional shapes of the nonlinear regression function equally. By doing so one obtains a distribution that exhaustively and uniformly covers the underlying potential shapes of the nonlinear function. On the parameter scale these priors will often result in quite nonuniform prior distributions. This paper gives hints on how to implement these priors in practice and illustrates them in realistic trial examples in the context of Phase II dose-response trials as well as Phase I first-in-human studies.

Unimodal regression using Bernstein-Schoenberg splines and penalties

Research in the field of nonparametric shape constrained regression has been intensive. However, only few publications explicitly deal with unimodality although there is need for such methods in applications, for example, in dose-response analysis. In this article, we propose unimodal spline regression methods that make use of Bernstein-Schoenberg splines and their shape preservation property. To achieve unimodal and smooth solutions we use penalized splines, and extend the penalized spline approach toward penalizing against general parametric functions, instead of using just difference penalties. For tuning parameter selection under a unimodality constraint a restricted maximum likelihood and an alternative Bayesian approach for unimodal regression are developed. We compare the proposed methodologies to other common approaches in a simulation study and apply it to a dose-response data set. All results suggest that the unimodality constraint or the combination of unimodality and a penalty can substantially improve estimation of the functional relationship.

Functional uniform priors for nonlinear modeling

This article considers the topic of finding prior distributions when a major component of the statistical model depends on a nonlinear function. Using results on how to construct uniform distributions in general metric spaces, we propose a prior distribution that is uniform in the space of functional shapes of the underlying nonlinear function and then back-transform to obtain a prior distribution for the original model parameters. The primary application considered in this article is nonlinear regression, but the idea might be of interest beyond this case. For nonlinear regression the so constructed priors have the advantage that they are parametrization invariant and do not violate the likelihood principle, as opposed to uniform distributions on the parameters or the Jeffrey's prior, respectively. The utility of the proposed priors is demonstrated in the context of design and analysis of nonlinear regression modeling in clinical dose-finding trials, through a real data example and simulation.

Stochastically ordered multiple regression

In various application areas, prior information is available about the direction of the effects of multiple predictors on the conditional response distribution. For example, in epidemiology studies of potentially adverse exposures and continuous health responses, one can typically assume a priori that increasing the level of an exposure does not lead to an improvement in the health response. Such an assumption can be formalized through a stochastic ordering assumption in each of the exposures, leading to a potentially large improvement in efficiency in nonparametric modeling of the conditional response distribution. This article proposes a Bayesian nonparametric approach to this problem based on characterizing the conditional response density as a Gaussian mixture, with the locations of the Gaussian means varying flexibly with predictors subject to minimal constraints to ensure stochastic ordering. Theoretical properties are considered and Markov chain Monte Carlo methods are developed for posterior computation. The methods are illustrated using simulation examples and a reproductive epidemiology application.