Reflecting on Andy Grieve's influence and innovation: A personal perspective

Throughout his career, Andy Grieve has developed and implemented many novel methods and has been involved in trial design and analysis for many trials that have broken new ground in the statistics field. His record of innovation is clear, but it is the way that he also applies these innovations in practice, reaches pragmatic solutions to problems and then shares and disseminates those findings that mark him out as a true leader in the statistical field. In this short article, I will discuss my own views of Andy's innovation, pragmatism and influence and how it has left its mark in my own career.

Comparing Go/No-Go Decision-Making Properties Between Single Arm Phase II Trial Designs in Oncology

INTRODUCTION

Simon's design has been widely used in oncology to conduct single arm phase II trials and to make Go/No-Go development decision. Other authors have proposed designs with decision-making frameworks that include a third, "Consider" outcome. For results in the Consider zone, a final Go/No-Go development decision must still be made; however it is typically a subjective decision based on the totality of data and the development landscape. Under this framework, the probability of continuing development when the candidate therapy is truly ineffective or the probability of stopping development when the candidate therapy is truly effective is undefined.

METHODS

We use a motivating example to compare end of trial decision-making between Simon's two-stage approach and a Multilevel outcome approach. We present the minimum and maximum development decision error probabilities by varying whether candidates that end in the Consider zone would ultimately continue with development or not.

RESULTS

The Multilevel approach typically requires fewer patients, but the risk of making an incorrect drug development decision is inflated above the statistically defined Type I and Type II error rates. Compared to a Type I error rate of 20%, the Multilevel trial's maximum probability of moving forward with an ineffective therapy is 22%, 27%, and 36% for Consider zone sizes of 10%, 20%, and 30%, respectively.

CONCLUSION

The Multilevel approach provides flexibility in interpreting moderate efficacy results. However, the flexibility is accomplished with a lower sample size and corresponding uncertainty in the trial outcome that increases the risk of incorrect drug development decisions.

A New Comprehensive Approach to Assess the Probability of Success of Development Programs Before Pivotal Trials

The point at which clinical development programs transition from early phase to pivotal trials is a critical milestone. Substantial uncertainty about the outcome of pivotal trials may remain even after seeing positive early phase data, and companies may need to make difficult prioritization decisions for their portfolio. The probability of success (PoS) of a program, a single number expressed as a percentage reflecting the multitude of risks that may influence the final program outcome, is a key decision-making tool. Despite its importance, companies often rely on crude industry benchmarks that may be "adjusted" by experts based on undocumented criteria and which are typically misaligned with the definition of success used to drive commercial forecasts, leading to overly optimistic expected net present value calculations. We developed a new framework to assess the PoS of a program before pivotal trials begin. Our definition of success encompasses the successful outcome of pivotal trials, regulatory approval and meeting the requirements for market access as outlined in the target product profile. The proposed approach is organized in four steps and uses an innovative Bayesian approach to synthesize all relevant evidence. The new PoS framework is systematic and transparent. It will help organizations to make more informed decisions. In this paper, we outline the rationale and elaborate on the structure of the proposed framework, provide examples, and discuss the benefits and challenges associated with its adoption.

Nerve guidance conduit development for primary treatment of peripheral nerve transection injuries: A commercial perspective

Commercial nerve guidance conduits (NGCs) for repair of peripheral nerve discontinuities are of little use in gaps larger than 30 mm, and for smaller gaps they often fail to compete with the autografts that they are designed to replace. While recent research to develop new technologies for use in NGCs has produced many advanced designs with seemingly positive functional outcomes in animal models, these advances have not been translated into viable clinical products. While there have been many detailed reviews of the technologies available for creating NGCs, none of these have focussed on the requirements of the commercialisation process which are vital to ensure the translation of a technology from bench to clinic. Consideration of the factors essential for commercial viability, including regulatory clearance, reimbursement processes, manufacturability and scale up, and quality management early in the design process is vital in giving new technologies the best chance at achieving real-world impact. Here we have attempted to summarise the major components to consider during the development of emerging NGC technologies as a guide for those looking to develop new technology in this domain. We also examine a selection of the latest academic developments from the viewpoint of clinical translation, and discuss areas where we believe further work would be most likely to bring new NGC technologies to the clinic. STATEMENT OF SIGNIFICANCE: NGCs for peripheral nerve repairs represent an adaptable foundation with potential to incorporate modifications to improve nerve regeneration outcomes. In this review we outline the regulatory processes that functionally distinct NGCs may need to address and explore new modifications and the complications that may need to be addressed during the translation process from bench to clinic.

Improved Decision-Making Confidence Using Item-Based Pharmacometric Model: Illustration with a Phase II Placebo-Controlled Trial

This study aimed to illustrate how a new methodology to assess clinical trial outcome measures using a longitudinal item response theory–based model (IRM) could serve as an alternative to mixed model repeated measures (MMRM). Data from the EXACT (Exacerbation of chronic pulmonary disease tool) which is used to capture frequency, severity, and duration of exacerbations in COPD were analyzed using an IRM. The IRM included a graded response model characterizing item parameters and functions describing symptom-time course. Total scores were simulated (month 12) using uncertainty in parameter estimates. The 50th (2.5th, 97.5th) percentiles of the resulting simulated differences in average total score (drug minus placebo) represented the estimated drug effect (95%CI), which was compared with published MMRM results. Furthermore, differences in sample size, sensitivity, specificity, and type I and II errors between approaches were explored. Patients received either oral danirixin 75 mg twice daily (n = 45) or placebo (n = 48) on top of standard of care over 52 weeks. A step function best described the COPD symptoms-time course in both trial arms. The IRM improved precision of the estimated drug effect compared to MMRM, resulting in a sample size of 2.5 times larger for the MMRM analysis to achieve the IRM precision. The IRM showed a higher probability of a positive predictive value (34%) than MMRM (22%). An item model–based analysis data gave more precise estimates of drug effect than MMRM analysis for the same endpoint in this one case study.

Friction and Decision Rules in Portfolio Decision Analysis

In portfolio decision analysis, features comprise the objectives, alternatives, physics, and information that define a decision context. By modeling features, decision analysts forecast the expected utilities of the alternatives. A model is complete if it contains all the features. A model is well-calibrated if it correctly predicts the probability distributions of each alternative’s utility, whereas ill-calibrated models, like those that suffer the optimizer’s curse, do not. Friction identifies qualities of a situation that prevent decision analysts from creating complete, well-calibrated models. When friction is significant, can maximizing expected utility be a suboptimal decision rule? Is satisfying decision theory’s axioms a necessary or sufficient condition for good decision making? Can rules that violate the axioms outperform rules that satisfy them? A simulation study of how unbiased, imprecise forecasts of payoffs affect project selection finds that, for the example tested, the answers are yes, no, and yes, which suggests that further studies of friction may be worthwhile. Discussions of friction bookend the study, starting the paper by defining friction and concluding by presenting three frameworks, each one from a different field of study, that provide mathematical tools for studying friction.

Go/No-Go Decision Model for Owners Using Exhaustive CHAID and QUEST Decision Tree Algorithms

Go/no-go execution decisions are one of the most important strategic decisions for owners during the early stages of construction projects. Restructuring the process of decision-making during these early stages may have sustainable results in the long run. The purpose of this paper is to establish proper go/no-go decision-tree models for owners. The decision-tree models were developed using Exhaustive Chi-square Automatic Interaction Detector (Exhaustive CHAID) and Quick, Unbiased, Efficient Statistical Tree (QUEST) algorithms. Twenty-three go/no-go key factors were collected through an extensive literature review. These factors were divided into four main risk categories: organizational, project/technical, legal, and financial/economic. In a questionnaire distributed among the construction professionals, the go/no-go variables were asked to be ranked according to their perceived significance. Split-sample validation was applied for testing and measuring the accuracy of the Exhaustive CHAID and QUEST models. Moreover, Spearman’s rank correlation and analysis of variance (ANOVA) tests were employed to identify the statistical features of the 100 responses received. The result of this study benchmarks the current assessment models and develops a simple and user-friendly decision model for owners. The model is expected to evaluate anticipated risk factors in the project and reduce the level of uncertainty. The Exhaustive CHAID and QUEST models are validated by a case study. This paper contributes to the current body of knowledge by identifying the factors that have the biggest effect on an owner’s decision and introducing Exhaustive CHAID and QUEST decision-tree models for go/no-go decisions for the first time, to the best of the authors’ knowledge. From the “sustainability” viewpoint, this study is significant since the decisions of the owner, based on a rigorous model, will yield sustainable and efficient projects.

Blood-Brain Barrier Disruption in Neuro-Oncology: Strategies, Failures, and Challenges to Overcome

The blood-brain barrier (BBB) presents a formidable challenge in the development of effective therapeutics in neuro-oncology. This has fueled several decades of efforts to develop strategies for disrupting the BBB, but progress has not been satisfactory. As such, numerous drug- and device-based methods are currently being investigated in humans. Through a focused assessment of completed, active, and pending clinical trials, our first aim in this review is to outline the scientific foundation, successes, and limitations of the BBBD strategies developed to date. Among 35 registered trials relevant to BBBD in neuro-oncology in the ClinicalTrials.gov database, mannitol was the most common drug-based method, followed by RMP-7 and regadenoson. MR-guided focused ultrasound was the most common device-based method, followed by MR-guided laser ablation, ultrasound, and transcranial magnetic stimulation. While most early-phase studies focusing on safety and tolerability have met stated objectives, advanced-phase studies focusing on survival differences and objective tumor response have been limited by heterogeneous populations and tumors, along with a lack of control arms. Based on shared challenges among all methods, our second objective is to discuss strategies for confirmation of BBBD, choice of systemic agent and drug design, alignment of BBBD method with real-world clinical workflow, and consideration of inadvertent toxicity associated with disrupting an evolutionarily-refined barrier. Finally, we conclude with a strategic proposal to approach future studies assessing BBBD.

Beyond p-values: A phase II dual-criterion design with statistical significance and clinical relevance

Background

Well-designed phase II trials must have acceptable error rates relative to a pre-specified success criterion, usually a statistically significant p-value. Such standard designs may not always suffice from a clinical perspective because clinical relevance may call for more. For example, proof-of-concept in phase II often requires not only statistical significance but also a sufficiently large effect estimate.

Purpose

We propose dual-criterion designs to complement statistical significance with clinical relevance, discuss their methodology, and illustrate their implementation in phase II.

Methods

Clinical relevance requires the effect estimate to pass a clinically motivated threshold (the decision value (DV)). In contrast to standard designs, the required effect estimate is an explicit design input, whereas study power is implicit. The sample size for a dual-criterion design needs careful considerations of the study’s operating characteristics (type I error, power).

Results

Dual-criterion designs are discussed for a randomized controlled and a single-arm phase II trial, including decision criteria, sample size calculations, decisions under various data scenarios, and operating characteristics. The designs facilitate GO/NO-GO decisions due to their complementary statistical–clinical criterion.

Limitations

While conceptually simple, implementing a dual-criterion design needs care. The clinical DV must be elicited carefully in collaboration with clinicians, and understanding similarities and differences to a standard design is crucial.

Conclusion

To improve evidence-based decision-making, a formal yet transparent quantitative framework is important. Dual-criterion designs offer an appealing statistical–clinical compromise, which may be preferable to standard designs if evidence against the null hypothesis alone does not suffice for an efficacy claim.

Quantitative decision-making in randomized Phase II studies with a time-to-event endpoint

One of the most critical decision points in clinical development is Go/No-Go decision-making after a proof-of-concept study. Traditional decision-making relies on a formal hypothesis testing with control of type I and type II error rates, which is limited by assessing the strength of efficacy evidence in a small isolated trial. In this article, we propose a quantitative Bayesian/frequentist decision framework for Go/No-Go criteria and sample size evaluation in Phase II randomized studies with a time-to-event endpoint. By taking the uncertainty of treatment effect into consideration, we propose an integrated quantitative approach for a program when both the Phase II and Phase III trials share a common endpoint while allowing a discount of the observed Phase II data. Our results confirm the argument that an increase in the sample size of a Phase II trial will result in greater increase in the probability of success of a Phase III trial than increasing the Phase III trial sample size by equal amount. We illustrate the steps in quantitative decision-making with a real example of a randomized Phase II study in metastatic pancreatic cancer.

Bayesian sample size determination for a Phase III clinical trial with diluted treatment effect

When Phase III treatment effect is diluted from what was observed from Phase II results, we propose to determine the Bayesian sample size for a Phase III clinical trial based on the normal, uniform, and truncated normal prior distributions of the treatment effects on an interval, which starts from an acceptable treatment effect to the observed treatment effect from Phase II. After incorporating the prior information of the treatment effects, the Bayesian sample size is the number of patients of the Phase III trial for a given Bayesian Predictive Power (BPP) or Bayesian Historical Predictive Power (BHPP). After that, the numerical simulations are carried out to determine the Bayesian sample size for the Phase III clinical trial. In particular, there exists a hook phenomenon for the BHPP when the number of patients of the Phase II trial equals 70 assuming the normal, uniform, or truncated normal treatment effect. Moreover, we add some sensitivity analysis of the Bayesian sample size about the parameters in the simulations. Finally, we determine the Bayesian sample size (number of events or deaths) of the Phase III trial for a fixed power, Bayesian Historical Power (BHP), and BHPP in the axitinib example.

A decision theoretical modeling for Phase III investments and drug licensing

For a new candidate drug to become an approved medicine, several decision points have to be passed. In this article, we focus on two of them: First, based on Phase II data, the commercial sponsor decides to invest (or not) in Phase III. Second, based on the outcome of Phase III, the regulator determines whether the drug should be granted market access. Assuming a population of candidate drugs with a distribution of true efficacy, we optimize the two stakeholders' decisions and study the interdependence between them. The regulator is assumed to seek to optimize the total public health benefit resulting from the efficacy of the drug and a safety penalty. In optimizing the regulatory rules, in terms of minimal required sample size and the Type I error in Phase III, we have to consider how these rules will modify the commercial optimization made by the sponsor. The results indicate that different Type I errors should be used depending on the rarity of the disease.

Simulation-based adjustment after exploratory biomarker subgroup selection in phase II

As part of the evaluation of phase II trials, it is common practice to perform exploratory subgroup analyses with the aim of identifying patient populations with a beneficial treatment effect. When investigating targeted therapies, these subgroups are typically defined by biomarkers. Promising results may lead to the decision to select the respective subgroup as the target population for a subsequent phase III trial. However, a selection based on a large observed treatment effect may potentially induce an upwards-bias leading to over-optimistic expectations on the success probability of the phase III trial. We describe how Approximate Bayesian Computation techniques can be used to derive a simulation-based bias adjustment method in this situation. Recommendations for the implementation of the approach are given. Simulation studies show that the proposed method reduces bias substantially compared with the maximum likelihood estimator. The procedure is illustrated with data from an oncology trial. Copyright © 2017 John Wiley & Sons, Ltd.

The Development and Validation of a Generic Instrument, QoDoS, for Assessing the Quality of Decision Making

INTRODUCTION

The impact of decision-making during the development and the regulatory review of medicines greatly influences the delivery of new medicinal products. Currently, there is no generic instrument that can be used to assess the quality of decision-making. This study describes the development of the Quality of Decision-Making Orientation Scheme QoDoS(©) instrument for appraising the quality of decision-making.

METHODS

Semi-structured interviews about decision-making were carried out with 29 senior decision makers from the pharmaceutical industry (10), regulatory authorities (9) and contract research organizations (10). The interviews offered a qualified understanding of the subjective decision-making approach, influences, behaviors and other factors that impact such processes for individuals and organizations involved in the delivery of new medicines. Thematic analysis of the transcribed interviews was carried out using NVivo8® software. Content validity was carried out using qualitative and quantitative data by an expert panel, which led to the developmental version of the QoDoS. Further psychometric evaluations were performed, including factor analysis, item reduction, reliability testing and construct validation.

RESULTS

The thematic analysis of the interviews yielded a 94-item initial version of the QoDoS(©) with a 5-point Likert scale. The instrument was tested for content validity using a panel of experts for language clarity, completeness, relevance and scaling, resulting in a favorable agreement by panel members with an intra-class correlation coefficient value of 0.89 (95% confidence interval = 0.56, 0.99). A 76-item QoDoS(©) (version 2) emerged from content validation. Factor analysis produced a 47-item measure with four domains. The 47-item QoDoS(©) (version 3) showed high internal consistency (n = 120, Cronbach's alpha = 0.89), high reproducibility (n = 20, intra-class correlation = 0.77) and a mean completion time of 10 min. Reliability testing and construct validation was successfully performed.

CONCLUSION

The QoDoS(©) is both reliable and valid for use. It has the potential for extensive use in medicines development by both the pharmaceutical industry and regulatory authorities. The QoDoS(©) can be used to assess the quality of decision-making and to inform decision makers of the factors that influence decision-making.

The shrinking or disappearing observed treatment effect

It is frequently noted that an initial clinical trial finding was not reproduced in a later trial. This is often met with some surprise. Yet, there is a relatively straightforward reason partially responsible for this observation. In this article, we examine this reason by first reviewing some findings in a recent publication in the Journal of the American Medical Association. To help explain the non-negligible chance of failing to reproduce a previous positive finding, we compare a series of trials to successive diagnostic tests used for identifying a condition. To help explain the suspicion that the treatment effect, when observed in a subsequent trial, seems to have decreased in magnitude, we draw a conceptual analogy between phases II-III development stages and interim analyses of a trial with a group sequential design. Both analogies remind us that what we observed in an early trial could be a false positive or a random high. We discuss statistical sources for these occurrences and discuss why it is important for statisticians to take these into consideration when designing and interpreting trial results.

The statistical interpretation of pilot trials: should significance thresholds be reconsidered?

BACKGROUND

In an evaluation of a new health technology, a pilot trial may be undertaken prior to a trial that makes a definitive assessment of benefit. The objective of pilot studies is to provide sufficient evidence that a larger definitive trial can be undertaken and, at times, to provide a preliminary assessment of benefit.

METHODS

We describe significance thresholds, confidence intervals and surrogate markers in the context of pilot studies and how Bayesian methods can be used in pilot trials. We use a worked example to illustrate the issues raised.

RESULTS

We show how significance levels other than the traditional 5% should be considered to provide preliminary evidence for efficacy and how estimation and confidence intervals should be the focus to provide an estimated range of possible treatment effects. We also illustrate how Bayesian methods could also assist in the early assessment of a health technology.

CONCLUSIONS

We recommend that in pilot trials the focus should be on descriptive statistics and estimation, using confidence intervals, rather than formal hypothesis testing and that confidence intervals other than 95% confidence intervals, such as 85% or 75%, be used for the estimation. The confidence interval should then be interpreted with regards to the minimum clinically important difference. We also recommend that Bayesian methods be used to assist in the interpretation of pilot trials. Surrogate endpoints can also be used in pilot trials but they must reliably predict the overall effect on the clinical outcome.

Evaluating and utilizing probability of study success in clinical development

Background

Drug development has become increasingly costly, lengthy, and risky. The call for better decision making in research and development has never been stronger. Analytic tools that utilize available data can inform decision makers of the risks and benefits of various decisions, which could lead to better and more informed decisions.

Purpose

Through some real oncology examples, we will demonstrate how using available data to analytically evaluate probability of study success (PrSS) can lead to better decisions in clinical development.

Methods

The predictive power, or average conditional power, is used to quantify the PrSS. To calculate the probability, we follow a general two-step process: (1) use Bayesian modeling and appropriate assumptions to synthesize relevant data to derive the distribution of treatment effect and (2) evaluate the PrSS analytically or via trial simulation.

Results

We applied the procedure to several compounds in our oncology pipeline. The analysis informed decision making where PrSS was an important factor to consider.

Limitations

When modeling the treatment effect, we made certain assumptions, including how two drugs work together and exchangeable treatment effects across studies. Those assumptions are reasonable for our specific situations but may not generalize well.

Conclusions

From our experience, PrSS based on available data can help decision making in drug development, particularly the Go/No-Go decision after the proof of concept trial is completed. When applicable, we recommend this evaluation be regularly done in addition to the routine data analysis for clinical trials.

Modelling and simulation in the pharmaceutical industry--some reflections

Modelling and simulation (M&S) is increasingly being applied in (clinical) drug development. It provides an opportune area for the community of pharmaceutical statisticians to pursue. In this article, we highlight useful principles behind the application of M&S. We claim that M&S should be focussed on decisions, tailored to its purpose and based in applied sciences, not relying entirely on data-driven statistical analysis. Further, M&S should be a continuous process making use of diverse information sources and applying Bayesian and frequentist methodology, as appropriate. In addition to forming a basis for analysing decision options, M&S provides a framework that can facilitate communication between stakeholders. Besides the discussion on modelling philosophy, we also describe how standard simulation practice can be ineffective and how simulation efficiency can often be greatly improved.