Estimands: discussion points from the PSI estimands and sensitivity expert group

A causal roadmap for generating high-quality real-world evidence

Increasing emphasis on the use of real-world evidence (RWE) to support clinical policy and regulatory decision-making has led to a proliferation of guidance, advice, and frameworks from regulatory agencies, academia, professional societies, and industry. A broad spectrum of studies use real-world data (RWD) to produce RWE, ranging from randomized trials with outcomes assessed using RWD to fully observational studies. Yet, many proposals for generating RWE lack sufficient detail, and many analyses of RWD suffer from implausible assumptions, other methodological flaws, or inappropriate interpretations. The Causal Roadmap is an explicit, itemized, iterative process that guides investigators to prespecify study design and analysis plans; it addresses a wide range of guidance within a single framework. By supporting the transparent evaluation of causal assumptions and facilitating objective comparisons of design and analysis choices based on prespecified criteria, the Roadmap can help investigators to evaluate the quality of evidence that a given study is likely to produce, specify a study to generate high-quality RWE, and communicate effectively with regulatory agencies and other stakeholders. This paper aims to disseminate and extend the Causal Roadmap framework for use by clinical and translational researchers; three companion papers demonstrate applications of the Causal Roadmap for specific use cases.

Defining estimands in clinical trials: A unified procedure

The ICH E9 (R1) addendum proposes five strategies to define estimands by addressing intercurrent events. However, mathematical forms of these targeted quantities are lacking, which might lead to discordance between statisticians who estimate these quantities and clinicians, drug sponsors, and regulators who interpret them. To improve the concordance, we provide a unified four-step procedure for constructing the mathematical estimands. We apply the procedure for each strategy to derive the mathematical estimands and compare the five strategies in practical interpretations, data collection, and analytical methods. Finally, we show that the procedure can help ease tasks of defining estimands in settings with multiple types of intercurrent events using two real clinical trials.

Evaluating how clear the questions being investigated in randomised trials are: systematic review of estimands

OBJECTIVES

To evaluate how often the precise research question being addressed about an intervention (the estimand) is stated or can be determined from reported methods, and to identify what types of questions are being investigated in phase 2-4 randomised trials.

DESIGN

Systematic review of the clarity of research questions being investigated in randomised trials in 2020 in six leading general medical journals.

DATA SOURCE

PubMed search in February 2021.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

Phase 2-4 randomised trials, with no restrictions on medical conditions or interventions. Cluster randomised, crossover, non-inferiority, and equivalence trials were excluded.

MAIN OUTCOME MEASURES

Number of trials that stated the precise primary question being addressed about an intervention (ie, the primary estimand), or for which the primary estimand could be determined unambiguously from the reported methods using statistical knowledge. Strategies used to handle post-randomisation events that affect the interpretation or existence of patient outcomes, such as intervention discontinuations or uses of additional drug treatments (known as intercurrent events), and the corresponding types of questions being investigated.

RESULTS

255 eligible randomised trials were identified. No trials clearly stated all the attributes of the estimand. In 117 (46%) of 255 trials, the primary estimand could be determined from the reported methods. Intercurrent events were reported in 242 (95%) of 255 trials; but the handling of these could only be determined in 125 (49%) of 255 trials. Most trials that provided this information considered the occurrence of intercurrent events as irrelevant in the calculation of the treatment effect and assessed the effect of the intervention regardless (96/125, 77%)-that is, they used a treatment policy strategy. Four (4%) of 99 trials with treatment non-adherence owing to adverse events estimated the treatment effect in a hypothetical setting (ie, the effect as if participants continued treatment despite adverse events), and 19 (79%) of 24 trials where some patients died estimated the treatment effect in a hypothetical setting (ie, the effect as if participants did not die).

CONCLUSIONS

The precise research question being investigated in most trials is unclear, mainly because of a lack of clarity on the approach to handling intercurrent events. Clear reporting of estimands is necessary in trial reports so that all stakeholders, including clinicians, patients and policy makers, can make fully informed decisions about medical interventions.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42021238053.

Estimands: bringing clarity and focus to research questions in clinical trials

Precise specification of the research question and associated treatment effect of interest is essential in clinical research, yet recent work shows that they are often incompletely specified. The ICH E9 (R1) Addendum on Estimands and Sensitivity Analysis in Clinical Trials introduces a framework that supports researchers in precisely and transparently specifying the treatment effect they aim to estimate in their clinical trial. In this paper, we present practical examples to demonstrate to all researchers involved in clinical trials how estimands can help them to specify the research question, lead to a better understanding of the treatment effect to be estimated and hence increase the probability of success of the trial.

Estimands and missing data in clinical trials of chronic pain treatments: advances in design and analysis

In clinical trials of treatments for chronic pain, the percentage of participants who withdraw early can be as high as 50%. Major reasons for early withdrawal in these studies include perceived lack of efficacy and adverse events. Commonly used strategies for accommodating missing data include last observation carried forward, baseline observation carried forward, and more principled methods such as mixed-model repeated-measures and multiple imputation. All these methods require strong and untestable assumptions concerning the conditional distribution of outcomes after dropout, given the observed data. We review recent developments in statistical methods for handling missing data in clinical trials, including implications of the increased emphasis being placed on precise formulation of the study objectives and the estimand (treatment effect to be estimated) of interest. A flexible method that seems to be well suited for the analysis of chronic pain clinical trials is control-based imputation, which allows a variety of assumptions to be made concerning the conditional distribution of postdropout outcomes that can be tailored to the estimand of interest. These assumptions can depend, for example, on the stated reasons for dropout. We illustrate these methods using data from 4 clinical trials of pregabalin for the treatment of painful diabetic peripheral neuropathy and postherpetic neuralgia. When planning chronic pain clinical trials, careful consideration of the trial objectives should determine the definition of the trial estimand, which in turn should inform methods used to accommodate missing data in the statistical analysis.

Essential statistical principles of clinical trials of pain treatments

This article presents an overview of fundamental statistical principles of clinical trials of pain treatments. Statistical considerations relevant to phase 2 proof of concept and phase 3 confirmatory randomized trials investigating efficacy and safety are discussed, including (1) research design; (2) endpoints and analyses; (3) sample size determination and statistical power; (4) missing data and trial estimands; (5) data monitoring and interim analyses; and (6) interpretation of results. Although clinical trials of pharmacologic treatments are emphasized, the key issues raised by these trials are also directly applicable to clinical trials of other types of treatments, including biologics, devices, nonpharmacologic therapies (eg, physical therapy and cognitive-behavior therapy), and complementary and integrative health interventions.

Estimands and inference in cluster-randomized vaccine trials

Cluster-randomized trials are often conducted to assess vaccine effects. Defining estimands of interest before conducting a trial is integral to the alignment between a study's objectives and the data to be collected and analyzed. This paper considers estimands and estimators for overall, indirect, and total vaccine effects in trials, where clusters of individuals are randomized to vaccine or control. The scenario is considered where individuals self-select whether to participate in the trial, and the outcome of interest is measured on all individuals in each cluster. Unlike the overall, indirect, and total effects, the direct effect of vaccination is shown in general not to be estimable without further assumptions, such as no unmeasured confounding. An illustrative example motivated by a cluster-randomized typhoid vaccine trial is provided.

Practical help for specifying the target difference in sample size calculations for RCTs: the DELTA2 five-stage study, including a workshop

BACKGROUND

The randomised controlled trial is widely considered to be the gold standard study for comparing the effectiveness of health interventions. Central to its design is a calculation of the number of participants needed (the sample size) for the trial. The sample size is typically calculated by specifying the magnitude of the difference in the primary outcome between the intervention effects for the population of interest. This difference is called the 'target difference' and should be appropriate for the principal estimand of interest and determined by the primary aim of the study. The target difference between treatments should be considered realistic and/or important by one or more key stakeholder groups.

OBJECTIVE

The objective of the report is to provide practical help on the choice of target difference used in the sample size calculation for a randomised controlled trial for researchers and funder representatives.

METHODS

The Difference ELicitation in TriAls2 (DELTA2) recommendations and advice were developed through a five-stage process, which included two literature reviews of existing funder guidance and recent methodological literature; a Delphi process to engage with a wider group of stakeholders; a 2-day workshop; and finalising the core document.

RESULTS

Advice is provided for definitive trials (Phase III/IV studies). Methods for choosing the target difference are reviewed. To aid those new to the topic, and to encourage better practice, 10 recommendations are made regarding choosing the target difference and undertaking a sample size calculation. Recommended reporting items for trial proposal, protocols and results papers under the conventional approach are also provided. Case studies reflecting different trial designs and covering different conditions are provided. Alternative trial designs and methods for choosing the sample size are also briefly considered.

CONCLUSIONS

Choosing an appropriate sample size is crucial if a study is to inform clinical practice. The number of patients recruited into the trial needs to be sufficient to answer the objectives; however, the number should not be higher than necessary to avoid unnecessary burden on patients and wasting precious resources. The choice of the target difference is a key part of this process under the conventional approach to sample size calculations. This document provides advice and recommendations to improve practice and reporting regarding this aspect of trial design. Future work could extend the work to address other less common approaches to the sample size calculations, particularly in terms of appropriate reporting items.

FUNDING

Funded by the Medical Research Council (MRC) UK and the National Institute for Health Research as part of the MRC-National Institute for Health Research Methodology Research programme.

Patient-reported outcomes of patients with advanced renal cell carcinoma treated with nivolumab plus ipilimumab versus sunitinib (CheckMate 214): a randomised, phase 3 trial

BACKGROUND

In the ongoing phase 3, CheckMate 214 trial, nivolumab plus ipilimumab improved overall survival compared with sunitinib in patients with intermediate or poor risk, previously untreated, advanced renal cell carcinoma. We aimed to assess whether health-related quality of life (HRQoL) could be used to further describe the benefit-risk profile of nivolumab plus ipilimumab versus sunitinib.

METHODS

In the phase 3, randomised, controlled, CheckMate 214 trial, patients aged 18 years and older with previously untreated, advanced or metastatic renal cell carcinoma with a clear-cell component were recruited from 175 hospitals and cancer centres in 28 countries. Patients were categorised by risk status into favourable, intermediate, and poor risk subgroups and randomly assigned (1:1) to open-label nivolumab 3 mg/kg plus ipilimumab 1 mg/kg every 3 weeks for four doses followed by nivolumab 3 mg/kg every 2 weeks, or sunitinib 50 mg/day for 4 weeks of each 6-week cycle. Randomisation was done with a block size of four and stratified by risk status and geographical region. Patient-reported outcomes (PROs) were assessed using the Functional Assessment of Cancer Therapy Kidney Symptom Index-19 (FKSI-19), Functional Assessment of Cancer Therapy-General (FACT-G), and EuroQol five dimensional three level (EQ-5D-3L) instruments. The coprimary endpoints of the trial, reported previously, were overall survival, progression-free survival, and the proportion of patients who had an objective response in those categorised as at intermediate or poor risk. PROs in all randomised participants were assessed as an exploratory endpoint; here we report this exploratory endpoint. This study is registered with ClinicalTrials.gov, number NCT02231749, and is ongoing but is now closed to recruitment.

FINDINGS

Between Oct 16, 2014, and Feb 23, 2016, of 1390 patients screened, 1096 (79%) were randomly assigned to treatment, of whom 847 (77%) were at intermediate or poor risk and randomly assigned to nivolumab plus ipilimumab (n=425) or sunitinib (n=422). Median follow-up was 25·2 months (IQR 23·0-27·4). PROs were more favourable with nivolumab plus ipilimumab than sunitinib throughout the first 103 weeks after baseline, with mean change from baseline at week 103 for FKSI-19 total score being 4·00 (95% CI 1·91 to 6·09) for nivolumab plus ipilimumab versus -3·14 (-6·03 to -0·25) for sunitinib (p<0·0001), and for FACT-G total score being 4·77 (1·73 to 7·82) for nivolumab plus ipilimumab versus -4·32 (-8·54 to -0·11) for sunitinib (p=0·0005). Significant differences were also seen for four of five FKSI-19 domains (disease-related symptoms, physical disease-related symptoms, treatment side-effects, and functional wellbeing) and FACT-G physical and functional wellbeing domains. However, there was no significant difference between the treatment groups at week 103 in EQ-5D-3L visual analogue rating scale (VAS) scores, with mean change from baseline to week 103 of 10·07 (95% CI 4·35 to 15·80) for nivolumab plus ipilimumab and 6·40 (-1·36 to 14·16) for sunitinib (p=0·45). Compared with sunitinib, nivolumab plus ipilimumab reduced risk of deterioration in FKSI-19 total score (hazard ratio [HR] 0·54; 95% CI 0·46-0·63), FACT-G total score (0·63, 0·52-0·75), and EQ-5D-3L VAS score (HR 0·75, 95% CI 0·63-0·89) and UK utility scores (0·67, 0·57-0·80).

INTERPRETATION

Nivolumab plus ipilimumab leads to fewer symptoms and better HRQoL than sunitinib in patients at intermediate or poor risk with advanced renal cell carcinoma. These results suggest that the superior efficacy of nivolumab plus ipilimumab over sunitinib comes with the additional benefit of improved HRQoL.

FUNDING

Bristol-Myers Squibb and ONO Pharmaceutical.

Development process of a consensus-driven CONSORT extension for randomised trials using an adaptive design

BACKGROUND

Adequate reporting of adaptive designs (ADs) maximises their potential benefits in the conduct of clinical trials. Transparent reporting can help address some obstacles and concerns relating to the use of ADs. Currently, there are deficiencies in the reporting of AD trials. To overcome this, we have developed a consensus-driven extension to the CONSORT statement for randomised trials using an AD. This paper describes the processes and methods used to develop this extension rather than detailed explanation of the guideline.

METHODS

We developed the guideline in seven overlapping stages: 1) Building on prior research to inform the need for a guideline; 2) A scoping literature review to inform future stages; 3) Drafting the first checklist version involving an External Expert Panel; 4) A two-round Delphi process involving international, multidisciplinary, and cross-sector key stakeholders; 5) A consensus meeting to advise which reporting items to retain through voting, and to discuss the structure of what to include in the supporting explanation and elaboration (E&E) document; 6) Refining and finalising the checklist; and 7) Writing-up and dissemination of the E&E document. The CONSORT Executive Group oversaw the entire development process.

RESULTS

Delphi survey response rates were 94/143 (66%), 114/156 (73%), and 79/143 (55%) in rounds 1, 2, and across both rounds, respectively. Twenty-seven delegates from Europe, the USA, and Asia attended the consensus meeting. The main checklist has seven new and nine modified items and six unchanged items with expanded E&E text to clarify further considerations for ADs. The abstract checklist has one new and one modified item together with an unchanged item with expanded E&E text. The E&E document will describe the scope of the guideline, the definition of an AD, and some types of ADs and trial adaptations and explain each reporting item in detail including case studies.

CONCLUSIONS

We hope that making the development processes, methods, and all supporting information that aided decision-making transparent will enhance the acceptability and quick uptake of the guideline. This will also help other groups when developing similar CONSORT extensions. The guideline is applicable to all randomised trials with an AD and contains minimum reporting requirements.

DELTA2 guidance on choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial

Background

A key step in the design of a RCT is the estimation of the number of participants needed in the study. The most common approach is to specify a target difference between the treatments for the primary outcome and then calculate the required sample size. The sample size is chosen to ensure that the trial will have a high probability (adequate statistical power) of detecting a target difference between the treatments should one exist.

The sample size has many implications for the conduct and interpretation of the study. Despite the critical role that the target difference has in the design of a RCT, the way in which it is determined has received little attention. In this article, we summarise the key considerations and messages from new guidance for researchers and funders on specifying the target difference, and undertaking and reporting a RCT sample size calculation. This article on choosing the target difference for a randomised controlled trial (RCT) and undertaking and reporting the sample size calculation has been dual published in the BMJ and BMC Trials journals

Methods

The DELTA² (Difference ELicitation in TriAls) project comprised five major components: systematic literature reviews of recent methodological developments (stage 1) and existing funder guidance (stage 2); a Delphi study (stage 3); a two-day consensus meeting bringing together researchers, funders and patient representatives (stage 4); and the preparation and dissemination of a guidance document (stage 5).

Results and Discussion

The key messages from the DELTA² guidance on determining the target difference and sample size calculation for a randomised caontrolled trial are presented. Recommendations for the subsequent reporting of the sample size calculation are also provided.

DELTA2 guidance on choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial

BACKGROUND

A key step in the design of a RCT is the estimation of the number of participants needed in the study. The most common approach is to specify a target difference between the treatments for the primary outcome and then calculate the required sample size. The sample size is chosen to ensure that the trial will have a high probability (adequate statistical power) of detecting a target difference between the treatments should one exist. The sample size has many implications for the conduct and interpretation of the study. Despite the critical role that the target difference has in the design of a RCT, the way in which it is determined has received little attention. In this article, we summarise the key considerations and messages from new guidance for researchers and funders on specifying the target difference, and undertaking and reporting a RCT sample size calculation. This article on choosing the target difference for a randomised controlled trial (RCT) and undertaking and reporting the sample size calculation has been dual published in the BMJ and BMC Trials journals METHODS: The DELTA² (Difference ELicitation in TriAls) project comprised five major components: systematic literature reviews of recent methodological developments (stage 1) and existing funder guidance (stage 2); a Delphi study (stage 3); a two-day consensus meeting bringing together researchers, funders and patient representatives (stage 4); and the preparation and dissemination of a guidance document (stage 5).

RESULTS AND DISCUSSION

The key messages from the DELTA² guidance on determining the target difference and sample size calculation for a randomised caontrolled trial are presented. Recommendations for the subsequent reporting of the sample size calculation are also provided.