Estimands in practice: Bridging the gap between study objectives and statistical analysis

Five-year efficacy outcomes of ocrelizumab in relapsing multiple sclerosis: A propensity-matched comparison of the OPERA studies with other disease-modifying therapies in real-world lines of treatments

Background

Clinical trials comparing the efficacy of ocrelizumab (OCR) with other disease-modifying therapies (DMTs) other than interferon (IFN) β-1a in relapsing multiple sclerosis (RMS) are lacking.

Objectives

To compare the treatment effect of OCR vs six DMTs' (IFN β-1a, glatiramer acetate, fingolimod, dimethyl fumarate, teriflunomide, natalizumab) treatment pathways used in clinical practice by combining clinical trial and real-world data.

Methods

Patient-level data from OPERA trials and open-label extension phase, and from the German NeuroTransData (NTD) MS registry, were used to build 1:1 propensity score-matched (PSM) cohorts controlling for seven baseline covariates, including brain imaging activity. Efficacy outcomes were time to first relapse and time to 24-week confirmed disability progression over 5.5 years of follow-up. Intention-to-treat analysis using all outcome data irrespective of treatment switch was applied.

Results

The analyses included 611 OPERA patients and 7141 NTD patients. We built 12 paired-matched cohorts (six for each outcome, two for each DMT) to compare efficacy of OCR in OPERA with each DMT treatment pathway in NTD. Post-matching, baseline covariates and PS were well balanced (standardized mean difference <.2 for all cohorts). Over 5.5 years, patients treated with OCR showed a statistically significant reduction in the risk of relapse (hazard ratios [HRs] .30 to .54) and disability progression (HRs .51 to .67) compared with all index therapies and their treatment switching pathways in NTD. Treatment switch and/or discontinuation occurred frequently in NTD cohorts.

Conclusion

OCR demonstrates superiority in controlling relapses and disability progression in RMS compared with real-world treatment pathways over a 5.5-year period. These analyses suggest that high-efficacy DMTs and high treatment persistence are critical to achieve greatest clinical benefit in RMS.

Registration

OPERA I (NCT01247324), OPERA II (NCT01412333).

Application of hypothetical strategies in acute pain

Since the publication of ICH E9 (R1), "Addendum to statistical principles for clinical trials: on choosing appropriate estimands and defining sensitivity analyses in clinical trials," there has been a lot of debate about the hypothetical strategy for handling intercurrent events. Arguments against the hypothetical strategy are twofold: (1) the clinical question has limited clinical/regulatory interest; (2) the estimation may need strong statistical assumptions. In this article, we provide an example of a hypothetical strategy handling use of rescue medications in the acute pain setting. We argue that the treatment effect of a drug that is attributable to the treatment alone is the clinical question of interest and is important to regulators. The hypothetical strategy is important when developing non-opioid treatment as it estimates the treatment effect due to treatment during the pre-specified evaluation period whereas the treatment policy strategy does not. Two widely acceptable and non-controversial clinical inputs are required to construct a reasonable estimator. More importantly, this estimator does not rely on additional strong statistical assumptions and is considered reasonable for regulatory decision making. In this article, we point out examples where estimators for a hypothetical strategy can be constructed without any strong additional statistical assumptions besides acceptable clinical inputs. We also showcase a new way to obtain estimation based on disease specific clinical knowledge instead of strong statistical assumptions. In the example presented, we clearly demonstrate the advantages of the hypothetical strategy compared to alternative strategies including the treatment policy strategy and a composite variable strategy.

Design and analysis considerations for early phase clinical trials in hepatitis B (HBV) cure research: the ACTG A5394 study in persons with both HIV and HBV

With growing interest and efforts to achieve a hepatitis B (HBV) cure, HBV therapeutics have increasingly entered the clinical testing phase. In designing an early phase clinical trial aimed at HBV cure, the heterogeneity in participants and the choice of a biomarker endpoint that signals a cure requires careful consideration. We describe the key elements to consider during the development of HBV clinical trials aimed at a functional cure, and how we have addressed them in the design of a phase II AIDS Clinical Trials Group (ACTG) study, A5394 (NCT05551273). The trial we present is for persons with both HIV and HBV, a unique population that has much to gain from an HBV cure. Our decisions on the design elements are specific to the study agent and the targeted population, but our deliberations may be informative in the emerging field of early phase HBV trials aimed at cure.

Tell me what you want, what you really really want: Estimands in observational pharmacoepidemiologic comparative effectiveness and safety studies

PURPOSE

Ideally, the objectives of a pharmacoepidemiologic comparative effectiveness or safety study should dictate its design and data analysis. This paper discusses how defining an estimand is instrumental to this process.

METHODS

We applied the ICH-E9 (Statistical Principles for Clinical Trials) R1 addendum on estimands - which originally focused on randomized trials - to three examples of observational pharmacoepidemiologic comparative effectiveness and safety studies. Five key elements specify the estimand: the population, contrasted treatments, endpoint, intercurrent events, and population-level summary measure.

RESULTS

Different estimands were defined for case studies representing three types of pharmacological treatments: (1) single-dose treatments using a case study about the effect of influenza vaccination versus no vaccination on mortality risk in an adult population of ≥60 years of age; (2) sustained-treatments using a case study about the effect of dipeptidyl peptidase 4 inhibitor versus glucagon-like peptide-1 agonist on hypoglycemia risk in treatment of uncontrolled diabetes; and (3) as needed treatments using a case study on the effect of nitroglycerin spray as-needed versus no nitroglycerin on syncope risk in treatment of stabile angina pectoris.

CONCLUSIONS

The case studies illustrated that a seemingly clear research question can still be open to multiple interpretations. Defining an estimand ensures that the study targets a treatment effect that aligns with the treatment decision the study aims to inform. Estimand definitions further help to inform choices regarding study design and data-analysis and clarify how to interpret study findings.

Important Considerations for Signal Detection and Evaluation

Safety clinicians have a wealth of resources describing how to perform signal detection. Nevertheless, there are some nuances concerning approaches taken by regulatory authorities and statistical considerations that should be appreciated. New approaches, such as the FDA Medical Queries, illustrate the value of considering medical concepts over individual adverse events. One area which would benefit from further clarity is how safety signals may be evaluated for evidence of a causal relationship to the drug of interest. Just as such safety signals can take many forms, the types of tools and methods required to interrogate these signals are equally as diverse. An understanding of the complexity of this process can aid the safety reviewer in successfully characterizing the emerging safety profile of a drug during the pre-marketing phase of development.

Evaluating how clear the questions being investigated in randomised trials are: systematic review of estimands

OBJECTIVES

To evaluate how often the precise research question being addressed about an intervention (the estimand) is stated or can be determined from reported methods, and to identify what types of questions are being investigated in phase 2-4 randomised trials.

DESIGN

Systematic review of the clarity of research questions being investigated in randomised trials in 2020 in six leading general medical journals.

DATA SOURCE

PubMed search in February 2021.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

Phase 2-4 randomised trials, with no restrictions on medical conditions or interventions. Cluster randomised, crossover, non-inferiority, and equivalence trials were excluded.

MAIN OUTCOME MEASURES

Number of trials that stated the precise primary question being addressed about an intervention (ie, the primary estimand), or for which the primary estimand could be determined unambiguously from the reported methods using statistical knowledge. Strategies used to handle post-randomisation events that affect the interpretation or existence of patient outcomes, such as intervention discontinuations or uses of additional drug treatments (known as intercurrent events), and the corresponding types of questions being investigated.

RESULTS

255 eligible randomised trials were identified. No trials clearly stated all the attributes of the estimand. In 117 (46%) of 255 trials, the primary estimand could be determined from the reported methods. Intercurrent events were reported in 242 (95%) of 255 trials; but the handling of these could only be determined in 125 (49%) of 255 trials. Most trials that provided this information considered the occurrence of intercurrent events as irrelevant in the calculation of the treatment effect and assessed the effect of the intervention regardless (96/125, 77%)-that is, they used a treatment policy strategy. Four (4%) of 99 trials with treatment non-adherence owing to adverse events estimated the treatment effect in a hypothetical setting (ie, the effect as if participants continued treatment despite adverse events), and 19 (79%) of 24 trials where some patients died estimated the treatment effect in a hypothetical setting (ie, the effect as if participants did not die).

CONCLUSIONS

The precise research question being investigated in most trials is unclear, mainly because of a lack of clarity on the approach to handling intercurrent events. Clear reporting of estimands is necessary in trial reports so that all stakeholders, including clinicians, patients and policy makers, can make fully informed decisions about medical interventions.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42021238053.

Estimators for handling COVID-19-related intercurrent events with a hypothetical strategy

The COVID-19 pandemic has affected clinical trials across disease areas, raising the questions how interpretable results can be obtained from impacted studies. Applying the estimands framework, analyses may seek to estimate the treatment effect in the hypothetical absence of such impact. However, no established estimators exist. This simulation study, based on an ongoing clinical trial in patients with Tourette syndrome, compares the performance of candidate estimators for estimands including either a continuous or binary variable and applying a hypothetical strategy for COVID-19-related intercurrent events (IE). The performance is investigated in a wide range of scenarios, under the null and the alternative hypotheses, including different modeling assumptions for the effect of the IE and proportions of affected patients ranging from 10% to 80%. Bias and type I error inflation were minimal or absent for most estimators under most scenarios, with only multiple imputation- and weighting-based methods displaying a type I error inflation in some scenarios. Of more concern, all methods that discarded post-IE data displayed a sharp decrease of power proportional to the proportion of affected patients, corresponding to both a reduced precision of estimation and larger confidence intervals. The simulation study shows that de-mediation via g-estimation is a promising approach. Besides showing the best performance in our simulation study, these approaches allow to estimate the effect of the IE on the outcome and cross-compare between different studies affected by similar IEs. Importantly, the results can be extrapolated to IEs not related to COVID-19 that follow a similar causal structure.

Estimands: bringing clarity and focus to research questions in clinical trials

Precise specification of the research question and associated treatment effect of interest is essential in clinical research, yet recent work shows that they are often incompletely specified. The ICH E9 (R1) Addendum on Estimands and Sensitivity Analysis in Clinical Trials introduces a framework that supports researchers in precisely and transparently specifying the treatment effect they aim to estimate in their clinical trial. In this paper, we present practical examples to demonstrate to all researchers involved in clinical trials how estimands can help them to specify the research question, lead to a better understanding of the treatment effect to be estimated and hence increase the probability of success of the trial.

Why ITT analysis is not always the answer for estimating treatment effects in clinical trials

For many years there has been a consensus among the Clinical Research community that ITT analysis represents the correct approach for the vast majority of trials. Recent worldwide regulatory guidance for pharmaceutical industry trials has allowed discussion of alternatives to the ITT approach to analysis; different treatment effects can be considered which may be more clinically meaningful and more relevant to patients and prescribers. The key concept is of a trial "estimand", a precise description of the estimated treatment effect. The strategy chosen to account for patients who discontinue treatment or take alternative medications which are not part of the randomised treatment regimen are important determinants of this treatment effect. One strategy to account for these events is treatment policy, which corresponds to an ITT approach. Alternative equally valid strategies address what the treatment effect is if the patient actually takes the treatment or does not use specific alternative medication. There is no single right answer to which strategy is most appropriate, the solution depends on the key clinical question of interest. The estimands framework discussed in the new guidance has been particularly useful in the context of the current COVID-19 pandemic and has clarified what choices are available to account for the impact of COVID-19 on clinical trials. Specifically, an ITT approach addresses a treatment effect that may not be generalisable beyond the current pandemic.

Confidence intervals for exposure-adjusted rate differences in randomized trials

Exposure-adjusted event rate is a quantity often used in clinical trials to describe average event count per unit of person-time. The event count may represent the number of patients experiencing first (incident) event episode, or the total number of event episodes, including recurring events. For inference about difference in the exposure-adjusted rates between interventions, many methods of interval estimation rely on the assumption of Poisson distribution for the event counts. These intervals may suffer from substantial undercoverage both, asymptotically due to extra-Poisson variation, and in the settings with rare events even when the Poisson assumption is satisfied. We review asymptotically robust methods of interval estimation for the rate difference that do not depend on distributional assumptions for the event counts, and propose a modification of one of these methods. The new interval estimator has asymptotically nominal coverage for the rate difference with an arbitrary distribution of event counts, and good finite sample properties, avoiding substantial undercoverage with small samples, rare events, or over-dispersed data. The proposed method can handle covariate adjustment and can be implemented with commonly available software. The method is illustrated using real data on adverse events in a clinical trial.