Eliminating Ambiguous Treatment Effects Using Estimands

Quasi-experimental methods for pharmacoepidemiology: difference-in-differences and synthetic control methods with case studies for vaccine evaluation

Difference-in-differences and synthetic control methods have become common study designs for evaluating the effects of changes in policies, including health policies. They also have potential for providing real-world effectiveness and safety evidence in pharmacoepidemiology. To effectively add to the toolkit of the field, however, designs-including both their benefits and drawbacks-must be well understood. Quasi-experimental designs provide an opportunity to estimate the average treatment effect on the treated without requiring the measurement of all possible confounding factors, and to assess population-level effects. This requires, however, other key assumptions, including the parallel trends or stable weighting assumptions, a lack of other concurrent events that could alter time trends, and an absence of contamination between exposed and unexposed units. The targeted estimands are also highly specific to the settings of the study, and combining across units or time periods can be challenging. Case studies are presented for 3 vaccine evaluation studies, showcasing some of these challenges and opportunities in a specific field of pharmacoepidemiology. These methods provide feasible and valuable sources of evidence in various pharmacoepidemiologic settings and can be improved through research to identify and weigh the advantages and disadvantages in those settings. This article is part of a Special Collection on Pharmacoepidemiology.

Composite event-free-survival as an endpoint in oncology drug evaluation: Review and guidance perspectives from the Haute Autorité de Santé (HAS)

BACKGROUND

The use of right-censored composite endpoints, such as progression-free survival, has been questioned in haemato-oncology trials due to potential bias in estimated treatment effect. This may impact the accuracy of health technology evaluations. We hypothesized that there is heterogeneity and potential sources of bias in the reporting of composite endpoints to health technology assessment (HTA) bodies.

METHODS

We reviewed the submissions for reimbursement of oncology drugs in 2021 and 2022 that used a composite endpoint in the pivotal trial, after appraisal by the French HTA body. The retrieved information included the clinical study report, protocol, and statistical analysis plan submitted by the industry. All events of the composite endpoint and all causes of censored observations were measured. The design characteristics and treatment effect estimates were recorded.

FINDINGS

Seventy-six submissions were selected, including seven without a right-censored endpoint and four evaluating associations, resulting in 65 analysed records: 17 for haematological and 48 for solid tumours. Out these 65 submissions, 47 (72·3%) used a randomized controlled design, and 18 (27·7%) a non-comparative design. The most frequently used composite endpoint was progression-free survival, used in 54 (83·1%) of the submissions. Censoring was possibly informative in 51 (92·7%) cases, mostly due to the onset of new treatment (44/51, 86·3%) and/or discontinuation of follow-up (33/51, 64·7%). In contrast, 38 (58·5%) trials reported a quantification of censored observations, with only 12/51 (23·5%) quantifying the informative ones. The estimated treatment effect on the composite outcome increased with the amount of censoring, suggesting a higher benefit of the drug, but remained below that on survival with poor evidence of surrogacy (R-squared=0·23).

INTERPRETATION

Clinical study reports should be improved in terms of reporting censoring, while stakeholders should be aware of this potential source of bias. At a minimum, sensitivity analysis that ignores intercurrent events should be requested.

Demystifying estimands in cluster-randomised trials

Estimands can help clarify the interpretation of treatment effects and ensure that estimators are aligned with the study's objectives. Cluster-randomised trials require additional attributes to be defined within the estimand compared to individually randomised trials, including whether treatment effects are marginal or cluster-specific, and whether they are participant- or cluster-average. In this paper, we provide formal definitions of estimands encompassing both these attributes using potential outcomes notation and describe differences between them. We then provide an overview of estimators for each estimand, describe their assumptions, and show consistency (i.e. asymptotically unbiased estimation) for a series of analyses based on cluster-level summaries. Then, through a re-analysis of a published cluster-randomised trial, we demonstrate that the choice of both estimand and estimator can affect interpretation. For instance, the estimated odds ratio ranged from 1.38 (p = 0.17) to 1.83 (p = 0.03) depending on the target estimand, and for some estimands, the choice of estimator affected the conclusions by leading to smaller treatment effect estimates. We conclude that careful specification of the estimand, along with an appropriate choice of estimator, is essential to ensuring that cluster-randomised trials address the right question.

Causal interpretation of the hazard ratio in randomized clinical trials

BACKGROUND

Although the hazard ratio has no straightforward causal interpretation, clinical trialists commonly use it as a measure of treatment effect.

METHODS

We review the definition and examples of causal estimands. We discuss the causal interpretation of the hazard ratio from a two-arm randomized clinical trial, and the implications of proportional hazards assumptions in the context of potential outcomes. We illustrate the application of these concepts in a synthetic model and in a model of the time-varying effects of COVID-19 vaccination.

RESULTS

We define causal estimands as having either an individual-level or population-level interpretation. Difference-in-expectation estimands are both individual-level and population-level estimands, whereas without strong untestable assumptions the causal rate ratio and hazard ratio have only population-level interpretations. We caution users against making an incorrect individual-level interpretation, emphasizing that in general a hazard ratio does not on average change each individual's hazard by a factor. We discuss a potentially valid interpretation of the constant hazard ratio as a population-level causal effect under the proportional hazards assumption.

CONCLUSION

We conclude that the population-level hazard ratio remains a useful estimand, but one must interpret it with appropriate attention to the underlying causal model. This is especially important for interpreting hazard ratios over time.

Estimands in CNS trials - A review of strategies for addressing intercurrent events

Background

The estimands framework represents a significant innovation for the design, conduct, analysis, and interpretation of clinical trials. An aim of the framework is to increase precision and transparency on the handling of intercurrent events (IEs), defined as events occurring after treatment initiation and affecting the endpoint. While the experience in constructing and reporting estimands in the published literature is limited, developers performing confirmatory studies are already making use of the new paradigm, allowing to survey the strategies proposed by applicants and endorsed by regulators.

Methods

To identify strategies for handling IEs in confirmatory central nervous system (CNS) trials, we searched scientific advice letters issued by the European Medicines Agency (EMA) between 2017 and 2022. We developed a categorisation of the IEs and classified, according to the strategies defined in the framework, the strategies proposed by the Applicants and recommended by the agency. Strategies proposed and recommended were summarised by category of IEs, and the rationale for the choices was analysed qualitatively.

Results

In total, 170 IEs were identified in 52 confirmatory trials. A clear preference for the treatment policy strategy for treatment discontinuation and for the hypothetical strategy for pandemic-related disruptions was identified. For other categories of IEs, there are more mixed patterns.

Discussion

This study highlights the multidimensional nature of choosing a strategy for an IE. For different occurring IEs in confirmatory CNS trials different strategies are of regulatory interest, depending on the trial objective, underlying disease properties, rarity of disease, as well as frequency and timing of IEs and their relatedness to the disease.

Optimal Heart Failure Medical Therapy and Mortality in Survivors of Cardiogenic Shock: Insights From the FRENSHOCK Registry

BACKGROUND

The effects of pharmacological therapy on cardiogenic shock (CS) survivors have not been extensively studied. Thus, this study investigated the association between guideline-directed heart failure (HF) medical therapy (GDMT) and one-year survival rate in patients who are post-CS.

METHODS AND RESULTS

FRENSHOCK (French Observatory on the Management of Cardiogenic Shock in 2016) registry was a prospective multicenter observational survey, conducted in metropolitan French intensive care units and intensive cardiac care units. Of 772 patients, 535 patients were enrolled in the present analysis following the exclusion of 217 in-hospital deaths and 20 patients with missing medical records. Patients with triple GDMT (beta-blockers, renin-angiotensin system inhibitors, and mineralocorticoid receptor antagonists) at discharge (n=112) were likely to have lower left ventricular ejection fraction on admission and at discharge compared with those without triple GDMT (n=423) (22% versus 28%, P<0.001 and 29% versus 37%, P<0.001, respectively). In the overall cohort, the one-year mortality rate was 23%. Triple GDMT prescription was significantly associated with a lower one-year all-cause mortality compared with non-triple GDMT (adjusted hazard ratio 0.44 [95% CI, 0.19-0.80]; P=0.007). Similarly, 2:1 propensity score matching and inverse probability treatment weighting based on the propensity score demonstrated a lower incidence of one-year mortality in the triple GDMT group. As the number of HF drugs increased, a stepwise decrease in mortality was observed (log rank; P<0.001).

CONCLUSIONS

In survivors of CS, the one-year mortality rate was significantly lower in those with triple GDMT. Therefore, this study suggests that intensive HF therapy should be considered in patients following CS.

Visualizing the target estimand in comparative effectiveness studies with multiple treatments

Aim: Comparative effectiveness research using real-world data often involves pairwise propensity score matching to adjust for confounding bias. We show that corresponding treatment effect estimates may have limited external validity, and propose two visualization tools to clarify the target estimand. Materials & methods: We conduct a simulation study to demonstrate, with bivariate ellipses and joy plots, that differences in covariate distributions across treatment groups may affect the external validity of treatment effect estimates. We showcase how these visualization tools can facilitate the interpretation of target estimands in a case study comparing the effectiveness of teriflunomide (TERI), dimethyl fumarate (DMF) and natalizumab (NAT) on manual dexterity in patients with multiple sclerosis. Results: In the simulation study, estimates of the treatment effect greatly differed depending on the target population. For example, when comparing treatment B with C, the estimated treatment effect (and respective standard error) varied from -0.27 (0.03) to -0.37 (0.04) in the type of patients initially receiving treatment B and C, respectively. Visualization of the matched samples revealed that covariate distributions vary for each comparison and cannot be used to target one common treatment effect for the three treatment comparisons. In the case study, the bivariate distribution of age and disease duration varied across the population of patients receiving TERI, DMF or NAT. Although results suggest that DMF and NAT improve manual dexterity at 1 year compared with TERI, the effectiveness of DMF versus NAT differs depending on which target estimand is used. Conclusion: Visualization tools may help to clarify the target population in comparative effectiveness studies and resolve ambiguity about the interpretation of estimated treatment effects.

Consensus Statement for Protocols of Factorial Randomized Trials: Extension of the SPIRIT 2013 Statement

Importance

Trial protocols outline a trial's objectives as well as the methods (design, conduct, and analysis) that will be used to meet those objectives, and transparent reporting of trial protocols ensures objectives are clear and facilitates appraisal regarding the suitability of study methods. Factorial trials, in which 2 or more interventions are assessed in the same set of participants, have unique methodological considerations. However, no extension of the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) 2013 Statement, which provides guidance on reporting of trial protocols, for factorial trials is available.

Objective

To develop a consensus-based extension to the SPIRIT 2013 Statement for factorial trials.

Evidence Review

The SPIRIT extension for factorial trials was developed using the Enhancing the Quality and Transparency of Health Research (EQUATOR) methodological framework. First, a list of reporting recommendations was generated using a scoping review of methodological articles identified using a MEDLINE search (inception to May 2019), which was supplemented with relevant articles from the personal collections of the authors. Second, a 3-round Delphi survey (January to June 2022, completed by 104 panelists from 14 countries) was conducted to assess the importance of items and identify additional recommendations. Third, a hybrid consensus meeting was held, attended by 15 panelists to finalize selection and wording of the checklist.

Findings

This SPIRIT extension for factorial trials modified 9 of the 33 items in the SPIRIT 2013 checklist. Key reporting recommendations were that the rationale for using a factorial design should be provided, including whether an interaction is hypothesized; the treatment groups that will form the main comparisons should be identified; and statistical methods for each main comparison should be provided, including how interactions will be assessed.

Conclusions and Relevance

In this consensus statement, 9 factorial-specific items were provided that should be addressed in all protocols of factorial trials to increase the trial's utility and transparency.

Association of SGLT-2 Inhibitors With Treatment Satisfaction and Diabetes-Specific and General Health Status in Adults With Cardiovascular Disease and Type 2 Diabetes

Background It is unknown if initiation of a sodium-glucose cotransporter-2 inhibitor (SGLT-2i) is associated with changes in patient-reported health status outside of clinical trials. Methods and Results Using a prospective observational study design, adults with type 2 diabetes and cardiovascular disease were recruited from 14 US hospitals between November 2019 and December 2021 if they were new users of noninsulin antidiabetic medications. The primary outcome was change in 6-month diabetes treatment satisfaction. Secondary outcomes included diabetes-related symptom distress, diabetes-specific quality of life, and general health status for all patients and based on cardiovascular disease type. Inverse probability of treatment weight using propensity score was performed to compare outcome changes based on medication use. Of 887 patients (SGLT-2i: n=242) included in the inverse probability of treatment weight analyses, there was no difference in changes in treatment satisfaction in SGLT-2i users compared with other diabetes medication users (0.99 [95% CI, -0.14 to 2.13] versus 1.54 [1.08 to 2.00], P=0.38). Initiating an SGLT-2i versus other diabetes medications was associated with a greater reduction in ophthalmological symptoms (-3.09 [95% CI, -4.99 to -1.18] versus -0.38 [-1.54 to 0.77], P=0.018) but less improvement in hyperglycemia (1.08 [-2.63 to 4.79] versus -3.60 [-5.34 to -1.86], P=0.026). In subgroup analyses by cardiovascular disease type, SGLT-2i use was associated with a greater reduction in total diabetes symptom burden and neurological sensory symptoms in patients with heart failure. Conclusions Among patients with type 2 diabetes and cardiovascular disease, initiating an SGLT-2i was not associated with changes in diabetes treatment satisfaction, total diabetes symptoms, diabetes-specific quality of life, or general health status.

Transcatheter Ductal Stents Versus Surgical Systemic-Pulmonary Artery Shunts in Neonates With Congenital Heart Disease With Ductal-Dependent Pulmonary Blood Flow: Trends and Associated Outcomes From the Pediatric Health Information System Database

Background Surgical systemic-to-pulmonary artery shunts have been the standard approach to establish stable pulmonary blood flow in neonates with congenital heart disease with ductal-dependent pulmonary blood flow. More recently, transcatheter ductal stents have been performed as an alternative, less invasive intervention. We aimed to characterize trends in the utilization of surgical shunts versus ductal stents and compare associated outcomes. Methods and Results Using data from the Pediatric Health Information System, we retrospectively analyzed neonates with congenital heart disease with ductal-dependent pulmonary blood flow who underwent surgical shunt or ductal stent placement between January 2016 and December 2021. Patients were identified by International Classification of Diseases, Tenth Revision (ICD-10) diagnosis and procedure codes. The primary outcome was length of hospital stay. Secondary outcomes were reintervention risk and adjusted hospital costs. Of 936 patients included, 65.2% underwent a surgical shunt over the 6-year period. The proportion who underwent ductal stenting increased from 19% to 53.4% from 2016 to 2021. The median adjusted difference in postintervention length of hospital stay was 11 days greater for the surgical shunt cohort (95% CI, 7.2-14.8; P<0.001). The adjusted reintervention risks within 3 (odds ratio [OR], 3.37 [95% CI, 1.91-5.95], P<0.001) and 6 months (OR, 2.43 [95% CI, 1.62-3.64], P<0.001) were significantly greater in the ductal stent group. Median adjusted index hospital costs were $198 300 ($11 6400-$340 000) versus $120 400 ($81 800-$192 400) for the surgical shunt and ductal stent cohorts, respectively (P<0.001). Conclusions Ductal stenting has become an increasingly utilized palliative approach to secure pulmonary blood flow in neonates with congenital heart disease with ductal-dependent pulmonary blood flow in the United States. Ductal stenting is associated with decreased length of hospital stay and reduced overall cost for the index hospitalization but with a greater reintervention risk than surgical shunting.

Tell me what you want, what you really really want: Estimands in observational pharmacoepidemiologic comparative effectiveness and safety studies

PURPOSE

Ideally, the objectives of a pharmacoepidemiologic comparative effectiveness or safety study should dictate its design and data analysis. This paper discusses how defining an estimand is instrumental to this process.

METHODS

We applied the ICH-E9 (Statistical Principles for Clinical Trials) R1 addendum on estimands - which originally focused on randomized trials - to three examples of observational pharmacoepidemiologic comparative effectiveness and safety studies. Five key elements specify the estimand: the population, contrasted treatments, endpoint, intercurrent events, and population-level summary measure.

RESULTS

Different estimands were defined for case studies representing three types of pharmacological treatments: (1) single-dose treatments using a case study about the effect of influenza vaccination versus no vaccination on mortality risk in an adult population of ≥60 years of age; (2) sustained-treatments using a case study about the effect of dipeptidyl peptidase 4 inhibitor versus glucagon-like peptide-1 agonist on hypoglycemia risk in treatment of uncontrolled diabetes; and (3) as needed treatments using a case study on the effect of nitroglycerin spray as-needed versus no nitroglycerin on syncope risk in treatment of stabile angina pectoris.

CONCLUSIONS

The case studies illustrated that a seemingly clear research question can still be open to multiple interpretations. Defining an estimand ensures that the study targets a treatment effect that aligns with the treatment decision the study aims to inform. Estimand definitions further help to inform choices regarding study design and data-analysis and clarify how to interpret study findings.

Fluid Optimisation in Emergency Laparotomy (FLO-ELA) Trial: study protocol for a multi-centre randomised trial of cardiac output-guided fluid therapy compared to usual care in patients undergoing major emergency gastrointestinal surgery

INTRODUCTION

Postoperative morbidity and mortality in patients undergoing major emergency gastrointestinal surgery are a major burden on healthcare systems. Optimal management of perioperative intravenous fluids may reduce mortality rates and improve outcomes from surgery. Previous small trials of cardiac-output guided haemodynamic therapy algorithms in patients undergoing gastrointestinal surgery have suggested this intervention results in reduced complications and a modest reduction in mortality. However, this existing evidence is based mainly on elective (planned) surgery, with little evaluation in the emergency setting. There are fundamental clinical and pathophysiological differences between the planned and emergency surgical setting which may influence the effects of this intervention. A large definitive trial in emergency surgery is needed to confirm or refute the potential benefits observed in elective surgery and to inform widespread clinical practice.

METHODS

The FLO-ELA trial is a multi-centre, parallel-group, open, randomised controlled trial. 3138 patients aged 50 and over undergoing major emergency gastrointestinal surgery will be randomly allocated in a 1:1 ratio using minimisation to minimally invasive cardiac output monitoring to guide protocolised administration of intra-venous fluid, or usual care without cardiac output monitoring. The trial intervention will be carried out during surgery and for up to 6 h postoperatively. The trial is funded through an efficient design call by the National Institute for Health and Care Research Health Technology Assessment (NIHR HTA) programme and uses existing routinely collected datasets for the majority of data collection. The primary outcome is the number of days alive and out of hospital within 90 days of randomisation. Participants and those delivering the intervention will not be blinded to treatment allocation. Participant recruitment started in September 2017 with a 1-year internal pilot phase and is ongoing at the time of publication.

DISCUSSION

This will be the largest contemporary randomised trial examining the effectiveness of perioperative cardiac output-guided haemodynamic therapy in patients undergoing major emergency gastrointestinal surgery. The multi-centre design and broad inclusion criteria support the external validity of the trial. Although the clinical teams delivering the trial interventions will not be blinded, significant trial outcome measures are objective and not subject to detection bias.

TRIAL REGISTRATION

ISRCTN 14729158. Registered on 02 May 2017.