The renaissance of field experimentation in evaluating interventions

Most experiments are done in laboratories. However, there is also a theory and practice of field experimentation. It has had its successes and failures over the past four decades but is now increasingly used for answering causal questions. This is true for both randomized and-perhaps more surprisingly-nonrandomized experiments. In this article, we review the history of the use of field experiments, discuss some of the reasons for their current renaissance, and focus the bulk of the article on the particular technical developments that have made this renaissance possible across four kinds of widely used experimental and quasi-experimental designs-randomized experiments, regression discontinuity designs in which those units above a cutoff get one treatment and those below get another, short interrupted time series, and nonrandomized experiments using a nonequivalent comparison group. We focus this review on some of the key technical developments addressing problems that previously stymied accurate effect estimation, the solution of which opens the way for accurate estimation of effects under the often difficult conditions of field implementation-the estimation of treatment effects under partial treatment implementation, the prevention and analysis of attrition, analysis of nested designs, new analytic developments for both regression discontinuity designs and short interrupted time series, and propensity score analysis. We also cover the key empirical evidence showing the conditions under which some nonrandomized experiments may be able to approximate results from randomized experiments.

A retrospective analysis of the effect of noncompliance on time to first major adverse cardiac event in LIPS

BACKGROUND

In the main publication for LIPS (Lescol Intervention Prevention Study), a 22% relative risk (RR) reduction for major adverse cardiac events (MACE) was found among those who used fluvastatin after a successful first percutaneous coronary intervention (PCI). However, intent-to-treat (ITT) analysis of clinical studies generally provides an observed treatment effect that is likely to underestimate what the treatment effect would be if compliance were perfect, because compliance in a clinical trial is invariably <100% during long-term follow-up.

OBJECTIVE

The aim of this study was to analyze the relationship between compliance and treatment effect in LIPS.

METHODS

In LIPS, patients who had undergone a successful first PCI were randomized to receive fluvastatin 40 mg BID or placebo BID for 3 to 5 years. The primary end point was survival time free of MACE (ie, cardiac death, nonfatal myocardial infarction, or reintervention procedure), and a Cox proportional hazards regression model with time-dependent covariates was used to predict the effect that fluvastatin would have had if trial medication had been continued. Logistic regression was used to determine factors influencing discontinuation of trial medication.

RESULTS

A total of 1677 patients were enrolled in LIPS: 844 in the fluvastatin group and 833 in the placebo group. In the fluvastatin group, 294 patients (34.8%) discontinued taking trial medication and 73 (8.6%) switched to another lipid-lowering medication, compared with 353 (42.4%) and 187 (22.4%) patients in the placebo group, respectively. The risk factor-adjusted RR of MACE with fluvastatin treatment was 0.74 (P = 0.004; 95% CI, 0.61-0.91). When also adjusted for noncompliance, the RR for fluvastatin versus placebo was 0.68 (P = 0.002; 95% CI, 0.53-0.86). Discontinuing fluvastatin without switching to another lipid-lowering medication increased the risk of MACE compared with that of patients who stayed on fluvastatin (RR = 2.27; P < 0.001; 95% CI, 1.60-3.23) and the increase in the risk of MACE was greater than that associated with discontinuing placebo (P = 0.032).

CONCLUSIONS

The present study found a 32% RR reduction for experiencing MACE during fluvastatin treatment after a successful PCI in LIPS, when analysis allowed for noncompliance. This suggests that the ITT analysis discussed in the main LIPS publication underestimated the benefit of fluvastatin treatment. Our survival model also provided tentative evidence that discontinuing lipid-lowering medication might lead to a potentially harmful rebound effect in this patient group.

Impact of treatment changes on the interpretation of the Concorde trial

BACKGROUND

The Concorde trial compared two policies of therapy with zidovudine (ZVD) in individuals with asymptomatic HIV infection: immediate or deferred ZDV. Participants in both groups could stop their blinded trial therapy for several reasons and/or could start open-label ZDV. The difference in survival and disease progression between the two groups was estimated allowing for treatment changes.

METHODS

The relationship between latest CD4 count, treatment changes and time to AIDS-related complex (ARC), AIDS or death was investigated using time-updated proportional hazards models, but these models gave seriously biased estimates of the effect of ZDV. Therefore, a method based on the comparison of the randomized groups was used. A model relating a participant's events times to the treatment actually received was used to estimate what would have been observed if the deferred group had not received ZDV before ARS or AIDS, and to explore alternative policies for starting Pneumocystis carinii pneumonia (PCP) prophylaxis.

RESULTS

The major treatment changes during the trial were the termination of blinded therapy because of adverse events or personal reasons (575 out of 1749 participants), starting open-label ZDV (745 participants), and starting PCP prophylaxis (613 participants). Starting open-label ZDV and PCP prophylaxis were strongly related to latest CD4 count. The uncorrected hazard ratios for immediate compared with deferred groups were 0.89 for time to ARC, AIDS or death [95% confidence interval (CI), 0.75-1.05], 1.01 for time to AIDS or death (95% CI, 0.82-1.24), and 1.26 for time to death (95% CI, 0.93-1.70). After correction for treatment changes, these hazard ratios were 0.79 (95% CI, 0.57-1.11), 1.01 (95% CI, 0.81-1.26), and 1.37 (95% CI, 0.91-2.08), respectively. Correction for PCP prophylaxis made little difference to the results.

CONCLUSIONS

Open-label ZDV before ARC or AIDS in the deferred group was likely to have diluted any differences between the immediate and deferred groups. After correction for this dilution, both the estimated benefit of immediate treatment in delaying progression to ARC, AIDS or death and the estimated disadvantage of immediate treatment in accelerating death were somewhat increased, but both remained consistent with chance alone. This study demonstrated the large potential bias inherent in non-randomization-based methods of analysis of clinical trials.