When are clinical trials beneficial for study patients and future patients? A factorial vignette-based survey of institutional review board members

IDENTIFICATION OF THRESHOLD FOR LARGE (DRAMATIC) EFFECTS THAT WOULD OBVIATE RANDOMIZED TRIALS IS NOT POSSIBLE

OBJECTIVE

To analyze distribution of "dramatic", large treatment effects.

STUDY DESIGN & SETTING

Pareto distribution modeling of previously reported cohorts of 3,486 randomized trials (RCTs) that enrolled 1,532,459 patients and 730 non-randomized studies (NRS) enrolling 1,650,658 patients.

RESULTS

We calculated the Pareto α parameter, which determines the tail of the distribution for various starting points of distribution [odds ratio_min (OR_min)]. In default analysis using all data at OR_min ≥1, Pareto distribution fit well to the treatment effects of RCTs favoring the new treatments (P = 0.21, Kolmogorov-Smirnov test) with best α = 2.32. For NRS, Pareto fit for OR_min ≥2 with best α = 1.91. For RCTs, theoretical 99th percentile OR was 32.7. The actual 99th percentile OR was 25; which converted into relative risk (RR) = 7.1. The maximum observed effect size was OR = 121 (RR = 11.45). For NRS, theoretical 99th percentile was OR = 315. The actual 99th percentile OR was 294 (RR = 13). The maximum observed effect size was OR = 1473 (RR = 66).

CONCLUSIONS

The effects sizes observed in RCTs and NRS considerably overlap. Large effects are rare and there is no clear threshold for dramatic effects that would obviate future RCTs.