Objecting to experiments even while approving of the policies or treatments they compare

Randomized Controlled Trial Aversion among Public Sector Leadership: A Survey Experiment

Background: While randomized controlled trials (RCTs) are typically considered the gold standard of program evaluation, they are infrequently chosen by public sector leaders, defined as government and nonprofit decision-makers, when an impact evaluation is required. Objectives: This study provides descriptive evidence on RCT aversion among public sector leaders and attempts to understand what factors affect their likelihood of choosing RCTs for impact evaluations. Research Design: The authors ask if public sector leaders follow similar preference patterns found among non-public sector leaders when choosing either an RCT or a quasi-experimental design and use a survey experiment to determine which factors affect the RCT choice. Subjects: The study sample includes 2050 public sector leaders and a comparison group of 2060 respondents who do not lead public sector organizations. Measures: The primary outcome measure is selecting an RCT as the preferred evaluation option. Results: When asked to make a decision about an impact evaluation, the majority of people do not choose an RCT. While also averse to RCTs, public sector leaders are about 13% more likely to prefer a RCT to a quasi-experimental evaluation compared to the general population. Public sector leaders are less likely to use RCTs for evaluations of more intense interventions, potentially because they are perceived to be superior to the options available for the control group. Conclusion: Funders should be aware that when given a choice, public sector leaders prefer other options to RCTs. Greater awareness of the benefits of RCTs could increase their use in the public sector.

Experiment aversion does not appear to generalize

Over the past decade, governments and organizations around the world have established behavioral insights teams advocating for randomized experiments. However, recent findings by M. N. Meyer et al., Proc. Natl. Acad. Sci. U.S.A. 116, 10723-10728 (2019) and P. R. Heck, C. F. Chabris, D. J. Watts, M. N. Meyer, Proc. Natl. Acad. Sci. U.S.A. 117, 18948-18950 (2020) suggest that people often rate randomized experiments as less appropriate than the policies they contain even when approving the implementation of either policy untested and when none of the individual policies is clearly superior. The authors warn that this could cause policymakers to avoid running large-scale field experiments or being transparent about running them and might contribute to an adverse heterogeneity bias in terms of who is participating in experiments. In one direct and six conceptual preregistered replications (total N = 5,200) of the previously published larger-effect studies, using the same main dependent variable but with variations in scenario wordings, recruitment platforms, and countries, and the addition of further measures to assess people's views, we test the generalizability and robustness of these findings. Together, we find that the original results do not appear to generalize. That is, our triangulation reveals insufficient evidence to conclude that people exhibit a common pattern of behavior that would be consistent with relative experiment aversion, thereby supporting recent findings by R. Mislavsky, B. Dietvorst, U. Simonsohn, Mark. Sci. 39, 1092-1104 (2020). Thus, policymakers may not need to be concerned about employing evidence-based practices more so than about universally implementing policies.

Experiment aversion among clinicians and the public - an obstacle to evidence-based medicine and public health

Background

Randomized controlled trials (RCTs) are essential for determining the safety and efficacy of healthcare interventions. However, both laypeople and clinicians often demonstrate experiment aversion: preferring to implement either of two interventions for everyone rather than comparing them to determine which is best. We studied whether clinician and layperson views of pragmatic RCTs for Covid-19 or other interventions became more positive early in the pandemic, which increased both the urgency and public discussion of RCTs.

Methods

We conducted several survey studies with laypeople (total n=2,909) and two with clinicians (n=895; n=1,254) in 2020 and 2021. Participants read vignettes in which a hypothetical decision-maker who sought to improve health could choose to implement intervention A for all, implement intervention B for all, or experimentally compare A and B and implement the superior intervention. Participants rated and ranked the appropriateness of each decision.

Results

Compared to our pre-pandemic results, we found no decrease in laypeople's aversion to non-Covid-19 experiments involving catheterization checklists and hypertension drugs. Nor were either laypeople or clinicians less averse to Covid-19 RCTs (concerning corticosteroid drugs, vaccines, intubation checklists, proning, school reopening, and mask protocols), on average. Across all vignettes and samples, levels of experiment aversion ranged from 28% to 57%, while levels of experiment appreciation (in which the RCT is rated higher than the participant's highest-rated intervention) ranged from only 6% to 35%.

Conclusions

Advancing evidence-based medicine through pragmatic RCTs will require anticipating and addressing experiment aversion among both patients and healthcare professionals.

Objecting to experiments even while approving of the policies or treatments they compare

We resolve a controversy over two competing hypotheses about why people object to randomized experiments: 1) People unsurprisingly object to experiments only when they object to a policy or treatment the experiment contains, or 2) people can paradoxically object to experiments even when they approve of implementing either condition for everyone. Using multiple measures of preference and test criteria in five preregistered within-subjects studies with 1,955 participants, we find that people often disapprove of experiments involving randomization despite approving of the policies or treatments to be tested.