A guide to megastudies

Using survey experiment pretesting to support future pandemic response

The world could witness another pandemic on the scale of COVID-19 in the future, prompting calls for research into how social and behavioral science can better contribute to pandemic response, especially regarding public engagement and communication. Here, we conduct a cost-effectiveness analysis of a familiar tool from social and behavioral science that could potentially increase the impact of public communication: survey experiments. Specifically, we analyze whether a public health campaign that pays for a survey experiment to pretest and choose between different messages for its public outreach has greater impact in expectation than an otherwise-identical campaign that does not. The main results of our analysis are 3-fold. First, we show that the benefit of such pretesting depends heavily on the values of several key parameters. Second, via simulations and an evidence review, we find that a campaign that allocates some of its budget to pretesting could plausibly increase its expected impact; that is, we estimate that pretesting is cost-effective. Third, we find pretesting has potentially powerful returns to scale; for well-resourced campaigns, we estimate pretesting is robustly cost-effective, a finding that emphasizes the benefit of public health campaigns sharing resources and findings. Our results suggest survey experiment pretesting could cost-effectively increase the impact of public health campaigns in a pandemic, have implications for practice, and establish a research agenda to advance knowledge in this space.

Effectiveness of ex ante honesty oaths in reducing dishonesty depends on content

Dishonest behaviours such as tax evasion impose significant societal costs. Ex ante honesty oaths-commitments to honesty before action-have been proposed as interventions to counteract dishonest behaviour, but the heterogeneity in findings across operationalizations calls their effectiveness into question. We tested 21 honesty oaths (including a baseline oath)-proposed, evaluated and selected by 44 expert researchers-and a no-oath condition in a megastudy involving 21,506 UK and US participants from Prolific.com who played an incentivized tax evasion game online. Of the 21 interventions, 10 significantly improved tax compliance by 4.5 to 8.5 percentage points, with the most successful nearly halving tax evasion. Limited evidence for moderators was found. Experts and laypeople failed to predict the most effective interventions, though experts' predictions were more accurate. In conclusion, honesty oaths were effective in curbing dishonesty, but their effectiveness varied depending on content. These findings can help design impactful interventions to curb dishonesty.

Megastudy shows that reminders boost vaccination but adding free rides does not

Encouraging routine COVID-19 vaccinations is likely to be a crucial policy challenge for decades to come. To avert hundreds of thousands of unnecessary hospitalizations and deaths, adoption will need to be higher than it was in the autumn of 2022 or 2023, when less than one-fifth of Americans received booster vaccines1,2. One approach to encouraging vaccination is to eliminate the friction of transportation hurdles. Previous research has shown that friction can hinder follow-through3 and that individuals who live farther from COVID-19 vaccination sites are less likely to get vaccinated4. However, the value of providing free round-trip transportation to vaccination sites is unknown. Here we show that offering people free round-trip Lyft rides to pharmacies has no benefit over and above sending them behaviourally informed text messages reminding them to get vaccinated. We determined this by running a megastudy with millions of CVS Pharmacy patients in the United States testing the effects of (1) free round-trip Lyft rides to CVS Pharmacies for vaccination appointments and (2) seven different sets of behaviourally informed vaccine reminder messages. Our results suggest that offering previously vaccinated individuals free rides to vaccination sites is not a good investment in the United States, contrary to the high expectations of both expert and lay forecasters. Instead, people in the United States should be sent behaviourally informed COVID-19 vaccination reminders, which increased the 30-day COVID-19 booster uptake by 21% (1.05 percentage points) and spilled over to increase 30-day influenza vaccinations by 8% (0.34 percentage points) in our megastudy. More rigorous testing of interventions to promote vaccination is needed to ensure that evidence-based solutions are deployed widely and that ineffective but intuitively appealing tools are discontinued.

Toolbox of individual-level interventions against online misinformation

The spread of misinformation through media and social networks threatens many aspects of society, including public health and the state of democracies. One approach to mitigating the effect of misinformation focuses on individual-level interventions, equipping policymakers and the public with essential tools to curb the spread and influence of falsehoods. Here we introduce a toolbox of individual-level interventions for reducing harm from online misinformation. Comprising an up-to-date account of interventions featured in 81 scientific papers from across the globe, the toolbox provides both a conceptual overview of nine main types of interventions, including their target, scope and examples, and a summary of the empirical evidence supporting the interventions, including the methods and experimental paradigms used to test them. The nine types of interventions covered are accuracy prompts, debunking and rebuttals, friction, inoculation, lateral reading and verification strategies, media-literacy tips, social norms, source-credibility labels, and warning and fact-checking labels.

Field testing the transferability of behavioural science knowledge on promoting vaccinations

As behavioural science is increasingly adopted by organizations, there is a growing need to assess the robustness and transferability of empirical findings. Here, we investigate the transferability of insights from various sources of behavioural science knowledge to field settings. Across three pre-registered randomized controlled trials (RCTs, N = 314,824) involving a critical policy domain-COVID-19 booster uptake-we field tested text-based interventions that either increased vaccinations in prior field work (RCT1, NCT05586204), elevated vaccination intentions in an online study (RCT2, NCT05586178) or were favoured by scientists and non-experts (RCT3, NCT05586165). Despite repeated exposure to COVID-19 vaccination messaging in our population, reminders and psychological ownership language increased booster uptake, replicating prior findings. However, strategies deemed effective by prediction or intention surveys, such as encouraging the bundling of COVID-19 boosters and flu shots or addressing misconceptions, yielded no detectable benefits over simple reminders. These findings underscore the importance of testing interventions' transferability to real-world settings.

Addressing climate change with behavioral science: A global intervention tournament in 63 countries

Effectively reducing climate change requires marked, global behavior change. However, it is unclear which strategies are most likely to motivate people to change their climate beliefs and behaviors. Here, we tested 11 expert-crowdsourced interventions on four climate mitigation outcomes: beliefs, policy support, information sharing intention, and an effortful tree-planting behavioral task. Across 59,440 participants from 63 countries, the interventions' effectiveness was small, largely limited to nonclimate skeptics, and differed across outcomes: Beliefs were strengthened mostly by decreasing psychological distance (by 2.3%), policy support by writing a letter to a future-generation member (2.6%), information sharing by negative emotion induction (12.1%), and no intervention increased the more effortful behavior-several interventions even reduced tree planting. Last, the effects of each intervention differed depending on people's initial climate beliefs. These findings suggest that the impact of behavioral climate interventions varies across audiences and target behaviors.

Psychological inoculation strategies to fight climate disinformation across 12 countries

Decades after the scientific debate about the anthropogenic causes of climate change was settled, climate disinformation still challenges the scientific evidence in public discourse. Here we present a comprehensive theoretical framework of (anti)science belief formation and updating to account for the psychological factors that influence the acceptance or rejection of scientific messages. We experimentally investigated, across 12 countries (N = 6,816), the effectiveness of six inoculation strategies targeting these factors-scientific consensus, trust in scientists, transparent communication, moralization of climate action, accuracy and positive emotions-to fight real-world disinformation about climate science and mitigation actions. While exposure to disinformation had strong detrimental effects on participants' climate change beliefs (δ = -0.16), affect towards climate mitigation action (δ = -0.33), ability to detect disinformation (δ = -0.14) and pro-environmental behaviour (δ = -0.24), we found almost no evidence for protective effects of the inoculations (all δ < 0.20). We discuss the implications of these findings and propose ways forward to fight climate disinformation.

Megastudies: A New Approach to Reducing Vaccine Hesitation Worldwide

Vaccine hesitancy is a considerable obstacle to achieving vaccine protection worldwide. There needs to be more evidence-based research for interventions for vaccine hesitancy. Existing effectiveness evaluations are limited to one particular hypothesis, and no studies have compared the effectiveness of different interventions. A megastudy takes a large-scale, multi-intervention, uniform participant and the same evaluation criteria approach to evaluate many interventions simultaneously and find the most effective ones. Therefore, megastudies can help us find the most effective interventions for vaccine hesitancy. Additionally, considering the complex causes of vaccine hesitancy, we design interventions that involve social factors in megastudies. Lastly, quality control and justice are critical issues for megastudies in the future.