Estimation of unobservable selection effects in on-line surveys through propensity score matching: An application to public acceptance of healthy eating policies

Measuring public opinion and acceptability of prevention policies: an integrative review and narrative synthesis of methods

Acceptability of and public support for prevention are an important part of facilitating policy implementation. This review aims to identify, summarize and synthesize the methods and study designs used to measure and understand public opinion, community attitudes and acceptability of strategies to prevent chronic noncommunicable disease (NCDs) in order to allow for examination of imbalances in methodological approaches and gaps in content areas. We searched four scientific databases (CINAHL, Embase, Ovid/MEDLINE and Scopus) for peer-reviewed, English-language studies published between January 2011 and March 2020 in high-income, democratic countries across North America, Europe and the Asia–Pacific region. Studies were included if they focused on opinions, attitudes and acceptability of primary prevention strategies and interventions addressing the key NCD risk factors of alcohol use, unhealthy diet, overweight/obesity, tobacco use and smoking, and physical inactivity. A total of 293 studies were included. Two thirds of studies (n = 194, 66%) used quantitative methods such as cross-sectional studies involving surveys of representative (n = 129, 44%) or convenience (n = 42, 14%) samples. A smaller number of studies used qualitative methods (n = 60, 20%) such as focus groups (n = 21, 7%) and interviews (n = 21, 7%). Thirty-nine studies (13%) used mixed methods such as content analysis of news media (n = 17, 6%). Tobacco control remains the dominant topic of public opinion literature about prevention (n = 124, 42%). Few studies looked solely at physical inactivity (n = 17, 6%). The results of this review suggest that public opinion and acceptability of prevention in the peer-reviewed literature is investigated primarily through cross-sectional surveys. Qualitative and mixed methods may provide more nuanced insights which can be used to facilitate policy implementation of more upstream strategies and policies to prevent NCDs.

An evaluation of whether propensity score adjustment can remove the self-selection bias inherent to web panel surveys addressing sensitive health behaviours

Background

In health research, population estimates are generally obtained from probability-based surveys. In market research surveys are frequently conducted from volunteer web panels. Propensity score adjustment (PSA) is often used at analysis to try to remove bias in the web survey, but empirical evidence of its effectiveness is mixed. We assess the ability of PSA to remove bias in the context of sensitive sexual health research and the potential of web panel surveys to replace or supplement probability surveys.

Methods

Four web panel surveys asked a subset of questions from the third British National Survey of Sexual Attitudes and Lifestyles (Natsal-3). Five propensity scores were generated for each web survey. The scores were developed from progressively larger sets of variables, beginning with demographic variables only and ending with demographic, sexual identity, lifestyle, attitudinal and sexual behaviour variables together. The surveys were weighted to match Natsal-3 based on propensity score quintiles. The performance of each survey and weighting was assessed by calculating the average ‘absolute’ odds ratio (inverse of the odds ratio if less than 1) across 22 pre-specified sexual behaviour outcomes of interest comparing the weighted web survey with Natsal-3. The average standard error across odds ratios was examined to assess the impact of weighting upon variance.

Results

Propensity weighting reduced bias relative to Natsal-3 as more variables were added for males, but had little effect for females, and variance increased for some surveys. Surveys with more biased estimates before propensity weighting showed greater reduction in bias from adjustment. Inconsistencies in performance were evident across surveys and outcomes. For most surveys and outcomes any reduction in bias was only partial and for some outcomes the bias increased.

Conclusions

Even after propensity weighting using a rich range of information, including some sexual behaviour variables, some bias remained and variance increased for some web surveys. Whilst our findings support the use of PSA for web panel surveys, the reduction in bias is likely to be partial and unpredictable, consistent with the findings from market research. Our results do not support the use of volunteer web panels to generate unbiased population health estimates.