Assessing and Improving Data Integrity in Web-Based Surveys: Comparison of Fraud Detection Systems in a COVID-19 Study

BACKGROUND

Web-based surveys increase access to study participation and improve opportunities to reach diverse populations. However, web-based surveys are vulnerable to data quality threats, including fraudulent entries from automated bots and duplicative submissions. Widely used proprietary tools to identify fraud offer little transparency about the methods used, effectiveness, or representativeness of resulting data sets. Robust, reproducible, and context-specific methods of accurately detecting fraudulent responses are needed to ensure integrity and maximize the value of web-based survey research.

OBJECTIVE

This study aims to describe a multilayered fraud detection system implemented in a large web-based survey about COVID-19 attitudes, beliefs, and behaviors; examine the agreement between this fraud detection system and a proprietary fraud detection system; and compare the resulting study samples from each of the 2 fraud detection methods.

METHODS

The PhillyCEAL Common Survey is a cross-sectional web-based survey that remotely enrolled residents ages 13 years and older to assess how the COVID-19 pandemic impacted individuals, neighborhoods, and communities in Philadelphia, Pennsylvania. Two fraud detection methods are described and compared: (1) a multilayer fraud detection strategy developed by the research team that combined automated validation of response data and real-time verification of study entries by study personnel and (2) the proprietary fraud detection system used by the Qualtrics (Qualtrics) survey platform. Descriptive statistics were computed for the full sample and for responses classified as valid by 2 different fraud detection methods, and classification tables were created to assess agreement between the methods. The impact of fraud detection methods on the distribution of vaccine confidence by racial or ethnic group was assessed.

RESULTS

Of 7950 completed surveys, our multilayer fraud detection system identified 3228 (40.60%) cases as valid, while the Qualtrics fraud detection system identified 4389 (55.21%) cases as valid. The 2 methods showed only "fair" or "minimal" agreement in their classifications (κ=0.25; 95% CI 0.23-0.27). The choice of fraud detection method impacted the distribution of vaccine confidence by racial or ethnic group.

CONCLUSIONS

The selection of a fraud detection method can affect the study's sample composition. The findings of this study, while not conclusive, suggest that a multilayered approach to fraud detection that includes conservative use of automated fraud detection and integration of human review of entries tailored to the study's specific context and its participants may be warranted for future survey research.

Leveraging Resilience-supportive Strategies to Enhance Protective Factors in Young Sexual Minority Men: A Scoping Review of HIV Behavioral Interventions Implemented in High-income Countries

Behavioral HIV interventions focused on strengthening young sexual minority men's (SMM) internal (assets) and external (resources) protective factors are promising, yet their evaluation as resilience-supportive strategies to minimize or negate HIV-related risks remain understudied. The objective of this scoping review was to describe resilience-supportive intervention strategies that have been used to achieve desired HIV behavioral outcomes and to identify how these strategies have been evaluated using a resilience analytic framework. Our scoping review uncovered 271 peer-reviewed articles, of which 38 were eligible for inclusion based on our review criteria. The majority of interventions relied on social support strategies as their primary resilience-supportive strategy. A third of interventions reviewed analyzed their findings from a deficits-focused model, another third used compensatory resilience models, and the remaining interventions employed a hybrid (i.e., deficit and compensatory model) strategy. None of the interventions evaluated their intervention effects using a risk-protective model. From our synthesis regarding the current state of research around resilience-informed interventions, we propose strategies to inform the design of resilience-supportive approaches and make recommendations to move the field forward on how to develop, implement, and measure young SMM's resiliency processes.