Outcomes of two randomized controlled trials, employing participants recruited through Mechanical Turk, of Internet interventions targeting unhealthy alcohol use

Rationale and Study Checklist for Ethical Rejection of Participants on Crowdsourcing Research Platforms

Online participant recruitment ("crowdsourcing") platforms are increasingly being used for research studies. While such platforms can rapidly provide access to large samples, there are concomitant concerns around data quality. Researchers have studied and demonstrated means to reduce the prevalence of low-quality data from crowdsourcing platforms, but approaches to doing so often involve rejecting work and/or denying payment to participants, which can pose ethical dilemmas. We write this essay as an associate professor and two institutional review board (IRB) directors to provide a perspective on the competing interests of participants/workers and researchers and to propose a checklist of steps that we believe may support workers' agency on the platform and lessen instances of unfair consequences to them while enabling researchers to definitively reject lower-quality work that might otherwise reduce the likelihood of their studies producing true results. We encourage further, explicit discussion of these issues among academics and among IRBs.

Digital interventions for reducing alcohol use in general populations: An updated systematic review and meta-analysis

This article updates a 2017 review on the effectiveness of digital interventions for reducing alcohol use in the general population. An updated systematic search of the MEDLINE database was performed in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses criteria to identify randomized controlled trials (RCTs) published from January 2017 to June 2022 that evaluated the effectiveness of digital interventions compared with no interventions, minimal interventions, and face-to-face interventions aimed at reducing alcohol use in the general population and, that also reported changes in alcohol use (quantity, frequency, quantity per drinking day, heavy episodic drinking (HED), or alcohol use disorders identification test (AUDIT) scores). A secondary analysis was performed that analyzed data from RCTs conducted in students. The review was not preregistered. The search produced 2224 articles. A total of 80 studies were included in the review, 35 of which were published after the last systematic review. A total of 66, 20, 18, 26, and 9 studies assessed the impact of digital interventions on alcohol quantity, frequency, quantity per drinking day, HED, and AUDIT scores, respectively. Individuals randomized to the digital interventions drank 4.12 (95% confidence interval (CI): 2.88, 5.36) fewer grams of alcohol per day, had 0.17 (95% CI 0.06, 0.29) fewer drinking days per week, drank approximately 3.89 (95% CI: 0.40, 7.38) fewer grams of alcohol per drinking day, had 1.11 (95% CI: 0.32, 1.91) fewer HED occasions per month, and had an AUDIT score 3.04 points lower (95% CI: 2.23, 3.85) than individuals randomized to the control condition. Significant reductions in alcohol quantity, frequency, and HED, but not quantity per drinking day, were observed among students. Digital interventions show potential for reducing alcohol use in general populations and could be used widely at the population level to reduce alcohol-attributable harms.

Measuring psychiatric symptoms online: A systematic review of the use of inventories on Amazon Mechanical Turk (mTurk)

Symptom measurement in psychiatric research increasingly uses digitized self-report inventories and is turning to crowdsourcing platforms for recruitment, e.g., Amazon Mechanical Turk (mTurk). The impact of digitizing pencil-and-paper inventories on the psychometric properties is underexplored in mental health research. Against this background, numerous studies report high prevalence estimates of psychiatric symptoms in mTurk samples. Here we develop a framework to evaluate the online implementation of psychiatric symptom inventories relative to two domains, that is, the adherence to (i) validated scoring and (ii) standardized administration. We apply this new framework to the online use of the Patient Health Questionnaire-9 (PHQ-9), Generalized Anxiety Disorder-7 (GAD-7), and Alcohol Use Disorder Identification Test (AUDIT). Our systematic review of the literature identified 36 implementations of these three inventories on mTurk across 27 publications. We also evaluated methodological approaches to enhance data quality, e.g., the use of bot detection and attention check items. Of the 36 implementations, 23 reported the applied diagnostic scoring criteria and only 18 reported the specified symptom timeframe. None of the 36 implementations reported adaptations made in their digitization of the inventories. While recent reports attribute higher rates of mood, anxiety, and alcohol use disorders on mTurk to data quality, our findings indicate that this inflation may also relate to the assessment methods. We provide recommendations to enhance both data quality and fidelity to validated administration and scoring methods.

(Non-) impact of task experience on behavioral economic decision-making

Behavioral economic research has been widely conducted via crowdsourcing resources to evaluate novel task designs or pilot interventions. One under recognized and yet-to-be tested concern is the impact of non-naïvety (i.e., prior task exposure) on behavioral economic task performance. We evaluated the influence of non-naïvety on task performance in two popular areas of behavioral economic research: behavioral economic demand and delay discounting. Participants (N = 485) recruited using Amazon Mechanical Turk (mTurk) completed alcohol and soda purchase tasks and delay discounting tasks for monetary and alcohol outcomes. Equivalence of responding and effect sizes with clinical variables were compared based on prior task experience. Over one quarter of participants reported demand task experience (26.9%) and nearly half endorsed delay discounting task experience (48.6%). Statistically equivalent responding was observed for alcohol purchase task data with less-than-small effect size differences based on task experience (d = 0.01-0.13). Similar results were observed for a soda purchase task thereby supporting generalization to a non-alcohol commodity. Measures of convergent and discriminant validity for behavioral economic demand indicated medium-to-large and stimulus-specific effect sizes with little variation based on prior task exposure. Delay discounting for money and alcohol showed some sensitivity to prior task experience (i.e., less steep discounting for non-naïve participants), however these effects were attenuated after accounting for group differences in alcohol use. These findings support the fidelity of behavioral economic task outcomes and emphasize that participant non-naïvety in crowdsourcing settings may minimally impact performance on behavioral economic assays commonly used in behavioral and addiction science. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Quality control questions on Amazon's Mechanical Turk (MTurk): A randomized trial of impact on the USAUDIT, PHQ-9, and GAD-7

Crowdsourced psychological and other biobehavioral research using platforms like Amazon's Mechanical Turk (MTurk) is increasingly common - but has proliferated more rapidly than studies to establish data quality best practices. Thus, this study investigated whether outcome scores for three common screening tools would be significantly different among MTurk workers who were subject to different sets of quality control checks. We conducted a single-stage, randomized controlled trial with equal allocation to each of four study arms: Arm 1 (Control Arm), Arm 2 (Bot/VPN Check), Arm 3 (Truthfulness/Attention Check), and Arm 4 (Stringent Arm - All Checks). Data collection was completed in Qualtrics, to which participants were referred from MTurk. Subjects (n = 1100) were recruited on November 20-21, 2020. Eligible workers were required to claim U.S. residency, have a successful task completion rate > 95%, have completed a minimum of 100 tasks, and have completed a maximum of 10,000 tasks. Participants completed the US-Alcohol Use Disorders Identification Test (USAUDIT), the Patient Health Questionnaire (PHQ-9), and a screener for Generalized Anxiety Disorder (GAD-7). We found that differing quality control approaches significantly, meaningfully, and directionally affected outcome scores on each of the screening tools. Most notably, workers in Arm 1 (Control) reported higher scores than those in Arms 3 and 4 for all tools, and a higher score than workers in Arm 2 for the PHQ-9. These data suggest that the use, or lack thereof, of quality control questions in crowdsourced research may substantively affect findings, as might the types of quality control items.

Actual versus perceived infection rates of COVID-19: Impact on distress, behavior and disability

OBJECTIVE

Accurate threat appraisal is central to survival. In the case of the coronavirus pandemic, accurate threat appraisal is difficult due to incomplete medical knowledge as well as complex social factors (e.g., mixed public health messages). The purpose of this study was to evaluate the degree to which individuals accurately perceive COVID-19 infection rates and to explore the role of COVID-19 threat perception on emotional and behavioral responses both cross sectionally and prospectively.

METHODS

A community sample (N = 249) was assessed using online crowdsourcing and followed for one month. COVID-19 threat appraisal was compared with actual COVID-19 infection rates and deaths at the time of data collection in each participant's county and state. It was predicted that actual versus perceived COVID-19 infection rates would only be modestly associated. Relative to actual infection rates, perceived infection rates were hypothesized to be a better predictor of COVID-related behaviors, distress, and impairment.

RESULTS

Findings indicated that relative to actual infection, perceived infection was a better predictor of COVID-related outcomes cross sectionally and longitudinally. Interestingly, actual infection rates were negatively related to behaviors cross sectionally (e.g., less stockpiling). Prospectively, these variables interacted to predict avoidance behaviors over time such that the relationship between perceived infection and avoidance was stronger as actual infection increased.

CONCLUSIONS

These data suggest that perceived COVID-19 infection is significantly associated with COVID-related behaviors, distress and impairment whereas actual infection rates have a less important and perhaps even paradoxical influence on behavioral responses to the pandemic.

Barriers to emergency department usage during the COVID-19 pandemic

OBJECTIVE

The objective of this study was to determine the public's likelihood of being willing to use an emergency department (ED) for urgent/emergent illness during the coronavirus disease 2019 (COVID-19) pandemic.

METHODS

An institutional review board-approved, cross-sectional survey of a non-probability sample from Amazon Mechanical Turk was administered May 24-25, 2020. Change in self-reported willingness to use an ED before and during the pandemic (primary outcome) was assessed via McNemar's test; COVID-19 knowledge and perceptions were secondary outcomes.

RESULTS

There were 855 survey participants (466 [54.5%] male; 699 [81.8%] White; median age 39). Proportion reporting likelihood to use the ED pre-pandemic (71% [604/855]) decreased significantly during the pandemic (49% [417/855]; P < 0.001); those unlikely to visit the ED increased significantly during the pandemic (41% [347/855] vs 22% [417/855], P < 0.001). Participants were unlikely to use the ED during the pandemic if they were unlikely to use it pre-pandemic (adjusted odds ratio, 4.55; 95% confidence interval, 3.09-6.7) or correctly answered more COVID-19 knowledge questions (adjusted odds ratio, 1.37; 95% confidence interval, 1.17-1.60). Furthermore, 23.4% (n = 200) of respondents believed the pandemic was not a serious threat to society. Respondents with higher COVID-19 knowledge scores were more likely to view the pandemic as serious (odds ratio, 1.57; 95% confidence interval, 1.36-1.82).

CONCLUSIONS

This survey study investigated the public's willingness to use the ED during the COVID-19 pandemic. Only 49% of survey respondents were willing to visit the ED during a pandemic if they felt ill compared with 71% before the pandemic.

Leveraging crowdsourcing methods to collect qualitative data in addiction science: Narratives of non-medical prescription opioid, heroin, and fentanyl use

BACKGROUND

Online crowdsourcing methods have proved useful for studies of diverse designs in the behavioral and addiction sciences. The remote and online setting of crowdsourcing research may provide easier access to unique participant populations and improved comfort for these participants in sharing sensitive health or behavioral information. To date, few studies have evaluated the use of qualitative research methods on crowdsourcing platforms and even fewer have evaluated the quality of data gathered. The purpose of the present analysis was to document the feasibility and validity of using crowdsourcing techniques for collecting qualitative data among people who use drugs.

METHODS

Participants (N = 60) with a history of non-medical prescription opioid use with transition to heroin or fentanyl use were recruited using Amazon Mechanical Turk (mTurk). A battery of qualitative questions was included indexing beliefs and behaviors surrounding opioid use, transition pathways to heroin and/or fentanyl use, and drug-related contacts with structural institutions (e.g., health care, criminal justice).

RESULTS

Qualitative data recruitment was feasible as evidenced by the rapid sampling of a relatively large number of participants from diverse geographic regions. Computerized text analysis indicated high ratings of authenticity for the provided narratives. These authenticity percentiles were higher than the average of general normative writing samples as well as than those collected in experimental settings.

CONCLUSIONS

These findings support the feasibility and quality of qualitative data collected in online settings, broadly, and crowdsourced settings, specifically. Future work among people who use drugs may leverage crowdsourcing methods and the access to hard-to-sample populations to complement existing studies in the human laboratory and clinic as well as those using other digital technology methods.

Association Between Public Trust and Provider Specialty Among Physicians With Financial Conflicts of Interest

OBJECTIVES

To characterize public perception of physicians' conflicts of interest (COIs) across medical and surgical specialties.

PATIENTS AND METHODS

A cross-sectional 6-arm randomized survey of a nonprobability sample from Amazon's Mechanical Turk occurred on December 11 to 16, 2018. Survey respondents were randomly assigned to vignettes that varied the physician specialty with COI. The primary outcome was mean difference in Mayer Trust, and the secondary outcome included the proportion who desire to discontinue care.

RESULTS

There were 1729 of 1920 respondents who completed the experiment (90.1% completion rate). Respondents were male (52.5%; n=907), white (71.4%; n=1234), and between the ages of 25 and 44 years (70.9%; n=1227). Mean ± SD Mayer Trust across the 6 specialties was 3.7±.60, with the only between-specialty differences observed for psychiatry compared with the other specialties (F=5.4; P<.001). The median dollar amount that would affect respondents' trust in a physician was $5000 (interquartile range, $100-$100,000). A total of 75.1% (n=1298) of respondents desired COI information, with 41.6% (n=720) discontinuing care. Age older than 34 years (adjusted odds ratio [aOR], 0.7; 95%, CI, 0.49-0.99; P=.047), nonwhite race (aOR, 1.3; 95% CI, 1.02-1.6; P=.03), educational attainment of 4 or more years of college (aOR, 1.31; 95% CI, 1.05-1.6; P=.016), and physician specialty as a psychiatrist (aOR, 1.5; 95% CI, 1.03-2.2; P=.034) were predictors for discontinuing care.

CONCLUSION

Public COI disclosure is a common method for managing financial conflicts. Although survey respondents were more likely to discontinue care with a physician with COI, they will act on this knowledge of COI differently depending on the specialty of the physician. The finding that psychiatry is an outlier may be a chance finding that warrants further confirmation. Continued efforts to ensure best practices for disclosure are required.

Public Concern About Monitoring Twitter Users and Their Conversations to Recruit for Clinical Trials: Survey Study

Background

Social networks such as Twitter offer the clinical research community a novel opportunity for engaging potential study participants based on user activity data. However, the availability of public social media data has led to new ethical challenges about respecting user privacy and the appropriateness of monitoring social media for clinical trial recruitment. Researchers have voiced the need for involving users’ perspectives in the development of ethical norms and regulations.

Objective

This study examined the attitudes and level of concern among Twitter users and nonusers about using Twitter for monitoring social media users and their conversations to recruit potential clinical trial participants.

Methods

We used two online methods for recruiting study participants: the open survey was (1) advertised on Twitter between May 23 and June 8, 2017, and (2) deployed on TurkPrime, a crowdsourcing data acquisition platform, between May 23 and June 8, 2017. Eligible participants were adults, 18 years of age or older, who lived in the United States. People with and without Twitter accounts were included in the study.

Results

While nearly half the respondents—on Twitter (94/603, 15.6%) and on TurkPrime (509/603, 84.4%)—indicated agreement that social media monitoring constitutes a form of eavesdropping that invades their privacy, over one-third disagreed and nearly 1 in 5 had no opinion. A chi-square test revealed a positive relationship between respondents’ general privacy concern and their average concern about Internet research (P<.005). We found associations between respondents’ Twitter literacy and their concerns about the ability for researchers to monitor their Twitter activity for clinical trial recruitment (P=.001) and whether they consider Twitter monitoring for clinical trial recruitment as eavesdropping (P<.001) and an invasion of privacy (P=.003). As Twitter literacy increased, so did people’s concerns about researchers monitoring Twitter activity. Our data support the previously suggested use of the nonexceptionalist methodology for assessing social media in research, insofar as social media-based recruitment does not need to be considered exceptional and, for most, it is considered preferable to traditional in-person interventions at physical clinics. The expressed attitudes were highly contextual, depending on factors such as the type of disease or health topic (eg, HIV/AIDS vs obesity vs smoking), the entity or person monitoring users on Twitter, and the monitored information.

Conclusions

The data and findings from this study contribute to the critical dialogue with the public about the use of social media in clinical research. The findings suggest that most users do not think that monitoring Twitter for clinical trial recruitment constitutes inappropriate surveillance or a violation of privacy. However, researchers should remain mindful that some participants might find social media monitoring problematic when connected with certain conditions or health topics. Further research should isolate factors that influence the level of concern among social media users across platforms and populations and inform the development of more clear and consistent guidelines.

How one small text change in a study document can impact recruitment rates and follow-up completions

BACKGROUND

The validity and reliability of longitudinal research is highly dependent on the recruitment and retention of representative samples. Various strategies have been developed and tested for improving recruitment and follow-up rates into health-behavioural research, but few have examined the role of linguistic choices and study document readability on participation rates. This study examined the impact of one small text change, assigning an inappropriate or grade-8 reading level password for intervention access, on participation rates and attrition in an online alcohol intervention trial.

METHODS

Participants were recruited into an online alcohol intervention study using Amazon's Mechanical Turk via a multi-step recruitment process which required participants to log into a study portal using a pre-assigned password. Passwords were qualitatively coded as grade-8 and/or inappropriate for use within a professional setting. Separate logistic regressions examined which demographic, clinical characteristics, and password categorizations were most strongly associated with recruitment rates and follow-up completions.

RESULTS

Inappropriate passwords were a barrier for recruitment among participants with post-secondary education as compared to those with less education (p = 0.044), while grade-8 passwords appeared to significantly facilitate the completion of 6-month follow-ups (p = 0.005).

CONCLUSIONS

Altogether, these findings suggest that some linguistic choices may play an important role in recruitment, while others, such as readability, may have longer-term effects on follow-up rates and attrition. Possible explanations for the findings, as well as, sample selection biases during recruitment and follow-up are discussed. Limitations of the study are stated and recommendations for researchers are provided.

TRIAL REGISTRATION

ClinicalTrials.gov NCT02977026. Registered 27 Nov 2016.