VOIS: A framework for recording Voice Over Internet Surveys

Verbal data provide researchers insight beyond that offered by text-based responses, including tone, reasoning elaboration, and experienced difficulty, among other processes. Additionally, it offers a less cognitively taxing way for participants to provide long responses. Verbal data collection methods are found in a variety of fields, mostly conducted in lab-based settings or requiring specialized hardware. Restricting verbal protocols to lab-based settings can have several drawbacks, including smaller sample sizes, biased populations, reduced adoption, and incompatibility with potential social distancing requirements. No method currently exists for researchers to collect verbal data within major online survey collection platforms. The current paper offers a user-friendly approach for collecting verbal data online, where a researcher can copy and paste JavaScript code into the desired survey platform. By providing a framework that does not require any advanced programming ability, researchers can collect verbal data in a scalable way using familiar modalities.

Prioritizing Native Breast Skin Preservation or Scar Symmetry in Autologous Breast Reconstruction? Using Crowdsourcing to Assess Preference

Background

Recent literature on autologous breast reconstruction suggests that such factors as scar symmetry and skin paddle size impact patient preferences more than preservation of native breast skin. Since patient satisfaction with plastic surgery procedures can be largely influenced by beauty standards set by the general public, this study used a novel crowdsourcing method to evaluate laypeople's aesthetic preferences for different bilateral autologous breast reconstructions to determine the relative importance of scar and skin paddle symmetry and preservation of native skin.

Methods

Using Amazon's Mechanical Turk crowdsourcing marketplace, participants ranked images of reconstructions based on overall aesthetic appearance. Images were digitally modified to reflect 4 types of reconstruction: immediate (IR), delayed symmetric (DS), delayed asymmetric (DA), or mixed (MR).

Results

DS was ranked most favorably (1.74), followed by IR (1.95), DA (2.93), and MR (3.34). Friedman rank sum and pairwise tests showed statistical significance for comparisons of all 4 reconstruction types. Likert ratings were higher for IR than for DA reconstructions for skin quality (P = .002), scar visibility (P < .001), scar position (P < .001), and breast symmetry, shape, and position (P < .001). Ratings increased for all aesthetic factors following nipple-areolar-complex reconstruction (P < .001).

Conclusions

More symmetric breast scars were rated aesthetically higher than nonsymmetric scarring, and our participants preferred maintenance of scar symmetry over preservation of native breast skin. These findings are consistent with previous studies that surveyed non-crowdsourced participants, which demonstrates the potential for crowdsourcing to be used to better understand the general public's preferences in plastic surgery.

How much are patients willing to pay for spine surgery? An evaluation of attitudes toward out-of-pocket expenses and cost-reducing measures

BACKGROUND CONTEXT

With rising healthcare expenditures in the United States, patients and providers are searching to maintain quality while reducing costs.

PURPOSE

The aim of this study was to investigate patient willingness to pay for anterior cervical discectomy and fusion (ACDF), degenerative lumbar spinal fusions (LF), and adult spine deformity (ASD) surgery.

STUDY DESIGN/SETTING

A survey was developed and distributed to anonymous respondents through Amazon Mechanical Turk (MTurk).

METHODS

The survey introduced 3 procedures: ACDF, LF, and ASD surgery. Respondents were asked sequentially if they would pay at each increasing price option. Respondents were then presented with various cost-saving methods and asked to select the options that made them most uncomfortable, even if those would save them out-of-pocket costs.

RESULTS

In total, 979 of 1,172 total responses (84%) were retained for analysis. The average age was 36.2 years and 44% of participants reported a household income of $50,000 to 100,000. A total of 63% used Medicare and 13% used Medicaid. A total of 40% stated they had high levels of financial stress. A total of 30.1% of participants were willing to undergo an ACDF, 30.3% were willing to undergo a LF, and 29.6% were willing to undergo ASD surgery for the cost of $3,000 (p=.98). Regression demonstrated that for ACDF surgery, a $100 increase in price resulted in a 2.1% decrease in willingness to pay. This is comparable to degenerative LF surgery (1.8% decrease), and ASD surgery (2%). When asked which cost-saving measures participants were least comfortable with for ACDF surgery, 60% stated "Use of the older generation implants/devices" (LF: 51%, ASD: 60%,), 61% stated "Having the surgery performed at a community hospital instead of at a major academic center" (LF: 49%, ASD: 56%), and 55% stated "Administration of anesthesia by a nurse anesthetist" (LF: 48.01%, ASD: 55%). Conversely, 36% of ACDF patients were uncomfortable with a "Video/telephone postoperative visit" to cut costs (LF: 51%, ASD: 39%).

CONCLUSIONS

Patients are unwilling to contribute larger copays for adult spinal deformity correction than for ACDF and degenerative lumbar spine surgery, despite significantly higher procedural costs and case complexity/invasiveness. Patients were most uncomfortable forfeiting newer generation implants, receiving the operation at a community rather than an academic center, and receiving care by physician extenders. Conversely, patients were more willing to convert postoperative visits to telehealth and forgo neuromonitoring, indicating a potentially poor understanding of which cost-saving measures may be implemented without increasing the risk of complications.

Crowd-sourced investigation of a potential relationship between Bartonella-associated cutaneous lesions and neuropsychiatric symptoms

Introduction

Preliminary studies suggest that infection with Bartonella bacteria can not only cause a characteristic rash, headache, fever, and fatigue but also neuropsychiatric symptoms. To date, this association has only been reported in case studies, and it remains unclear if this association generalizes to larger samples.

Methods

We used Amazon's Mechanical Turk (MTurk) to crowdsource a large sample (N = 996) of individuals to ascertain the extent to which the presence of participant-identified Bartonella-associated cutaneous lesions (BACL) was associated with self-reported measures of anxiety, depression, and schizotypy. Participants were asked to select images of cutaneous lesions they had seen on their own bodies and complete a battery of self-report questionnaires to assess psychiatric symptoms. Participants were not informed that the focus of the study was on potential dermatological lesions associated with Bartonella. Point-biserial correlations were used to determine the potential relationship between selecting a BACL image and the severity of self-reported psychiatric symptoms.

Results

Scores of anxiety, depression, and schizotypy were positively and significantly correlated with selecting a BACL image. Furthermore, self-report scores of 10 or higher on the GAD-7 and PHQ-9, which represent the suggested clinical cutoffs for meeting criteria for a depressive or anxiety-related disorder, were also significantly associated with selecting a BACL image. Non-Bartonella-associated cutaneous legions were also significantly associated with self-reported measures of psychiatric symptoms.

Discussion

The current study broadens the link between the presence of BACL and the presence of psychiatric symptoms of anxiety, depression, and schizotypy and extends a potential relationship beyond the small sample sizes of previous case studies and case series. Further investigation is recommended to address limitations and expand on these findings.

Response Consistency of Crowdsourced Web-Based Surveys on Type 1 Diabetes

Although Amazon Mechanical Turk facilitates the quick surveying of a large sample from various demographic and socioeconomic backgrounds, it may not be an optimal platform for obtaining reliable diabetes-related information from the online type 1 diabetes population.

Effects of Excluding Those Who Report Having "Syndomitis" or "Chekalism" on Data Quality: Longitudinal Health Survey of a Sample From Amazon's Mechanical Turk

BACKGROUND

Researchers have implemented multiple approaches to increase data quality from existing web-based panels such as Amazon's Mechanical Turk (MTurk).

OBJECTIVE

This study extends prior work by examining improvements in data quality and effects on mean estimates of health status by excluding respondents who endorse 1 or both of 2 fake health conditions ("Syndomitis" and "Chekalism").

METHODS

Survey data were collected in 2021 at baseline and 3 months later from MTurk study participants, aged 18 years or older, with an internet protocol address in the United States, and who had completed a minimum of 500 previous MTurk "human intelligence tasks." We included questions about demographic characteristics, health conditions (including the 2 fake conditions), and the Patient Reported Outcomes Measurement Information System (PROMIS)-29+2 (version 2.1) preference-based score survey. The 3-month follow-up survey was only administered to those who reported having back pain and did not endorse a fake condition at baseline.

RESULTS

In total, 15% (996/6832) of the sample endorsed at least 1 of the 2 fake conditions at baseline. Those who endorsed a fake condition at baseline were more likely to identify as male, non-White, younger, report more health conditions, and take longer to complete the survey than those who did not endorse a fake condition. They also had substantially lower internal consistency reliability on the PROMIS-29+2 scales than those who did not endorse a fake condition: physical function (0.69 vs 0.89), pain interference (0.80 vs 0.94), fatigue (0.80 vs 0.92), depression (0.78 vs 0.92), anxiety (0.78 vs 0.90), sleep disturbance (-0.27 vs 0.84), ability to participate in social roles and activities (0.77 vs 0.92), and cognitive function (0.65 vs 0.77). The lack of reliability of the sleep disturbance scale for those endorsing a fake condition was because it includes both positively and negatively worded items. Those who reported a fake condition reported significantly worse self-reported health scores (except for sleep disturbance) than those who did not endorse a fake condition. Excluding those who endorsed a fake condition improved the overall mean PROMIS-29+2 (version 2.1) T-scores by 1-2 points and the PROMIS preference-based score by 0.04. Although they did not endorse a fake condition at baseline, 6% (n=59) of them endorsed at least 1 of them on the 3-month survey and they had lower PROMIS-29+2 score internal consistency reliability and worse mean scores on the 3-month survey than those who did not report having a fake condition. Based on these results, we estimate that 25% (1708/6832) of the MTurk respondents provided careless or dishonest responses.

CONCLUSIONS

This study provides evidence that asking about fake health conditions can help to screen out respondents who may be dishonest or careless. We recommend this approach be used routinely in samples of members of MTurk.

Women's preferences for a new contraceptive under development: an exploratory study

Objective

Currently available contraceptive methods do not meet the needs of all users. We sought to explore preferences of potential end-users regarding an on-demand, non-hormonal female contraceptive currently under development, using a web-based survey.

Study design

We recruited respondents for an exploratory survey via web link on Amazon Mechanical Turk (MTurk). Individuals were eligible if they were 18-44 years of age, identified as cis-gender female, were English-speaking, not pregnant, and had used barrier contraception previously. Respondents provided demographic characteristics and a basic reproductive history. We then provided a brief description of the potential contraceptive. Respondents were asked about their interest in the proposed contraceptive and preferences for method attributes.

Results

A total of 500 respondents completed the survey. Three-quarters of respondents were <35 years of age and 48.2% were currently using a barrier contraceptive method. Three-fourths of respondents (73.8%) expressed interest in using the contraceptive under development. The majority wanted the method to be small (≤2 inches), rod-shaped, and low cost (<$5 per use). More than half (59.4%) said it was important to be able to use the method without partners' knowledge. The most reported potential concerns were vaginal irritation (51.6%) and lack of effectiveness (46.4%). Sixty percent of respondents were confident they could use the method correctly.

Discussion

Available contraceptive methods lack attributes preferred by some users. Development of new contraceptives frequently does not involve end-user input early in the development process. Individuals in this sample displayed interest in the proposed contraceptive and expressed preferences that can inform the further development of this method.

A survey study of adults with chronic conditions: Examining the correlation between patient activation and health locus of control

OBJECTIVES

This study aimed to examine (1) the association between patient activation (PA), health locus of control (HLOC), sociodemographic and clinical factors, and (2) the effect of HLOC dimensions, sociodemographic and clinical factors on PA.

METHODS

Three hundred U.S. adults, with at least one chronic condition (CC) were recruited through Amazon Mechanical Turk and completed an online survey which included sociodemographic questions, the Patient Activation Measure^® - 10, and the Multidimensional Locus of Control (MHLC) - Form B. Statistical analyses, including descriptive, correlation, and multiple linear regression, were conducted using IBM SPSS v25.

RESULTS

Of the 300 participants, more than half were male (66.3%), White (70.7%), with at least a college degree (76.0%), and employed full-time (79.0%). The average PA score was 68.8 ± 14.5. Multiple linear regression indicated that participants who reported they were Black, retired, with a greater number of CCs, and with higher scores in Chance MHLC had higher PA, while participants with higher scores in Internal MHLC, were unemployed and reported to have been affected by COVID-19-related worry or fear to manage their CC, had lower PA.

DISCUSSION

HLOC dimensions should be addressed concurrently with PA for patients with CCs, thus adding to a more patient-centered clinical approach.

Gig Work and the Pandemic: Looking for Good Pay from Bad Jobs During the COVID-19 Crisis

COVID-19 led to work hour reductions and layoffs for many Americans with wage/salary jobs. Some gig work, however, which is usually considered precarious, remained available. We examine whether people doing gig microtasks right before the pandemic increased their microtask hours during COVID-19 and whether those changes helped them financially. Using data from workers on Amazon's Mechanical Turk platform from February, March, and April of 2020, we find that roughly one third of existing workers increased their microtask hours. Increases were larger for people who lost household income or wage/salary hours. Spending more time on microtasks, however, did little to help workers financially. Furthermore, the people most reliant on microtasks before the pandemic had worse financial outcomes than others. In short, even though microtask work might seem like a good way for people to recoup lost income during the pandemic, it was of limited utility even for the experienced workers in our sample.

Using Young Adult Language to Describe the Effects of Simultaneous Alcohol and Marijuana Use: Implications for Assessment

Introduction: Prevalence of alcohol and marijuana use is highest in young adulthood and an increasing number of young adults report simultaneous alcohol and marijuana (SAM) use, which is consistently linked with numerous negative consequences. To better understand reasons for engaging in SAM use and to refine measurement of subjective effects of SAM use, this study aimed to identify (1) how young adults describe subjective experiences during a SAM use occasion and (2) how language describing subjective effects changes as a function of level of alcohol and marijuana use. Methods: Using Amazon's Mechanical Turk (MTurk), 323 participants (53.6% women, 68.4% White, M age = 23.0 years) who reported past-month heavy episodic drinking and past-month SAM use were asked to list words to describe how they feel when using only alcohol, only marijuana, and various combinations of alcohol and marijuana. Results: SAM use language varied as a function of age and substance use behavior but was not associated with sex or race. Large differences in the terms used to describe subjective effects were observed when comparing different combinations of alcohol and marijuana use; most notably the term "cross-faded" appeared primarily when engaging at the heaviest combinations of alcohol and marijuana. Conclusion: Young adults have a wide range of vocabulary for describing subjective effects of SAM use, and subjective effects vary as a function of the level of each substance used. Future research should consider integrating such contemporary language when measuring subjective effects of SAM use.

The Benefits of Crowdsourcing to Seed and Align an Algorithm in an mHealth Intervention for African American and Hispanic Adults: Survey Study

BACKGROUND

The lack of publicly available and culturally relevant data sets on African American and bilingual/Spanish-speaking Hispanic adults' disease prevention and health promotion priorities presents a major challenge for researchers and developers who want to create and test personalized tools built on and aligned with those priorities. Personalization depends on prediction and performance data. A recommender system (RecSys) could predict the most culturally and personally relevant preventative health information and serve it to African American and Hispanic users via a novel smartphone app. However, early in a user's experience, a RecSys can face the "cold start problem" of serving untailored and irrelevant content before it learns user preferences. For underserved African American and Hispanic populations, who are consistently being served health content targeted toward the White majority, the cold start problem can become an example of algorithmic bias. To avoid this, a RecSys needs population-appropriate seed data aligned with the app's purposes. Crowdsourcing provides a means to generate population-appropriate seed data.

OBJECTIVE

Our objective was to identify and test a method to address the lack of culturally specific preventative personal health data and sidestep the type of algorithmic bias inherent in a RecSys not trained in the population of focus. We did this by collecting a large amount of data quickly and at low cost from members of the population of focus, thereby generating a novel data set based on prevention-focused, population-relevant health goals. We seeded our RecSys with data collected anonymously from self-identified Hispanic and self-identified non-Hispanic African American/Black adult respondents, using Amazon Mechanical Turk (MTurk).

METHODS

MTurk provided the crowdsourcing platform for a web-based survey in which respondents completed a personal profile and a health information-seeking assessment, and provided data on family health history and personal health history. Respondents then selected their top 3 health goals related to preventable health conditions, and for each goal, reviewed and rated the top 3 information returns by importance, personal utility, whether the item should be added to their personal health library, and their satisfaction with the quality of the information returned. This paper reports the article ratings because our intent was to assess the benefits of crowdsourcing to seed a RecSys. The analysis of the data from health goals will be reported in future papers.

RESULTS

The MTurk crowdsourcing approach generated 985 valid responses from 485 (49%) self-identified Hispanic and 500 (51%) self-identified non-Hispanic African American adults over the course of only 64 days at a cost of US $6.74 per respondent. Respondents rated 92 unique articles to inform the RecSys.

CONCLUSIONS

Researchers have options such as MTurk as a quick, low-cost means to avoid the cold start problem for algorithms and to sidestep bias and low relevance for an intended population of app users. Seeding a RecSys with responses from people like the intended users allows for the development of a digital health tool that can recommend information to users based on similar demography, health goals, and health history. This approach minimizes the potential, initial gaps in algorithm performance; allows for quicker algorithm refinement in use; and may deliver a better user experience to individuals seeking preventative health information to improve health and achieve health goals.

Automated Bot Detection Using Bayesian Latent Class Models in Online Surveys

Behavioral scientists have become increasingly reliant on online survey platforms such as Amazon's Mechanical Turk (Mturk). These platforms have many advantages, for example it provides ease of access to difficult to sample populations, a large pool of participants, and an easy to use implementation. A major drawback is the existence of bots that are used to complete online surveys for financial gain. These bots contaminate data and need to be identified in order to draw valid conclusions from data obtained with these platforms. In this article, we will provide a Bayesian latent class joint modeling approach that can be routinely applied to identify bots and simultaneously estimate a model of interest. This method can be used to separate the bots' response patterns from real human responses that were provided in line with the item content. The model has the advantage that it is very flexible and is based on plausible assumptions that are met in most empirical settings. We will provide a simulation study that investigates the performance of the model under several relevant scenarios including sample size, proportion of bots, and model complexity. We will show that ignoring bots will lead to severe parameter bias whereas the Bayesian latent class model results in unbiased estimates and thus controls this source of bias. We will illustrate the model and its capabilities with data from an empirical political ideation survey with known bots. We will discuss the implications of the findings with regard to future data collection via online platforms.

(2021)

We respond to commentaries on our 2021 paper "Concerns and recommendations for using Amazon MTurk for eating disorder research." The commentators raised many thoughtful and nuanced points regarding data validity and ethical means of online data collection. We echo concerns about the ethics of recruiting via platforms such as MTurk, and highlight tensions between recommendations for ethical data collection and ensuring data integrity. Especially, we highlight the consistent finding that MTurk workers display elevated (often remarkably so) rates of psychopathology, and argue such findings merit further scrutiny to ensure both data are valid and workers not exploited.

(2021)

Burnette et al. reported a study that they sought to undertake to validate common eating disorder questionnaires in sexual and gender minorities. The researchers took advantage of the online recruitment platform Amazon Mechanical Turk (MTurk). Contrary to their expectations, the study proved not feasible due to invalid answering. Thus, Burnette et al. raise concerns against the trustworthiness of crowd-sourced data that may be undermined by financial interests and other kinds of motivations. Our commentary highlights the potential of the COVID-19 pandemic to inflate especially those intentions, which are monetary. Against the background of the COVID-19 pandemic, a further problem seems to be that the anonymity of online crowd sourcing platforms might tempt participants to provide inconsistent answers, possibly reflecting tendencies of reactance. The reported pattern of paradoxical responses in Burnette et al.'s work does not reflect malingering; rather we believe that the study might have served some participants as an outlet for negative emotions. We discuss mechanisms of quality control and highlight the lack of interpersonal interaction associated with online data collections.

Concerns and recommendations for using Amazon MTurk for eating disorder research

OBJECTIVE

Our original aim was to validate and norm common eating disorder (ED) symptom measures in a large, representative community sample of transgender adults in the United States. We recruited via Amazon Mechanical Turk (MTurk), a popular online recruitment and data collection platform both within and outside of the ED field. We present an overview of our experience using MTurk.

METHOD

Recruitment began in Spring 2020; our original target N was 2,250 transgender adults stratified evenly across the United States. Measures included a demographics questionnaire, the Eating Disorder Examination-Questionnaire, and the Eating Attitudes Test-26. Consistent with current literature recommendations, we implemented a comprehensive set of attention and validity measures to reduce and identify bot responding, data farming, and participant misrepresentation.

RESULTS

Recommended validity and attention checks failed to identify the majority of likely invalid responses. Our collection of two similar ED measures, thorough weight history assessment, and gender identity experiences allowed us to examine response concordance and identify impossible and improbable responses, which revealed glaring discrepancies and invalid data. Furthermore, qualitative data (e.g., emails received from MTurk workers) raised concerns about economic conditions facing MTurk workers that could compel misrepresentation.

DISCUSSION

Our results strongly suggest most of our data were invalid, and call into question results of recently published MTurk studies. We assert that caution and rigor must be applied when using MTurk as a recruitment tool for ED research, and offer several suggestions for ED researchers to mitigate and identify invalid data.

Economic anxiety among contingent survey workers

Psychologists and other social scientists increasingly conduct experiments with online convenience samples from Amazon's Mechanical Turk Marketplace (MTurk). MTurk and population-based samples differ in well-documented ways, but whether or not compositional differences are problematic for experiments remains controversial. We highlight a critically important characteristic that is likely to interact with many experimental treatments in the psychological and behavioral sciences, and that has not been identified by other studies of MTurk samples: economic anxiety. We document a sizable difference between contingent survey workers and the general population and explain the ways in which economic anxiety is likely to interact with experimental treatments. In an era of rapidly growing economic anxiety and group disparities in economic wellbeing, awareness of this compositional difference is essential, especially in cases where experimental stimuli may interact with economic anxiety.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s12144-021-02535-4.

Reducing Internalized Homonegativity: Refinement and Replication of an Online Intervention for Gay Men

We refined and replicated an efficacious brief intervention to reduce internalized homonegativity (IH) with a sample of gay and exclusively same-sex attracted men recruited from outside of LGBT community networks using Amazon Mechanical Turk. We sought to 1) determine if levels of IH differed between the original study's community-based sample and our non-community-based sample, 2) examine the efficacy of the replicated intervention, and 3) assess for longitudinal effects of the intervention at a 30-day follow-up. Four hundred eighty-four participants completed either the intervention or a stress management control condition. Mean levels of IH were higher in the current sample compared with the earlier study's community sample. The intervention was efficacious at reducing global IH, reducing personal homonegativity, and increasing gay affirmation. Ninety-six participants completed the follow-up; follow-up results were not significant and may have been affected by high rates of attrition. Implications for research and practice are discussed.

Cognitive Interference in the Context of Daily Stressors, Daily Awareness of Age-Related Change, and General Aging Attitudes

OBJECTIVES

Previous diary work indicates that older people experience more intrusive and unwanted thoughts (i.e., cognitive interference) on days with stressors. We examined additional predictors of daily cognitive interference to enhance understanding of the psychological context surrounding this link. We specifically focused on factors related to subjective experiences of aging based on studies that have related higher stress and impairments in cognition such as executive control processes (working memory) to negative age stereotypes. Consistent with these findings, we generally expected stronger stress effects on cognitive interference when daily self-perceptions of aging (i.e., within-person fluctuations in awareness of age-related losses [AARC losses]) and general aging attitudes (i.e., individual differences in attitudes toward own aging [ATOA]) were more negative.

METHODS

Participants (n = 91; aged 60-80) on Amazon's Mechanical Turk completed surveys on 9 consecutive days, reporting on their ATOA (Day 1) as well as their stressors, AARC losses, and cognitive interference (Days 2-9).

RESULTS

Multilevel models showed that people reported more cognitive interference on days with more AARC losses. Individuals with positive ATOA also experienced less cognitive interference on days with more stressors, whereas those with negative ATOA experienced more.

DISCUSSION

Both individual differences and fluctuating daily perceptions of aging appear to be important for older adults' cognitive interference. Consistent with other work, positive ATOA protected against daily stressor effects. Further elucidating these relationships can increase understanding of and facilitate efforts to improve (daily) cognitive experiences in older adults.

Quality control questions on Amazon's Mechanical Turk (MTurk): A randomized trial of impact on the USAUDIT, PHQ-9, and GAD-7

Crowdsourced psychological and other biobehavioral research using platforms like Amazon's Mechanical Turk (MTurk) is increasingly common - but has proliferated more rapidly than studies to establish data quality best practices. Thus, this study investigated whether outcome scores for three common screening tools would be significantly different among MTurk workers who were subject to different sets of quality control checks. We conducted a single-stage, randomized controlled trial with equal allocation to each of four study arms: Arm 1 (Control Arm), Arm 2 (Bot/VPN Check), Arm 3 (Truthfulness/Attention Check), and Arm 4 (Stringent Arm - All Checks). Data collection was completed in Qualtrics, to which participants were referred from MTurk. Subjects (n = 1100) were recruited on November 20-21, 2020. Eligible workers were required to claim U.S. residency, have a successful task completion rate > 95%, have completed a minimum of 100 tasks, and have completed a maximum of 10,000 tasks. Participants completed the US-Alcohol Use Disorders Identification Test (USAUDIT), the Patient Health Questionnaire (PHQ-9), and a screener for Generalized Anxiety Disorder (GAD-7). We found that differing quality control approaches significantly, meaningfully, and directionally affected outcome scores on each of the screening tools. Most notably, workers in Arm 1 (Control) reported higher scores than those in Arms 3 and 4 for all tools, and a higher score than workers in Arm 2 for the PHQ-9. These data suggest that the use, or lack thereof, of quality control questions in crowdsourced research may substantively affect findings, as might the types of quality control items.

Renewal and resurgence phenomena generalize to Amazon's Mechanical Turk

Amazon's Mechanical Turk (MTurk) is a crowdsourcing platform that provides researchers with the potential for obtaining behavioral data for very little cost. However, the extent to which the results of common behavioral phenomena found in basic, translational, and applied laboratories may be reproduced (as a first step towards prospective research) via MTurk remains relatively unexplored. We evaluated renewal and resurgence arrangements using MTurk as the subject recruitment platform as a first step to determining the generality of the obtained data. Results suggested that MTurk participants produced renewal and resurgence data similar to those reported in basic, translational, and applied studies.

Utility estimation for neurogenic bowel dysfunction in the general population

BACKGROUND

Neurogenic bowel dysfunction (NBD) affects over 80% of individuals with spina bifida causing bowel incontinence and/or constipation. NBD is also associated with decreased quality of life, depression, anxiety, and decreased employment/educational attainment. Because NBD is a life-altering condition without a cure, understanding the utility of different health states related to NBD would aid clinicians as they try to counsel families regarding management options and to better understand the quality of life associated with disease management.

OBJECTIVE

To elicit utility scores for NBD using an online community sample.

STUDY DESIGN

A cross-sectional anonymous survey was completed by 1534 voluntary participants via an online platform (Amazon Mechanical Turk (MTurk, http://www.mturk.com/)), representing an 87% response rate. The survey presented hypothetical scenarios that asked respondents to imagine themselves as an individual living with NBD or as the caretaker of a child with NBD. The time trade-off (TTO) method was used to estimate a utility score, and outcomes for each scenario were calculated using median and IQR. Univariate comparisons of distributions of TTO for demographic data were made using Kruskal-Wallis tests.

RESULTS

The median utility score for NBD was 0.84 [0.70-0.92]. Participants reported that they would give up a median of 5 years of their own life, to prevent NBD in themselves of their child. Utility values for child scenarios were significantly different when stratified by age, gender, race, parental status, marital status, and income. Stratification by current health status did not yield significantly different utility values.

DISCUSSION

Study findings are comparable with other TTO-determined utility values of moderately severe disease states, including severe persistent asthma (0.83), moderate seizure disorder (0.84) and mild mental retardation (0.84). The significant variations in utility values based on age, gender, race, parent status, partner/marital status and income variables existed in our study, which is similar to findings in other health fields. Study limitations include lack of unanimous agreement about TTO's validity in measuring utility values, and MTurk participant reports can be generalized to greater population.

CONCLUSION

NBD is perceived by the community as having a substantial impact on the lives of children with spina bifida, representing a 16% reduction from perfect health. In general, health state utilities have been increasingly used in healthcare systems to understand how burdensome a population perceives a disease is and to evaluate whether interventions improve quality of life years.

Rise in Use of Digital Mental Health Tools and Technologies in the United States During the COVID-19 Pandemic: Survey Study

BACKGROUND

Accompanying the rising rates of reported mental distress during the COVID-19 pandemic has been a reported increase in the use of digital technologies to manage health generally, and mental health more specifically.

OBJECTIVE

The objective of this study was to systematically examine whether there was a COVID-19 pandemic-related increase in the self-reported use of digital mental health tools and other technologies to manage mental health.

METHODS

We analyzed results from a survey of 5907 individuals in the United States using Amazon Mechanical Turk (MTurk); the survey was administered during 4 week-long periods in 2020 and survey respondents were from all 50 states and Washington DC. The first set of analyses employed two different logistic regression models to estimate the likelihood of having symptoms indicative of clinical depression and anxiety, respectively, as a function of the rate of COVID-19 cases per 10 people and survey time point. The second set employed seven different logistic regression models to estimate the likelihood of using seven different types of digital mental health tools and other technologies to manage one's mental health, as a function of symptoms indicative of clinical depression and anxiety, rate of COVID-19 cases per 10 people, and survey time point. These models also examined potential interactions between symptoms of clinical depression and anxiety, respectively, and rate of COVID-19 cases. All models controlled for respondent sociodemographic characteristics and state fixed effects.

RESULTS

Higher COVID-19 case rates were associated with a significantly greater likelihood of reporting symptoms of depression (odds ratio [OR] 2.06, 95% CI 1.27-3.35), but not anxiety (OR 1.21, 95% CI 0.77-1.88). Survey time point, a proxy for time, was associated with a greater likelihood of reporting clinically meaningful symptoms of depression and anxiety (OR 1.19, 95% CI 1.12-1.27 and OR 1.12, 95% CI 1.05-1.19, respectively). Reported symptoms of depression and anxiety were associated with a greater likelihood of using each type of technology. Higher COVID-19 case rates were associated with a significantly greater likelihood of using mental health forums, websites, or apps (OR 2.70, 95% CI 1.49-4.88), and other health forums, websites, or apps (OR 2.60, 95% CI 1.55-4.34). Time was associated with increased odds of reported use of mental health forums, websites, or apps (OR 1.20, 95% CI 1.11-1.30), phone-based or text-based crisis lines (OR 1.20, 95% CI 1.10-1.31), and online, computer, or console gaming/video gaming (OR 1.12, 95% CI 1.05-1.19). Interactions between COVID-19 case rate and mental health symptoms were not significantly associated with any of the technology types.

CONCLUSIONS

Findings suggested increased use of digital mental health tools and other technologies over time during the early stages of the COVID-19 pandemic. As such, additional effort is urgently needed to consider the quality of these products, either by ensuring users have access to evidence-based and evidence-informed technologies and/or by providing them with the skills to make informed decisions around their potential efficacy.

PROFILES OF ALCOHOL AND MARIJUANA USE AMONG SIMULTANEOUS ALCOHOL AND MARIJUANA USERS: INDIVIDUAL DIFFERENCES IN DEMOGRAPHICS AND SUBSTANCE USE

Simultaneous alcohol and marijuana (SAM) use - or use of both substances with overlapping effects - is common among emerging adults and is linked to increased risk for problematic substance use outcomes. The current study identified subgroups of emerging adult SAM users based on their typical alcohol and marijuana use patterns and compared groups on key individual characteristics. Latent profile analysis uncovered four profiles of SAM users (n=522): Light Users (LU; 49.0%), Moderate Drinkers with Frequent Marijuana Use (MDFM; 37.9%), Moderate Drinkers with High Peak Levels (MDHP; 5.4%), and Heavy/Frequent Users (HFU; 7.7%). Group differences by demographic characteristics were found, with LU more likely to be college attendees/graduates than MDFM. Additionally, HFU were more likely to be Greek-affiliated than both LU and MDFM. Groups also differed based on other drug use behavior and preferred route of marijuana administration. Findings demonstrate diversity among SAM users based on typical substance use patterns.

Willingness to Use Cannabis for Gynecological Conditions: A National Survey

Objective: Expanded legal access to cannabis in the United States has led to its increased use for treating medical conditions. We assessed the use of and attitudes toward cannabis for treating gynecological conditions. Materials and Methods: We utilized Amazon.com Inc.'s Mechanic Turk platform to administer a survey to U.S. women 18 years and older about cannabis use for recreational and medicinal purposes and willingness to use cannabis to treat 17 gynecological conditions. We collected sociodemographic data and views about the legal status of cannabis. We used logistic regression to identify factors associated with willingness to use cannabis for gynecological conditions. Results: In our analytical sample (N = 995), women who reported ever using cannabis were more willing to use cannabis to treat a gynecological condition compared with never users (91.6% vs. 64.6%, p < 0.01). Women willing to use cannabis for gynecological conditions were most interested in using cannabis for treating gynecological pain (61.2% of never users vs. 90.0% of ever users; p < 0.001) compared with procedural pain (38.2% vs. 79.0%, respectively; p < 0.001) or other conditions (38.0% vs. 79.8%, respectively; p < 0.001). In multivariate analysis, willingness to use cannabis for a gynecological condition was associated only with a history of ever using cannabis and views that cannabis should be legal in some capacity and not by age, race, or education. Conclusions: The majority of women would consider using cannabis to treat gynecological conditions. Overall, respondents who had a history of cannabis use were more likely to report willingness to use cannabis for all gynecological conditions, but a large proportion of those who reported never using cannabis were also willing to use it.

An Exploration of Stress: Leveraging Online Data from Crowdsourcing Platforms

Background: Early detection of community health risk factors such as stress is of great interest to health policymakers, but representative data collection is often expensive and time-consuming. It is important to investigate the use of alternative means of data collection such as crowdsourcing platforms.

Methods: An online sample of Amazon Mechanical Turk (MTurk) workers (N = 500) filled out, for themselves and their child, demographic information and the 10-item Perceived Stress Scale (PSS-10), designed to measure the degree to which situations in one’s life are appraised as stressful. Internal consistency reliability of the PSS-10 was examined via Cronbach’s alpha. Analysis of variance (ANOVA) was utilized to explore trends in the average perceived stress of both adults and their children. Last, Rasch trees were utilized to detect differential item functioning (DIF) in the set of PSS-10 items.

Results: The PSS-10 showed adequate internal consistency reliability (Cronbach’s alpha = 0.73). ANOVA results suggested that stress scores significantly differed by education (p = 0.024), employment status (p = 0.0004), and social media usage (p = 0.015). Rasch trees, a recursive partitioning technique based on the Rasch model, indicated that items on the PSS-10 displayed DIF attributable to physical health for adults and social media usage for children.

Conclusion: The key conclusion is that this data collection scheme shows promise, allowing public health officials to examine health risk factors such as perceived stress quickly and cost effectively.

Realistic precision and accuracy of online experiment platforms, web browsers, and devices

Due to increasing ease of use and ability to quickly collect large samples, online behavioural research is currently booming. With this popularity, it is important that researchers are aware of who online participants are, and what devices and software they use to access experiments. While it is somewhat obvious that these factors can impact data quality, the magnitude of the problem remains unclear. To understand how these characteristics impact experiment presentation and data quality, we performed a battery of automated tests on a number of realistic set-ups. We investigated how different web-building platforms (Gorilla v.20190828, jsPsych v6.0.5, Lab.js v19.1.0, and psychoJS/PsychoPy3 v3.1.5), browsers (Chrome, Edge, Firefox, and Safari), and operating systems (macOS and Windows 10) impact display time across 30 different frame durations for each software combination. We then employed a robot actuator in realistic set-ups to measure response recording across the aforementioned platforms, and between different keyboard types (desktop and integrated laptop). Finally, we analysed data from over 200,000 participants on their demographics, technology, and software to provide context to our findings. We found that modern web platforms provide reasonable accuracy and precision for display duration and manual response time, and that no single platform stands out as the best in all features and conditions. In addition, our online participant analysis shows what equipment they are likely to use.

Evaluating MTurk as a recruitment tool for rural people with disabilities

BACKGROUND

Recruitment of people with disabilities often occurs through disability organizations, advocacy groups, service providers, and patient registries. Recruitment that relies exclusively on established relationships can produce samples that may miss important information. The MTurk online marketplace offers a convenient option for recruitment.

OBJECTIVE

The paper compares samples recruited through (1) conventional and (2) MTurk methods to better understand how these samples contrast with one another and with national estimates of people with disabilities.

METHODS

In 2019, researchers recruited 1374 participants through conventional methods and 758 through MTurk to complete the National Survey on Health and Disability (NSHD). We analyzed sample differences between recruitment groups with t-tests, Chi-square, and logistic regression.

RESULTS

With the exception of race/ethnicity, the conventional and MTurk samples were significantly different on several dimensions including age, gender, education level, marital status, children living at home, and sexual orientation. The MTurk sample was overrepresented in lower income brackets. A significantly higher percentage of the conventional sample received SSI, SSDI, or both, compared to the MTurk sample (36.2% vs 12.8%) and had significantly higher rates of insurance coverage. Comparisons with American Community Survey data show that the conventional and MTurk samples aligned more closely with the general population of people with disabilities on different characteristics.

CONCLUSIONS

MTurk is a viable complement to conventional recruitment methods, but it should not be a replacement. A combination of strategies builds a more robust dataset that allows for more varied examination of issues relevant to people with disabilities.

Examining the effectiveness of an online program to cultivate mindfulness and self-compassion skills (Mind-OP): Randomized controlled trial on Amazon's Mechanical Turk

OBJECTIVES

The demand for effective psychological treatments for depression, anxiety, and heightened stress is far outstripping their supply. Accordingly, internet delivered, self-help interventions offer hope to many people, as they can be easily accessed and at a fraction of the price of face-to-face options. Mindfulness and self-compassion are particularly exciting approaches, as evidence suggests interventions that cultivate these skills are effective in reducing depression, anxiety, and heightened stress. We examined the effectiveness of a newly developed program that combines mindfulness, self-compassion, and goal-setting exercises into a brief self-guided intervention (Mind-OP). The secondary aim of this study was to investigate the feasibility of conducting a randomized-controlled trial entirely on a popular crowdsourcing platform, Amazon's Mechanical Turk (MTurk).

METHODS

We randomized 456 participants reporting heightened depression, anxiety, or stress to one of two conditions: the 4-week Mind-OP intervention (n = 227) or to an active control condition (n = 229) where participants watched nature videos superimposed onto relaxing meditation music for four consecutive weeks. We administered measures of anxiety, depression, perceived stress, dispositional and state mindfulness, self-compassion, and nonattachment.

RESULTS

Intent-to-treat and per-protocol analyses revealed that, compared to participants in the control condition, participants in the Mind-OP intervention condition reported significantly less anxiety and stress at the end of the trial, as well as significantly greater mindfulness, self-compassion, and nonattachment.

CONCLUSIONS

Mind-OP appears effective in reducing anxiety symptoms and perceived stress among MTurk participants. We highlight issues (e.g., attrition) related to feasibility of conducting randomized trials on crowdsourcing platforms such as MTurk.

How Do Older Adults Recruited Using MTurk Differ From Those in a National Probability Sample?

A growing number of studies within the field of gerontology have included samples recruited from Amazon's Mechanical Turk (MTurk), an online crowdsourcing portal. While some research has examined how younger adult participants recruited through other means may differ from those recruited using MTurk, little work has addressed this question with older adults specifically. In the present study, we examined how older adults recruited via MTurk might differ from those recruited via a national probability sample, the Health and Retirement Study (HRS), on a battery of outcomes related to health and cognition. Using a Latin-square design, we examined the relationship between recruitment time, remuneration amount, and measures of cognitive functioning. We found substantial differences between our MTurk sample and the participants within the HRS, most notably within measures of verbal fluency and analogical reasoning. Additionally, remuneration amount was related to differences in time to complete recruitment, particularly at the lowest remuneration level, where recruitment completion required between 138 and 485 additional hours. While the general consensus has been that MTurk samples are a reasonable proxy for the larger population, this work suggests that researchers should be wary of overgeneralizing research conducted with older adults recruited through this portal.

The timing mega-study: comparing a range of experiment generators, both lab-based and online

Many researchers in the behavioral sciences depend on research software that presents stimuli, and records response times, with sub-millisecond precision. There are a large number of software packages with which to conduct these behavioral experiments and measure response times and performance of participants. Very little information is available, however, on what timing performance they achieve in practice. Here we report a wide-ranging study looking at the precision and accuracy of visual and auditory stimulus timing and response times, measured with a Black Box Toolkit. We compared a range of popular packages: PsychoPy, E-Prime®, NBS Presentation®, Psychophysics Toolbox, OpenSesame, Expyriment, Gorilla, jsPsych, Lab.js and Testable. Where possible, the packages were tested on Windows, macOS, and Ubuntu, and in a range of browsers for the online studies, to try to identify common patterns in performance. Among the lab-based experiments, Psychtoolbox, PsychoPy, Presentation and E-Prime provided the best timing, all with mean precision under 1 millisecond across the visual, audio and response measures. OpenSesame had slightly less precision across the board, but most notably in audio stimuli and Expyriment had rather poor precision. Across operating systems, the pattern was that precision was generally very slightly better under Ubuntu than Windows, and that macOS was the worst, at least for visual stimuli, for all packages. Online studies did not deliver the same level of precision as lab-based systems, with slightly more variability in all measurements. That said, PsychoPy and Gorilla, broadly the best performers, were achieving very close to millisecond precision on several browser/operating system combinations. For response times (measured using a high-performance button box), most of the packages achieved precision at least under 10 ms in all browsers, with PsychoPy achieving a precision under 3.5 ms in all. There was considerable variability between OS/browser combinations, especially in audio-visual synchrony which is the least precise aspect of the browser-based experiments. Nonetheless, the data indicate that online methods can be suitable for a wide range of studies, with due thought about the sources of variability that result. The results, from over 110,000 trials, highlight the wide range of timing qualities that can occur even in these dedicated software packages for the task. We stress the importance of scientists making their own timing validation measurements for their own stimuli and computer configuration.

Using an Online Platform to Administer the Single-Session Point Subtraction Aggression Paradigm: An Initial Examination of Feasibility and Validity

Online platforms represent a cost-effective option for data collection; however, it is unclear whether online administration of certain kinds of tasks (e.g., behavioral measures of aggression) poses validity threats. The present study provided a preliminary examination of effort (as indexed by total number of presses), differential drop-out, and believability of an online version of the single-session point subtraction aggression paradigm (PSAP). Two subsamples of participants were recruited; a sample recruited through Amazon's Mechanical Turk (n = 758) and an in-person undergraduate sample (n = 88). All participants completed the PSAP, along with measures of trait hostility and state anger. The online sample did not differ from the in-person sample on effort (i.e., total number of presses), and did not find the task less believable. Higher scores on state anger were associated with lower likelihood of beginning the online PSAP, but were not associated with prematurely closing the task. State anger was related to aggressive responding on the PSAP. Limitations of the online PSAP and considerations for future research are discussed.

An Evaluation of Amazon's Mechanical Turk, Its Rapid Rise, and Its Effective Use

Over the past 2 decades, many social scientists have expanded their data-collection capabilities by using various online research tools. In the 2011 article "Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data?" in Perspectives on Psychological Science, Buhrmester, Kwang, and Gosling introduced researchers to what was then considered to be a promising but nascent research platform. Since then, thousands of social scientists from seemingly every field have conducted research using the platform. Here, we reflect on the impact of Mechanical Turk on the social sciences and our article's role in its rise, provide the newest data-driven recommendations to help researchers effectively use the platform, and highlight other online research platforms worth consideration.

Public Concern About Monitoring Twitter Users and Their Conversations to Recruit for Clinical Trials: Survey Study

Background

Social networks such as Twitter offer the clinical research community a novel opportunity for engaging potential study participants based on user activity data. However, the availability of public social media data has led to new ethical challenges about respecting user privacy and the appropriateness of monitoring social media for clinical trial recruitment. Researchers have voiced the need for involving users’ perspectives in the development of ethical norms and regulations.

Objective

This study examined the attitudes and level of concern among Twitter users and nonusers about using Twitter for monitoring social media users and their conversations to recruit potential clinical trial participants.

Methods

We used two online methods for recruiting study participants: the open survey was (1) advertised on Twitter between May 23 and June 8, 2017, and (2) deployed on TurkPrime, a crowdsourcing data acquisition platform, between May 23 and June 8, 2017. Eligible participants were adults, 18 years of age or older, who lived in the United States. People with and without Twitter accounts were included in the study.

Results

While nearly half the respondents—on Twitter (94/603, 15.6%) and on TurkPrime (509/603, 84.4%)—indicated agreement that social media monitoring constitutes a form of eavesdropping that invades their privacy, over one-third disagreed and nearly 1 in 5 had no opinion. A chi-square test revealed a positive relationship between respondents’ general privacy concern and their average concern about Internet research (P<.005). We found associations between respondents’ Twitter literacy and their concerns about the ability for researchers to monitor their Twitter activity for clinical trial recruitment (P=.001) and whether they consider Twitter monitoring for clinical trial recruitment as eavesdropping (P<.001) and an invasion of privacy (P=.003). As Twitter literacy increased, so did people’s concerns about researchers monitoring Twitter activity. Our data support the previously suggested use of the nonexceptionalist methodology for assessing social media in research, insofar as social media-based recruitment does not need to be considered exceptional and, for most, it is considered preferable to traditional in-person interventions at physical clinics. The expressed attitudes were highly contextual, depending on factors such as the type of disease or health topic (eg, HIV/AIDS vs obesity vs smoking), the entity or person monitoring users on Twitter, and the monitored information.

Conclusions

The data and findings from this study contribute to the critical dialogue with the public about the use of social media in clinical research. The findings suggest that most users do not think that monitoring Twitter for clinical trial recruitment constitutes inappropriate surveillance or a violation of privacy. However, researchers should remain mindful that some participants might find social media monitoring problematic when connected with certain conditions or health topics. Further research should isolate factors that influence the level of concern among social media users across platforms and populations and inform the development of more clear and consistent guidelines.

Outcomes of two randomized controlled trials, employing participants recruited through Mechanical Turk, of Internet interventions targeting unhealthy alcohol use

Background

Two randomized controlled trials (RCTs) were conducted to explore the utility of the Mechanical Turk (MTurk) crowdsourcing platform to conduct rapid trials evaluating online interventions for unhealthy alcohol use.

Methods

Both trials employed a staged recruitment procedure where participants who drank in an unhealthy fashion were identified using a baseline survey and then invited to take part in a 6-month follow-up. Participants in both trials were randomized to receive one of several different online interventions or to a no intervention control condition. In study 1, the online interventions were password protected and only those who accessed the study portal were randomized to condition. In study 2, participants were directed to free-of charge interventions and asked to send a screenshot of the intervention to demonstrate that they had complied.

Results

Participants reporting unhealthy alcohol use were recruited fairly rapidly. Large numbers of screeners were completed (Study 1: n = 4910; Study 2: n = 5812), found eligible (Study 1: n = 3741; Study 2: n = 4095), and randomized to condition (Study 1: n = 511; Study 2: n = 878). Fair follow-up rates were observed at 6 months for each study (Study 1: 82%; Study 2: 66%). Neither trial was able to clearly demonstrate that providing access to the online interventions lead to increased reductions in alcohol use as compared to the control group.

Conclusions

While recruitment through a crowdsourcing platform is rapid and relatively low cost, it is possible that the lack of impact of the online websites employed in these trials could be due to the source of participants rather than the lack of efficacy of the interventions.

Trial registration

ClinicalTrials.gov # NCT02977026 and NCT03060135.

Assessing and overcoming participant dishonesty in online data collection

Crowdsourcing services, such as MTurk, have opened a large pool of participants to researchers. Unfortunately, it can be difficult to confidently acquire a sample that matches a given demographic, psychographic, or behavioral dimension. This problem exists because little information is known about individual participants and because some participants are motivated to misrepresent their identity with the goal of financial reward. Despite the fact that online workers do not typically display a greater than average level of dishonesty, when researchers overtly request that only a certain population take part in an online study, a nontrivial portion misrepresent their identity. In this study, a proposed system is tested that researchers can use to quickly, fairly, and easily screen participants on any dimension. In contrast to an overt request, the reported system results in significantly fewer (near zero) instances of participant misrepresentation. Tests for misrepresentations were conducted by using a large database of past participant records (~45,000 unique workers). This research presents and tests an important tool for the increasingly prevalent practice of online data collection.

Generalizability of Total Worker Health® Online Training for Young Workers

Young workers (under 25-years-old) are at risk of workplace injuries due to inexperience, high-risk health behaviors, and a lack of knowledge about workplace hazards. Training based on Total Worker Health^® (TWH) principles can improve their knowledge of and ability to identify hazards associated with work organization and environment. In this study, we assessed changes to knowledge and behavior following an online safety and health training between two groups by collecting information on the demographic characteristics, knowledge, and self-reported behaviors of workplace health and safety at three different points in time. The participants’ age ranged from 15 to 24 years. Age adjusted results exhibited a significant increase in knowledge immediately after completing the training, although knowledge decreased in both groups in the follow-up. Amazon Marketplace Mechanical Turk (MTurk) participants demonstrated a greater increase in knowledge, with a significantly higher score compared to the baseline, indicating retention of knowledge three months after completing the training. The majority of participants in both groups reported that they liked the Promoting U through Safety and Health (PUSH) training for improving health and safety and that the training should be provided before starting a job. Participants also said that the training was interactive, informative and humorous. The participants reported that the PUSH training prepared them to identify and control hazards in their workplace and to communicate well with the supervisors and coworkers about their rights. Training programs based on TWH improves the safety, health and well-being of young workers.

MTurk Participants Have Substantially Lower Evaluative Subjective Well-Being Than Other Survey Participants

Amazon's MTurk platform has become a popular site for obtaining relatively inexpensive and convenient adult samples for use in behavioral research. Concerns have been raised about selection issues, because MTurk workers chose to participate in the platform and select the tasks they perform (of many offered to them). Prior studies have documented demographic and psychological differences with national samples. In this paper we studied evaluative subjective well-being (the Cantril Ladder) in an MTurk sample, a national Internet panel sample, and a national telephone survey conducted by Gallup-Sharecare. A surprising finding was that MTurk participants' Ladder scores were substantial lower than the other two samples. Analyses controlling for six demographic differences among the samples only slightly reduced the mean differences. However, patterns of demographic-well-being associations were similar within the samples. To corroborate these results, we conducted a secondary analysis on another three samples, one MTurk sample and two Internet panel samples. The same group differences in Ladder scores were observed. These findings add to the growing literature documenting the characteristics of MTurk samples and we discuss the implications for future research with such samples.

Obtaining quality data using behavioral measures of impulsivity in gambling research with Amazon's Mechanical Turk

BACKGROUND AND AIMS

To date, no research has examined the viability of using behavioral tasks typical of cognitive and neuropsychology within addiction populations through online recruitment methods. Therefore, we examined the reliability and validity of three behavioral tasks of impulsivity common in addiction research in a sample of individuals with a current or past history of problem gambling recruited online.

METHODS

Using a two-stage recruitment process, a final sample of 110 participants with a history of problem or disordered gambling were recruited through MTurk and completed self-report questionnaires of gambling involvement symptomology, a Delay Discounting Task (DDT), Balloon Analogue Risk Task (BART), Cued Go/No-Go Task, and the UPPS-P.

RESULTS

Participants demonstrated logically consistent responding on the DDT. The area under the empirical discounting curve (AUC) ranged from 0.02 to 0.88 (M = 0.23). The BART demonstrated good split-third reliability (ρs = 0.67 to 0.78). The tasks generally showed small correlations with each other (ρs = ±0.06 to 0.19) and with UPPS-P subscales (ρs = ±0.01 to 0.20).

DISCUSSION AND CONCLUSIONS

The behavioral tasks demonstrated good divergent validity. Correlation magnitudes between behavioral tasks and UPPS-P scales and mean scores on these measures were generally consistent with the existing literature. Behavioral tasks of impulsivity appear to have utility for use with problem and disordered gambling samples collected online, allowing researchers a cost efficient and rapid avenue for conducting behavioral research with gamblers. We conclude with best-practice recommendations for using behavioral tasks using crowdsourcing samples.

Incorporating public values into evaluative criteria: Using crowdsourcing to identify criteria and standards

At its core, evaluation involves the generation of value judgments. These evaluative judgments are based on comparing an evaluand's performance to what the evaluand is supposed to do (criteria) and how well it is supposed to do it (standards). The aim of this four-phase study was to test whether criteria and standards can be set via crowdsourcing, a potentially cost- and time-effective approach to collecting public opinion data. In the first three phases, participants were presented with a program description, then asked to complete a task to either identify criteria (phase one), weigh criteria (phase two), or set standards (phase three). Phase four found that the crowd-generated criteria were high quality; more specifically, that they were clear and concise, complete, non-overlapping, and realistic. Overall, the study concludes that crowdsourcing has the potential to be used in evaluation for setting stable, high-quality criteria and standards.

The MTurkification of Social and Personality Psychology

The potential role of brief online studies in changing the types of research and theories likely to evolve is examined in the context of earlier changes in theory and methods in social and personality psychology, changes that favored low-difficulty, high-volume studies. An evolutionary metaphor suggests that the current publication environment of social and personality psychology is a highly competitive one, and that academic survival and reproduction processes (getting a job, tenure/promotion, grants, awards, good graduate students) can result in the extinction of important research domains. Tracking the prevalence of brief online studies, exemplified by studies using Amazon Mechanical Turk, in three top journals ( Journal of Personality and Social Psychology, Personality and Social Psychology Bulletin, Journal of Experimental Social Psychology) reveals a dramatic increase in their frequency and proportion. Implications, suggestions, and questions concerning this trend for the field and questions for its practitioners are discussed.

Comparing Amazon's Mechanical Turk Platform to Conventional Data Collection Methods in the Health and Medical Research Literature

BACKGROUND

The goal of this article is to conduct an assessment of the peer-reviewed primary literature with study objectives to analyze Amazon.com 's Mechanical Turk (MTurk) as a research tool in a health services research and medical context.

METHODS

Searches of Google Scholar and PubMed databases were conducted in February 2017. We screened article titles and abstracts to identify relevant articles that compare data from MTurk samples in a health and medical context to another sample, expert opinion, or other gold standard. Full-text manuscript reviews were conducted for the 35 articles that met the study criteria.

RESULTS

The vast majority of the studies supported the use of MTurk for a variety of academic purposes.

DISCUSSION

The literature overwhelmingly concludes that MTurk is an efficient, reliable, cost-effective tool for generating sample responses that are largely comparable to those collected via more conventional means. Caveats include survey responses may not be generalizable to the US population.

Are all "research fields" equal? Rethinking practice for the use of data from crowdsourcing market places

New technologies like large-scale social media sites (e.g., Facebook and Twitter) and crowdsourcing services (e.g., Amazon Mechanical Turk, Crowdflower, Clickworker) are impacting social science research and providing many new and interesting avenues for research. The use of these new technologies for research has not been without challenges, and a recently published psychological study on Facebook has led to a widespread discussion of the ethics of conducting large-scale experiments online. Surprisingly little has been said about the ethics of conducting research using commercial crowdsourcing marketplaces. In this article, I focus on the question of which ethical questions are raised by data collection with crowdsourcing tools. I briefly draw on the implications of Internet research more generally, and then focus on the specific challenges that research with crowdsourcing tools faces. I identify fair pay and the related issue of respect for autonomy, as well as problems with the power dynamic between researcher and participant, which has implications for withdrawal without prejudice, as the major ethical challenges of crowdsourced data. Furthermore, I wish to draw attention to how we can develop a "best practice" for researchers using crowdsourcing tools.

Towards program theory validation: Crowdsourcing the qualitative analysis of participant experiences

This exploratory study examines a novel tool for validating program theory through crowdsourced qualitative analysis. It combines a quantitative pattern matching framework traditionally used in theory-driven evaluation with crowdsourcing to analyze qualitative interview data. A sample of crowdsourced participants are asked to read an interview transcript and identify whether program theory components (Activities and Outcomes) are discussed and to highlight the most relevant passage about that component. The findings indicate that using crowdsourcing to analyze qualitative data can differentiate between program theory components that are supported by a participant's experience and those that are not. This approach expands the range of tools available to validate program theory using qualitative data, thus strengthening the theory-driven approach.

Exploring the Great Schism in the Social Sciences: Confirmation Bias and the Interpretation of Results Relating to Biological Influences on Human Behavior and Psychology

The nature-nurture debate is one that biologists often dismiss as a false dichotomy, as all phenotypic traits are the results of complex processes of gene and environment interactions. However, such dismissiveness belies the ongoing debate that is unmistakable throughout the biological and social sciences concerning the role of biological influences in the development of psychological and behavioral traits in humans. Many have proposed that this debate is due to ideologically driven biases in the interpretation of results. Those favoring biological approaches have been accused of a greater willingness to accept biological explanations so as to rationalize or justify the status quo of inequality. Those rejecting biological approaches have been accused of an unwillingness to accept biological explanations so as to attribute inequalities solely to social and institutional factors, ultimately allowing for the possibility of social equality. While it is important to continue to investigate this topic through further research and debate, another approach is to examine the degree to which the allegations of bias are indeed valid. To accomplish this, a convenience sample of individuals with relevant postgraduate degrees was recruited from Mechanical Turk and social media. Participants were asked to rate the inferential power of different research designs and of mock results that varied in the degree to which they supported different ideologies. Results were suggestive that researchers harbor sincere differences of opinion concerning the inferential value of relevant research. There was no suggestion that ideological confirmation biases drive these differences. However, challenges associated with recruiting a large enough sample of experts as well as identifying believable mock scenarios limit the study's inferential scope.

Comparing Self-Reported Demographic and Sexual Behavioral Factors Among Men Who Have Sex with Men Recruited Through Mechanical Turk, Qualtrics, and a HIV/STI Clinic-Based Sample: Implications for Researchers and Providers

Recruitment for HIV research among gay, bisexual, and other men who have sex with men (MSM) has increasingly moved to the online sphere. However, there are limited data comparing the characteristics of clinic-based respondents versus those recruited via online survey platforms. MSM were recruited from three sampling sites (STI clinic, MTurk, and Qualtrics) to participate in a survey from March 2015 to April 2016. Respondents were compared between each of the sampling sites on demographics, sexual history, substance use, and attention filter passage. Attention filter passage was high for the online sampling sites (MTurk = 93%; Qualtrics = 86%), but significantly lower for the clinic-based sampling site (72%). Clinic-based respondents were significantly more racially/ethnically diverse, reported lower income, and reported more unemployment than online respondents. Clinic-based respondents reported significantly more male sexual partners in the previous 3 months (M clinic-based = 6; MTurk = 3.6; Qualtrics = 4.5), a higher proportion of gonorrhea, chlamydia, and/or syphilis in the last year, and a greater proportion of methamphetamine use (clinic-based = 21%; MTurk = 5%), and inhaled nitrates use (clinic-based = 41%; MTurk = 11%). The clinic-based sample demonstrated more demographic diversity and a greater proportion of HIV risk behaviors when compared to the online samples, but also a relatively low attention filter passage rate. We recommend the use of attention filters across all modalities to assess response validity and urge caution with online survey engines as samples may differ demographically and behaviorally when compared to clinic-based respondents.

The Promise and Pitfalls of Using Crowdsourcing in Research Prioritization for Back Pain: Cross-Sectional Surveys

Background

The involvement of patients in research better aligns evidence generation to the gaps that patients themselves face when making decisions about health care. However, obtaining patients’ perspectives is challenging. Amazon’s Mechanical Turk (MTurk) has gained popularity over the past decade as a crowdsourcing platform to reach large numbers of individuals to perform tasks for a small reward for the respondent, at small cost to the investigator. The appropriateness of such crowdsourcing methods in medical research has yet to be clarified.

Objective

The goals of this study were to (1) understand how those on MTurk who screen positive for back pain prioritize research topics compared with those who screen negative for back pain, and (2) determine the qualitative differences in open-ended comments between groups.

Methods

We conducted cross-sectional surveys on MTurk to assess participants’ back pain and allow them to prioritize research topics. We paid respondents US $0.10 to complete the 24-point Roland Morris Disability Questionnaire (RMDQ) to categorize participants as those “with back pain” and those “without back pain,” then offered both those with (RMDQ score ≥7) and those without back pain (RMDQ <7) an opportunity to rank their top 5 (of 18) research topics for an additional US $0.75. We compared demographic information and research priorities between the 2 groups and performed qualitative analyses on free-text commentary that participants provided.

Results

We conducted 2 screening waves. We first screened 2189 individuals for back pain over 33 days and invited 480 (21.93%) who screened positive to complete the prioritization, of whom 350 (72.9% of eligible) did. We later screened 664 individuals over 7 days and invited 474 (71.4%) without back pain to complete the prioritization, of whom 397 (83.7% of eligible) did. Those with back pain who prioritized were comparable with those without in terms of age, education, marital status, and employment. The group with back pain had a higher proportion of women (234, 67.2% vs 229, 57.8%, P=.02). The groups’ rank lists of research priorities were highly correlated: Spearman correlation coefficient was .88 when considering topics ranked in the top 5. The 2 groups agreed on 4 of the top 5 and 9 of the top 10 research priorities.

Conclusions

Crowdsourcing platforms such as MTurk support efforts to efficiently reach large groups of individuals to obtain input on research activities. In the context of back pain, a prevalent and easily understood condition, the rank list of those with back pain was highly correlated with that of those without back pain. However, subtle differences in the content and quality of free-text comments suggest supplemental efforts may be needed to augment the reach of crowdsourcing in obtaining perspectives from patients, especially from specific populations.

Assessing Diabetes-Relevant Data Provided by Undergraduate and Crowdsourced Web-Based Survey Participants for Honesty and Accuracy

Background

To eliminate health disparities, research will depend on our ability to reach select groups of people (eg, samples of a particular racial or ethnic group with a particular disease); unfortunately, researchers often experience difficulty obtaining high-quality data from samples of sufficient size.

Objective

Past studies utilizing MTurk applaud its diversity, so our initial objective was to capitalize on MTurk’s diversity to investigate psychosocial factors related to diabetes self-care.

Methods

In Study 1, a “Health Survey” was posted on MTurk to examine diabetes-relevant psychosocial factors. The survey was restricted to individuals who were 18 years of age or older with diabetes. Detection of irregularities in the data, however, prompted an evaluation of the quality of MTurk health-relevant data. This ultimately led to Study 2, which utilized an alert statement to improve conscientious behavior, or the likelihood that participants would be thorough and diligent in their responses. Trap questions were also embedded to assess conscientious behavior.

Results

In Study 1, of 4165 responses, 1246 were generated from 533 unique IP addresses completing the survey multiple times within close temporal proximity. Ultimately, only 252 responses were found to be acceptable. Further analyses indicated additional quality concerns with this subsample. In Study 2, as compared with the MTurk sample (N=316), the undergraduate sample (N=300) included more females, and fewer individuals who were married. The samples did not differ with respect to race. Although the presence of an alert resulted in fewer trap failures (mean=0.07) than when no alert was present (mean=0.11), this difference failed to reach significance: F_1,604=2.5, P=.11, ƞ²=.004, power=.35. The modal trap failure response was zero, while the mean was 0.092 (SD=0.32). There were a total of 60 trap failures in a context where the potential could have exceeded 16,000.

Conclusions

Published studies that utilize MTurk participants are rapidly appearing in the health domain. While MTurk may have the potential to be more diverse than an undergraduate sample, our efforts did not meet the criteria for what would constitute a diverse sample in and of itself. Because some researchers have experienced successful data collection on MTurk, while others report disastrous results, Kees et al recently identified that one essential area of research is of the types and magnitude of cheating behavior occurring on Web-based platforms. The present studies can contribute to this dialogue, and alternately provide evidence of disaster and success. Moving forward, it is recommended that researchers employ best practices in survey design and deliberately embed trap questions to assess participant behavior. We would strongly suggest that standards be in place for publishing the results of Web-based surveys—standards that protect against publication unless there are suitable quality assurance tests built into the survey design, distribution, and analysis.

Validation of a Seven-Factor Structure for the Motives for Playing Drinking Games Measure

Playing drinking games can be characterized as a high-risk drinking activity because games are typically designed to promote heavy alcohol consumption. While research suggests that young adults are motivated to play drinking games for a variety of reasons (e.g., for thrills/fun, for the competition), the Motives for Playing Drinking Games measure has received limited empirical attention. We examined the psychometric properties of this measure with a confirmation sample of young adults recruited from Amazon's MTurk ( N = 1,809, ages 18-25 years, 47% men; 41% not currently enrolled in college) and a validation sample of college students ( N = 671; ages 18-23 years; 26% men). Contrary to the 8-factor model obtained by Johnson and Sheets in a study published in 2004, examination of the factor structure with our confirmation sample yielded a revised 7-factor model that was invariant across race/ethnicity and college student status. This model was also validated with the college student sample. In the confirmation sample, enhancement/thrills and sexual pursuit motives for playing drinking games were positively associated with gaming frequency/consumption and negative gaming consequences. Furthermore, conformity motives for playing drinking games were positively associated with negative gaming consequences, while competition motives were positively associated with gaming frequency. These findings have significant implications for research and prevention/intervention efforts.

Latent-level relations between DSM-5 PTSD symptom clusters and problematic smartphone use

Common mental health consequences following the experience of potentially traumatic events include Posttraumatic Stress Disorder (PTSD) and addictive behaviors. Problematic smartphone use is a newer manifestation of addictive behaviors. People with anxiety severity (such as PTSD) may be at risk for problematic smartphone use as a means of coping with their symptoms. Unique to our knowledge, we assessed relations between PTSD symptom clusters and problematic smartphone use. Participants (N = 347), recruited through Amazon's Mechanical Turk (MTurk), completed measures of PTSD and smartphone addiction. Results of the Wald tests of parameter constraints indicated that problematic smartphone use was more related to PTSD's negative alterations in cognitions and mood (NACM) than to PTSD's avoidance factor, Wald χ²(1, N = 347) = 12.51, p = 0.0004; and more to PTSD's arousal compared to PTSD's avoidance factor, Wald χ²(1, N = 347) = 14.89, p = 0.0001. Results indicate that problematic smartphone use is most associated with negative affect and arousal among trauma-exposed individuals. Implications include the need to clinically assess problematic smartphone use among trauma-exposed individuals presenting with higher NACM and arousal severity; and targeting NACM and arousal symptoms to mitigate the effects of problematic smartphone use.