1
|
Kolc KL, Tan YXK, Lo AZY, Shvetcov A, Mitchell PB, Perkes IE. Measuring psychiatric symptoms online: A systematic review of the use of inventories on Amazon Mechanical Turk (mTurk). J Psychiatr Res 2023; 163:118-126. [PMID: 37209617 DOI: 10.1016/j.jpsychires.2023.05.027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/07/2022] [Revised: 02/20/2023] [Accepted: 05/01/2023] [Indexed: 05/22/2023]
Abstract
Symptom measurement in psychiatric research increasingly uses digitized self-report inventories and is turning to crowdsourcing platforms for recruitment, e.g., Amazon Mechanical Turk (mTurk). The impact of digitizing pencil-and-paper inventories on the psychometric properties is underexplored in mental health research. Against this background, numerous studies report high prevalence estimates of psychiatric symptoms in mTurk samples. Here we develop a framework to evaluate the online implementation of psychiatric symptom inventories relative to two domains, that is, the adherence to (i) validated scoring and (ii) standardized administration. We apply this new framework to the online use of the Patient Health Questionnaire-9 (PHQ-9), Generalized Anxiety Disorder-7 (GAD-7), and Alcohol Use Disorder Identification Test (AUDIT). Our systematic review of the literature identified 36 implementations of these three inventories on mTurk across 27 publications. We also evaluated methodological approaches to enhance data quality, e.g., the use of bot detection and attention check items. Of the 36 implementations, 23 reported the applied diagnostic scoring criteria and only 18 reported the specified symptom timeframe. None of the 36 implementations reported adaptations made in their digitization of the inventories. While recent reports attribute higher rates of mood, anxiety, and alcohol use disorders on mTurk to data quality, our findings indicate that this inflation may also relate to the assessment methods. We provide recommendations to enhance both data quality and fidelity to validated administration and scoring methods.
Collapse
Affiliation(s)
- Kristy L Kolc
- School of Psychology, Faculty of Science, University of New South Wales, Sydney, Australia; Discipline of Psychiatry and Mental Health, School of Clinical Medicine, Faculty of Medicine and Health, University of New South Wales, Sydney, Australia.
| | - Yue Xuan Karen Tan
- Discipline of Psychiatry and Mental Health, School of Clinical Medicine, Faculty of Medicine and Health, University of New South Wales, Sydney, Australia
| | - Alys Z Y Lo
- Discipline of Psychiatry and Mental Health, School of Clinical Medicine, Faculty of Medicine and Health, University of New South Wales, Sydney, Australia
| | - Artur Shvetcov
- Discipline of Psychiatry and Mental Health, School of Clinical Medicine, Faculty of Medicine and Health, University of New South Wales, Sydney, Australia; Black Dog Institute, Sydney, Australia
| | - Philip B Mitchell
- Discipline of Psychiatry and Mental Health, School of Clinical Medicine, Faculty of Medicine and Health, University of New South Wales, Sydney, Australia
| | - Iain E Perkes
- School of Psychology, Faculty of Science, University of New South Wales, Sydney, Australia; Discipline of Psychiatry and Mental Health, School of Clinical Medicine, Faculty of Medicine and Health, University of New South Wales, Sydney, Australia; Discipline of Paediatrics and Child Health, School of Clinical Medicine, Faculty of Medicine and Health, University of New South Wales, Sydney, Australia; Department of Psychological Medicine, Sydney Children's Hospital Network, Sydney, Australia
| |
Collapse
|
2
|
Alarefi A, Wang X, Tao R, Rui Q, Gao G, Wang Y, Pang L, Liu C, Zhang X. Depicting People in Visual Cues Affects Alcohol Cue Reactivity in Male Alcohol-Dependent Patients. Brain Sci 2022; 12:brainsci12030307. [PMID: 35326264 PMCID: PMC8946691 DOI: 10.3390/brainsci12030307] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2021] [Revised: 01/15/2022] [Accepted: 01/31/2022] [Indexed: 12/22/2022] Open
Abstract
Cue reactivity is often used to study alcohol cues brain responses. Standardized image sets are used, but the effect of viewing people interacting with the alcohol drink remains unclear, which is associated with the factors of alcohol cues that influence the degree of response to alcohol stimuli. The present study used fMRI to investigate the reactivity of alcohol dependence (AD) inpatients to alcohol cues with or without human drinking behavior. Cues with a human interacting with a drink were hypothesized to increase sensorimotor activation. In total, 30 AD inpatients were asked to view pictures with a factorial design of beverage types (alcoholic vs. non-alcoholic beverages) and cue types (with or without drink action). Whole-brain analyses were performed. A correlation analysis was conducted to confirm whether the whole-brain analysis revealed cue-related brain activations correlated with problem drinking duration. The left lingual gyrus showed significant beverage types through cue type interaction, and the bilateral temporal cortex showed significant activation in response to alcohol cues depicting human drinking behavior. The right and left lingual gyrus regions and left temporal cortex were positively correlated with problem drinking duration. Sensorimotor activations in the temporal cortex may reflect self-referential and memory-based scene processing. Thus, our findings indicate these regions are associated with alcohol use and suggest them for cue exposure treatment of alcohol addiction.
Collapse
Affiliation(s)
- Abdulqawi Alarefi
- Department of Radiology, The First Affiliated Hospital of USTC, School of Life Science, Division of Life Science and Medicine, University of Science & Technology of China, Hefei 230027, China; (A.A.); (Y.W.); (C.L.)
| | - Xunshi Wang
- Affiliated Psychological Hospital of Anhui Medical University, Hefei Fourth People’s Hospital, Anhui Mental Health Center, Hefei 230017, China; (X.W.); (R.T.); (Q.R.); (G.G.)
| | - Rui Tao
- Affiliated Psychological Hospital of Anhui Medical University, Hefei Fourth People’s Hospital, Anhui Mental Health Center, Hefei 230017, China; (X.W.); (R.T.); (Q.R.); (G.G.)
| | - Qinqin Rui
- Affiliated Psychological Hospital of Anhui Medical University, Hefei Fourth People’s Hospital, Anhui Mental Health Center, Hefei 230017, China; (X.W.); (R.T.); (Q.R.); (G.G.)
| | - Guoqing Gao
- Affiliated Psychological Hospital of Anhui Medical University, Hefei Fourth People’s Hospital, Anhui Mental Health Center, Hefei 230017, China; (X.W.); (R.T.); (Q.R.); (G.G.)
| | - Ying Wang
- Department of Radiology, The First Affiliated Hospital of USTC, School of Life Science, Division of Life Science and Medicine, University of Science & Technology of China, Hefei 230027, China; (A.A.); (Y.W.); (C.L.)
| | - Liangjun Pang
- Affiliated Psychological Hospital of Anhui Medical University, Hefei Fourth People’s Hospital, Anhui Mental Health Center, Hefei 230017, China; (X.W.); (R.T.); (Q.R.); (G.G.)
- Correspondence: (L.P.); (X.Z.); Tel.: +86-551-63607295 (X.Z.)
| | - Chialun Liu
- Department of Radiology, The First Affiliated Hospital of USTC, School of Life Science, Division of Life Science and Medicine, University of Science & Technology of China, Hefei 230027, China; (A.A.); (Y.W.); (C.L.)
| | - Xiaochu Zhang
- Department of Radiology, The First Affiliated Hospital of USTC, School of Life Science, Division of Life Science and Medicine, University of Science & Technology of China, Hefei 230027, China; (A.A.); (Y.W.); (C.L.)
- Affiliated Psychological Hospital of Anhui Medical University, Hefei Fourth People’s Hospital, Anhui Mental Health Center, Hefei 230017, China; (X.W.); (R.T.); (Q.R.); (G.G.)
- Centers for Biomedical Engineering, School of Information Science and Technology, University of Science & Technology of China, Hefei 230027, China
- Department of Psychology, School of Humanities & Social Science, University of Science & Technology of China, Hefei 230026, China
- Correspondence: (L.P.); (X.Z.); Tel.: +86-551-63607295 (X.Z.)
| |
Collapse
|
3
|
Pennington CR, Jones A, Bartlett JE, Copeland A, Shaw DJ. Raising the bar: improving methodological rigour in cognitive alcohol research. Addiction 2021; 116:3243-3251. [PMID: 33999479 DOI: 10.1111/add.15563] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/22/2020] [Revised: 01/29/2021] [Accepted: 04/28/2021] [Indexed: 01/31/2023]
Abstract
BACKGROUND AND AIMS A range of experimental paradigms claim to measure the cognitive processes underpinning alcohol use, suggesting that heightened attentional bias, greater approach tendencies and reduced cue-specific inhibitory control are important drivers of consumption. This paper identifies methodological shortcomings within this broad domain of research and exemplifies them in studies focused specifically on alcohol-related attentional bias. ARGUMENT AND ANALYSIS We highlight five main methodological issues: (i) the use of inappropriately matched control stimuli; (ii) opacity of stimulus selection and validation procedures; (iii) a credence in noisy measures; (iv) a reliance on unreliable tasks; and (v) variability in design and analysis. This is evidenced through a review of alcohol-related attentional bias (64 empirical articles, 68 tasks), which reveals the following: only 53% of tasks use appropriately matched control stimuli; as few as 38% report their stimulus selection and 19% their validation procedures; less than 28% used indices capable of disambiguating attentional processes; 22% assess reliability; and under 2% of studies were pre-registered. CONCLUSIONS Well-matched and validated experimental stimuli, the development of reliable cognitive tasks and explicit assessment of their psychometric properties, and careful consideration of behavioural indices and their analysis will improve the methodological rigour of cognitive alcohol research. Open science principles can facilitate replication and reproducibility in alcohol research.
Collapse
Affiliation(s)
| | - Andrew Jones
- Institute of Population Health Sciences, University of Liverpool, Liverpool, UK
| | | | - Amber Copeland
- Department of Psychology, University of Sheffield, Sheffield, UK
| | - Daniel J Shaw
- School of Psychology, Aston University, Birmingham, UK
| |
Collapse
|