Evaluating Amazon's Mechanical Turk for psychological research on the symbolic control of attention

Influence of respondents' Differentiation of subjective response on water knowledge stock test scale: Evaluation based on two-parameter-multidimensional IRT model

Insufficient awareness of water issues is a crucial bottleneck restricting the sustainable utilization of water resources. To accurately measure citizens' water knowledge stock and overcome the differences between scales and respondents' characteristic levels on test results, the research focuses on developing and evaluating water knowledge stock test scales. The mechanism for identifying indicators is designed based on the grounded theory, and as a result, the water knowledge stock test indicator system is derived. The data was collected by the form of survey questionnaire developed with the test indicator system. A two-parameter multidimensional item response theoretical model is constructed based on item parameter estimation, data model fitting, and item information function. The survey data and optimization model are used to optimize the water knowledge stock test scale and verify the fitting degree with the characteristics of the respondents. The test information function and standard error function indicate that the scale is most informative for individuals with characteristic levels ranging from -2 to 3, resulting in a highly reliable test effect. The research has established a measurement indicator system, methodology, and presented results that serve as a foundation for measuring the stock of water knowledge.

The Active Suppression of a Distractor's Location Can Be Elusive

Our visual system is inundated with distracting objects that vie for our attention. While visual attention selects relevant information, inhibitory mechanisms might be useful to suppress the locations occupied by irrelevant distractors. Yet, there is a dearth of behavioral evidence for the active suppression of a distractor's location (ASDL) using central cues that provide preliminary information about a distractor's location. In the first two experiments, we attempt to conceptually replicate, using an online platform, experiments that provide evidence of the ASDL. We replicate the distractor cueing effect in a localization task (Experiment 1) wherein responses to targets were faster when a central arrow cued the location of an impending distractor than an empty location. This effect was larger in the first block of trials than it was in the second. In a discrimination task (Experiment 2), unlike previous studies, we found no evidence for an effect of distractor cueing. In Experiment 3, we replaced the central arrow cues with central number cues because arrow cues may elicit a symbolic shift of attention that might offset the ASDL. Once again, the best model was one in which the distractor cueing effect was absent. We replicate these failures to find evidence of the ASDL in two more experiments. The results suggest that the ASDL can be elusive and may be tied to the response system, not attention.

From Lab-Testing to Web-Testing in Cognitive Research: Who You Test is More Important than how You Test

The transition to web-testing, although promising, entails many new concerns. Web-testing is harder to monitor, so researchers need to ensure that the quality of the data collected is comparable to the quality of data typically achieved by lab-testing. Our study yields a novel contribution to this issue, by being the first to distinguish between the impact of web-testing and the impact of sourcing individuals from different participant pools, including crowdsourcing platforms. We presented a fairly general working memory task to 196 MTurk participants, 300 Prolific participants, and 255 students from the University of Geneva, allowing for a comparison of data quality across different participant pools. Among university students, 215 were web-tested, and 40 were lab-tested, allowing for a comparison of testing modalities within the same participant pool. Data quality was measured by assessing multiple data characteristics (i.e., reaction time, accuracy, anomalous values) and the presence of two behavioral benchmark effects. Our results revealed that who you test (i.e., participant pool) is more important than how you test (i.e., testing modality). Concerning how you test, our results showed that web-testing incurs a small, yet acceptable loss of data quality compared to lab-testing. Concerning who you test, Prolific participants were almost indistinguishable from web-tested students, but MTurk participants differed drastically from the other pools. Our results therefore encourage the use of web-testing in the domain of cognitive psychology, even when using complex paradigms. Nevertheless, these results urge for caution regarding how researchers select web-based participant pools when conducting online research.

Measurement invariance of the Multidimensional Depression Assessment Scale (MDAS) across gender and ethnic groups of Asian, Caucasian, Black, and Hispanic

BACKGROUND

Major Depressive Disorder is a severe and highly disabling mental illness. Almost all self-reported questionnaires have overlooked the interpersonal symptoms of depression which are important across gender and culture. The Multidimensional Depression Assessment Scale (MDAS) developed by Cheung and Power (2012) entails comprehensive emotional, cognitive, somatic, and interpersonal subscales. It addresses the criticism that existing self-report depression scales might not cover sufficient phenomenological heterogeneity of depression. The current study aimed to evaluate the psychometric properties of the MDAS across gender and four major ethnic groups of Caucasian, Black, Asian and Hispanic, including reliability and concurrent validity against the Centre for Epidemiological Studies Depression Scale (CESD) and Patients Health Questionnaire (PHQ-9). It also aimed to establish a stable factor structure across gender and ethnic groups and test the measurement invariance to enhance its potential for clinical use.

METHODS

A community sample of 3499 participants from four ethnic groups were recruited via online crowdsourcing sites of Qualtrics and Amazon M Turk. Each individual completed a demographic questionnaire, the MDAS, CESD and PHQ-9.

RESULTS

There was good internal consistency (Cronbach's alpha >0.90) and concurrent reliability across gender and ethnic groups. Strict measurement invariance was established for MDAS over a four-factor factor structure corresponding to the four subscales.

CONCLUSIONS

The MDAS showed good psychometric properties and measurement invariance of a four-factor structure, suggesting its potential to be used in clinical settings across gender and ethnic groups.

LIMITATIONS

Participants all answered the questionnaires in English, which could hinder cultural variations in their expression of symptoms.

Designing Environmental Messages to Discourage Red Meat Consumption: An Online Experiment

Reducing red meat consumption in high-consuming countries is critical for mitigating climate change and preventing chronic disease. This study tested the effectiveness of messages conveying the worsening or reduction of environmental harms at discouraging red meat consumption. 1078 U.S. adults viewed seven messages in an online survey highlighting the reduction or worsening of environmental harms associated with eating red meat (between-subjects factor) and rated the messages on how much they discouraged them from wanting to buy beef. Each message highlighted a different environmental harm: deforestation, climate change, water shortages, biodiversity loss, carbon footprint, greenhouse gas emissions, or environment (within-subjects factor). No statistically significant difference was found between the reduction and worsening of environmental harms conditions for most topics, though the worsening of harms frame slightly outperformed the reduction of harms frame for the 'environment' topic. 'Environment' was also the message topic that elicited the strongest response from participants overall. Latino participants, those with more than a high school degree, and those who consume beef once a week or less rated messages as more effective than non-Latino participants, those who completed high school or less, and those who consumed beef more than once a week. Future research should explore the effect of messages on behavioral outcomes.

The misrepresentation of spatial uncertainty in visual search: Single- versus joint-distribution probability cues

The present study used information theory to quantify the extent to which different spatial cues conveyed the entropy associated with the identity and location of a visual search target. Single-distribution cues reflected the probability that the target would appear at one fixed location whereas joint-distribution cues reflected the probability that the target would appear at the location where another cue (arrow) pointed. The present study used a novel demand-selection paradigm to examine the extent to which individuals explicitly preferred one type of probability cue over the other. Although both cues conveyed equal entropy, the main results suggested representation of greater target entropy for joint- than for single-distribution cues based on a comparison between predicted and observed probability cue choices across four experiments. The present findings emphasize the importance of understanding how individuals represent basic information-theoretic quantities that underlie more complex decision-theoretic processes such as Bayesian and active inference.

A new jsPsych plugin for psychophysics, providing accurate display duration and stimulus onset asynchrony

A JavaScript framework named 'jsPsych' developed by de Leeuw (2015) is widely used for conducting Web-based experiments, and its functionality can be enhanced by using plugins. This article introduces a new jsPsych plugin which enables experimenters to set different onset times for geometric figures, images, sounds, and moving objects, and present them synchronized with the refresh of the display. Moreover, this study evaluated the stimulus onset asynchronies (SOAs) using visual and audio stimuli. The results showed that: (i) the deviations from the intended SOAs between two visual stimuli were less than 10 ms, (ii) the variability across browser-computer combinations was reduced compared with the no-plugin condition, and (iii) the variability of the SOAs between visual and audio stimuli was relatively large (about 50 ms). This study concludes that although the use of audio stimuli is somewhat limited, the new plugin provides experimenters with useful and accurate methods for conducting psychophysical experiments online. The latest version of the plugin can be downloaded freely from https://jspsychophysics.hes.kyushu-u.ac.jp under the MIT license.

Methods of Detecting Insufficient Effort Responding: Comparisons and Practical Recommendations

Insufficient effort responding (IER) affects many forms of assessment in both educational and psychological contexts. Much research has examined different types of IER, IER's impact on the psychometric properties of test scores, and preprocessing procedures used to detect IER. However, there is a gap in the literature in terms of practical advice for applied researchers and psychometricians when evaluating multiple sources of IER evidence, including the best strategy or combination of strategies when preprocessing data. In this study, we demonstrate how the use of different IER detection methods may affect psychometric properties such as predictive validity and reliability. Moreover, we evaluate how different data cleansing procedures can detect different types of IER. We provide evidence via simulation studies and applied analysis using the ACT's Engage assessment as a motivating example. Based on the findings of the study, we provide recommendations and future research directions for those who suspect their data may contain responses reflecting careless, random, or biased responding.

Robust maximum marginal likelihood (RMML) estimation for item response theory models

Self-report data are common in psychological and survey research. Unfortunately, many of these samples are plagued with careless responses, due to unmotivated participants. The purpose of this study was to propose and evaluate a robust estimation method to detect careless or unmotivated responders, while leveraging item response theory (IRT) person-fit statistics. First, we outlined a general framework for robust estimation specific for IRT models. Subsequently, we conducted a simulation study covering multiple conditions in order to evaluate the performance of the proposed method. Ultimately, we showed that robust maximum marginal likelihood (RMML) estimation significantly improves detection rates for careless responders and reduces bias in item parameters across conditions. Furthermore, we applied our method to a real data set, to illustrate the utility of the proposed method. Our findings suggest that robust estimation coupled with person-fit statistics offers a powerful procedure to identify careless respondents for further review and to provide more accurate item parameter estimates in the presence of careless responses.

Script differences and masked translation priming: Evidence from Hindi-English bilinguals

This study reports on two experiments investigating the effects of script differences on masked translation priming in highly proficient early Hindi-English bilinguals. In Experiment 1 (the cross-script experiment), L1 Hindi was presented in the standard Devanagari script, while L2 English was presented in the Roman alphabet. In Experiment 2 (the same-script experiment), both L1 Hindi and L2 English were presented in the Roman alphabet. Both experiments revealed translation priming in the L1-L2 direction. However, L2-L1 priming was obtained in the same-script experiment, but not in the cross-script experiment. These findings are discussed in relation to the orthographic cue hypothesis as well as hypotheses that hold that script differences influence the distance between the L1 and L2 in lexical space and/or cross-language lateral inhibition. We also provide alternative accounts for these results in terms of how orthographic cues provided by L1 targets might lead to the discontinuation or disruption of processing for L2 primes.