The inter-item standard deviation (ISD): An index that discriminates between conscientious and random responders

Detecting inattentive respondents by machine learning: A generic technique that substitutes for the directed questions scale and compensates for its shortcomings

Web surveys are often used to collect data for psychological research. However, the inclusion of many inattentive respondents can be a problem. Various methods for detecting inattentive respondents have been proposed, most of which require the inclusion of additional items in the survey for detection or the calculation of variables for detection after data collection. This study proposes a method for detecting inattentive respondents in web surveys using machine learning. The method requires only the collection of response time and the inclusion of a Likert scale, eliminating the need to include special detection items in the survey. Based on data from 16 web surveys, a method was developed using predictor variables not included in existing methods. While previous machine learning methods for detecting inattentive respondents can only be applied to the same surveys as the data on which the models were developed, the proposed model is generic and can be applied to any questionnaire as long as response time is available, and a Likert scale is included. In addition, the proposed method showed partially higher accuracy than existing methods.

Validating the IDRIS and IDRIA: Two infrequency/frequency scales for detecting careless and insufficient effort survey responders

To detect careless and insufficient effort (C/IE) survey responders, researchers can use infrequency items - items that almost no one agrees with (e.g., "When a friend greets me, I generally try to say nothing back") - and frequency items - items that almost everyone agrees with (e.g., "I try to listen when someone I care about is telling me something"). Here, we provide initial validation for two sets of these items: the 14-item Invalid Responding Inventory for Statements (IDRIS) and the 6-item Invalid Responding Inventory for Adjectives (IDRIA). Across six studies (N1 = 536; N2 = 701; N3 = 500; N4 = 499; N5 = 629, N6 = 562), we found consistent evidence that the IDRIS is capable of detecting C/IE responding among statement-based scales (e.g., the HEXACO-PI-R) and the IDRIA is capable of detecting C/IE responding among both adjective-based scales (e.g., the Lex-20) and adjective-derived scales (e.g., the BFI-2). These findings were robust across different analytic approaches (e.g., Pearson correlations; Spearman rank-order correlations), different indices of C/IE responding (e.g., person-total correlations; semantic synonyms; horizontal cursor variability), and different sample types (e.g., US undergraduate students; Nigerian survey panel participants). Taken together, these results provide promising evidence for the utility of the IDRIS and IDRIA in detecting C/IE responding.

Predictors of Change in Vaccination Decisions Among the Vaccine Hesitant: Examining the Roles of Age and Intolerance of Uncertainty

BACKGROUND

Vaccine hesitancy and resistance pose significant threats to controlling pandemics and preventing infectious diseases. In a group of individuals unvaccinated against the disease caused by the SARS-CoV-2 coronavirus (COVID-19), we investigated how age, intolerance of uncertainty (IU), and their interaction affected the likelihood of having changed one's vaccination decision a year later. We hypothesized that higher IU would increase the likelihood of becoming vaccinated, particularly among individuals of younger age. We predicted that this effect would remain significant, even after controlling for delay discounting and trust in science.

PURPOSE

The goal of this research was to understand the factors influencing changes in vaccination decisions among the vaccine hesitant.

METHODS

In a larger longitudinal study, ~7,500 participants from Prolific.co completed demographic and vaccination status questions, a delay discounting task, and the Intolerance of Uncertainty Scale in June-August 2021. Approximately 3,200 participants completed a follow-up survey in July-August 2022, answering questions about vaccination status, reasons for vaccination decision, and trust in science. We analyzed data from 251 participants who initially had no intention of getting vaccinated and completed the follow-up survey; 38% reported becoming vaccinated in the intervening year.

RESULTS

Data were analyzed using multilevel logistic regression. Over and above other factors related to vaccination decisions (delay discounting, trust in science), younger participants were more likely to change their decision and become vaccinated a year later, especially if they had higher IU, confirming our predictions. Primary reasons for becoming vaccinated were necessity and seeking protection against the virus.

CONCLUSIONS

These findings highlight the complex interplay between age, uncertainty, and vaccination decisions, and inform health policies by suggesting the need for tailoring interventions to specific concerns in different age groups.

Careless responding detection revisited: Accuracy of direct and indirect measures

To screen for careless responding, researchers have a choice between several direct measures (i.e., bogus items, requiring the respondent to choose a specific answer) and indirect measures (i.e., unobtrusive post hoc indices). Given the dearth of research in the area, we examined how well direct and indirect indices perform relative to each other. In five experimental studies, we investigated whether the detection rates of the measures are affected by contextual factors: severity of the careless response pattern, type of item keying, and type of item presentation. We fully controlled the information environment by experimentally inducing careless response sets under a variety of contextual conditions. In Studies 1 and 2, participants rated the personality of an actor that presented himself in a 5-min-long videotaped speech. In Studies 3, 4, and 5, participants had to rate their own personality across two measurements. With the exception of maximum longstring, intra-individual response variability, and individual contribution to model misfit, all examined indirect indices performed better than chance in most of the examined conditions. Moreover, indirect indices had detection rates as good as and, in many cases, better than the detection rates of direct measures. We therefore encourage researchers to use indirect indices, especially within-person consistency indices, instead of direct measures.

How Terminology Affects Users' Responses to System Failures

OBJECTIVE

The objective of our research is to advance the understanding of behavioral responses to a system's error. By examining trust as a dynamic variable and drawing from attribution theory, we explain the underlying mechanism and suggest how terminology can be used to mitigate the so-called algorithm aversion. In this way, we show that the use of different terms may shape consumers' perceptions and provide guidance on how these differences can be mitigated.

BACKGROUND

Previous research has interchangeably used various terms to refer to a system and results regarding trust in systems have been ambiguous.

METHODS

Across three studies, we examine the effect of different system terminology on consumer behavior following a system failure.

RESULTS

Our results show that terminology crucially affects user behavior. Describing a system as "AI" (i.e., self-learning and perceived as more complex) instead of as "algorithmic" (i.e., a less complex rule-based system) leads to more favorable behavioral responses by users when a system error occurs.

CONCLUSION

We suggest that in cases when a system's characteristics do not allow for it to be called "AI," users should be provided with an explanation of why the system's error occurred, and task complexity should be pointed out. We highlight the importance of terminology, as this can unintentionally impact the robustness and replicability of research findings.

APPLICATION

This research offers insights for industries utilizing AI and algorithmic systems, highlighting how strategic terminology use can shape user trust and response to errors, thereby enhancing system acceptance.

Sense of coherence moderates job demand-resources and impact on burnout among nurses and midwives in the context of the COVID-19 pandemic: A cross-sectional survey

AIM

This study aimed to test the propositions using the job demands-resources (JD-R) model for main/moderation/mediation effects of a sense of coherence and practice environment support on mental well-being (anxiety, depression and burnout) outcomes in nurses and midwives in Australia during the COVID-19 pandemic.

DESIGN

Cross-sectional quantitative survey.

DATA SOURCES

The study was a cross-sectional design using self-report questionnaires reported as per the Reporting of Observational Studies in Epidemiology Guidelines. Following human research ethics approval (2020.ETH.00121) participants were recruited to take part in an online anonymous survey using self-report instruments to test the JD-R model in Australia.

RESULTS

156 participant nurses and midwives experienced anxiety, depression and emotional burnout during COVID-19. While a considerable proportion of participants indicated high levels of emotional exhaustion, their responses showed low levels of depersonalization (detached response to other people) and high levels of personal accomplishment (high levels of work performance and competence). A sense of coherence was a significant protective factor for mental health well-being for the participants, which is to say, high levels of sense of coherence were predictive of lower levels of anxiety, depression and burnout in this study sample.

CONCLUSION

It is evident that both nursing and midwifery professions require psychosocial support to preserve their health both in the short and long term. Ensuring individualized tailored support will require a layered response within organizations aimed at individual self-care and collegial peer support.

PATIENT OR PUBLIC CONTRIBUTION

There was no patient or public contribution in this study, as the focus was on nurses and midwives.

Consistency checks to improve measurement with the Young Mania Rating Scale (YMRS)

BACKGROUND

Mitigating rating inconsistency can improve measurement fidelity and detection of treatment response.

METHODS

The International Society for CNS Clinical Trials and Methodology convened an expert Working Group that developed logical consistency (LC) checks for ratings of the Young Mania Rating Scale (YMRS), which is widely used in studies of mood and bipolar disorders. LC and statistical outlier-response pattern checks (SC) were applied to 63,228 YMRS administrations from 14 clinical trials evaluating treatments for bipolar disorder. Checks were also applied to Monte Carlo-simulated data as a proxy for their use under conditions of inconsistency.

RESULTS

42 LC flags were developed, and four SC flags were created from the data set (n = 14). Almost 20 % of the rating administrations had at least one LC flag, 6.7 % had two or more, 1.7 % had three or more; 17.3 % percent of the administrations had at least one SC flag and 4.6 % percent had two or more. Overall, 31 % of administrations had at least one flag of any type, 12.1 % had two or more and 5.3 % had three or more. In acute antimanic treatment trials (n = 10) there were more flags of any type compared to relapse prevention trials (n = 4).

LIMITATIONS

Flagged ratings may represent less-common presentations assessed correctly.

CONCLUSIONS

Using established methods, we illustrate development and application of consistency flags for YMRS ratings. Applying flags and mitigation during trials may improve the value of YMRS data, help focus attention on rater training, and improve reliability and validity of trial data.

Development of an Inconsistent Responding Scale for the HEXACO Personality Inventory-Revised

Inconsistent or careless responding is a significant threat to the validity of self-reported personality data. Using archival samples of undergraduate and community participants, we developed an inconsistent responding scale using items that appear on both the 60- and 100-item versions of the HEXACO Personality Inventory-Revised-two widely used measures of the HEXACO model of personality trait structure. We identified pairs of correlated HEXACO items in Sample 1 and created a total inconsistent responding score by summing absolute differences between each item pair. The Brief Response Inconsistency Evaluation (BRIE) for the HEXACO effectively differentiated between genuine and randomly generated responses across samples. The BRIE also correlated as expected with other measures of careless responding and relevant personality traits (e.g., conscientiousness). Tentative cut scores for the BRIE that appear to provide a reasonable balance between sensitivity and specificity in Sample 1 were investigated. Future research should examine the BRIE with different populations and translations of the HEXACO inventories and further investigate the effectiveness of the recommended cut scores.

Inattentive responding can induce spurious associations between task behaviour and symptom measures

Although online samples have many advantages for psychiatric research, some potential pitfalls of this approach are not widely understood. Here we detail circumstances in which spurious correlations may arise between task behaviour and symptom scores. The problem arises because many psychiatric symptom surveys have asymmetric score distributions in the general population, meaning that careless responders on these surveys will show apparently elevated symptom levels. If these participants are similarly careless in their task performance, this may result in a spurious association between symptom scores and task behaviour. We demonstrate this pattern of results in two samples of participants recruited online (total N = 779) who performed one of two common cognitive tasks. False-positive rates for these spurious correlations increase with sample size, contrary to common assumptions. Excluding participants flagged for careless responding on surveys abolished the spurious correlations, but exclusion based on task performance alone was less effective.

Using heterogeneous sources of data and interpretability of prediction models to explain the characteristics of careless respondents in survey data

Prior to further processing, completed questionnaires must be screened for the presence of careless respondents. Different people will respond to surveys in different ways. Some take the easy path and fill out the survey carelessly. The proportion of careless respondents determines the survey's quality. As a result, identifying careless respondents is critical for the quality of obtained results. This study aims to explore the characteristics of careless respondents in survey data and evaluate the predictive power and interpretability of different types of data and indices of careless responding. The research question focuses on understanding the behavior of careless respondents and determining the effectiveness of various data sources in predicting their responses. Data from a three-month web-based survey on participants' personality traits such as honesty-humility, emotionality, extraversion, agreeableness, conscientiousness and openness to experience was used in this study. Data for this study was taken from Schroeders et al.. The gradient boosting machine-based prediction model uses data from the answers, time spent for answering, demographic information on the respondents as well as some indices of careless responding from all three types of data. Prediction models were evaluated with tenfold cross-validation repeated a hundred times. Prediction models were compared based on balanced accuracy. Models' explanations were provided with Shapley values. Compared with existing work, data fusion from multiple types of information had no noticeable effect on the performance of the gradient boosting machine model. Variables such as "I would never take a bribe, even if it was a lot", average longstring, and total intra-individual response variability were found to be useful in distinguishing careless respondents. However, variables like "I would be tempted to use counterfeit money if I could get away with it" and intra-individual response variability of the first section of a survey showed limited effectiveness. Additionally, this study indicated that, whereas the psychometric synonym score has an immediate effect and is designed with the goal of identifying careless respondents when combined with other variables, it is not necessarily the optimal choice for fitting a gradient boosting machine model.

Development of the reading literacy questionnaire for EFL learners at primary schools

Previous studies have indicated that there are a variety of factors influencing reading literacy assessment, including linguistic, cognitive, and affective factors, but little has been done on how to integrate these influential factors reasonably in a reading literacy instrument. As such, the purpose of this study is to develop and validate an English Reading Literacy Questionnaire (ERLQ) for English as foreign language (EFL) learners at the elementary level. The ERLQ was designed and revised through three rounds of validation with a sample of 784 pupils (Grades 3-6) in six primary schools from six provinces in China. Validity and reliability tests of the questionnaire were conducted with item analysis, Exploratory Factor Analysis (EFA), Confirmatory Factor Analysis (CFA), reliability test, and the analysis of criterion validity in SPSS 26.0 and AMOS 23.0. Results indicated that the revised ERLQ had high internal consistency, ranging from 0.729 to 0.823. The criterion validity of the ERLQ was supported by significant correlations to the Chinese Students' English Rating Scale verified by the authoritative department, with a correlation coefficient of 0.871. The study shows that the revised questionnaire, with 3 dimensions of 14 items, has high reliability and validity, which can be used as an assessment instrument for the intended audience. It also suggests that modifications may be made for further use in other regions and countries, depending on the background information of the learners.

Consistency checks to improve measurement with the Hamilton Rating Scale for Anxiety (HAM-A)

BACKGROUND

Mitigating rating inconsistency can improve measurement fidelity and detection of treatment response.

METHODS

The International Society for CNS Clinical Trials and Methodology convened an expert Working Group that developed consistency checks for ratings of the Hamilton Anxiety Rating Scale (HAM-A) and Clinical Global Impression of Severity of anxiety (CGIS) that are widely used in studies of mood and anxiety disorders. Flags were applied to 40,349 HAM-A administrations from 15 clinical trials and to Monte Carlo-simulated data as a proxy for applying flags under conditions of inconsistency.

RESULTS

Thirty-three flags were derived these included logical consistency checks and statistical outlier-response pattern checks. Twenty-percent of the HAM-A administrations had at least one logical scoring inconsistency flag, 4 % had two or more. Twenty-six percent of the administrations had at least one statistical outlier flag and 11 % had two or more. Overall, 35 % of administrations had at least one flag of any type, 19 % had one and 16 % had 2 or more. Most of administrations in the Monte Carlo- simulated data raised multiple flags.

LIMITATIONS

Flagged ratings may represent less-common presentations of administrations done correctly. Conclusions-Application of flags to clinical ratings may aid in detecting imprecise measurement. Flags can be used for monitoring of raters during an ongoing trial and as part of post-trial evaluation. Appling flags may improve reliability and validity of trial data.

Studies in the Mentality of Literates: 3. Conceptual Structure and Nonsense of Personality Testing

In this study, third in a series of studies of the relationships between the dominant type of the Word Meaning Structure (WMS) and various psychic processes, response patterns on personality questionnaires with Likert-type response format of individuals with different levels of education (including adult illiterates) in Brazil (N = 102) and in Estonia (N = 520) were assessed with person oriented methods of data analysis. We found that responses to two personality questionnaires (International Personality Item Pool Questionnaire, IPIP-Q60 and Estonian Collectivism Scale, ESTCOL) are inconsistent and do not correspond to theories that underlie construction and interpretation of such assessment tools. Two novel ways to assess inconsistent response patterns were developed. The Consistency Index (CI) characterizes between-item inconsistency and the Determinacy Index (DI) characterizes within-item inconsistency. The dominant type of the WMS and the level of education were related to both CI and DI. Higher level of between-item inconsistency characterizes everyday conceptual thinkers with lower levels of education and higher level of within-item inconsistency was observed among logical conceptual thinkers with higher levels of education. Systematic relationships between WMS and inconsistent patterns of responses indicate that responses on personality questionnaires cannot be interpreted in terms of personality characteristics. The results of our study also provide further support to the idea that dominant type of the WMS is a pervasive characteristic of the psyche and determines qualitatively possibilities and limits of the psychic processes. The results of this study are in agreement with the idea that WMS defines the "Great Divide."

Sexual Communal Strength Partially Mediates the Associations between Mate Retention Behaviors and Relationship Satisfaction

Mate retention behaviors are associated with several aspects of romantic relationships, but different categories of mate retention behaviors have divergent associations with relationship satisfaction. The present studies examined whether sexual communal strength - which refers to the motivation to meet the sexual needs of one's partner - mediated the associations that benefit-provisioning and cost-inflicting mate retention behaviors had with relationship satisfaction. We investigated the associations that mate retention behaviors had with relationship satisfaction through sexual communal strength using men's self-reports (Study 1) and women's partner-reports (Study 2). We found that sexual communal strength partially mediated the associations that mate retention behaviors had with relationship satisfaction in both studies. Benefit-provisioning mate retention behaviors had a positive indirect association with relationship satisfaction through sexual communal strength in both studies, whereas cost-inflicting mate retention behaviors had a negative indirect association with relationship satisfaction through sexual communal strength in Study 1 but not in Study 2. We explore the reasons why sexual communal strength may play an important role in the connections that mate retention behaviors have with relationship satisfaction.

Developing Early Markers of Cognitive Decline and Dementia Derived From Survey Response Behaviors: Protocol for Analyses of Preexisting Large-scale Longitudinal Data

BACKGROUND

Accumulating evidence shows that subtle alterations in daily functioning are among the earliest and strongest signals that predict cognitive decline and dementia. A survey is a small slice of everyday functioning; nevertheless, completing a survey is a complex and cognitively demanding task that requires attention, working memory, executive functioning, and short- and long-term memory. Examining older people's survey response behaviors, which focus on how respondents complete surveys irrespective of the content being sought by the questions, may represent a valuable but often neglected resource that can be leveraged to develop behavior-based early markers of cognitive decline and dementia that are cost-effective, unobtrusive, and scalable for use in large population samples.

OBJECTIVE

This paper describes the protocol of a multiyear research project funded by the US National Institute on Aging to develop early markers of cognitive decline and dementia derived from survey response behaviors at older ages.

METHODS

Two types of indices summarizing different aspects of older adults' survey response behaviors are created. Indices of subtle reporting mistakes are derived from questionnaire answer patterns in a number of population-based longitudinal aging studies. In parallel, para-data indices are generated from computer use behaviors recorded on the backend server of a large web-based panel study known as the Understanding America Study (UAS). In-depth examinations of the properties of the created questionnaire answer pattern and para-data indices will be conducted for the purpose of evaluating their concurrent validity, sensitivity to change, and predictive validity. We will synthesize the indices using individual participant data meta-analysis and conduct feature selection to identify the optimal combination of indices for predicting cognitive decline and dementia.

RESULTS

As of October 2022, we have identified 15 longitudinal ageing studies as eligible data sources for creating questionnaire answer pattern indices and obtained para-data from 15 UAS surveys that were fielded from mid-2014 to 2015. A total of 20 questionnaire answer pattern indices and 20 para-data indices have also been identified. We have conducted a preliminary investigation to test the utility of the questionnaire answer patterns and para-data indices for the prediction of cognitive decline and dementia. These early results are based on only a subset of indices but are suggestive of the findings that we anticipate will emerge from the planned analyses of multiple behavioral indices derived from many diverse studies.

CONCLUSIONS

Survey response behaviors are a relatively inexpensive data source, but they are seldom used directly for epidemiological research on cognitive impairment at older ages. This study is anticipated to develop an innovative yet unconventional approach that may complement existing approaches aimed at the early detection of cognitive decline and dementia.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/44627.

Dealing with Careless Responding in Survey Data: Prevention, Identification, and Recommended Best Practices

Surveys administered online have several benefits, but they are particularly prone to careless responding, which occurs when respondents fail to read item content or give sufficient attention, resulting in raw data that may not accurately reflect respondents' true levels of the constructs being measured. Careless responding can lead to various psychometric issues, potentially impacting any area of psychology that uses self-reported surveys and assessments. This review synthesizes the careless responding literature to provide a comprehensive understanding of careless responding and ways to prevent, identify, report, and clean careless responding from data sets. Further, we include recommendations for different levels of screening for careless responses. Finally, we highlight some of the most promising areas for future work on careless responding.

Sperm Competition Risk: The Connections That Partner Attractiveness and Infidelity Risk Have with Mate Retention Behaviors and Semen-Displacing Behaviors

The present studies investigated the relationships between men's perceived risk of experiencing sperm competition (i.e., when the ejaculates of two or more men simultaneously occupy the reproductive tract of a single woman), and their use of strategies to detect, prevent, and correct their partner's sexual infidelity. We investigated these associations using self-reports provided by men (Study 1, n = 113), partner-reports provided by women (Study 2, n = 136), and dyadic reports (Study 3, n = 103 couples). The results of these studies indicated that the attractiveness of women was consistently associated with men's use of benefit-provisioning mate retention behaviors (e.g., buying expensive gifts for one's partner, showing signs of physical affection) and semen-displacing behaviors (e.g., deeper copulatory thrusting, more thrusts during copulation), whereas the infidelity risk of women was often associated with men's use of cost-inflicting mate retention behaviors (e.g., threatening to end the relationship, monopolization of partner's free time). Discussion addresses the evolutionary implications of these results, including the possibility that men use both benefit-provisioning mate retention behaviors and semen-displacing behaviors when they perceive their partner to be more attractive, ostensibly as a way to mitigate their risk of sperm competition. Discussion also explores the extent to which these results extend those of previous studies concerning sperm competition risk.

Effects of Response Option Order on Likert-Type Psychometric Properties and Reactions

The effects of different response option orders on survey responses have been studied extensively. The typical research design involves examining the differences in response characteristics between conditions with the same item stems and response option orders that differ in valence-either incrementally arranged (e.g., strongly disagree to strongly agree) or decrementally arranged (e.g., strongly agree to strongly disagree). The present study added two additional experimental conditions-randomly incremental or decremental and completely randomized. All items were presented in an item-by-item format. We also extended previous studies by including an examination of response option order effects on: careless responding, correlations between focal predictors and criteria, and participant reactions, all the while controlling for false discovery rate and focusing on the size of effects. In a sample of 1,198 university students, we found little to no response option order effects on a recognized personality assessment vis-à-vis measurement equivalence, scale mean differences, item-level distributions, or participant reactions. However, the completely randomized response option order condition differed on several careless responding indices suggesting avenues for future research.

An explanatory mixture IRT model for careless and insufficient effort responding in self-report measures

Careless and insufficient effort responding (C/IER) on self-report measures results in responses that do not reflect the trait to be measured, thereby posing a major threat to the quality of survey data. Reliable approaches for detecting C/IER aid in increasing the validity of inferences being made from survey data. First, once detected, C/IER can be taken into account in data analysis. Second, approaches for detecting C/IER support a better understanding of its occurrence, which facilitates designing surveys that curb the prevalence of C/IER. Previous approaches for detecting C/IER are limited in that they identify C/IER at the aggregate respondent or scale level, thereby hindering investigations of item characteristics evoking C/IER. We propose an explanatory mixture item response theory model that supports identifying and modelling C/IER at the respondent-by-item level, can detect a wide array of C/IER patterns, and facilitates a deeper understanding of item characteristics associated with its occurrence. As the approach only requires raw response data, it is applicable to data from paper-and-pencil and online surveys. The model shows good parameter recovery and can well handle the simultaneous occurrence of multiple types of C/IER patterns in simulated data. The approach is illustrated on a publicly available Big Five inventory data set, where we found later item positions to be associated with higher C/IER probabilities. We gathered initial supporting validity evidence for the proposed approach by investigating agreement with multiple commonly employed indicators of C/IER.

Erectile Dysfunction and Sexual Coercion: The Role of Sperm Competition Risk

Erectile dysfunction (ED) has been shown to be associated with several negative aspects of intimate relationships. Our goal for the present research was to examine whether ED was associated with men's use of sexual coercion in their intimate relationships (which may include the use of physical coercion, psychological manipulation, or emotional manipulation to gain sexual access) and if perceived sperm competition risk (i.e., perceived risk of partner infidelity, which may place a man's sperm into competition with sperm from another man) played a role in this association. These associations were examined in Study 1 using self-reports provided by men (N = 202) who had a mean age of 30.48 years (SD = 5.03) and were recruited through Amazon's Mechanical Turk (MTurk). ED was found to have a large positive association with sexual coercion. However, men's self-reports did not provide support for sperm competition risk moderating the association between ED and sexual coercion, but an exploratory analysis revealed that sperm competition risk mediated this association. We attempted to replicate and extend these results in Study 2 by using partner-reports provided by women (N = 151) who had a mean age of 30.41 years (SD = 4.77) and were recruited through MTurk. Women's partner-reports provided support for sperm competition risk moderating the association between ED and sexual coercion. In addition, an exploratory analysis found that sperm competition risk also mediated the association between ED and sexual coercion, similar to Study 1. Discussion explores the implications of these results for understanding the role that sperm competition risk may play in the connection between ED and sexual coercion.

Short-sighted decision-making by those not vaccinated against COVID-19

Widespread vaccination is necessary to minimize or halt the effects of many infectious diseases, including COVID-19. Stagnating vaccine uptake can prolong pandemics, raising the question of how we might predict, prevent, and correct vaccine hesitancy and unwillingness. In a multinational sample (N = 4,452) recruited from 13 countries that varied in pandemic severity and vaccine uptake (July 2021), we examined whether short-sighted decision-making as exemplified by steep delay discounting-choosing smaller immediate rewards over larger delayed rewards-predicts COVID-19 vaccination status. Delay discounting was steeper in unvaccinated individuals and predicted vaccination status over and above demographics or mental health. The results suggest that delay discounting, a personal characteristic known to be modifiable through cognitive interventions, is a contributing cause of differences in vaccine compliance.

A methodology for preprocessing structured big data in the behavioral sciences

The characteristics of big data, including high volume, increased variety, and velocity, pose special challenges for data analysis. As these characteristics generally preclude manual data inspection and processing, researchers must often use computational methodologies to deal with this type of data; techniques that may be unfamiliar to nonspecialists, including behavioral scientists. However, previous data analytics methodologies within the field of computer science, developed to handle the generic tasks of data collection, preprocessing, and analysis, can be appropriated for use in other disciplines. These methodologies involve a sequential pipeline of quality checks to prepare data sets for analysis and application. Building upon these methodologies, this paper describes the Big Data Quality & Statistical Assurance (BDQSA) model, applicable for researchers in the behavioral sciences. It involves a series of data preprocessing tasks, to achieve data understanding, as well as data screening, cleaning, and transformation. These are followed by a statistical quality phase, which includes extraction of the relevant data subset, type conversions, ensuring sample representativeness when appropriate, and assessing statistical assumptions. The resulting model thereby provides methodological guidance for the preprocessing of behavioral science big data, aimed at ensuring acceptable data quality before analysis is undertaken. Sample R code snippets demonstrating the application of this model are provided throughout the paper.

Erectile Dysfunction and Partner-Directed Behaviors in Romantic Relationships: The Mediating Role of Suspicious Jealousy

Erectile dysfunction (ED) has been shown to have a considerable impact on romantic relationships. The purpose of the present research was to examine whether the associations that ED had with men's mate retention behaviors, partner-directed insults, partner-directed violence, and injuries inflicted on the partner were mediated by suspicious jealousy but not reactive jealousy. These associations were examined using self-reports of men (Study 1) and partner-reports provided by women about their perceptions of their male partner (Study 2). The results of both studies indicated that suspicious jealousy (but not reactive jealousy) mediated the associations that ED had with the partner-directed behaviors of men in their romantic relationships. Specifically, higher levels of ED were associated with men experiencing (or being perceived to experience) more suspicious jealousy which, in turn, predicted their use of mate retention behaviors, partner-directed insults, and partner-directed violence as well as the infliction of injuries on their female partners. In our discussion, we address the evolutionary implications of these associations, as well as limitations and directions for future research on ED.

Consistency checks to improve measurement with the Hamilton Rating Scale for Depression (HAM-D)

BACKGROUND

Symptom manifestations in mood disorders can be subtle. Cumulatively, small imprecisions in measurement can limit our ability to measure treatment response accurately. Logical and statistical consistency checks between item responses (i.e., cross-sectionally) and across administrations (i.e., longitudinally) can contribute to improving measurement fidelity.

METHODS

The International Society for CNS Clinical Trials and Methodology convened an expert Working Group that assembled flags indicating consistency/inconsistency ratings for the Hamilton Rating Scale for Depression (HAM-D17), a widely-used rating scale in studies of depression. Proposed flags were applied to assessments derived from the NEWMEDS data repository of 95,468 HAM-D administrations from 32 registration trials of antidepressant medications and to Monte Carlo-simulated data as a proxy for applying flags under conditions of known inconsistency.

RESULTS

Two types of flags were derived: logical consistency checks and statistical outlier-response pattern checks. Almost thirty percent of the HAMD administrations had at least one logical scoring inconsistency flag. Seven percent had flags judged to suggest that a thorough review of rating is warranted. Almost 22% of the administrations had at least one statistical outlier flag and 7.9% had more than one. Most of the administrations in the Monte Carlo- simulated data raised multiple flags.

LIMITATIONS

Flagged ratings may represent less-common presentations of administrations done correctly.

CONCLUSIONS

Application of flags to clinical ratings may aid in detecting imprecise measurement. Reviewing and addressing these flags may improve reliability and validity of clinical trial data.

Outlier-response pattern checks to improve measurement with the Montgomery-Asberg depression rating scale (MADRS)

Symptom manifestations in affective disorders can be subtle. Small imprecisions in measurement can lead to incorrect estimation of change. Previously, expert-derived scoring inconsistency flags were developed for MADRS. Currently, we derive empirically based outlier-pattern flags, to further detect imprecisions in ratings. NEWMEDS data repository of almost 25,000 MADRS administrations from 11 registration trials of antidepressants was used to identify outlier response patterns reflecting potentially careless responses. Coverage of these flags was compared to previously published expert derived flags. Both sets of flags were also further tested in Monte Carlo simulated data as a proxy to applying flags under conditions of known inconsistency. The outlier flags derived provide cutting points to identify: (1) under and overuse of values (e.g., Scoring "1″ on 6 or more items), (2) disproportionate use of even or odd response choices (e.g., 8 or more odd values), (3) longest consecutive use of value (e.g., more than 5 items in a row scored with same value), (4) high variability within administration (standard deviation greater than 1.8), (5) outlier responses on multiple items (i.e., multivariate outliers), and (6) outlier scoring (e.g., scoring 4,5 or 6 on item 1). Outlier response flags were raised in 26% of the MADRS administration and in 97% of the Monte Carlo data. Of administrations with no expert flag, 21.7% had an outlier flag and of administrations with at least one expert flag, 27.7% also had an outlier flag. Outlier-pattern flags appear to be a useful adjunct to expert derived flags in the quest to improve measurement in clinical trials.

Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting

Careless responding is a bias in survey responses that disregards the actual item content, constituting a threat to the factor structure, reliability, and validity of psychological measurements. Different approaches have been proposed to detect aberrant responses such as probing questions that directly assess test-taking behavior (e.g., bogus items), auxiliary or paradata (e.g., response times), or data-driven statistical techniques (e.g., Mahalanobis distance). In the present study, gradient boosted trees, a state-of-the-art machine learning technique, are introduced to identify careless respondents. The performance of the approach was compared with established techniques previously described in the literature (e.g., statistical outlier methods, consistency analyses, and response pattern functions) using simulated data and empirical data from a web-based study, in which diligent versus careless response behavior was experimentally induced. In the simulation study, gradient boosting machines outperformed traditional detection mechanisms in flagging aberrant responses. However, this advantage did not transfer to the empirical study. In terms of precision, the results of both traditional and the novel detection mechanisms were unsatisfactory, although the latter incorporated response times as additional information. The comparison between the results of the simulation and the online study showed that responses in real-world settings seem to be much more erratic than can be expected from the simulation studies. We critically discuss the generalizability of currently available detection methods and provide an outlook on future research on the detection of aberrant response patterns in survey research.

The Infertility-Related Stress Scale: Validation of a Brazilian-Portuguese Version and Measurement Invariance Across Brazil and Italy

Infertility constitutes an essential source of stress in the individual and couple's life. The Infertility-Related Stress Scale (IRSS) is of clinical interest for exploring infertility-related stress affecting the intrapersonal and interpersonal domains of infertile individuals' lives. In the present study, the IRSS was translated into Brazilian-Portuguese, and its factor structure, reliability, and relations to sociodemographic and infertility-related characteristics and depression were examined. A sample of 553 Brazilian infertile individuals (54.2% female, mean aged 36 ± 6 years) completed the Brazilian-Portuguese IRSS (IRSS-BP), and a subsample of 222 participants also completed the BDI-II. A sample of 526 Italian infertile individuals (54.2% female, mean aged 38 ± 6 years) was used to test for the IRSS measurement invariance across Brazil and Italy. Results of exploratory structural equation modeling (ESEM) indicated that a bifactor solution best represented the structure underlying the IRSS-BP. Both the general and the two specific intrapersonal and interpersonal IRSS-BP factors showed satisfactory levels of composite reliability. The bifactor ESEM solution replicated well across countries. As evidence of relations to other variables, female gender, a longer duration of infertility, and higher depression were associated with higher scores in global and domain-specific infertility-related stress. The findings offer initial evidence of validity and reliability of the IRSS-BP, which could be used by fertility clinic staff to rapidly identify patients who need support to deal with the stressful impact of infertility in the intrapersonal and interpersonal life domains, as recommended by international guidelines for routine psychosocial care in infertility settings.

Careless responses and construct validity of Wong-Law Emotional Intelligence Scale

Careless response in the investigation of emotional intelligence has not been explicitly addressed. In a sample of 180 Spanish adults responding to the Wong-Law Emotional Intelligence Scale, a small prevalence (7.2%) was detected. The impact was small on the psychometric parameters, but they were less inflated, more realistic, and precise.

Outlier-response pattern checks to improve measurement with the Positive and Negative Syndrome Scale (PANSS)

We derived outlier-response pattern checks to flag possible careless PANSS (Positive and Negative Syndrome Scale) administrations based on analysis of 122,000 administrations from 29 registration trials of antipsychotics from NEWMEDS data repository. Flags identify outlier administrations based on frequency of endorsing a given response value, use of even or odd values, consecutive use of same value, variability of values, responses per specific item, and values on multiple items. Outlier flags were compared to published expert derived scoring inconsistency flags and tested in Monte Carlo simulated data, with known inconsistency, and appear to be useful at identifying administrations that require review.

Respondent Burden Effects on Item Non-Response and Careless Response Rates: An Analysis of Two Types of Surveys

The respondent burden refers to the effort required by a respondent to answer a questionnaire. Although this concept was introduced decades ago, few studies have focused on the quantitative detection of such a burden. In this paper, a face-to-face survey and a telephone survey conducted in Valencia (Spain) are analyzed. The presence of burden is studied in terms of both item non-response rates and careless response rates. In particular, two moving-window statistics based on the coefficient of unalikeability and the average longstring index are proposed for characterizing careless responding. Item non-response and careless response rates are modeled for each survey by using mixed-effects models, including respondent-level and question-level covariates and also temporal random effects to assess the existence of respondent burden during the questionnaire. The results suggest that the sociodemographic characteristics of the respondents and the typology of the question impact item non-response and careless response rates. Moreover, the estimates of the temporal random effects indicate that item non-response and careless response rates are time-varying, suggesting the presence of respondent burden. In particular, an increasing trend in item non-response rates in the telephone survey has been found, which supports the hypothesis of the burden. Regarding careless responding, despite the presence of some temporal variation, no clear trend has been identified.

Validation of the Psychometric Properties of the NEO-FFI-3 in an Arabic Context

Aim

The present study aims to assess the validity and reliability of the psychometric properties of the Arabic adaptation of the NEO Five-Factor Inventory (NEO-FFI-3) to be used in research of personality inventory in the Arabic context.

Methods

It is a validation study of an Arabic version (NEO-AR) of the NEO-FFI-3. An online version was used to collect 1306 samples using convenient sampling in two phases. The collected data tested the validity of the inventory by using exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), while reliability was verified using Cronbach’s alpha after an Arabic version was conjured using forward translation, testing, and back-translation.

Results

The principal axis factoring (PAF) and the confirmatory factor analysis (CFA) are the analyses used to validate the factorial structure of the NEO-FFI-3 (NEO-AR). The reproduction of the original structure of NEO-FFI was proven using PAF, and CFA revealed a satisfactory fit of single-factor models for the five dimensions. The reliability analysis showed high values of internal consistency, which are congruent with previous international adaptations of the NEO-FFI-3.

Conclusion

The findings of the current study suggest that the Arabic version of the NEO-FFI-3 (NEO-AR) is a reliable and robust instrument to measure the five dimensions of personality of the Five-Factor Model facets as the original North America NEO-FFI-3. The results imply that the approach measuring psychological characteristics is successful across different cultures as in previous studies and the Arab World in this study.

Narcissism and Exercise Addiction: The Mediating Roles of Exercise-Related Motives

The present research examined whether the associations that narcissistic personality features had with exercise addiction were mediated by particular motives for engaging in exercise in a large Israeli community sample (N = 2629). The results revealed that each aspect of narcissism was positively associated with exercise addiction. Narcissistic admiration and narcissistic rivalry had similar positive indirect associations with exercise addiction through the interpersonal motive for exercise. However, these aspects of narcissism diverged in their indirect associations with exercise addiction through psychological motives, body-related motives, and fitness motives for exercise such that these indirect associations were positive for narcissistic admiration but negative for narcissistic rivalry. Narcissistic vulnerability had positive indirect associations with exercise addiction through body-related motives and fitness motives that were similar to those observed for narcissistic admiration. These results suggest that exercise-related motives may play important roles in the associations that narcissistic personality features have with exercise addiction. The discussion will focus on the implications of these results for understanding the complex connections between narcissism and exercise addiction.

Getting better without memory

Does the tendency to adjust appraisals of ourselves in the past and future in order to maintain a favourable view of ourselves in the present require episodic memory? A developmental amnesic person with impaired episodic memory (HC) was compared with two groups of age-matched controls on tasks assessing the Big Five personality traits and social competence in relation to the past, present and future. Consistent with previous research, controls believed that their personality had changed more in the past 5 years than it will change in the next 5 years (i.e. the end-of-history illusion), and rated their present and future selves as more socially competent than their past selves (i.e. social improvement illusion), although this was moderated by self-esteem. Despite her lifelong episodic memory impairment, HC also showed these biases of temporal self-appraisal. Together, these findings do not support the theory that the temporal extension of the self-concept requires the ability to recollect richly detailed memories of the self in the past and future.

Robust maximum marginal likelihood (RMML) estimation for item response theory models

Self-report data are common in psychological and survey research. Unfortunately, many of these samples are plagued with careless responses, due to unmotivated participants. The purpose of this study was to propose and evaluate a robust estimation method to detect careless or unmotivated responders, while leveraging item response theory (IRT) person-fit statistics. First, we outlined a general framework for robust estimation specific for IRT models. Subsequently, we conducted a simulation study covering multiple conditions in order to evaluate the performance of the proposed method. Ultimately, we showed that robust maximum marginal likelihood (RMML) estimation significantly improves detection rates for careless responders and reduces bias in item parameters across conditions. Furthermore, we applied our method to a real data set, to illustrate the utility of the proposed method. Our findings suggest that robust estimation coupled with person-fit statistics offers a powerful procedure to identify careless respondents for further review and to provide more accurate item parameter estimates in the presence of careless responses.

The Assessment and Impact of Careless Responding in Routine Outcome Monitoring within Mental Health Care

Careless responding by mental health patients on self-report assessments is rarely investigated in routine care despite the potential for serious consequences such as faulty clinical decisions. We investigated validity indices most appropriate for detecting careless responding in routine outcome monitoring (ROM) in mental health-care. First, we reviewed indices proposed in previous research for their suitability in ROM. Next, we evaluated six selected indices using data of the Brief Symptom Inventory and the Mood and Anxiety Symptom Questionnaire from 3,483 outpatients. Simulations showed that for typical ROM scales the Lmax index, Mahalanobis distance, and inter-item standard deviation may be too strongly confounded with the latent trait value to compare careless responding across patients with different symptom severity. Application of two different classification methods to the validity indices did not converge in similar prevalence estimates of careless responding. Finally, results suggest that careless responding does not have a substantial biasing effect on scale-score statistics. We recommend the lzp person-fit index to screen for random careless responding in large ROM data sets. However, additional research should further investigate methods for detecting repetitive responding in typical ROM data and assess whether there are specific circumstances in which simpler validity statistics or direct screening methods perform similarly as the lzp index.

Psychometric Evaluation of the BFI-10 and the NEO-FFI-3 in Indian Adolescents

The Five-Factor Model (FFM) is one of the most commonly examined constructs of personality across cultures in recent times. However, there is a lacuna of evidence for the suitability of FFM measures for Indian adolescent school students below the age of 17 years. We carried out two independent studies for the psychometric evaluation of the measures BFI-10 and NEO-FFI-3 on Indian adolescent school students. Both studies examined two socio-culturally distinct linguistic groups of secondary and senior secondary school students with a total sample of N = 1117 students. There was very limited support for a five-factor solution in both cases. Model fit was poor when applying FFM measures to our samples, whether applying confirmatory factor analysis or exploratory structural equation models. The results provide evidence against using adult personality measures with adolescents without separate psychometric validation and applying the Western age norms to Indian students without considering that the process of personality consolidation during adolescence may not be identical across cultures.

Know Your Heritage: Exploring the Effects of Fit in Cultural Knowledge on Chinese Canadians' Heritage Identification

In the present research, we introduce the notion of fit in cultural knowledge (FICK) – which we define as a match between the self and others in representing a cultural tradition. For ethnic minorities, FICK can be manifested in different degrees of matching their personal beliefs about their heritage culture with outgroup as well as ingroup beliefs about their heritage culture. We conducted two studies with the objective of exploring the potentially negative effects of FICK on Chinese Canadians’ heritage identification. In both studies, Chinese Canadian university students (N = 102; N = 156) indicated their personal beliefs about what values are normative in Chinese culture. Ingroup beliefs were assessed by beliefs about Chinese values that Chinese Canadians ascribed to their parents (Study 2), whereas outgroup beliefs were assessed by beliefs about Chinese values that were held by Euro-Canadians (Study 1) or that Chinese Canadians ascribed to Euro-Canadians (Study 2). The main findings based on a series of path models are as follows: (1) a stronger FICK generally predicted lower Chinese identification (centrality, ingroup ties, and affect), yet those negative effects were largely manifested in the openness to change versus conservation rather than in the self-transcendence versus self-enhancement value dimension. (2) The negative effects could be explained by Chinese Canadians’ experience of bicultural conflict (Study 1) and the frustration of continuity, meaning, and belonging identity motives (Study 2), suggesting that it matters which specific views of Chinese culture are matched in FICK. 3) Individuals who agreed with the perceived outgroup beliefs, and parental beliefs to a lesser extent, were more likely to apply the model minority stereotype to other Chinese Canadians (Study 2). Taken together, those findings demonstrate the challenges FICK presents to heritage identity maintenance among Chinese Canadian young adults. Implications for enculturation and cultural fit are discussed.

The Conscientious Responders Scale Helps Researchers Verify the Integrity of Personality Questionnaire Data

The Conscientious Responders Scale is a five-item embeddable validity scale that differentiates between conscientious and indiscriminate responding in personality-questionnaire data (CR & IR). This investigation presents further evidence of its validity and generalizability across two experiments. Study 1 tests its sensitivity to questionnaire length, a known cause of IR, and tries to provoke IR by manipulating psychological reactance. As expected, short questionnaires produced higher Conscientious Responders Scale scores than long questionnaires, and Conscientious Responders Scale scores were unaffected by reactance manipulations. Study 2 tests concerns that the Conscientious Responders Scale's unusual item content could potentially irritate and baffle responders, ironically increasing rates of IR. We administered two nearly identical questionnaires: one with an embedded Conscientious Responders Scale and one without the Conscientious Responders Scale. Psychometric comparisons revealed no differences across questionnaires' means, variances, interitem response consistencies, and Cronbach's alphas. In sum, the Conscientious Responders Scale is highly sensitive to questionnaire length-a known correlate of IR-and can be embedded harmlessly in questionnaires without provoking IR or changing the psychometrics of other measures.