Agree or Disagree? Cognitive Processes in Answering Contrastive Survey Questions

An evaluation of the psychometric properties of the Australian Collaborative Practice Assessment Tool

OBJECTIVES

This study aimed to validate the Collaborative Practice Assessment Tool (CPAT) in the Australian setting and provide a quality instrument in terms of psychometric properties that can be used to measure interprofessional outcomes for both healthcare practitioners and students. The outcomes evaluated include the capacity to work in an interprofessional team, good interprofessional communication skills, leadership skills, ensuring clear division of tasks and roles in a team, effective conflict management, and being actively involved with patients and their families/communities in care.

METHODS

The COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments) taxonomy and standards were used as guides for evaluating the psychometric properties of the Australian CPAT, which include evaluations regarding instrument development requirements of sample target and size, content validity, internal structure (structural validity, internal consistency reliability and measurement invariance), and hypotheses testing. CPAT Australia was developed through two stages involving pilot studies and a validation study, both of which included healthcare practitioners and students as participants. A pilot study examined content validity regarding item relevance, item comprehensibility, and instrument comprehensiveness. The validation study was carried out to assess the internal structure of CPAT Australia for aspects of structural validity, internal consistency reliabilities, and configural, metric and scalar measurement invariance. The structural validity was explored using the following three steps: exploratory, confirmatory, and multi-group factor analysis. Construct validity was evaluated to confirm direct and indirect paths of assumptions based on a previously validated model. Data collected between August 2021 and May 2022.

RESULTS

The content validity evaluation confirmed that all items were relevant, understandable and comprehensive for measuring interprofessional collaborative care in Australia. Three hundred ninety-nine participants contributed to the validation study (n=152 practitioners; n=247 students). The original instrument model of 8-Factor 56-Item was improved in the Australian CPAT. Two items, Item 27 (Physicians assume the ultimate responsibility) and Item 49 (Final decision rest with the physician), were consistently rejected and therefore discarded. The internal structure of the 7-Factor 54-Item solution was confirmed as a suitable model with fit indices meeting COSMIN standards for a good model in practitioner and student cohorts. Configural, metric and scalar invariances were confirmed, indicating the invariance of the instruments when used for the practitioner and student cohorts. The construct validity evaluation indicated that 81.3% of direct and indirect assumptions were accepted, fulfilling the COSMIN requirement of >75% of proposed assumptions being accepted.

CONCLUSION

The Australian CPAT with a 7-factor 54-item solution was confirmed as a quality measure for assessing interprofessional education and collaborative practice for both healthcare practitioners and students in Australia with robust psychometric properties.

Community perceptions matter: a mixed-methods study using local knowledge to define features of success for a community intervention to improve quality of care for children under-5 in Jigawa, Nigeria

OBJECTIVES

In this study, we used the information generated by community members during an intervention design process to understand the features needed for a successful community participatory intervention to improve child health.

DESIGN

We conducted a concurrent mixed-methods study (November 2019-March 2020) to inform the design and evaluation of a community-facility linkage participatory intervention.

SETTING

Kiyawa Local Government Area (Jigawa State, Nigeria)-population of 230 000 (n=425 villages).

PARTICIPANTS

Qualitative data included 12 community conversations with caregivers of children under-5 (men, older and younger women; n=9 per group), 3 focus group discussions (n=10) with ward development committee members and interviews with facility heads (n=3). Quantitative data comprised household surveys (n=3464) with compound heads (n=1803) and women (n=1661).

RESULTS

We analysed qualitative data with thematic network analysis and the surveys with linear regression-results were triangulated in the interpretation phase. Participants identified the following areas of focus: community health education; facility infrastructure, equipment and staff improvements; raising funds to make these changes. Community involvement, cooperation and empowerment were recognised as a strategy to improve child health, and the presence of intermediate bodies (development committees) was deemed important to improve communication and solve problems between community and facility members. The survey showed functional community relations' dynamics, with high levels of internal cohesion (78%), efficacy in solving problems together (79%) and fairness of the local leaders (82%).

CONCLUSIONS

Combining the results from this study and critical theories on successful participation identified community-informed features for a contextually tailored community-facility link intervention. The need to promote a more inclusive approach to future child health interventions was highlighted. In addition to health education campaigns, the relationship between community and healthcare providers needs strengthening, and development committees were identified as an essential feature for successfully linking communities and facilities for child health.

TRIAL REGISTRATION NUMBER

ISRCTN39213655.

Wording Effects in Assessment: Missing the Trees for the Forest

This article examines wording effects when positive and negative worded items are included in psychological assessment. Wordings effects have been analyzed in the literature using statistical approaches based on population homogeneity assumptions (i.e. CFA, SEM), commonly adopting the bifactor model to separate trait variance and wording effects. This article presents an alternative approach by explicitly modeling population heterogeneity through a latent profile model, based on the idea that a subset of individuals exhibits wording effects. This kind of mixture model allows simultaneously to classify respondents, substantively characterize the differences in their response profiles, and report respondents' results in a comparable manner. Using the Rosenberg's self-esteem scale data from the LISS Panel (N = 6,762) in three studies, we identify a subgroup of participants who respond differentially according to item-wording and examine the impact of its responses in the estimation of the RSES measurement model, in terms of global and individual fit, under one-factor and bifactor models.The results of these analyses support the interpretation of wording effects in terms of a theoretically-proposed differential pattern of response to positively and negatively worded items, introducing a valuable tool for examining the artifactual or substantive interpretations of such wording effects.

Do People Perceive the Disagreement in Straw Man Fallacies? An Experimental Investigation

So far, experimental studies on the straw man have targeted the misrepresentational dimension of this fallacy. In order to provide a more detailed understanding of the way the straw man is perceived, the focus of this paper lies on the refutational dimension. In two experiments, I will assess (1) if people are sensitive to the underlying disagreement expressed through the use of a straw man and (2) if question wording plays a role for the perception of disagreement. The results of the experiment show that participants indeed notice easily that the person performing a straw man disagrees with his opponent. It also emerges from the experiment that the difference between a positive or negative formulation of the experimental questions does not affect the perception of disagreement in the straw man. The underlying disagreement in the straw man is thus perceived either way.

Barriers and facilitators to clinical implementation of radiotherapy treatment planning automation: A survey study of medical dosimetrists

PURPOSE

Little is known about the scale of clinical implementation of automated treatment planning techniques in the United States. In this work, we examine the barriers and facilitators to adoption of commercially available automated planning tools into the clinical workflow using a survey of medical dosimetrists.

METHODS/MATERIALS

Survey questions were developed based on a literature review of automation research and cognitive interviews of medical dosimetrists at our institution. Treatment planning automation was defined to include auto-contouring and automated treatment planning. Survey questions probed frequency of use, positive and negative perceptions, potential implementation changes, and demographic and institutional descriptive statistics. The survey sample was identified using both a LinkedIn search and referral requests sent to physics directors and senior physicists at 34 radiotherapy clinics in our state. The survey was active from August 2020 to April 2021.

RESULTS

Thirty-four responses were collected out of 59 surveys sent. Three categories of barriers to use of automation were identified. The first related to perceptions of limited accuracy and usability of the algorithms. Eighty-eight percent of respondents reported that auto-contouring inaccuracy limited its use, and 62% thought it was difficult to modify an automated plan, thus limiting its usefulness. The second barrier relates to the perception that automation increases the probability of an error reaching the patient. Third, respondents were concerned that automation will make their jobs less satisfying and less secure. Large majorities reported that they enjoyed plan optimization, would not want to lose that part of their job, and expressed explicit job security fears.

CONCLUSION

To our knowledge this is the first systematic investigation into the views of automation by medical dosimetrists. Potential barriers and facilitators to use were explicitly identified. This investigation highlights several concrete approaches that could potentially increase the translation of automation into the clinic, along with areas of needed research.

The Impact of Item Feature and Response Preference in a Mixed-Format Design

A combination of positively and negatively worded items (termed a mixed-format design) has been widely adopted in personality and attitude assessments. While advocates claim that the inclusion of positively and negatively worded items will encourage respondents to process the items more carefully and avoid response preference, others have reported that negatively worded (NW) items may induce a nuisance factor and contaminate scale scores. The present study examined the extent of the impact of the NW-item feature and further investigated whether a mixed-format design could effectively control acquiescence and the preference for extreme response options using two datasets (Attitude toward Peace Walls, and International Personality Item Pool). A proposed multidimensional item response model was implemented to simultaneously estimate the impact of item feature and response preference. The results suggested that NW items induced an impact on item responses and that affirmative preference was negligible, regardless of the proportion of NW items in a scale. However, participants' extremity preference was large in both balanced and imbalanced mixed-format designs. It concludes that the impact of the NW-item feature is not negligible in a mixed-format scale, which exhibits good control of acquiescence but not extremity preference.

The Influence of Item Characteristics on Acquiescence among Latino Survey Respondents

Acquiescence is often defined as the systematic selection of agreeable ("strongly agree") or affirmative ("yes") responses to survey items, regardless of item content or directionality. This definition implies that acquiescence is immune to item characteristics; however, the influence of item characteristics on acquiescence remains largely unexplored. We examined the influence of eight item characteristics on acquiescence in a telephone survey of 400 Latinos and non-Latino whites: qualified wording, mental comparisons, negated wording, unfamiliar terms, ambiguous wording, knowledge accessibility, item length, and polysyllabic wording. Negated and ambiguous wording was associated with reduced acquiescence for the full sample, as well as subsamples stratified by ethnicity and sociodemographic characteristics. This effect was strongest among younger, more educated, and non-Latino white respondents. No other item characteristics had a significant influence on respondent acquiescence. Findings from this study suggest that acquiescence may be affected by interactions between respondent and item characteristics.

Method effects associated with negatively and positively worded items on the 12-item General Health Questionnaire (GHQ-12): results from a cross-sectional survey with a representative sample of Catalonian workers

OBJECTIVE

Recent studies into the factorial structure of the 12-item version of the General Health Questionnaire (GHQ-12) have shown that it was best represented by a single substantive factor when method effects associated with negatively worded (NW) items are considered. The purpose of the present study was to examine the presence of method effects, and their relationships with demographic covariates, associated with positively worded (PW) and/or NW items.

DESIGN

A cross-sectional, observational study to compare a comprehensive set of confirmatory factor models, including method effects associated with PW and/or NW items with GHQ-12 responses.

SETTING

Representative sample of all employees living in Catalonia (Spain).

PARTICIPANTS

3050 participants (44.6% women) who responded the Second Catalonian Survey of Working Conditions.

RESULTS

A confirmatory factor analysis showed that the best fitting model was a unidimensional model with two additional uncorrelated method factors associated with PW and NW items. Furthermore, structural equation modelling (SEM) revealed that method effects were differentially related to both the sex and age of the respondents.

CONCLUSION

Individual differences related to sex and age can help to identify respondents who are prone to answering PW and NW items differently. Consequently, it is desirable that both the constructs of interest as well as the effects of method factors are considered in SEM models as a means of avoiding the drawing of inaccurate conclusions about the relationships between the substantive factors.

A good tennis player does not lose matches. The effects of valence congruency in processing stance-argument pairs

According to the principle of Argumentative Orientation (AO), speakers and writers adjust their frame choice to the opinion they want to convey and hearers and readers are sensitive to this profile choice. In three reaction time studies (N = 68; N = 97; N = 60) we investigated whether, in line with AO, stance-argument pairs congruent in valence are easier to process and to verify than incongruent pairs. Second, we tested whether, in line with predictions from the Markedness Principle (MP), positive congruent pairs are easier to process than negative congruent pairs. In line with AO, participants made faster and more accurate judgments of congruent pairs than of incongruent pairs. This effect was observed when controlling for word length and word frequency, and occurred irrespective of the distance between the evaluative word in the stance and argument. No unambiguous effect of Markedness was found.

The Relative Power of Negativity: The Influence of Language Intensity on Perceived Strength

Negative utterances and words have been found to be stronger than positive utterances and words, but what happens if positive and negative utterances are intensified? Two online experiments were carried out in which participants judged the strength of (un)intensified positive and negative evaluations in written dialogues. Both studies showed intensified language was perceived as stronger than unmarked language (i.e., language that was not intensified), and negative evaluations were stronger than positive evaluations. What is more, intensification and polarity interact; the increment of perceived strength for intensified positive adjectives (Study 1) and purely intensified adverbs (really, very; Study 2) was bigger than the increment in perceived strength of intensified negative adjective and adverbs. When a meaningful intensifier (deliciously, disgustingly) was used, the negativity effect remained. The findings were discussed within cognitive frameworks such as relevance theory, theory of mind, and theory on verbal aggression.

Positive vs. Negative: The Impact of Question Polarity in Voting Advice Applications

Online Voting Advice Applications (VAAs) are survey-like instruments that help citizens to shape their political preferences and compare them with those of political parties. Especially in multi-party democracies, their increasing popularity indicates that VAAs play an important role in opinion formation for citizens, as well as in the public debate prior to elections. Hence, the objectivity and transparency of VAAs are crucial. In the design of VAAs, many choices have to be made. Extant research in survey methodology shows that the seemingly arbitrary choice to word questions positively (e.g., 'The city council should allow cars into the city centre') or negatively ('The city council should ban cars from the city centre') systematically affects the answers. This asymmetry in answers is in line with work on negativity bias in other areas of linguistics and psychology. Building on these findings, this study investigated whether question polarity also affects the answers to VAA statements. In a field experiment (N = 31,112) during the Dutch municipal elections we analysed the effects of polarity for 16 out of 30 VAA statements with a large variety of linguistic contrasts. Analyses show a significant effect of question wording for questions containing a wide range of implicit negations (such as 'forbid' vs. 'allow'), as well as for questions with explicit negations (e.g., 'not'). These effects of question polarity are found especially for VAA users with lower levels of political sophistication. As these citizens are an important target group for Voting Advice Applications, this stresses the need for VAA builders to be sensitive to wording choices when designing VAAs. This study is the first to show such consistent wording effects not only for political attitude questions with implicit negations in VAAs, but also for political questions containing explicit negations.

The effects of authentic leadership and organizational commitment on turnover intention

Purpose

– The purpose of this paper is to explore whether authentic leadership in hospitality is composed of four distinctive but related substantive components (i.e. self-awareness, relational transparency, balanced processing, and internalized moral); the impact of authentic leadership on employees’ organizational commitment (OC); the impact of employees’ OC on their turnover intention (TI); and the indirect effect of authentic leadership on employees’ TI via OC.

Design/methodology/approach

– The authors tested a sample of 236 students working as employees in hospitality in the USA, on the idea that authentic leadership increases OC which in turn decreases TI. The participants were asked to rate the manager’s leadership style and the frequency of their leadership behavior.

Findings

– Results of structural equation modeling provide support for the positive effect of authentic leadership on OC in the hospitality industry, and suggest that OC mediates reduced TI.

Practical implications

– The findings in the present study are extremely useful to managers, human resource managers, and organizations as a whole. Practitioners looking to increase employee OC and decrease TI can do so by augmenting the authentic leadership qualities of managers.

Originality/value

– The results of this study suggests a variety of significant theoretical contributions as well as critical leadership and organizational implications. The effects of authentic leadership were empirically tested on employees’ OC and the effects of that OC on TI.

Learning and satisficing: an analysis of sequence effects in health valuation

OBJECTIVE

To estimate the effect of sequence on response precision and response behavior in health valuation studies.

METHODS

Time trade-off (TTO) and paired comparison responses from six health valuation studies-four US, one Spanish, and one Dutch-were examined (22,225 respondents) to test whether task sequence influences response precision (e.g., rounding), response changes, and median response times. Each study used a computer-based instrument that randomized task sequence among a national sample of adults, age 18 years or older, from the general population.

RESULTS

For both TTO and paired comparisons, median response times decreased with sequence (i.e., learning), but tended to flatten after the first three tasks. Although the paired comparison evidence demonstrated that sequence had no effect on response precision, the frequency of rounded TTO responses (to either 1-year or 5-year units) increased with sequence.

CONCLUSIONS

Based on these results, randomizing or reducing the number of paired comparison tasks does not appear to influence response precision; however, generalizability, practicality, and precautionary considerations remain. Overall, participants learned to respond efficiently within the first three tasks and did not resort to satisficing, but may have rounded their TTO responses.

Item Response Theory Models for Wording Effects in Mixed-Format Scales

Many scales contain both positively and negatively worded items. Reverse recoding of negatively worded items might not be enough for them to function as positively worded items do. In this study, we commented on the drawbacks of existing approaches to wording effect in mixed-format scales and used bi-factor item response theory (IRT) models to test the assumption of reverse coding and evaluate the magnitude of the wording effect. The parameters of the bi-factor IRT models can be estimated with existing computer programs. Two empirical examples from the Program for International Student Assessment and the Trends in International Mathematics and Science Study were given to demonstrate the advantages of the bi-factor approach over traditional ones. It was found that the wording effect in these two data sets was substantial and that ignoring the wording effect resulted in overestimated test reliability and biased person measures.