Effects of situation descriptions on the construct-related validity of construct-driven situational judgment tests

Presenting the direct intercultural effectiveness simulation: an implicit trait policy on intercultural competence

Background

An implicit trait policy (ITP) represents the interaction between a personal disposition and general domain knowledge on how to effectively handle a specific (intercultural) situation. Such an ITP is a proven construct to create instruments that can predict future effective behavior. Moreover, such a simulation can provide valuable proxies for actual (future) behavior, as measures of (future) real life intercultural interactions are not always available.

Methods

In a series of three studies (N₁ = 224, N₂ = 291, N₃ = 478), the present research introduces a "Direct Intercultural Effectiveness Simulation" or DIES, an instrument that simulates intercultural effectiveness by directly tapping into an ITP on intercultural competence.

Results

First and foremost, the present research demonstrates that the DIES instrument generates reliable and construct-valid measures of intercultural effectiveness. Second, the DIES instrument also shows expected converging and diverging patterns when correlated with a nomological network on intercultural effectiveness. And third, the DIES measure is further validated by integration into an ITP framework of intercultural effectiveness based on theoretical and empirical accounts from literature.

Conclusion

The DIES instrument generates a reliable and valid measure of intercultural effectiveness by tapping into an ITP on intercultural competence. Theoretically, the present research integrates the instrument into literature by empirically verifying an ITP framework of intercultural effectiveness. In practice, the DIES instrument can be used as an awareness or training proxy for actual behavior to tackle important problems like ethnic prejudice and discrimination.

Use of situational judgment tests for assessing non-cognitive attributes of final year dental students

OBJECTIVE

Situational judgment tests (SJT) have been previously used in the admission process of medical and dental undergraduate programs, but their utilization as an assessment tool is fairly new. The objective of this study was to explore essential non-cognitive attributes of dental graduates. We also aimed to assess the attributes of final year dental students utilizing SJT and later collected participants' reactions on the test.

METHODS

A mixed-method exploratory sequential study was conducted. One-on-one and group discussions were done with subject matter experts (SMEs) to explore the essential attributes of fresh dental graduates. Based on the thematic analysis, 10 SJT items per domain were constructed by one group of SMEs and expert validated by a different group of SMEs and vice versa. The final dental-SJT was piloted online on final year dental students and their perceptions were collected using a five-point Likert scale questionnaire. Descriptive statistics and Cronbach's alpha were calculated for the test.

RESULTS

The consultation with SMEs generated three main themes, namely, Conventional skills, Tribal skills, and Occupational skills. The 70-item dental-SJT piloted on 150 final year dental students revealed data were normally distributed. The internal reliability of test was good (0.738). Students scored highest in "management skills" (>70%) and lowest in "professional ethics" (56%) and "diverse experiences" (58%). The overall student perception about the test was good.

CONCLUSION

SJT is an effective tool to assess non-cognitive attributes of dental students. It has an overall positive perception about perceived predictive validity, perceived fairness, face and content validity.

On the Construct-Related Validity of Implicit Trait Policies

In response to recent calls to incorporate Implicit Trait Policies (ITPs) into personality research, the current study examined the construct-related validity of ITP measures. ITPs are defined as implicit beliefs about the effectiveness of behaviors that reflect a certain trait. They are assessed by utilizing the methodology of Situational Judgment Tests. We empirically examined ( N = 339) several underlying key assumptions of ITP theory, including trait-specificity, the relation to personality traits, their context-independence, and the relation to general domain knowledge. Overall, our results showed little support for these assumptions. Although we found some confirmation for expected correlations between ITPs and personality traits, most of the observed variance in ITP measures was either method-specific or due to measurement error. We conclude that the herein examined ITP measures lack construct-related validity and discuss implications for ITP theory and assessment.