Measuring teacher innovative behavior: a validated multidimensional inventory for use with public school teachers

Purpose

The role of Teacher Innovative Behavior (TIB), in responding to systemic problems in educational systems and promoting “intrapreneurial” behavior has been recognized in recent times. A robust instrument that can help administrators and teacher educators gauge the levels of TIB among their teachers will facilitate the promotion of innovative behavior.

Design/methodology/approach

This study tested a multidimensional innovative behavior inventory (IBI), innovation support inventory (ISI) and innovation output (IO) in a developing nation (India) context with public school teachers (n = 34,754), for reliability, validity, measurement invariance and structural invariance across caste, gender and subject groups.

Findings

The IBI, ISI and IO showed good reliability and validity along with full measurement invariance at configural, metric and scalar levels. With respect to the structural parameters, the inventories exhibited invariance of factor variance and covariance, but not of factor means.

Practical implications

Teacher innovative behavior (TIB) is seen by developing country education administrators as a tool to address difficult problems. With better measurement, it will be possible to identify teachers who need training in creativity and entrepreneurial behavior, teachers who might have developed innovative practices that could be used for teacher development, and ways of promoting competition among teachers.

Originality/value

The study validates inventories, which were earlier tested in non-educational domains, for use with public school teachers of a developing country across gender, caste and subject groups.

Large studies reveal how reference bias limits policy applications of self-report measures

There is growing policy interest in identifying contexts that cultivate self-regulation. Doing so often entails comparing groups of individuals (e.g., from different schools). We show that self-report questionnaires-the most prevalent modality for assessing self-regulation-are prone to reference bias, defined as systematic error arising from differences in the implicit standards by which individuals evaluate behavior. In three studies, adolescents (N = 229,685) whose peers performed better academically rated themselves lower in self-regulation and held higher standards for self-regulation. This effect was not observed for task measures of self-regulation and led to paradoxical predictions of college persistence 6 years later. These findings suggest that standards for self-regulation vary by social group, limiting the policy applications of self-report questionnaires.

Comparing the reliability and predictive power of child, teacher, and guardian reports of noncognitive skills

Children’s noncognitive or socioemotional skills (e.g., persistence and self-control) are typically measured using surveys in which either children rate their own skills or adults rate the skills of children. For many purposes—including program evaluation and monitoring school systems—ratings are often collected from multiple perspectives about a single child (e.g., from both the child and an adult). Collecting data from multiple perspectives is costly, and there is limited evidence on the benefits of this approach. Using a longitudinal survey, this study compares children’s noncognitive skills as reported by themselves, their guardians, and their teachers. Although reports from all three types of respondents are correlated with each other, teacher reports have the highest internal consistency and are the most predictive of children’s later cognitive outcomes and behavior in school. The teacher reports add predictive power beyond baseline measures of Intelligence Quotient (IQ) for most outcomes in schools. Measures collected from children and guardians add minimal predictive power beyond the teacher reports.

Personality traits and academic performance: Correcting self-assessed traits with vignettes

In this study, we investigate whether Conscientiousness, Emotional Stability and Risk Preference relate to student performance in higher education. We employ anchoring vignettes to correct for heterogeneous scale use in these non-cognitive skills. Our data are gathered among first-year students at a Dutch university. The results show that Conscientiousness is positively related to student performance, but the estimates are strongly biased upward if we use the uncorrected variables. We do not find significant relationships for Emotional Stability but find that the point estimates are larger when using the uncorrected variables. Measured Risk Preference is negatively related to student performance, yet this is fully explained by heterogeneous scale use. These results indicate the importance of using more objective measurements of personality traits.

The importance of development standards for anchoring vignettes: an illustrative example from pediatric localized scleroderma quality of life

PURPOSE

Anchoring vignettes (AVs) are a promising measurement technique to reduce bias in patient-reported outcome (PRO) measures by helping researchers understand differences in how individuals and groups interpret response options. However, little attention has been paid to ensure quality development of AVs, and their performance has not been well assessed in pediatric populations. In this study, we explore the application of a rigorous development process for AVs based upon current standards for PROs, as well as feasibility of AVs when administered to children and adolescents.

METHODS

We developed AVs using a rigorous, patient-centered mixed methods process including three phases: (1) development, (2) a pilot study, and (3) a field test. Our proposed process included the generation of a conceptual framework based on the PRO, the Localized Scleroderma Quality of Life Instrument, and numerous vignette-specific considerations. We qualitatively explored readability and comprehension of the AVs (pilot study) and then analyzed ranking patterns within vignette sets (field test).

RESULTS

Four sets of four vignettes were developed. Revisions were suggested at each phase of development. The pilot study demonstrated that children ≥ 10 years had no trouble indicating understanding of the AVs. In the field test, although appropriate rankings of vignettes were generally demonstrated by participants, the percentage of tied rankings was higher than expected in this pediatric group.

CONCLUSIONS

This work supports the need for rigorous developmental standards for AVs, as each stage of development suggested revisions. Additionally, AVs showed initial promise for use with pediatric populations; general feasibility and understanding were supported.

Associations between self-rated health and the assessments of anchoring vignettes in cardiovascular patients

Background/Objective

Assessments of health can be biased by response shift effects. One method for detecting such effects is the use of anchoring vignettes. The aim of this study was to analyze the relationship between participants' self-assessed health state and their assessments of these vignettes.

Method

A total of 342 cardiovascular patients assessed their own state of health on a 0-100 visual analogue scale. The patients additionally assessed two vignettes featuring fictional persons suffering from specific complaints. A sample of the general population (N = 1,236) served as controls.

Results

The participants rated the health state of the vignette character featuring physical problems as being significantly better than the general population did (effect size: d = 0.53). The group difference in the assessment of the vignette featuring primarily mental health problems was lower (d = -0.17). Participants' assessments of the vignettes were positively correlated with their assessments of their own health state (r = .26 and r = .10) and with several quality of life variables.

Conclusions

Anchoring vignettes are a useful tool for detecting response shift effects.

The Effects of Vignette Scoring on Reliability and Validity of Self-Reports

The research presented in this article combines mathematical derivations and empirical results to investigate effects of the nonparametric anchoring vignette approach proposed by King, Murray, Salomon, and Tandon on the reliability and validity of rating data. The anchoring vignette approach aims to correct rating data for response styles to improve comparability across individuals and groups. Vignettes are used to adjust self-assessment responses on the respondent level but entail significant assumptions: They are supposed to be invariant across respondents, and the responses to vignette prompts are supposed to be without error and strictly ordered. This article shows that these assumptions are not always met and that the anchoring vignette approach leads to higher Cronbach's alpha values and increased correlations among adjusted variables regardless of whether the assumptions of the approach are met or violated. Results suggest that the underlying assumptions and effects of the anchoring vignette approach should be carefully examined as the increased correlations and reliability estimates can be observed even for response variables that are independent random draws and uncorrelated with any other variable.

Using Anchoring Vignettes to Adjust Self-Reported Personality: A Comparison Between Countries

Data from self-report tools cannot be readily compared between cultures due to culturally specific ways of using a response scale. As such, anchoring vignettes have been proposed as a suitable methodology for correcting against this difference. We developed anchoring vignettes for the Big Five Inventory-44 (BFI-44) to supplement its Likert-type response options. Based on two samples (Rwanda: n = 423; Philippines: n = 143), we evaluated the psychometric properties of the measure both before and after applying the anchoring vignette adjustment. Results show that adjusted scores had better measurement properties, including improved reliability and a more orthogonal correlational structure, relative to scores based on the original Likert scale. Correlations of the Big Five Personality Factors with life satisfaction were essentially unchanged after the vignette-adjustment while correlations with counterproductive were noticeably lower. Overall, these changed findings suggest that the use of anchoring vignette methodology improves the cross-cultural comparability of self-reported personality, a finding of potential interest to the field of global workforce research and development as well as educational policymakers.

A Note on Construct Validity of the Anchoring Method in PISA 2012

We examine construct validity of the anchoring method used with 12 noncognitive scales from the Programme for International Student Assessment (PISA) 2012 project. This method combines individuals’ responses to vignettes and self-rated scores based on Likert-type items. It has been reported that the use of anchoring vignettes can reverse country-level correlations between academic achievement scores and noncognitive measures from negative to positive, and therefore align them with the typically reported individual-level correlations. Using the PISA 2012 data, we show that construct validity of this approach may be open to question because the anchored scales produce a different set of latent dimensions in comparison with nonanchored scales, even though both scales were created from the same set of individual responses. We also demonstrate that only one of three vignettes may be responsible for the resolution of the “paradox” highlighting that the choice of vignettes may be more important than what was previously reported.

Effects of Anchoring Vignettes on Comparability and Predictive Validity of Student Self-Reports in 64 Cultures

Anchoring vignettes are item batteries especially designed for correcting responses that might be affected by incomparability. This article investigates the effects of anchoring vignettes on the validity of student self-report data in 64 cultures. Using secondary data analysis from the 2012 Programme for International Student Assessment (PISA), we checked the validity of ratings on vignette questions, and investigated how rescaled item responses of two student scales, Teacher Support and Classroom Management, enhanced comparability and predictive validity. The main findings include that (a) responses to vignette questions represent valid individual and cultural differences; in particular, violations in these responses (i.e., misorderings) are related to low socioeconomic status and low cognitive sophistication; (b) the rescaled responses tend to show higher levels of comparability; and (c) the associations of rescaled Teacher Support and Classroom Management with math achievement, Student-Oriented Instruction, and Teacher-Directed Instruction are slightly different from raw scores of the two target constructs, and the associations with rescaled scores seem to be more in line with the literature. Namely, the associations among all self-report Likert-type scales are weaker with rescaled scores, presumably reducing common method variance, and both rescaled scale scores are more positively related to math achievement. The country ranking also changes substantially; in particular, Asian cultures top the ranking on Teacher Support after rescaling. However, anchoring vignettes are not a cure-all in solving measurement bias in cross-cultural surveys; we discuss the technicality and directions for further research on this technique.