The TeamSTEPPS for Improving Diagnosis Team Assessment Tool: Scale Development and Psychometric Evaluation

INTRODUCTION

One in three patients is affected by diagnosis-related communication failures. Only a few valid and reliable instruments that measure teamwork and communication exist, and none of those focus on improving diagnosis. The authors developed, refined, and psychometrically evaluated the TeamSTEPPSⓇ for Improving Diagnosis Team Assessment Tool (TAT), which assesses diagnostic teamwork and communication in five critical teamwork domains and can be used to identify strengths and opportunities for improvement and monitor performance.

METHODS

The TAT was administered as a cross-sectional survey to 360 health professionals across nine diverse US health systems. Content and construct validity were evaluated through pilot implementation and subject matter expert review. Reliability and internal consistency were assessed with Cronbach's alpha. To understand sources of variation in TAT scores and assess the tool's consistency across diverse health care organizations, generalizability theory (G-theory) was used. Best practices in screening for careless responding identified participants with random or nonvarying responses.

RESULTS

Analyses indicated strong support for the tool. Content validity findings indicated that the TAT encompassed relevant diagnostic improvement teamwork and communication content. Construct validity, evaluated through pilot implementations, demonstrated the tool's effectiveness in assessing teamwork categories. Reliability analyses confirmed the TAT's internal consistency, with an overall Cronbach's alpha of 0.97. Each dimension of the TAT exhibited good reliability coefficients, ranging from 0.83 to 0.95. G-theory analysis showed that variations in TAT scores were primarily attributed to respondents (28.0%) and scale dimensions (59.6%); both are desirable facets of variation. Further, examination of careless respondents ensured the accuracy and quality of the results, enhancing the TAT's credibility as a valuable diagnostic improvement tool.

CONCLUSION

Psychometric evaluation demonstrated that the TAT is a reliable and valid instrument for assessing teamwork and communication among and across diagnostic teams. The TAT adds a novel, evidence-based, psychometrically sound measurement tool to help advance diagnostic teamwork and communication to improve patient care and outcomes.

Evaluation of a Measurement System to Assess ICU Team Performance

OBJECTIVE

Measuring teamwork is essential in critical care, but limited observational measurement systems exist for this environment. The objective of this study was to evaluate the reliability and validity of a behavioral marker system for measuring teamwork in ICUs.

DESIGN

Instances of teamwork were observed by two raters for three tasks: multidisciplinary rounds, nurse-to-nurse handoffs, and retrospective videos of medical students and instructors performing simulated codes. Intraclass correlation coefficients were calculated to assess interrater reliability. Generalizability theory was applied to estimate systematic sources of variance for the three observed team tasks that were associated with instances of teamwork, rater effects, competency effects, and task effects.

SETTING

A 15-bed surgical ICU at a large academic hospital.

SUBJECTS

One hundred thirty-eight instances of teamwork were observed. Specifically, we observed 88 multidisciplinary rounds, 25 nurse-to-nurse handoffs, and 25 simulated code exercises.

INTERVENTIONS

No intervention was conducted for this study.

MEASUREMENTS AND MAIN RESULTS

Rater reliability for each overall task ranged from good to excellent correlation (intraclass correlation coefficient, 0.64-0.81), although there were seven cases where reliability was fair and one case where it was poor for specific competencies. Findings from generalizability studies provided evidence that the marker system dependably distinguished among teamwork competencies, providing evidence of construct validity.

CONCLUSIONS

Teamwork in critical care is complex, thereby complicating the judgment of behaviors. The marker system exhibited great potential for differentiating competencies, but findings also revealed that more context specific guidance may be needed to improve rater reliability.

Perspectives on the Relationship Between Job Performance and Ratings of Job Performance

The comments and suggestions prompted by K. R. Murphy’s (2008) description of alternate models of the relationship between job performance and ratings of job performance reflect 3 broad themes: (a) the relationship between performance appraisal and performance measurement, (b) the best psychometric models for understanding performance ratings, and (c) the appropriateness of static measures for dynamic phenomena. This paper comments on these 3 themes and suggests directions for future research and practice in performance appraisal that focuses on rater goals, organizational interventions to improve the accuracy and value of ratings, and assessments of the value of performance ratings as criteria.

Assessing measurement equivalence across rating sources: a multitrait-multirater approach

Research examining the structure of multisource performance ratings has demonstrated that ratings are a direct function of both who is doing the rating (rating source) as well as what is being rated (performance dimension). A separate line of research has focused on the extent to which performance ratings are equivalent across sources. To date no research has examined the measurement equivalence of multisource ratings within the context of both dimension and rating source direct effects on ratings. We examine the impact of both performance dimension and rating source as well as the degree of measurement equivalence across sources. Results indicate that (a) the impact of the underlying performance dimension is the same across rating sources, (b) the impact of rating source is substantial and only slightly smaller than the impact of the underlying performance dimension, and (c) the impact of rating source differs substantially depending on the source.

Trying to understand the different pieces of the construct validity puzzle of assessment centers: an examination of assessor and assessee effects

This study examined the effects of assessor-related factors (i.e., type of assessor) and assessee-related factors (i.e., type of assessee profile) on the construct validity of assessment center ratings. In particular, 3 types of assessors (26 industrial/organizational [I/O] psychologists, 20 managers, and 27 students), rated assessee performances that varied according to cross-exercise consistency (i.e., relatively inconsistent vs. relatively consistent) and dimension differentiation (relatively undifferentiated vs. relatively differentiated). Construct validity evidence was established for only one assessee profile and only in the I/O psychologist and managerial samples. More generally, these results indicate that 3 factors (poor design, assessor unreliability, and especially cross-situational inconsistent assessee performances) may explain why construct validity evidence is often not established in operational assessment centers.

Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity

This study compares the effects of data-driven assessor training with schema-driven assessor training and control training. The sample consisted of 229 industrial and organizational psychology students and 161 managers who were randomly assigned to 1 of these training strategies. Participants observed and rated candidates in an assessment center exercise. The data-driven and schema-driven assessor training approaches outperformed the control training on all 3 dependent variables. The schema-driven assessor training resulted in the largest values of interrater reliability, dimension differentiation, and accuracy. Managers provided significantly more accurate ratings than students but distinguished less between the dimensions. Practical implications regarding the design of assessor trainings and the composition of assessor teams are proposed.