The Teamwork Mini-Clinical Evaluation Exercise (T-MEX): a workplace-based assessment focusing on collaborative competencies in health care

Does participation in sports competitions enhance interprofessional teamwork among medical students? Evidence from a medical school curriculum experiment

BACKGROUND

Effective interprofessional teamwork is essential for the efficiency, safety and quality of healthcare system services and requires interprofessional education for medical students. Physical education is a simple and easy way to teach teamwork, which translates into team performance in the work environment. This study was conducted to examine the effectiveness of the physical education competition model, instead of the exams model, for improving teamwork skills among medical students.

METHODS

A quasiexperimental intervention design was used to measure the effect of a 16-week cheerleading programme on subjects' teamwork skills by completing a teamwork scale comprising four subdimensions, namely, personal characteristics, teamwork, leadership, and conflict management, before the start and at the end of the programme, and by comparing nonwinning to winning students to measure the effect of teamwork skills on team performance.

RESULTS

A total of 179 students completed the valid baseline and posttest (effective rate = 95.21%). The teamwork scale scores (B M = 4.81, R M = 5.05, p < 0.001) and 4 subdimension scores (personal characteristics p = 0.002, teamwork p = 0.028, leadership p < 0.001, conflict management p < 0.001) were statistically significant. Twenty-two of the 44 items in the scale improved significantly. The differences between students who won the competition and those who did not (N M=4.86, W M=5.14, p<0.01) were statistically significant, with no significant differences in personal characteristics p = 0.183; significant differences in the 3 subdimensions of teamwork p < 0.01, leadership p = 0.024, and conflict management p = 0.037; and a significant increase in 13 out of 44 self-efficacy items on the scale.

CONCLUSIONS

The "race for exams" physical education programme improved teamwork among medical students, and increased teamwork improved team performance. The "competition instead of examination" physical education programme provides a quantifiable method for improving interprofessional teamwork among medical students.

Collaboration cognizance: Development of a self-assessment tool to measure intra-professional collaborative practices (IPCP) in postgraduate medical residents at tertiary care hospitals

BACKGROUND

The ever-evolving healthcare system of today demands physicians who steer their roles as treatment providers, managers and collaborators. Professionals are highly interdependent due to increased complexity of health problems and risk of errors increases with transitions in care. In hospitals, the main workforce is postgraduate residents; therefore, intraprofessional collaboration amongst residents is essential. Several instruments are available to evaluate interprofessional collaboration amongst physicians, nurses and hospital teams but none specifically assessed intra-professional collaborative practices amongst residents working in tertiary care hospitals in multi-disciplinary teams. This study intends to develop and validate an instrument to self-assess intraprofessional collaborative practices in postgraduate residents undergoing residency in multiple specialties at tertiary care hospitals.

APPROACH

This study on Instrument Development employed mixed method study design executed in two phases. In phase 1, six domains of intraprofessional collaborative practices were identified from literature and 35 items were developed. Fifteen experts participated in qualitative content validation and provided comments. To establish content validity in phase 2, content validity index (CVI) and content clarity average (CCA) were assessed by seventeen experts. Response process validity was established by cognitive interviewing of 5 postgraduate residents. Pilot testing was done on a sample of 407 residents. Cronbach's alpha was determined, and confirmatory factor analysis established construct validity.

RESULTS

During phase 1, items were modified based on qualitative feedback from 15 experts. In round 2, CVI and CCA were determined based on responses of 17 experts. The items having an I-CVI greater than 0.90 were accepted and six items underwent modifications as their I-CVI fell between 0.78 and 0.90. Similarly, four items with a CCA of less than 2.4 were modified to increase clarity. Cognitive interviews of participants on 30 items resulted in the deletion of 1 item and changes in 5 items. The final instrument had 29 items categorized under six constructs. All items had good factor loadings during CFA, so none was deleted. Cronbach's Alpha α was 0.937.

CONCLUSION

Intraprofessional collaborative practices in residents is a valid and reliable self-assessment tool comprising 29 items measuring six constructs. It may be used by residents to assess their collaborative practices and incorporated in curricula to help develop collaborative practices and their assessment during training of postgraduate residents.

A study of validity and usability evidence for non-technical skills assessment tools in simulated adult resuscitation scenarios

BACKGROUND

Non-technical skills (NTS) assessment tools are widely used to provide formative and summative assessment for healthcare professionals and there are now many of them. This study has examined three different tools designed for similar settings and gathered evidence to test their validity and usability.

METHODS

Three NTS assessment tools designed for use in the UK were used by three experienced faculty to review standardized videos of simulated cardiac arrest scenarios: ANTS (Anesthetists' Non-Technical Skills), Oxford NOTECHS (Oxford NOn-TECHnical Skills) and OSCAR (Observational Skill based Clinical Assessment tool for Resuscitation). Internal consistency, interrater reliability and quantitative and qualitative analysis of usability were analyzed for each tool.

RESULTS

Internal consistency and interrater reliability (IRR) varied considerably for the three tools across NTS categories and elements. Intraclass correlation scores of three expert raters ranged from poor (task management in ANTS [0.26] and situation awareness (SA) in Oxford NOTECHS [0.34]) to very good (problem solving in Oxford NOTECHS [0.81] and cooperation [0.84] and SA [0.87] in OSCAR). Furthermore, different statistical tests of IRR produced different results for each tool. Quantitative and qualitative examination of usability also revealed challenges in using each tool.

CONCLUSIONS

The lack of standardization of NTS assessment tools and training in their use is unhelpful for healthcare educators and students. Educators require ongoing support in the use of NTS assessment tools for the evaluation of individual healthcare professionals or healthcare teams. Summative or high-stakes examinations using NTS assessment tools should be undertaken with at least two assessors to provide consensus scoring. In light of the renewed focus on simulation as an educational tool to support and enhance training recovery in the aftermath of COVID-19, it is even more important that assessment of these vital skills is standardized, simplified and supported with adequate training.

Perceived positive social interdependence in online versus face-to-face team-based learning styles of collaborative learning: a randomized, controlled, mixed-methods study

BACKGROUND

Collaborative learning is a group learning approach in which positive social interdependence within a group is key to better learning performance and future attitudes toward team practice. Recent attempts to replace a face-to-face environment with an online one have been developed using information communication technology. However, this raises the concern that online collaborative learning (OCL) may reduce positive social interdependence. Therefore, this study aimed to compare the degree of social interdependence in OCL with face-to-face environments and clarify aspects that affect social interdependence in OCL.

METHODS

We conducted a crossover study comparing online and face-to-face collaborative learning environments in a clinical reasoning class using team-based learning for medical students (n = 124) in 2021. The participants were randomly assigned to two cohorts: Cohort A began in an online environment, while Cohort B began in a face-to-face environment. At the study's midpoint, the two cohorts exchanged the environments as a washout. The participants completed surveys using the social interdependence in collaborative learning scale (SOCS) to measure their perceived positive social interdependence before and after the class. Changes in the mean SOCS scores were compared using paired t-tests. Qualitative data related to the characteristics of the online environment were obtained from the focus groups and coded using thematic analysis.

RESULTS

The matched-pair tests of SOCS showed significant progression between pre- and post-program scores in the online and face-to-face groups. There were no significant differences in overall SOCS scores between the two groups. Sub-analysis by subcategory showed significant improvement in boundary (discontinuities among individuals) and means interdependence (resources, roles, and tasks) in both groups, but outcome interdependence (goals and rewards) improved significantly only in the online group. Qualitative analysis revealed four major themes affecting social interdependence in OCL: communication, task-sharing process, perception of other groups, and working facilities.

CONCLUSIONS

There is a difference in the communication styles of students in face-to-face and online environments, and these various influences equalize the social interdependence in a face-to-face and online environment.

Factors and Interactions Influencing Direct Observation: A Literature Review Guided by Activity Theory

PhenomenonEnsuring that future physicians are competent to practice medicine is necessary for high quality patient care and safety. The shift toward competency-based education has placed renewed emphasis on direct observation via workplace-based assessments in authentic patient care contexts. Despite this interest and multiple studies focused on improving direct observation, challenges regarding the objectivity of this assessment approach remain underexplored and unresolved. Approach: We conducted a literature review of direct observation in authentic patient contexts by systematically searching databases PubMed, Embase, Web of Science, and ERIC. Included studies comprised original research conducted in the patient care context with authentic patients, either as a live encounter or a video recording of an actual encounter, which focused on factors affecting the direct observation of undergraduate medical education (UME) or graduate medical education (GME) trainees. Because the patient care context adds factors that contribute to the cognitive load of the learner and of the clinician-observer we focused our question on such contexts, which are most useful in judgments about advancement to the next level of training or practice. We excluded articles or published abstracts not conducted in the patient care context (e.g., OSCEs) or those involving simulation, allied health professionals, or non-UME/GME trainees. We also excluded studies focused on end-of-rotation evaluations and in-training evaluation reports. We extracted key data from the studies and used Activity Theory as a lens to identify factors affecting these observations and the interactions between them. Activity Theory provides a framework to understand and analyze complex human activities, the systems in which people work, and the interactions or tensions between multiple associated factors. Findings: Nineteen articles were included in the analysis; 13 involved GME learners and 6 UME learners. Of the 19, six studies were set in the operating room and four in the Emergency department. Using Activity Theory, we discovered that while numerous studies focus on rater and tool influences, very few study the impact of social elements. These are the rules that govern how the activity happens, the environment and members of the community involved in the activity and how completion of the activity is divided up among the members of the community. Insights: Viewing direct observation via workplace-based assessment through the lens of Activity Theory may enable educators to implement curricular changes to improve direct observation of assessment. Activity Theory may allow researchers to design studies to focus on the identified underexplored interactions and influences in relation to direct observation.

Evaluation of Communication and Safety Behaviors During Hospital-Wide Code Response Simulation

INTRODUCTION

To understand the baseline quality of team communication behaviors at our organization, we implemented institution-wide simulation training and measured the performance of safety behaviors of ad hoc teams in emergent situations.

METHODS

Clinicians participated in 2 interprofessional video-recorded simulation scenarios, each followed by debriefing. Using a standardized evaluation instrument, 2 reviewers independently evaluated the presence or absence of desired team safety behaviors, including escalating care, sharing a mental model, establishing leadership, thinking out loud, and identifying roles and responsibilities. We also scored the quality of sharing the mental model, closed-loop communication, and overall team performance on a 7-point scale. Discordant reviews were resolved with scoring by an additional reviewer.

RESULTS

A total of 1404 clinicians participated in 398 simulation scenarios, resulting in 257 usable videos. Overall, teams exhibited desired behaviors at the following frequencies: escalating care, 85%; sharing mental models, 66%; verbally establishing leadership, 6%; thinking out loud, 87%; and identifying roles and responsibilities, 27%. Across all reviews, the quality of the graded behaviors (of 7 points) was 2.8 for shared mental models, 3.3 for closed-loop communication, and 3.2 for overall team performance.

CONCLUSIONS

In a simulation setting with ad hoc teams, there was variable performance on completing safety behaviors and only a fair quality of graded communication behaviors. These results establish a baseline assessment of communication and teamwork behaviors and will guide future quality improvement interventions.

Building a Shared Mental Model of Competence Across the Continuum: Trainee Perceptions of Subinternships for Residency Preparation

INTRODUCTION

Toward a vision of competency-based medical education (CBME) spanning the undergraduate to graduate medical education (GME) continuum, University of Minnesota Medical School (UMMS) developed the Subinternship in Critical Care (SICC) offered across specialties and sites. Explicit course objectives and assessments focus on internship preparedness, emphasizing direct observation of handovers (Core Entrustable Professional Activity, "EPA," 8) and cross-cover duties (EPA 10).

METHODS

To evaluate students' perceptions of the SICC's and other clerkships' effectiveness toward internship preparedness, all 2016 and 2017 UMMS graduates in GME training (n = 440) were surveyed regarding skill development and assessment among Core EPAs 1, 4, 6, 8, 9, 10. Analysis included descriptive statistics plus chi-squared and Kappa agreement tests.

RESULTS

Respondents (n = 147, response rate 33%) rated the SICC as a rotation during which they gained most competence among EPAs both more (#4, 57% rated important; #8, 75%; #10, 70%) and less explicit (#6, 53%; #9, 69%) per rotation objectives. Assessments of EPA 8 (80% rated important) and 10 (76%) were frequently perceived as important toward residency preparedness. Agreement between importance of EPA development and assessment was moderate (Kappa = 0.40-0.59, all surveyed EPAs).

CONCLUSIONS

Graduates' perceptions support the SICC's educational utility and assessments. Based on this and other insight from the SICC, the authors propose implications toward collectively envisioning the continuum of physician competency.

Faculty and Resident Engagement With a Workplace-Based Assessment Tool: Use of Implementation Science to Explore Enablers and Barriers

PURPOSE

Implementation of workplace-based assessment programs has encountered significant challenges. Faculty and residents alike often have a negative view of these programs as "tick-box" or "jump through the hoops" exercises. A number of recommendations have been made to address these challenges. To understand the experience with a workplace-based assessment tool that follows many of these recommendations, the authors conducted a qualitative study using the Consolidated Framework for Implementation Research (CFIR) to identify enablers and barriers to engagement with the tool.

METHOD

The Psychopharmacotherapy-Structured Clinical Observation (P-SCO) is a direct observation tool designed to assess resident performance during a psychiatric medication management visit. From August 2017 to February 2018, the P-SCO was implemented in the outpatient continuity clinics for second- and third-year residents at Zucker Hillside Hospital/Northwell Health. In February and March 2019, the authors conducted semistructured interviews of participating faculty and residents. Interview guides based on the CFIR were used to capture the enablers and barriers to engagement. Interview transcripts were independently coded. Codes were then organized into themes relevant to the domains of the CFIR.

RESULTS

Ten faculty and 10 residents were interviewed. Overall, participants had a positive experience with the P-SCO. Enabling factors for faculty and residents included the ongoing training, design features of the P-SCO, predisposing beliefs, dedicated faculty time, and the perception that the P-SCO improved verbal feedback quality. Barriers for faculty included checklist length and discomfort with feedback that threatens identity, and barriers for residents included faculty variability in timeliness and quality of feedback and minimal review of the feedback after initial receipt.

CONCLUSIONS

This study demonstrates that the negative experience of faculty and residents with workplace-based assessment tools shown in prior studies can be overcome, at least in part, when specific implementation strategies are pursued. The findings provide guidance for future research and implementation efforts.

Stimulating Students' Interprofessional Teamwork Skills Through Community-Based Education: A Mixed Methods Evaluation

Background

Interprofessional education (IPE) is suggested as a good means to prepare future healthcare professionals for collaborative work in interprofessional teams enabling them to solve complex health problems. Previous studies have advocated experiential IPE, including community-based IPE (CBIPE). This study aims to evaluate a CBIPE programme by exploring the students’ perception toward CBIPE design and toward groups’ teamwork.

Methods

To identify students’ perceptions of teamwork, the Interprofessional Teamwork Evaluation questionnaire was administered to 254 students of medical, nursing and midwifery programme. Three uni-professional focus group (FG) discussions were conducted to analyse the students’ perception of the design of community-based education and underlying reasons for teamwork.

Results

FGs reported three aspects that influence skills development in collaborative practice among students that shed light on why midwifery and nursing students held less positive perceptions of communication and mutual support: 1) communication gap due to lack of confidence, 2) contrasting ways of thinking affect communication in decision-making, and 3) the leadership culture in the health services.

Conclusion

A CBIPE programme was successfully implemented at Universitas Islam Sultan Agung. It demonstrated that students in the health professions can develop skills in collaborative practice despite having some problems with communication and mutual support.

A mobile app to capture EPA assessment data: Utilizing the consolidated framework for implementation research to identify enablers and barriers to engagement

INTRODUCTION

Mobile apps that utilize the framework of entrustable professional activities (EPAs) to capture and deliver feedback are being implemented. If EPA apps are to be successfully incorporated into programmatic assessment, a better understanding of how they are experienced by the end-users will be necessary. The authors conducted a qualitative study using the Consolidated Framework for Implementation Research (CFIR) to identify enablers and barriers to engagement with an EPA app.

METHODS

Structured interviews of faculty and residents were conducted with an interview guide based on the CFIR. Transcripts were independently coded by two study authors using directed content analysis. Differences were resolved via consensus. The study team then organized codes into themes relevant to the domains of the CFIR.

RESULTS

Eight faculty and 10 residents chose to participate in the study. Both faculty and residents found the app easy to use and effective in facilitating feedback immediately after the observed patient encounter. Faculty appreciated how the EPA app forced brief, distilled feedback. Both faculty and residents expressed positive attitudes and perceived the app as aligned with the department's philosophy. Barriers to engagement included faculty not understanding the EPA framework and scale, competing clinical demands, residents preferring more detailed feedback and both faculty and residents noting that the app's feedback should be complemented by a tool that generates more systematic, nuanced, and comprehensive feedback. Residents rarely if ever returned to the feedback after initial receipt.

DISCUSSION

This study identified key enablers and barriers to engagement with the EPA app. The findings provide guidance for future research and implementation efforts focused on the use of mobile platforms to capture direct observation feedback.

Advancing Our Understanding of Narrative Comments Generated by Direct Observation Tools: Lessons From the Psychopharmacotherapy-Structured Clinical Observation

BACKGROUND

While prior research has focused on the validity of quantitative ratings generated by direct observation tools, much less is known about the written comments.

OBJECTIVE

This study examines the quality of written comments and their relationship with checklist scores generated by a direct observation tool, the Psychopharmacotherapy-Structured Clinical Observation (P-SCO).

METHODS

From 2008 to 2012, faculty in a postgraduate year 3 psychiatry outpatient clinic completed 601 P-SCOs. Twenty-five percent were randomly selected from each year; the sample included 8 faculty and 57 residents. To assess quality, comments were coded for valence (reinforcing or corrective), behavioral specificity, and content. To assess the relationship between comments and scores, the authors calculated the correlation between comment and checklist score valence and examined the degree to which comments and checklist scores addressed the same content.

RESULTS

Ninety-one percent of the comments were behaviorally specific. Sixty percent were reinforcing, and 40% were corrective. Eight themes were identified, including 2 constructs not adequately represented by the checklist. Comment and checklist score valence was moderately correlated (Spearman's rho = 0.57, P < .001). Sixty-seven percent of high and low checklist scores were associated with a comment of the same valence and content. Only 50% of overall comments were associated with a checklist score of the same valence and content.

CONCLUSIONS

A direct observation tool such as the P-SCO can generate high-quality written comments. Narrative comments both explain checklist scores and convey unique content. Thematic coding of comments can improve the content validity of a checklist.

Observer-based tools for non-technical skills assessment in simulated and real clinical environments in healthcare: a systematic review

Background

Over the past three decades multiple tools have been developed for the assessment of non-technical skills (NTS) in healthcare. This study was designed primarily to analyse how they have been designed and tested but also to consider guidance on how to select them.

Objectives

To analyse the context of use, method of development, evidence of validity (including reliability) and usability of tools for the observer-based assessment of NTS in healthcare.

Design

Systematic review.

Data sources

Search of electronic resources, including PubMed, Embase, CINAHL, ERIC, PsycNet, Scopus, Google Scholar and Web of Science. Additional records identified through searching grey literature (OpenGrey, ProQuest, AHRQ, King’s Fund, Health Foundation).

Study selection

Studies of observer-based tools for NTS assessment in healthcare professionals (or undergraduates) were included if they: were available in English; published between January 1990 and March 2018; assessed two or more NTS; were designed for simulated or real clinical settings and had provided evidence of validity plus or minus usability. 11,101 articles were identified. After limits were applied, 576 were retrieved for evaluation and 118 articles included in this review.

Results

One hundred and eighteen studies describing 76 tools for assessment of NTS in healthcare met the eligibility criteria. There was substantial variation in the method of design of the tools and the extent of validity, and usability testing. There was considerable overlap in the skills assessed, and the contexts of use of the tools.

Conclusion

This study suggests a need for rationalisation and standardisation of the way we assess NTS in healthcare and greater consistency in how tools are developed and deployed.

Development of a Simulation-Based Interprofessional Teamwork Assessment Tool

BACKGROUND

The Accreditation Council for Graduate Medical Education (ACGME) Milestone projects required each specialty to identify essential skills and develop means of assessment with supporting validity evidence for trainees. Several specialties rate trainees on a milestone subcompetency related to working in interprofessional teams. A tool to assess trainee competence in any role on an interprofessional team in a variety of scenarios would be valuable and suitable for simulation-based assessment.

OBJECTIVE

We developed a tool for simulation settings that assesses interprofessional teamwork in trainees.

METHODS

In 2015, existing tools that assess teamwork or interprofessionalism using direct observation were systematically reviewed for appropriateness, generalizability, adaptability, ease of use, and resources required. Items from these tools were included in a Delphi method with multidisciplinary pediatrics experts using an iterative process from June 2016 to January 2017 to develop an assessment tool.

RESULTS

Thirty-one unique tools were identified. A 2-stage review narrowed this list to 5 tools, and 81 items were extracted. Twenty-two pediatrics experts participated in 4 rounds of Delphi surveys, with response rates ranging from 82% to 100%. Sixteen items reached consensus for inclusion in the final tool. A global 4-point rating scale from novice to proficient was developed.

CONCLUSIONS

A novel tool to assess interprofessional teamwork for individual trainees in a simulated setting was developed using a systematic review and Delphi methodology. This is the first step to establish the validity evidence necessary to use this tool for competency-based assessment.

Assessment Tools for Feedback and Entrustment Decisions in the Clinical Workplace: A Systematic Review

BACKGROUND: Entrustable Professional Activities (EPAs) combine feedback and evaluation with a permission to act under a specified level of supervision and the possibility to schedule learners for clinical service. This literature review aims to identify workplace-based assessment tools that indicate progression toward unsupervised practice, suitable for entrustment decisions and feedback to learners. METHODS: A systematic search was performed in the PubMed, Embase, ERIC, and PsycINFO databases. Based on title/abstract and full text, articles were selected using predetermined inclusion and exclusion criteria. Information on workplace-based assessment tools was extracted using data coding sheets. The methodological quality of studies was assessed using the medical education research study quality instrument (MERSQI). RESULTS: The search yielded 6,371 articles (180 were evaluated in full text). In total, 80 articles were included, identifying 67 assessment tools. Only a few studies explicitly mentioned assessment tools used as a resource for entrustment decisions. Validity evidence was frequently reported, and the MERSQI score was 10.0 on average. CONCLUSIONS: Many workplace-based assessment tools were identified that potentially support learners with feedback on their development and support supervisors with providing feedback. As expected, only few articles referred to entrustment decisions. Nevertheless, the existing tools or the principals could be used for entrustment decisions, supervision level, or autonomy.

Evidence for the Validity of the Psychopharmacotherapy-Structured Clinical Observation Tool: Results of a Factor and Time Series Analysis

OBJECTIVE

The Psychopharmacotherapy-Structured Clinical Observation (P-SCO) is a direct observation tool designed to assess resident performance of a medication visit. This study examines two dimensions of validity for the P-SCO: internal structure and how scores correlate with another variable associated with competence (experience).

METHODS

The faculty completed 601 P-SCOs over 4 years. Multilevel exploratory factor analysis was performed with minimum thresholds for eigenvalue (≥ 1.0) and proportion of variance explained (≥ 5.0%). Internal reliability was assessed with Cronbach alpha. To examine how scores changed with experience, mean ratings (1-4 scale) were calculated for each factor by quarter of the academic year. Separate linear mixed models were also performed.

RESULTS

The analysis yielded three factors that explained 50% of the variance and demonstrated high internal reliability: affective tasks (alpha = 0.90), cognitive tasks (alpha = 0.84), and hard tasks (alpha = 0.74). Items within "hard tasks" were assessment of substance use, violence risk, and adherence, and inquiry about interactions with other providers. Monitoring adverse effects did not load on the hard task factor but also had overall low mean ratings. Compared to the first quarter, fourth quarter scores for affective tasks (b = 0.54, p < 0.01) and hard tasks (b = 0.46, p = 0.02) were significantly improved while cognitive tasks had a non-significant increase. For the hard tasks, the proportion of residents with a low mean rating improved but was still over 30% during the fourth quarter.

CONCLUSIONS

The results provide evidence for the validity of the P-SCO with respect to its internal structure and how scores correlate with experience. Curricular implications are explored, especially for the tasks that were hard to learn.

Considering the interdependence of clinical performance: implications for assessment and entrustment

INTRODUCTION

Our ability to assess independent trainee performance is a key element of competency-based medical education (CBME). In workplace-based clinical settings, however, the performance of a trainee can be deeply entangled with others on the team. This presents a fundamental challenge, given the need to assess and entrust trainees based on the evolution of their independent clinical performance. The purpose of this study, therefore, was to understand what faculty members and senior postgraduate trainees believe constitutes independent performance in a variety of clinical specialty contexts.

METHODS

Following constructivist grounded theory, and using both purposive and theoretical sampling, we conducted individual interviews with 11 clinical teaching faculty members and 10 senior trainees (postgraduate year 4/5) across 12 postgraduate specialties. Constant comparative inductive analysis was conducted. Return of findings was also carried out using one-to-one sessions with key informants and public presentations.

RESULTS

Although some independent performances were described, participants spoke mostly about the exceptions to and disclaimers about these, elaborating their sense of the interdependence of trainee performances. Our analysis of these interdependence patterns identified multiple configurations of coupling, with the dominant being coupling of trainee and supervisor performance. We consider how the concept of coupling could advance workplace-based assessment efforts by supporting models that account for the collective dimensions of clinical performance.

CONCLUSION

These findings call into question the assumption of independent performance, and offer an important step toward measuring coupled performance. An understanding of coupling can help both to better distinguish independent and interdependent performances, and to consider revising workplace-based assessment approaches for CBME.

Education in Pediatrics Across the Continuum (EPAC): First Steps Toward Realizing the Dream of Competency-Based Education

The Education in Pediatrics Across the Continuum (EPAC) Study Group is developing the first competency-based, time-variable progression from undergraduate medical education (UME) to graduate medical education (GME) in the history of medical education in the United States. EPAC, an innovation project sponsored by the Association of American Medical Colleges and supported by the Josiah Macy Jr. Foundation, was developed through a collaboration between five medical schools and multiple professional organizations with an interest in undergraduate and graduate medical education. The planning and implementation process demanded cooperatively addressing practical barriers such as education requirements for licensure and developing approaches to learner assessment that provided meaningful information about competency. Each participating school now has at least three cohorts of learners participating, and the program is transitioning its first cohort of students from UME to GME based on achievement of predetermined competencies that allow this transition. Members of the first cohort of learners in this program have begun their pediatric residency training at different times beginning in late 2016, confirming the feasibility of competency-based advancement from UME to GME in pediatrics. Although there is still much to learn about the outcomes of EPAC learners' professional development in residency training and beyond, EPAC has defined an operational approach to a different path through medical school and into residency training, based on the attainment of competence.

A Systematic Review of Assessment Tools Measuring Interprofessional Education Outcomes Relevant to Pharmacy Education

Objective. To identify and describe the available quantitative tools that assess interprofessional education (IPE) relevant to pharmacy education. Methods. A systematic approach was used to identify quantitative IPE assessment tools relevant to pharmacy education. The search strategy included the National Center for Interprofessional Practice and Education Resource Exchange (Nexus) website, a systematic search of the literature, and a manual search of journals deemed likely to include relevant tools. Results. The search identified a total of 44 tools from the Nexus website, 158 abstracts from the systematic literature search, and 570 abstracts from the manual search. A total of 36 assessment tools met the criteria to be included in the summary, and their application to IPE relevant to pharmacy education was discussed. Conclusion. Each of the tools has advantages and disadvantages. No single comprehensive tool exists to fulfill assessment needs. However, numerous tools are available that can be mapped to IPE-related accreditation standards for pharmacy education.

Priority Areas and Potential Solutions for Successful Integration and Sustainment of Health Systems Science in Undergraduate Medical Education

Educators, policy makers, and health systems leaders are calling for significant reform of undergraduate medical education (UME) and graduate medical education (GME) programs to meet the evolving needs of the health care system. Nationally, several schools have initiated innovative curricula in both classroom and workplace learning experiences to promote education in health systems science (HSS), which includes topics such as value-based care, health system improvement, and population and public health. However, the successful implementation of HSS curricula across schools is challenged by issues of curriculum design, assessment, culture, and accreditation, among others. In this report of a working conference using thematic analysis of workshop recommendations and experiences from 11 U.S. medical schools, the authors describe seven priority areas for the successful integration and sustainment of HSS in educational programs, and associated challenges and potential solutions. In 2015, following regular HSS workgroup phone calls and an Accelerating Change in Medical Education consortium-wide meeting, the authors identified the priority areas: partner with licensing, certifying, and accrediting bodies; develop comprehensive, standardized, and integrated curricula; develop, standardize, and align assessments; improve the UME to GME transition; enhance teachers' knowledge and skills, and incentives for teachers; demonstrate value added to the health system; and address the hidden curriculum. These priority areas and their potential solutions can be used by individual schools and HSS education collaboratives to further outline and delineate the steps needed to create, deliver, study, and sustain effective HSS curricula with an eye toward integration with the basic and clinical sciences curricula.

When less is more: validating a brief scale to rate interprofessional team competencies

BACKGROUND

There is a need for validated and easy-to-apply behavior-based tools for assessing interprofessional team competencies in clinical settings. The seven-item observer-based Modified McMaster-Ottawa scale was developed for the Team Objective Structured Clinical Encounter (TOSCE) to assess individual and team performance in interprofessional patient encounters.

OBJECTIVE

We aimed to improve scale usability for clinical settings by reducing item numbers while maintaining generalizability; and to explore the minimum number of observed cases required to achieve modest generalizability for giving feedback.

DESIGN

We administered a two-station TOSCE in April 2016 to 63 students split into 16 newly-formed teams, each consisting of four professions. The stations were of similar difficulty. We trained sixteen faculty to rate two teams each. We examined individual and team performance scores using generalizability (G) theory and principal component analysis (PCA).

RESULTS

The seven-item scale shows modest generalizability (.75) with individual scores. PCA revealed multicollinearity and singularity among scale items and we identified three potential items for removal. Reducing items for individual scores from seven to four (measuring Collaboration, Roles, Patient/Family-centeredness, and Conflict Management) changed scale generalizability from .75 to .73. Performance assessment with two cases is associated with reasonable generalizability (.73). Students in newly-formed interprofessional teams show a learning curve after one patient encounter. Team scores from a two-station TOSCE demonstrate low generalizability whether the scale consisted of four (.53) or seven items (.55).

CONCLUSION

The four-item Modified McMaster-Ottawa scale for assessing individual performance in interprofessional teams retains the generalizability and validity of the seven-item scale. Observation of students in teams interacting with two different patients provides reasonably reliable ratings for giving feedback. The four-item scale has potential for assessing individual student skills and the impact of IPE curricula in clinical practice settings.

ABBREVIATIONS

IPE: Interprofessional education; SP: Standardized patient; TOSCE: Team objective structured clinical encounter.

Team-Based Simulation for Medical Student Handoff Education

INTRODUCTION

Good handoffs require teamwork, clear communication, a cognitively safe environment, and a good understanding of the patient's medical needs. Complex tertiary care training institutions require multiple handoffs over a patient's time in the hospital. Medical students need better handoff education. We designed a handoff training exercise using a simulated patient care environment for students to practice and observe multiple handoffs over time.

METHODS

An initial large-group didactic session provides direct instruction on handoffs. In small groups, students are subsequently assigned roles as individual physicians in a chain of providers during a simulated patient hospitalization over several days, with an additional student as an observer. Blinded to any prior discussion, student physicians sequentially give and receive handoffs about the patient from a previous physician as their simulated hospital course evolves. The observer shares his or her insights, and a large-group structured debriefing exercise follows.

RESULTS

In both 2015 and 2016, we implemented this session with a cohort of 30 fourth-year medical students. Most recently we implemented this with chief residents, and a group of 20 third-year pediatric clerkship students. We reviewed a selection of the discussion guides and found reporters/ observers noted that participants stated the primary problem 95% of the time (19 of 20 handoffs), the patient acuity 90% of the time (18 of 20 handoffs), and a clear contingency plan 85% of the time (17 of 20 handoffs).

DISCUSSION

We found students initially to be preoccupied with making correct clinical decisions instead of giving effective handoffs, and consequently we clarified details and adjusted clinical information to be more transparent. Results suggest the session achieves our goals of using a structured handoff method to communicate higher-level information and debrief effectively with peers. We envision this activity to be applicable to teaching handoffs to residents, nurses, and other health care professionals, as well as possibly in diverse clinical environments or in collaboration with outpatient providers.

For the General Internist: A Summary of Key Innovations in Medical Education

We conducted a review of published medical education articles to identify high-quality research and innovation relevant to educators in general medicine. Our review team consisted of six general internists with expertise in medical education and a professional medical librarian. We manually searched 15 journals in pairs (a total of 3062 citations) for original research articles in medical education published in 2014. Each pair of reviewers independently rated the relevance, importance, and generalizability of articles on medical education in their assigned journals using a 27-point scale (maximum of 9 points for each characteristic). From this list, each team member independently reviewed the 22 articles that received a score of 20 or higher from both initial reviewers, and for each selected article rated the quality and global relevance for the generalist educator. We included the seven top-rated articles for presentation in this review, and categorized the studies into four general themes: continuity clinic scheduling, remediation, interprofessional education, and quality improvement and patient safety. We summarized key findings and identified significant limitations of each study. Further studies assessing patient outcomes are needed to strengthen the literature in medical education. This summary of relevant medical education articles can inform future research, teaching, and practice.

Addressing the Interprofessional Collaboration Competencies of the Association of American Medical Colleges: A Systematic Review of Assessment Instruments in Undergraduate Medical Education

PURPOSE

To summarize characteristics and validity evidence of tools that assess teamwork in undergraduate medical education (UME), and provide recommendations for addressing the interprofessional collaboration competencies of the Association of American Medical Colleges (AAMC).

METHOD

The authors conducted a systematic review, searching MEDLINE, MEDLINE In-process, CINAHL, and PsycINFO from January 1, 1979, through April 1, 2014; they searched reference lists and national meeting abstracts. They included original research reports that described a quantitative tool used to assess teamwork in UME. They abstracted characteristics and validity evidence for the tools, plus study quality, according to established frameworks. Two authors independently abstracted 25% of articles and calculated agreement. Authors then applied predefined criteria to identify tools best suited to address the AAMC's teamwork competencies.

RESULTS

Of 13,549 citations, 70 articles describing 64 teamwork assessment tools were included. Of these 64 tools, 27 (42%) assessed teamwork in classroom, 31 (48%) in simulation, and only 7 (11%) in actual clinical settings. The majority (47; 73%) of tools assessed medical students' teamwork in interprofessional teams. On the basis of content concordance, strength of validity evidence, generalizability of scores, and level of outcomes, four published tools were recommended to assess the AAMC's teamwork competencies: the Collaborative Healthcare Interdisciplinary Relationship Planning Scale, Readiness for Interprofessional Learning Scale, Communication and Teamwork Skills assessment, and Teamwork Mini-Clinical Evaluation Exercise.

CONCLUSIONS

Substantial validity evidence supports the use of several UME teamwork assessments. Four tools have been appropriately designed and sufficiently studied to constitute appropriate assessments of the AAMC's teamwork competencies.

Workplace-based assessment: a review of user perceptions and strategies to address the identified shortcomings

Workplace based assessments (WBAs) are now commonplace in postgraduate medical training. User acceptability and engagement is essential to the success of any medical education innovation. To this end, possessing an insight into trainee and trainer perceptions towards WBAs will help identify the major problems, permitting strategies to be introduced to improve WBA implementation. A review of literature was performed to identify studies examining trainee and trainer perceptions towards WBAs. Studies were excluded if non-English or sampling a non-medical/dental population. The identified literature was synthesised for the purpose of this critical narrative review. It is clear that there is widespread negativity towards WBAs in the workplace. This has negatively impacted on the effectiveness of WBA tools as learning aids. This negativity exists in trainees but also to an extent in their trainers. Insight gained from the literature reveals three dominant problems with WBA implementation: poor understanding as to the purpose of WBAs; insufficient time available for undertaking these assessments; and inadequate training of trainers. Approaches to addressing these three problems with WBA implementation are discussed. It is likely that a variety of solutions will be required. The prevalence of negativity towards WBAs is substantial in both trainees and trainers, eroding the effectiveness of learning that is consequent upon them. The educational community must now listen to the concerns being raised by the users and consider the range of strategies being proposed to improve the experiences of trainees, and their trainers.

Advancing Competency-Based Medical Education: A Charter for Clinician-Educators

The International Competency-Based Medical Education (ICBME) Collaborators have been working since 2009 to promote understanding of competency-based medical education (CBME) and accelerate its uptake worldwide. This article presents a charter, supported by a literature-based rationale, which is meant to provide a shared mental model of CBME that will serve as a path forward in its widespread implementation.At a 2013 summit, the ICBME Collaborators laid the groundwork for this charter. Here, the fundamental principles of CBME and professional responsibilities of medical educators in its implementation process are described. The authors outline three fundamental principles: (1) Medical education must be based on the health needs of the populations served; (2) the primary focus of education and training should be the desired outcomes for learners rather than the structure and process of the educational system; and (3) the formation of a physician should be seamless across the continuum of education, training, and practice.Building on these principles, medical educators must demonstrate commitment to teaching, assessing, and role modeling the range of identified competencies. In the clinical setting, they must provide supervision that balances patient safety with the professional development of learners, being transparent with stakeholders about level of supervision needed. They must use effective and efficient assessment strategies and tools for basing transition decisions on competence rather than time in training, empowering learners to be active participants in their learning and assessment. Finally, advancing CBME requires program evaluation and research, faculty development, and a collaborative approach to realize its full potential.

Interprofessional collaboration milestones: advocating for common assessment criteria in graduate medical education

BACKGROUND

Milestone-based assessments of resident physicians inform critical decisions regarding resident competence and advancement. Thus, it is essential that milestone evaluations are based upon strong validity evidence and that consistent evaluation criteria are used across residency programs. A common approach to assessment of interprofessional collaboration milestones is particularly important since standardized measures of individual resident competence in interprofessional collaboration have not been established.

DISCUSSION

We propose that assessments of interprofessional collaboration in graduate medical education meet common criteria, namely, these assessments should: 1) measure competency of an individual resident, 2) occur in the context of an interprofessional team, 3) be ascertained via direct observation of the resident, 4) be performed in a real-world clinical practice setting (such as a hospital ward, outpatient clinic, or operating room). We present the evidence-based rationale for these criteria and cite examples of published assessment instruments that fulfill one or more of the criteria, however further research is needed to ensure fidelity of assessments. The proposed criteria may assist residency educators as they endeavor to provide robust and consistent assessments of interprofessional collaboration milestones.