How to assess communication, professionalism, collaboration and the other intrinsic CanMEDS roles in orthopedic residents: use of an objective structured clinical examination (OSCE)

Re-examining single-moment-in-time high-stakes examinations in specialist training: A critical narrative review

In this critical narrative review, we challenge the belief that single-moment-in-time high-stakes examinations (SMITHSEx) are an essential component of contemporary specialist training. We explore the arguments both for and against SMITHSEx, examine potential alternatives, and discuss the barriers to change.SMITHSEx are viewed as the "gold standard" assessment of competence but focus excessively on knowledge assessment rather than capturing essential competencies required for safe and competent workplace performance. Contrary to popular belief, regulatory bodies do not mandate SMITHSEx in specialist training. Though acting as significant drivers of learning and professional identity formation, these attributes are not exclusive to SMITHSEx.Skills such as crisis management, procedural skills, professionalism, communication, collaboration, lifelong learning, reflection on practice, and judgement are often overlooked by SMITHSEx. Their inherent design raises questions about the validity and objectivity of SMITHSEx as a measure of workplace competence. They have a detrimental impact on trainee well-being, contributing to burnout and differential attainment.Alternatives to SMITHSEx include continuous low-stakes assessments throughout training, ongoing evaluation of competence in the workplace, and competency-based medical education (CBME) concepts. These aim to provide a more comprehensive and context-specific assessment of trainees' competence while also improving trainee welfare.Specialist training colleges should evolve from exam providers to holistic education sources. Assessments should emphasise essential practical knowledge over trivia, align with clinical practice, aid learning, and be part of a diverse toolkit. Eliminating SMITHSEx from specialist training will foster a competency-based approach, benefiting future medical professionals' well-being and success.

Developing an innovative national ACP-OSCE program in Taiwan: a mixed method study

OBJECTIVES

To evaluate the process and the comprehensiveness of advance care planning (ACP), we designed a national ACP-OSCE (Objective Structured Clinical Examination) program.

METHODS

The program was designed as a 40-minute OSCE test. Participants were categorized as different ACP team members to illustrate realistic scenarios. Preceptors were asked to observe ACP professionals' actions, responses, and communication skills during ACP with standardized patients (SP) through a one-way mirror. Participants' communication skills, medical expertise, legal knowledge, empathetic response and problem-solving skills of ACP were also self-evaluated before and after OSCE. Thematic analysis was used for qualitative analysis.

RESULTS

In Nov 2019, a total of 18 ACP teams with 38 ACP professionals completed the ACP-OSCE program, including 15 physicians, 15 nurses, 5 social workers, and 3 psychologists. After the ACP-OSCE program, the average score of communication skills, medical expertise, legal knowledge, empathetic response, ACP problem-solving all increased. Nurses felt improved in medical expertise, legal knowledge, and problem-solving skills, psychologists and social workers felt improved in legal knowledge, while physicians felt no improved in all domain, statistically. Thematic analysis showed professional skills, doctoral-patient communication, benefit and difficulties of ACP were the topics which participants care about. Meanwhile, most participants agreed that ACP-OSCE program is an appropriate educational tool.

CONCLUSION

This is the first national ACP-OSCE program in Asia. We believe that this ACP-OSCE program could be applied in other countries to improve the ACP process and quality.

Evaluating the Construct Validity of Competencies: A Retrospective Analysis

Background

A competency-based framework focuses on alignment between professional standards and assessment design. This alignment implies improved measurement validity, yet it has not been established that competence in one context predicts performance in another context. High-stakes competence assessments offer insights into the relationship between assessment design and competencies.

Methods/Analyses

The internationally educated nurses competency assessment program (IENCAP) was developed at Touchstone Institute in collaboration with the College of Nurses of Ontario (CNO) and includes a 12-station OSCE. Each station evaluated the same 10 competencies. We submitted competency scores to a multi-trait multi-method matrix analysis to evaluate the convergent and discriminant validity of competencies.

Results/Observations

All correlations were significant and positive; however, we did not find evidence of convergent or discriminant validity. Correlations were higher between different competencies evaluated within the same station (mean correlation = 0.60) compared to identical competencies evaluated across different stations (mean correlation = 0.19).

Discussion

The results do not provide evidence of construct validity for competencies. While competency-based approaches emphasize various generalized knowledge, skills, and attitudes, these findings indicate that the clinical context is a major determinant of performance.

Conclusion

The context-dependent nature of competencies requires multiple assessments in varied contexts. Performance on a single competency cannot be determined in a single occasion.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-023-01794-z.

Associations between communication OSCE performance and admission interviews in medical education

OBJECTIVE

To investigate associations between medical students' communication and other clinical skills assessed in OSCEs and MMIs performance upon admission by developing an assessment strategy based on an internationally acknowledged curriculum framework.

METHODS

Between July 2019 and March 2020, 365 medical students in the 4th and 5th semester took two OSCEs containing 10 5-minute stations examining communication and other clinical skills. We used a European conceptual framework to determine the content validity of the communication score and calculated scores of communication and other clinical skills. We assessed students' OSCE performance and estimated multiple regressions to predict its association with MMIs.

RESULTS

The multiple linear regressions showed that students' MMI performance upon admission is significantly associated with the communication score (b = 0.32, p = 0.006) but not the clinical score (b = 0.19, p = 0.121), when controlling for gender and cognitive criteria.

CONCLUSIONS

Our assessment strategy designed to distinguish between different areas of competence provides a more thorough description of the positive relationship between OSCE performance and MMIs.

PRACTICE IMPLICATIONS

We developed a communication skills assessment strategy that can be easily applied by medical schools that use OSCEs as a training or assessment method.

"You Never Get a Second Chance": First Impressions of Physicians Depend on Their Body Posture and Gender

A first impression matters, in particular when encounters are brief as in most doctor-patient interactions. In this study, we investigate how physicians’ body postures impact patients’ first impressions of them and extend previous research by exploring posture effects on the perception of all roles of a physician – not just single aspects such as scholarly expertise or empathy. In an online survey, 167 participants ranked photographs of 4 physicians (2 female, 2 male) in 4 postures (2 open, 2 closed). The results show that male physicians were rated more positively when assuming open rather than closed postures with respect to all professional physician roles. Female physicians in open postures were rated similarly positive for items related to medical competence, but they tended to be rated less favorably with respect to social skills (such as the ability to communicate with and relate to the patient). These findings extend what is known about the effects of physicians’ body postures on the first impressions patients form to judge physicians’ medical versus social competencies. We discuss practical implications and the need for more research on interaction effects of body postures and physician gender on first impressions.

Unobserved Observers: Nurses' Perspectives About Sharing Feedback on the Performance of Resident Physicians

PURPOSE

Postgraduate training programs are incorporating feedback from registered nurses (RNs) to facilitate holistic assessments of resident performance. RNs are a potentially rich source of feedback because they often observe trainees during clinical encounters when physician supervisors are not present. However, RN perspectives about sharing feedback have not been deeply explored. This study investigated RN perspectives about providing feedback and explored the facilitators and barriers influencing their engagement.

METHOD

Constructivist grounded theory methodology was used in interviewing 11 emergency medicine and 8 internal medicine RNs at 2 campuses of a tertiary care academic medical center in Ontario, Canada, between July 2019 and March 2020. Interviews explored RN experiences working with and observing residents in clinical practice. Data collection and analysis were conducted iteratively. Themes were identified using constant comparative analysis.

RESULTS

RNs felt they could observe authentic day-to-day behaviors of residents often unwitnessed by supervising physicians and offer unique feedback related to patient advocacy, communication, leadership, collaboration, and professionalism. Despite a strong desire to contribute to resident education, RNs were apprehensive about sharing feedback and reported barriers related to hierarchy, power differentials, and a fear of overstepping professional boundaries. Although infrequent, a key stimulus that enabled RNs to feel safe in sharing feedback was an invitation from the supervising physician to provide input.

CONCLUSIONS

Perceived hierarchy in academic medicine is a critical barrier to engaging RNs in feedback for residents. Accessing RN feedback on authentic resident behaviors requires dismantling the negative effects of hierarchy and fostering a collaborative interprofessional working environment. A critical step toward this goal may require supervising physicians to model feedback-seeking behavior by inviting RNs to share feedback. Until a workplace culture is established that validates nurses' input and creates safe opportunities for them to contribute to resident education, the voices of nurses will remain unheard.

How good are medical students at communicating risk? An implementation study at three German medical schools

OBJECTIVE

To assess the objective and subjective risk communication skills of medical students in three universities in Germany METHODS: We developed a risk communication skills Objective Structured Clinical Examination (OSCE) station and implemented it in three medical schools in Germany. 596 students contributed data to a risk communication checklist and a risk communication self-assessment. Multiple linear regression models were used to identify factors associated with the risk communication OSCE performance.

RESULTS

Participants in our study achieved on average 73.5% of the total risk communication skills score, which did not differ between locations (F(2-595) = 1.96; p = 0.142). The mean objective performance of students who assessed their skills as poor was significantly worse than the performance of students who assessed their skills as good (t(520) = -5.01, p < 0.001). The risk communication skills score was associated with native language but not with gender nor General Point Average (p < 0.001).

CONCLUSIONS

Medical students demonstrated acceptable risk communication skills scores and were able to self-assess their performance. However, selected communication techniques should be re-emphasised in the undergraduate medical curriculum.

PRACTICE IMPLICATIONS

Our research identified shortcomings in particular subgroups that can be addressed through tailored curriculum interventions.

Challenges for Family Medicine Residents in Attaining the CanMEDS Professional Role: A Thematic Analysis of Preceptor Field Notes

PURPOSE

Among the roles of the competent physician is that of a professional, according to the Canadian Medical Education Directives for Specialists (CanMEDS) framework, which describes the abilities physicians require to effectively meet the health care needs of the people they serve. Through examination of preceptor field notes on resident performance, the authors identified aspects of this role with which family medicine residents struggle.

METHOD

The authors used a structured thematic analysis in this qualitative study to explore the written feedback postgraduate medical learners receive at the University of Toronto Department of Family and Community Medicine. Seventy field notes written between 2015 and 2017 by clinical educators for residents who scored "below expectation" in the CanMEDS professional role were analyzed. From free-text comments, the authors derived inductive codes, amalgamated the codes into themes, and measured the frequency of the occurrence of the codes. The authors then mapped the themes to the key competencies of the CanMEDS professional role.

RESULTS

From the field notes, 7 themes emerged that described reasons for poor performance. Lack of collegiality, failure to adhere to standards of practice or legal guidelines, and lack of reflection or self-learning were identified as major issues. Other themes were failure to maintain boundaries, taking actions that could have a negative impact on patient care, failure to maintain patient confidentiality, and failure to engage in self-care. When the themes were mapped to the key competencies in the CanMEDS professional role, most related to the competency "commitment to the profession."

CONCLUSIONS

This study highlights aspects of professional conduct with which residents struggle and suggests that the way professionalism is taught in residency programs-and at all medical training levels-should be reassessed. Educational interventions that emphasize learners' commitment to the profession could enhance the development of more practitioners who are consummate professionals.

An Objective Structured Clinical Examination for Medical Student Radiology Clerkships: Reproducibility Study

BACKGROUND

Objective structured clinical examinations (OSCEs) are a useful method to evaluate medical students' performance in the clerkship years. OSCEs are designed to assess skills and knowledge in a standardized clinical setting and through use of a preset standard grading sheet, so that clinical knowledge can be evaluated at a high level and in a reproducible way.

OBJECTIVE

This study aimed to present our OSCE assessment tool designed specifically for radiology clerkship medical students, which we called the objective structured radiology examination (OSRE), with the intent to advance the assessment of clerkship medical students by providing an objective, structured, reproducible, and low-cost method to evaluate medical students' radiology knowledge and the reproducibility of this assessment tool.

METHODS

We designed 9 different OSRE cases for radiology clerkship classes with participating third- and fourth-year medical students. Each examination comprises 1 to 3 images, a clinical scenario, and structured questions, along with a standardized scoring sheet that allows for an objective and low-cost assessment. Each medical student completed 3 of 9 random examination cases during their rotation. To evaluate for reproducibility of our scoring sheet assessment tool, we used 5 examiners to grade the same students. Reproducibility for each case and consistency for each grader were assessed with a two-way mixed effects intraclass correlation coefficient (ICC). An ICC below 0.4 was deemed poor to fair, an ICC of 0.41 to 0.60 was moderate, an ICC of 0.6 to 0.8 was substantial, and an ICC greater than 0.8 was almost perfect. We also assessed the correlation of scores and the students' clinical experience with a linear regression model and compared mean grades between third- and fourth-year students.

RESULTS

A total of 181 students (156 third- and 25 fourth-year students) were included in the study for a full academic year. Moreover, 6 of 9 cases demonstrated average ICCs more than 0.6 (substantial correlation), and the average ICCs ranged from 0.36 to 0.80 (P<.001 for all the cases). The average ICC for each grader was more than 0.60 (substantial correlation). The average grade among the third-year students was 11.9 (SD 4.9), compared with 12.8 (SD 5) among the fourth-year students (P=.005). There was no correlation between clinical experience and OSRE grade (-0.02; P=.48), adjusting for the medical school year.

CONCLUSIONS

Our OSRE is a reproducible assessment tool with most of our OSRE cases showing substantial correlation, except for 3 cases. No expertise in radiology is needed to grade these examinations using our scoring sheet. There was no correlation between scores and the clinical experience of the medical students tested.

Comparison of Resident Self-Evaluation to Standardized Patient Evaluators in a Multi-Institutional Objective Structured Clinical Examination: Objectively Measuring Residents' Communication and Counseling Skills

BACKGROUND

For the past 30 years, there has been a growing emphasis on communication and self-evaluation skills training in graduate medical education. This is reflected in the Next Accreditation System. The Objective Structured Clinical Examination (OSCE) is widely used in graduate medical education for assessing dimensions of interpersonal communication and counseling skills. The OSCEs may be developed to target challenging clinical scenarios difficult to capture in clinical practice and can be used as a medium for resident self-evaluation.

OBJECTIVES

The aims of the study were to evaluate residents' interpersonal, communication, and counseling skills using Kalamazoo Essential Elements Communication Checklist in 4 clinically challenging scenarios and to compare standardized patient (SP) evaluations to residents' self-evaluation by category of medical school.

METHODS

South East Michigan Center for Medical Education is a consortium of teaching hospitals. Member residents participate in 4 OSCEs as part of their postgraduate 1 curriculum. The OSCEs were developed to evaluate clinically relevant but difficult to capture scenarios including: (a) error disclosure/counseling an angry patient; (b) delivering bad news/end of life; (c) domestic violence; and (d) counseling a patient with colon cancer requesting alternative treatments. At the conclusion of each OSCE, SPs evaluated and residents self-evaluated their performance. Once evaluations were completed, SPs provided residents with feedback.

RESULTS

Six member institutions and 344 residents participated during the 2014, 2015, and 2016 academic years. There were more international medical graduates (59%) than graduates of Liaison Committee for Medical Education-accredited medical schools. There were more males (62.2%) than females. Standardized patients rated residents higher than residents rated themselves in 2014 (<0.001), but not in 2015 or 2016. When combining all years and all residents, there was no correlation of SP and resident scores. Standardized patients rated female residents higher than female residents rated themselves (P < 0.0001). Male residents scored themselves similarly to the SPs, but male residents rated themselves higher than female residents rated themselves (P < 0.001). Standardized patient scores for male and female residents were not significantly different.

CONCLUSIONS

Targeted OSCEs provide an objective format to evaluate residents in challenging clinical scenarios. Resident self-evaluations did not correlate with SPs. In addition, female residents rated themselves lower than male residents and lower than SPs. There is need to develop interventions and curricula to improve resident's self-evaluation skills and in particular address lower self-evaluation by female trainees.

The state of health advocacy training in postgraduate medical education: a scoping review

CONTEXT

Health advocacy is an essential component of postgraduate medical education, and is part of many physician competency frameworks such as the Canadian Medical Education Directives for Specialists (CanMEDS) roles. There is little consensus about how advocacy should be taught and assessed in the postgraduate context. There are no consolidated guides to assist in the design and implementation of postgraduate health advocacy curricula.

OBJECTIVES

This scoping review aims to identify and analyse existing literature pertaining to health advocacy education and assessment in postgraduate medicine. We specifically sought to summarise themes from the literature that may be useful to medical educators to inform further health advocacy curriculum interventions.

METHODS

MEDLINE, Embase and ERIC were searched using MeSH (medical student headings) and non-MeSH search terms. Additional articles were found using forward snowballing. The grey literature search included Google and relevant stakeholder websites, regulatory bodies, physician associations, government agencies and academic institutions. We followed a stepwise scoping review methodology, followed by thematic analysis using an inductive approach.

RESULTS

Of the 123 documents reviewed in full, five major themes emerged: (i) conceptions of health advocacy have evolved towards advocating with rather than for patients, communities and populations; (ii) longitudinal curricula were less common but appeared the most promising, often linked to scholarly or policy objectives; (iii) hands-on, immersive opportunities build competence and confidence; (iv) community-identified needs and partnerships are increasingly considered in designing curriculum, and (v) resident-led and motivated programmes appear to engage residents and allow for achievement of stated outcomes. There remain significant challenges to assessment of health advocacy competencies, and assessment tools for macro-level health advocacy were notably absent.

CONCLUSIONS

There is considerable heterogeneity in the way health advocacy is taught, assessed and incorporated into postgraduate curricula across programmes and disciplines. We consolidated recommendations from the literature to inform further health advocacy curriculum design, implementation and assessment.

Competency-based medical education in ophthalmology residency training: a review

Competency-based medical education (CBME) is an outcomes-based training model that has become the new standard of medical education internationally. Regulatory and accrediting bodies have responded by establishing key competencies that residency programs may use to reshape their curricula. Available literature on the implementation of CBME in ophthalmology residency training programs across North America was examined. Ophthalmology-specific residency training objectives and milestones have been proposed within North America in the last 4 years. Curriculum guidelines and implementation blueprints and principles have also been proffered from various sources internationally but have met with some barriers. Assessment tools within ophthalmology have seen the most innovative development within the domain of surgical skills objectives. Competencies outside of the medical expert and patient care roles have proven more challenging to teach and assess. One ophthalmology program in Canada has undergone early implementation of a novel CBME curriculum. There is still considerable work to be done to successfully implement CBME curricula within ophthalmology residency programs in North America. Collaborative efforts to develop customizable curricula, tackle implementation barriers, and create specific assessment modalities will assist programs in meeting the competency mandates of CBME.

Evaluation of inter-professional communication and leadership skills among graduating Canadian urology residents

INTRODUCTION

The importance of developing inter-professional communication and leadership skills among residents is well-recognized; however, formal tools to assess these skills are lacking. The goal of our study was to assess the leader and communicator roles in graduating urology residents using a validated self-assessment form developed for business students that focuses on inter-professional skills.

METHODS

Chief residents (n=36) were evaluated with surveys of communication and leadership skills. The same surveys were administered through email to the residents' program directors (PDs). Resident self-assessment and PD assessment were compared using paired and non-paired t-tests.

RESULTS

Graduating urology residents' self-assessment of their communication and leadership skills did not differ from assessments made by their PDs (77.6 vs. 74.4%; p=0.19); however, there were outlier candidates in whom PD assessment differed substantially from self-assessment on both surveys. Graduating urology residents scored themselves higher on self-awareness (82.6 vs. 77%; p= 0.05) and lower on stress management (67.7 vs. 77%; p= 0.01) compared to their PDs. Resident self-assessment scores were similar to business students on both communication and leadership surveys. Limitations were the small sample size and lack of survey evaluation by those surveyed.

CONCLUSIONS

Graduating urology residents' self-assessment of their own communication and leadership skills did not differ greatly from assessment by their PDs or a sample of business students. Comparison of self-assessment evaluations and evaluations by PDs allowed us to identify outliers in whom self-assessment and PD-assessment markedly differed, which may allow for more focused and meaningful feedback.

The education avenue of the clinical learning environment: A pragmatic approach

Aim: This perspective is part of a series of articles that are outcomes of a consensus conference, that seek to offer a comprehensive examination of the clinical learning environment (CLE), using different academic disciplines and areas of focus termed "avenues." The education dimensions of the CLE are discussed in detail in this perspective, along with critical linkages to the other avenues. Methods: Using iterative presentations, discussion and small group work, in October 2018, the consensus conference participants explored the education, psychological, sociocultural, diversity and inclusion, digital and architectural aspects of the CLE. Results: The education avenue of the CLE includes elements of teaching and learning such as the curriculum, clinical experiences, the assessment system, educational program governance, trainee selection, faculty development, and program evaluation and improvement. Within the educational domain, we focus on organizational and personal/social aspects of the CLE, including (1) curriculum design and deployment, including placement of trainees in clinical settings, organizational culture, practices and policies, and accreditation and regulatory requirements from the organizational domain (2) the education system, including assessment, program evaluation and organization and governance; and (3) elements from the personal and social domains, including peer-to-peer, trainee-faculty, and trainee-patient relationships that influence how and what postgraduate trainees learn, trainee selection, informal and hidden curricula, and trainees' perceptions of their learning environment. Conclusions: We provide suggestions for further research and recommendations for addressing challenges and facilitating improvement in the educational aspects of the CLE, along with actionable practice points.

Selected Abstracts From the Proceedings of the 2017 Society of Teachers of Family Medicine Conference on Medical Student Education

The Society of Teachers of Family Medicine (STFM) is an organization made up of educators devoted to teaching family medicine to learners of all levels. This multidisciplinary group of physicians, behavioral scientists, researchers, and educators from other health professions works to further STFM's mission of improving the health of all people through education, research, patient care, and advocacy. STFM held its 43rd Conference on Medical Student Education in Anaheim, California from February 9 to 12, 2017. Abstracts for conference sessions can be viewed online.1 The conference was held concurrently with the 2017 Society of Student-Run Free Clinics Annual Conference. This partnership empowered many passionate medical students to participate in STFM sessions and present their posters. A wide variety of topics were explored by STFM conference presenters and attendees. The plenary speakers addressed physician wellness (Dike Drummond, MD), family medicine as a career (Wanda Filer, MD), and the future of family medicine (Aaron Michelfelder, MD, and Michelle Byrne, MD). The STFM Education Committee reviewed and selected eight exemplary abstracts from 22 presented educational research papers. Criteria for selection included strength of contribution to medical student education, topic of interest within and beyond family medicine, and quality of study, including well-described rationale, appropriate methods, clear results, and thoughtful conclusions. The areas covered are related to new educational methods and tools, faculty development, and interprofessional learning and assessment.

Core Professionalism Education in Surgery: A Systematic Review

BACKGROUND

Professionalism education is one of the major elements of surgical residency education.

AIMS

To evaluate the studies on core professionalism education programs in surgical professionalism education.

STUDY DESIGN

Systematic review.

METHODS

This systematic literature review was performed to analyze core professionalism programs for surgical residency education published in English with at least three of the following features: program developmental model/instructional design method, aims and competencies, methods of teaching, methods of assessment, and program evaluation model or method. A total of 27083 articles were retrieved using EBSCOHOST, PubMed, Science Direct, Web of Science, and manual search.

RESULTS

Eight articles met the selection criteria. The instructional design method was presented in only one article, which described the Analysis, Design, Development, Implementation, and Evaluation model. Six articles were based on the Accreditation Council for Graduate Medical Education criterion, although there was significant variability in content. The most common teaching method was role modeling with scenario- and case-based learning. A wide range of assessment methods for evaluating professionalism education were reported. The Kirkpatrick model was reported in one article as a method for program evaluation.

CONCLUSION

It is suggested that for a core surgical professionalism education program, developmental/instructional design model, aims and competencies, content, teaching methods, assessment methods, and program evaluation methods/models should be well defined, and the content should be comparable.

Strategies identified by program directors to improve adoption of the CanMEDS framework

BACKGROUND

Challenges associated with the use of the CanMEDS physician competency framework (CanMEDS) have been the subject of several studies. Most of these have focused on the adoption of specific roles in an Anglophone context. This study aims to investigate how Francophone postgraduate medical education (PGME) program directors have integrated the CanMEDS framework into their programs.

METHODS

We invited Francophone PGME program directors to participate in group interviews aimed at exploring their experiences using the CanMEDS framework. We used an open-ended interview guide and realized a thematic analysis of the transcripts.

RESULTS

We held five group interviews between February and December 2014 with 17 Francophone program directors representing 13 out of a maximum of 62 different specialties/subspecialties. Although program directors endorsed the framework, its integration was seen as challenging, particularly the assessment of non-medical expert roles. To overcome these challenges, they relied on common strategies including a longitudinal approach to the framework, improving inter-program collaboration, and subcontracting the teaching of certain roles.

CONCLUSION

While integrating the CanMEDS framework into their programs, Francophone program directors struggled with teaching and assessing non-medical expert roles and ensuring their longitudinal integration over time. Directors relied on various strategies, some of which (e.g., subcontracting) may ultimately limit the adoption of the framework as a whole. ___.

A Comparison of Assessment Tools: Is Direct Observation an Improvement Over Objective Structured Clinical Examinations for Communications Skills Evaluation?

BACKGROUND

Evaluation of resident physicians' communications skills is a challenging task and is increasingly accomplished with standardized examinations. There exists a need to identify the effective, efficient methods for assessment of communications skills.

OBJECTIVE

We compared objective structured clinical examination (OSCE) and direct observation as approaches for assessing resident communications skills.

METHODS

We conducted a retrospective cohort analysis of orthopaedic surgery resident physicians at a single tertiary care academic institution, using the Institute for Healthcare Communication "4 Es" model for effective communication. Data were collected between 2011 and 2015. A total of 28 residents, each with OSCE and complete direct observation assessment checklists, were included in the analysis. Residents were included if they had 1 OSCE assessment and 2 or more complete direct observation assessments.

RESULTS

There were 28 of a possible 59 residents (47%) included. A total of 89% (25 of 28) of residents passed the communications skills OSCE; only 54% (15 of 28) of residents passed the direct observation communications assessment. There was a positive, moderate correlation between OSCE and direct observation scores overall (r = 0.415, P = .028). There was no agreement between OSCE and direct observation in categorizing residents into passing and failing scores (κ = 0.205, P = .16), after adjusting for chance agreement.

CONCLUSIONS

Our results suggest that OSCE and direct observation tools provide different insights into resident communications skills (simulation of rare and challenging situations versus real-life daily encounters), and may provide useful perspectives on resident communications skills in different contexts.

Current Practices in Assessing Professionalism in United States and Canadian Allopathic Medical Students and Residents

Professionalism is a critically important competency that must be evaluated in medical trainees but is a complex construct that is hard to assess. A systematic review was undertaken to give insight into the current best practices for assessment of professionalism in medical trainees and to identify new research priorities in the field. A search was conducted on PubMed for behavioral assessments of medical students and residents among the United States and Canadian allopathic schools in the last 15 years. An initial search yielded 594 results, 28 of which met our inclusion criteria. Our analysis indicated that there are robust generic definitions of the major attributes of medical professionalism. The most commonly used assessment tools are survey instruments that use Likert scales tied to attributes of professionalism. While significant progress has been made in this field in recent years, several opportunities for system-wide improvement were identified that require further research. These include a paucity of information about assessment reliability, the need for rater training, a need to better define competency in professionalism according to learner level (preclinical, clerkship, resident etc.) and ways to remediate lapses in professionalism. Student acceptance of assessment of professionalism may be increased if assessment tools are shifted to better incorporate feedback. Tackling the impact of the hidden curriculum in which students may observe lapses in professionalism by faculty and other health care providers is another priority for further study.

Visions and reality: the idea of competence-oriented assessment for German medical students is not yet realised in licensing examinations

Objective: Competence orientation, often based on the CanMEDS model, has become an important goal for modern curricula in medical education. The National Competence Based Catalogue of Learning Objectives for Undergraduate Medical Education (NKLM) has been adopted in Germany. However, it is currently unknown whether the vision of competence orientation has also reached the licensing examination procedures. Methods: Therefore, a prospective, descriptive, single-centre, exemplary study design was applied to evaluate 4051 questions/tasks (from 28 examiners at 7 two-day licensing oral-practical exams) for undergraduate medical students at the University of Ulm. The oral and practical questions/tasks as well as the real bedside assessment were assigned to specific competence roles (NKLM section I), categories (NKLM section II) and taxonomy levels of learning domains. Results: Numerous questions/tasks were set per candidate (day 1/2: 70±24/86±19 questions) in the licensing oral-practical exam. Competence roles beyond the "medical expert" were scarcely considered. Furthermore, practical and communication skills at the bedside were hardly addressed (less than 3/15 min). Strikingly, there was a significant predominance of questions with a low-level taxonomy. Conclusions: The data indicate a misalignment of competence-oriented frameworks and the "real world" licensing practical-oral medical exam, which needs improvement in both evaluation and education processes.

Assessing medical professionalism: A systematic review of instruments and their measurement properties

Background

Over the last three decades, various instruments were developed and employed to assess medical professionalism, but their measurement properties have yet to be fully evaluated. This study aimed to systematically evaluate these instruments’ measurement properties and the methodological quality of their related studies within a universally acceptable standardized framework and then provide corresponding recommendations.

Methods

A systematic search of the electronic databases PubMed, Web of Science, and PsycINFO was conducted to collect studies published from 1990–2015. After screening titles, abstracts, and full texts for eligibility, the articles included in this study were classified according to their respective instrument’s usage. A two-phase assessment was conducted: 1) methodological quality was assessed by following the COnsensus-based Standards for the selection of health status Measurement INstruments (COSMIN) checklist; and 2) the quality of measurement properties was assessed according to Terwee’s criteria. Results were integrated using best-evidence synthesis to look for recommendable instruments.

Results

After screening 2,959 records, 74 instruments from 80 existing studies were included. The overall methodological quality of these studies was unsatisfactory, with reasons including but not limited to unknown missing data, inadequate sample sizes, and vague hypotheses. Content validity, cross-cultural validity, and criterion validity were either unreported or negative ratings in most studies. Based on best-evidence synthesis, three instruments were recommended: Hisar’s instrument for nursing students, Nurse Practitioners’ Roles and Competencies Scale, and Perceived Faculty Competency Inventory.

Conclusion

Although instruments measuring medical professionalism are diverse, only a limited number of studies were methodologically sound. Future studies should give priority to systematically improving the performance of existing instruments and to longitudinal studies.

Use of an Objective Structured Assessment of Technical Skill After a Sports Medicine Rotation

PURPOSE

The purpose of this study was to determine if the use of an Objective Structured Assessment of Technical skill (OSATS), using dry models, would be a valid method of assessing residents' ability to perform sports medicine procedures after training in a competency-based model.

METHODS

Over 18 months, 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4-5]) sat the OSATS after their rotation, in addition to 14 sports medicine staff and fellows. Each resident was provided a list of 10 procedures in which they were expected to show competence. At the end of the rotation, each resident undertook an OSATS composed of 6 stations sampled from the 10 procedures using dry models-faculty used the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task-specific checklists, as well as an overall 5-point global rating scale (GRS) to score each resident. Each procedure was videotaped for blinded review.

RESULTS

The overall reliability of the OSATS (0.9) and the inter-rater reliability (0.9) were both high. A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each technical procedure (P < .001). Further analysis revealed a significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), as well as between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05).

CONCLUSIONS

The results of this study show that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching.

CLINICAL RELEVANCE

As postgraduate medical training shifts to a competency-based model, methods of assessing performance of technical procedures become necessary.

An objective structured clinical exam to measure intrinsic CanMEDS roles

BACKGROUND

The CanMEDS roles provide a comprehensive framework to organize competency-based curricula; however, there is a challenge in finding feasible, valid, and reliable assessment methods to measure intrinsic roles such as Communicator and Collaborator. The objective structured clinical exam (OSCE) is more commonly used in postgraduate medical education for the assessment of clinical skills beyond medical expertise.

METHOD

We developed the CanMEDS In-Training Exam (CITE), a six-station OSCE designed to assess two different CanMEDS roles (one primary and one secondary) and general communication skills at each station. Correlation coefficients were computed for CanMEDS roles within and between stations, and for general communication, global rating, and total scores. One-way analysis of variance (ANOVA) was used to investigate differences between year of residency, sex, and the type of residency program.

RESULTS

In total, 63 residents participated in the CITE; 40 residents (63%) were from internal medicine programs, whereas the remaining 23 (37%) were pursuing other specialties. There was satisfactory internal consistency for all stations, and the total scores of the stations were strongly correlated with the global scores r=0.86, p<0.05. Noninternal medicine residents scored higher in terms of the Professional competency overall, whereas internal medicine residents scored significantly higher in the Collaborator competency overall.

DISCUSSION

The OSCE checklists developed for the assessment of intrinsic CanMEDS roles were functional, but the specific items within stations required more uniformity to be used between stations. More generic types of checklists may also improve correlations across stations.

CONCLUSION

An OSCE measuring intrinsic competence is feasible; however, further development of our cases and checklists is needed. We provide a model of how to develop an OSCE to measure intrinsic CanMEDS roles that educators may adopt as residency programs move into competency-based medical education.

Direct Observation: Assessing Orthopaedic Trainee Competence in the Ambulatory Setting

The Accreditation Council of Graduate Medical Education requires that residency programs teach and assess trainees in six core competencies. Assessments are imperative to determine trainee competence and to ensure that excellent care is provided to all patients. A structured, direct observation program is feasible for assessing nontechnical core competencies and providing trainees with immediate constructive feedback. Direct observation of residents in the outpatient setting by trained faculty allows assessment of each core competency. Checklists are used to document residents' basic communication skills, clinical reasoning, physical examination methods, and medical record keeping. Faculty concerns regarding residents' professionalism, medical knowledge, fatigue, or ability to self-assess are tracked. Serial observations allow for the reinforcement and/or monitoring of skills and attitudes identified as needing improvement. Residents who require additional coaching are identified early in training. Progress in educational milestones is recorded, allowing an individualized educational program that ensures that future orthopaedic surgeons excel across all domains of medical and surgical competence.

Measuring medical students' empathy using direct verbal expressions

PURPOSE

Empathy is an important trait in physicians and a key element in the physician-patient relationship. Accordingly, one of the goals in medical education is developing empathy in students. We attempted to practically assess medical students' empathy through their direct verbal expressions.

METHODS

The medical students' empathy was measured using the modified Pencil-and-Paper Empathy Rating Test by Winefield and Chur-Hansen (2001). The students took 15 minutes or so to complete the scale, and it was then scored by one of two trained evaluators (0 to 4 points for each item, for a total score of 40). The subjects were 605 medical students, and the data were analyzed using descriptive analysis, independent t-test, and one-way analysis of variance in SPSS version 21.0.

RESULTS

The students' empathy scores were low (mean, 12.13; standard deviation, 2.55); their most common responses (78.6%) registered as non-empathetic. Differences in empathy were observed by gender (female students>male students; t=-5.068, p<0.001), school system (medical school>medical college; t=-1.935, p=0.053), and academic level (pre-medical 1 year < other years; t=-4.050, p<0.001).

CONCLUSION

Our findings lead us to the significant conclusion that there is the need for empathy enhancement training programs with practical content.

Exclusion of Residents From Surgery-Intensive Care Team Communication: A Qualitative Study

OBJECTIVE

Communication competency is an important aspect of postgraduate training and patient care delivery in all specialties and clinical domains. This study explored staff surgeon and intensivist perceptions of and experiences with residents' communication with a view toward fostering high quality interspecialty team communication in the surgical intensive care unit.

DESIGN

A qualitative study using semistructured interviews. Data were analyzed iteratively and inductively as per standard qualitative thematic approach.

SETTING

University of Toronto, Toronto, Canada.

PARTICIPANTS

A total of 15 staff surgeons and intensivists who collaborate in patient care in the surgical intensive care unit.

RESULTS

The phenomenon of "resident bypass" emerged, resulting from staff surgeon and intensivist perceptions that residents threaten the quality of interspecialty team communication. Clear patterns and preferences for resident exclusion from this communication were present. A total of 5 interrelated drivers of resident bypass were discovered: lack of trust, lack of specialized knowledge, poor system design, need for timely communication, and residents' inadequate contribution to decision-making. Surgical and intensive care staff were dissatisfied with the structure of residents' roles in interspecialty team communication. Concerns about communication gaps, patient care continuity, and patient safety were expressed.

CONCLUSIONS

Surgical and intensive care staff exclude residents from interspecialty team communication for the benefit of patient safety and care continuity, but this limits opportunities for residents to develop communication skill and competence. Efforts are needed to effectively integrate surgery and intensive care residents in interspecialty attending-resident communication in ways that are meaningful for both patient care and postgraduate training. The implications for medical education are discussed.

Reflections in a time of transition: orthopaedic faculty and resident understanding of accreditation schemes and opinions on surgical skills feedback

INTRODUCTION

Orthopaedic surgery is one of the first seven specialties that began collecting Milestone data as part of the Accreditation Council for Graduate Medical Education's Next Accreditation System (NAS) rollout. This transition from process-based advancement to outcome-based education is an opportunity to assess resident and faculty understanding of changing paradigms, and opinions about technical skill evaluation.

METHODS

In a large academic orthopaedic surgery residency program, residents and faculty were anonymously surveyed. A total of 31/32 (97%) residents and 29/53 (55%) faculty responded to Likert scale assessments and provided open-ended responses. An internal end-of-rotation audit was conducted to assess timeliness of evaluations. A mixed-method analysis was utilized, with nonparametric statistical testing and a constant-comparative qualitative method.

RESULTS

There was greater familiarity with the six core competencies than with Milestones or the NAS (p<0.05). A majority of faculty and residents felt that end-of-rotation evaluations were not adequate for surgical skills feedback. Fifty-eight per cent of residents reported that end-of-rotation evaluations were rarely or never filled out in a timely fashion. An internal audit demonstrated that more than 30% of evaluations were completed over a month after rotation end. Qualitative analysis included themes of resident desire for more face-to-face feedback on technical skills after operative cases, and several barriers to more frequent feedback.

DISCUSSION

The NAS and outcome-based education have arrived. Residents and faculty need to be educated on this changing paradigm. This transition period is also a window of opportunity to address methods of evaluation and feedback. In our orthopaedic residency, trainees were significantly less satisfied than faculty with the amount of technical and surgical skills feedback being provided to trainees. The quantitative and qualitative analyses converge on one theme: a desire for frequent, explicit, timely feedback after operative cases. To overcome the time-limited clinical environment, feedback tools need to be easily integrated and efficient. Creative solutions may be needed to truly achieve outcome-based graduate medical education.

Cognitive and Psychomotor Entrustable Professional Activities: Can Simulators Help Assess Competency in Trainees?

BACKGROUND

An entrustable professional activity describes a professional task that postgraduate residents must master during their training. The use of simulation to assess performance of entrustable professional activities requires further investigation.

QUESTIONS/PURPOSES

(1) Is simulation-based assessment of resident performance of entrustable professional activities reliable? (2) Is there evidence of important differences between Postgraduate Year (PGY)-1 and PGY-4 residents when performing simulated entrustable professional activities?

METHODS

Three entrustable professional activities were chosen from a list of competencies: management of the patient for total knee arthroplasty (TKA); management of the patient with an intertrochanteric hip fracture; and management of the patient with an ankle fracture. Each assessment of entrustable professional activity was 40 minutes long with three components: preoperative management of a patient (history-taking, examination, image interpretation); performance of a technical procedure on a sawbones model; and postoperative management of a patient (postoperative orders, management of complications). Residents were assessed by six faculty members who used checklists based on a modified Delphi technique, an overall global rating scale as well as a previously validated global rating scale for the technical procedure component of each activity. Nine PGY-1 and nine PGY-4 residents participated in our simulated assessment. We assessed reliability by calculating the internal consistency of the mean global rating for each activity as well as the interrater reliability between the faculty assessment and blinded review of videotaped encounters. We sought evidence of a difference in performance between PGY-1 and PGY-4 residents on the overall global rating scale for each station of each entrustable professional activity.

RESULTS

The reliability (Cronbach's α) for the hip fracture activity was 0.88, it was 0.89 for the ankle fracture activity, and it was 0.84 for the TKA activity. A strong correlation was seen between blinded observer video review and faculty scores (mean 0.87 [0.07], p < 0.001). For the hip fracture entrustable professional activity, the PGY-4 group had a higher mean global rating scale than the PGY-1 group for preoperative management (3.56 [0.5] versus 2.33 [0.5], p < 0.001), postoperative management (3.67 [0.5] versus 2.22 [0.7], p < 0.001), and technical procedures (3.11 [0.3] versus 3.67 [0.5], p = 0.015). For the TKA activity, the PGY-4 group scored higher for postoperative management (3.5 [0.8] versus 2.67 [0.5], p = 0.016) and technical procedures (3.22 [0.9] versus 2.22 [0.9], p = 0.04) than the PGY-1 group, but no difference for preoperative management with the numbers available (PGY-4, 3.44 [0.7] versus PGY-1 2.89 [0.8], p = 0.14). For the ankle fracture activity, the PGY-4 group scored higher for postoperative management (3.22 [0.8] versus 2.33 [0.7], p = 0.18) and technical procedures (3.22 [1.2] versus 2.0 [0.7], p = 0.018) than the PGY-1 groups, but no difference for preoperative management with the numbers available (PGY-4, 3.22 [0.8] versus PGY-1, 2.78 [0.7], p = 0.23).

CONCLUSIONS

The results of our study show that simulated assessment of entrustable professional activities may be used to determine the ability of a resident to perform professional tasks that are critical components of medical training. In this manner, educators can ensure that competent performance of these skills in the simulated setting occurs before actual practice with patients in the clinical setting.

How to maintain equity and objectivity in assessing the communication skills in a large group of student nurses during a long examination session, using the Objective Structured Clinical Examination (OSCE)

BACKGROUND

While development, testing, and innovation of the Objective Structured Clinical Examination (OSCE) are common in the international literature, studies from the United States of America (USA), Australia, and the United Kingdom (UK) predominate. There is little known about OSCE use in European countries, such as Italy, where other than cost analysis, there is little reporting of OSCE use or validation.

OBJECTIVES

This paper reports on one Italian initiative, which evaluated the equity and objectivity of the OSCE method of assessing communication skills.

DESIGN

An OSCE method was used to assess the communication skills of first-year students of the Degree Course in Nursing. A method of simulation was implemented through role-playing with standardized patients. An observational method was used to collect data.

PARTICIPANTS AND SETTINGS

Four hundred and twenty-one first-year undergraduate nursing students at one university site in Italy took part.

METHODS

Ten examination sessions were carried out. The students' performances were assessed by two examiners who used a structured observation grid and conducted their assessment separately. A situation simulated by four nurses with experience as actors was used as the topic for the students' examination.

RESULTS

Calculation of the daily rate of students who passed the examination revealed a random distribution over time. The nonparametric correlation indexes referring to the assessments and to the scores assigned by the two examiners proved statistically significant (P≤0.001).

CONCLUSIONS

The study confirmed the validity of the OSCE method in ensuring equity and objectivity of communication skills assessment in a large population of nursing students for the purpose of certification throughout the duration of the examination. This has important implications for nurse education and practice as the extent to which OSCE approaches, while deemed objective, are culturally sensitive or valid and reliable across cultures is not clear. This is something that requires further research and examination in this field.

An assessment of the Chilean National Examination of Orthopaedic Surgery

BACKGROUND

The Chilean National Examination of Orthopaedic Surgery (EMNOT) has been administered since 2009. It was developed to determine whether individual residents are meeting minimal knowledge standards at the end of their training programs.

METHODS

We performed a retrospective evaluation of the EMNOT for all years it has been administered (2009-2015). The test was analyzed for content, taxonomy of questions asked (1: direct recall; 2: diagnosis; 3: evaluation/decision-making), residents' performance, difficulty index and discrimination index.

RESULTS

During the years of EMNOT administration, the most frequently tested areas have been pediatric orthopaedics (22.9 %), spine (13.8 %), general orthopaedics (13.8 %) and musculoskeletal trauma (9.9 %). A significant increase in questions with images was observed, as well as a significant decrease in the percentage of Type 1 and an increase in Type 3 questions. The Difficulty Index showed a medium level of difficulty for all years the examination has been administered. The Discrimination Index showed good discrimination in 2009, fair discrimination from 2010 through 2012, and excellent discrimination from 2013 through 2015.

CONCLUSION

The EMNOT has evolved over several years to include better quality questions, better discrimination, and a more representative distribution of questions covering the different orthopaedic sub-specialties. This examination represents an effective instrument for quality assurance of orthopaedic residency programs in Chile.

One-Year Progress in Ophthalmic Education: Annual Review

PURPOSE

The aim of this study was to update the practicing ophthalmologist on the English language literature about medical education from the prior year.

DESIGN

A search of English language literature was performed on PubMed from January 1, 2014, to December 31, 2014.

METHODS

Because the search using the main topic of the review "medical education" came up with 7394 citations, authors finally decided to narrow the search to 3 topics of their interest:1. Current state of competency-based education and teaching methods of competencies. This section included ophthalmic/ophthalmology education, core competencies, competency-based education, teaching strategies, tools and methods in medical education.2. E-learning. This section included e-learning, online learning, online teaching, Web-based teaching, Web-based learning, and flipped classroom.3.

ASSESSMENTS

This section included assessment of medical students, residents, fellows, faculty, attending physicians, and medical teachers, assessment of medical student ophthalmology programs, ophthalmology residency programs, residency programs, and fellowship programs.

RESULTS

The authors reviewed and summarized articles published in 2014 examining or describing the 3 main areas of the review described previously.

CONCLUSIONS

This review updates the comprehensive ophthalmologist on advances in ophthalmic medical education. Ophthalmic educators could apply the ideas presented in this review according to their possibilities in their own settings and programs.