CanMEDS evaluation in Canadian postgraduate training programmes: tools used and programme director satisfaction

Making assessment a team sport: a qualitative study of facilitated group feedback in internal medicine residency

Purpose

Competency-based medical education relies on feedback from workplace-based assessment (WBA) to direct learning. Unfortunately, WBAs often lack rich narrative feedback and show bias towards Medical Expert aspects of care. Building on research examining interactive assessment approaches, the Queen's University Internal Medicine residency program introduced a facilitated, team-based assessment initiative ("Feedback Fridays") in July 2017, aimed at improving holistic assessment of resident performance on the inpatient medicine teaching units. In this study, we aim to explore how Feedback Fridays contributed to formative assessment of Internal Medicine residents within our current model of competency-based training.

Method

A total of 53 residents participated in facilitated, biweekly group assessment sessions during the 2017 and 2018 academic year. Each session was a 30-minute facilitated assessment discussion done with one inpatient team, which included medical students, residents, and their supervising attending. Feedback from the discussion was collected, summarized, and documented in narrative form in electronic WBA forms by the program's assessment officer for the residents. For research purposes, verbatim transcripts of feedback sessions were analyzed thematically.

Results

The researchers identified four major themes for feedback: communication, intra- and inter-personal awareness, leadership and teamwork, and learning opportunities. Although feedback related to a broad range of activities, it showed strong emphasis on competencies within the intrinsic CanMEDS roles. Additionally, a clear formative focus in the feedback was another important finding.

Conclusions

The introduction of facilitated team-based assessment in the Queen's Internal Medicine program filled an important gap in WBA by providing learners with detailed feedback across all CanMEDS roles and by providing constructive recommendations for identified areas for improvement.

Designing the Future of Medical Education: The EPA Framework as a Catalyst to Inform Family Planning and Reproductive Health Fellowship Training Program in Ethiopia Medical Education: An Exploratory Sequential Mixed Method Study

Introduction

Entrustable Professional Activities (EPAs) are tasks or responsibilities within a specific field that can be given to a learner once they are competent to perform them independently. EPAs are being used in various specialty programs and serving as valuable tool to inform educational program. However, due to disparities in professional practice between different contexts, the automatic transfer of a set of core EPAs is not feasible. Hence, our study aims to develop an EPA framework to inform the Family Planning and Reproductive Health Fellowship Program in the local context of Ethiopia.

Methods

We employed an exploratory mixed-method design, which involved the collection of qualitative data using the Nominal Group Technique and quantitative data through a nationwide survey in all residency training institutions across the country. Qualitative data analysis involved several steps, including compiling a list of tasks, removing duplicate tasks, reviewing EPAs using criteria and an equal rubric tool. For quantitative data analysis, descriptive statistics, validity index analysis, and intra-class correlation coefficients, were used.

Results

Seven senior panelists were able to propose a total of 57 EPAs, with 17 remaining after qualitative data analysis. The panelist evaluated the relevance of each EPA in the second phase. As a result, 17 EPAs received a content validity index of >0.83, indicating satisfactory relevance. In the national survey, experts reached a high level of final agreement regarding the relevance and representativeness of all 17 EPAs (ICC = 0.815, 95% CI [0.0.756,0.865], p.0001).

Conclusion

The final set of 17 end-of-training EPAs is valid, acceptable and representative of the discipline, and they can be used as a framework to inform Family planning and Reproductive Health Fellowship Program in Ethiopian medical education once these core EPA statements are described in sufficient detail. This can contribute to raise the quality of training and hence the quality of patient care.

Integrating the social determinants of health into curriculum: AMEE Guide No. 162

The World Health Organization (WHO) defines the Social Determinants of Health (SDOH) as the non-medical factors influencing health outcomes. SDOH is associated with conditions in which people are born, grow, work, and live. Medical schools and licensing bodies are increasingly recognizing the need for doctors and healthcare professionals to be aware of their patient's social context and how it impacts their states of health and disease. However, there is considerable variation in the approaches of different institutions and countries to incorporating SDOH into their curricula. In order to allow clinicians to adopt a holistic approach to patient health, equipping them with extensive knowledge of SDOH would give learners the confidence, skills, knowledge, and attitudes needed to effectively engage with patients and their families. This approach aids health professionals with knowledge of the influence of the social context and cultural factors that affect patients' behaviors in relation to health. Incorporating the SDOH in medical and health professional school curricula would contribute towards adequately preparing future healthcare practitioners to provide effective, comprehensive, and equitable care, especially to marginalized and underserved populations. The Guide will take an evidence-based approach grounded in the available contemporary literature and case studies. The focus will be on integrating SDOH into undergraduate and postgraduate medical curricula to promote an understanding of the social factors that influence patients' and communities' health. Ultimately, this guide seeks to contribute to the reduction of inequalities in health.

The CanMEDS Competency Framework in laboratory medicine: a phenomenographic study exploring how professional roles are applied outside the clinical environment

Background

The CanMEDS Competency Framework is an internationally recognized model used to outline the proficiencies of a physician. It has predominantly been studied in clinical environments but not all medical specialties take part in direct patient contact. In laboratory medicine, the role of the physician is to promote and enhance patient diagnostics by managing and overseeing the functions of a diagnostic laboratory.

Methods

This phenomenographic study explores the lived experiences of biochemistry, microbiology, and pathology residency program directors to better understand how they utilize the CanMEDS competencies. Eight laboratory medicine program directors from across Canada were individually interviewed using a semi-structured interview, and the data was analysed using inductive thematic analysis.

Results

The findings show that the current framework is disconnected from the unique context of laboratory medicine with some competencies appearing unrelatable using the current standardized definitions and expectations. Nevertheless, participants considered the framework to be an appropriate blueprint of the competencies necessary for their professional environment, but to make it accessible more autonomy is required to adapt the framework to their needs.

Conclusion

Newer renditions of the CanMEDS Competency Framework should better consider the realities of non-clinical disciplines.

5 Slides in 5 Minutes: Incorporating CanMEDS Competencies on Acute Care Surgery

BACKGROUND

The University of British Columbia (UBC) Division of General Surgery developed an initiative entitled "5-in-5s" to improve educational opportunities on the Acute Care Surgery (ACS) service. We examined whether 5-in-5s are felt to be a valuable teaching tool, and evaluated their ability to incorporate CanMEDS competencies within the General Surgery program.

METHODS

A web-based survey was distributed to all general surgery trainees and staff on ACS that have participated in 5-in-5s.

RESULTS

A total of 37 responses were collected (62% response rate). All respondents felt 5-in-5s were valuable overall. Four of the seven CanMEDS competencies were evaluated. About 100% felt their knowledge was positively impacted by presenting, and 80% by attending alone. About 71% of respondents agreed that 5-in5s provided opportunities for health advocacy, 50% for collaboration, and 36% for leadership.

CONCLUSION

We identified 5-in-5s as a valuable teaching method and a novel approach to integrate CanMEDS competencies into ACS training.

Teaching in radiation oncology: now and 2025-results of a focus group with medical students

PURPOSE

In Germany, the new Licensing Regulations for Physicians 2025 (Ärztliche Approbationsordnung, ÄApprO) define a binding legal framework on the basis of which medical faculties modernize their curricula. Since 2015, the National Competence Based Learning Objectives Catalogue for Medicine 2.0 (Nationaler Kompetenzbasierter Lernzielkatalog 2.0., NKLM) formulates competencies and learning objectives to be achieved in the course of studies as curriculum orientation for the medical faculties. In addition, about 80% of the areas of a new core curriculum are to be made compulsory. A needs analysis in the target group of students has not yet taken place for the subject of radiation therapy (RT) or radiation oncology (RO). This study therefore surveys the experiences and requirements of students regarding medical education in RT.

METHODS

Qualitative single-center study using a semistructured in-depth focus group with 11 medical students (20-26 years; 6 female, 5 male) was conducted. Brainstorming sessions were conducted in small groups and individually; oral contributions were recorded, transcribed, and analyzed using qualitative content analysis according to Mayring. Results were compared with the content of the future curriculum and reviewed for congruence with current expert recommendations of the German Society of Radiation Oncology (Deutsche Gesellschaft für Radioonkologie, DEGRO).

RESULTS

The plans to develop a longitudinal and practice-oriented curriculum was positively received by students. Specifically, students wanted to introduce the basics of RT as an early link to practice in preclinical teaching units. The necessary acquisition of communicative skills should also be taught by lecturers in RO. Methodologically, regular digital survey tools for self-monitoring, discussion rooms, and problem-based learning were named. In the perception of students, the subject appears underrepresented in relation to its relevance in the multimodal therapy of oncological diseases.

CONCLUSION

Results of the needs analysis for the subject of RT are consistent with ÄApprO, NKLM, and DEGRO. Moreover, they complement them and should be considered in the curriculum development of Masterplan Medical Education 2020 (Masterplan Medizinstudium 2020). The results contribute to high-quality and target-group-oriented medical training in the subject of RT, increased visibility, and thus early bonding of future physicians to RO in Germany.

Grappling with key questions about assessment of the Health Advocate role

Introduction

Although the CanMEDS framework sets the standard for Canadian training, health advocacy competence does not appear to factor heavily into high stakes assessment decisions. Without forces motivating uptake, there is little movement by educational programs to integrate robust advocacy teaching and assessment practices. However, by adopting CanMEDS, the Canadian medical education community endorses that advocacy is required for competent medical practice. It's time to back up that endorsement with meaningful action. Our purpose was to aid this work by answering the key questions that continue to challenge training for this intrinsic physician role.

Methods

We used a critical review methodology to both examine literature relevant to the complexities impeding robust advocacy assessment, and develop recommendations. Our review moved iteratively through five phases: focusing the question, searching the literature, appraising and selecting sources, and analyzing results.

Results

Improving advocacy training relies, in part, on the medical education community developing a shared vision of the Health Advocate (HA) role, designing, implementing, and integrating developmentally appropriate curricula, and considering ethical implications of assessing a role that may be risky to enact.

Conclusion

Changes to assessment could be a key driver of curricular change for the HA role, provided implementation timelines and resources are sufficient to make necessary changes meaningful. To truly be meaningful, however, advocacy first needs to be perceived as valuable. Our recommendations are intended as a roadmap for transforming advocacy from a theoretical and aspirational value into one viewed as having both practical relevance and consequential implications.

Benchmarking a Canadian Anesthesiology Resident Research Program against national norms using a logic model framework: a quality improvement study

Background

Canadian specialty training programs are expected to deliver curriculum content and assess competencies related to the CanMEDS Scholar role. We evaluated our residency research program and benchmarked it against national norms for quality improvement purposes.

Methods

In 2021 we reviewed departmental curriculum documents and surveyed current and recently graduated residents. We applied a logic model framework to assess if our program's inputs, activities, and outputs addressed the relevant CanMeds Scholar competencies. We then descriptively benchmarked our results against a 2021 environmental scan of Canadian anesthesiology resident research programs.

Results

Local program content was successfully mapped to competencies. The local survey response rate was 40/55 (73%). In benchmarking, our program excelled in providing milestone-related assessments, research funding, administrative, supervisory, and methodologic support, and requiring a literature review, proposal presentation, and local abstract submission as output. Acceptable activities to meet research requirements vary greatly among programs. Balancing competing clinical and research responsibilities was a frequently reported challenge.

Conclusions

The logic model framework was easily applied and demonstrated our program benchmarked well against national norms. National level dialogue is needed to develop specific, consistent scholar role activities and competency assessments to bridge the gap between expected outcome standards and education practice.

Scholarly practice in healthcare professions: findings from a scoping review

Scholarly practitioners are broadly defined as healthcare professionals that address critical practice problems using theory, scientific evidence, and practice-based knowledge. Though scholarly practice is included in most competency frameworks, it is unclear what scholarly practice is, how it develops and how it is operationalized in clinical practice. The aim of this review was to determine what is known about scholarly practice in healthcare professionals. We conducted a scoping review and searched MEDLINE, EMBASE, CINAHL from inception to May 2020. We included papers that explored, described, or defined scholarly practice, scholar or scholarly practitioner, and/or related concepts in healthcare professionals. We included a total of 90 papers. Thirty percent of papers contained an explicit definition of scholarly practice. Conceptualizations of scholarly practice were organized into three themes: the interdependent relationship between scholarship and practice; advancing the profession's field; and core to being a healthcare practitioner. Attributes of scholarly practitioners clustered around five themes: commitment to excellence in practice; collaborative nature; presence of virtuous characteristics; effective communication skills; and adaptive change ethos. No single unified definition of scholarly practice exists within the literature. The variability in terms used to describe scholarly practice suggests that it is an overarching concept rather than a definable entity. There are similarities between scholarly practitioners and knowledge brokers regarding attributes and how scholarly practice is operationalized. Individuals engaged in the teaching, research and/or assessment of scholarly practice should make explicit their definitions and expectations for healthcare professionals.

Exploring the Nexus of Healthcare Employees' Professional Quality, Health Psychology and Service Value: A Qualitative Study

While the implementation of the "graded diagnosis and treatment" system highlights the important role of general practitioners as "residents' health gatekeepers", it brings the problem of insufficient service capacity and difficulty in realizing the service value. At present, the service value of general practitioners is a relatively new topic in the field of general medicine. Therefore, few studies discuss the specific path that affects the realization of their service value. According to literature analysis, the professional quality of general practitioners plays a positive role in improving their service quality. So it can be inferred that the main reason for this phenomenon is that the professional quality level of general practitioners as the service subject is low and they have not been trusted and recognized by the residents of the service object. So far, it is difficult for most residents to change their willingness to go to large hospitals. Training is the most critical link to improving the professional quality of general practitioners. Therefore, how to enhance the professional quality of general practitioners through effective training so as to realize the service value is a problem worth discussing. Our study took 37 general practitioners from 12 Community Health Service hospitals as the interviewees and used grounded theory to mine the internal correlation between variables. The results show that: (1) the professional quality of general practitioners mainly includes three dimensions: professional ethics, theoretical knowledge, and professional skills; (2) through training, the professional quality of general practitioners has been effectively improved; (3) the improvement of general practitioners' professional quality directly affects the realization of their technical value, environmental value and information value; (4) the professional quality of general practitioners can be improved through training, which will affect the realization of their service value. Our research contribution is to break through the previous research paradigm of analyzing the relationship between variables based on the existing literature. This paper uses the procedural grounded theory method to analyze the concept of general practitioners' professional quality from scratch through continuous refinement and summary and constructs a theoretical model of the training path from general practitioners' professional quality to service value. On the one hand, the research results can realize their service value by improving the professional quality of general practitioners. On the other hand, the realization of the service value of general practitioners can provide effective support for patients to create a good medical environment.

Intrinsic or Invisible? An Audit of CanMEDS Roles in Entrustable Professional Activities

PURPOSE

Postgraduate medical education in Canada has quickly transformed to a competency-based model featuring new entrustable professional activities (EPAs) and associated milestones. It remains unclear, however, how these milestones are distributed between the central medical expert role and 6 intrinsic roles of the larger CanMEDS competency framework. A document review was thus conducted to measure how many EPA milestones are classified under each CanMEDS role, focusing on the overall balance between representation of intrinsic roles and that of medical expert.

METHOD

Data were extracted from the EPA guides of 40 Canadian specialties in 2021 to measure the percentage of milestones formally linked to each role. Subsequent analyses explored for differences when milestones were separated by stage of postgraduate training, weighted by an EPA's minimum number of observations, or sorted by surgical and medical specialties.

RESULTS

Approximately half of all EPA milestones (mean = 48.6%; 95% confidence interval [CI] = 45.9, 51.3) were classified under intrinsic roles overall. However, representation of the health advocate role was consistently low (mean = 2.95%; 95% CI = 2.49, 3.41), and some intrinsic roles-mainly leader, scholar, and professional-were more heavily concentrated in the final stage of postgraduate training. These findings held true under all conditions examined.

CONCLUSIONS

The observed distribution of roles in EPA milestones fits with high-level descriptions of CanMEDS in that intrinsic roles are viewed as inextricably linked to medical expertise, implying both are equally important to cultivate through curricula. Yet a fine-grained analysis suggests that a low prevalence or late emphasis of some intrinsic roles may hinder how they are taught or assessed. Future work must explore whether the quantity or timing of milestones shapes the perceived value of each role, and other factors determining the optimal distribution of roles throughout training.

The 2021 European Training Requirements in Paediatric Endocrinology and Diabetes

The aims of the 2021 European Training Requirements (ETR) in Paediatric Endocrinology and Diabetes (PED) are to (1) provide standards to harmonize training programmes in PED between different European countries, (2) establish clearly defined standards of knowledge and skills required to practice PED at the tertiary care level, (3) foster the development of a network of competent tertiary care centres for PED in Europe and globally, and (4) improve the quality of care for children and adolescents requiring PED services. This ETR in PED specifies the requirements for training institutions, trainers, and trainees. It also provides the detailed syllabus/core content that trainees are expected to achieve in order to become competent independent clinicians in PED. References to consensus guidelines produced and/or endorsed by ESPE are included. The target users are trainees in PED, trainers, and all involved with quality assurance and accreditation. The process to develop and approve this 2021 ETR has been rigorous and involved trainees and consultants in paediatric and adult Endocrinology, ESPE (Syllabus Task Force, Education and Training Committee, Council), European Academy of Paediatrics (Tertiary Care Council, Assembly), European Board of Paediatrics, and Union of European Medical Specialists. Implementing the ETR will complement professional regulatory requirements for postgraduate training in PED in different countries and allow harmonizing standards across Europe. ETR is publicly available at www.eurospe.org/education/education-training-syllabus and at https://www.uems.eu/__data/assets/pdf_file/0007/133990/UEMS-2021.17-European-Training-Requirement-in-Paediatric-Endocrinology.pdf.

[Project report on fostering scientific competencies in pain medicine in the context of student education]

AIM

The training of scientific skills and competencies is an essential part of academic medical studies. As part of the MaReCuM model study program at Heidelberg University's Mannheim Medical School, a fifth-year rotation on scientific skills in the field of pain medicine was implemented. This paper describes this competence-oriented rotation as well as the investigation of the educational effect.

METHOD

A total of 114 fifth-year medical students participated in the survey (response rate: 83%). The control group completed the fifth year prior to the implementation of the rotation. The experimental group was required to participate in the rotation and the real healthcare research study "Case management program: low back pain". A survey of both groups was conducted on the first day of the rotation and at the end of the module.

RESULTS

The innovative and competency-based learning unit was successfully implemented as part of the MaReCuM model study program and carried out with partners in general practice as well as the Mannheim Institute of Public Health. The participating students accepted the rotation well. There was no measurable effect on the subjective learning success of the rotation in the evaluation.

DISCUSSION

To the authors' knowledge, this educational approach has never been tested before in a German study program. The presented rotation offers an additional option for the training of scientific competencies as part of medical studies. The missing of a measurable effect could be due to the extensive experience of the medical students as well as the limitations on participation in a real healthcare study. An additional learning opportunity could be created by connecting the preexisting lectures to a longitudinal module on scholarly competencies. The implementation of the program also offers a unique opportunity for educational research on the acquisition of scientific competencies in medical students.

Development of an Entrustable Professional Activities (EPA) Framework to Inform Surgical Residency Training Programs in Ethiopia: A Three-round National Delphi Method Study

BACKGROUND

Entrustable Professional Activities (EPAs) have been proposed as a means to translate competencies into clinical practice. Although EPAs for residency training have become available, 1 set of core EPAs cannot automatically be transferred from one context to another due to cultural variability. Further, there is a lack of African- and Asian-based EPA development and implementation studies. We developed an end-of-training EPAs framework to inform surgical residency training programs in the local context of Ethiopian medical education.

METHODS

A three-round Delphi method was used to establish consensus about important surgical EPAs among experts. A total of 136 experts representing all surgical residency training institutions in Ethiopia were invited to participate. Round 1 & 2 consisted of senior expert panelists (n = 8) to identify potential EPAs and determine the content validity. Round 3 consisted of a survey (n = 128) to further validate the identified EPAs by attending surgeons who work with them. Each EPA had to achieve at least 80% or higher agreement among experts to be considered having acceptable content validity.

RESULTS

In round 1, a total of 272 EPAs were proposed, reduced, and grouped to 39 consented EPAs. In round 2, the same experts rated each EPA's relevance, resulting in 32 EPAs with a satisfactory item-level content validity index (I-CVI > 0.83). Overall, in the survey in round 3, 29 EPAs met the standard criterion for acceptability (S-CVI/Ave = 0.90) and achieved a high degree of final consensus (ICC = 0.998, 95% CI [0.996, 0.999]; (F = 439.2, p < 0.0001).

CONCLUSIONS

The framework of 29 validated and accepted EPAs can guide future surgical residency training programs in the Ethiopian medical education context. The framework allows programs to move from a time-dependent to an outcome-based model and transforms traditional assessment into entrustment decisions. Thus, the use of the framework can improve the quality of training and patient care in Ehtiopia.

Challenges for Family Medicine Residents in Attaining the CanMEDS Professional Role: A Thematic Analysis of Preceptor Field Notes

PURPOSE

Among the roles of the competent physician is that of a professional, according to the Canadian Medical Education Directives for Specialists (CanMEDS) framework, which describes the abilities physicians require to effectively meet the health care needs of the people they serve. Through examination of preceptor field notes on resident performance, the authors identified aspects of this role with which family medicine residents struggle.

METHOD

The authors used a structured thematic analysis in this qualitative study to explore the written feedback postgraduate medical learners receive at the University of Toronto Department of Family and Community Medicine. Seventy field notes written between 2015 and 2017 by clinical educators for residents who scored "below expectation" in the CanMEDS professional role were analyzed. From free-text comments, the authors derived inductive codes, amalgamated the codes into themes, and measured the frequency of the occurrence of the codes. The authors then mapped the themes to the key competencies of the CanMEDS professional role.

RESULTS

From the field notes, 7 themes emerged that described reasons for poor performance. Lack of collegiality, failure to adhere to standards of practice or legal guidelines, and lack of reflection or self-learning were identified as major issues. Other themes were failure to maintain boundaries, taking actions that could have a negative impact on patient care, failure to maintain patient confidentiality, and failure to engage in self-care. When the themes were mapped to the key competencies in the CanMEDS professional role, most related to the competency "commitment to the profession."

CONCLUSIONS

This study highlights aspects of professional conduct with which residents struggle and suggests that the way professionalism is taught in residency programs-and at all medical training levels-should be reassessed. Educational interventions that emphasize learners' commitment to the profession could enhance the development of more practitioners who are consummate professionals.

Not wanted on the voyage: highlighting intrinsic CanMEDS gaps in Competence by Design curricula

BACKGROUND

As governing bodies design new curricula that seek to further incorporate principles of competency-based medical education within time-based models of training, questions have been raised regarding the continued centrality of existing CanMEDS competencies. Although efforts have been made to align these new curricula with CanMEDS, we don't yet know to what extent these competencies are meaningfully integrated.

METHODS

A content analysis approach was used to systematically evaluate national Canadian curricula for 18 residency-training programs and determine the number of times each enabling CanMEDS competency was represented.

RESULTS

Clear trends persisted across all programs. Medical Expert and Collaborator competencies were well integrated into curriculum (81% and 86% mapped to assessment) while competencies related to the Leader, Professional, and Health Advocate roles were less frequently mapped to assessment (41%, 36%, and 40%) and were often absent from the new curricula altogether (59%, 64%, and 60%).

CONCLUSION

Deliberate planning in curriculum development affords the early identification of gaps. These gaps can inform current assessment practice and future curricular development by providing direction for innovation. If we are to ensure that any new curricula meaningfully address all CanMEDS roles, we need to think carefully about how to best teach and assess underrepresented competencies.

"When in Doubt, Ask the Patient": A Quantitative, Patient-Oriented Approach to Formative Assessment of CanMEDS Roles

INTRODUCTION

Since the introduction of competency-based frameworks into postgraduate medical curricula, educators have struggled to implement robust assessment tools that document the progression of necessary skills. The global movement towards competency-based medical education demands validated assessment tools. Our objective was to provide validity evidence for the Ottawa CanMEDS Competency Assessment Tool (OCCAT), designed to assess clinical performance in the communicator, professional, and health advocate CanMEDS roles.

METHODS

We developed the OCCAT, a 29-item questionnaire informed by specialty-specific Entrustable Professional Activities and consultation with stakeholders, including patients. Our sample included nine neonatal-perinatal medicine and maternal fetal medicine fellows rotating through antenatal high-risk clinics at the Ottawa Hospital. Following 70 unique encounters, the OCCAT was completed by patients and learners. Generalizability theory was used to determine overall reliability of scores. Differences in self and patient ratings were assessed using analyses of variance.

RESULTS

Generalizability analysis demonstrated that both questionnaires produced reliable scores (G-coefficient > 0.9). Self-scores were significantly lower than patient scores across all competencies, F(1, 6) = 13.9, p = .007. Variability analysis demonstrated that trainee scores varied across all competencies, suggesting both groups were able to recognize competencies as distinct and discriminate favorable behaviors belonging to each.

DISCUSSION

Our findings lend support to the movement to integrate self-assessment and patient feedback in formal evaluations for the purpose of enriched learner experiences and improved patient outcomes. We anticipate that the OCCAT will facilitate bridging to competency-based medical education.

Fundamentos para un currículo nacional en Cirugía General basado en competencia: Consenso Delphi de la División de Educación de la Asociación Colombiana de Cirugía

Frente a los desafíos de la Educación Basada en Competencia, este consenso Delphi de la División de Educación de la Asociación Colombiana de Cirugía informa sobre las mínimas competencias profesionales esperadas del cirujano egresado de los veinte programas de Especialización en Cirugía General en Colombia. Un total de 105 profesores de los programas de especialización evaluaron tres áreas de competencia profesional: 1) atributos profesionales generales del residente durante su formación, 2) competencias prácticas (procedimientos quirúrgicos) que los residentes deben realizar al final de su entrenamiento y 3) Actividades Profesionales Confiables (APC) que los residentes deben ejecutar sin supervisión al final de su entrenamiento. Los resultados informan un alto nivel de consenso en el 100 % los atributos profesionales generales y APC, y del 75 % en diferentes procedimientos quirúrgicos. El consenso abre la puerta para el desarrollo de un currículo nacional de la especialidad y tiene implicaciones para la práctica educativa e investigación futura.

Program directors' reflections on national policy change in medical education: insights on decision-making, accreditation, and the CanMEDS framework

BACKGROUND

Outcomes of national policy change impact all levels of the organizational hierarchy. The medical education literature is sparse on how reflections from program directors (PDs) on past large-scale policy changes can inform future policy initiatives. To fill this gap, we conducted a national survey on PDs' perceptions of, and reflections on, decision-making in medical education, accreditation procedures, and the CanMEDS framework implementation.

METHODS

The survey was distributed to former Canadian specialty medicine PDs (N = 684). Descriptive analysis was performed on quantitative data, thematic analysis was performed on qualitative comments, and comparisons between the quantitative and qualitative findings were performed to identify areas of convergence and/or divergence.

RESULTS

A total of 265 (38.7%) former PDs participated. Quantitative analysis revealed that 52.8% of respondents did not feel involved in decision-making regarding policy changes, 45.1% of respondents did not feel prepared to assess the CanMEDS Roles, and PDs were divided on the reasonableness of accreditation documentation. Qualitative analysis produced four themes: communication, resources, expectations of outcomes, and buy-in. Nine sub-themes were also identified. A high level of convergence was identified across the content, with only four areas of divergence identified.

CONCLUSIONS

Our findings have the potential to inform future policy and/or accreditation changes. Without the lens of those charged with overseeing the implementation, policy evaluation and quality improvement will remain uninformed. PDs, therefore, bring unique insights into our understanding of national policy changes, and without the voices of these frontline implementers, the true success of policy change implementation will be hindered.

Which Aspects of the CanMEDS Competencies are Most Valued in Radiologists? Perspectives of Trainees From Other Specialties

PURPOSE

Radiologists work primarily in collaboration with other healthcare professionals. As such, these stakeholder perspectives are of value to the development and assessment of educational outcomes during the transition to competency-based medical education. Our aim in this study was to determine which aspects of the Royal College CanMEDS competencies for diagnostic radiology are considered most important by future referring physicians.

METHODS

Institutional ethics approval was obtained. After pilot testing, an anonymous online survey was sent to all residents and clinical fellows at our university. Open-ended questions asked respondents to describe the aspects of radiologist service they felt were most important. Thematic analysis of the free-text responses was performed using a grounded theory approach. The resulting themes were mapped to the 2015 CanMEDS Key Competencies.

RESULTS

115 completed surveys were received from residents and fellows from essentially all specialties and years of training (out of 928 invited). Major themes were 1) timeliness and accessibility of service, 2) quality of reporting, and 3) acting as a valued team member. The competencies identified as important by resident physicians were largely consistent with the CanMEDS framework, although not all key competencies were covered in the responses.

CONCLUSIONS

This study illustrates how CanMEDS roles and competencies may be exemplified in a concrete and specialty-specific manner from the perspective of key stakeholders. Our survey results provide further insight into specific objectives for teaching and assessing these competencies in radiology residency training, with the ultimate goal of improving patient care through strengthened communication and working relationships.

Exploring patients' and physicians' perspectives about competent health advocacy

INTRODUCTION

Many residency programmes struggle to demonstrate how they prepare trainees to become competent health advocates. To meaningfully teach and assess it, we first need to understand what 'competent' health advocacy (HA) is and what competently enacting it requires. Attempts at clarifying HA have largely centred around the perspectives of consultant physicians and trainees. Without patients' perspectives, we risk training learners to advocate in ways that may be misaligned with patients' needs and goals. Therefore, the purpose of our research was to generate a multi-perspective understanding about the meaning of competence for the HA role.

METHODS

We used constructivist grounded theory to explore patients' and physicians' perspectives about competent health advocacy. Data were collected using photo elicitation; patients (n = 10) and physicians (n = 14) took photographs depicting health advocacy that were used to inform semi-structured interviews. Themes were identified using constant comparative analysis.

RESULTS

Physician participants associated HA with disruption or political activism, suggesting that competence hinged on medical and systems expertise, a conducive learning environment, and personal and professional characteristics including experience, status and political savvy. Patient participants, however, equated physician advocacy with patient centredness, perceiving that competent HAs are empathetic and attentive listeners. In contrast to patients, few physicians identified as advocates, raising questions about their ability to train or to thoughtfully assess learners' abilities.

CONCLUSION

Few participants perceived HA as a fundamental physician role-at least not as it is currently defined in curricular frameworks. Misperceptions that HA is primarily disruptive may be the root cause of the HA problem; solving it may rely on focusing training on bolstering skills like empathy and listening not typically associated with the HA role. Since there may be no competency where the patient voice is more critical, we need to explore opportunities for patients to facilitate learning for the HA role.

Utilization of evidence-based tools and medical education literature by Canadian postgraduate program directors in the teaching and assessment of the CanMEDS roles

BACKGROUND

Researchers have shown that clinical educators feel insufficiently informed about how to teach and assess the CanMEDS roles. Thus, our objective was to examine the extent to which program directors utilize evidence-based tools and the medical education literature in teaching and assessing the CanMEDS roles.

METHODS

In 2016, the authors utilized an online questionnaire to survey 747 Canadian residency program directors (PD's) of Royal College of Physicians and Surgeons of Canada (RCPSC) accredited programs.

RESULTS

Overall, 186 PD's participated (24.9%). 36.6% did not know whether the teaching strategies they used were evidence-based and another third (31.9%) believed they were "not at all" or "to a small extent" evidence-based. Similarly, 31.8% did not know whether the assessment tools they used were evidence-based and another third (39.7%) believed they were "not at all" or "to a small extent" evidence-based. PD's were aware of research on teaching strategies (62.4%) and assessment tools (51.9%), but felt they did not have sufficient time to review relevant literature (72.1% for teaching and 64.1% for assessment).

CONCLUSIONS

Canadian PD's reported low awareness of evidence-based tools for teaching and assessment, implying a potential knowledge translation gap in medical education research.

Analysis of Supervisors' Feedback to Residents on Communicator, Collaborator, and Professional Roles During Case Discussions

BACKGROUND

Literature examining the feedback supervisors give to residents during case discussions in the realms of communication, collaboration, and professional roles (intrinsic roles) focuses on analyses of written feedback and self-reporting.

OBJECTIVES

We quantified how much of the supervisors' verbal feedback time targeted residents' intrinsic roles and how well feedback time was aligned with the role targeted by each case. We analyzed the educational goals of this feedback. We assessed whether feedback content differed depending on whether the residents implied or explicitly expressed a need for particular feedback.

METHODS

This was a mixed-methods study conducted from 2017 to 2019. We created scripted cases for radiology and internal medicine residents to present to supervisors, then analyzed the feedback given both qualitatively and quantitatively. The cases were designed to highlight the CanMEDS intrinsic roles of communicator, collaborator, and professional.

RESULTS

Radiologists (n = 15) spent 22% of case discussions providing feedback on intrinsic roles (48% aligned): 28% when the case targeted the communicator role, 14% for collaborator, and 27% for professional. Internists (n = 15) spent 70% of discussions on intrinsic roles (56% aligned): 66% for communicator, 73% for collaborator, and 72% for professional. Radiologists' goals were to offer advice (66%), reflections (21%), and agreements (7%). Internists offered advice (41%), reflections (40%), and clarifying questions (10%). We saw no consistent effects when residents explicitly requested feedback on an intrinsic role.

CONCLUSIONS

Case discussions represent frequent opportunities for substantial feedback on intrinsic roles, largely aligned with the clinical case. Supervisors predominantly offered monologues of advice and agreements.

Residents' perspectives: Does the Saudi general surgery residency training program apply CanMEDS competencies and prepare future leaders?

BACKGROUND

This study investigates leadership skills and Canadian Medical Education Directives for Specialists (CanMEDS) competencies acquisition within the General Surgery Residency Training Program (GSRTP). The Saudi Commission for Health Specialties (SCFHS) incorporates the CanMEDS Competency Framework into its curriculum to prepare the resident for healthcare needs.

METHODS

This is a descriptive-analytical study. A questionnaire was used to collect data from 117 General Surgery residents (GS) at seven institutes in Jeddah, Saudi Arabia.

RESULTS

The GS residents reported an acceptable self-perceived level of Clinical Leadership Skills (mean ± standard deviation). The most dominant skill was working with others (1.98 ± 1.03), followed by demonstrating personal qualities (2.07 ± 0.88), the ability to manage services (2.21 ± 1.37), improving services (2.22 ± 1.84) and last, setting directions (2.39 ± 0.95). Regarding the CanMEDS competencies, the respondents showed a generally positive perception with an "agree" level (Mean = 1.83). Of the CanMEDS competency roles, Collaborator ranked first followed by Professional and then Communicator. Leader competency ranked fourth followed by Health Advocate, Medical Expert and last, Scholar.

CONCLUSION

The GSRTP residents showed satisfactory self-assessed clinical leadership skills and acquirement of the CanMEDS competencies during their training, which will prepare them to lead in the future.

The impact of a global health elective on CanMEDS competencies and future practice

BACKGROUND

There is evidence that participating in global health electives generates positive educational outcomes and personal benefits for medical trainees. The objective of this study was to examine the effect and impact that a global health elective has on CanMEDS competencies and anticipated future practice.

RESULTS

The medical expert, collaborator, leader, scholar, and professional CanMEDS competencies were self-perceived to be strongly impacted through this elective. A total of 94% of participants indicated it increased their strengths as a medical expert and leader, 82% indicated a major impact on the scholar competency, 88% of participants reported a strong impact as a professional, and 76% of participants indicated that it strongly impacted them as a collaborator. The majority of participants continue to have involvement in global health, and 88% of respondents found this elective to be influential on their current practice and beliefs.

CONCLUSIONS

These results suggest that individuals who participated in this global health elective perceived value in their experience. These findings support our hypothesis that participation in this global health elective would generate self-perceived positive impacts. Global health electives may provide an opportunity for physicians to expand on their CanMEDS competencies and become more proficient in caring for diverse patient populations.

Competencies and Feedback on Internal Medicine Residents' End-of-Rotation Assessments Over Time: Qualitative and Quantitative Analyses

PURPOSE

To examine how qualitative narrative comments and quantitative ratings from end-of-rotation assessments change for a cohort of residents from entry to graduation, and explore associations between comments and ratings.

METHOD

The authors obtained end-of-rotation quantitative ratings and narrative comments for 1 cohort of internal medicine residents at the University of Illinois at Chicago College of Medicine from July 2013-June 2016. They inductively identified themes in comments, coded orientation (praising/critical) and relevance (specificity and actionability) of feedback, examined associations between codes and ratings, and evaluated changes in themes and ratings across years.

RESULTS

Data comprised 1,869 assessments (828 comments) on 33 residents. Five themes aligned with ACGME competencies (interpersonal and communication skills, professionalism, medical knowledge, patient care, and systems-based practice), and 3 did not (personal attributes, summative judgment, and comparison to training level). Work ethic was the most frequent subtheme. Comments emphasized medical knowledge more in year 1 and focused more on autonomy, leadership, and teaching in later years. Most comments (714/828 [86%]) contained high praise, and 412/828 (50%) were very relevant. Average ratings correlated positively with orientation (β = 0.46, P < .001) and negatively with relevance (β = -0.09, P = .01). Ratings increased significantly with each training year (year 1, mean [standard deviation]: 5.31 [0.59]; year 2: 5.58 [0.47]; year 3: 5.86 [0.43]; P < .001).

CONCLUSIONS

Narrative comments address resident attributes beyond the ACGME competencies and change as residents progress. Lower quantitative ratings are associated with more specific and actionable feedback.

The state of health advocacy training in postgraduate medical education: a scoping review

CONTEXT

Health advocacy is an essential component of postgraduate medical education, and is part of many physician competency frameworks such as the Canadian Medical Education Directives for Specialists (CanMEDS) roles. There is little consensus about how advocacy should be taught and assessed in the postgraduate context. There are no consolidated guides to assist in the design and implementation of postgraduate health advocacy curricula.

OBJECTIVES

This scoping review aims to identify and analyse existing literature pertaining to health advocacy education and assessment in postgraduate medicine. We specifically sought to summarise themes from the literature that may be useful to medical educators to inform further health advocacy curriculum interventions.

METHODS

MEDLINE, Embase and ERIC were searched using MeSH (medical student headings) and non-MeSH search terms. Additional articles were found using forward snowballing. The grey literature search included Google and relevant stakeholder websites, regulatory bodies, physician associations, government agencies and academic institutions. We followed a stepwise scoping review methodology, followed by thematic analysis using an inductive approach.

RESULTS

Of the 123 documents reviewed in full, five major themes emerged: (i) conceptions of health advocacy have evolved towards advocating with rather than for patients, communities and populations; (ii) longitudinal curricula were less common but appeared the most promising, often linked to scholarly or policy objectives; (iii) hands-on, immersive opportunities build competence and confidence; (iv) community-identified needs and partnerships are increasingly considered in designing curriculum, and (v) resident-led and motivated programmes appear to engage residents and allow for achievement of stated outcomes. There remain significant challenges to assessment of health advocacy competencies, and assessment tools for macro-level health advocacy were notably absent.

CONCLUSIONS

There is considerable heterogeneity in the way health advocacy is taught, assessed and incorporated into postgraduate curricula across programmes and disciplines. We consolidated recommendations from the literature to inform further health advocacy curriculum design, implementation and assessment.

Competency-based medical education in ophthalmology residency training: a review

Competency-based medical education (CBME) is an outcomes-based training model that has become the new standard of medical education internationally. Regulatory and accrediting bodies have responded by establishing key competencies that residency programs may use to reshape their curricula. Available literature on the implementation of CBME in ophthalmology residency training programs across North America was examined. Ophthalmology-specific residency training objectives and milestones have been proposed within North America in the last 4 years. Curriculum guidelines and implementation blueprints and principles have also been proffered from various sources internationally but have met with some barriers. Assessment tools within ophthalmology have seen the most innovative development within the domain of surgical skills objectives. Competencies outside of the medical expert and patient care roles have proven more challenging to teach and assess. One ophthalmology program in Canada has undergone early implementation of a novel CBME curriculum. There is still considerable work to be done to successfully implement CBME curricula within ophthalmology residency programs in North America. Collaborative efforts to develop customizable curricula, tackle implementation barriers, and create specific assessment modalities will assist programs in meeting the competency mandates of CBME.

How clinical teaching teams deal with educational change: 'we just do it'

BACKGROUND

In postgraduate medical education, program directors are in the lead of educational change within clinical teaching teams. As change is part of a social process, it is important to not only focus on the program director but take their other team members into account. The purpose of this study is to provide an in-depth insight into how clinical teaching teams manage and organize curriculum change processes, and implement curriculum change in daily practice.

METHODS

An explorative qualitative semi-structured interview study was conducted between October 2016 and March 2017. A total of six clinical teaching teams (n = 6) participated in this study, i.e. one program director, one clinical staff member, and one trainee from each clinical teaching team (n = 18). Data were analysed and structured by means of thematic analysis.

RESULTS

The analysis yielded to five factors that positively impact change: shared commitment, reinvention, ownership, supportive structure and open culture. Factors that negatively impact change were: resistance, behaviour change, balance between different tasks, lack of involvement, lack of consensus, and unsafe culture and hierarchy. Overall, no clear change strategy could be recognized.

CONCLUSIONS

Insight was gathered in factors facilitating and hindering the implementation of change. It seems particularly important for clinical teaching teams to be able to create a sense of ownership among all team members by making a proposed change valuable for their local context as well as to be capable of working together as a team. Cultural factors seem to be particularly relevant in a team's ability to accomplish this.

Evaluation of inter-professional communication and leadership skills among graduating Canadian urology residents

INTRODUCTION

The importance of developing inter-professional communication and leadership skills among residents is well-recognized; however, formal tools to assess these skills are lacking. The goal of our study was to assess the leader and communicator roles in graduating urology residents using a validated self-assessment form developed for business students that focuses on inter-professional skills.

METHODS

Chief residents (n=36) were evaluated with surveys of communication and leadership skills. The same surveys were administered through email to the residents' program directors (PDs). Resident self-assessment and PD assessment were compared using paired and non-paired t-tests.

RESULTS

Graduating urology residents' self-assessment of their communication and leadership skills did not differ from assessments made by their PDs (77.6 vs. 74.4%; p=0.19); however, there were outlier candidates in whom PD assessment differed substantially from self-assessment on both surveys. Graduating urology residents scored themselves higher on self-awareness (82.6 vs. 77%; p= 0.05) and lower on stress management (67.7 vs. 77%; p= 0.01) compared to their PDs. Resident self-assessment scores were similar to business students on both communication and leadership surveys. Limitations were the small sample size and lack of survey evaluation by those surveyed.

CONCLUSIONS

Graduating urology residents' self-assessment of their own communication and leadership skills did not differ greatly from assessment by their PDs or a sample of business students. Comparison of self-assessment evaluations and evaluations by PDs allowed us to identify outliers in whom self-assessment and PD-assessment markedly differed, which may allow for more focused and meaningful feedback.

Teaching and assessment strategies for nursing self-care competencies in Ontario's nursing education programs

In Canada, nurses comprise half the healthcare employment sector, yet comparably, they are twice as likely to experience absenteeism due to occupational burnout. This issue is pronounced for entry-level nurses. The College of Nurses of Ontario (CNO) mandates professional practice standards and entry-level practice competencies; notably, these documents lack explicit performance expectations linked to nursing self-care. This lack of acknowledgment is reflected in what is known, which is little, about how self-care competencies are taught and assessed in nursing education programs. From a Program Director perspective, this study examined the strategies used to teach and assess self-care competencies in Ontario's nursing education programs. Survey results (n = 8), reported intention-action gaps highest (i.e., there is a need for increased teaching), in self-care competencies not mandated by the CNO (i.e., relationship, emotional, and spiritual self-care), whereas competencies mandated by the CNO (i.e., professionalism), were rated higher in relative teaching than importance. Given that self-care strategies (e.g. maintaining healthy interpersonal relationships and engaging in spiritual growth and mindfulness) have shown to be protective factors against workload stress, burnout, and job attrition, regulatory colleges need to consider mandating these self-care competencies within their professional practice standards and entry-level registered nurse practice guidelines.

Navigating complexity in team-based clinical settings

CONTEXT

Educators must prepare learners to navigate the complexities of clinical care. Training programmes have, however, traditionally prioritised teaching around the biomedical and the technical, not the socio-relational or systems issues that create complexity. If we are to transform medical education to meet the demands of 21st century practice, we need to understand how clinicians perceive and respond to complex situations.

METHODS

Constructivist grounded theory informed data collection and analysis; during semi-structured interviews, we used rich pictures to elicit team members' perspectives about clinical complexity in neurology and in the intensive care unit. We identified themes through constant comparative analysis.

RESULTS

Routine care became complex when the prognosis was unknown, when treatment was either non-existent or had been exhausted or when being patient and family centred challenged a system's capabilities, or participants' training or professional scope of practice. When faced with complexity, participants reported that care shifted from relying on medical expertise to engaging in advocacy. Some physician participants, however, either did not recognise their care as advocacy or perceived it as outside their scope of practice. In turn, advocacy was often delegated to others.

CONCLUSIONS

Our research illuminates how expert clinicians manoeuvre moments of complexity; specifically, navigating complexity may rely on mastering health advocacy. Our results suggest that advocacy is often negotiated or collectively enacted in team settings, often with input from patients and families. In order to prepare learners to navigate complexity, we suggest that programmes situate advocacy training in complex clinical encounters, encourage reflection and engage non-physician team members in advocacy training.

Passion projects and disorienting dilemmas

None

Evaluation of virtual patient cases for teaching diagnostic and management skills in internal medicine: a mixed methods study

Objective

The virtual patient (VP) is a computer program that simulates real-life clinical scenarios and allows learners to make diagnostic and therapeutic decisions in a safe environment. Although many VP cases are available, few focus on junior trainees as their target audience. In addition, there is wide variability in trainees’ clinical rotation experiences, based on local practice and referral patterns, duty hour restrictions, and competing educational requirements. In order to standardize clinical exposure and improve trainees’ knowledge and perceived preparedness to manage core internal medicine cases, we developed a pool of VP cases to simulate common internal medicine presentations. We used quantitative and qualitative analyses to evaluate the effectiveness of one of our VP cases among medical trainees at University of Toronto. We also evaluated the role of VP cases in integrated teaching of non-medical expert competencies.

Results

Despite modest effects on knowledge acquisition, a majority of participants enjoyed using VP cases as a resource to help them prepare for and reinforce clinical experiences. Cognitive interactivity and repetitive practice were particularly appreciated by study participants. Trainees perceived VP cases as a useful resource as their learning can be customized to their actions within the case, resulting in unique learning trajectories.

Electronic supplementary material

The online version of this article (10.1186/s13104-018-3463-x) contains supplementary material, which is available to authorized users.

How are competency frameworks perceived and taught? : An exploratory study in the context of pharmacy education

INTRODUCTION

Faculties of Pharmacy worldwide have to adapt their curriculum to accreditation criteria. The present study explored how the Association of Faculties of Pharmacy of Canada's (AFPC's) 2010 Educational Outcomes are perceived and taught at the Leslie Dan Faculty of Pharmacy (LDFP). These outcomes were adapted from the CanMeds Physician Competency Framework which describes both medical expert and non-expert roles.

METHODS

We wondered if pharmacy would struggle, as medicine has, to integrate these roles into curricula in meaningful ways, given the absence of previous studies from Pharmacy. We conducted an exploratory interview study with 10 core faculty members in charge of courses where non-expert roles were taught. Data were analysed using conventional content analysis.

RESULTS

Faculty members understood that the AFPC Outcomes are important for students, patients, and the profession of pharmacy, and some saw the roles as knowledge-based and teachable using content from academic disciplines. However, most saw them as skills taught informally or through clinical experience. They used the roles as a framing device to legitimize their course content and relied on informal role modelling to do most of the teaching. The few faculty members who taught content related to these roles had postgraduate education in a social science or humanities discipline.

DISCUSSION

Similar to studies of Faculties of Medicine, our study highlights the difficulty of translating a role-based, competency framework into concrete, integrated curricula for students. Competency development should be explicitly embedded into the curriculum and cannot be left to individual instructors.

Integrated and implicit: how residents learn CanMEDS roles by participating in practice

CONTEXT

Learning outcomes for residency training are defined in competency frameworks such as the CanMEDS framework, which ultimately aim to better prepare residents for their future tasks. Although residents' training relies heavily on learning through participation in the workplace under the supervision of a specialist, it remains unclear how the CanMEDS framework informs practice-based learning and daily interactions between residents and supervisors.

OBJECTIVES

This study aimed to explore how the CanMEDS framework informs residents' practice-based training and interactions with supervisors.

METHODS

Constructivist grounded theory guided iterative data collection and analyses. Data were collected by direct observations of residents and supervisors, combined with formal and field interviews. We progressively arrived at an explanatory theory by coding and interpreting the data, building provisional theories and through continuous conversations. Data analysis drew on sensitising insights from communities of practice theory, which provided this study with a social learning perspective.

RESULTS

CanMEDS roles occurred in an integrated fashion and usually remained implicit during interactions. The language of CanMEDS was not adopted in clinical practice, which seemed to impede explicit learning interactions. The CanMEDS framework seemed only one of many factors of influence in practice-based training: patient records and other documents were highly influential in daily activities and did not always correspond with CanMEDS roles. Additionally, the position of residents seemed too peripheral to allow them to learn certain aspects of the Health Advocate and Leader roles.

CONCLUSIONS

The CanMEDS framework did not really guide supervisors' and residents' practice or interactions. It was not explicitly used as a common language in which to talk about resident performance and roles. Therefore, the extent to which CanMEDS actually helps improve residents' learning trajectories and conversations between residents and supervisors about residents' progress remains questionable. This study highlights the fact that the reification of competency frameworks into the complexity of practice-based learning is not a straightforward exercise.

Using In-Training Evaluation Report (ITER) Qualitative Comments to Assess Medical Students and Residents: A Systematic Review

PURPOSE

In-training evaluation reports (ITERs) constitute an integral component of medical student and postgraduate physician trainee (resident) assessment. ITER narrative comments have received less attention than the numeric scores. The authors sought both to determine what validity evidence informs the use of narrative comments from ITERs for assessing medical students and residents and to identify evidence gaps.

METHOD

Reviewers searched for relevant English-language studies in MEDLINE, EMBASE, Scopus, and ERIC (last search June 5, 2015), and in reference lists and author files. They included all original studies that evaluated ITERs for qualitative assessment of medical students and residents. Working in duplicate, they selected articles for inclusion, evaluated quality, and abstracted information on validity evidence using Kane's framework (inferences of scoring, generalization, extrapolation, and implications).

RESULTS

Of 777 potential articles, 22 met inclusion criteria. The scoring inference is supported by studies showing that rich narratives are possible, that changing the prompt can stimulate more robust narratives, and that comments vary by context. Generalization is supported by studies showing that narratives reach thematic saturation and that analysts make consistent judgments. Extrapolation is supported by favorable relationships between ITER narratives and numeric scores from ITERs and non-ITER performance measures, and by studies confirming that narratives reflect constructs deemed important in clinical work. Evidence supporting implications is scant.

CONCLUSIONS

The use of ITER narratives for trainee assessment is generally supported, except that evidence is lacking for implications and decisions. Future research should seek to confirm implicit assumptions and evaluate the impact of decisions.

Equipping future doctors: incorporating management and leadership into medical curriculums in the United Kingdom

Throughout their careers, doctors are likely to come across complex management and leadership scenarios that many would not have had prior training in. Expectations of doctors are rising and it is becoming increasingly necessary to be able to astutely handle a variety of situations. Medical curricula must reflect this change and adapt to include the teaching of key management and leadership skills. Despite budgeting pressures, the National Health Service continues to spend vast sums of money on external management consultants. The 2013 Francis Report stressed the need for better management skills and leadership, especially in doctors who were identified as the spearheads of change. This view is backed up by senior professionals who stress that by incorporating it into undergraduate curricula, doctors will be equipped with the skills to flourish in the future. The challenges of doing so must be highlighted, since the teaching of managerial and leadership concepts must effectively combine theoretical approaches with practical applications. Empowering students of today will enable them as tomorrow's doctors to tackle the challenges of modern medicine.

Health advocacy. MEDICAL TEACHER 2017;39:128-135. [PMID: 27866451 DOI: 10.1080/0142159x.2017.1245853]

In the medical profession, activities related to ensuring access to care, navigating the system, mobilizing resources, addressing health inequities, influencing health policy and creating system change are known as health advocacy. Foundational concepts in health advocacy include social determinants of health and health inequities. The social determinants of health (i.e. the conditions in which people live and work) account for a significant proportion of an individual's and a population's health outcomes. Health inequities are disparities in health between populations, perpetuated by economic, social, and political forces. Although it is clear that efforts to improve the health of an individual or population must consider "upstream" factors, how this is operationalized in medicine and medical education is controversial. There is a lack of clarity around how health advocacy is delineated, how physicians' scope of responsibility is defined and how teaching and assessment is conceptualized and enacted. Numerous curricular interventions have been described in the literature; however, regardless of the success of isolated interventions, understanding health advocacy instruction, assessment and evaluation will require a broader examination of processes, practices and values throughout medicine and medical education. To support the instruction, assessment and evaluation of health advocacy, a novel framework for health advocacy is introduced. This framework was developed for several purposes: defining and delineating different types and approaches to advocacy, generating a "roadmap" of possible advocacy activities, establishing shared language and meaning to support communication and collaboration across disciplines and providing a tool for the assessment of learners and for the evaluation of teaching and programs. Current approaches to teaching and assessment of health advocacy are outlined, as well as suggestions for future directions and considerations.

Epistemology, culture, justice and power: non-bioscientific knowledge for medical training

CONTEXT

While medical curricula were traditionally almost entirely comprised of bioscientific knowledge, widely accepted competency frameworks now make clear that physicians must be competent in far more than biomedical knowledge and technical skills. For example, of the influential CanMEDS roles, six are conceptually based in the social sciences and humanities (SSH). Educators frequently express uncertainty about what to teach in this area. This study concretely identifies the knowledge beyond bioscience needed to support the training of physicians competent in the six non-Medical Expert CanMEDS roles.

METHODS

We interviewed 58 non-clinician university faculty members with doctorates in over 20 SSH disciplines. We abstracted our transcripts (meaning condensation, direct quotations) resulting in approximately 300 pages of data which we coded using top-down (by CanMEDS role) and bottom-up (thematically) approaches and analysed within a critical constructivist framework. Participants and clinicians with SSH PhDs member-checked and refined our results.

RESULTS

Twelve interrelated themes were evident in the data. An understanding of epistemology, including the constructed nature of social knowledge, was seen as the foundational theme without which the others could not be taught or understood. Our findings highlighted three anchoring themes (Justice, Power, Culture), all of which link to eight more specific themes concerning future physicians' relationships to the world and the self. All 12 themes were cross-cutting, in that each related to all six non-Medical Expert CanMEDS roles. The data also provided many concrete examples of potential curricular content.

CONCLUSIONS

There is a definable body of SSH knowledge that forms the academic underpinning for important physician competencies and is outside the experience of most medical educators. Curricular change incorporating such content is necessary if we are to strengthen the non-Medical Expert physician competencies. Our findings, particularly our cross-cutting themes, also provide a pedagogically useful mechanism for holistically teaching the underpinnings of physician competence. We are now implementing our findings into medical curricula.

Measuring interprofessional competencies and attitudes among health professional students creating family planning virtual patient cases

BACKGROUND

The Virtual Interprofessional Patients-Computer-Assisted Reproductive Health Education for Students (VIP-CARES) Project took place during the summers of 2010-2012 for eight weeks each year at the University of British Columbia (UBC). Undergraduate health care students worked collaboratively to develop virtual patient case-based learning modules on the topic of family planning. The purpose of this study was to evaluate the changes in perception towards interprofessional collaboration (IPC) among the participants, before and after the project.

METHODS

This study utilized a mixed methods evaluation using self-assessment survey instruments, semi-structured interviews, and reflective essays. Pre- and post- project surveys were adapted from the Canadian Medical Education Determinants (CanMEDS) and Canadian Interprofessional Health Collaborative (CIHC) frameworks, as well as the Memorial University Interprofessional Attitudes (IPA) questionnaire. The survey results were analyzed as mean (M) and standard deviation (SD) on Likert scales. The non-parametric Wilcoxon signed-rank test was used to determine if any significant changes were measured between each participant's differences in score (p ≤ 0.05). Post-project interview transcripts and essays were analyzed using recursive abstraction to elicit any themes.

RESULTS

Altogether, 26 students in medicine, pharmacy, nursing, midwifery, dentistry, counselling psychology, and computer science participated in VIP-CARES, during the three years. Student attitudes toward IPC were positive before and after the project. At the project's conclusion, there was a statistically significant increase in the participants' self-assessment competency scores in the CanMEDS roles of health advocate (p = 0.05), manager (p = 0.02), and medical expert (p = 0.03), as well as the CIHC domains of interprofessional communication (p = 0.04), role clarification (p = 0.01), team functioning (p = 0.05), and collaborative leadership (p = 0.01). Qualitative evaluations yielded three major themes: communication and respect as key to team functioning, importance of role clarification within the team, and existence of inherent challenges to IPC. From the reflections, students generally felt more comfortable with their improvements in the CIHC domains of interprofessional communication, team functioning, and role clarification.

CONCLUSION

After working within an interdisciplinary team developing virtual patient learning modules on family planning, the student participants of the VIP-CARES Project indicated general improvement in the skills necessary for effective interprofessional collaboration. Triangulation of the overall data suggests this was especially observed within the areas of interprofessional communication, team functioning, and role clarification.

Content validity of workplace-based portfolios: A multi-centre study

BACKGROUND

Portfolios are used as tools to coach and assess students in the workplace. This study sought to evaluate the content validity of portfolios as reflected in their capacity to adequately assess achieved competences of medical students during clerkships.

METHODS

We reviewed 120 workplace portfolios at three medical universities (Belgium and the Netherlands). To validate their content, we developed a Validity Inventory for Portfolio Assessment (VIPA) based on the CanMEDS roles. Two raters evaluated each portfolio and indicated for each VIPA item whether the portfolio provided sufficient information to enable satisfactory assessment of the item. We ran a descriptive analysis on the validation data and computed Cohen's Kappa to investigate interrater agreement.

RESULTS

The portfolios adequately covered the items pertaining to the communicator (90%) and professional (87%) roles. Coverage of the medical expert, collaborator, scholar and manager roles ranged between 75% and 85%. The health advocate role, covering 59%, was clearly less well represented. This role also exhibited little interrater agreement (Kappa < 0.4).

CONCLUSIONS

This study lends further credence to the evidence that portfolios can indeed adequately assess the different CanMEDS roles during clerkships, the health advocate role, which was less well represented in the portfolio content, excepted.

A feedback system in residency to evaluate CanMEDS roles and provide high-quality feedback: Exploring its application

INTRODUCTION

Residents benefit from regular, high quality feedback on all CanMEDS roles during their training. However, feedback mostly concerns Medical Expert, leaving the other roles behind. A feedback system was developed to guide supervisors in providing feedback on CanMEDS roles. We analyzed whether feedback was provided on the intended roles and explored differences in quality of written feedback.

METHODS

In the feedback system, CanMEDS roles were assigned to five authentic situations: Patient Encounter, Morning Report, On-call, CAT, and Oral Presentation. Quality of feedback was operationalized as specificity and inclusion of strengths and improvement points. Differences in specificity between roles were tested with Mann-Whitney U tests with a Bonferroni correction (α = 0.003).

RESULTS

Supervisors (n = 126) provided residents (n = 120) with feedback (591 times). Feedback was provided on the intended roles, most frequently on Scholar (78%) and Communicator (71%); least on Manager (47%), and Collaborator (56%). Strengths (78%) were mentioned more frequently than improvement points (52%), which were lacking in 40% of the feedback on Manager, Professional, and Collaborator. Feedback on Scholar was more frequently (p = 0.000) and on Reflective Professional was less frequently (p = 0.003) specific.

DISCUSSION AND CONCLUSION

Assigning roles to authentic situations guides supervisors in providing feedback on different CanMEDS roles. We recommend additional supervisor training on how to observe and evaluate the roles.

Evaluating the reliability of surgical assessment methods in an orthopedic residency program

BACKGROUND

Orthopedic surgical education in Canada has seen major change in the last 15 years. Work hour restrictions and external influence have led to new approaches for surgical training. With a change toward competency-based educational models under the CanMEDS headings there is a need to ensure the validity of modern assessment methods. Our objective was to evaluate the reliability of a currently used surgical skill assessment tool within an orthopedic residency program, as measured by the Surgical Encounters Form.

METHODS

A surgical assessment tool has previously been created at our institution that comprises 15 items spanning 4 of the CanMEDS competencies. Results were blinded to the primary investigator and coded by a third party. The assessments were collected, and we measured percent agreement using Cronbach's α and Fleiss κ.

RESULTS

Over a 5-month period 11 staff members assessed 10 residents. Eighty-eight assessments were completed in total. Weighted percent agreement was 90.9%. Cronbach's α averaged 0.865 for the medical expert role, 0.920 for technical skills, 0.934 for the communicator role, 1.00 for the collaborator role and 1.00 for the health advocate role. The mean Fleiss κ score was 0.147 (95% confidence interval -0.071 to 0.364), demonstrating low interrater reliability.

CONCLUSION

Despite the development of a validated assessment tool to evaluate surgical skills acquisition, interrater reliability results suggest low levels of agreement among assessors.

When patients and surgeons disagree about surgical outcome: investigating patient factors and chart note communication

OBJECTIVE

Effective physician-patient communication is a critical component of a clinical practice and in order to achieve optimal patient outcomes. We aimed to investigate indirect effects of physician-patient communication by examining the relationship between a physician-patient mismatch in perceived outcomes and content in the medical record's clinical note. We compared patient records whose perceived subjective assessment of surgery outcomes agreed or disagreed with the surgeon's perception of that outcome (Subjective Disagreement).

METHODS

This study included 172 spine surgery patients at a teaching hospital. Patient-reported outcomes included the Oswestry Disability Index; the Short-Form 36; and a Visual Analogue Scale items for leg and back pain. We content-analyzed the clinical note in the medical record, and used logistic regression to evaluate predictors of Subjective Disagreement (n = 41 disagreed vs. 131 agreed).

RESULTS

Patient and surgeon agreed in 76% of cases and disagreed in 24% of cases. Patients who assessed their outcome worse than their surgeons tended to be less educated and involved in litigation. They also tended to report worsened mental health and leg pain. Content analysis revealed group differences in surgeon communication patterns in the chart notes related to how symptom change was emphasized, how follow-up was described, and a specific word reference. Specifically, disagreement was predicted by using "much" to emphasize the findings and noting long-term prognosis. Agreement was predicted by use of positive emphasis terms, having an "as-needed" follow-up plan, and using "happy" in the chart note.

CONCLUSION

The nature of measuring outcomes of surgery is based on patient perception. In surgeon-patient perspective mismatches, patient factors may serve as barriers to improvement. Worsened change on patient-reported mental health may be an independent factor which colors the patient's general perceptions. This aspect of treatment may be missed by the spine surgeon. Chart note communication styles reflect the subjective disagreement. Investigating and/ or treating mental health deterioration may be valuable in resolving this mismatch and for overall outcome.

Specialty Training's Organizational Readiness for curriculum Change (STORC): development of a questionnaire in a Delphi study

BACKGROUND

In postgraduate medical education (PGME), programs have been restructured according to competency-based frameworks. The scale and implications of these adjustments justify a comprehensive implementation plan. Organizational Readiness for Change (ORC) is seen as a critical precursor for a successful implementation of change initiatives. Though, ORC in health care settings is mostly assessed in small scale settings and in relation to new policies and practices rather than educational change. Therefore our aim with this work was to develop an instrument to asses Specialty Training's Organizational Readiness for curriculum Change (STORC).

METHODS

A Delphi procedure was conducted to examine the applicability of a preliminary questionnaire in PGME, which was based on existing instruments designed for business and health care organizations. The 41 panellists (19 trainees and 22 supervisors from 6 specialties) from four different countries who were confronted with an apparent curriculum change, or would be in the near future, were asked to rate the relevance of a 89-item web-based questionnaire with regard to changes in specialty training on a 5-point Likert scale. Furthermore, they were invited to make qualitative comments on the items.

RESULTS

In two rounds the 89-item preliminary questionnaire was reduced to 44 items. Items were either removed, kept, adapted or added based on individual item scores and qualitative comments. In the absence of a gold standard, this Delphi procedure was considered complete when the overall questionnaire rating exceeded 4.0 (scale 0-5). The overall item score reached 4.1 in the second round, meeting our criteria for completion of this Delphi procedure.

CONCLUSIONS

This Delphi study describes the initial validating step in the development of an instrument to asses Specialty Training's Organisational Readiness for curriculum Change (STORC). Since ORC is measured on various subscales and presented as such, its strength lies in analysing these subscales. The latter makes it possible for educational leaders to identify and anticipate on hurdles in the implementation process and subsequently optimize efforts for successful curriculum change.

Not just 'for' but 'with': health advocacy as a partnership process

CONTEXT

Health advocacy is often framed as an activity that physicians do for others. A physician uses her expertise to identify and address the health needs of patients or communities on their behalves. As part of a larger study, we uncovered data to suggest that effective health advocates work not just for but often with others to understand and address their health needs.

OBJECTIVES

This paper explores and elaborates the important distinction between advocating with and for others.

METHODS

We interviewed 10 physicians, identified by others as successful health advocates, about their advocacy activities. Informed by constructivist grounded theory, we gathered and evaluated data iteratively, continually revising the interview outline and concurrently refining our evolving themes. Once it had stabilised, the coding scheme was applied to the full set of transcripts.

RESULTS

Health advocacy was framed by participants as an activity that was more often done with others, than for others. This manifested in two ways: (i) joining other voices: rather than always feeling a need to plan and act alone, our participants often described making efforts to find and join existing initiatives and to work collaboratively, and (ii) amplifying other voices: rather than authoritatively determining needs and enacting solutions on behalf of others, our participants often described making efforts to empower others to find their own voices, thereby fostering autonomy rather than reliance. Participants described factors and mechanisms that enabled them to approach advocacy in this manner.

CONCLUSIONS

Successful health advocates often enact health advocacy with others, rather than exclusively for them. This partnership-based facilitative approach enables them to better appreciate the needs of those requiring support, and to ask: 'How can I help?' If this approach were more effectively reflected in formal constructions of the process, health advocacy might not only be practised more effectively, but might also be perceived as more achievable by trainees and physicians.

Designing faculty development to support the evaluation of resident competency in the intrinsic CanMEDS roles: practical outcomes of an assessment of program director needs

BACKGROUND

The Royal College of Physicians and Surgeons of Canada and the College of Family Physicians of Canada mandate that faculty members demonstrate they are evaluating residents on all CanMEDS (Canadian Medical Education Directions for Specialists) roles as part of the accreditation process. Postgraduate Medical Education at the University of Ottawa initiated a 5-year project to develop and implement a comprehensive system to assess the full spectrum of CanMEDS roles. This paper presents the findings from a needs assessment with Program Directors, in order to determine how postgraduate medical faculty can be motivated and supported to evaluate residents on the intrinsic CanMEDS roles.

METHODS

Semi-structured individual interviews were conducted with 60 Postgraduate Program Directors in the Faculty of Medicine. Transcribed interviews were analyzed using qualitative analysis. Once the researchers were satisfied the identified themes reflected the views of the participants, the data was assigned to categories to provide rich, detailed, and comprehensive information that would indicate what faculty need in order to effectively evaluate their residents on the intrinsic roles.

RESULTS

Findings indicated faculty members need faculty development and shared point of care resources to support them with how to not only evaluate, but also teach, the intrinsic roles. Program Directors expressed the need to collaborate and share resources across departments and national specialty programs. Based on our findings, we designed and delivered workshops with companion eBooks to teach and evaluate residents at the point of care (Developing the Professional, Health Advocate and Scholar).

CONCLUSIONS

Identifying stakeholder needs is essential for designing effective faculty development. By sharing resources, faculties can prevent 'reinventing the wheel' and collaborate to meet the Colleges' accreditation requirements more efficiently.

Frontline over ivory tower: key competencies in community-based curricula

BACKGROUND

The Royal College of Physicians and Surgeons of Canada mandates that community experiences be incorporated into medicine-based specialties. Presently there is wide variability in community endocrine experiences across Canadian training programs. This is complicated by the paucity of literature providing guidance on what constitutes a 'community' rotation.

METHOD

A modified Delphi technique was used to determine the CanMEDS competencies best taught in a community endocrinology curriculum. The Delphi technique is a qualitative-research method that uses a series of questionnaires sent to a group of experts with controlled feedback provided by the researchers after each survey round. The experts in this study included endocrinology program directors, community endocrinologists, endocrinology residents and recent endocrinology graduates.

RESULTS

Thirty four out of 44 competencies rated by the panel were deemed suitable for a community curriculum. The experts considered the "Manager" role best taught in the community, while they considered the community least suitable to learn the "Medical Expert" competency.

CONCLUSIONS

To our knowledge, this is the first time the content of a community-based subspecialty curriculum was determined using the Delphi process in Canada. These findings suggest that community settings have potential to fill in gaps in residency training in regards to the CanMEDS Manager role. The results will aid program directors in designing competency-based community endocrinology rotations and competency-based community rotations in other medical subspecialty programs.