Entrustment Decision Making: Extending Miller's Pyramid

A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees

INTRODUCTION

The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why?

METHODS

Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs.

RESULTS

PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes.

DISCUSSION

PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.

A modified Delphi approach to nurturing professionalism in postgraduate medical education in Singapore

INTRODUCTION

Nurturing professional identities instils behavioural standards of physicians, and this in turn facilitates consistent professional attitudes, practice and patient care. Identities are socioculturally constructed efforts; therefore, we must account for the social, cultural and local healthcare factors that shape physicians' roles, responsibilities and expectations. This study aimed to forward a programme to nurture professionalism among physicians in Singapore.

METHODS

A three-phase, evidenced-based approach was used. First, a systematic scoping review (SSR) was conducted to identify professionalism elements. Second, a questionnaire was created based on the findings of the SSR. Third, a modified Delphi approach, which involved local experts to identify socioculturally appropriate elements to nurture professionalism, was used.

RESULTS

A total of 124 articles were identified from the SSR; these articles revealed definitions, knowledge, skills and approaches to nurturing professionalism. Through the modified Delphi approach, we identified professional traits, virtues, communication, ethical, self-care, teaching and assessment methods, and support mechanisms.

CONCLUSION

The results of this study formed the basis for a holistic and longitudinal programme focused on instilling professional traits and competencies over time through personalised and holistic support of physicians. The findings will be of interest to medical communities in the region and beyond.

OWIED System to Facilitate the Entrustment Decision in EPA Implementation

BACKGROUND

Entrustable professional activities (EPAs) are gaining attention as an approach for adequate implementation of a competency-based evaluation framework in nursing education. This study sought to develop an Online Web-assessment Interface for Entrustment Decision (OWIED) to facilitate the entrustment decision in EPA implementation.

METHOD

A participatory qualitative action research design consisting of two phases was used for this study. The exploration phase was conducted in close collaboration with stakeholders. Following the exploration phase, the primary researcher and a team of subject-matter experts in academic and information technology developed the OWIED system according to the exploration phase results.

RESULTS

The necessary features that met the expectations of the stakeholders were identified and assisted in developing the OWIED system.

CONCLUSION

OWIED may provide a valid tool to track and validate nursing student acquisition of core competencies and assist in making entrustment decisions as students fulfill their training requirements in their academic program. [J Nurs Educ. 2024;63(6):356-365.].

Assessment of Entrustable Professional Activities Among Dutch Endocrine Supervisors

Entrustable Professional Activities (EPAs) are an important tool to support individualisation of medical training in a competency-based setting and are increasingly implemented in the clinical speciality training for endocrinologist. This study aims to assess interrater agreement and factors that potentially impact EPA scores. Five known factors that affect entrustment decisions in health profesions training (capability, integrity, reliability, humility, agency) were used in this study. A case-vignette study using standardised written cases. Case vignettes (n = 6) on the topics thyroid disease, pituitary disease, adrenal disease, calcium and bone disorders, diabetes mellitus, and gonadal disorders were written by two endocrinologists and a medical education expert and assessed by endocrinologists experienced in the supervision of residents in training. Primary outcome is the inter-rater agreement of entrustment decisions for endocrine EPAs among raters. Secondary outcomes included the dichotomous interrater agreement (entrusted vs. non-entrusted), and an exploration of factors that impact decision-making. The study protocol was registered and approved by the Ethical Review Board of the Netherlands Association for Medical Education (NVMO-ERB # 2020.2.5). Nine endocrinologists from six different academic regions participated. Overall, the Fleiss Kappa measure of agreement for the EPA level was 0.11 (95% CI: 0.03-0.22) and for the entrustment decision 0.24 (95% CI 0.11-0.37). Of the five features that impacted the entrustment decision, capability was ranked as the most important by a majority of raters (56%-67%) in every case. There is a considerable discrepancy between the EPA levels assigned by different raters. These findings emphasise the need to base entrustment decisions on multiple observations, made by a team of supervisors and enriched with factors other than direct medical competence.

Domains Influencing Faculty Decisions on the Level of Supervision Required for Anesthesiology EPAs with Analysis of Feedback Comments

OBJECTIVE

The purpose of this qualitative study was to examine responses related to entrustment and feedback comments from an assessment tool.

DESIGN

Qualitative analyses using semi-structured interviews and analysis of narrative comments.

SETTING

Main hospital OR suite at a large academic medical center.

PARTICIPANTS

faculty, and residents who work in the OR suite.

RESULTS

Seven of the 14 theoretical domains from the Theoretical Domains Framework were identified as influencing faculty decision on entrustment: knowledge, skills, intention, memory/attention/decision processes, environmental context, and resources, beliefs of capabilities, and reinforcement. The majority (651/1116 (58.4%)) of faculty comments were critical/modest praise and relevant, consistent across all 6 EPAs. The written in feedback comments for all 1,116 Web App EPA assessments yielded a total of 1,599 sub-competency specific responses. These responses were mapped to core competencies, and at least once to 13 of the 23 ACGME subcompetencies.

CONCLUSIONS

Domains identified as influencing faculty decision on entrustment were knowledge, skills, intention, memory/attention/decision processes, environmental context, and resources, beliefs of capabilities, and reinforcement. Most narrative feedback comments were critical/modest praise and relevant, consistent across each of the EPAs.

Assessing clinical competence: a multitrait-multimethod matrix construct validity study

Education in Doctor of Medicine programs has moved towards an emphasis on clinical competency, with entrustable professional activities providing a framework of learning objectives and outcomes to be assessed within the clinical environment. While the identification and structured definition of objectives and outcomes have evolved, many methods employed to assess clerkship students' clinical skills remain relatively unchanged. There is a paucity of medical education research applying advanced statistical design and analytic techniques to investigate the validity of clinical skills assessment. One robust statistical method, multitrait-multimethod matrix analysis, can be applied to investigate construct validity across multiple assessment instruments and settings. Four traits were operationalized to represent the construct of critical clinical skills (professionalism, data gathering, data synthesis, and data delivery). The traits were assessed using three methods (direct observations by faculty coaches, clinical workplace-based evaluations, and objective structured clinical examination type clinical practice examinations). The four traits and three methods were intercorrelated for the multitrait-multimethod matrix analysis. The results indicated reliability values in the adequate to good range across the three methods with the majority of the validity coefficients demonstrating statistical significance. The clearest evidence for convergent and divergent validity was with the professionalism trait. The correlations on the same method/different traits analyses indicated substantial method effect; particularly on clinical workplace-based assessments. The multitrait-multimethod matrix approach, currently underutilized in medical education, could be employed to explore validity evidence of complex constructs such as clinical skills. These results can inform faculty development programs to improve the reliability and validity of assessments within the clinical environment.

Exploring adaptive expertise in residency: the (missed) opportunity of uncertainty

Preparing novice physicians for an unknown clinical future in healthcare is challenging. This is especially true for emergency departments (EDs) where the framework of adaptive expertise has gained traction. When medical graduates start residency in the ED, they must be supported in becoming adaptive experts. However, little is known about how residents can be supported in developing this adaptive expertise. This was a cognitive ethnographic study conducted at two Danish EDs. The data comprised 80 h of observations of 27 residents treating 32 geriatric patients. The purpose of this cognitive ethnographic study was to describe contextual factors that mediate how residents engage in adaptive practices when treating geriatric patients in the ED. Results showed that all residents fluidly engaged in both adaptive and routine practices, but they were challenged when engaging in adaptive practices in the face of uncertainty. Uncertainty was often observed when residents' workflows were disrupted. Furthermore, results highlighted how residents construed professional identity and how this affected their ability to shift between routine and adaptive practices. Residents reported that they thought that they were expected to perform on par with their more experienced physician colleagues. This negatively impacted their ability to tolerate uncertainty and hindered the performance of adaptive practices. Thus, aligning clinical uncertainty with the premises of clinical work, is imperative for residents to develop adaptive expertise.

The identification of requirements for competency development during work-integrated learning in healthcare education

BACKGROUND

Work-integrated learning (WIL) is widely accepted and necessary to attain the essential competencies healthcare students need at their future workplaces. Yet, competency-based education (CBE) remains complex. There often is a focus on daily practice during WIL. Hereby, continuous competency development is at stake. Moreover, the fact that competencies need to continuously develop is often neglected.

OBJECTIVES

To ultimately contribute to the optimization of CBE in healthcare education, this study aimed at examining how competency development during WIL in healthcare education could be optimized, before and after graduation.

METHODS

Fourteen semi-structured interviews with 16 experts in competency development and WIL were carried out. Eight healthcare disciplines were included namely associate degree nursing, audiology, family medicine, nursing (bachelor), occupational therapy, podiatry, pediatrics, and speech therapy. Moreover, two independent experts outside the healthcare domain were included to broaden the perspectives on competency development. A qualitative research approach was used based on an inductive thematic analysis using Nvivo12© where 'in vivo' codes were clustered as sub-themes and themes.

RESULTS

The analysis revealed eight types of requirements for effective and continuous competency development, namely requirements in the context of (1) competency frameworks, (2) reflection and feedback, (3) assessment, (4) the continuity of competency development, (5) mentor involvement, (6) ePortfolios, (7) competency development visualizations, and (8) competency development after graduation. It was noteworthy that certain requirements were fulfilled in one educational program whereas they were absent in another. This emphasizes the large differences in how competence-based education is taking shape in different educational programs and internship contexts. Nevertheless, all educational programs seemed to recognize the importance of ongoing competency development.

CONCLUSION

The results of this study indicate that identifying and meeting the requirements for effective and continuous competency development is essential to optimize competency development during practice in healthcare education.

Leveraging faculty development to support validation of entrustable professional activities assessment tools in anatomic and clinical pathology training

Entrustable professional activities (EPAs) are observable activities that define the practice of medicine and provide a framework of evaluation that has been incorporated into US medical school curricula in both undergraduate and graduate medical education. This manuscript describes the development of an entrustment scale and formative and summative evaluations for pathology EPAs, outlines a process for faculty development that was employed in a pilot study implementing two Anatomic Pathology and two Clinical Pathology EPAs in volunteer pathology residency programs, and provides initial validation data for the proposed pathology entrustment scales. Prior to implementation, faculty development was necessary to train faculty on the entrustment scale for each given activity. A "train the trainer" model used performance dimension training and frame of reference training to train key faculty at each institution. The session utilized vignettes to practice determination of entrustment ratings and development of feedback for trainees as to strengths and weaknesses in the performance of these activities. Validity of the entrustment scale is discussed using the Messick framework, based on concepts of content, response process, and internal structure. This model of entrustment scales, formative and summative assessments, and faculty development can be utilized for any pathology EPA and provides a roadmap for programs to design and implement EPA assessments into pathology residency training.

Beyond the Procedure Log: Using Individualized Learning Plans to Set Learner-Specific Milestones for Procedural Skills Acquisition

ABSTRACT

Procedural training for nonsurgical fields, such as internal medicine, is an important component of medical education. However, recent changes to accreditation guidelines have resulted in less formal guidance on procedural competency, not only leading to opportunities for individualizing training but also creating potential problems for trainees and training programs. In this article, the authors use internal medicine as an exemplar to review current strategies for procedural education in nonsurgical fields, including procedural simulation, dedicated procedural rotations, and advanced subspecialty training, and highlight an emerging need for learner-specific terminal milestones in procedural training. Individualized learning plans (ILPs), collections of trainee-specific objectives for learning, are arguably a useful strategy for organizing procedural training. The role of ILPs as a framework to support setting learner-specific terminal milestones, guide skill acquisition, and allocate procedural learning opportunities based on trainees' anticipated career plans is subsequently explored, and how an ILP-based approach might be implemented within the complex educational milieu of a clinical training program is examined. The limitations and pitfalls of an ILP-based approach, including the need for development of coaching programs, are considered. The authors conclude that, despite the limitations of ILPs, when combined with other current strategies for building trainees' procedural competence, these plans may help trainees maximize the educational benefits of their training period and can encourage effective, safer, and equitable allocation of procedural practice opportunities.

Immunization Simulation Using Actual-Size Infant Manikins for Medical Students in Indonesia: A Quasi-Experimental Study

INTRODUCTION

Manikins are tools used in simulation training for medical students to develop important skills, one of which is immunization. However, conventional manikins often do not resemble the actual size of an infant's arm or leg. This study aimed to determine the impact of using actual-size manikins on students' knowledge and practical skills, evaluate their confidence in immunization practice, and assess students' responses and feedback regarding the practice using actual-size manikins.

METHODS

This was a quasi-experimental study involving medical students at the Faculty of Medicine, Universitas Indonesia, from October 2020 to April 2021. Students in the intervention group used newly developed actual-size infant arm and leg manikins, while the control group used conventional manikins. All students underwent the objective structured clinical examination (OSCE) and the scores were compared between the 2 groups. Within the intervention group, data on pretest and posttest scores, feedback questionnaires, and self-confidence assessments were also obtained and analyzed.

RESULTS

A total of 205 students were included. Statistically significant difference was found in the OSCE scores between the intervention and control groups (P < 0.01). Students in the intervention group (n = 108) showed significant improvement in knowledge scores after the workshop (P < 0.01). Most students (81.7%) expressed confidence in administering vaccines to live patients after practicing with manikins. In addition, 98.2% of students (n = 107) acknowledged the benefits of practicing with actual-size manikins in accurately determining the injection sites.

CONCLUSIONS

Simulation with the actual-size manikins significantly improved students' knowledge and practical immunization skills, leading to increased confidence and competence in their immunization skills.

The Next Era of Assessment: Building a Trustworthy Assessment System

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

Multi-stakeholder validation of entrustable professional activities for a family medicine care of the elderly residency program: A focus group study

Entrustable Professional Activities (EPAs) have become widely used within Competency-Based Medical Education (CBME) for the training and evaluation of residents. Little is known about the effectiveness of incorporating multiple stakeholder groups in the validation of EPAs. Here, we seek to validate an EPA framework developed for the University of Manitoba Care of the Elderly Enhanced Skills program using online focus groups consisting of five stakeholder groups. Participants were recruited to take part in one of five online focus groups, one for each stakeholder group (physician faculty, residents, non-physician healthcare professionals, administrators/managers, and patients). Each group met one time for 90 minutes over ZOOM®. The themes arising from stakeholder feedback suggest that successful EPAs must neither be too specific nor too expansive in scope, clearly delineate appropriate means of evaluation, and indicate specific clinical settings in which each EPA should be evaluated. Cross-cutting themes included requiring trainees to collaborate with other professionals when it would optimize patient care, and preparing trainees to advocate for their patients' health (Advocacy). The present study demonstrates that multi-stakeholder analysis yields diverse feedback that can help make EPAs more clear, easier to use in evaluation, and more socially accountable.

Exploring Factors for Implementation of EPAs in Pediatric Subspecialty Fellowships: A Qualitative Study of Program Directors

OBJECTIVE

To understand fellowship program directors' (FPDs) perspectives on facilitators and barriers to using entrustable professional activities (EPAs) in pediatric subspecialty training.

METHODS

We performed a qualitative study of FPDs, balancing subspecialty, program size, geographic region and current uses of EPAs. A study coordinator conducted 1-on-1 interviews using a semistructured approach to explore EPA use or nonuse and factors supporting or preventing their use. Investigators independently coded transcribed interviews using an inductive approach and the constant comparative method. Group discussion informed code structure development and refinement. Iterative data collection and analysis continued until theoretical sufficiency was achieved, yielding a thematic analysis.

RESULTS

Twenty-eight FPDs representing 11 pediatric subspecialties were interviewed, of whom 16 (57%) reported current EPA use. Five major themes emerged: (1) facilitators including the intuitive nature and simple wording of EPAs; (2) barriers such as workload burden and lack of a regulatory requirement; (2) variable knowledge and training surrounding EPAs, leading to differing levels of understanding; (3) limited current use of EPAs, even among self-reported users; and (4) complementary nature of EPAs and milestones. FPDs acknowledged the differing strengths of both EPAs and milestones but sought additional knowledge about the value added by EPAs for assessing trainees, including the impact on outcomes.

CONCLUSIONS

Identified themes can inform effective and meaningful EPA implementation strategies: Supporting and educating FPDs, ongoing assessment of the value of EPAs in training, and practical integration with current workflow. Generating additional data and engaging stakeholders is critical for successful implementation for the pediatric subspecialties.

Comparison of attitudes toward community-based medicine between regional-quota and general-selected medical student in Japan

Objective: This study compared the regional-quota and general-selected medical students' understanding, interest, and confidence in the community medicine practice and their attitudes toward the concept guidelines. Methods: We conducted a Web-based questionnaire survey regarding the understanding, interest, and confidence in future community medicine practice and attitudes toward concept guidelines among medical students of all grades (regional-quota and general-selected: n=82 and n=617, respectively). Results: The overall response rates were 68.5% (56/82) and 66.0% (409/617) in the regional-quota and general-selected groups, respectively. Although there was no significant difference between the groups in terms of understanding (P=0.998), interest and confidence in future practice were significantly higher in the regional-quota group (both P<0.001). There was no significant difference between the two groups for any of the six questions regarding community medicine guidelines. Conclusion: The understanding of community medicine or its conceptual guidelines did not significantly differ between the two groups; however, interest and confidence in future practice were significantly higher in the regional-quota group. These results suggest that the regional-quota system positively upregulates the interest in community medicine, which could be associated with confidence in future practice. Comprehensive and longitudinal improvements in the regional-quota system may be effective in cultivating community medicine.

Medical competence as a multilayered construct

BACKGROUND

The conceptualisation of medical competence is central to its use in competency-based medical education. Calls for 'fixed standards' with 'flexible pathways', recommended in recent reports, require competence to be well defined. Making competence explicit and measurable has, however, been difficult, in part due to a tension between the need for standardisation and the acknowledgment that medical professionals must also be valued as unique individuals. To address these conflicting demands, a multilayered conceptualisation of competence is proposed, with implications for the definition of standards and approaches to assessment.

THE MODEL

Three layers are elaborated. This first is a core layer of canonical knowledge and skill, 'that, which every professional should possess', independent of the context of practice. The second layer is context-dependent knowledge, skill, and attitude, visible through practice in health care. The third layer of personalised competence includes personal skills, interests, habits and convictions, integrated with one's personality. This layer, discussed with reference to Vygotsky's concept of Perezhivanie, cognitive load theory, self-determination theory and Maslow's 'self-actualisation', may be regarded as the art of medicine. We propose that fully matured professional competence requires all three layers, but that the assessment of each layer is different.

IMPLICATIONS

The assessment of canonical knowledge and skills (Layer 1) can be approached with classical psychometric conditions, that is, similar tests, circumstances and criteria for all. Context-dependent medical competence (Layer 2) must be assessed differently, because conditions of assessment across candidates cannot be standardised. Here, multiple sources of information must be merged and intersubjective expert agreement should ground decisions about progression and level of clinical autonomy of trainees. Competence as the art of medicine (Layer 3) cannot be standardised and should not be assessed with the purpose of permission to practice. The pursuit of personal excellence in this level, however, can be recognised and rewarded.

A Changing Landscape for Lifelong Learning in Health Globally

On 25 July 2022, the Continuing Professional Development (CPD) Special Interest Group of the Association for Medical Education in Europe came together to open up discussions during a live webinar on 'Exploring the Evolution of CPD'. The objective was to bring together global medical educators to consider perspectives of CPD from the role of global lifelong learners, the role of educators and the role of education providers and health regulators. The landscape of CPD is evolving, and the roles of each key player must include specific actions for facilitated change. Delivering competency outcomes-based learning, fit for purpose, to lifelong learners in health will require (1) learner agency, (2) leadership from educators and (3) providers of lifelong learning to come together to improve delivery of CPD that leads to meaningful change in practice care delivery.

Development of complex pedagogical competencies and reflexivity in clinical teachers via distance learning: a mixed methods study

Clinical reasoning is the cornerstone to healthcare practice and teaching it appropriately is of utmost importance. Yet there is little formal training for clinical supervisors in supervising this reasoning process. Distance education provides interesting opportunities for continuous professional development of healthcare professionals. This mixed methods study aimed at gaining in-depth understanding about whether and how clinical teachers can develop complex pedagogical competencies through participation in a Massive Open Online Course on the supervision of clinical reasoning (MOOC SCR). Participants self-assed their clinical supervision skills before and after partaking in the MOOC SCR through the Maastricht Clinical Teachers Questionnaire. Item scores and the distribution of response proportions before and after participation were compared using paired t-tests and McNemar's tests respectively. In parallel, the evolution of a subset of MOOC participants' pedagogical practice and posture was explored via semi-structured interviews throughout and beyond their MOOC participation using simulated and personal situational recalls. The verbatim were analysed with standard thematic analysis. Quantitative and qualitative findings converged and their integration demonstrated that partaking in the MOOC SCR promoted the development of complex pedagogical competencies and reflexivity with the participants. This was quantitatively evidenced by significantly higher self-assessed supervision skills and corresponding attitudes after completing the MOOC. The qualitative data provided rich descriptions of how this progression in pedagogical practice and posture occurred in the field and how it was shaped by participants' interaction with the MOOC's content and their motivations to progress. Our findings provide evidence for the development of pedagogical skills and corresponding attitudes for the supervision of clinical reasoning through participation in the MOOC SCR and contribute to the literature body on the opportunities that distance learning provides for the development of pedagogical competencies. The extent to which the pedagogical underpinnings of the MOOC contributed to these developments remains to be determined.

Contradictions and Opportunities: Reconciling Professional Identity Formation and Competency-Based Medical Education

The widespread adoption of Competency-Based Medical Education (CBME) has resulted in a more explicit focus on learners' abilities to effectively demonstrate achievement of the competencies required for safe and unsupervised practice. While CBME implementation has yielded many benefits, by focusing explicitly on what learners are doing, curricula may be unintentionally overlooking who learners are becoming (i.e., the formation of their professional identities). Integrating professional identity formation (PIF) into curricula has the potential to positively influence professionalism, well-being, and inclusivity; however, issues related to the definition, assessment, and operationalization of PIF have made it difficult to embed this curricular imperative into CBME. This paper aims to outline a path towards the reconciliation of PIF and CBME to better support the development of physicians that are best suited to meet the needs of society. To begin to reconcile CBME and PIF, this paper defines three contradictions that must and can be resolved, namely: (1) CBME attends to behavioral outcomes whereas PIF attends to developmental processes; (2) CBME emphasizes standardization whereas PIF emphasizes individualization; (3) CBME organizes assessment around observed competence whereas the assessment of PIF is inherently more holistic. Subsequently, the authors identify curricular opportunities to address these contradictions, such as incorporating process-based outcomes into curricula, recognizing the individualized and contextualized nature of competence, and incorporating guided self-assessment into coaching and mentorship programs. In addition, the authors highlight future research directions related to each contradiction with the goal of reconciling 'doing' and 'being' in medical education.

A Feasible Web-Conference-Style Remote Simulation using Demonstration Video Clips in Anaesthesia under the COVID-19 Outbreaks: A Preliminary Survey Study

Objective

The Coronavirus disease-2019 (COVID-19) outbreak has deprived simulation-based medical education for health care workers. Attendees are strictly prohibited to converge at a simulation training venue because of the COVID-19 outbreaks. To address this issue, we developed a web-conference-style remote simulation program using demonstration video clips. This report introduced the method and described participants' satisfaction.

Methods

This preliminary survey study evaluated learners' satisfaction in multiple institutions. The Satisfaction Scale Questionnaire with High-Fidelity Clinical Simulation (SSHF) by a 5-degree scale was used to evaluate participants' satisfaction. The survey was conducted immediately after completing the simulation sessions.

Results

Ten (100%) participants (7 anaesthesia individuals, 2 anaesthesia residents and 1 nurse anaesthetist) from nine institutions responded to the survey. All median values of the satisfaction scores were ≥4.0, whereas the median values of scores for environmental fidelity and psychological insecurity were 3.5 and 3.0, respectively (P=0.005).

Conclusion

A web-conference-style remote simulation using demonstration video clips is a feasible method for conducting simulation-based medical education under COVID-19 that showed high satisfaction scores. Further, additional studies are required to explore the internal and external validity and the effectiveness of mastery learning.

Entrustment versus performance scale in high-stakes OSCEs: Rater insights and psychometric properties

BACKGROUND

Although entrustment scales are increasingly applied in workplace-based assessments, their role in OSCEs remains unclear. We investigated raters' perceptions using an entrustment scale and psychometric analyses.

METHOD

A mixed-methods design was used. OSCE raters' (n = 162) perceptions were explored via questionnaire and four focus groups (n = 14). Psychometric OSCE properties were analyzed statistically.

RESULTS

Raters (n = 53, response rate = 41%) considered the entrustment scale comprehensible (89%) and applicable (60%). A total of 43% preferred the entrustment scale, 21% preferred the global performance scale, and 36% were undecided. Raters' written comments indicated that while they appreciated the authenticity of entrustment levels, they considered them subjective. The focus groups highlighted three main themes: (1) recollections of the clinical workplace as a cognitive reference triggered by entrustment scales; (2) factors influencing entrustment decisions; and (3) cognitive load is reduced at the perceived cost of objectivity. Psychometric analyses (n = 480 students) revealed improvements in some OSCE metrics when entrustment and global performance scales were combined.

CONCLUSION

Entrustment scales are beneficial for high-stakes OSCEs and have greater clinical relevance from the raters' perspective. Our findings support the use of entrustment and global performance scales in combination.

Navigating Challenging Conversations: The Interplay Between Inquiry and Knowledge Drives Preparation for Future Learning

Introduction

While some physicians hone their skills through informal learning in clinical practice, others do not. There is a lack of understanding of why some physicians seek improvement and how they use the workplace context to build their capabilities. Because physicians rarely pursue formal professional development activities to improve communication skills, examining physician-patient communication offers a powerful opportunity to illuminate important aspects of preparation for future learning in the workplace.

Methods

This qualitative observational study involved over 100 hours of observation of eight pediatric rehabilitation physicians as they interacted with patients and families at an academic teaching hospital in 2018-2020. Detailed field notes of observations, post-observation interviews, and exit interviews were the data sources. Data collection and analysis using a constructivist grounded theory approach occurred iteratively, and themes were identified through constant comparative analysis.

Results

Through their daily work, experienced physicians employ 'habits of inquiry' by constantly seeking a better understanding of how to navigate challenging conversations in practice through monitoring and attuning to situational and contextual cues, taking risks and navigating uncertainty while exploring new and varied ways of practicing, and seeking why their strategies are successful or not.

Discussion

Engaging in communication challenges drives physician learning through an interplay between habits of inquiry and knowledge: inquiry into how to improve their communication supported by existing conceptual knowledge to generate new strategies. These 'habits of inquiry' prompt continual reinvestment in problem solving to refine existing knowledge and to build new skills for navigating communication challenges in practice.

Pyramids in medical education: Their conceptualization and utility summarized

Objectives

To examine the conceptual underpinnings of learning pyramids, their limitations and some proposed modifications, and to describe their utility.

Materials and methods

Starting with a selection of pyramids based on whether graphic design was part of their original design and on their conceptual description, we examined the criticisms they have received and the modifications derived from those observations, and the use given to the pyramids most commonly used in medical education.

Results

Five pyramids were included, namely, George Miller, Edgar Dale, Donald Kirkpatrick, Benjamín Bloom and Abraham Maslow. Pyramids describe different aspects of medical education evaluation, either of individuals or of training programs, including competencies, identity, reliability, learning, behavior, results, cognitive complexity and self-realization.

Conclusions

As theoretical models, the pyramids examined have contributed to support learning processes in health professions. Their practical utility extends to different specialties and education levels given that they can help faculty optimize curricular design, teaching and evaluation processes. It is important to conduct a local assessment of the effects on health science programs built on these theoretical models.

The Resident Learning Journey in the Medical Intensive Care Unit

Background

The medical intensive care unit (MICU) offers rich resident learning opportunities, but traditional teaching strategies can be difficult to employ in this fast-paced, high-acuity environment. Resident perspectives of learning within this environment may improve our understanding of the common challenges residents face and inform novel approaches to transform the MICU educational experience.

Objective

We conducted a qualitative study of internal medicine residents to better understand their approach to learning the critical care activities that they are entrusted to perform in the MICU.

Methods

Using a thematic analysis approach, we conducted six focus group interviews with 15 internal medicine residents, separated by postgraduate year. A trained investigator led each interview, which was audio-recorded and transcribed verbatim for analysis. Our diverse research team, representing different career stages across the continuum of learning to minimize interpretive bias, identified codes and subsequent themes inductively. We refined these themes through group discussion and sensitizing social learning theory concepts using Wenger's community of practice and organized them to create learner archetypes and a conceptual framework of resident learning in the MICU.

Results

We identified three thematic resident learning categories: learning goals and motivation, clinical engagement, and interprofessional collaboration. We distinguished three learner archetypes, the novice, experiential learner, and practicing member, to describe progressive resident development within the interprofessional MICU team, the challenges they frequently encounter, and potential teaching strategies to facilitate learning.

Conclusion

We developed a conceptual framework that describes the resident's journey to becoming a trusted, collaborating member of the interprofessional MICU team. We identified common developmental challenges residents face and offer educational strategies that may support their progress. These findings should inform future efforts to develop novel teaching strategies to promote resident learning in the MICU.

Inconsistently reporting post-licensure EPA specifications in different clinical professions hampers fidelity and practice translation: a scoping review

BACKGROUND

Entrustable Professional Activities (EPAs) are defined units of professional practice entrusted to professionals once they have attained the specific competencies required to complete the end-to-end task. They provide a contemporary framework for capturing real-world clinical skillsets and integrating clinical education with practice. Our scoping review question was: how are post-licensure EPAs reported in peer reviewed literature, in different clinical professions?

METHOD

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) checklist, Arksey and O'Malley and Joanna Briggs Institute (JBI) methodology. Searching ten electronic databases returned 1622 articles, with 173 articles included. Data extracted included demographics, EPA discipline, titles and further specifications.

RESULTS

All articles were published between 2007-2021 across sixteen country contexts. The majority were from North America (n = 162, 73%) describing medical sub-specialty EPAs (n = 126, 94%). There were comparably few EPA frameworks reported in clinical professions other than medicine (n = 11, 6%). Many articles reported only EPA titles without further explanation and limited content validation. The majority did not include information about the EPA design process. Few EPAs and frameworks were reported according to all the recommended EPA attributes. There was unclear distinction between specialty-specific EPAs and those that could be useful across disciplines.

DISCUSSION

Our review highlights the large volume of EPAs reported in post-licensure medicine, including the volume disparity compared to other clinical professions. Basing our enquiry upon existing guidelines for EPA attributes and features, our experience in conducting the review and our primary finding demonstrated heterogeneity of EPA reporting according to these specifications. To promote EPA fidelity, and quality appraisal, and to reduce interpretation subjectivity, we advocate: diligently reporting EPA attributes and features; including reference or citation to EPA design and content validity information; and considering distinguishing EPAs as specialty-specific or transdisciplinary.

CONCLUSION

A large volume of post-licensure EPAs were identified in medicine relative to other clinical professions. EPA specifications were absent or variously reported in the literature, risking ambiguous interpretation. The authors recommend that future EPAs are reported with reference to established and evolving construct recommendations, which is integral to concept fidelity and translation to practice and education.

"A whole other competence story": exploring faculty perspectives on the process of workplace-based assessment of entrustable professional activities

The centrality of entrustable professional activities (EPAs) in competency-based medical education (CBME) is predicated on the assumption that low-stakes, high-frequency workplace-based assessments used in a programmatic approach will result in accurate and defensible judgments of competence. While there have been conversations in the literature regarding the potential of this approach, only recently has the conversation begun to explore the actual experiences of clinical faculty in this process. The purpose of this qualitative study was to explore the process of EPA assessment for faculty in everyday practice. We conducted 18 semi-structured interviews with Anesthesia faculty at a Canadian academic center. Participants were asked to describe how they engage in EPA assessment in daily practice and the factors they considered. Interviews were audio-recorded, transcribed, and analysed using the constant comparative method of grounded theory. Participants in this study perceived two sources of tension in the EPA assessment process that influenced their scoring on official forms: the potential constraints of the assessment forms and the potential consequences of their assessment outcome. This was particularly salient in circumstances of uncertainty regarding the learner's level of competence. Ultimately, EPA assessment in CBME may be experienced as higher-stakes by faculty than officially recognized due to these tensions, suggesting a layer of discomfort and burden in the process that may potentially interfere with the goal of assessment for learning. Acknowledging and understanding the nature of this burden and identifying strategies to mitigate it are critical to achieving the assessment goals of CBME.

Developing entrustable professional activities for family medicine training in South Africa

Workplace-based assessment (WPBA) is becoming part of high-stake assessments in specialist training. Entrustable professional activities (EPAs) are a recent addition to WPBA. This is the first South African publication on developing EPAs for postgraduate family medicine training. An EPA is a unit of practice, observable in the workplace, constituting several tasks with underlying knowledge, skills and professional behaviours. Entrustable professional activities allow for entrustable decisions regarding competence in a described work context. A national workgroup representing all nine postgraduate training programmes in South Africa has developed 19 EPAs. This new concept needs change management to understand the theory and practice of EPAs. Family medicine departments with large clinical workloads are small, necessitating navigating logistical issues to develop EPAs. It has unmasked existing workplace learning and assessment challenges.Contribution: This article contributes new thinking to developing EPAs for family medicine in an effort to understand more authentic WPBA nationally.

Clinical decision-making and adaptive expertise in residency: a think-aloud study

Clinical decision-making (CDM) is the ability to make clinical choices based on the knowledge and information available to the physician. It often refers to individual cognitive processes that becomes more dependent with the acquisition of experience and knowledge. Previous research has used dual-process theory to explain the cognitive processes involved in how physicians acquire experiences that help them develop CDM. However, less is known about how CDM is shaped by the physicians' situated cognition in the clinical environment. This is especially challenging for novice physicians, as they need to be adaptive to compensate for the lack of experience. The adaptive expert framework has been used to explain how novice physicians learn, but it has not yet been explored, how adaptive expertise is linked to clinical decision-making amongst novice physicians.This study aimed to analyse how residents utilize and develop adaptive expert cognition in a natural setting. By describing cognitive processes through verbalization of thought processes, we sought to explore their CDM strategies considering the adaptive expert framework.We used concurrent and retrospective think-aloud interviews in a natural setting of an emergency department (ED) at a university hospital, to query residents about their reasoning during a patient encounter. We analysed data using protocol analysis to map cognitive strategies from these verbalizations. Subsequently in a narrative analysis, we compared these strategies with the literature on adaptive expertise.Fourteen interviews were audio recorded over the course for 17 h of observation. We coded 78 informational concepts and 46 cognitive processes. The narrative analysis demonstrated how epistemic distance was prevalent in the initial CDM process and self-regulating processes occurred during hypothesis testing. However, residents who too quickly moved on to hypothesis testing tended to have to redirect their hypothesis more often, and thus be more laborious in their CDM. Uncertainty affected physicians' CDM when they did not reconcile their professional role with being allowed to be uncertain. This allowance is an important feature of orientation to new knowledge as it facilitates the evaluation of what the physician does not know.For the resident to learn to act as an adaptive decision-maker, she relied on contextual support. The professional role was crucial in decisional competency. This supports current literature, which argues that role clarification helps decisional competency. This study adds that promoting professional development by tolerating uncertainty may improve adaptive decisional competency.

The effect of breast self-examination training on nursing students by using hybrid-based simulation on knowledge, skills, and ability to correctly evaluate pathological findings: Randomized Controlled Study

AIM

To investigate the effectiveness of hybrid simulation-based training and classical-based training with the Miller pyramids' ability to evaluate students' knowledge, skills and pathologies in breast self-examination.

BACKGROUND

Breast cancer incidence is becoming more common among women in developed and developing countries. Nurses should be capable of evaluating breast self-examination knowledge, abilities and pathologies.

METHOD

A randomized full experimental design was used. Two groups were formed: a hybrid simulation-based training group (n = 36) and a classical-based training group (n = 36)-students in the hybrid simulation-based group trained on a standardized patient with a wearable breast attachment. The classical-based training group received training on the breast model. The laboratory performance of the students was evaluated one week after the training. Their performance on the standardized patient was evaluated against the checklist in the clinical setting one week after the laboratory. Before and after the training, each student's knowledge of basic and Application steps was evaluated. Students completed the self-description form regarding the clinical process after the clinical performance. All participants received 60 min of theoretical and 60 min of demonstration training. While the students in the hybrid simulation-based training group completed the 55-minute simulation training, the students in the Classic-based training group practised on the desktop breast model for 10 min. The hybrid simulation-based training group received 175 min of total training time, whereas the classical-based training group received 130 min.

RESULTS

Compared with classical-based training, teaching BSE with hybrid simulation-based training raised students' total knowledge score, application step score, skill score and pathological evaluation score (p < 0.05). Students who trained using hybrid simulation-based training experienced low levels of anxiety but increased levels of excitement and motivation (p < 0.05). The education and counselling scores of students who received hybrid simulation -based simulation training were higher than those who received classical-based training (p < 0.05). There was no statistically significant difference between the groups' BSE mean pre-test basic knowledge scores (p = 0.186). BSE mean application steps knowledge scores difference (pre-test-post-test) of the groups was found to be statistically significant (p < 0.001) CONCLUSION: The ability to accurately evaluate breast self-examination on knowledge, skills and pathological findings of students in the hybrid simulation-based training group was higher than in the classical-based training group. Besides, the students' perceived communication, education and counselling efficiency in the hybrid simulation-based training group were higher compared with the classical-based training group.

Development of entrustable professional activities for post-doctorate pediatric dentistry education

PURPOSE

To identify the core components of pediatric dentistry defining entrustable professional activities (EPAs) representing the profession.

METHODS

Potential core components of pediatric dentistry and corresponding domains were identified through review of literature and existing pediatric dentistry standards. A modified Delphi technique was utilized to rate these candidate EPAs to achieve consensus around prioritized EPAs.

RESULTS

Eleven participants participated in all three rounds of the Delphi. After three rounds, 16 candidate EPAs reached consensus for pediatric dentistry. Each EPA fell into one of four domains: "assessment and planning," "provision of care," "behavior guidance," and "professional development." An original candidate EPA focused on non-pharmacological behavior guidance was deemed too broad by the Delphi. This EPA was subsequently developed into three separate components on nitrous oxide analgesia, moderate sedation, and general anesthesia.

CONCLUSIONS

Prioritized EPAs will help define the essential activities of the profession and provide a framework for creating assessments to ensure that graduating pediatric residents are ready for unsupervised practice.

Balancing closure and discovery: adaptive expertise in the workplace

Residents must develop knowledge, skills, and attitudes to handle a rapidly developing clinical environment. To address this need, adaptive expertise has been suggested as an important framework for health professions education. However, research has yet to explore the relationship between workplace learning and adaptive expertise. This study sought to investigate how clinical supervision might support the development of adaptive expertise. The present study used a focused ethnography in two emergency departments. We observed 75 supervising situations with the 27 residents resulting in 116 pages of field notes. The majority of supervision was provided by senior physicians, but also included other healthcare professionals. We found that supervision could serve two purposes: closure and discovery. Supervision aimed at discovery included practices that reflected instructional approaches said to promote adaptive expertise, such as productive struggle. Supervision aimed at closure-included practices with instructional approaches deemed important for efficient and safe patient care, such as verifying information. Our results suggest that supervision is a shared practice and responsibility. We argue that setting and aligning expectations before engaging in supervision is important. Furthermore, results demonstrated that supervision was a dynamic process, shifting between both orientations, and that supervision aimed at discovery could be an an appropriate mode of supervision, even in the most demanding clinical situations.

'Next steps are…': An exploration of coaching and feedback language in EPA assessment comments

PURPOSE

Entrustable Professional Activities (EPA) assessments are intended to facilitate meaningful, low-stakes coaching and feedback, partly through the provision of written comments. We sought to explore EPA assessment comments provided to internal medicine (IM) residents for evidence of feedback and coaching language as well as politeness.

METHODS

We collected all written comments from EPA assessments of communication from a first-year IM resident cohort at the University of Toronto. Sensitized by politeness theory, we analyzed data using principles of constructivist grounded theory.

RESULTS

Nearly all EPA assessments (94%) contained written feedback based on focused clinical encounters. The majority of comments demonstrated coaching language, including phrases like 'don't forget to,' and 'next steps are,' followed by specific suggestions for improvement. A variety of words, including 'autonomy' and 'independence' denoted entrustment decisions. Linguistic politeness strategies such as hedging were pervasive, seemingly to minimize harm to the supervisor-trainee relationship.

CONCLUSION

Evidence of written coaching feedback suggests that EPA assessment comments are being used as intended as a means of formative feedback to promote learning. Yet, the frequent use of polite language suggests that EPAs may be higher-stakes than expected, highlighting a need for changes to the assessment culture and improved feedback literacy.

Fidelity to Best Practices in EPA Implementation: Outcomes Supporting Use of the Core Components Framework From the University of Virginia Entrustable Professional Activity Program

PROBLEM

The rapid expansion of entrustable professional activity (EPA) assessment programs has led to calls to ensure fidelity in implementation and integrity in meeting the goals of competency-based medical education. Initiated in July 2017, in advance of the articulated core components of EPA implementation, this article describes the structure and outcomes of the University of Virginia (UVA) EPA Program and provides support for the identified essential components.

APPROACH

The UVA EPA Program includes workplace assessments by residents/fellows, attending faculty, and master assessors (MAs), experienced clinicians who assess students across disciplines and clinical settings. All assessors participate in formal professional development and provide verbal and written comments to support their supervision ratings. The Entrustment Committee, composed of 12 MAs, uses a shared mental model and aggregates all assessor data to make a high-stakes summative entrustment decision about students' readiness to assume the role of an acting intern.

OUTCOMES

Since 2017, over 2,000 assessors have completed 56,969 EPA assessments for 1,479 students. Ninety-four percent of assessments have been done during the clerkship phase. Residents/fellows have completed a mean of 18 assessments, attending faculty a mean of 27, and MAs a mean of 882. Seventy-four percent of observed encounters involved patients with acute concerns with or without a co-morbid condition. Fifty percent of assessments occurred in inpatient and 32% in ambulatory settings. Eighty-seven percent of assessments contained narrative comments with more than 100 characters.

NEXT STEPS

Planned next steps will include earlier identification of students who require individualized learning to promote the development of skills related to EPAs, expansion of the remediation program to enable more students to engage in a clinical performance mastery elective, and creation of targeted professional development for assessors to reinforce the tenets of the EPA program.

Assessment of final-year medical students' entrustable professional activities after education on an interprofessional training ward: A case-control study

INTRODUCTION

Interprofessional training wards (ITWs) are implemented to provide medical students with a holistic and authentic health care experience to improve their clinical competencies. Controlled outcome studies assessing students' competencies after ITW-training are uncommon. In this case-control study, we assessed final-year medical students who received ITW-training regarding entrustable professional activities (EPAs) and communicative as well as social competencies.

METHODS

In March 2021, 32 final-year students, 16 with (ITW group) and 16 without (control group) a previous four-week placement on an ITW participated in a training simulating the first day of residency. The simulated patients assessed students' communication and interpersonal skills for history taking with the ComCare index after every consultation. Twelve prospective EPAs were assessed by three senior physicians after watching videos of the students' case presentations.

RESULTS

While baseline characteristics and ComCare index ratings were not significantly different between the two groups, the overall mean entrustment level for the 12 EPAs was significantly higher (p < 0.001) in the ITW group compared to the control group (median = 3.15 versus 2.22). The interrater reliability for all EPAs was high and entrustment in students from the ITW group was significantly higher in 10 out of 12 EPAs.

DISCUSSION

ITW training seems to prepare medical students well to practice competencies which are relevant for prospective entrustment decisions and can be deduced by senior physicians from case presentations. Further studies with larger student cohorts are needed to corroborate this finding and observable EPAs could also be defined to assess students' competencies after ITW training.

Adaptations in clinical examinations of medical students in response to the COVID-19 pandemic: a systematic review

INTRODUCTION

Clinical examinations (assessments) are integral to ensuring that medical students can treat patients safely and effectively. The COVID-19 pandemic disrupted traditional formats of clinical examinations. This prompted Medical Schools to adapt their approaches to conducting these examinations to make them suitable for delivery in the pandemic. This systematic review aims to identify the approaches that Medical Schools, internationally, adopted in adapting their clinical examinations of medical students in response to the COVID-19 pandemic.

METHODS

Three databases and four key medical education journals were systematically searched up to 22 October 2021; a grey literature search was also undertaken. Two reviewers independently screened at title, abstract stage and full text stage against predefined eligibility criteria. Discrepancies were resolved by discussion and involvement of senior authors. Risk of bias assessment was performed using an adapted version of a pre-existing risk of bias assessment tool for medical education developments. Results were summarised in a narrative synthesis.

RESULTS

A total of 36 studies were included, which documented the approaches of 48 Medical Schools in 17 countries. Approaches were categorised into in-person clinical examinations (22 studies) or online clinical examinations (14 studies). Authors of studies reporting in-person clinical examinations described deploying enhanced infection control measures along with modified patient participation. Authors of studies reporting online clinical examinations described using online software to create online examination circuits. All authors reported that adapted examinations were feasible, scores were comparable to previous years' student cohorts, and participant feedback was positive. Risk of bias assessment highlighted heterogeneity in reporting of the clinical examinations.

CONCLUSIONS

This review identified two broad approaches to adapting clinical examinations in the pandemic: in-person and online. Authors reported it was feasible to conduct clinical examinations in the pandemic where medical educators are given sufficient time and resources to carefully plan and introduce suitable adaptations. However, the risk of bias assessment identified few studies with high reporting quality, which highlights the need for a common framework for reporting of medical education developments to enhance reproducibility across wider contexts. Our review provides medical educators with the opportunity to reflect on past practises and facilitate the design and planning of future examinations.

Implementing an entrustable professional activities programmatic assessments for nurse practitioner training in emergency care: A pilot study

BACKGROUND

Entrustable professional activities (EPAs), as new methods used to operationalize competency-based education, are gaining acceptance in medical professions. However, no EPA-based framework exists to evaluate emergency care nurse practitioners' competencies in clinical practice.

OBJECTIVES

To develop, implement, and evaluate an EPA-based framework for emergency care nurse practitioners.

DESIGN

Prospective observational study.

SETTING

An accredited nurse practitioner training program in the emergency department of a University-affiliated Hospital.

PARTICIPANTS

Eight novice nurse practitioners and eighteen clinical instructors.

METHODS

A working group of academic educators, nurse practitioner leaders, and clinical instructors developed a set of EPAs using a participatory design method. We scheduled weekly ad hoc evaluations for the nurse practitioners in emergency care, and we collected several observations from multiple assessors for analysis.

RESULTS

Four nested EPAs were developed that adopted a 1-5 supervision-entrustment scale to measure performance. The results of the 20-month assessments illustrated that the average scores of EPA1, which measured the assessment and management of patients with acute medical presentations in the emergency department, differed significantly but were not positively correlated with the nurse practitioners' increasing subspecialty and department seniority levels (Kruskal-Wallis test P = .011 and 0.006, respectively). In addition, the most time required for the nurse practitioners to achieve a stable level 5 entrustability score for all EPAs for at least 3 consecutive months was 18 months. Clinical instructors noted that recognizing limits is a skill that is essential for a nurse practitioner to earn clinical instructors' trust.

CONCLUSIONS

Entrustability scores may not correlate with nurse practitioners' increasing seniority, and patient care requires the longest training time (18 months) for emergency care nurse practitioners to achieve autonomy. Program directors should adjust the supervision-entrustment level expected at various stages of training according to the complexity of individual EPAs. An EPA-based assessment framework in a nurse practitioner training program may assist in overseeing the development of competencies for emergency care nurse practitioners.

How can Entrustable Professional Activities serve the quality of health care provision through licensing and certification?

This paper about Entrustable Professional Activities (EPAs) was solicited to support the discussion about the future of licensing within the Medical Council of Canada. EPAs, units of professional practice to be entrusted to learners or professionals once they have shown to possess sufficient competence, were proposed in 2005 to operationalize competency-based postgraduate medical education and have become widely popular for various health professions education programs in many countries. EPAs break the breadth of competence for license down to units of practice that can be overseen, assessed, monitored, documented, and entrusted. EPAs together may constitute an individual's portfolio of qualifications, and define a scope of practice. A medical license and a specialty certification can then be defined as the required combination of EPAs for which one is qualified at any specific moment in time. That 'snapshot' could change over time and reflect the professional development of the individual, both in their competence and in their privileges to practice. Micro-credentialing and digital badges might become an adequate option to show-case one's scope of practice at any time and operationalize the idea of a dynamic portfolio of EPAs.

Concordance of Narrative Comments with Supervision Ratings Provided During Entrustable Professional Activity Assessments

BACKGROUND

Use of EPA-based entrustment-supervision ratings to determine a learner's readiness to assume patient care responsibilities is expanding.

OBJECTIVE

In this study, we investigate the correlation between narrative comments and supervision ratings assigned during ad hoc assessments of medical students' performance of EPA tasks.

DESIGN

Data from assessments completed for students enrolled in the clerkship phase over 2 academic years were used to extract a stratified random sample of 100 narrative comments for review by an expert panel.

PARTICIPANTS

A review panel, comprised of faculty with specific expertise related to their roles within the EPA program, provided a "gold standard" supervision rating using the comments provided by the original assessor.

MAIN MEASURES

Interrater reliability (IRR) between members of review panel and correlation coefficients (CC) between expert ratings and supervision ratings from original assessors.

KEY RESULTS

IRR among members of the expert panel ranged from .536 for comments associated with focused history taking to .833 for complete physical exam. CC (Kendall's correlation coefficient W) between panel members' assignment of supervision ratings and the ratings provided by the original assessors for history taking, physical examination, and oral presentation comments were .668, .697, and .735 respectively. The supervision ratings of the expert panel had the highest degree of correlation with ratings provided during assessments done by master assessors, faculty trained to assess students across clinical contexts. Correlation between supervision ratings provided with the narrative comments at the time of observation and supervision ratings assigned by the expert panel differed by clinical discipline, perhaps reflecting the value placed on, and perhaps the comfort level with, assessment of the task in a given specialty.

CONCLUSIONS

To realize the full educational and catalytic effect of EPA assessments, assessors must apply established performance expectations and provide high-quality narrative comments aligned with the criteria.

Variable or Fixed? Exploring Entrustment Decision Making in Workplace- and Simulation-Based Assessments

PURPOSE

Many models of competency-based medical education (CBME) emphasize assessing entrustable professional activities (EPAs). Despite the centrality of EPAs, researchers have not compared rater entrustment decisions for the same EPA across workplace- and simulation-based assessments. This study aimed to explore rater entrustment decision making across these 2 assessment settings.

METHOD

An interview-based study using a constructivist grounded theory approach was conducted. Gastroenterology faculty at the University of Toronto and the University of Calgary completed EPA assessments of trainees' endoscopic polypectomy performance in both workplace and simulation settings between November 2019 and January 2021. After each assessment, raters were interviewed to explore how and why they made entrustment decisions within and across settings. Transcribed interview data were coded iteratively using constant comparison to generate themes.

RESULTS

Analysis of 20 interviews with 10 raters found that participants (1) held multiple meanings of entrustment and expressed variability in how they justified their entrustment decisions and scoring, (2) held personal caveats for making entrustment decisions "comfortably" (i.e., authenticity, task-related variability, opportunity to assess trainee responses to adverse events, and the opportunity to observe multiple performances over time), (3) experienced cognitive tensions between formative and summative purposes when assessing EPAs, and (4) experienced relative freedom when using simulation to formatively assess EPAs but constraint when using only simulation-based assessments for entrustment decision making.

CONCLUSIONS

Participants spoke about and defined entrustment variably, which appeared to produce variability in how they judged entrustment across participants and within and across assessment settings. These rater idiosyncrasies suggest that programs implementing CBME must consider how such variability affects the aggregation of EPA assessments, especially those collected in different settings. Program leaders might also consider how to fulfill raters' criteria for comfortably making entrustment decisions by ensuring clear definitions and purposes when designing and integrating workplace- and simulation-based assessments.

Who can do this procedure? Using entrustable professional activities to determine curriculum and entrustment in anesthesiology - An international survey

INTRODUCTION

As competency-based curricula get increasing attention in postgraduate medical education, Entrustable Professional Activities (EPAs) are gaining in popularity. The aim of this survey was to determine the use of EPAs in anesthesiology training programs across Europe and North America.

METHODS

A survey was developed and distributed to anesthesiology residency training program directors in Switzerland, Germany, Austria, Netherlands, USA and Canada. A convergent design mixed-methods approach was used to analyze both quantitative and qualitative data.

RESULTS

The survey response rate was 38% (108 of 284). Seven percent of respondents used EPAs for making entrustment decisions. Fifty-three percent of institutions have not implemented any specific system to make such decisions. The majority of respondents agree that EPAs should become an integral part of the training of residents in anesthesiology as they are universal and easy to use.

CONCLUSION

Although recommended by several national societies, EPAs are used in few anesthesiology training programs. Over half of responding programs have no specific system for making entrustment decisions. Although several countries are adopting or planning to adopt EPAs and national societies are recommending the use of EPAs as a framework in their competency-based programs, few are yet using these to make "competence" decisions.

A Literature Review: Entrustable Professional Activities, an assessment tool for postgraduate dental training?

Assessing when dental trainees are ready to independently undertake clinical procedures at specialist level is critical for dental postgraduate programmes to determine when a trainee is 'work ready', in addition to ensuring patient safety. Entrustable professional activities (EPA) are a novel method of competency-based assessment. An EPA is a unit of professional practice or critical clinical activity identified within dental training programmes, which should be assessed during training, to establish if trainees are ready for independent practice, with a progressive decrease in supervision, based on supervisors' entrustment decisions. This article describes EPAs, entrustment decisions, including entrustment supervision scales and the process recommended to develop EPAs within dental curricula. EPAs have not been formally introduced for assessment within dental education programmes in the United Kingdom, but recent developments have been described in undergraduate dental education globally. Clinical significance: Competency-based assessments need to be continually developed to adapt to rapidly changing population health care and dental needs, to determine when dental trainees are ready for independent clinical practice. Early development of entrustable professional activities for assessment in undergraduate dental programmes has been well received by both trainees and supervisors. Further investigation is required to consider formal development of EPAs within postgraduate dental programmes.

Techniques to Teach Students Effectively Using Telemedicine. Comment on "Incorporating Medical Students Into Primary Care Telehealth Visits: Tutorial"

Collapse

Measuring the Impact of a Faculty Development Program on Clinical Educators

INTRODUCTION

An Academy of Clinical Educators (ACE) was established at the University of Newcastle, to support and build capacity among existing and prospective medical educators. ACE established a Certificate of Clinical Teaching and Supervision (CCTS) program, the final assessment of which was a reflective piece on how the course has affected participants' practice as clinical teachers or supervisors and how changes are expected to impact learner achievement. We conducted a qualitative evaluation of these to explore the impact of the CCTS on participants' teaching.

METHODS

Thirty-one participants (of 90 completers to date) consented for their written reflections to undergo qualitative thematic analysis and completed a survey exploring their preparation for, and experience of the program, and application of skills learnt.

RESULTS

Most participants reported applying the skills gained through the CCTS to their teaching practice to a large (n=23; 72%) or very large (n=5; 16%) extent. Four themes emerged from the qualitative data, aligned with the topics of the CCTS: teaching structure; feedback; orientation; and assessment. Participants described application of more structured approaches to orientation, teaching and feedback, positive student responses, and self-reported satisfaction with adopted changes.

DISCUSSION

The CCTS has motivated change in the teaching practice of participants. Although evidence presented here is limited by the self-reported nature, descriptions of actual changes in practice were detailed and specific enough to suggest they could act as a proxy for objectively measured change in behaviour and outcome.

CONCLUSION

A faculty development program delivered to clinicians with a range of teaching and education-related roles, from varied clinical disciplines and professions, can promote improved, structured teaching and feedback.

Development of a national pain management competency profile to guide entry-level physiotherapy education in Canada

BACKGROUND

National strategies from North America call for substantive improvements in entry-level pain management education to help reduce the burden of chronic pain. Past work has generated a valuable set of interprofessional pain management competencies to guide the education of future health professionals. However, there has been very limited work that has explored the development of such competencies for individual professions in different regions. Developing profession-specific competencies tailored to the local context is a necessary first step to integrate them within local regulatory systems. Our group is working toward this goal within the context of entry-level physiotherapy (PT) programs across Canada.

AIMS

This study aimed to create a consensus-based competency profile for pain management, specific to the Canadian PT context.

METHODS

A modified Delphi design was used to achieve consensus across Canadian university-based and clinical pain educators.

RESULTS

Representatives from 14 entry-level PT programs (93% of Canadian programs) and six clinical educators were recruited. After two rounds, a total of 15 competencies reached the predetermined endorsement threshold (75%). Most participants (85%) reported being "very satisfied" with the process.

CONCLUSIONS

This process achieved consensus on a novel pain management competency profile specific to the Canadian PT context. The resulting profile delineates the necessary abilities required by physiotherapists to manage pain upon entry to practice. Participants were very satisfied with the process. This study also contributes to the emerging literature on integrated research in pain management by profiling research methodology that can be used to inform related work in other health professions and regions.

Trust and risk pitfalls in medical education: A qualitative study of clinical teachers

PURPOSE

Learning opportunities in teaching hospitals are gated by clinical teachers. One way to unpack their decision-making is to employ a 'trust and risk' model. This study aimed to uncover clinical teachers' experience of trust, risk and vulnerability as they participate in medical education.

METHODS

Hospital-based clinical teachers were interviewed about trust, risk and vulnerability in medical education. Data analysis was undertaken using a constructivist, qualitative framework.

RESULTS

Twenty demographically diverse clinical teachers participated. Trust and risk were regarded as fundamental workplace and teaching concepts. Their concerns fell into three domains of risk: clinical, teaching and personal. Being trusted unlocked clinical learning opportunities, whereas trust failure limited future participation. Feeling trusted or not affected wellbeing and self-efficacy. Trust and risk pitfalls in education included bias, asymmetry and sidelining.

CONCLUSIONS

This study adds to the literature by voicing clinical teachers' personal risks and vulnerabilities. Attention was drawn to the benefits of being perceived as trustworthy, and to the clinical, teaching and personal vulnerabilities of trust failure.If expert judgement of trustworthiness is to be legitimised as meaningful assessment, clinical teachers must be aware not only of how trust is built, but also the pitfalls of trust failure.

Exploring the Association Between USMLE Scores and ACGME Milestone Ratings: A Validity Study Using National Data From Emergency Medicine

PURPOSE

The United States Medical Licensing Examination (USMLE) sequence and the Accreditation Council for Graduate Medical Education (ACGME) milestones represent 2 major components along the continuum of assessment from undergraduate through graduate medical education. This study examines associations between USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores and ACGME emergency medicine (EM) milestone ratings.

METHOD

In February 2019, subject matter experts (SMEs) provided judgments of expected associations for each combination of Step examination and EM subcompetency. The resulting sets of subcompetencies with expected strong and weak associations were selected for convergent and discriminant validity analysis, respectively. National-level data for 2013-2018 were provided; the final sample included 6,618 EM residents from 158 training programs. Empirical bivariate correlations between milestone ratings and Step scores were calculated, then those correlations were compared with the SMEs' judgments. Multilevel regression analyses were conducted on the selected subcompetencies, in which milestone ratings were the dependent variable, and Step 1 score, Step 2 CK score, and cohort year were independent variables.

RESULTS

Regression results showed small but statistically significant positive relationships between Step 2 CK score and the subcompetencies (regression coefficients ranged from 0.02 [95% confidence interval (CI), 0.01-0.03] to 0.12 [95% CI, 0.11-0.13]; all P < .05), with the degree of association matching the SMEs' judgments for 7 of the 9 selected subcompetencies. For example, a 1 standard deviation increase in Step 2 CK score predicted a 0.12 increase in MK-01 milestone rating, when controlling for Step 1. Step 1 score showed a small statistically significant effect with only the MK-01 subcompetency (regression coefficient = 0.06 [95% CI, 0.05-0.07], P < .05).

CONCLUSIONS

These results provide incremental validity evidence in support of Step 1 and Step 2 CK score and EM milestone rating uses.