Comparability of outcome frameworks in medical education: Implications for framework development

Conceptualizing Educational Comparability in Distributed Health Professions Education: A Scoping Review

PURPOSE

This study aimed to create greater clarity about the current understanding and formulate a model of how educational comparability has been used in the literature to inform practice.

METHOD

The authors conducted a literature search of 9 online databases, seeking articles published on comparability in distributed settings in health professions education before August 2021, with an updated search conducted in May 2023. Using a structured scoping review approach, 2 reviewers independently screened articles for eligibility with inclusion criteria and extracted key data. All authors participated in the descriptive analysis of the extracted data.

RESULTS

Twenty-four articles published between 1987 and 2021 met the inclusion criteria. Most articles were focused on medical education programs (n = 21) and located in North America (n = 18). The main rationale for discussing comparability was accreditation. These articles did not offer definitions or discussions about what comparability means. The program logic model was used as an organizing framework to synthesize the literature on practices that schools undertake to facilitate and demonstrate comparability in the design (inputs), implementation (activities), and evaluation (outcomes) of distributed education. Inputs include common learning objectives, identical assessment tools and policies, governance models that enable clear communication, and reporting structure that is supported by technological infrastructure. Activities include faculty planning meetings and faculty development training. Outcomes include student experiences and academic performances.

CONCLUSIONS

This study demonstrated that a more complex understanding of the dynamics of educational processes and practices is required to better guide the practice of educational comparability within distributed education programs. In addition to highlighting the need to develop an accepted definition of educational comparability, further elucidation of the underlying dynamics among input, activities, and outcomes would help to better determine what drivers should be prioritized when considering educational change with attention to context within distributed education.

Cracking the Nut on LCME Standard 8.7: Innovations to Ensure Comparability Across Geographically Distributed Campuses

Problem: A large state university in the southeastern United States and state Area Health Education Centers (AHEC) collaborated to establish branch campuses to increase clinical capacity for medical student education. Prior to formally becoming branch campuses, two AHEC sites had established innovative curricular structures different than the central campus. These sites worked with the central campus as clinical training sites. Upon becoming formal campuses, their unique clinical experiences were maintained. A third campus established a curricular structure identical to the central campus. Little exists in the literature regarding strategies that ensure comparability yet allow campuses to remain unique and innovative. Intervention: We implemented a balanced matrix organizational structure, well-defined communication plan, and newly developed tool to track comparability. A balanced matrix organization model framed the campus relationships. Adopting this model led to identifying reporting structures, developing multidirectional communication strategies, and the Campus Comparability Tool. Context: The UNC School of Medicine central campus is in Chapel Hill. All 192 students complete basic science course work on central campus. For required clinical rotations, approximately 140 students are assigned to the central campus, which includes rotations in Raleigh or Greensboro. The remaining students are assigned to Asheville (25-30), Charlotte (25-30), or Wilmington (5-7). Chapel Hill and Wilmington follow identical rotation structures, 16 weeks each of (a) combined surgery and adult inpatient experiences; (b) combined obstetrics/gynecology, psychiatry, and inpatient pediatrics; and (c) longitudinal clinical experiences in adult and pediatric medicine. Asheville offers an 8-month longitudinal integrated outpatient experience with discreet inpatient experiences in surgery and adult care. Charlotte offers a 6-month longitudinal integrated experiences and 6 months of block inpatient experiences. Aside from Charlotte and Raleigh, the other sites are urban but surrounded by rural counties. Chapel Hill is 221 miles from Asheville, 141 from Charlotte, and 156 from Wilmington. Outcome: Using the balanced matrix organization, various reporting structures and lines of communication ensured the educational objectives for students were clear on all campuses. The communication strategies facilitated developing consistent evaluation metrics across sites to compare educational experiences. Lessons Learned: The complexities of different healthcare systems becoming regional campuses require deliberate planning and understanding the culture of those sites. Recognizing how size and location of the organization affects communication, the central campus took the lead centralizing functions when appropriate. Adopting uniform educational technology has played an essential role in evaluating the comparability of core educational content on campuses delivering content in very distinct ways.

Clinical Skills Simulation Complementing Core Content: Development of the Simulation Lab Integrated Curriculum Experience (SLICE)

Simulation is emerging as an essential component of the medical school curriculum. Simulation Lab Integrated Curriculum Experience (SLICE) is a student-organized program at the University of North Carolina School of Medicine (UNC SOM) for medical students that provides skills-based training sessions to augment didactic learning experiences. During its pilot year, SLICE conducted five events with respondents completing pre-and post-surveys evaluating participants' level of comfort with procedures. There was a significant increase in self-reported confidence after each session, with students providing overwhelmingly positive feedback regarding SLICE's ability to contextualize material presented in traditional lectures.

Selection into medicine: the predictive validity of an outcome-based procedure

BACKGROUND

Medical schools must select students from a large pool of well-qualified applicants. A challenging issue set forward in the broader literature is that of which cognitive and (inter)personal qualities should be measured to predict diverse later performance. To address this gap, we designed a 'backward chaining' approach to selection, based on the competences of a 'good doctor'. Our aim was to examine if this outcome-based selection procedure was predictive of study success in a medical bachelor program.

METHODS

We designed a multi-tool selection procedure, blueprinted to the CanMEDS competency framework. The relationship between performance at selection and later study success across a three-year bachelor program was examined in three cohorts. Study results were compared between selection-positive and selection-negative (i.e. primarily rejected) students.

RESULTS

Selection-positive students outperformed their selection-negative counterparts throughout the entire bachelor program on assessments measuring cognitive (e.g. written exams), (inter)personal and combined outcomes (i.e. OSCEs). Of the 30 outcome variables, selection-positive students scored significantly higher in 11 cases. Fifteen other, non-significant between-group differences were also in favor of the selection-positives. An overall comparison using a sign test indicated a significant difference between both groups (p < 0.001), despite equal pre-university GPAs.

CONCLUSIONS

The use of an outcome-based selection approach seems to address some of the predictive validity limitations of commonly-used selection tools. Selection-positive students significantly outperformed their selection-negative counterparts across a range of cognitive, (inter)personal, and mixed outcomes throughout the entire three-year bachelor in medicine.

Development, implementation and first insights of a time- and location-independent longitudinal postgraduate curriculum in emergency medicine

Introduction, background and context: There have been few reports on the implementation of a structured curriculum for emergency medicine, as emergency medicine is not yet an established medical specialty for training in many European countries, including Switzerland and Germany. Because of the non-plannable workload in the emergency setting, common training approaches are often difficult to implement. Need-assessments of emergency medicine trainees commonly identify a need for interactive, time-independent ways of learning that integrate modern forms of knowledge transfer. Methods: In the present study, we assess the local needs of emergency medicine specialists and trainees for a curriculum in emergency medicine and elaborate possible solutions for the implementation of this curriculum, taking into account the special needs in a highly dynamic, unplannable environment, such as an interdisciplinary emergency department. Results: We describe the development of the emergency medicine curriculum on the basis of the six steps proposed by Kern for curriculum development in medical education, as well as the implementation, lessons learned and interval evaluation. Conclusions: The combination of multiple teaching formats, ranging from time- and location-independent solutions such as podcasted lectures to simulation-based training sessions, as well as small-group workshops and skill training sessions, might be a valuable approach to implementing a state-of-the-art curriculum in a busy emergency department.

Toward a shared language for competency-based medical education

The paradigm shift brought about by the advent of competency-based medical education (CBME) can be characterized as an adaptive change. Currently, its development and implementation suffer from the lack of a lingua franca. A shared language is needed to support collaboration and dissemination across the world community of medical educators. The International CBME Collaborators held a second summit in 2013 to explore this and other contemporary CBME issues. We present the resulting International CBME Collaborator's glossary of CBME terms. Particular attention is given to the terms competency, entrustable professional activity (EPA), and milestone and their interrelationships. Medical education scholars and enthusiasts of the competency-based approach are encouraged to adopt these terms and definitions, although no doubt the vocabulary of CBME will continue to evolve.

What makes a doctor a scholar: a systematic review and content analysis of outcome frameworks

BACKGROUND

Many national outcome frameworks (OF) call for a sound scholarship education and scholarly behaviour of physicians. Educators however are known to interpret the scholar role in markedly different ways and at least one major initiative to unify several national outcome frameworks failed to agree on a common definition of the scholar role. Both circumstances currently limit the development of educational and assessment strategies specific for the scholar role. Given increasing physician mobility together with the global perspective inherent in a doctor's role as a scholar, we were interested in what different OFs define as the scholar role and attempted to identify communalities and differences between them.

METHODS

We conducted a systematic review for OF in medical education in PubMed and google. After in- and exclusion processes, we extracted all content listed under the scholar role (if present) and categorized it based on Boyer's established model of scholarship. Next, we extracted all content related to scholarship from OFs not explicitly defining a scholar role and used it to validate the categories resulting from step one.

RESULTS

From 1816 search results, we identified 13 eligible OFs, seven of which explicitly specified a scholar role. The outcomes only partly map onto Boyer's definition of scholarship: Discovery, Integration, Application, and Teaching. We adapted and validated a model extending this definition to contain Common Basics (partly overlapping with Integration and Teaching), Clinical Application (specifying Application), Research (Discovery and partly Integration), Teaching and Education (partly overlapping with Teaching) and Lifelong Learning (no equivalent in Boyer's model). Whereas almost all OFs cover Common Basics, Clinical Application, and Lifelong Learning, fewer and less specific outcomes relate to Research or Teaching.

CONCLUSIONS

The need to adapt existing models of scholarship may result from the changing demands directed at medical scholars. The considerable differences identified between OFs may explain why educators have difficulties defining the scholar role and why the role is rarely assessed. We may have missed OFs due to our in- and exclusion criteria but the results provide a solid basis on which to build a common understanding of what makes a doctor a scholar.

The future of practical skills in undergraduate medical education - an explorative Delphi-Study

BACKGROUND

64% of young medical professionals in Germany do not feel adequately prepared for the practical requirements of the medical profession. The goal of "outcome-orientated training" is to structure medical curricula based on the skills needed when entering the workforce after completing undergraduate medical education, and thus to bridge the gap between the skills graduates have attained and those necessary for a career in the medical profession. Outcome frameworks (OFs) are used for this purpose. In preparation for developing the National Competence-Based Catalogue of Learning Objectives for Medicine (NKLM) - the German OF - the "Consensus Statement of Practical Skills in Undergraduate Medical Education" (which structures the teaching and acquisition of practical skills in Germany and which strongly influenced the "Clinical-Practical Skills" chapter of the NKLM) was published in 2011. It is not uncommon for at least a decade to elapse between the definition and implementation of an OF and the students' graduation, which can further increase the gap between necessary and acquired skills. Thus, the purpose of this paper is to posit theses for future development in healthcare and to apply these theses to a current OF.

METHODOLOGY

Partially structured interviews with experts were used to generate theses pertaining to general, future development in healthcare. These theses were assessed by physician experts based on the likelihood of implementation by the year 2025. The 288 learning goals of the consensus statement were assessed for their relevance for medical education in the interim.

RESULTS

11 theses were generated for the development of medicine, and these theses were assessed and discussed by 738 experts. These theses include the increase in diseases associated with old age, the increasing significance of interprofessional cooperation, and the growing prevalence of telemedicine applications. Of the 288 learning goals of the consensus statement, 231 of the goals were assessed as relevant, and 57 were deemed irrelevant for the short-term future.

DISCUSSION

The theses on the future of healthcare, which were generated in this study and which were validated by numerous experts, provide indications of future developments of overall requirements for medical school graduates. For example, when applied to the content of the "Clinical-Practical Skills" NKLM chapter, they largely validate the future relevance of developing practical skills while also providing indications for their further development as applied to the consensus statement.

The future of practical skills in undergraduate medical education - an explorative Delphi-Study

BACKGROUND

64% of young medical professionals in Germany do not feel adequately prepared for the practical requirements of the medical profession. The goal of "outcome-orientated training" is to structure medical curricula based on the skills needed when entering the workforce after completing undergraduate medical education, and thus to bridge the gap between the skills graduates have attained and those necessary for a career in the medical profession. Outcome frameworks (OFs) are used for this purpose. In preparation for developing the National Competence-Based Catalogue of Learning Objectives for Medicine (NKLM) - the German OF - the "Consensus Statement of Practical Skills in Undergraduate Medical Education" (which structures the teaching and acquisition of practical skills in Germany and which strongly influenced the "Clinical-Practical Skills" chapter of the NKLM) was published in 2011. It is not uncommon for at least a decade to elapse between the definition and implementation of an OF and the students' graduation, which can further increase the gap between necessary and acquired skills. Thus, the purpose of this paper is to posit theses for future development in healthcare and to apply these theses to a current OF.

METHODOLOGY

Partially structured interviews with experts were used to generate theses pertaining to general, future development in healthcare. These theses were assessed by physician experts based on the likelihood of implementation by the year 2025. The 288 learning goals of the consensus statement were assessed for their relevance for medical education in the interim.

RESULTS

11 theses were generated for the development of medicine, and these theses were assessed and discussed by 738 experts. These theses include the increase in diseases associated with old age, the increasing significance of interprofessional cooperation, and the growing prevalence of telemedicine applications. Of the 288 learning goals of the consensus statement, 231 of the goals were assessed as relevant, and 57 were deemed irrelevant for the short-term future.

DISCUSSION

The theses on the future of healthcare, which were generated in this study and which were validated by numerous experts, provide indications of future developments of overall requirements for medical school graduates. For example, when applied to the content of the "Clinical-Practical Skills" NKLM chapter, they largely validate the future relevance of developing practical skills while also providing indications for their further development as applied to the consensus statement.

The scholar role in the National Competence Based Catalogues of Learning Objectives for Undergraduate Medical Education (NKLM) compared to other international frameworks

BACKGROUND

In Germany, a national competence based catalogue of learning objectives in medicine (NKLM) was developed by the Society for Medical Education and the Council of Medical Faculties. As many of its international counterparts the NKLM describes the qualifications of medical school graduates. The definition of such outcome frameworks indents to make medical education transparent to students, teachers and society. The NKLM aims to amend existing lists of medical topics for assessment with learnable competencies. All outcome frameworks are structured into chapters, domains or physician roles. The definition of the scholar-role poses a number of questions such as: What distinguishes necessary qualifications of a scientifically qualified physician from those of a medical scientist?

METHODS

13 outcome frameworks were identified through a systematic three-step literature review and their content compared to the scholar role in the NKLM by means of a qualitative text analysis. The three steps consist of (1) search for outcome frameworks, (2) in- and exclusion, and (3) data extraction, categorization, and validation. The results were afterwards matched with the scholar role of the NKLM.

RESULTS

Extracted contents of all frameworks may be summarized into the components Common Basics, Clinical Application, Research, Teaching and Education, and Lifelong Learning. Compared to the included frameworks the NKLM emphasises competencies necessary for research and teaching while clinical application is less prominently mentioned.

CONCLUSION

The scholar role of the NKLM differs from other international outcome frameworks. Discussing these results shall increase propagation and understanding of the NKLM and thus contribute to the qualification of future medical graduates in Germany.