Measuring Graduate Medical Education Outcomes to Honor the Social Contract

From admissions to licensure: education data associations from a multi-centre undergraduate medical education collaboration

This paper reports the findings of a Canada based multi-institutional study designed to investigate the relationships between admissions criteria, in-program assessments, and performance on licensing exams. The study's objective is to provide valuable insights for improving educational practices across different institutions. Data were gathered from six medical schools: McMaster University, the Northern Ontario School of Medicine University, Queen's University, University of Ottawa, University of Toronto, and Western University. The dataset includes graduates who undertook the Medical Council of Canada Qualifying Examination Part 1 (MCCQE1) between 2015 and 2017. The data were categorized into five distinct sections: demographic information as well as four matrices: admissions, course performance, objective structured clinical examination (OSCE), and clerkship performance. Common and unique variables were identified through an extensive consensus-building process. Hierarchical linear regression and a manual stepwise variable selection approach were used for analysis. Analyses were performed on data set encompassing graduates of all six medical schools as well as on individual data sets from each school. For the combined data set the final model estimated 32% of the variance in performance on licensing exams, highlighting variables such as Age at Admission, Sex, Biomedical Knowledge, the first post-clerkship OSCE, and a clerkship theta score. Individual school analysis explained 41-60% of the variance in MCCQE1 outcomes, with comparable variables to the analysis from of the combined data set identified as significant independent variables. Therefore, strongly emphasising the need for variety of high-quality assessment on the educational continuum. This study underscores the importance of sharing data to enable educational insights. This study also had its challenges when it came to the access and aggregation of data. As such we advocate for the establishment of a common framework for multi-institutional educational research, facilitating studies and evaluations across diverse institutions. This study demonstrates the scientific potential of collaborative data analysis in enhancing educational outcomes. It offers a deeper understanding of the factors influencing performance on licensure exams and emphasizes the need for addressing data gaps to advance multi-institutional research for educational improvements.

Meeting the Health Needs of Society Through Curricular Innovation in Physical Therapist Education: Examples From 3 Entry-Level Programs

BACKGROUND AND PURPOSE

Entry-level physical therapist (PT) education programs play a critical role in inspiring future leaders to become moral change agents, capable of understanding and addressing evolving societal health needs. Social reconstructionism represents an educational philosophy focused on alleviating pervasive inequities and improving the health of society; however, its application in PT education is not well understood. The purpose of this article is to describe the approach 3 entry-level PT programs used to manifest social reconstructionism within their curricula to foster social consciousness and strengthen moral agency.

METHOD/MODEL DESCRIPTION AND EVALUATION

The Framework for Educating Health Professionals to Address the Social Determinants of Health informed the development of a conceptual framework that illustrates the role of PT education programs in creating transformative learning environments and preparing learners to meet societal health needs. Concepts within this framework were operationalized according to 5 domains put forth by Jensen et al. and related competencies put forth by the Association of American Medical Colleges.

OUTCOMES

Narratives from each program demonstrate how the design and implementation of curricula, grounded in social reconstructionism, can be accomplished through 1) ongoing modifications, 2) thoughtful reconstruction, and 3) initial design. Sample learning activities, objectives, and assessment strategies are provided.

DISCUSSION AND CONCLUSION

The proposed method was effective in guiding meaningful objectives, learning activities, and assessment strategies grounded in social reconstructionism. Such findings can inform the design of curricula that inspire the next generation of transformative leaders who work to alleviate pervasive inequities and improve the health of society.

Postresidency Practice Setting and Clinical Care Features According to 3 Versus 4 Years of Training in Family Medicine: A Length of Training Pilot Study

BACKGROUND AND OBJECTIVES

Factors associated with physician practice choice include residency location, training experiences, and financial incentives. How length of training affects practice setting and clinical care features postgraduation is unknown.

METHODS

In this Length of Training Pilot (LoTP) study, we surveyed 366 graduates of 3-year (3YR) and 434 graduates of 4-year (4YR) programs 1 year after completion of training between 2013 and 2021. Variables assessed included reasons for practice setting choice, practice type, location, practice and community size, specialty mix, and clinical care delivery features (eg, integrated behavioral health, risk stratified care management). We compared different length of training models using χ2 or Fisher's exact tests for categorical variables and independent samples, and t test (unequal variances) for continuous variables.

RESULTS

Response rates ranged from 50% to 88% for 3YR graduates and 68% to 95% for 4YR graduates. Scope of practice was a predominant reason for graduates choosing their eventual practice, and salary was a less likely reason for those completing 4 years versus 3 years of training (scope, 72% vs 55%, P=.001; salary, 15% vs 22%, P=.028). Community size, practice size, practice type, specialty mix, and practice in a federally designated underserved site did not differ between the two groups. We found no differences in patient-centered medical home features when comparing the practices of 3YR to 4YR graduates.

CONCLUSIONS

Training length did not affect practice setting or practice features for graduates of LoTP programs. Future LoTP analyses will examine how length of training affects scope of practice and clinical preparedness, which may elucidate other elements associated with practice choice.

Ten-Year Outcomes: Community Health Center/Academic Medicine Partnership for Rural Family Medicine Training

BACKGROUND AND OBJECTIVES

The widening gap between urban and rural health outcomes is exacerbated by physician shortages that disproportionately affect rural communities. Rural residencies are an effective mechanism to increase physician placement in rural and medically underserved areas yet are limited in number due to funding. Community health center/academic medicine partnerships (CHAMPs) can serve as a collaborative framework for expansion of academic primary care residencies outside of traditional funding models. This report describes 10-year outcomes of a rural training pathway developed as part of a CHAMP collaboration.

METHODS

Using data from internal registries and public sources, our retrospective study examined demographic and postgraduation practice characteristics for rural pathway graduates. We identified the rates of postgraduation placement in rural (Federal Office of Rural Health Policy grant-eligible) and federally designated Medically Underserved Areas/Populations (MUA/Ps). We assessed current placement for graduates >3 years from program completion.

RESULTS

Over a 10-year period, 25 trainees graduated from the two residency expansion sites. Immediately postgraduation, 84% (21) were in primary care Health Professional Shortage Areas (HPSAs), 80% (20) in MUA/Ps, and 60% (15) in rural locations. Sixteen graduates were >3 years from program completion, including 69% (11) in primary care HPSAs, 69% (11) in MUA/Ps, and 50% (5) in rural locations.

CONCLUSIONS

This CHAMP collaboration supported development of a rural pathway that embedded family medicine residents in community health centers and effectively increased placement in rural and MUA/Ps. This report adds to national research on rural workforce development, highlighting the role of academic-community partnerships in expanding rural residency training outside of traditional funding models.

Exploring the Validity and Reliability of Neurosurgery Residency Program Rankings: A Quantitative Analysis

OBJECTIVE

To evaluate the relationships between Doximity rankings (Doximity, Inc.) of residency programs and 2 new ranking systems based on publication rates and academic pursuits.

METHODS

We collected data on 550 neurosurgery graduates over 3 years. We analyzed the median number of published manuscripts per resident and the percentage of residents pursuing academic careers and compared them across the Doximity Research Productivity and Reputation Rankings. We used logistic regression to evaluate the relationships among the rankings, publication rates, and academic pursuits.

RESULTS

Neurosurgery residents published a median of 10 manuscripts per person (IQR: 6-17), and 50% (IQR: 33%-67%) of residents in a given program pursued an academic career. The distributions of the median number of published manuscripts across the Doximity Research Productivity Ranking and the Doximity Reputation Ranking tiers differed significantly (all P < 0.001). Similarly, the distribution of the percentage of residents pursuing an academic career across both published Doximity ranking systems' tiers differed significantly (all P = 0.02). Moreover, we found moderate agreement between the 2 Doximity rankings, fair agreement between the publication and the other 3 rankings, and slight agreement between the academic pursuit and the Doximity rankings.

CONCLUSIONS

We introduced 2 new methods to rank residency programs based on the number of graduates pursuing an academic position and the median number of published manuscripts per resident. By taking a comprehensive approach, neurosurgery applicants can ensure that they select a residency program that meets their needs and offers them the best opportunity for success.

Growing a rural family physician workforce: The contributions of rural background and rural place of residency training

OBJECTIVE

To determine the distinct influences of rural background and rural residency training on rural practice choice among family physicians.

DATA SOURCES AND STUDY SETTING

We used a subset of The RTT Collaborative rural residency list and longitudinal data on family physicians from the American Board of Family Medicine National Graduate Survey (NGS; three cohorts, 2016-2018) and American Medical College Application Service (AMCAS).

STUDY DESIGN

We conducted a logistic regression, computing predictive marginals to assess associations of background and residency location with physician practice location 3 years post-residency.

DATA COLLECTION/EXTRACTION METHODS

We merged NGS data with residency type-rural or urban-and practice location with AMCAS data on rural background.

PRINCIPAL FINDINGS

Family physicians from a rural background were more likely to choose rural practice (39.2%, 95% CI = 35.8, 42.5) than those from an urban background (13.8%, 95% CI = 12.5, 15.0); 50.9% (95% CI = 43.0, 58.8) of trainees in rural residencies chose rural practice, compared with 18.0% (95% CI = 16.8, 19.2) of urban trainees.

CONCLUSIONS

Increasing rural programs for training residents from both rural and urban backgrounds, as well as recruiting more rural students to medical education, could increase the number of rural family physicians.

An Ideal System of Assessment to Support Competency-Based Graduate Medical Education: Key Attributes and Proposed Next Steps

Competency-based medical education (CBME) is the future of medical education and relies heavily on high quality assessment. However, the current assessment practices employed by many general surgery graduate medical education training programs are subpar. Assessments often lack reliability and validity evidence, have low faculty engagement, and differ from program to program. Given the importance of assessment in CBME, it is critical that we build a better assessment system for measuring trainee competency. We propose that an ideal system of assessment is standardized, evidence-based, comprehensive, integrated, and continuously improving. In this article, we explore these characteristics and propose next steps to achieve such a system of assessment in general surgery.

The Next Era of Assessment: Building a Trustworthy Assessment System

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

Consistency and quality in written accreditation protocols for pediatrician training programs: a mixed-methods analysis of a global sample, and directions for improvement

BACKGROUND

The World Federation for Medical Education (WFME) defines accreditation as 'certification of the suitability of medical education programs, and of…competence…in the delivery of medical education.' Accreditation bodies function at national, regional and global levels. In 2015, WFME published quality standards for accreditation of postgraduate medical education (PGME). We compared accreditation of pediatric PGME programs to these standards to understand variability in accreditation and areas for improvement.

METHODS

We examined 19 accreditation protocols representing all country income levels and world regions. For each, two raters assessed 36 WFME-defined accreditation sub-areas as present, partially present, or absent. When rating "partially present" or "absent", raters noted the rationale for the rating. Using an inductive approach, authors qualitatively analyzed notes, generating themes in reasons for divergence from the benchmark.

RESULTS

A median of 56% (IQR 43-77%) of WFME sub-areas were present in individual protocols; 22% (IQR 15-39%) were partially present; and 8.3% (IQR 5.5-21%) were absent. Inter-rater agreement was 74% (SD 11%). Sub-areas least addressed included number of trainees, educational expertise, and performance of qualified doctors. Qualitative themes of divergence included (1) variation in protocols related to heterogeneity in program structure; (2) limited engagement with stakeholders, especially regarding educational outcomes and community/health system needs; (3) a trainee-centered approach, including equity considerations, was not universal; and (4) less emphasis on quality of education, particularly faculty development in teaching.

CONCLUSIONS

Heterogeneity in accreditation can be appropriate, considering cultural or regulatory context. However, we identified broadly applicable areas for improvement: ensuring equitable access to training, taking a trainee-centered approach, emphasizing quality of teaching, and ensuring diverse stakeholder feedback.

Neurology Education in 2035: The Neurology Future Forecasting Series

In the past decade, there have been dramatic changes in all aspects of neurologic care, and along with this, neurology education has transformed. These changes have affected all aspects of education across the educational continuum, including learners, teachers, educators, content, delivery methods, assessments, and outcomes. Health systems science, health humanities, diversity, equity, and inclusion and health disparities are becoming core components of neurology curricula, and, in the future, will be integrated into every aspect of our educational mission. The ways in which material is taught and learned have been influenced by technologic innovations and a growing understanding of the science of learning. We forecast that this trend will continue, with learners choosing from an array of electronic resources to engage with fundamental topics, allowing front-line clinical teachers to spend more time supporting critical reasoning and teaching students how to learn. There has been a growing differentiation of educational roles (i.e., teachers, educators, and scholars). We forecast that these roles will become more distinct, each with an individualized pattern of support and expectations. Assessment has become more aligned with the work of the learners, and there are growing calls to focus more on the impact of educational programs on patient care. We forecast that there will be an increased emphasis on educational outcomes and public accountability for training programs. In this article, we reflect on the history of medical education in neurology and explore the current state to forecast the future of neurology education and discuss ways in which we can prepare.

A Gap in Mission: The Disparate Missions of Medical Schools and Teaching Hospitals

Objective

The social contract mandates that in return for the government-funded labor of residents and fellows, the medical profession trains, to the best of its ability, the physicians of our future. This contract obligates graduate medical education (GME) to utilize all available information to create an optimal learning environment for its trainees. Business research has determined that a clearly defined mission statement is associated with improved employee engagement, retention, and wellness. Given that GME trainees are situated at the intersection of at least two institutions, each with its own separate mission, trainees could potentially be hindered by incongruent missions in the learning environment. The literature on mission statements has analyzed medical schools and hospitals separately; however, investigations comparing the statements of these affiliated institutions have not been conducted. Therefore, we plan to compare the content and assess the consistency of mission statements from affiliated medical schools and hospitals to determine if incongruencies exist.

Methods

In 2023, the mission statements from the Association of American Medical Colleges (AAMC) medical schools and affiliated teaching hospitals (n = 163) were aggregated from their public websites. The content of each mission statement was thematically analyzed to assess variation.

Results

According to content analysis of the mission statements from 163 AAMC medical school members and affiliated teaching hospitals, less than half of their top priorities are shared by their affiliated hospitals (45%). Additionally, themes of diversity, religion, and global care were found to be contrasting priorities between affiliated institutions.

Conclusion

Given the precedence within the business and the observed discrepancies in mission, further research is needed to determine whether collaborating medical schools and hospitals could provide a more favorable graduate training environment by uniting their priorities and identifying shared goals.

Current Situation and Demand for Continuing Medical Education (CME) for Obstetricians and Gynecologists

Objective

To explore the needs of obstetricians and gynecologists with different working years for the contents and forms of CME.

Methods

The online questionnaire was distributed on the largest academic training platform for obstetrics and gynecology in China from April 2020 to May 2020. The survey contents mainly included the training forms and training contents of CME, as well as the mastery of diseases by doctors with different working years.

Results

4458 questionnaires were returned, of which 3954 questionnaires were included in the study. There was a significant (p < 0.001) positive correlation between the mastery of 19 diseases by obstetricians and gynecologists with different working years. After adjusting for specialist departments, hospital grades, nature of units and ages, it was found that there was no statistically significant difference between the groups in gynecological endocrine-related diseases, such as precocious puberty/delayed puberty, and there was also no statistically significant difference between the groups of shorter working years in infertility, menopausal syndrome, hyperprolactinemia and premature ovarian failure.

Conclusion

The most popular training content for obstetricians and gynecologists with different working years is common gynecological endocrine diseases, of which abnormal uterine bleeding, menopausal syndrome, polycystic ovarian syndrome and cervical lesions are the most selected diseases, and the most popular form of training is online.