1
|
Versalle RL, Todd BR, Chen NW, Turner-Lawrence DE. Early Emergency Medicine Milestone Assessment for Predicting First-Year Resident Performance. MEDEDPORTAL : THE JOURNAL OF TEACHING AND LEARNING RESOURCES 2024; 20:11386. [PMID: 38476297 PMCID: PMC10928014 DOI: 10.15766/mep_2374-8265.11386] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/06/2023] [Accepted: 11/28/2023] [Indexed: 03/14/2024]
Abstract
Introduction The Accreditation Council for Graduate Medical Education (ACGME) requires emergency medicine (EM) residency training programs to monitor residents' progress using standardized milestones. The first assessment of PGY 1 resident milestones occurs midway through the first year and could miss initial deficiencies. Early assessment of PGY 1 EM resident milestones has potential to identify at-risk residents prior to standard midyear evaluations. We developed an orientation syllabus for PGY 1 residents followed by a milestone assessment. Assessment scores helped predict future milestone scores and American Board of Emergency Medicine (ABEM) In-Training Examination (ITE) scores for PGY 1 residents. Methods From 2013 to 2020, we developed and implemented Milestone Evaluation Day (MED), a simulation-based day and written exam assessing PGY 1 EM residents during their first month on the 23 ACGME 1.0 milestones. MED stations included a history and physical with verbal presentation, patient simulation, vascular access, wound management, and airway management. MED, Clinical Competency Committee-generated (CCC-generated) milestone, and ABEM ITE scores were averaged and compared utilizing Pearson's correlation coefficient. Results Of 112 PGY 1 EM residents, 110 (98%) were analyzed over an 8-year period. We observed a moderate positive correlation of MED and CCC-generated milestone scores (r = .34, p < .001). There was a nonstatistically significant weak positive correlation of MED and ABEM ITE scores (r = .13, p = .17). Discussion An early assessment of EM milestones in the PGY 1 year can assist in the prediction of CCC-generated milestone scores for PGY 1 residents.
Collapse
Affiliation(s)
- Rochelle L. Versalle
- Third-Year Resident, Department of Emergency Medicine, Oakland University William Beaumont School of Medicine
| | - Brett R. Todd
- Associate Professor, Department of Emergency Medicine, Oakland University William Beaumont School of Medicine
| | - Nai-Wei Chen
- Statistician, Division of Informatics and Biostatistics, Beaumont Institute
| | - Danielle E. Turner-Lawrence
- Associate Professor, Department of Emergency Medicine, Oakland University William Beaumont School of Medicine
| |
Collapse
|
2
|
Suliman S, Könings KD, Allen M, Al-Moslih A, Carr A, Koopmans RP. Sailing the boat together: Co-creation of a model for learning during transition. MEDICAL TEACHER 2023; 45:193-202. [PMID: 36044884 DOI: 10.1080/0142159x.2022.2118037] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
PURPOSE Medical students' transition to postgraduate training, given the complexity of new roles and responsibilities, requires the engagement of all involved stakeholders. This study aims to co-create a transition curriculum and determine the value of involving the key stakeholders throughout such transition in its design process. METHODS We conducted a mixed-methods study involving faculty/leaders (undergraduate/postgraduate), final-year medical students, and chief residents. It commenced with eight co-creation sessions (CCS), qualitative results of which were used to draft a quantitative survey sent to non-participants, followed by two consensus-building CCS with the original participants. We applied thematic analysis for transcripts of all CCS, and mean scores with standard deviations for survey analysis. RESULTS We identified five themes: adaptation, authenticity, autonomy, connectedness, and continuity, embedded in the foundation of a supportive environment, to constitute a Model of Learning during Transition (MOLT). Inclusion of various stakeholders and optimizing their representation brought rich perspectives to the design process. This was reinforced through active students' participation enabling a final consensus. CONCLUSIONS Bringing perspectives of key stakeholders in the transition spectrum enriches transition curricula. The proposed MOLT can provide a guide for curriculum designers to optimize the final year of undergraduate medical training in preparing students for postgraduate training with essential competencies to be trained.
Collapse
Affiliation(s)
- Shireen Suliman
- Medicine Department, Hamad Medical Corporation, Doha, Qatar
- College of Medicine, QU Health, Qatar University, Doha, Qatar
- School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| | - Karen D Könings
- School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| | - Margaret Allen
- Medical Education Department, Hamad Medical Corporation, Doha, Qatar
| | - Ayad Al-Moslih
- College of Medicine, QU Health, Qatar University, Doha, Qatar
| | - Alison Carr
- College of Medicine, QU Health, Qatar University, Doha, Qatar
| | - Richard P Koopmans
- Internal Medicine, Maastricht University Medical Center, Maastricht, The Netherlands
| |
Collapse
|
3
|
Ryan MS, Iobst W, Holmboe ES, Santen SA. Competency-based medical education across the continuum: How well aligned are medical school EPAs to residency milestones? MEDICAL TEACHER 2022; 44:510-518. [PMID: 34807793 DOI: 10.1080/0142159x.2021.2004303] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
INTRODUCTION Competency-based medical education (CBME) provides a framework for describing learner progression throughout training. However, specific approaches to CBME implementation vary widely across educational settings. Alignment between various methods used across the continuum is critical to support transitions and assess learner performance. The purpose of this study was to investigate alignment between CBME frameworks used in undergraduate medical education (UME) and graduate medical education (GME) settings using the US context as a model. METHOD The authors analyzed content from the core entrustable professional activities for entering residency (Core EPAs; UME model) and residency milestones (GME model). From that analysis, they performed a series of cross-walk activities to investigate alignment between frameworks. After independent review, authors discussed findings until consensus was reached. RESULTS Some alignment was found for activities associated with history taking, physical examination, differential diagnosis, patient safety, and interprofessional care; however, there were far more examples of misalignment. CONCLUSIONS These findings highlight challenges creating alignment of assessment frameworks across the continuum of training. The importance of these findings includes implications for assessment and persistence of the educational gap across UME and GME. The authors provide four next steps to improve upon the continuum of education.
Collapse
Affiliation(s)
- Michael S Ryan
- Department of Pediatrics, Virginia Commonwealth University School of Medicine, Richmond, VA, USA
| | - William Iobst
- Accreditation Council for Graduate Medical Education, Chicago, IL, USA
| | - Eric S Holmboe
- Accreditation Council for Graduate Medical Education, Chicago, IL, USA
| | - Sally A Santen
- Department of Emergency Medicine, Virginia Commonwealth University School of Medicine, Richmond, VA, USA
| |
Collapse
|
4
|
Schreyer KE, Healy ME, Repanshek Z, Satz WA, Ufberg JW. Implementation of a pilot novel objective peer comparison evaluation system in an emergency medicine residency program. AEM EDUCATION AND TRAINING 2022; 6:e10736. [PMID: 35434444 PMCID: PMC9005166 DOI: 10.1002/aet2.10736] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/07/2021] [Revised: 02/24/2022] [Accepted: 03/04/2022] [Indexed: 06/14/2023]
Abstract
OBJECTIVES Emergency medicine (EM) residents are currently evaluated via The Milestones, which have been shown to be imperfect and subjective. There is also a need for residents to achieve competency in patient safety and quality improvement processes, which can be accomplished through provision of peer comparison metrics. This pilot study aimed to evaluate the implementation of an objective peer comparison system for metrics that quantified aspects of quality and safety, efficiency and throughput, and utilization. METHODS This pilot study took place at an academic, tertiary care center with a 3-year residency and 14 residents per postgraduate year (PGY) class. Metrics were compared within each PGY class using Wilcoxon signed-rank and rank-order analyses. RESULTS Significant changes were seen in the majority of the metrics for all PGY classes. PGY3s accounted for the significant change in EKG and X-ray reads, while PGY1s and PGY2s accounted for the significant change in disposition to final note share. Physician evaluation to disposition decision was the only metric that did not reach significance in any class. CONCLUSIONS These preliminary data suggest that providing objective metrics is possible. Peer comparison metrics could provide an effective objective addition to the milestone evaluation system currently in use.
Collapse
Affiliation(s)
- Kraftin E. Schreyer
- Department of Emergency MedicineTemple University HospitalPhiladelphiaPennsylvaniaUSA
| | - Megan E. Healy
- Department of Emergency MedicineTemple University HospitalPhiladelphiaPennsylvaniaUSA
| | - Zachary Repanshek
- Department of Emergency MedicineTemple University HospitalPhiladelphiaPennsylvaniaUSA
| | - Wayne A. Satz
- Department of Emergency MedicineTemple University HospitalPhiladelphiaPennsylvaniaUSA
| | - Jacob W. Ufberg
- Department of Emergency MedicineTemple University HospitalPhiladelphiaPennsylvaniaUSA
| |
Collapse
|
5
|
Cooney RR, Murano T, Ring H, Starr R, Beeson MS, Edgar L. The Emergency Medicine Milestones 2.0: Setting the stage for 2025 and beyond. AEM EDUCATION AND TRAINING 2021; 5:e10640. [PMID: 34471793 PMCID: PMC8325436 DOI: 10.1002/aet2.10640] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/06/2021] [Accepted: 06/23/2021] [Indexed: 06/13/2023]
Abstract
INTRODUCTION Beginning in 1999, residents in emergency medicine have been expected to demonstrate competence in the six Accreditation Council on Graduate Medical Education (ACGME) Core Competencies. Expectations were further refined and clarified through the introduction of the Milestones in 2013. Emerging research and data from milestone reporting has illustrated the need for modification of the original milestones. Against this backdrop, the ACGME convened a committee to review and revise the original milestones. METHODS The working group was convened in December 2018 and consisted of representatives from the American Board of Emergency Medicine, American Osteopathic Association, Council of Residency Directors in Emergency Medicine, Association of American Medical Colleges, ACGME-Emergency Medicine Review Committee, three community members, a resident member, and a public member. This group also included members from both academic and community emergency medicine programs. The group was overseen by the ACGME vice president for milestones development and met in person one time followed by four virtual sessions to revise and draft the Emergency Medicine Milestones and Supplemental Guide as part of the ACGME Milestones 2.0 Project. RESULTS Using data from milestones reporting, needs assessment data, stakeholder interviews, and community commentary, the working group engaged in revisions and updates for the Emergency Medicine Milestones and created a supplemental guide to aid programs in the design of programmatic assessment for the milestones. CONCLUSION The Emergency Medicine Milestones 2.0 provide updated specialty-specific, competency-based behavioral anchors to guide the assessment of residents, the design of curricula, and the advancement of emergency medicine training programs.
Collapse
Affiliation(s)
| | | | - Hope Ring
- St. Mary Mercy HospitalLivoniaMichiganUSA
| | - Ryan Starr
- Cape Fear Valley Medical CenterFayettevilleNorth CarolinaUSA
| | | | | |
Collapse
|
6
|
Gottlieb M, Jordan J, Siegelman JN, Cooney R, Stehman C, Chan TM. Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature. AEM EDUCATION AND TRAINING 2021; 5:e10519. [PMID: 34041428 PMCID: PMC8138102 DOI: 10.1002/aet2.10519] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/03/2020] [Revised: 07/31/2020] [Accepted: 08/09/2020] [Indexed: 05/07/2023]
Abstract
OBJECTIVES Direct observation is important for assessing the competency of medical learners. Multiple tools have been described in other fields, although the degree of emergency medicine-specific literature is unclear. This review sought to summarize the current literature on direct observation tools in the emergency department (ED) setting. METHODS We searched PubMed, Scopus, CINAHL, the Cochrane Central Register of Clinical Trials, the Cochrane Database of Systematic Reviews, ERIC, PsycINFO, and Google Scholar from 2012 to 2020 for publications on direct observation tools in the ED setting. Data were dual extracted into a predefined worksheet, and quality analysis was performed using the Medical Education Research Study Quality Instrument. RESULTS We identified 38 publications, comprising 2,977 learners. Fifteen different tools were described. The most commonly assessed tools included the Milestones (nine studies), Observed Structured Clinical Exercises (seven studies), the McMaster Modular Assessment Program (six studies), Queen's Simulation Assessment Test (five studies), and the mini-Clinical Evaluation Exercise (four studies). Most of the studies were performed in a single institution, and there were limited validity or reliability assessments reported. CONCLUSIONS The number of publications on direct observation tools for the ED setting has markedly increased. However, there remains a need for stronger internal and external validity data.
Collapse
Affiliation(s)
- Michael Gottlieb
- Department of Emergency MedicineRush University Medical CenterChicagoILUSA
| | - Jaime Jordan
- Department of Emergency MedicineRonald Reagan UCLA Medical CenterLos AngelesCAUSA
| | | | - Robert Cooney
- Department of Emergency MedicineGeisinger Medical CenterDanvillePAUSA
| | | | - Teresa M. Chan
- Department of MedicineDivision of Emergency MedicineMcMaster UniversityHamiltonOntarioCanada
| |
Collapse
|
7
|
Schut S, Maggio LA, Heeneman S, van Tartwijk J, van der Vleuten C, Driessen E. Where the rubber meets the road - An integrative review of programmatic assessment in health care professions education. PERSPECTIVES ON MEDICAL EDUCATION 2021; 10:6-13. [PMID: 33085060 PMCID: PMC7809087 DOI: 10.1007/s40037-020-00625-w] [Citation(s) in RCA: 42] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/26/2020] [Revised: 09/21/2020] [Accepted: 09/29/2020] [Indexed: 05/12/2023]
Abstract
INTRODUCTION Programmatic assessment was introduced as an approach to design assessment programmes with the aim to simultaneously optimize the decision-making and learning function of assessment. An integrative review was conducted to review and synthesize results from studies investigating programmatic assessment in health care professions education in practice. METHODS The authors systematically searched PubMed, Web of Science, and ERIC to identify studies published since 2005 that reported empirical data on programmatic assessment. Characteristics of the included studies were extracted and synthesized, using descriptive statistics and thematic analysis. RESULTS Twenty-seven studies were included, which used quantitative methods (n = 10), qualitative methods (n = 12) or mixed methods (n = 5). Most studies were conducted in clinical settings (77.8%). Programmatic assessment was found to enable meaningful triangulation for robust decision-making and used as a catalyst for learning. However, several problems were identified, including overload in assessment information and the associated workload, counterproductive impact of using strict requirements and summative signals, lack of a shared understanding of the nature and purpose of programmatic assessment, and lack of supportive interpersonal relationships. Thematic analysis revealed that the success and challenges of programmatic assessment were best understood by the interplay between quantity and quality of assessment information, and the influence of social and personal aspects on assessment perceptions. CONCLUSION Although some of the evidence may seem compelling to support the effectiveness of programmatic assessment in practice, tensions will emerge when simultaneously stimulating the development of competencies and assessing its result. The identified factors and inferred strategies provide guidance for navigating these tensions.
Collapse
Affiliation(s)
- Suzanne Schut
- School of Health Professions Education, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands.
| | - Lauren A Maggio
- Department of Medicine, Uniformed Services, University of the Health Sciences, Bethesda, MD, USA
| | - Sylvia Heeneman
- School of Health Professions Education, Department of Pathology, Cardiovascular Research Institute Maastricht, Maastricht University, Maastricht, The Netherlands
| | - Jan van Tartwijk
- Department of Education, Utrecht University, Utrecht, The Netherlands
| | - Cees van der Vleuten
- School of Health Professions Education, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands
| | - Erik Driessen
- School of Health Professions Education, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
8
|
Cifuni M, Stoddard C, Witt S, Pfennig-Bass C, Pittman M. A Standardized Patient Experience: Elevating Interns to Expected Level of Clinical Competency. West J Emerg Med 2020; 22:37-40. [PMID: 33439801 PMCID: PMC7806322 DOI: 10.5811/westjem.2020.10.49092] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2020] [Accepted: 10/16/2020] [Indexed: 11/11/2022] Open
Abstract
Introduction Medical students transition to intern year with significant variability in prior clinical experience depending on their medical school education. This leads to notable differences in the interns’ ability to perform focused histories and physical exams, develop reasoned differentials, and maximize care plans. Providing a foundational experience for these essential skills will help to establish standardized expectations despite variable medical school experiences. Methods During an orientation block, interns participated in a standardized patient experience. Interns were presented with three common chief complaints: abdominal pain; chest pain; and headache. Faculty observed the three patient encounters and provided immediate verbal and written feedback to the interns based on a standardized grading rubric. Results All residents that participated “agreed” or “strongly agreed” that the experience was a meaningful educational experience. 90% of the interns reported the experience would change their clinical practice. Additionally, 75% of residents survyed one year after the experience felt the experience changed their clinical practice. Faculty felt the learning experience allowed them to address knowledge gaps early and provide early guidance where needed. Conclusion This article describes an emergency medicine residency program’s effort to provide a foundational experience for interns in evaluating emergency department patients. The intent was to “level the playing field” and establish “good habits” early in intern year with the realization that prior experiences vary significantly in July of intern year.
Collapse
Affiliation(s)
- Megan Cifuni
- Prisma Health Upstate, Department of Emergency Medicine, Greenville, South Carolina
| | - Caroline Stoddard
- Prisma Health Upstate, Department of Emergency Medicine, Greenville, South Carolina
| | - Scott Witt
- Prisma Health Upstate, Department of Emergency Medicine, Greenville, South Carolina
| | - Camiron Pfennig-Bass
- Prisma Health Upstate, Department of Emergency Medicine, Greenville, South Carolina
| | - Mark Pittman
- Prisma Health Upstate, Department of Emergency Medicine, Greenville, South Carolina
| |
Collapse
|
9
|
Chung AS, Bogoch S, Mody S, Smith C, Pushkar I, Drapkin J, Saloum D. Videotaped Unannounced Standardized Patient Encounters to Evaluate Interpersonal and Communication Skills in Emergency Medicine Residents. AEM EDUCATION AND TRAINING 2020; 4:419-422. [PMID: 33150286 PMCID: PMC7592816 DOI: 10.1002/aet2.10401] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/30/2019] [Revised: 10/02/2019] [Accepted: 10/14/2019] [Indexed: 06/11/2023]
Abstract
BACKGROUND Communication and interpersonal skills are important for effective patient care but are difficult to measure. Unannounced standardized patient encounters (USPEs) have the benefit of providing a standardized situation and provide a representation of usual care rather than best behavior, while also allowing for video recording without violating patient privacy. We conducted a feasibility pilot study to examine the use of videotaped USPEs in resident education of interpersonal and communication skills and specifically empathy. METHODS This was a pilot study conducted at an urban community teaching hospital. About 16 first-year emergency medicine engaged in four videotaped USPEs during their normal clinical shifts during 1 academic year. All visible recording equipment was concealed from the residents. The standardized patients completed two assessment forms after each encounter to measure empathy. RESULTS All 16 residents engaged in four videotaped USPEs for a total of 64 encounters. CONCLUSIONS We were able to successfully demonstrate the feasibility of using USPEs for assessment of resident interpersonal and communication skills.
Collapse
Affiliation(s)
- Arlene S. Chung
- Department of Emergency MedicineMaimonides Medical CenterBrooklynNY
| | - Sally Bogoch
- Department of Emergency MedicineMaimonides Medical CenterBrooklynNY
| | - Shivani Mody
- Department of Emergency MedicineMaimonides Medical CenterBrooklynNY
| | - Colleen Smith
- Department of Emergency MedicineIcahn School of Medicine at Mount SinaiNew YorkNY
| | | | | | - David Saloum
- Department of Emergency MedicineMaimonides Medical CenterBrooklynNY
| |
Collapse
|
10
|
Bell SG, Kobernik EK, Burk-Rafel J, Hughes DT, Schiller J, Heidemann LA, Morgan HK. Trainees' Perceptions of the Transition From Medical School to Residency. J Grad Med Educ 2020; 12:611-614. [PMID: 33149831 PMCID: PMC7594769 DOI: 10.4300/jgme-d-20-00183.1] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/03/2020] [Revised: 05/18/2020] [Accepted: 07/15/2020] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND There is emerging evidence that learners may be suboptimally prepared for the expectations of residency. In order to address these concerns, many medical schools are implementing residency preparation courses (RPCs). OBJECTIVE We aimed to determine trainees' perceptions of their transition to residency and whether they felt that they benefited from participation in an RPC. METHODS All residents and fellows at the University of Michigan (n = 1292) received an electronic survey in July 2018 that queried respondents on demographics, whether medical school had prepared them for intern year, and whether they had participated in an RPC. RESULTS The response rate was 44% (563 of 1292) with even distribution across gender and postgraduate years (PGYs). Most (78%, 439 of 563) felt that medical school prepared them well for intern year. There were no differences in reported preparedness for intern year across PGY, age, gender, or specialty. Overall, 28% (156 of 563) of respondents participated in an RPC and endorsed feeling prepared for intern year, which was more than RPC non-participants (85% [133 of 156] vs 70% [306 of 439], P = .029). Participation in longer RPCs was also associated with higher perceived preparedness for residency. CONCLUSIONS This study found that residents from multiple specialties reported greater preparedness for residency if they participated in a medical school fourth-year RPC, with greater perceptions of preparedness for longer duration RPCs, which may help to bridge the medical school to residency gap.
Collapse
|
11
|
Morgan HK, Mejicano GC, Skochelak S, Lomis K, Hawkins R, Tunkel AR, Nelson EA, Henderson D, Shelgikar AV, Santen SA. A Responsible Educational Handover: Improving Communication to Improve Learning. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2020; 95:194-199. [PMID: 31464734 DOI: 10.1097/acm.0000000000002915] [Citation(s) in RCA: 37] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/26/2023]
Abstract
An important tenet of competency-based medical education is that the educational continuum should be seamless. The transition from undergraduate medical education (UME) to graduate medical education (GME) is far from seamless, however. Current practices around this transition drive students to focus on appearing to be competitively prepared for residency. A communication at the completion of UME-an educational handover-would encourage students to focus on actually preparing for the care of patients. In April 2018, the American Medical Association's Accelerating Change in Medical Education consortium meeting included a debate and discussion on providing learner performance measures as part of a responsible educational handover from UME to GME. In this Perspective, the authors describe the resulting 5 recommendations for developing such a handover: (1) The purpose of the educational handover should be to provide medical school performance data to guide continued improvement in learner ability and performance, (2) the process used to create an educational handover should be philosophically and practically aligned with the learner's continuous quality improvement, (3) the educational handover should be learner driven with a focus on individualized learning plans that are coproduced by the learner and a coach or advisor, (4) the transfer of information within an educational handover should be done in a standardized format, and (5) together, medical schools and residency programs must invest in adequate infrastructure to support learner improvement. These recommendations are shared to encourage implementation of the educational handover and to generate a potential research agenda that can inform policy and best practices.
Collapse
Affiliation(s)
- Helen K Morgan
- H.K. Morgan is clinical associate professor of obstetrics and gynecology and learning health sciences, University of Michigan Medical School, Ann Arbor, Michigan. G.C. Mejicano is senior associate dean for education and professor of medicine, Oregon Health & Science University, Portland, Oregon. S. Skochelak is group vice president, Medical Education, American Medical Association, Chicago, Illinois. K. Lomis is vice president of undergraduate medical education innovations, American Medical Association, Chicago, Illinois. R. Hawkins is president and CEO, American Board of Medical Specialties, Chicago, Illinois. A.R. Tunkel is senior associate dean for medical education and professor of medicine and medical science, Warren Alpert Medical School of Brown University, Providence, Rhode Island. E.A. Nelson is associate dean of undergraduate medical education and distinguished teaching professor, Dell Medical School, University of Texas at Austin, Austin, Texas. D. Henderson is associate dean for student affairs, associate dean for multicultural and community affairs, and associate professor of family medicine, University of Connecticut School of Medicine, Farmington, Connecticut. A.V. Shelgikar is clinical associate professor of neurology, University of Michigan Medical School, Ann Arbor, Michigan. S.A. Santen is senior associate dean of assessment, evaluation, and scholarship and professor of emergency medicine, Virginia Commonwealth University School of Medicine, Richmond, Virginia
| | | | | | | | | | | | | | | | | | | |
Collapse
|
12
|
Santen SA, Yamazaki K, Holmboe ES, Yarris LM, Hamstra SJ. Comparison of Male and Female Resident Milestone Assessments During Emergency Medicine Residency Training: A National Study. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2020; 95:263-268. [PMID: 31517688 PMCID: PMC7004441 DOI: 10.1097/acm.0000000000002988] [Citation(s) in RCA: 38] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/09/2023]
Abstract
PURPOSE A previous study found that milestone ratings at the end of training were higher for male than for female residents in emergency medicine (EM). However, that study was restricted to a sample of 8 EM residency programs and used individual faculty ratings from milestone reporting forms that were designed for use by the program's Clinical Competency Committee (CCC). The objective of this study was to investigate whether similar results would be found when examining the entire national cohort of EM milestone ratings reported by programs after CCC consensus review. METHOD This study examined longitudinal milestone ratings for all EM residents (n = 1,363; 125 programs) reported to the Accreditation Council for Graduate Medical Education every 6 months from 2014 to 2017. A multilevel linear regression model was used to estimate differences in slope for all subcompetencies, and predicted marginal means between genders were compared at time of graduation. RESULTS There were small but statistically significant differences between males' and females' increase in ratings from initial rating to graduation on 6 of the 22 subcompetencies. Marginal mean comparisons at time of graduation demonstrated gender effects for 4 patient care subcompetencies. For these subcompetencies, males were rated as performing better than females; differences ranged from 0.048 to 0.074 milestone ratings. CONCLUSIONS In this national dataset of EM resident milestone assessments by CCCs, males and females were rated similarly at the end of their training for the majority of subcompetencies. Statistically significant but small absolute differences were noted in 4 patient care subcompetencies.
Collapse
Affiliation(s)
- Sally A. Santen
- S.A. Santen is professor and senior associate dean, Virginia Commonwealth University School of Medicine, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-8327-8002
| | - Kenji Yamazaki
- K. Yamazaki is senior analyst, Milestones Research and Evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois
| | - Eric S. Holmboe
- E.S. Holmboe is chief research, milestones development and evaluation officer, Accreditation Council for Graduate Medical Education, Chicago, Illinois; ORCID: https://orcid.org/0000-0003-0108-6021
| | - Lalena M. Yarris
- L.M. Yarris is professor and residency director, Department of Emergency Medicine, Oregon Health & Science University, Portland, Oregon
| | - Stanley J. Hamstra
- S.J. Hamstra is vice president, Department of Research, Milestone Development and Evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois, adjunct professor, Faculty of Education, University of Ottawa, Ottawa, Ontario, Canada, and adjunct professor, Department of Medical Education, Feinberg School of Medicine, Northwestern University, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-0680-366X
| |
Collapse
|
13
|
van der Vleuten CPM, Schuwirth LWT. Assessment in the context of problem-based learning. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2019; 24:903-914. [PMID: 31578642 PMCID: PMC6908559 DOI: 10.1007/s10459-019-09909-1] [Citation(s) in RCA: 25] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/03/2019] [Accepted: 08/07/2019] [Indexed: 05/29/2023]
Abstract
Arguably, constructive alignment has been the major challenge for assessment in the context of problem-based learning (PBL). PBL focuses on promoting abilities such as clinical reasoning, team skills and metacognition. PBL also aims to foster self-directed learning and deep learning as opposed to rote learning. This has incentivized researchers in assessment to find possible solutions. Originally, these solutions were sought in developing the right instruments to measure these PBL-related skills. The search for these instruments has been accelerated by the emergence of competency-based education. With competency-based education assessment moved away from purely standardized testing, relying more heavily on professional judgment of complex skills. Valuable lessons have been learned that are directly relevant for assessment in PBL. Later, solutions were sought in the development of new assessment strategies, initially again with individual instruments such as progress testing, but later through a more holistic approach to the assessment program as a whole. Programmatic assessment is such an integral approach to assessment. It focuses on optimizing learning through assessment, while at the same gathering rich information that can be used for rigorous decision-making about learner progression. Programmatic assessment comes very close to achieving the desired constructive alignment with PBL, but its wide adoption-just like PBL-will take many years ahead of us.
Collapse
Affiliation(s)
- Cees P M van der Vleuten
- School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands.
| | - Lambert W T Schuwirth
- Prideaux Centre for Research in Health Professions Education, College of Medicine and Public Health, Flinders University, Sturt Road, Bedford Park, SA, 5042, Australia
| |
Collapse
|
14
|
Heidemann LA, Keilin CA, Santen SA, Fitzgerald JT, Zaidi NL, Whitman L, Jones EK, Lypson ML, Morgan HK. Does Performance on Evidence-Based Medicine and Urgent Clinical Scenarios Assessments Deteriorate During the Fourth Year of Medical School? Findings From One Institution. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2019; 94:731-737. [PMID: 30640259 DOI: 10.1097/acm.0000000000002583] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
PURPOSE The fourth year of medical school (M4) should prepare students for residency yet remains generally unstructured, with ill-defined goals. The primary aim of this study was to determine whether there were performance changes in evidence-based medicine (EBM) and urgent clinical scenarios (UCS) assessments before and after M4 year. METHOD University of Michigan Medical School graduates who matched into internship at Michigan Medicine completed identical assessments on EBM and UCS at the beginning of M4 year and 13 months later during postgraduate year 1 (PGY1) orientation. Individual scores on these assessments were compared using paired t test analysis. The associations of academic performance, residency specialty classification, and initial performance on knowledge changes were analyzed. RESULTS During academic years 2014 and 2015, 76 students matched into a Michigan Medicine internship; 52 completed identical EBM stations and 53 completed UCS stations. Learners' performance on the EBM assessment decreased from M4 to PGY1 (mean 93% [SD = 7%] vs. mean 80% [SD = 13%], P < .01), while performance on UCS remained stable (mean 80% [SD = 9%] vs. mean 82% [SD = 8%], P = .22). High M4 performers experienced a greater rate of decline in knowledge level compared with low M4 performers for EBM (-20% vs. -4%, P = .01). Residency specialty and academic performance did not affect performance. CONCLUSIONS This study demonstrated degradation of performance in EBM during the fourth year and adds to the growing literature that highlights the need for curricular reform during this year.
Collapse
Affiliation(s)
- Lauren A Heidemann
- L.A. Heidemann is clinical assistant professor of internal medicine, University of Michigan Medical School, Ann Arbor, Michigan. C.A. Keilin is a medical student, University of Michigan Medical School, Ann Arbor, Michigan. S.A. Santen was assistant dean of evaluation and assessment and professor of emergency medicine and learning health sciences, University of Michigan Medical School, Ann Arbor, Michigan, at the time the study was conducted. She is currently senior associate dean, Evaluation, Assessment, and Scholarship of Learning, Virginia Commonwealth University School of Medicine, Richmond, Virginia. J.T. Fitzgerald is professor, Department of Learning Health Sciences, University of Michigan, and Geriatric Research Education and Clinical Center, Arbor VA Medical Center, Ann Arbor, Michigan. N.L. Zaidi is associate director of advancing scholarship, Office of Medical Student Education, University of Michigan Medical School, Ann Arbor, Michigan. L. Whitman is standardized patient program manager and educator, University of Michigan Medical School, Ann Arbor, Michigan. E.K. Jones is clinical assistant professor of family medicine, University of Michigan Medical School, Ann Arbor, Michigan. M.L. Lypson is director of medical and dental education, Office of Academic Affiliations, Department of Veterans Affairs, Washington, DC, and adjunct professor of internal medicine and learning health sciences, University of Michigan Medical School, Ann Arbor, Michigan. H.K. Morgan is director, Comprehensive Clinical Assessment, and clinical associate professor of obstetrics and gynecology and learning health sciences, University of Michigan Medical School, Ann Arbor, Michigan
| | | | | | | | | | | | | | | | | |
Collapse
|
15
|
O'Brien BC. What to Do About the Transition to Residency? Exploring Problems and Solutions From Three Perspectives. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2018; 93:681-684. [PMID: 29419551 DOI: 10.1097/acm.0000000000002150] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
Transitions are both a blessing and a curse for learning and professional development. While transitions can afford valuable opportunities for growth, they can also burden learners in ways that stymie performance and development. In this Invited Commentary, the author focuses on the transition from medical school to residency-a transition that many believe has tipped too far toward burden. The commentary explores three ways of problematizing the transition to residency. The first is as a transaction problem involving a complex exchange of information among programs and people. The second is as a transfer problem in which learners must apply knowledge in new contexts. The third is a trajectory problem concerning each learner's progression through medical education. The author describes current efforts aligned with each framing of the problem and discusses potential gaps in these efforts that might be addressed through an overarching framing of transitions in medical education as transformative processes. This framing may help the medical educational community think more holistically about the problems and solutions with transitions.
Collapse
Affiliation(s)
- Bridget C O'Brien
- B.C. O'Brien is associate professor, Department of Medicine, and educational researcher, Center for Faculty Educators, University of California, San Francisco, San Francisco, California; ORCID: http://orcid.org/0000-0001-9591-5243
| |
Collapse
|
16
|
Abstract
The aim of this section is to expand and accelerate advances in curriculum developments and in methods of teaching bioethics.
Collapse
|
17
|
Perry M, Linn A, Munzer BW, Hopson L, Amlong A, Cole M, Santen SA. Programmatic Assessment in Emergency Medicine: Implementation of Best Practices. J Grad Med Educ 2018; 10:84-90. [PMID: 29467979 PMCID: PMC5821020 DOI: 10.4300/jgme-d-17-00094.1] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/08/2017] [Revised: 07/17/2017] [Accepted: 10/31/2017] [Indexed: 01/20/2023] Open
Abstract
BACKGROUND Programmatic assessment is the intentional collection of key data from multiple sources for both assessment of learning and assessment for learning. OBJECTIVE We developed a system of programmatic assessment (PA) to identify competency progression (summative) and assessment for learning to assist residents in their formative development. METHODS The programmatic assessment was designed iteratively from 2014 through 2016. All assessments were first categorized by competency domain and source of assessment. The number of assessment modalities for each competency domain was collected. These multisource assessments were then mapped by program leadership to the milestones to develop a master PA blueprint. A resident learning management system provided the platform for aggregating formative and summative data, allowing residents and faculty ongoing access to guide learning and assessment. A key component of programmatic assessment was to support resident integration of assessment information through feedback by faculty after shifts and during monthly formal assessments, semiannual resident reviews, and summative judgments by the Clinical Competency Committee. RESULTS Through the PA, the 6 competency domains are assessed through multiple modalities: patient care (22 different assessments), professionalism (18), systems-based practice (17), interprofessional and communication skills (16), medical knowledge (11), and practice-based learning and improvement (6). Each assessment provides feedback to the resident in various formats. Our programmatic assessment has been utilized for more than 2 years with iterative improvements. CONCLUSIONS The implementation of programmatic assessment allowed our program to organize diverse, multisourced feedback to drive both formative and summative assessments.
Collapse
|
18
|
Barrie MG, Amick C, Mitzman J, Way DP, King AM. Bringing the Flipped Classroom to Day 1: A Novel Didactic Curriculum for Emergency Medicine Intern Orientation. West J Emerg Med 2017; 19:145-147. [PMID: 29383072 PMCID: PMC5785184 DOI: 10.5811/westjem.2017.11.35286] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2017] [Revised: 10/31/2017] [Accepted: 11/03/2017] [Indexed: 11/11/2022] Open
Abstract
Most emergency medicine (EM) residency programs provide an orientation program for their incoming interns, with the lecture being the most common education activity during this period. Our orientation program is designed to bridge the gap between undergraduate and graduate medical education by ensuring that all learners demonstrate competency on Level 1 Milestones, including medical knowledge (MK). To teach interns core medical knowledge in EM, we reformulated orientation using the flipped-classroom model by replacing lectures with small group, case-based discussions. Interns demonstrated improvement in medical knowledge through higher scores on a posttest. Evaluation survey results were also favorable for the flipped-classroom teaching format.
Collapse
Affiliation(s)
- Michael G Barrie
- The Ohio State University, Wexner Medical Center, Department of Emergency Medicine, Columbus, Ohio
| | - Christopher Amick
- The Ohio State University, Wexner Medical Center, Department of Emergency Medicine, Columbus, Ohio
| | - Jennifer Mitzman
- The Ohio State University, Wexner Medical Center, Department of Emergency Medicine, Columbus, Ohio
| | - David P Way
- The Ohio State University, Wexner Medical Center, Department of Emergency Medicine, Columbus, Ohio
| | - Andrew M King
- The Ohio State University, Wexner Medical Center, Department of Emergency Medicine, Columbus, Ohio
| |
Collapse
|
19
|
Davis R, Davis J, Berg K, Berg D, Morgan CJ, Russo S, Riesenberg LA. Patient Handoff Education: Are Medical Schools Catching Up? Am J Med Qual 2017; 33:140-146. [PMID: 28728430 DOI: 10.1177/1062860617719128] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Communication errors during shift-to-shift handoffs are a leading cause of preventable adverse events. Nevertheless, handoff skills are variably taught at medical schools. The authors administered questionnaires on handoffs to interns during orientation. Questions focused on medical school handoff education, experiences, and perceptions. The majority (546/718) reported having some form of education on handoffs during medical school, with 48% indicating this was 1 hour or less. Most respondents (98%) reported that they believe patients experience adverse events because of inadequate handoffs, and more than one third had witnessed a patient safety issue. Results show that medical school graduates are not receiving adequate handoff training. Yet graduates are expected to conduct safe patient handoffs at the start of residency. Given that ineffective handoffs pose a significant patient safety risk, medical school graduates should have a baseline competency in handoff skills. This will require medical schools to develop, implement, and study handoff education.
Collapse
Affiliation(s)
- Robyn Davis
- 1 The University of Alabama at Birmingham, AL
| | - Joshua Davis
- 2 Pennsylvania State College of Medicine, Hershey, PA
| | - Katherine Berg
- 3 Sidney Kimmel Medical College at Thomas Jefferson University, Philadelphia, PA
| | - Dale Berg
- 3 Sidney Kimmel Medical College at Thomas Jefferson University, Philadelphia, PA
| | | | - Stefani Russo
- 3 Sidney Kimmel Medical College at Thomas Jefferson University, Philadelphia, PA
| | | |
Collapse
|
20
|
Kinnear B, Bensman R, Held J, O'Toole J, Schauer D, Warm E. Critical Deficiency Ratings in Milestone Assessment: A Review and Case Study. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2017; 92:820-826. [PMID: 28557948 DOI: 10.1097/acm.0000000000001383] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
PURPOSE The Accreditation Council for Graduate Medical Education (ACGME) requires programs to report learner progress using specialty-specific milestones. It is unclear how milestones can best identify critical deficiencies (CDs) in trainee performance. Specialties developed milestones independently of one another; not every specialty included CDs within milestones ratings. This study examined the proportion of ACGME milestone sets that include CD ratings, and describes one residency program's experiences using CD ratings in assessment. METHOD The authors reviewed ACGME milestones for all 99 specialties in November 2015, determining which rating scales contained CDs. The authors also reviewed three years of data (July 2012-June 2015) from the University of Cincinnati Medical Center (UCMC) internal medicine residency assessment system based on observable practice activities mapped to ACGME milestones. Data were analyzed by postgraduate year, assessor type, rotation, academic year, and core competency. The Mantel-Haenszel chi-square test was used to test for changes over time. RESULTS Specialties demonstrated heterogeneity in accounting for CDs in ACGME milestones, with 22% (22/99) of specialties having no language describing CDs in milestones assessment. Thirty-three percent (63/189) of UCMC internal medicine residents received at least one CD rating, with CDs accounting for 0.18% (668/364,728) of all assessment ratings. The authors identified CDs across multiple core competencies and rotations. CONCLUSIONS Despite some specialties not accounting for CDs in milestone assessment, UCMC's experience demonstrates that a significant proportion of residents may be rated as having a CD during training. Identification of CDs may allow programs to develop remediation and improvement plans.
Collapse
Affiliation(s)
- Benjamin Kinnear
- B. Kinnear is assistant professor and residency assistant program director, Medicine-Pediatrics, Department of Internal Medicine, University of Cincinnati College of Medicine, and Department of Pediatrics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio. R. Bensman is clinical fellow, Department of Pediatrics, Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio. J. Held is assistant professor and residency associate program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio. J. O'Toole is associate professor and residency associate program director, Medicine-Pediatrics, Department of Internal Medicine, University of Cincinnati College of Medicine, and Department of Pediatrics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio. D. Schauer is associate professor and residency associate program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio. E. Warm is Richard W. Vilter Professor of Medicine and residency program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio
| | | | | | | | | | | |
Collapse
|
21
|
Tanksley AL, Farnan JM, Arora VM. A Solution to the Problem of Sustainability of Opioid Initiatives in Graduate Medical Education. J Grad Med Educ 2017; 9:133. [PMID: 28261411 PMCID: PMC5319615 DOI: 10.4300/jgme-d-16-00514.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/06/2022] Open
Affiliation(s)
- Audrey L Tanksley
- Assistant Professor of Medicine, University of Chicago Medical Center
| | - Jeanne M Farnan
- Associate Professor, Department of Medicine, University of Chicago Pritzker School of Medicine
| | - Vineet M Arora
- Associate Professor and Director of Graduate Medical Education Clinical Learning Environment Innovation, Department of Medicine, University of Chicago Pritzker School of Medicine
| |
Collapse
|
22
|
Fischer JP, House JB, Hopson LR, Perry MA, Theyyuni N, Wolff MS, Sozener CB, Santen SA. Incorporation of a Graduate Student Writer into a Productive Research Team. West J Emerg Med 2017; 18:84-85. [PMID: 28116014 PMCID: PMC5226770 DOI: 10.5811/westjem.2016.9.31253] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2016] [Revised: 07/25/2016] [Accepted: 09/15/2016] [Indexed: 11/11/2022] Open
Affiliation(s)
- Jonathan P Fischer
- Alumnus of University of Michigan, School of Public Health, Ann Arbor, Michigan; University of Chicago Pritzker School of Medicine, Chicago, Illinois
| | - Joseph B House
- University of Michigan, Department of Emergency Medicine, Ann Arbor, Michigan
| | - Laura R Hopson
- University of Michigan, Department of Emergency Medicine, Ann Arbor, Michigan
| | - Marcia A Perry
- University of Michigan, Department of Emergency Medicine, Ann Arbor, Michigan
| | - Nikhil Theyyuni
- University of Michigan, Department of Emergency Medicine, Ann Arbor, Michigan
| | - Margaret S Wolff
- University of Michigan, Department of Emergency Medicine, Ann Arbor, Michigan
| | - Cemal B Sozener
- University of Michigan, Department of Emergency Medicine, Ann Arbor, Michigan
| | - Sally A Santen
- University of Michigan, Department of Emergency Medicine, Ann Arbor, Michigan; University of Michigan, Department of Learning Health Sciences, Ann Arbor, Michigan
| |
Collapse
|
23
|
Goldman RH, Tuomala RE, Bengtson JM, Stagg AR. How Effective are New Milestones Assessments at Demonstrating Resident Growth? 1 Year of Data. JOURNAL OF SURGICAL EDUCATION 2017; 74:68-73. [PMID: 27395399 DOI: 10.1016/j.jsurg.2016.06.009] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/28/2016] [Revised: 05/09/2016] [Accepted: 06/07/2016] [Indexed: 06/06/2023]
Abstract
OBJECTIVE Assessment tools that accrue data for the Accreditation Council for Graduate Medical Education Milestones must evaluate residents across multiple dimensions, including medical knowledge, procedural skills, teaching, and professionalism. Our objectives were to: (1) develop an assessment tool to evaluate resident performance in accordance with the Milestones and (2) review trends in resident achievements during the inaugural year of Milestone implementation. DESIGN A novel venue and postgraduate year (PGY) specific assessment tool was built, tested, and implemented for both operating room and labor and delivery "venues." Resident development of competence and independence was captured over time. To account for variable rotation schedules, the year was divided into thirds and compared using two-tailed Fisher's exact test. SETTING Brigham and Women's and Massachusetts General Hospitals, Boston MA. PARTICIPANTS Faculty evaluators and obstetrics and gynecology residents. RESULTS A total of 822 assessments of 44 residents were completed between 9/2014 and 6/2015. The percentage of labor and delivery tasks completed "independently" increased monotonically across the start of all years: 8.4% for PGY-1, 60.3% for PGY-2, 73.7% for PGY-3, and 87.5% for PGY-4. Assessments of PGY-1 residents demonstrated a significant shift toward "with minimal supervision" and "independent" for the management of normal labor (p = 0.03). PGY-3 residents demonstrated an increase in "able to be primary surgeon" in the operating room, from 36% of the time in the first 2/3 of the year, to 62.3% in the last 1/3 (p < 0.01). CONCLUSION Assessment tools developed to assist with Milestone assignments capture the growth of residents over time and demonstrate quantifiable differences in achievements between PGY classes. These tools will allow for targeted teaching opportunities for both individual residents and residency programs.
Collapse
Affiliation(s)
- Randi H Goldman
- Department of Obstetrics, Gynecology and Reproductive Biology, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts.
| | - Ruth E Tuomala
- Department of Obstetrics, Gynecology and Reproductive Biology, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts; Vincent Department of Obstetrics and Gynecology, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
| | - Joan M Bengtson
- Department of Obstetrics, Gynecology and Reproductive Biology, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
| | - Amy R Stagg
- Vincent Department of Obstetrics and Gynecology, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
| |
Collapse
|
24
|
McGrath J, Barrie M, Way DP. Emergency Medicine Resident Orientation: How Training Programs Get Their Residents Started. West J Emerg Med 2016; 18:97-104. [PMID: 28116017 PMCID: PMC5226773 DOI: 10.5811/westjem.2016.10.31275] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2016] [Revised: 09/01/2016] [Accepted: 10/01/2016] [Indexed: 01/07/2023] Open
Abstract
Introduction The first formal orientation program for incoming emergency medicine (EM) residents was started in 1976. The last attempt to describe the nature of orientation programs was by Brillman in 1995. Now almost all residencies offer orientation to incoming residents, but little is known about the curricular content or structure of these programs. The purpose of this project was to describe the current composition and purpose of EM resident orientation programs in the United States. Methods In autumn of 2014, we surveyed all U.S. EM residency program directors (n=167). We adapted our survey instrument from one used by Brillman (1995). The survey was designed to assess the orientation program’s purpose, structure, content, and teaching methods. Results The survey return rate was 63% (105 of 167). Most respondents (77%) directed three-year residencies, and all but one program offered intern orientation. Orientations lasted an average of nine clinical (Std. Dev.=7.3) and 13 non-clinical days (Std. Dev.=9.3). The prototypical breakdown of program activities was 27% lectures, 23% clinical work, 16% skills training, 10% administrative activities, 9% socialization and 15% other activities. Most orientations included activities to promote socialization among interns (98%) and with other members of the department (91%). Many programs (87%) included special certification courses (ACLS, ATLS, PALS, NRP). Course content included the following: use of electronic medical records (90%), physician wellness (75%), and chief complaint-based lectures (72%). Procedural skill sessions covered ultrasound (94%), airway management (91%), vascular access (90%), wound management (77%), splinting (67%), and trauma skills (62%). Conclusion Compared to Brillman (1995), we found that more programs (99%) are offering formal orientation and allocating more time to them. Lectures remain the most common educational activity. We found increases in the use of skills labs and specialty certifications. We also observed increases in time dedicated to clinical work during orientation. Only a few programs reported engaging in baseline or milestone assessments, an activity that could offer significant benefits to the residency program.
Collapse
Affiliation(s)
- Jillian McGrath
- The Ohio State University College of Medicine, Department of Emergency Medicine, Columbus, Ohio
| | - Michael Barrie
- The Ohio State University College of Medicine, Department of Emergency Medicine, Columbus, Ohio
| | - David P Way
- The Ohio State University College of Medicine, Department of Emergency Medicine, Columbus, Ohio
| |
Collapse
|
25
|
Gee DW, Phitayakorn R, Khatri A, Butler K, Mullen JT, Petrusa ER. A Pilot Study to Gauge Effectiveness of Standardized Patient Scenarios in Assessing General Surgery Milestones. JOURNAL OF SURGICAL EDUCATION 2016; 73:e1-e8. [PMID: 27886969 DOI: 10.1016/j.jsurg.2016.08.012] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/13/2016] [Revised: 08/18/2016] [Accepted: 08/22/2016] [Indexed: 06/06/2023]
Abstract
PURPOSE Some General Surgery Milestones can be difficult to assess in traditional clinical settings and especially difficult to assess in junior residents. The purpose of this pilot study was to METHODS: A total of 9 categorical interns participated in a comprehensive, 4-module, SP scenario designed to evaluate and manage right upper quadrant pain. SP checklist scores (SP%) were converted to Milestone-equivalent scores for direct comparison (SP-C). Milestone scores were analyzed from 3 different sources: SP, faculty (FAC), and CCC. Interns completed course evaluations at the end of each session. Spearman's rho was used to determine correlations. Wilcoxon signed rank tests were used to test for differences between scores from different sources. RESULTS Individual intern Milestone scores from the 3 sources (SP-C, FAC, and CCC) did not correlate. All 7 mean Milestone scores from SPs were significantly higher than from FAC and CCC. FAC and CCC scores were statistically equivalent except for Systems-Based Practice 1 (SBP1) and Patient Care 3 (PC3) where CCC scores were significantly higher than FAC. Mean SP% scores for PC1 were significantly lower than for PROF1, MK1, MK2, and ICS1 (p < 0.05). Interns felt the modules were moderately to very useful. CONCLUSIONS Developing an SP scenario for Milestones evaluation is feasible. SPs, faculty observers, and CCC each use different data to provide a unique source of Milestone assessment. SP scenarios may be ideally suited to assess specific resident strengths and weaknesses and provide individualized feedback, thus augmenting traditional evaluations. Additional SP scenarios, assessing a broader range of skills and Milestones, are advisable for more reliable estimates of resident performance.
Collapse
Affiliation(s)
- Denise W Gee
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts.
| | - Roy Phitayakorn
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - Avni Khatri
- Laboratory of Computer Science, Massachusetts General Hospital, Boston, Massachusetts
| | - Kathryn Butler
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - John T Mullen
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - Emil R Petrusa
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| |
Collapse
|
26
|
Lawson L, Jung J, Franzen D, Hiller K. Clinical Assessment of Medical Students in Emergency Medicine Clerkships: A Survey of Current Practice. J Emerg Med 2016; 51:705-711. [PMID: 27614539 DOI: 10.1016/j.jemermed.2016.06.045] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2016] [Revised: 06/15/2016] [Accepted: 06/29/2016] [Indexed: 11/17/2022]
Abstract
BACKGROUND Assessment practices in emergency medicine (EM) clerkships have not been previously described. Clinical assessment frequently relies on global ratings of clinical performance, or "shift cards," although these tools have not been standardized or studied. OBJECTIVE We sought to characterize assessment practices in EM clerkships, with particular attention to shift cards. METHODS A survey regarding assessment practices was administered to a national sample of EM clerkship directors (CDs). Descriptive statistics were compiled and regression analyses were performed. RESULTS One hundred seventy-two CDs were contacted, and 100 (58%) agreed to participate. The most heavily weighted assessment methods in final grades were shift cards (66%) and written examinations (21-26%), but there was considerable variability in grading algorithms. EM faculty (100%) and senior residents (69%) were most commonly responsible for assessment, and assessors were often preassigned (71%). Forty-four percent of CDs reported immediate completion of shift cards, 27% within 1 to 2 days, and 20% within a week. Only 40% reported return rates >75%. Thirty percent of CDs do not permit students to review individual evaluations, and 54% of the remainder deidentify evaluations before student review. Eighty-six percent had never performed psychometric analysis on their assessment tools. Sixty-five percent of CDs were satisfied with their shift cards, but 90% supported the development of a national tool. CONCLUSION There is substantial variability in assessment practices between EM clerkships, raising concern regarding the comparability of grades between institutions. CDs rely on shift cards in grading despite the lack of evidence of validity and inconsistent process variables. Standardization of assessment practices may improve the assessment of EM students.
Collapse
Affiliation(s)
- Luan Lawson
- Department of Emergency Medicine, East Carolina University Brody School of Medicine, Greenville, North Carolina
| | - Julianna Jung
- Department of Emergency Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland
| | - Douglas Franzen
- Division of Emergency Medicine, University of Washington School of Medicine, Seattle, Washington
| | - Katherine Hiller
- Department of Emergency Medicine, University of Arizona College of Medicine, Tucson, Arizona
| |
Collapse
|
27
|
Quinn SM, Worrilow CC, Jayant DA, Bailey B, Eustice E, Kohlhepp J, Rogers R, Kane BG. Using Milestones as Evaluation Metrics During an Emergency Medicine Clerkship. J Emerg Med 2016; 51:426-431. [PMID: 27473442 DOI: 10.1016/j.jemermed.2016.06.014] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2016] [Revised: 05/18/2016] [Accepted: 06/04/2016] [Indexed: 11/19/2022]
Abstract
BACKGROUND The Accreditation Council for Graduate Medical Education's (ACGME) Milestones presumes graduating medical students will enter residency proficient at Milestone level 1 for 23 skills. The Next Accreditation System now includes Milestones for each postgraduate specialty, and it is unlikely that schools will document every emergency medicine (EM) applicant's EM-specific skills in their performance evaluation. OBJECTIVES The goals of this research were to determine if assessment of the Milestones was feasible during a medical student clerkship and examine the proportion of medical students performing at Milestone level 1. METHODS This study was conducted at a center with Liaison Committee on Medical Education-approved medical training and a 4-year EM residency. Using traditional clerkship, we studied the feasibility of an ACGME EM Milestones-based clerkship assessment. Data led to redesign of the clerkship and its evaluation process, including all level 1 anchor(s) to add "occasionally" (>60%), "usually" (>80%), and "always" (100%) on a Likert scale to on-shift assessment forms. RESULTS During the feasibility phase (2013-14), 75 students rotated though the clerkship; 55 evaluations were issued and 50 contained the Milestone summary. Eight deficiencies were noted in Milestone 12 and three in Milestone 14. After changes, 49 students rotated under the new evaluation rubric. Of 575 completed on-shift evaluations, 16 Milestone deficiencies were noted. Of 41 institutional evaluations issued, only one student had deficiencies noted, all of which pertained to patient care. All evaluations in this second cohort contained each student's Milestone proficiency. CONCLUSIONS Assessment of the Milestones is feasible. Communication of ACGME EM Milestone proficiency may identify students who require early observation or remediation. The majority of students meet the anchors for the Milestones, suggesting that clerkship assessment with the ACGME EM Milestones does not adequately differentiate students.
Collapse
Affiliation(s)
- Shawn M Quinn
- Department of Emergency Medicine, Lehigh Valley Health Network/University of South Florida Morsani College of Medicine, Allentown, Pennsylvania
| | - Charles C Worrilow
- Department of Emergency Medicine, Lehigh Valley Health Network/University of South Florida Morsani College of Medicine, Allentown, Pennsylvania
| | - Deepak A Jayant
- Department of Emergency Medicine, Lehigh Valley Health Network/University of South Florida Morsani College of Medicine, Allentown, Pennsylvania
| | - Blake Bailey
- Department of Emergency Medicine, Lehigh Valley Health Network/University of South Florida Morsani College of Medicine, Allentown, Pennsylvania
| | - Eric Eustice
- Department of Emergency Medicine, Lehigh Valley Health Network/University of South Florida Morsani College of Medicine, Allentown, Pennsylvania
| | - Jared Kohlhepp
- Department of Emergency Medicine, Lehigh Valley Health Network/University of South Florida Morsani College of Medicine, Allentown, Pennsylvania
| | - Ryan Rogers
- Department of Emergency Medicine, Lehigh Valley Health Network/University of South Florida Morsani College of Medicine, Allentown, Pennsylvania
| | - Bryan G Kane
- Department of Emergency Medicine, Lehigh Valley Health Network/University of South Florida Morsani College of Medicine, Allentown, Pennsylvania
| |
Collapse
|
28
|
Baker-Genaw K, Kokas MS, Ahsan SF, Darnley-Fisch D, Drake S, Goyal N, Inamdar K, Moutzouros V, Prabhakar D, Rolland L, Sangha R, Shreve M, Woodward A. Mapping Direct Observations From Objective Structured Clinical Examinations to the Milestones Across Specialties. J Grad Med Educ 2016; 8:429-34. [PMID: 27413450 PMCID: PMC4936865 DOI: 10.4300/jgme-d-15-00385.1] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND Little is known about residents' performance on the milestones at the institutional level. Our institution formed a work group to explore this using an institutional-level curriculum and residents' evaluation of the milestones. OBJECTIVE We assessed whether beginner-level milestones for interpersonal and communication skills (ICS) related to observable behaviors in ICS-focused objective structured clinical examinations (OSCEs) for postgraduate year (PGY) 1 residents across specialties. METHODS The work group compared ICS subcompetencies across 12 programs to identify common beginner-level physician-patient communication milestones. The selected ICS milestone sets were compared for common language with the ICS-OSCE assessment tool-the Kalamazoo Essential Elements of Communication Checklist-Adapted (KEECC-A). To assess whether OSCE scores related to ICS milestone scores, all PGY-1 residents from programs that were part of Next Accreditation System Phase 1 were identified; their OSCE scores from July 2013 to June 2014 and ICS subcompetency scores from December 2014 were compared. RESULTS The milestones for 10 specialties and the transitional year had at least 1 ICS subcompetency that related to physician-patient communication. The language of the ICS beginner-level milestones appears similar to behaviors outlined in the KEECC-A. All 60 residents with complete data received at least a beginner-level ICS subcompetency score and at least a satisfactory score on all 3 OSCEs. CONCLUSIONS The ICS-OSCE scores for PGY-1 residents appear to relate to beginner-level milestones for physician-patient communication across multiple specialties.
Collapse
Affiliation(s)
- Kimberly Baker-Genaw
- Corresponding author: Kimberly Baker-Genaw, MD, Henry Ford Hospital, Department of Medical Education, CFP-1, 2799 West Grand Boulevard, Detroit, MI 48202, 313.916.3829, fax 313.916.1394,
| | | | | | | | | | | | | | | | | | | | | | | | | |
Collapse
|
29
|
Bradley KE, Andolsek KM. A pilot study of orthopaedic resident self-assessment using a milestones' survey just prior to milestones implementation. INTERNATIONAL JOURNAL OF MEDICAL EDUCATION 2016; 7:11-8. [PMID: 26752012 PMCID: PMC4715902 DOI: 10.5116/ijme.5682.6dfd] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/26/2015] [Accepted: 12/29/2015] [Indexed: 05/17/2023]
Abstract
OBJECTIVE To pilot test if Orthopaedic Surgery residents could self-assess their performance using newly created milestones, as defined by the Accreditation Council on Graduate Medical Education. METHODS In June 2012, an email was sent to Program Directors and administrative coordinators of the 154 accredited Orthopaedic Surgery Programs, asking them to send their residents a link to an online survey. The survey was adapted from the Orthopaedic Surgery Milestone Project. Completed surveys were aggregated in an anonymous, confidential database. SAS 9.3 was used to perform the analyses. RESULTS Responses from 71 residents were analyzed. First and second year residents indicated through self-assessment that they had substantially achieved Level 1 and Level 2 milestones. Third year residents reported they had substantially achieved 30/41, and fourth year residents, all Level 3 milestones. Fifth year, graduating residents, reported they had substantially achieved 17 Level 4 milestones, and were extremely close on another 15. No milestone was rated at Level 5, the maximum possible. Earlier in training, Patient Care and Medical Knowledge milestones were rated lower than the milestones reflecting the other four competencies of Practice Based Learning and Improvement, Systems Based Practice, Professionalism, and Interpersonal Communication. The gap was closed by the fourth year. CONCLUSIONS Residents were able to successfully self-assess using the 41 Orthopaedic Surgery milestones. Respondents' rate improved proficiency over time. Graduating residents report they have substantially, or close to substantially, achieved all Level 4 milestones. Milestone self-assessment may be a useful tool as one component of a program's overall performance assessment strategy.
Collapse
Affiliation(s)
- Kendall E. Bradley
- Orthopaedic Surgery Residency Training Program, Duke University Hospital, Box 3000 DUMC, Durham, NC 27710, USA
| | - Kathryn M. Andolsek
- Community and Family Medicine, Duke University School of Medicine, Box 3648 DUMC, 201 Trent Drive, Durham, NC 27710, USA
| |
Collapse
|
30
|
Yarris LM, Juve AM, Coates WC, Fisher J, Heitz C, Shayne P, Farrell SE. Critical Appraisal of Emergency Medicine Education Research: The Best Publications of 2014. Acad Emerg Med 2015; 22:1327-36. [PMID: 26473396 DOI: 10.1111/acem.12801] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2015] [Accepted: 07/22/2015] [Indexed: 11/29/2022]
Abstract
OBJECTIVES The objective was to critically appraise and highlight rigorous education research study articles published in 2014 whose outcomes advance the science of emergency medicine (EM) education. METHODS A search of the English language literature in 2014 querying Education Resources Information Center (ERIC), PsychINFO, PubMed, and Scopus identified 243 EM-related articles using either quantitative (hypothesis-testing or observational investigations of educational interventions) or qualitative (exploring important phenomena in EM education) methods. Two reviewers independently screened all of the publications using previously established exclusion criteria. Six reviewers then independently scored the 25 selected publications using either a qualitative or a quantitative scoring system. Each scoring system consisted of nine criteria. Selected criteria were based on accepted educational review literature and chosen a priori. Both scoring systems use parallel scoring metrics and have been used previously within this annual review. RESULTS Twenty-five medical education research papers (22 quantitative, three qualitative) met the criteria for inclusion and were reviewed. Five quantitative and two qualitative studies were ranked most highly by the reviewers as exemplary and are summarized in this article. CONCLUSIONS This annual critical appraisal series highlights seven excellent EM education research studies, meeting a priori criteria and published in 2014. Methodologic strengths in the 2014 papers are noted, and current trends in medical education research in EM are discussed.
Collapse
Affiliation(s)
- Lalena M. Yarris
- Department of Emergency Medicine; Oregon Health & Science University; Portland OR
| | - Amy Miller Juve
- Department of Anesthesiology & Perioperative Medicine; Oregon Health & Science University; Portland OR
| | - Wendy C. Coates
- Department of Emergency Medicine and Los Angeles Biomedical Research Institute at Harbor-UCLA Medical Center; University of California; Los Angeles David Geffen School of Medicine; Los Angeles CA
| | - Jonathan Fisher
- Departments of Emergency Medicine; Maricopa Medical Center & University of Arizona COM-Phoenix Campus; Phoenix AZ
| | - Corey Heitz
- Department of Emergency Medicine; Virginia Tech Carilion School of Medicine; Roanoke VA
| | - Philip Shayne
- Department of Emergency Medicine; Emory University School of Medicine; Atlanta GA
| | - Susan E. Farrell
- Partners Healthcare International and Department of Emergency Medicine; Harvard Medical School; Boston MA
| |
Collapse
|
31
|
Dehon E, Jones J, Puskarich M, Sandifer JP, Sikes K. Use of Emergency Medicine Milestones as Items on End-of-Shift Evaluations Results in Overestimates of Residents' Proficiency Level. J Grad Med Educ 2015. [PMID: 26221433 PMCID: PMC4512788 DOI: 10.4300/jgme-d-14-00438.1] [Citation(s) in RCA: 21] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND The emergency medicine milestones were developed to provide more objective resident assessment than current methods. However, little is known about the best practices for applying the milestones in resident assessment. OBJECTIVE We examined the utility of end-of-shift evaluations (ESEs) constructed using the milestones in resident assessment. METHODS We developed 14 daily ESEs, each of which included 9 or 10 emergency medicine milestones. Postgraduate year (PGY)-1 and PGY-2 residents were assessed on milestone levels 1 through 3; PGY-3 and PGY-4 residents were assessed on levels 3 through 5. Each milestone was rated on a nominal scale (yes, no, or not applicable). The Clinical Competency Committee combined the ESE data with data from other assessments to determine each resident's proficiency level for the emergency medicine subcompetencies. We used descriptive statistics to summarize resident ESEs and milestone levels. We analyzed differences in ESE score across PGY levels using t tests and analyses of variance. RESULTS Faculty completed 763 ESEs on 33 residents with a range of 2 to 54 (median=22) ESEs per resident. Faculty rarely (8%, 372 of 4633) rated a resident as not achieving a milestone on the ESEs. Analyses of variance revealed that ESE scores on level 3 milestones did not differ significantly by PGY level. There was poor agreement between ESE scores and Clinical Competency Committee ratings. CONCLUSIONS The ESEs constructed using the milestones resulted in grade or milestone inflation. Our results do not support using milestones as a stand-alone assessment tool.
Collapse
Affiliation(s)
- Erin Dehon
- Corresponding author: Erin Dehon, PhD, Department of Emergency Medicine, University of Mississippi Medical Center, 2500 N State Street, Jackson, MS 39216, 504.710.5368,
| | | | | | | | | |
Collapse
|