Centralized assessment in graduate medical education: cents and sensibilities

An Interactive Mock Paging Curriculum to Prepare New Internal Medicine Interns for Inpatient Wards

INTRODUCTION

The July effect refers to an increase in adverse outcomes during periods of physician trainee turnover in teaching hospitals. We created an interactive resident-led curriculum to train new internal medicine interns for routine encounters on inpatient wards by role-playing through mock paging scenarios and focusing on practical information relevant to intern year.

METHODS

A formal assessment of the academic year 2018 intern boot camp curriculum revealed that interns preferred sessions that involved active learning strategies and covered common issues. In the first week of academic year 2019, interns participated in two 1-hour small-group sessions involving mock paging scenarios. Interns were divided into small groups with one facilitator who was a senior medicine resident. Within these groups, facilitators acted as the nurse and provided pages. Interns took turns answering these mock pages based on a sign-out of patients. The facilitator emphasized desired learner actions and teaching points using a provided guide.

RESULTS

Twenty interns participated in the curriculum. Interns rated the curriculum highly and felt that the sessions improved their knowledge, comfort, and skills in managing routine inpatient encounters. On a 2-week follow-up knowledge test to determine if they retained the information from the curriculum, interns scored an average of 85% (response rate: 60%, N = 12), indicating that they could apply the knowledge/skills learned to new scenarios.

DISCUSSION

This curriculum prepares medicine interns to manage common inpatient issues at the beginning of their residency. After completing the curriculum, interns reported increased confidence in handling these issues.

Making Sense of Trainee Performance: Entrustment Decision-Making in Internal Medicine Program Directors

Background: Competency-based assessment is an important but challenging aspect of residency education but determines trainees' progression towards the ultimate goal of graduation. Entrustment decision making has been proposed as a supplementary metric to assess trainee competence. This study explores the process by which Program Directors (PDs) make entrustment decisions in Internal Medicine (IM) training programs. Study Design: Purposive sampling was used to recruit PDs from ACGME-accredited IM training programs to participate in a semi-structured interview. We analyzed interviews using an iterative, grounded theory-based approach to allow identification of themes that define the process of trainee entrustment. Results: Sixteen PDs were interviewed. Qualitative analysis showed that PDs use a dynamic process to understand trainee entrustability and progression towards competence, including construction of assessment networks, comparing performance to expected trajectory of trainee competence development, and bidirectional filtering and weighing of assessment data. Conclusions: PDs serve as a central processor by which assessment data on trainees is filtered, weighted, and compared an expected trajectory, all to gain understanding of trainee performance. Assessment networks are crucial to understanding trainee competence. While expected trajectory is an important tool to determine how trainees are progressing, its continued use may inject bias into the assessment process and slow transition to true competency-based assessment.

Mitigating "Educational Groundhog Day" - The Role of Learner Handoffs Within Clinical Rotations: A Survey of Pediatric Educational Leaders

BACKGROUND

Medical students decry frequent changes in faculty supervision, leading to the experience of "educational groundhog day." The discontinuity in supervision, cursory relationships, and uncoordinated feedback impede students' skill acquisition and delay entrustment decisions. Whereas patient handoff bundles are common, little is known about similarly structured approaches to learner handoffs (LHs).

OBJECTIVE

To describe current LH procedures and practices within pediatric clerkships and subinternships and to gauge interest in a future LH bundle.

METHODS

Nine items included in the 2016 Council on Medical Student Education in Pediatrics annual member survey were analyzed using mixed-methods.

RESULTS

The response rates were 66% (101 of 152) and 40% (165 of 411) for institutions and individuals, respectively. After limiting data to complete responses to programs with traditional block rotations, 54% of individual respondents (76 of 141) identified as inpatient faculty and about a quarter endorsed providing LHs. Inpatient faculty most commonly supervise medical students for 5 to 7 days. Most endorsed needing 1 to 3 days to determine a student's baseline performance and 5 days or more to make entrustment decisions. Three-quarters of inpatient faculty endorsed interest in LHs, while fewer than 16% of course directors currently provide LH expectations. Four themes emerged: instrument features, stakeholder buy-in, impact, and utility.

CONCLUSIONS

Typical inpatient faculty service days approximate the time required for making entrustment decisions about clinical students. While most inpatient faculty desire a LH bundle for use within a clinical rotation, few institutions and faculty currently use LHs. LHs could accelerate entrustment decisions by allowing coordinated feedback that might hasten learner clinical-skill development.

The New Internal Medicine Subinternship Curriculum Guide: a Report from the Alliance for Academic Internal Medicine

The internal medicine (IM) subinternship has been a long-established clinical experience in the final phase of medical school deemed by key stakeholders as a crucial rotation to prepare senior medical students for internship. Medical education has changed greatly since the first national curriculum for this course was developed in 2002 by the Clerkship Directors in Internal Medicine (CDIM). Most notably, competency-based medical education (CBME) has become a fixture in graduate medical education and has gradually expanded into medical school curricula. Still, residency program directors and empirical studies have identified gaps and inconsistencies in knowledge and skills among new interns. Recognizing these gaps, the Association of Program Directors in Internal Medicine (APDIM) surveyed its members in 2010 and identified four core skills essential for intern readiness. The Association of American Medical Colleges (AAMC) also published 13 core entrustable professional activities (EPAs) for entering residency to be expected of all medical school graduates. Results from the APDIM survey along with the widespread adoption of CBME informed this redesign of the IM subinternship curriculum. The authors provide an overview of this new guide developed by the Alliance for Academic Internal Medicine (AAIM) Medical Student-to-Resident Interface Committee (MSRIC).

Use of an objective structured clinical exam (OSCE) for early identification of communication skills deficits in interns

INTRODUCTION

There is limited information about whether OSCE during GME orientation can identify trainee communication deficits before these become evident via clinical performance evaluations.

METHODS

Ninety-seven interns matriculating to eight residency programs in six specialties at four hospitals participated in a nine-station communication skills OSCE. Ratings were based on the "Kalamazoo, adapted" communication skills checklist. Possible association with intern performance evaluations was assessed by repeated-measures logistic regression and ROC curves were generated.

RESULTS

The mean OSCE score was 4.08 ± 0.27 with a range of 3.3-4.6. Baseline OSCE scores were associated with subsequent communication concerns recorded by faculty, based on 1591 evaluations. A 0.1-unit decrease in the OSCE communication score was associated with an 18% higher odds of being identified with a communication concern by faculty evaluation (odds ratio 1.18, 95% CI 1.01-1.36, p = 0.034). ROC curves did not demonstrate a "cut-off" score (AUC= 0.558). Non-faculty evaluators were 3-5 times more likely than faculty evaluators to identify communication deficits, based on 1900 evaluations.

CONCLUSIONS

Lower OSCE performance was associated with faculty communication concerns on performance evaluations; however, a "cut-off" score was not demonstrated that could identify trainees for potential early intervention. Multi-source evaluation also identified trainees with communication skills deficits.

Evaluation of a multiple-encounter in situ simulation for orientation of staff to a new paediatric emergency service: a single-group pretest/post-test study

AIM

To assess the utility of a multiple-encounter in-situ (MEIS) simulation as an orientation tool for multidisciplinary staff prior to opening a new paediatric emergency service.

METHODS

A single-group pretest/post-test study was conducted. During the MEIS simulation, multidisciplinary staff with participant or observer roles managed eight children (mannequins) who attended triage with their parent/guardians (clinical facilitators) for a range of emergency presentations (structured scenarios designed to represent the expected range of presentations plus test various clinical pathways/systems). Participants were debriefed to explore clinical, systems and crisis-resource management issues. Participants also completed a pre-intervention and post-intervention questionnaire comprising statements about role confidence and orientation adequacy. Pre-test and post-test results were analysed using t-test and Wilcoxon signed rank test.

RESULTS

Eighty-nine staff participated in the MEIS simulation, with the majority completing the pre-simulation and post-simulation questionnaire. There was a significant improvement in post-intervention versus pre-intervention Likert scores for role confidence and orientation adequacy (p=0.001 and <0.001, respectively); effect sizes suggested the greatest impact was on orientation adequacy. Nearly all scenarios resulted in significant increases in participants' confidence levels.

CONCLUSIONS

The MEIS simulation was of utility in orientation of staff, at least with respect to self-reported role confidence and orientation adequacy. Its effectiveness in practice or compared with other orientation techniques was not assessed, but it did identify several flaws in planned systems allowing remediation prior to opening.

Assessing Residents' Competency at Baseline: How Much Does the Medical School Matter?

BACKGROUND

Although there is some consensus about the competencies needed to enter residency, the actual skills of graduating medical students may not meet expectations. In addition, little is known about the association between undergraduate medical education and clinical performance at entry into and during residency.

OBJECTIVE

We explored the association between medical school of origin and clinical performance using a multi-station objective structured clinical examination for incoming residents at the University of Michigan Health System.

METHODS

Prior to assuming clinical duties, all first-year residents at the University of Michigan Health System participate in the Postgraduate Orientation Assessment (POA). This assesses competencies needed during the first months of residency. Performance data for 1795 residents were collected between 2002 and 2012. We estimated POA variance by medical school using linear mixed models.

RESULTS

Medical school predicted the following amounts of variance in performance-data gathering scores: 1.67% (95% confidence interval [CI] 0.36-2.93); assessment scores: 4.93% (95% CI 1.84-6.00); teamwork scores: 0.80% (95% CI 0.00-1.82); communication scores: 2.37% (95% CI 0.66-3.83); and overall POA scores: 4.19% (95% CI 1.59-5.35).

CONCLUSIONS

The results show that residents' medical school of origin is weakly associated with clinical competency, highlighting a potential source of variability in undergraduate medical education. The practical significance of these findings needs further evaluation.

Internal Medicine Residency Program Directors' Views of the Core Entrustable Professional Activities for Entering Residency: An Opportunity to Enhance Communication of Competency Along the Continuum

PURPOSE

To examine internal medicine (IM) residency program directors' (PDs') perspectives on the Core Entrustable Professional Activities for Entering Residency (Core EPAs)-introduced into undergraduate medical education to further competency-based assessment-and on communicating competency-based information during transitions.

METHOD

A spring 2015 Association of Program Directors in Internal Medicine survey asked PDs of U.S. IM residency programs for their perspectives on which Core EPAs new interns must or should possess on day 1, which are most essential, and which have the largest gap between expected and observed performance. Their views and preferences were also requested regarding communicating competency-based information at transitions from medical school to residency and residency to fellowship/employment.

RESULTS

The response rate was 57% (204/361 programs). The majority of PDs felt new interns must/should possess 12 of the 13 Core EPAs. PDs' rankings of Core EPAs by relative importance were more varied than their rankings by the largest gaps in performance. Although preferred timing varied, most PDs (82%) considered it important for medical schools to communicate Core EPA-based information to PDs; nearly three-quarters (71%) would prefer a checklist format. Many (60%) would be willing to provide competency-based evaluations to fellowship directors/employers. Most (> 80%) agreed that there should be a bidirectional communication mechanism for programs/employers to provide feedback on competency assessments.

CONCLUSIONS

The gaps identified in Core EPA performance may help guide medical schools' curricular and assessment tool design. Sharing competency-based information at transitions along the medical education continuum could help ensure production of competent, practice-ready physicians.

‘We might as well be speaking different languages’: an innovative interprofessional education tool to teach and assess communication skills critical to patient safety

Introduction

All practicing health professionals must be able to communicate effectively with their colleagues around the care of patients. Better communication between health professionals not only improves patient outcomes, but also cuts down on costly and unnecessary tests or healthcare services. At New York University (NYU), we have addressed the need for interprofessional education (IPE) by incorporating a set of interprofessional objective structured clinical examinations (OSCEs) cases into our performance-based assessment programme to expand the educational tools for interprofessional collaborative (IPC) practice, assessment and feedback.

Methods

We identified and operationalised IPC competencies to create an assessment tool for use in IPC clinical cases, delineating core domains and then identifying observable behaviours that represented the broader competencies. IPC cases (for use in OSCEs) were designed in a way that required medical students and residents to collaborate effectively with a health professional from another discipline (standardised registered nurse (RN)) in order to provide quality care to a (standardised) patient. Feedback from the standardised RN and the participants was content analysed and our own experience in implementing was described.

Results

This method demonstrates that IPC practice can effectively be incorporated into medical education training and assessment, at the undergraduate and graduate level. We found high internal consistency among items within each of the core IPC competency domains (Cronbach's α 0.80–0.85). Based on both standardised RN and faculty feedback, the cases were effective in discriminating among learners within and across undergraduate medical education (UME) and graduate medical education (GME) levels, and within learners, in identifying individual strengths and weaknesses. Learners found these cases to be realistic, challenging and stimulating.

Conclusions

OSCE-based IPC training is a feasible and useful methodology. Ultimately, IPC OSCE cases are training tools that provide learners with a safe environment to practice, receive feedback and develop the critical skills needed for our evolving healthcare system. The next steps are to expand the scope of IPE cases to include more team members, and team work to also incorporate faculty development to ensure that our teachers and role models are effective in providing feedback on IPC practice.

Residents' ability to interpret radiology images: development and improvement of an assessment tool

RATIONALE AND OBJECTIVES

Despite increasing radiology coverage, nonradiology residents continue to preliminarily interpret basic radiologic studies independently, yet their ability to do so accurately is not routinely assessed.

MATERIALS AND METHODS

An online test of basic radiologic image interpretation was developed through an iterative process. Educational objectives were established, then questions and images were gathered to create an assessment. The test was administered online to first-year interns (postgraduate year [PGY] 1) from 14 different specialties, as well as a sample of third- and fourth-year radiology residents (PGY3/R2 and PGY4/R3).

RESULTS

Over a 2-year period, 368 residents were assessed, including PGY1 (n = 349), PGY3/R2 (n = 14), and PGY4/R3 (n = 5) residents. Overall, the test discriminated effectively between interns (average score = 66%) and advanced residents (R2 = 86%, R3 = 89%; P < .05). Item analysis indicated discrimination indices ranging from -0.72 to 48.3 (mean = 3.12, median 0.58) for individual questions, including four questions with negative discrimination indices. After removal of the negatively indexed questions, the overall predictive value of the instrument persisted and discrimination indices increased for all but one of the remaining questions (range 0.027-70.8, mean 5.76, median 0.94).

CONCLUSIONS

Validation of an initial iteration of an assessment of basic image-interpretation skills led to revisions that improved the test. The results offer a specific test of radiologic reading skills with validation evidence for residents. More generally, results demonstrate a principled approach to test development.

Test of integrated professional skills: objective structured clinical examination/simulation hybrid assessment of obstetrics-gynecology residents' skill integration

BACKGROUND

Assessment of obstetrics-gynecology residents' ability to integrate clinical judgment, interpersonal skills, and technical ability in a uniform fashion is required to document achievement of benchmarks of competency. An observed structured clinical examination that incorporates simulation and bench models uses direct observation of performance to generate formative feedback and standardized evaluation.

METHODS

The Test of Integrated Professional Skills (TIPS) is a 5-station performance-based assessment that uses standardized patients and complex scenarios involving ultrasonography, procedural skills, and evidence-based medicine. Standardized patients and faculty rated residents by using behaviorally anchored checklists. Mean scores reflecting performance in TIPS were compared across competency domains and by developmental level (using analysis of variance) and then compared to standard faculty clinical evaluations (using Spearman ρ). Participating faculty and residents were also asked to evaluate the usefulness of the TIPS.

RESULTS

Twenty-four residents participated in the TIPS. Checklist items used to assess competency were sufficiently reliable, with Cronbach α estimates from 0.69 to 0.82. Performance improved with level of training, with wide variation in performance. Standard faculty evaluations did not correlate with TIPS performance. Several residents who were rated as average or above average by faculty performed poorly on the TIPS (> 1 SD below the mean). Both faculty and residents found the TIPS format useful, providing meaningful evaluation and opportunity for feedback.

CONCLUSIONS

A simulation-based observed structured clinical examination facilitates observation of a range of skills, including competencies that are difficult to observe and measure in a standardized way. Debriefing with faculty provides an important interface for identification of performance gaps and individualization of learning plans.

Making July safer: simulation-based mastery learning during intern boot camp

PURPOSE

Patient care quality worsens during academic year turnover. Incoming interns' uneven clinical skills likely contribute to this phenomenon, known as the "July effect." The aim of this study was to evaluate the impact of a simulation-based mastery learning (SBML) boot camp on internal medicine interns' clinical skills.

METHOD

This was a cohort study of an SBML intervention at Northwestern University. In 2011, 47 interns completed boot camp; 109 interns (from 2009 and 2010) who did not participate in boot camp were historical controls. Boot-camp-trained interns participated in three days of small-group teaching sessions, deliberate practice, and individualized feedback. Primary outcome measures were performance of boot-camp-trained interns and historical controls on five parts of a clinical skills examination (CSE). Assessments included recognition of physical examination findings (cardiac auscultation), performance of procedures (paracentesis and lumbar puncture), management of critically ill patients (intensive care unit skills), and communication with patients (code status discussion). Boot camp participants were required to meet or exceed a minimum passing standard (MPS) before beginning their internship.

RESULTS

Boot-camp-trained interns all eventually met or exceeded the MPS and performed significantly better than historical control interns on all skills (P < .01), even after controlling for age, gender, and USMLE Step 1 and 2 scores (P < .001). The authors detected no relationship between CSE scores and age, gender, prior experience, self-confidence, or USMLE Step 1 and 2 scores.

CONCLUSIONS

An SBML boot camp allows for individualized training, assessment, and documentation of competence before interns begin providing medical care.

Clinical skills assessment of procedural and advanced communication skills: performance expectations of residency program directors

BACKGROUND

High stakes medical licensing programs are planning to augment and adapt current examinations to be relevant for a two-decision point model for licensure: entry into supervised practice and entry into unsupervised practice. Therefore, identifying which skills should be assessed at each decision point is critical for informing examination development, and gathering input from residency program directors is important.

METHODS

Using data from previously developed surveys and expert panels, a web-delivered survey was distributed to 3,443 residency program directors. For each of the 28 procedural and 18 advanced communication skills, program directors were asked which clinical skills should be assessed, by whom, when, and how. Descriptive statistics were collected, and Intraclass Correlations (ICC) were conducted to determine consistency across different specialties.

RESULTS

Among 347 respondents, program directors reported that all advanced communication and some procedural tasks are important to assess. The following procedures were considered 'important' or 'extremely important' to assess: sterile technique (93.8%), advanced cardiovascular life support (ACLS) (91.1%), basic life support (BLS) (90.0%), interpretation of electrocardiogram (89.4%) and blood gas (88.7%). Program directors reported that most clinical skills should be assessed at the end of the first year of residency (or later) and not before graduation from medical school. A minority were considered important to assess prior to the start of residency training: demonstration of respectfulness (64%), sterile technique (67.2%), BLS (68.9%), ACLS (65.9%) and phlebotomy (63.5%).

DISCUSSION

Results from this study support that assessing procedural skills such as cardiac resuscitation, sterile technique, and phlebotomy would be amenable to assessment at the end of medical school, but most procedural and advanced communications skills would be amenable to assessment at the end of the first year of residency training or later.

CONCLUSIONS

Gathering data from residency program directors provides support for developing new assessment tools in high-stakes licensing examinations.

Review article: closing the research gap at the interface of learning and clinical practice

PURPOSE

The dialogue at the interface of education and clinical practice highlights areas of critical importance to the development of new approaches for educating anesthesiologists. The purpose of this article is to examine the literature on education and acquisition of competence in three areas relevant to the interface of learning and clinical practice, with the aim to suggest a research agenda that adds to the evidence on preparing physicians for independent practice. The three areas are: 1) transitions across the continuum of education; 2) the effect of reductions in hours of clinical training on competence; and 3) efforts to incorporate the competencies and CanMEDS roles into teaching and evaluation.

PRINCIPAL FINDINGS

Fifty-six articles relevant to one or more of the themes were identified in the review, including 21 studies of transitions (in, during, and after residency education), 19 studies on the effects of duty hour limits on residents' acquisition of competence, and 16 articles that assessed competency-based teaching and assessment in anesthesiology. Overall, the findings suggested a relative paucity of scientific evidence and a need for research and the development of new scientific theory. Studies generally treated one of the themes in isolation, while in actuality they interact to produce optimal as well as suboptimal learning situations, while medical education research often is limited by small samples, brief follow-up, and threats to validity. This suggests a "research gap" where editorials and commentaries have moved ahead of an evidence base for education. Promising areas for research include preparation for care deemed important by society, work to apply knowledge about the development of expertise in other disciplines to medicine, and ways to embed the competencies in teaching and evaluation more effectively.

CONCLUSION

Closing the research gap in medical education will require clear direction for future work. The starting point, at an institution or nationally, is dialogue within the specialty to achieve consensus on some of the most pressing questions.

Feasibility of implementing a standardized multisource feedback program in the graduate medical education environment

BACKGROUND

Multisource feedback (MSF) is emerging as a central assessment method for several medical education competencies. Planning and resource requirements for a successful implementation can be significant. Our goal is to examine barriers and challenges to a successful multisite MSF implementation, and identify the benefits of MSF as perceived by participants.

METHODS

We analyzed the 2007-2008 field trial implementation of the Assessment of Professional Behaviors, an MSF program of the National Board of Medical Examiners, conducted with 8 residency and fellowship programs at 4 institutions. We use a multimethod analysis that draws on quantitative process indicators and qualitative participant experience data. Process indicators include program attrition, completion of implementation milestones, number of participants at each site, number of MSF surveys assigned and completed, and adherence to an experimental rater training protocol. Qualitative data include communications with each program and semistructured interviews conducted with key field trial staff to elicit their experiences with implementation.

RESULTS

Several implementation challenges are identified, including communication gaps and difficulty scheduling implementation and training workshops. Participant interviews indicate several program changes that should enhance feasibility, including increasing communication and streamlining the training process.

CONCLUSIONS

Multisource feedback is a complex educational intervention that has the potential to provide users with a better understanding of performance expectations in the graduate medical education environment. Standardization of the implementation processes and tools should reduce the burden on program administrators and participants. Further study is warranted to broaden our understanding of the resource requirements for a successful MSF implementation and to show how outcomes change as MSF gains broader acceptance.

An interdisciplinary infection control education intervention: necessary but not sufficient

BACKGROUND

Patient care environments are struggling to eradicate health care associated infections and studies of undergraduate and graduate medical trainees have revealed significant gaps in their performance of proper hand hygiene and aseptic technique (HH/AT), suggesting the need for improved curriculum. High-reliability industries have provided a model using standardization of approach and interprofessional training, and both are particularly suited to the teaching and assessment of these life-saving skills. The Infection Control Education project is a grant-funded, multi-institutional pilot launched to improve the teaching and assessment of HH/AT in our community.

METHODS

An interprofessional team of leaders and educators from 2 local hospital systems and 3 health colleges developed a 9-component "ICE PACK," which includes a unanimously endorsed, detailed HH/AT checklist. This teaching and assessment module was delivered to nursing/medical student and postgraduate year 1 resident/nurse intern pairs. Retention of checklist skills was retested 2 to 5 months after participation in the module.

RESULTS

Learner pairs participating in the 2-hour module mastered the HH/AT checklist and rated the experience highly. Retention after several weeks was disappointingly low in 2 of the 3 participant groups.

CONCLUSIONS

A community-wide HH/AT checklist was developed and an ICE PACK of materials created that is portable, standardizes the teaching and assessment of HH/AT skills, and is designed for interprofessional pairs of learners. Retention of checklist steps was disappointing in most of participant groups. Multiple, simultaneous strategies for improving compliance with infection control mandates appear necessary.

Better data for teachers, better data for learners, better patient care: college-wide assessment at Michigan State University's College of Human Medicine

When our school organized the curriculum around a core set of medical student competencies in 2004, it was clear that more numerous and more varied student assessments were needed. To oversee a systematic approach to the assessment of medical student competencies, the Office of College-wide Assessment was established, led by the Associate Dean of College-wide Assessment. The mission of the Office is to 'facilitate the development of a seamless assessment system that drives a nimble, competency-based curriculum across the spectrum of our educational enterprise.' The Associate Dean coordinates educational initiatives, developing partnerships to solve common problems, and enhancing synergy within the College. The Office also works to establish data collection and feedback loops to guide rational intervention and continuous curricular improvement. Aside from feedback, implementing a systems approach to assessment provides a means for identifying performance gaps, promotes continuity from undergraduate medical education to practice, and offers a rationale for some assessments to be located outside of courses and clerkships. Assessment system design, data analysis, and feedback require leadership, a cooperative faculty team with medical education expertise, and institutional support. The guiding principle is 'Better Data for Teachers, Better Data for Learners, Better Patient Care.' Better data empowers faculty to become change agents, learners to create evidence-based improvement plans and increases accountability to our most important stakeholders, our patients.

Efficacy and efficiency of webcast orientations versus live resident orientations: results of a 2-year survey

BACKGROUND

Beginning a graduate medical education training program is associated with a steep learning curve for incoming residents.

OBJECTIVE

To compare the efficacy and efficiency of live versus webcast formats for Institutional Orientation.

METHODS

This 2-year non-blinded study, with a nonrandomized cohort, compares outcomes for trainees oriented Summer 2005 in a ''live-lecture'' format with trainees oriented Summer 2006 using a webcast format. Outcomes include posttest success, the time required, presentation quality and utility, and cost.

RESULTS

In 2005, 249 trainees attended the live orientation. Of the 211 who completed the posttest; 132 (63%) passed it within 3 attempts. Of the 241 trainees in 2006, 236 completed the posttest. Of these, 215 (91%) passed it within 3 attempts. Compared to the live-lecture cohort, the webcast cohort rated the posttest as more difficult. Despite performing better, significantly fewer trainees in the webcast cohort rated the posttest as "appropriate" (χ(2) =  5 28.57, df 5 1, P , .001). There were no significant differences between the 2 groups on their perceptions of quality and utility of the presentations. While the first year cost of the webcast exceeded that of live lectures, the amortized cost was nearly identical to the live-lecture costs.

DISCUSSION

As corroborated by resident comments, the web-based approach was more effective because it provided trainees flexibility regarding when to study, options on how to view the material, and opportunities to review it if needed for mastery. We plan to continue using the webcast strategy, revising the content as needed.