Evaluation, grading, and use of the RIME vocabulary on internal medicine clerkships: results of a national survey and comparison to other clinical clerkships

Frontline assessors' opinions about grading committees in a medicine clerkship

BACKGROUND

Collective decision-making by grading committees has been proposed as a strategy to improve the fairness and consistency of grading and summative assessment compared to individual evaluations. In the 2020-2021 academic year, Washington University School of Medicine in St. Louis (WUSM) instituted grading committees in the assessment of third-year medical students on core clerkships, including the Internal Medicine clerkship. We explored how frontline assessors perceive the role of grading committees in the Internal Medicine core clerkship at WUSM and sought to identify challenges that could be addressed in assessor development initiatives.

METHODS

We conducted four semi-structured focus group interviews with resident (n = 6) and faculty (n = 17) volunteers from inpatient and outpatient Internal Medicine clerkship rotations. Transcripts were analyzed using thematic analysis.

RESULTS

Participants felt that the transition to a grading committee had benefits and drawbacks for both assessors and students. Grading committees were thought to improve grading fairness and reduce pressure on assessors. However, some participants perceived a loss of responsibility in students' grading. Furthermore, assessors recognized persistent challenges in communicating students' performance via assessment forms and misunderstandings about the new grading process. Interviewees identified a need for more training in formal assessment; however, there was no universally preferred training modality.

CONCLUSIONS

Frontline assessors view the switch from individual graders to a grading committee as beneficial due to a perceived reduction of bias and improvement in grading fairness; however, they report ongoing challenges in the utilization of assessment tools and incomplete understanding of the grading and assessment process.

Competency-Based Medical Education in a Norm-Referenced World: A Root Cause Analysis of Challenges to the Competency-Based Paradigm in Medical School

Competency-based medical education (CBME) requires a criterion-referenced approach to assessment. However, despite best efforts to advance CBME, there remains an implicit, and at times, explicit, demand for norm-referencing, particularly at the junction of undergraduate medical education (UME) and graduate medical education (GME). In this manuscript, the authors perform a root cause analysis to determine the underlying reasons for continued norm-referencing in the context of the movement toward CBME. The root cause analysis consisted of 2 processes: (1) identification of potential causes and effects organized into a fishbone diagram and (2) identification of the 5 whys. The fishbone diagram identified 2 primary drivers: the false notion that measures such as grades are truly objective and the importance of different incentives for different key constituents. From these drivers, the importance of norm-referencing for residency selection was identified as a critical component. Exploration of the 5 whys further detailed the reasons for continuation of norm-referenced grading to facilitate selection, including the need for efficient screening in residency selection, dependence upon rank-order lists, perception that there is a best outcome to the match, lack of trust between residency programs and medical schools, and inadequate resources to support progression of trainees. Based on these findings, the authors argue that the implied purpose of assessment in UME is primarily stratification for residency selection. Because stratification requires comparison, a norm-referenced approach is needed. To advance CBME, the authors recommend reconsideration of the approach to assessment in UME to maintain the purpose of selection while also advancing the purpose of rendering a competency decision. Changing the approach will require a collaboration between national organizations, accrediting bodies, GME programs, UME programs, students, and patients/societies. Details are provided regarding the specific approaches required of each key constituent group.

Factors Influencing Effective Communication between Stakeholder Groups during DVM Work-Based Learning Program in Bangladesh

This study aimed to review the existing communication systems between the universities, placement providers, and students during the DVM final year work-based learning (WBL) program in Bangladesh. The intention was to identify what factors impact the effectiveness of the communication system and to explore ways to enhance communication to better support the program. A questionnaire was used to collect details about the WBL program and the communication systems from all universities in Bangladesh. The questionnaire was completed on paper at a meeting of the National Veterinary Dean Council and online with a member of each university's WBL coordination team. A summary of the current WBL programs in Bangladesh was produced. Focus group discussions were used to collect more detailed information about the communication systems and were held via Zoom with recent graduates (n = 16) and placement providers (n = 7). Effective means of communication between all stakeholders were identified as an initial letter, phone calls, and spot visits by teachers. However, the frequency of formal communication before and during placements was variable and the ways of providing feedback on the communication systems were insufficient. These issues sometimes undermined the student learning experience. Suggestions for improvements included increased resourcing, greater use of online communication systems, and a national committee to oversee WBL. Other ways to motivate placement providers included a better honorarium and continuing education courses. The results suggest that existing communication systems for veterinary WBL in Bangladesh are not completely satisfactory. Measures are needed to improve communication to optimize the student learning experience and capitalize on the many benefits of the WBL program for all stakeholders.

Hawks and Doves: Perceptions and Reality of Faculty Evaluations

OBJECTIVES

Internal medicine clerkship grades are important for residency selection, but inconsistencies between evaluator ratings threaten their ability to accurately represent student performance and perceived fairness. Clerkship grading committees are recommended as best practice, but the mechanisms by which they promote accuracy and fairness are not certain. The ability of a committee to reliably assess and account for grading stringency of individual evaluators has not been previously studied.

METHODS

This is a retrospective analysis of evaluations completed by faculty considered to be stringent, lenient, or neutral graders by members of a grading committee of a single medical college. Faculty evaluations were assessed for differences in ratings on individual skills and recommendations for final grade between perceived stringency categories. Logistic regression was used to determine if actual assigned ratings varied based on perceived faculty's grading stringency category.

RESULTS

"Easy graders" consistently had the highest probability of awarding an above-average rating, and "hard graders" consistently had the lowest probability of awarding an above-average rating, though this finding only reached statistical significance only for 2 of 8 questions on the evaluation form (P = .033 and P = .001). Odds ratios of assigning a higher final suggested grade followed the expected pattern (higher for "easy" and "neutral" compared to "hard," higher for "easy" compared to "neutral") but did not reach statistical significance.

CONCLUSIONS

Perceived differences in faculty grading stringency have basis in reality for clerkship evaluation elements. However, final grades recommended by faculty perceived as "stringent" or "lenient" did not differ. Perceptions of "hawks" and "doves" are not just lore but may not have implications for students' final grades. Continued research to describe the "hawk and dove effect" will be crucial to enable assessment of local grading variation and empower local educational leadership to correct, but not overcorrect, for this effect to maintain fairness in student evaluations.

Determining Grades in the Internal Medicine Clerkship: Results of a National Survey of Clerkship Directors

PURPOSE

Trust in and comparability of assessments are essential in clerkships in undergraduate medical education for many reasons, including ensuring competency in clinical skills and application of knowledge important for the transition to residency and throughout students' careers. The authors examined how assessments are used to determine internal medicine (IM) core clerkship grades across U.S. medical schools.

METHODS

A multisection web-based survey of core IM clerkship directors at 134 U.S. medical schools with membership in the Clerkship Directors in Internal Medicine was conducted in October through November 2018. The survey included a section on assessment practices to characterize current grading scales used, who determines students' final clerkship grades, the nature/type of summative assessments, and how assessments are weighted. Respondents were asked about perceptions of the influence of the National Board of Medical Examiners (NBME) Medicine Subject Examination (MSE) on students' priorities during the clerkship.

RESULTS

The response rate was 82.1% (110/134). There was considerable variability in the summative assessments and their weighting in determining final grades. The NBME MSE (91.8%), clinical performance (90.9%), professionalism (70.9%), and written notes (60.0%) were the most commonly used assessments. Clinical performance assessments and the NBME MSE accounted for the largest percentage of the total grade (on average 52.8% and 23.5%, respectively). Eighty-seven percent of respondents were concerned that students' focus on the NBME MSE performance detracted from patient care learning.

CONCLUSIONS

There was considerable variability in what IM clerkships assessed and how those assessments were translated into grades. The NBME MSE was a major contributor to the final grade despite concerns about the impact on patient care learning. These findings underscore the difficulty in comparing learners across institutions and serve to advance discussions for how to improve accuracy and comparability of grading in the clinical environment.

Are We Measuring What Matters? How Student and Clerkship Characteristics Influence Clinical Grading

PURPOSE

Given the growing emphasis placed on clerkship performance for residency selection, clinical evaluation and its grading implications are critically important; therefore, the authors conducted this study to determine which evaluation components best predict a clinical honors recommendation across 3 core clerkships.

METHOD

Student evaluation data were collected during academic years 2015-2017 from the third-year internal medicine (IM), pediatrics, and surgery clerkships at the University of Alabama at Birmingham School of Medicine. The authors used factor analysis to examine 12 evaluation components (12 items), and they applied multilevel logistic regression to correlate evaluation components with a clinical honors recommendation.

RESULTS

Of 3,947 completed evaluations, 1,508 (38%) recommended clinical honors. The top item that predicted a clinical honors recommendation was clinical reasoning skills for IM (odds ratio [OR] 2.8; 95% confidence interval [CI], 1.9 to 4.2; P < .001), presentation skills for surgery (OR 2.6; 95% CI, 1.6 to 4.2; P < .001), and knowledge application for pediatrics (OR 4.8; 95% CI, 2.8 to 8.2; P < .001). Students who spent more time with their evaluators were more likely to receive clinical honors (P < .001), and residents were more likely than faculty to recommend clinical honors (P < .001). Of the top 5 evaluation items associated with clinical honors, 4 composed a single factor for all clerkships: clinical reasoning, knowledge application, record keeping, and presentation skills.

CONCLUSIONS

The 4 characteristics that best predicted a clinical honors recommendation in all disciplines (clinical reasoning, knowledge application, record keeping, and presentation skills) correspond with traditional definitions of clinical competence. Structural components, such as contact time with evaluators, also correlated with a clinical honors recommendation. These findings provide empiric insight into the determination of clinical honors and the need for heightened attention to structural components of clerkships and increased scrutiny of evaluation rubrics.

Evaluating the Reliability and Validity Evidence of the RIME (Reporter-Interpreter-Manager-Educator) Framework for Summative Assessments Across Clerkships

PURPOSE

The ability of medical schools to accurately and reliably assess medical student clinical performance is paramount. The RIME (reporter-interpreter-manager-educator) schema was originally developed as a synthetic and intuitive assessment framework for internal medicine clerkships. Validity evidence of this framework has not been rigorously evaluated outside of internal medicine. This study examined factors contributing to variability in RIME assessment scores using generalizability theory and decision studies across multiple clerkships, thereby contributing to its internal structure validity evidence.

METHOD

Data were collected from RIME-based summative clerkship assessments during 2018-2019 at Virginia Commonwealth University. Generalizability theory was used to explore variance attributed to different facets through a series of unbalanced random-effects models by clerkship. For all analyses, decision (D-) studies were conducted to estimate the effects of increasing the number of assessments.

RESULTS

From 231 students, 6,915 observations were analyzed. Interpreter was the most common RIME designation (44.5%-46.8%) across all clerkships. Variability attributable to students ranged from 16.7% in neurology to 25.4% in surgery. D-studies showed the number of assessments needed to achieve an acceptable reliability (0.7) ranged from 7 in pediatrics and surgery to 11 in internal medicine and 12 in neurology. However, depending on the clerkship each student received between 3 and 8 assessments.

CONCLUSIONS

This study conducted generalizability- and D-studies to examine the internal structure validity evidence of RIME clinical performance assessments across clinical clerkships. Substantial proportion of variance in RIME assessment scores was attributable to the rater, with less attributed to the student. However, the proportion of variance attributed to the student was greater than what has been demonstrated in other generalizability studies of summative clinical assessments. Overall, these findings support the use of RIME as a framework for assessment across clerkships and demonstrate the number of assessments required to obtain sufficient reliability.

Clerkship Grading Committees: the Impact of Group Decision-Making for Clerkship Grading

BACKGROUND

Faculty and students debate the fairness and accuracy of medical student clerkship grades. Group decision-making is a potential strategy to improve grading.

OBJECTIVE

To explore how one school's grading committee members integrate assessment data to inform grade decisions and to identify the committees' benefits and challenges.

DESIGN

This qualitative study used semi-structured interviews with grading committee chairs and members conducted between November 2017 and March 2018.

PARTICIPANTS

Participants included the eight core clerkship directors, who chaired their grading committees. We randomly selected other committee members to invite, for a maximum of three interviews per clerkship.

APPROACH

Interviews were recorded, transcribed, and analyzed using inductive content analysis.

KEY RESULTS

We interviewed 17 committee members. Within and across specialties, committee members had distinct approaches to prioritizing and synthesizing assessment data. Participants expressed concerns about the quality of assessments, necessitating careful scrutiny of language, assessor identity, and other contextual factors. Committee members were concerned about how unconscious bias might impact assessors, but they felt minimally impacted at the committee level. When committee members knew students personally, they felt tension about how to use the information appropriately. Participants described high agreement within their committees; debate was more common when site directors reviewed students' files from other sites prior to meeting. Participants reported multiple committee benefits including faculty development and fulfillment, as well as improved grading consistency, fairness, and transparency. Groupthink and a passive approach to bias emerged as the two main threats to optimal group decision-making.

CONCLUSIONS

Grading committee members view their practices as advantageous over individual grading, but they feel limited in their ability to address grading fairness and accuracy. Recommendations and support may help committees broaden their scope to address these aspirations.

Using the Clerkship Shelf Exam Score as a Qualification for an Overall Clerkship Grade of Honors: A Valid Practice or Unfair to Students?

Most clerkships require medical students to pass the National Board of Medical Examiners (NBME) subject (shelf) exam to pass the clerkship. Many use the NBME's recommended honors cut score on the shelf exam to determine medical student eligibility for an overall clerkship grade of honors. This use of a conjunctive scoring model for determining honors is inconsistent with the logic behind the intended use of this model for making pass-fail determinations. Further, many clerkships use grading systems that employ both this conjunctive model for honors eligibility and a compensatory scoring model for determining the overall clerkship grade. For students who fall short of the shelf exam honors cut score, such a grading system effectively increases the weighting of shelf exam performance and decreases the clerkship's transparency about the weighting of performance on other assessments toward the clerkship composite score and overall grade. It may also lead to contradictory grading results in which student B has a higher composite score than student A, yet student B receives a lower overall grade. The author illustrates how to calculate a weight for shelf exam performance that would be fairer to students and would help create a more transparent weighting scheme for the grading system. The author recommends that clerkships restructure their grading systems so that shelf exam honors-eligibility cut scores are not used as conjunctive criteria for determining overall clerkship grades of honors. A reexamination of the NBME's practice of suggesting honors-eligibility cut scores for shelf exams is also recommended.

Clinical Performance Evaluations of Third-Year Medical Students and Association With Student and Evaluator Gender

PURPOSE

Clinical performance evaluations are major components of medical school clerkship grades. But are they sufficiently objective? This study aimed to determine whether student and evaluator gender is associated with assessment of overall clinical performance.

METHOD

This was a retrospective analysis of 4,272 core clerkship clinical performance evaluations by 829 evaluators of 155 third-year students, within the Alpert Medical School grading database for the 2013-2014 academic year. Overall clinical performance, assessed on a three-point scale (meets expectations, above expectations, exceptional), was extracted from each evaluation, as well as evaluator gender, age, training level, department, student gender and age, and length of observation time. Hierarchical ordinal regression modeling was conducted to account for clustering of evaluations.

RESULTS

Female students were more likely to receive a better grade than males (adjusted odds ratio [AOR] 1.30, 95% confidence interval [CI] 1.13-1.50), and female evaluators awarded lower grades than males (AOR 0.72, 95% CI 0.55-0.93), adjusting for department, observation time, and student and evaluator age. The interaction between student and evaluator gender was significant (P = .03), with female evaluators assigning higher grades to female students, while male evaluators' grading did not differ by student gender. Students who spent a short time with evaluators were also more likely to get a lower grade.

CONCLUSIONS

A one-year examination of all third-year clerkship clinical performance evaluations at a single institution revealed that male and female evaluators rated male and female students differently, even when accounting for other measured variables.

Referral for competency committee review for poor performance on the internal medicine clerkship is associated with poor performance in internship

PURPOSE

To determine how students who are referred to a competency committee for concern over performance, and ultimately judged not to require remediation, perform during internship.

METHODS

Uniformed Services University of the Health Sciences' students who graduated between 2007 and 2011 were included in this study. We compared the performance during internship of three groups: students who were referred to the internal medicine competency committee for review who met passing criterion, students who were reviewed by the internal medicine competency committee who were determined not to have passed the clerkship and were prescribed remediation, and students who were never reviewed by this competency committee. Program Director survey results and United States Medical Licensing Examination (USMLE) Step 3 examination results were used as the outcomes of interest.

RESULTS

The overall survey response rate for this 5-year cohort was 81% (689/853). 102 students were referred to this competency committee for review. 63/102 students were reviewed by this competency committee, given passing grades in the internal medicine clerkship, and were not required to do additional remediation. 39/102 students were given less than passing grades by this competency committee and required to perform additional clinical work in the department of medicine to remediate their performance. 751 students were never presented to this competency committee. Compared to students who were never presented for review, the group of reviewed students who did not require remediation was 5.6 times more likely to receive low internship survey ratings in the realm of professionalism, 8.6 times more likely to receive low ratings in the domain of medical expertise, and had a higher rate of USMLE Step 3 failure (9.4% vs. 2.8%). When comparing the reviewed group to students who were reviewed and also required remediation, the only significant difference between groups regarding professionalism ratings with 50% of the group requiring remediation garnering low ratings compared to 18% of the reviewed group.

CONCLUSIONS

Students who are referred to a committee for review following completion of their internal medicine clerkship are more likely to receive poor ratings in internship and fail USMLE Step 3 compared to students whose performance in the medicine clerkship does not trigger a committee review. These findings provide validity evidence for our competency committee review in that the students identified as requiring further clinical work had significantly higher rates of poor ratings in professionalism than students who were reviewed by the competency committee but not required to remediate. Additionally, students reviewed but not required to remediate were nonetheless at risk of low internship ratings, suggesting that these students might need some intervention prior to graduation.

Midclerkship feedback in the surgical clerkship: the "Professionalism, Reporting, Interpreting, Managing, Educating, and Procedural Skills" application utilizing learner self-assessment

BACKGROUND

The Liaison Committee on Medical Education requires midclerkship formative (low stakes) feedback to students regarding their clinical skills. Student self-assessment is not commonly incorporated into this evaluation. We sought to determine the feasibility of collecting and comparing student self-assessment with that of their preceptors using an iPad application. These student self-ratings and preceptor ratings are jointly created and reviewed as part of a face-to-face midclerkship feedback session.

METHODS

Using our iPad application for Professionalism, Reporting, Interpreting, Managing, Educating, and Procedural Skills ("PRIMES"), students answer 6 questions based on their self-assessment of performance at midclerkship. Each skill is rated on a 3-point scale (beginning, competent, and strong) with specific behavioral anchors. The faculty preceptors then complete the same PRIMES form during the face-to-face meeting. The application displays a comparison of the 2 sets of ratings, facilitating a discussion to determine individualized learning objectives for the second half of the clerkship.

RESULTS

A total of 209 student-preceptor pairs completed PRIMES ratings. On average, student-preceptor ratings were in agreement for 38% of the time. Agreement between students and preceptors was highest for Professionalism (70%) and lowest for Procedural Skills (22%). On average, 60% of student-preceptor ratings did not agree. Students rated themselves lower than preceptors 52% of the time, while only 8% of students rated themselves higher than their preceptors' ratings (this difference is significant at the P value <.05 level).

CONCLUSIONS

This study demonstrates the value of using the PRIMES framework to incorporate surgery clerkship students' self-assessment into formative face-to-face midclerkship feedback sessions with their preceptors with the goal to improve performance during the second half of the clerkship.

Grading Practices and Distributions Across Internal Medicine Clerkships

THEORY

Clerkship evaluation and grading practices vary widely between U.S. medical schools. Grade inflation continues to exist, and grade distribution is likely to be different among U.S. medical schools.

HYPOTHESES

Increasing the number of available grades curtails "grade inflation."

METHOD

A national survey of all Clerkship Directors in Internal Medicine members was administered in 2011. The authors assessed key aspects of grading.

RESULTS

Response rate was 76%. Among clerkship directors (CDs), 61% of respondents agreed that grade inflation existed in the internal medicine clerkship at their school, and 43% believed that it helped students obtain better residency positions. With respect to grading practices, 79% of CDs define specific behaviors needed to achieve each grade, and 36% specify an ideal grade distribution. In addition, 44% have a trained core faculty responsible for evaluating students, 35% describe formal grading meetings, and 39% use the Reporter-Interpreter-Manager-Educator (RIME) scheme. Grading scales were described as follows: 4% utilize a pass/fail system, 13% a 3-tier (e.g., Honors/Pass/Fail), 45% 4-tier, 35% 5-tier, and 4% 6+-tier system. There was a trend to higher grades with more tiers available.

CONCLUSIONS

Grade inflation continues in the internal medicine clerkship. Almost half of CDs feel that this practice assists students to obtain better residency positions. A minority of programs have a trained core faculty who are responsible for evaluation. About one third have formal grading meetings and use the RIME system; both have been associated with more robust and balanced grading practices. In particular, there is a wide variation between schools in the percentage of students who are awarded the highest grade, which has implications for residency applications. Downstream users of clinical clerkship grades must be fully aware of these variations in grading in order to appropriately judge medical student performance.

Regular Formal Evaluation Sessions are Effective as Frame-of-Reference Training for Faculty Evaluators of Clerkship Medical Students

BACKGROUND

Face-to-face formal evaluation sessions between clerkship directors and faculty can facilitate the collection of trainee performance data and provide frame-of-reference training for faculty.

OBJECTIVE

We hypothesized that ambulatory faculty who attended evaluation sessions at least once in an academic year (attendees) would use the Reporter-Interpreter-Manager/Educator (RIME) terminology more appropriately than faculty who did not attend evaluation sessions (non-attendees).

DESIGN

Investigators conducted a retrospective cohort study using the narrative assessments of ambulatory internal medicine clerkship students during the 2008-2009 academic year.

PARTICIPANTS

The study included assessments of 49 clerkship medical students, which comprised 293 individual teacher narratives.

MAIN MEASURES

Single-teacher written and transcribed verbal comments about student performance were masked and reviewed by a panel of experts who, by consensus, (1) determined whether RIME was used, (2) counted the number of RIME utterances, and (3) assigned a grade based on the comments. Analysis included descriptive statistics and Pearson correlation coefficients.

KEY RESULTS

The authors reviewed 293 individual teacher narratives regarding the performance of 49 students. Attendees explicitly used RIME more frequently than non-attendees (69.8 vs. 40.4 %; p < 0.0001). Grades recommended by attendees correlated more strongly with grades assigned by experts than grades recommended by non-attendees (r = 0.72; 95 % CI (0.65, 0.78) vs. 0.47; 95 % CI (0.26, 0.64); p = 0.005). Grade recommendations from individual attendees and non-attendees each correlated significantly with overall student clerkship clinical performance [r = 0.63; 95 % CI (0.54, 0.71) vs. 0.52 (0.36, 0.66), respectively], although the difference between the groups was not statistically significant (p = 0.21).

CONCLUSIONS

On an ambulatory clerkship, teachers who attended evaluation sessions used RIME terminology more frequently and provided more accurate grade recommendations than teachers who did not attend. Formal evaluation sessions may provide frame-of-reference training for the RIME framework, a method that improves the validity and reliability of workplace assessment.

The IDEA Assessment Tool: Assessing the Reporting, Diagnostic Reasoning, and Decision-Making Skills Demonstrated in Medical Students' Hospital Admission Notes

Construct: Clinical skills are used in the care of patients, including reporting, diagnostic reasoning, and decision-making skills. Written comprehensive new patient admission notes (H&Ps) are a ubiquitous part of student education but are underutilized in the assessment of clinical skills. The interpretive summary, differential diagnosis, explanation of reasoning, and alternatives (IDEA) assessment tool was developed to assess students' clinical skills using written comprehensive new patient admission notes.

BACKGROUND

The validity evidence for assessment of clinical skills using clinical documentation following authentic patient encounters has not been well documented. Diagnostic justification tools and postencounter notes are described in the literature (1,2) but are based on standardized patient encounters. To our knowledge, the IDEA assessment tool is the first published tool that uses medical students' H&Ps to rate students' clinical skills.

APPROACH

The IDEA assessment tool is a 15-item instrument that asks evaluators to rate students' reporting, diagnostic reasoning, and decision-making skills based on medical students' new patient admission notes. This study presents validity evidence in support of the IDEA assessment tool using Messick's unified framework, including content (theoretical framework), response process (interrater reliability), internal structure (factor analysis and internal-consistency reliability), and relationship to other variables.

RESULTS

Validity evidence is based on results from four studies conducted between 2010 and 2013. First, the factor analysis (2010, n = 216) yielded a three-factor solution, measuring patient story, IDEA, and completeness, with reliabilities of .79, .88, and .79, respectively. Second, an initial interrater reliability study (2010) involving two raters demonstrated fair to moderate consensus (κ = .21-.56, ρ =.42-.79). Third, a second interrater reliability study (2011) with 22 trained raters also demonstrated fair to moderate agreement (intraclass correlations [ICCs] = .29-.67). There was moderate reliability for all three skill domains, including reporting skills (ICC = .53), diagnostic reasoning skills (ICC = .64), and decision-making skills (ICC = .63). Fourth, there was a significant correlation between IDEA rating scores (2010-2013) and final Internal Medicine clerkship grades (r = .24), 95% confidence interval (CI) [.15, .33].

CONCLUSIONS

The IDEA assessment tool is a novel tool with validity evidence to support its use in the assessment of students' reporting, diagnostic reasoning, and decision-making skills. The moderate reliability achieved supports formative or lower stakes summative uses rather than high-stakes summative judgments.

Narrative descriptions should replace grades and numerical ratings for clinical performance in medical education in the United States

Background: In medical education, evaluation of clinical performance is based almost universally on rating scales for defined aspects of performance and scores on examinations and checklists. Unfortunately, scores and grades do not capture progress and competence among learners in the complex tasks and roles required to practice medicine. While the literature suggests serious problems with the validity and reliability of ratings of clinical performance based on numerical scores, the critical issue is not that judgments about what is observed vary from rater to rater but that these judgments are lost when translated into numbers on a scale. As the Next Accreditation System of the Accreditation Council on Graduate Medical Education (ACGME) takes effect, medical educators have an opportunity to create new processes of evaluation to document and facilitate progress of medical learners in the required areas of competence.

Proposal and initial experience: Narrative descriptions of learner performance in the clinical environment, gathered using a framework for observation that builds a shared understanding of competence among the faculty, promise to provide meaningful qualitative data closely linked to the work of physicians. With descriptions grouped in categories and matched to milestones, core faculty can place each learner along the milestones' continua of progress. This provides the foundation for meaningful feedback to facilitate the progress of each learner as well as documentation of progress toward competence.

Implications: This narrative evaluation system addresses educational needs as well as the goals of the Next Accreditation System for explicitly documented progress. Educators at other levels of education and in other professions experience similar needs for authentic assessment and, with meaningful frameworks that describe roles and tasks, may also find useful a system built on descriptions of learner performance in actual work settings.

Conclusions: We must place medical learning and assessment in the contexts and domains in which learners do clinical work. The approach proposed here for gathering qualitative performance data in different contexts and domains is one step along the road to moving learners toward competence and mastery.

Observer-Reporter-Interpreter-Manager-Educator (ORIME) Framework to Guide Formative Assessment of Medical Students

The Observer-Reporter-Interpreter-Manager-Educator (ORIME) is adapted from RIME, an intuitive, self-explanatory and “synthetic” framework that assesses formatively, a student’s ability to synthesise knowledge, skills and attitude during a clinical encounter with a patient. The “O” refers to a student’s ability to pay attention and perceive with open-mindedness, people and events around him or her. The framework is suitable for definition of interim outcomes in a 5-year undergraduate programme. To align students’ and clinical teachers’ expectations further, selection of case complexity that is commensurate with student’s seniority and competence should be guided and an adapted version of the Minnesota Complexity Assessment Tool is proposed. Key words: Case Complexity, Clinical Teaching, Undergraduate Medical Education

Frameworks for learner assessment in medicine: AMEE Guide No. 78

In any evaluation system of medical trainees there is an underlying set of assumptions about what is to be evaluated (i.e., which goals reflect the values of the system or institution), what kind of observations or assessments are useful to allow judgments 1 ; and how these are to be analyzed and compared to a standard of what is to be achieved by the learner. These assumptions can be conventionalized into a framework for evaluation. Frameworks encompass, or "frame," a group of ideas or categories to reflect the educational goals against which a trainee's level of competence or progress is gauged. Different frameworks provide different ways of looking at the practice of medicine and have different purposes. In the first place, frameworks should enable educators to determine to what extent trainees are ready for advancement, that is, whether the desired competence has been attained. They should provide both a valid mental model of competence and also terms to describe successful performance, either at the end of training or as milestones during the curriculum. Consequently, such frameworks drive learning by providing learners with a guide for what is expected. Frameworks should also enhance consistency and reliability of ratings across staff and settings. Finally, they determine the content of, and resources needed for, rater training to achieve consistency of use. This is especially important in clinical rotations, in which reliable assessments have been most difficult to achieve. Because the limitations of workplace-based assessment have persisted despite the use of traditional frameworks (such as those based on knowledge, skills, and attitudes), this Guide will explore the assumptions and characteristics of traditional and newer frameworks. In this AMEE Guide, we make a distinction between analytic, synthetic, and developmental frameworks. Analytic frameworks deconstruct competence into individual pieces, to evaluate each separately. Synthetic frameworks attempt to view competence holistically, focusing evaluation on the performance in real-world activities. Developmental frameworks focus on stages of, or milestones, in the progression toward competence. Most frameworks have one predominant perspective; some have a hybrid nature.

The required written history and physical is alive, but not entirely well, in internal medicine clerkships

BACKGROUND

Effective written communication is a core competency for medical students, but it is unclear whether or how this skill is evaluated in clinical clerkships.

PURPOSE

This study identifies current requirements and practices regarding required written work during internal medicine clerkships.

METHODS

In 2010, Clerkship Directors of Internal Medicine (CDIM) surveyed its institutional members; one section asked questions about students' written work. RESULTS were compared to similar, unpublished CDIM 2001 survey questions.

RESULTS

Requirements for student-written work were nearly universal (96% in 2001 and 100% in 2010). Only 23% used structured evaluation forms and 16% reported written work was weighted as a percentage of the final grade, although 72% of respondents reported that written work was "factored" into global ratings.

CONCLUSIONS

Despite near universal requirements for student written work, structured evaluation was not commonly performed, raising concern about the validity of factoring these assessments into grades.

Variation and imprecision of clerkship grading in U.S. medical schools

PURPOSE

Despite standardized curricula and mandated accreditation, concern exists regarding the variability and imprecision of medical student evaluation. The authors set out to perform a complete review of clerkship evaluation in U.S. medical schools.

METHOD

Clerkship evaluation data were obtained from all Association of American Medical Colleges-affiliated medical schools reporting enrollment during 2009-2010. Deidentified reports were analyzed to define the grading system and the percentage of each class within each grading tier. Inter- and intraschool grading variation was assessed in part by comparing the proportion of students receiving the top grade.

RESULTS

Data were analyzed from 119 of 123 accredited medical schools. Dramatic variation was detected. Specifically, the authors documented eight different grading systems using 27 unique sets of descriptive terminology. Imprecision of grading was apparent. Institutions frequently used the same wording (e.g., "honors") to imply different meanings. The percentage of students awarded the top grade in any clerkship exhibited extreme variability (range 2%-93%) from school to school, as well as from clerkship to clerkship within the same school (range 18%-81%). Ninety-seven percent of all U.S. clerkship students were awarded one of the top three grades regardless of the number of grading tiers. Nationally, less than 1% of students failed any required clerkship.

CONCLUSIONS

There exists great heterogeneity of grading systems and imprecision of grade meaning throughout the U.S. medical education system. Systematic changes seeking to increase consistency, transparency, and reliability of grade meaning are needed to improve the student evaluation process at the national level.

How and why internal medicine clerkship directors use locally developed, faculty-written examinations: results of a national survey

PURPOSE

To describe how and why internal medicine clerkship directors (CDs) use locally developed, faculty-written (LFW) examinations.

METHOD

In 2009, the Clerkship Directors in Internal Medicine conducted an annual, online, confidential survey of its 107 U.S. and Canadian institutional members, including questions about LFW examinations. Data were analyzed using descriptive statistics and coding of free text.

RESULTS

Sixty-nine of 107 members (64.5%) responded. The National Board of Medical Examiners (NBME) examination was administered by 93% (63/68), LFW examinations were used by 33% (22/67), and both types were used by 22% (17/67)-compared with 85%, 36%, and 12% in 2005. LFW examinations were frequently created by the CD alone (9/22; 41%) and consisted of one test (12/22; 52.2%), but some schools gave two (6/22; 26.1%), three (2/22; 8.6%), or four or more (3/22; 13%). Multiple-choice examinations were most common (26/38; 68.4%), followed by short-answer (8/38; 21.1%) and essay (4/38; 10.5%). Most were graded using preestablished criteria; half required a minimum passing score (60% most common). LFW exams were most commonly 5% to 10% of the total grade. Only a minority of CDs reported having reliability estimates or a control group for their exams. Most (70%) reported using LFW exams to cover content felt to be underrepresented by the NBME.

CONCLUSIONS

Findings strongly suggest that a minority of internal medicine CDs use LFW examinations, mostly to measure achievement not assessed by the NBME. However, validity evidence is not consistently being gathered, which may limit judgments based on exam results.

The use of skills inventories to assess and grade practice: Part 2--evaluation of assessment strategy

This paper evaluates the introduction of an assessment tool to grade clinical competence in post-registration critical care courses using a skills-based assessment strategy. An audit of skills assessors was conducted alongside an analysis of theory and practice marks. Findings showed marks awarded for practice were generally higher than those awarded for theory which may be expected in a clinically-based profession. Whilst grading of practice requires further exploration, our experience shows that competence can be defined, measured and the resultant marks incorporated into a degree classification with relative ease. Consistency between assessors remains an issue but can be assisted by the use of clear skills templates and a user-friendly grading tool.

Medical education in the United States of America

This article was written to provide a brief history of the medical educational system in the USA, the current educational structure, and the current topics and challenges facing USA medical educators today. The USA is fortunate to have a robust educational system, with over 150 medical schools, thousands of graduate medical education programs, well-accepted standardized examinations throughout training, and many educational research programs. All levels of medical education, from curriculum reform in medical schools and the integration of competencies in graduate medical education, to the maintenance of certification in continuing medical education, have undergone rapid changes since the turn of the millennium. The intent of the changes has been to involve the patient sooner in the educational process, use better educational strategies, link educational processes more closely with educational outcomes, and focus on other skills besides knowledge. However, with the litany of changes have come increased regulation without (as of yet) clear evidence as to which of the changes will result in better physicians. In addition, the USA governmental debt crisis threatens the current educational structure. The next wave of changes in the USA medical system needs to focus on what particular educational strategies result in the best physicians and how to fund the system over the long term.

Comparing a script concordance examination to a multiple-choice examination on a core internal medicine clerkship

BACKGROUND

Script concordance (SC) questions, in which a learner is given a brief clinical scenario then asked if additional information makes one hypothesis more or less likely, with answers compared to a panel of experts, are designed to reflect a learner's clinical reasoning.

PURPOSE

The purpose is to compare reliability, validity, and learner satisfaction between a three-option modified SC examination to a multiple-choice question (MCQ) examination among medical students during a 3rd-year internal medicine clerkship, to compare reliability and learner satisfaction of SC between medical students and a convenience sample of house staff, and to compare learner satisfaction with SC between 1st- and 4th-quarter medical students.

METHODS

Using a prospective cohort design, we compared the reliability of 20-item SC and MCQ examinations, sequentially administered on the same day. To measure validity, scores were compared to scores on the National Board of Medical Examiners (NBME) subject examination in medicine and to a clinical performance measure. SC and MCQ were also administered to a convenience sample of internal medicine house staff. Medical student and house staff were anonymously surveyed regarding satisfaction with the examinations.

RESULTS

There were 163 students who completed the examinations. With students, the initial reliability of the SC was half that of MCQ (KR20 0.19 vs. 0.41), but with house staff (n = 15), reliability was the same (KR20 = 0.52 for both examinations). SC performance correlated with student clinical performance, whereas MCQ did not (r = .22, p = .005 vs. .11, p = .159). Students reported that SC questions were no more difficult and were answered more quickly than MCQ questions. Both exams were considered easier than NBME, and all 3 were considered equally fair. More students preferred MCQ over SC (55.8% vs. 18.0%), whereas house staff preferred SC (46% vs. 23%; p = .03).

CONCLUSIONS

This SC examination was feasible and was more valid than the MCQ examination because of better correlation with clinical performance, despite being initially less reliable and less preferred by students. SC was more reliable and preferred when administered to house staff.

Reporter-Interpreter-Manager-Educator (RIME) descriptive ratings as an evaluation tool in an emergency medicine clerkship

BACKGROUND

Emergency Medicine (EM) clerkships traditionally assess students using numerical ratings of clinical performance. The descriptive ratings of the Reporter, Interpreter, Manager, and Educator (RIME) method have been shown to be valuable in other specialties.

OBJECTIVES

We hypothesized that the RIME descriptive ratings would correlate with clinical performance and examination scores in an EM clerkship, indicating that the RIME ratings are a valid measure of performance.

METHODS

This was a prospective cohort study of an evaluation instrument for 4(th)-year medical students completing an EM rotation. This study received exempt Institutional Review Board status. EM faculty and residents completed shift evaluation forms including both numerical and RIME ratings. Students completed a final examination. Mean scores for RIME and clinical evaluations were calculated. Linear regression models were used to determine whether RIME ratings predicted clinical evaluation scores or final examination scores.

RESULTS

Four hundred thirty-nine students who completed the EM clerkship were enrolled in the study. After excluding items with missing data, there were 2086 evaluation forms (based on 289 students) available for analysis. There was a clear positive relationship between RIME category and clinical evaluation score (r(2)=0.40, p<0.01). RIME ratings correlated most strongly with patient management skills and least strongly with humanistic qualities. A very weak correlation was seen with RIME and final examination.

CONCLUSION

We found a positive association between RIME and clinical evaluation scores, suggesting that RIME is a valid clinical evaluation instrument. RIME descriptive ratings can be incorporated into EM evaluation instruments and provides useful data related to patient management skills.

Faculty development in assessment: the missing link in competency-based medical education

As the medical education community celebrates the 100th anniversary of the seminal Flexner Report, medical education is once again experiencing significant pressure to transform. Multiple reports from many of medicine's specialties and external stakeholders highlight the inadequacies of current training models to prepare a physician workforce to meet the needs of an increasingly diverse and aging population. This transformation, driven by competency-based medical education (CBME) principles that emphasize the outcomes, will require more effective evaluation and feedback by faculty.Substantial evidence suggests, however, that current faculty are insufficiently prepared for this task across both the traditional competencies of medical knowledge, clinical skills, and professionalism and the newer competencies of evidence-based practice, quality improvement, interdisciplinary teamwork, and systems. The implication of these observations is that the medical education enterprise urgently needs an international initiative of faculty development around CBME and assessment. In this article, the authors outline the current challenges and provide suggestions on where faculty development efforts should be focused and how such an initiative might be accomplished. The public, patients, and trainees need the medical education enterprise to improve training and outcomes now.

Commentary: The RIME/EMR scheme: an educational approach to clinical documentation in electronic medical records

Electronic medical records (EMRs) increasingly are used to document the delivery of patient care. Clinical practices that are involved in medical education are more likely to employ EMRs. Yet, the growing use of EMRs presents a new set of challenges for undergraduate and graduate medical education. EMRs can significantly impact how trainees learn and develop medical decision-making strategies and clinical documentation skills. EMRs also affect how clinical notes are evaluated and how feedback is provided to the learner. To use EMRs effectively, students must learn how narrative elements (how to take and record a medical history and physician examination), data elements (laboratory, radiology, medication, and information from ancillary and consultative services), and system elements (how EMRs function within the context of the health care or hospital system where the student trains) combine in the context of compassionate, competent, and safe patient care. This commentary specifically addresses educational issues surrounding student and resident use of EMR systems. The Reporter-Interpreter-Manager-Educator scheme is one approach to teach and evaluate clinical documentation skills using EMRs in the context of the Accreditation Council for Graduate Medical Education core educational competencies.

Faculty verbal evaluations reveal strategies used to promote medical student performance

BACKGROUND

Preceptors rarely follow medical students' developing clinical performance over time and across disciplines. This study analyzes preceptors' descriptions of longitudinal integrated clerkship (LIC) students' clinical development and their identification of strategies to guide students' progress.

METHODS

We used a common evaluation framework, reporter-interpreter-manager-educator, to guide multidisciplinary LIC preceptors' discussions of students' progress. We conducted thematic analysis of transcripts from preceptors' (seven longitudinal ambulatory preceptors per student) quarterly group discussions of 15 students' performance over one year.

RESULTS

All students' clinical development progressed, although most experienced obstacles. Lack of structure in the history and physical exam commonly obstructed progression. Preceptors used templates for data gathering, and modeling or experiences in the inpatient setting to provide time and solidify structure. To advance students' knowledge acquisition, many preceptors identified focused learning topics with their students; to promote application of knowledge, preceptors used reasoning strategies to teach the steps involved in synthesizing clinical data. Preceptors shared accountability for helping students advance as the LIC allowed them to follow students' response to teaching strategies.

DISCUSSION

These results depict preceptors' perceptions of LIC students' developmental continuum and illustrate how multidisciplinary preceptors can use a common evaluation framework to identify strategies to improve performance and follow students' performance longitudinally.

Lost in translation: the challenges of global communication in medical education publishing

CONTEXT

An academic journal serves its purpose by being read and understood. International medical education journals that want to reach a wider readership must be accessible to a multitude of cultures and contexts. It is therefore important that authors and editors consider how their use of language will be interpreted by health care education colleagues who work in different settings. Given the increasing importance of communicating research findings in health care education, it is surprising that no surveys of the comprehensibility of medical education publications have been published in the medical education literature.

METHODS

We (a group of education researchers from Europe and North America) set out to examine the comprehensibility of a defined set of recently published medical education papers. We surveyed all the articles published in four major international journals on medical education during the first 5 months of 2008 and searched for terminology that might prove obscure or confusing to an international readership.

RESULTS

We found that many of the articles surveyed included terminology, contextual descriptions, acronyms and titles that assumed a shared understanding of setting between authors and readers. We include illustrative examples in the text.

DISCUSSION

Terminological and contextual challenges for international readers are common features of the research publications surveyed. In order that the findings of education research may be more widely disseminated and understood, it is important that authors, referees and editors pay attention to the comprehensibility of the language they use in articles selected for publication.

Clerkship directors' practices with respect to preparing students for and using the National Board of Medical Examiners Subject Exam in medicine: results of a United States and Canadian Survey

PURPOSE

Clerkship directors' practices regarding the National Board of Medical Examiners (NBME) subject exam in medicine are important in enhancing educational evaluation policy. The study's purpose was to determine clerkship directors' use of the subject exam in medicine and related learning activities in the context of curricula and outcomes of the directors' internal medicine clerkships.

METHOD

The authors conducted a survey of directors of internal medicine clerkships in 2007. They performed descriptive statistical and multivariate analyses on all responses.

RESULTS

Of 110 clerkship directors, 82 responded to the survey, for an overall response rate of 75%. Eighty-eight percent of the clerkship directors required the NBME subject examination in medicine. The mean minimum passing score was 62 (SD = 4.2); this score was not adjusted throughout the academic year, and it contributed 20% to 25% of the final grade. Most (89%) clerkships allowed students a retake after a failed first attempt. Most clerkship directors prepared students for the NBME subject exam in their programs through some combination of lectures, independent self-study, and review sessions with exam-preparation review books. However, 42% of clerkship directors lacked a specific strategy for a retake after a failure.

CONCLUSION

Clerkship directors' use of the NBME subject exam in medicine is high. Most allow a retake after a first failure, and a combination of strategies is currently provided to help students prepare. A need exists to develop remediation plans for students who fail the exam. This report may serve as a reference for curricular and programmatic clerkship decisions.

The association of student examination performance with faculty and resident ratings using a modified RIME process

BACKGROUND

RIME is a descriptive framework in which students and their teachers can gauge progress throughout a clerkship from R (reporter) to I (interpreter) to M (manager) to E (educator). RIME, as described in the literature, is complemented by residents and attending physicians meeting with a clerkship director to discuss individual student progress, with group discussion resulting in assignment of a RIME stage.

OBJECTIVE

1) to determine whether a student's RIME rating is associated with end-of-clerkship examination performance; and 2) to determine whose independent RIME rating is most predictive of a student's examination performance: attendings, residents, or interns.

DESIGN

Prospective cohort study.

PARTICIPANTS

Third year medical students from academic years 2004-2005 and early 2005-2006 at 1 medical school.

MEASUREMENTS AND MAIN RESULTS

Each attending, resident, and intern independently assessed the student's final RIME stage attained. For the purpose of analysis, R stage=1, I=2, M=3, and E=4. Regression analyses were performed with examination scores as dependent variables (National Board of Medical Examiners [NBME] medicine subject examination and a clinical performance examination [CPE]), with independent variables of mean attending RIME score, mean resident score, and mean intern score. For the 122 students, significant predictors of NBME subject exam score were resident RIME rating (p = .008) and intern RIME rating (p = .02). Significant predictor of CPE performance was resident RIME rating (p = .01).

CONCLUSION

House staff RIME ratings of students are associated with student performance on written and clinical skills examinations.