Farewell to the Step 2 Clinical Skills Exam: New Opportunities, Obligations, and Next Steps

What have we learned about constructed response short-answer questions from students and faculty? A multi-institutional study

PURPOSE

The purpose of this study was to enrich understanding about the perceived benefits and drawbacks of constructed response short-answer questions (CR-SAQs) in preclerkship assessment using Norcini's criteria for good assessment as a framework.

METHODS

This multi-institutional study surveyed students and faculty at three institutions. A survey using Likert scale and open-ended questions was developed to evaluate faculty and student perceptions of CR-SAQs using the criteria of good assessment to determine the benefits and drawbacks. Descriptive statistics and Chi-square analyses are presented, and open responses were analyzed using directed content analysis to describe benefits and drawbacks of CR-SAQs.

RESULTS

A total of 260 students (19%) and 57 faculty (48%) completed the survey. Students and faculty report that the benefits of CR-SAQs are authenticity, deeper learning (educational effect), and receiving feedback (catalytic effect). Drawbacks included feasibility, construct validity, and scoring reproducibility. Students and faculty found CR-SAQs to be both acceptable (can show your reasoning, partial credit) and unacceptable (stressful, not USMLE format).

CONCLUSIONS

CR-SAQs are a method of aligning innovative curricula with assessment and could enrich the assessment toolkit for medical educators.

Progress testing of an objective structured clinical examination during undergraduate clinical clerkship: a mixed-methods pilot study

BACKGROUND

Progress testing is an assessment method in which an examination reflecting competencies at graduation is regularly administered to students over multiple years, thereby facilitating self-directed learning. However, the significance of the objective structured clinical examination as a progress test in undergraduate education, needs to be determined. This study provides evidence of the role of the objective structured clinical examination for progress testing and optimal scoring methods for assessing students in different academic years.

METHODS

We conducted a sequential explanatory mixed-methods pilot study. Participants were assessed using the Item Rating Scale, the year-adjusted Global Rating Scale, and the Training Level Rating Scale. The characteristics of each scale were compared quantitatively. In addition, the influence of the objective structured clinical examination as a progress test on learning attitudes was examined. Qualitative data from a post-examination questionnaire were analyzed, using content analysis to explore influences on self-directed learning.

RESULTS

Sixth and fifth year clinical students (n = 235) took the objective structured clinical examination progress test. The total Item Rating Scales were recorded (%) as 59.03 ± 5.27 and 52.64 ± 5.08 (p < 0.01); Training Level Rating Scale was 3.94 ± 0.39 vs 3.22 ± 0.42 (p < 0.01); and the year-adjusted Global Rating Scale was 4.25 ± 0.44 vs 4.32 ± 0.52 (no significant difference), for the sixth and fifth year students, respectively. The correlations across stations and the reliability of each station were satisfactory. Four categories were identified in the qualitative analysis: "motivation to learn during the clinical clerkship was promoted," "dissatisfied with being asked about things they had not experienced," "confusion about being unable to use conventional test-taking strategies," and "insufficient understanding of competencies at graduation." The scores indicated significant differences in performance according to training year.

CONCLUSIONS

This study provides evidence that the objective structured clinical examination can be used as a progress testing tool for undergraduate clinical clerkships. Further enhancement of training opportunities and dissemination of performance competency goals in clerkship curricula are required if we intend to promote self-directed learning through progress testing.

Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments

The United States Medical Licensing Examination (USMLE) has been a subject of performance study for artificial intelligence (AI) models. However, their performance on questions involving USMLE soft skills remains unexplored. This study aimed to evaluate ChatGPT and GPT-4 on USMLE questions involving communication skills, ethics, empathy, and professionalism. We used 80 USMLE-style questions involving soft skills, taken from the USMLE website and the AMBOSS question bank. A follow-up query was used to assess the models' consistency. The performance of the AI models was compared to that of previous AMBOSS users. GPT-4 outperformed ChatGPT, correctly answering 90% compared to ChatGPT's 62.5%. GPT-4 showed more confidence, not revising any responses, while ChatGPT modified its original answers 82.5% of the time. The performance of GPT-4 was higher than that of AMBOSS's past users. Both AI models, notably GPT-4, showed capacity for empathy, indicating AI's potential to meet the complex interpersonal, ethical, and professional demands intrinsic to the practice of medicine.

Demise of the USMLE Step-2 CS exam: Rationalizing a way forward

The COVID-19 pandemic has compelled rethinking and changes in medical education, the most controversial perhaps being the cancelation of USMLE Step-2 Clinical Skills exam (Step-2 CS). What started in March of 2020 as suspension of this professional licensure exam, because of concerns about infection risk for examinees, standardized patients (SPs), and administrators, soon became permanent cancelation in January 2021. Expectedly, it triggered debate in medical education circles. Positively, however, the USMLE regulatory agencies (NBME and FSMB) saw an opportunity to innovate an exam tainted with perceptions of validity deficits, cost, examinee inconvenience, and worries about future pandemics; they therefore called for a public debate to fashion a way forward. We have approached the issue by defining Clinical Skills (CS), exploring its epistemology and historic evolution, including assessment modalities from Hippocratic times to the modern era. We defined CS as the art of medicine manifest in the physician-patient encounter as history taking (driven by communication skills and cultural competence) and physical examination. We classified CS components into knowledge and psychomotor skill domains, established their relative importance in the physician process (clinical reasoning) of diagnosis, thus establishing a theoretical framework for developing valid, reliable, feasible, fair, and verifiable CS assessment. Given the concerns for COVID-19 and future pandemics, we established that CS can largely be assessed remotely, and what could not, can be assessed locally (school/regional consortia level) as part of a USMLE-regulated/supervised assessment regimen with established national standards, thus maintaining USMLE's fiduciary responsibilities. We have suggested a national/regional program for faculty development in CS curriculum development, and assessment, including standard setting skills. This pool of expert faculty will form the nucleus of our proposed USMLE-regulated External Peer Review Initiative (EPRI). Finally, we suggest that CS evolves into an academic discipline/department of its own, rooted in scholarship.

After the Discontinuation of Step 2 CS: A Collaborative Statement from the Directors of Clinical Skills Education (DOCS)

Issue: The United States Medical Licensing Exam (USMLE) Step 2 Clinical Skills Examination (Step 2 CS), the only clinical skills competency testing required for licensure in the United States, has been discontinued. Evidence: This exam, though controversial, propelled a movement emphasizing the value of clinical skills instruction and assessment in undergraduate medical education. While disappointed by the loss of this national driver that facilitated standardization of clinical skills education, the Directors of Clinical Skills Education (DOCS) see prospects for educational innovation and growth. DOCS is a national organization and inclusive community of clinical skills education leaders. This statement from DOCS regarding the discontinuation of USMLE Step 2 CS has been informed by DOCS meetings, listserv discussions, an internal survey, and a review of recent literature. Implications: Rigorous clinical skills assessment remains central to effective and patient-centered healthcare. DOCS shares specific concerns as well as potential solutions. Now free from the external pressure to prepare students for success on Step 2 CS, clinical skills educators can reprioritize content and restructure clinical skills programs to best meet the needs of learners and the ever-evolving healthcare landscape. DOCS, as an organization of clinical skills leaders, makes the following recommendations: 1) Collaboration amongst institutions must be prioritized; clinical skills assessment consortia should be expanded. 2) Governing, accrediting, and licensing organizations should leverage their influence to support and require high quality clinical skills assessments. 3) UME clinical skills leaders should develop ways to identify students who perform with exceptional, borderline, and poor clinical skills at their local institutions. 4) UME leadership should fully commit resources and curricular time to graduate students with excellent clinical skills.

Master clinical teachers and personalised learning

Clinical history taking and physical examination are two of the most important competencies of physicians. In addition to informing diagnoses, these activities build rapport and establish relationships between caregivers and patients. Despite this, emphasis on the assessment of bedside clinical skills is declining. To prepare our students for clinical work, we began a clinical competency, personalised teaching programme in which students perform a history and physical examination in front of a master clinical teacher (MCT) approximately every 2 weeks throughout their core clerkship year. The MCT works with the student in a clinical encounter, providing personalised bedside instruction on all features of being a clinician including bedside manner, history-taking skills, physical examination skills, and clinical reasoning. The MCT then provides an assessment of student's competency development and gives feedback to the student about what they do well and where they have opportunities for growth. Assessment data are collected and tracked longitudinally across the clerkship phase to ensure that each student is progressing developmentally. With over 6000 observations of student performance, we are able to discern competency development and growth over time. We can identify if a student is not improving as expected during their clerkship phase and intervene by providing extra practice and training. This core clerkship teaching programme has been well received by both students and instructors and has led us to pilot this approach during the post-clerkship phase of our medical training.

Evolution of Educational Commission for Foreign Medical Graduates Certification in the Absence of the USMLE Step 2 Clinical Skills Examination

The United States Medical Licensing Examination Step 2 Clinical Skills (CS) was paused in 2020 because of the ongoing COVID-19 pandemic and discontinued in 2021. Step 2 CS was an important tool to assess readiness of international medical graduates (IMGs) to enter graduate medical education (GME) in the United States. This article describes the Educational Commission for Foreign Medical Graduates' (ECFMG's) response to the discontinuation of Step 2 CS. ECFMG certifies IMGs who seek eligibility for GME and licensure in the United States. Requirements for ECFMG certification include, among other factors, demonstration of adequate clinical skills and English proficiency, which were previously assessed as part of Step 2 CS. Beginning in June 2020 and during the next year, ECFMG modified the certification process with the introduction of 6 opportunities (pathways) for IMGs to demonstrate adequate clinical skills and a new test of English proficiency. In addition, permanent ECFMG certification is now granted only after the successful completion of the first year of residency, as determined by the program director. The COVID-19 pandemic and discontinuation of Step 2 CS caused a significant crisis for many IMGs who sought entrance into the United States, impacting the careers of those who had planned entry and those who would be eligible for U.S. training and the future workforce. Despite challenges due to the ongoing global pandemic, ECFMG certification continues to allow qualified physicians to enter U.S. GME and ensures that these individuals are ready to begin supervised training.

The Florida Clinical Skills Collaborative: A New Regional Consortium for the Assessment of Clinical Skills

Discontinuation of the United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills (CS) exam and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 2 Performance Evaluation (2-PE) raised questions about the ability of medical schools to ensure the clinical skills competence of graduating students. In February 2021, representatives from all Florida, United States, allopathic and osteopathic schools initiated a collaboration to address this critically important issue in the evolving landscape of medical education. A 5-point Likert scale survey of all members (n=18/20 individuals representing 10/10 institutions) reveals that initial interest in joining the collaboration was high among both individuals (mean 4.78, SD 0.43) and institutions (mean 4.69, SD 0.48). Most individuals (mean 4.78, SD 0.55) and institutions (mean 4.53, SD 0.72) are highly satisfied with their decision to join. Members most commonly cited a "desire to establish a shared assessment in place of Step 2 CS/2-PE" as their most important reason for joining. Experienced benefits of membership were ranked as the following: 1) Networking, 2) Shared resources for curriculum implementation, 3) Scholarship, and 4) Work towards a shared assessment in place of Step 2 CS/2-PE. Challenges of membership were ranked as the following: 1) Logistics such as scheduling and technology, 2) Agreement on common goals, 3) Total time commitment, and 4) Large group size. Members cited the "administration of a joint assessment pilot" as the highest priority for the coming year. Florida has successfully launched a regional consortium for the assessment of clinical skills competency with high levels of member satisfaction which may serve as a model for future regional consortia.

Optimizing Clinical Reasoning Assessments With Analytic and Holistic Ratings: Examining the Validity, Reliability, and Cost of a Simplified Patient Note Scoring Procedure

PURPOSE

Post-standardized patient (SP) encounter patient notes used to assess students' clinical reasoning represent a significant time burden for faculty who traditionally score them. To reduce this burden, the authors previously reported a complex faculty-developed scoring method to assess patient notes rated by nonclinicians. The current study explored whether a simplified scoring procedure for nonclinician raters could further optimize patient note assessments by reducing time, cost, and creating additional opportunities for formative feedback.

METHOD

Ten nonclinician raters scored patient notes of 141 students across 5 SP cases by identifying case-specific patient note checklist items. The authors identified the bottom quintile of students using the proportion of correct items identified in the note (percent-scores) and case-specific faculty-generated scoring formulas (formula-scores). Five faculty raters scored a subset of notes from low, borderline, and high-performing students (n = 30 students) using a global rating scale. The authors performed analyses to gather validity evidence for percent-scores (i.e., relationship to other variables), investigate its reliability (i.e., generalizability study), and evaluate its costs (i.e., faculty time).

RESULTS

Nonclinician percent- and formula-scores were highly correlated ( r = .88) and identified similar lists of low-performing students. Both methods demonstrated good agreement for pass-fail determinations with each other (Kappa = .68) and with faculty global ratings (Kappa percent =.61; Kappa formula =.66). The G-coefficient of percent-scores was .52, with 38% of variability attributed to checklist items nested in cases. Using percent-scores saved an estimated $746 per SP case (including 6 hours of faculty time) in development costs over formula-scores.

CONCLUSIONS

Nonclinician percent-scores reliably identified low-performing students without the need for complex faculty-developed scoring formulas. Combining nonclinician analytic and faculty holistic ratings can reduce the time and cost of patient note scoring and afford faculty more time to coach at-risk students and provide targeted assessment input for high-stakes summative exams.

Cancel culture: exploring the unintended consequences of cancelling the Canadian national licensing clinical examination

Assessment drives learning. However, when it comes to high-stakes examinations (e.g., for licensure or certification), these assessments of learning may be seen as unnecessary hurdles by some. Licensing clinical skills assessment in particular have come under fire over the years. Recently, assessments such as the Medical Council of Canada Qualifying Examination Part II, a clinical skills objective structured clinical examination, have been permanently cancelled. The authors explore potential consequences of this cancellation including those that are inadvertent and undesirable. Future next steps for clinical skills assessment are explored.

Student-Led Initiatives and Advocacy in Academic Medicine: Empowering the Leaders of Tomorrow

Trainees' value as effective leaders within academic medicine has been increasingly recognized. From their perspective as adult learners who enter medical education from diverse backgrounds, trainees offer significant value to the teaching, learning, and practice of medicine. As such, trainees have developed and led various successful initiatives throughout academic medicine. In this Invited Commentary, 2 medical students with national leadership roles provide their perspectives on how student-led initiatives and advocacy can help push academic medicine forward. The authors first provide an overview of the success of student-led initiatives throughout medical education as evidenced by the Trainee-Authored Letters to the Editor in this issue, highlighting the unique contributions and perspectives of trainees in the development and implementation of new initiatives or ways of thinking. Although trainees add value to many areas in academic medicine, here the authors highlight 4 current areas that align with Association of American Medical Colleges priorities: (1) public health emergencies including the COVID-19 pandemic; (2) equity, inclusion, and diversity; (3) wellness and resilience amongst trainees and health care providers; and (4) recent changes to the United States Medical Licensing Examination and the transition to residency. By sharing their experiences with student-led initiatives within each of these domains, the authors provide lessons learned and discuss successes and obstacles encountered along the way. Overall, there is a critical need for increased engagement of trainees in medical education. Empowering trainees now ensures the academic medicine leaders of tomorrow are prepared to face the challenges that await them.