Residency Program Director Perspectives on Changes to US Medical Licensing Examination

This survey study examines the perspectives of residency program directors regarding reporting the US Medical Licensing Examination (USMLE) Step 1 as pass/fail and discontinuing Step 2 Clinical Skills.

The End of Step 2 CS Should Be the Beginning of a New Approach to Clinical Skills Assessment

The discontinuation of the United States Medical Licensing Examination Step 2 Clinical Skills (CS) in 2020 in response to the COVID-19 pandemic marked the end of a decades-long debate about the utility and value of the exam. For all its controversy, the implementation of Step 2 CS in 2004 brought about profound changes to the landscape of medical education, altering the curriculum and assessment practices of medical schools to ensure students were prepared to take and pass this licensing exam. Its elimination, while celebrated by some, is not without potential negative consequences. As the responsibility for assessing students' clinical skills shifts back to medical schools, educators must take care not to lose the ground they have gained in advancing clinical skills education. Instead, they need to innovate, collaborate, and share resources; hold themselves accountable; and ultimately rise to the challenge of ensuring that physicians have the necessary clinical skills to safely and effectively practice medicine.

Multiple United States Medical Licensing Examination Attempts and the Estimated Risk of Disciplinary Actions Among Graduates of U.S. and Canadian Medical Schools

PURPOSE

The United States Medical Licensing Examination (USMLE) recently announced 2 policy changes: shifting from numeric score reporting on the Step 1 examination to pass/fail reporting and limiting examinees to 4 attempts for each Step component. In light of these policies, exam measures other than scores, such as the number of examination attempts, are of interest. Attempt limit policies are intended to ensure minimum standards of physician competency, yet little research has explored how Step attempts relate to physician practice outcomes. This study examined the relationship between USMLE attempts and the likelihood of receiving disciplinary actions from state medical boards.

METHOD

The sample population was 219,018 graduates from U.S. and Canadian MD-granting medical schools who passed all USMLE Step examinations by 2011 and obtained a medical license in the United States, using data from the NBME and the Federation of State Medical Boards. Logistic regressions estimated how attempts on Steps 1, 2 Clinical Knowledge (CK), and 3 examinations influenced the likelihood of receiving disciplinary actions by 2018, while accounting for physician characteristics.

RESULTS

A total of 3,399 physicians (2%) received at least 1 disciplinary action. Additional attempts needed to pass Steps 1, 2 CK, and 3 were associated with an increased likelihood of receiving disciplinary actions (odds ratio [OR]: 1.07, 95% confidence interval [CI]: 1.01, 1.13; OR: 1.09, 95% CI: 1.03, 1.16; OR: 1.11, 95% CI: 1.04, 1.17, respectively), after accounting for other factors.

CONCLUSIONS

Physicians who took multiple attempts to pass Steps 1, 2 CK, and 3 were associated with higher estimated likelihood of receiving disciplinary actions. This study offers support for licensure and practice standards to account for physicians' USMLE attempts. The relatively small effect sizes, however, caution policy makers from placing sole emphasis on this relationship.

Evolution of Clinical Skills Assessment in the USMLE: Looking to the Future After Step 2 CS Discontinuation

The COVID-19 pandemic interrupted administration of the United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills (CS) exam in March 2020 due to public health concerns. As the scope and magnitude of the pandemic became clearer, the initial plans by the USMLE program's sponsoring organizations (NBME and Federation of State Medical Boards) to resume Step 2 CS in the short-term shifted to long-range plans to relaunch an exam that could harness technology and reduce infection risk. Insights about ongoing changes in undergraduate and graduate medical education and practice environments, coupled with challenges in delivering a transformed examination during a pandemic, led to the January 2021 decision to permanently discontinue Step 2 CS. Despite this, the USMLE program considers assessment of clinical skills to be critically important. The authors believe this decision will facilitate important advances in assessing clinical skills. Factors contributing to the decision included concerns about achieving desired goals within desired time frames; a review of enhancements to clinical skills training and assessment that have occurred since the launch of Step 2 CS in 2004; an opportunity to address safety and health concerns, including those related to examinee stress and wellness during a pandemic; a review of advances in the education, training, practice, and delivery of medicine; and a commitment to pursuing innovative assessments of clinical skills. USMLE program staff continue to seek input from varied stakeholders to shape and prioritize technological and methodological enhancements to guide development of clinical skills assessment. The USMLE program's continued exploration of constructs and methods by which communication skills, clinical reasoning, and physical examination may be better assessed within the remaining components of the exam provides opportunities for examinees, educators, regulators, the public, and other stakeholders to provide input.

Consistency Between State Medical License Applications and Recommendations Regarding Physician Mental Health

This study examines the consistency of medical license applications with recommendations from the Federation of State Medical Boards on physician wellness and burnout.

Internal medicine residency program director perceptions of USMLE Step 1 pass/fail scoring: A cross-sectional survey

The United States Medical Licensing Examination Step 1 will transition to a pass/fail exam starting no earlier than January 2022. Internal medicine residency programs will need to adapt to these changes. The purpose of this study was to investigate: 1. internal medicine residency program directors’ perceptions on the change of Step 1 to a pass/fail exam, and 2. the impact on other factors considered for internal medicine residency selection. A validated REDCap survey was sent to 548 program directors at active Accreditation Council for Graduate Medical Education internal medicine residency programs. Contact information from the American Medical Association's Fellowship and Residency Electronic Interactive Database was used. The survey had 123 respondents (22.4%). Most internal medicine program directors do not support the pass/fail change. A greater importance will be placed on Step 2 Clinical Knowledge exam, personal knowledge of the applicant, clerkship grades, and audition electives. Allopathic students from less highly regarded medical schools, as well as osteopathic and international students, will be disadvantaged. About half believe that schools should adopt a graded pre-clinical curriculum (51.2%) and that there should be residency application caps (54.5%). Internal medicine program directors mostly disagree with the pass/fail Step 1 transition. Residency programs will need to reevaluate how applicants are evaluated. Other factors, such as Step 2 Clinical Knowledge score, personal knowledge of the applicant, grades in clerkships, and audition rotations will now be emphasized more heavily.

Association of Use of the NEJM Knowledge+ Product and Performance on the ABIM IM-MOC Exam

Phenomenon: Internal medicine physicians in the United States must pass the American Board of Internal Medicine Internal Medicine Maintenance of Certification (ABIM IM-MOC) examination as part of their ABIM IM-MOC requirements. Many of these physicians use an examination product to help them prepare, such as e-Learning products, including the ACP's MKSAP, UpToDate, and NEJM Knowledge+, yet their effectiveness remains largely unstudied. Approach: We compared ABIM IM-MOC examination performance among 177 physicians who attempted an ABIM IM-MOC examination between 2014-2017 and completed at least 75% of the NEJM Knowledge+ product prior to the ABIM IM-MOC examination and 177 very similar matched control physicians who did not use NEJM Knowledge+. Our measures of ABIM IM-MOC exam performance for NEJM Knowledge+ users were based on the results of the first attempt immediately following the NEJM Knowledge+ use and for non-users were based on the applicable matched examination performance. The three dichotomous examination performance outcomes measured on the first attempt at the ABIM IM-MOC examination included pass rate, scoring in the upper quartile, and scoring in the lower quartile. Findings: Use of NEJM Knowledge+ was associated with a regression adjusted 10.6% (5.37% to 15.8%) greater likelihood of passing the MOC examination (p < .001), 10.7% (2.61% to 18.7%) greater likelihood of having an examination score in the top quartile (p = .009), and -10.8% (-16.8% to -4.86%) lower likelihood of being in the bottom quartile of the MOC examination (p < .001) as compared to similar physicians who did not use NEJM Knowledge+. Insight: Physicians who used NEJM Knowledge+ had better ABIM IM-MOC exam performance. Further research is needed to determine what aspects of e-Learning products best prepare physicians for MOC examinations.

A Crisis of Trust Between U.S. Medical Education and the National Board of Medical Examiners

This Invited Commentary is an independent opinion piece and companion to the Perspective by Carmody and Rajasekaran that appears in this issue of Academic Medicine. The National Board of Medical Examiners (NBME), a 501(c)(3) nonprofit, is a powerful gatekeeper to the medical profession in the United States. According to publicly available tax data, the NBME, which has increased its number of income-enhancing products, had revenues of $153.9 million (M) and net assets of $177.6M in 2017, earnings (revenue less expenses) of $39.7M in 2013-2017, and a highly compensated management team. Medical students are ultimately the source of nearly all the NBME's revenue, and the NBME has contributed to the growth of medical student debt. The NBME has operated as a monopoly since its agreement in the early 1990s with the Federation of State Medical Boards to cosponsor the United States Medical Licensing Examination (USMLE). Although the NBME has developed valuable products and is ostensibly governed by a capable board, the NBME has inherent financial conflicts of interest and may be benefiting from the current "Step 1 mania" undermining undergraduate medical education. Here, the author makes 4 recommendations to reestablish the trust of the U.S. medical education community in the NBME: (1) the NBME should recuse itself from current discussions and policy-making decisions related to changes in the score reporting of the USMLE Step 1 exam; (2) the NBME should disclose and be transparent about all aspects of its finances; (3) new NBME products, changes in pricing, and changes to pass thresholds should be approved by an oversight committee, independent of the NBME; and (4) the NBME (and USMLE) should not charge students or residents for retaking any of its licensing examinations.

How is increased selectivity of medical school admissions associated with physicians' career choice? A Japanese experience

BACKGROUND

During the long-lasting economic stagnation, the popularity of medical school has dramatically increased among pre-medical students in Japan. This is primarily due to the belief that medicine is generally a recession-proof career. As a result, pre-medical students today who want to enter medical school have to pass a more rigorous entrance examination than that in the 1980s. This paper explores the association between the selectivity of medical school admissions and graduates' later career choices.

METHODS

A unique continuous measure of the selectivity of medical school admissions from 1980 to 2017, which is defined as the deviation value of medical schools, was merged with cross-sectional data of 122 990 physicians aged 35 to 55 years. The association between the deviation value of medical schools and various measures of physicians' career choices was explored by logistic and ordinary least square regression models. Graduates from medical schools in which the deviation value was less than 55 were compared with those from more competitive medical schools, after controlling for fixed effects for the medical school attended by binary variables.

RESULTS

From 1980 to 2017, the average deviation value increased from 58.3 to 66.3, indicating a large increase in admission selectivity. Empirical results suggest that increasing selectivity of a medical school is associated with graduates having a higher probability of choosing a career in an acute hospital as well as having a lower probability of opening their own clinic and choosing a career in primary health care. Graduating from a highly competitive medical school (i.e., deviation value of more than 65) significantly increases the probability of working at typical acute hospitals such as so-called 7:1 hospitals (OR 1.665 2, 95%CI 1.444 0-1.920 4) and decreases the probability of working at primary care facilities (OR 0.602 6, 95%CI 0.441 2-0.823 0). It is also associated with graduates having a higher probability of becoming medical board certified (OR 1.294 6, 95%CI 1.108 8-1.511 4).

CONCLUSION

Overall, this paper concludes that increased selectivity of medical school admissions predicts a higher quality of physicians in their own specialty, but at the same time, it is associated with a lower supply of physicians who go into primary care.

The Validity of Scores From the New MCAT Exam in Predicting Student Performance: Results From a Multisite Study

PURPOSE

The new Medical College Admission Test (MCAT) was introduced in April 2015. This report presents findings from the first study of the validity of scores from the new MCAT exam in predicting student performance in the first year of medical school (M1).

METHOD

The authors analyzed data from the national population of 2016 matriculants with scores from the new MCAT exam (N = 7,970) and the sample of 2016 matriculants (N = 955) from 16 medical schools who volunteered to participate in the validity research. They examined correlations of students' MCAT total scores and total undergraduate grade point averages (UGPAs), alone and together, with their summative performance in M1, and the success rate of students with different MCAT scores in their on-time progression to the second year of medical school (M2). They assessed whether MCAT scores provided comparable prediction of performance in M1 by students' race/ethnicity, socioeconomic background, and gender.

RESULTS

Correlations of MCAT scores with summative performance in M1 ranged from medium to large. Although MCAT scores and UGPAs provided similar prediction of performance in M1, using both metrics provided better prediction than either alone. Additionally, students with a wide range of MCAT scores progressed to M2 on time. Finally, MCAT scores provided comparable prediction of performance in M1 for students from different sociodemographic backgrounds.

CONCLUSIONS

This study provides early evidence that scores from the new MCAT exam predict student performance in M1. Future research will examine the validity of MCAT scores in predicting performance in later years.

Investigation of answer changes on the USMLE® Step 2 Clinical Knowledge examination

BACKGROUND

Examinees often believe that changing answers will lower their scores; however, empirical studies suggest that allowing examinees to change responses may improve their performance in classroom assessments. To date, no studies have been able to examine answer changes during large scale professional credentialing or licensing examinations.

METHODS

In this study, we expand the research on answer changes by analyzing responses from 27,830 examinees who completed the Step 2 Clinical Knowledge (CK) examination between August of 2015 and August of 2016.

RESULTS

The results showed that although 68% of examinees changed at least one item, the overall average number of changes was small. Among the examinees who changed answers, approximately 45% increased their scores and approximately 28% decreased their scores. On average, examinees spent shortest time on the item changes from wrong to right and they were more likely to change their scores from wrong to right than right to wrong.

CONCLUSIONS

Consistent with previous studies, these findings support the beneficial effects of answer changes in high-stakes medical examinations and suggest that examinees who are overly cautious about changing answers may put themselves at a disadvantage.

Factors influencing the National License Examination step 1 score in preclinical medical students

The National License Examination step 1 (NLE1), which tests basic medical sciences knowledge in Thailand, is considered to be tough and stressful for medical students due to the large amount of content. This study aimed to determine factors influencing the NLE1 score (NLE1S). The NLE1S, academic achievement, and class attendance were obtained officially. Other factors, including study habits, were obtained via a questionnaire, with 81.97% (241/294) being returned. Students were divided into four groups according to the central passing score and Z-score of the NLE1S, including the fail (<52%; n = 13), low-pass (52 to <70%; n = 121), high-pass (70 to <80%; n = 89), and excellent (≥80%; n = 18) groups. Men had higher NLE1S (P < 0.001) and comprehensive examination scores (P < 0.001) than women. Students with high motivation to study medicine had higher NLE1S. Daily preparation time (h/day) was lower, but stress was higher, in the fail group. In the excellent group, internet for academic use and achievement of study targets were higher; internet for nonacademic use, instance of absence, and stress were lower; and check-in time was earlier. The NLE1S had strong positive correlations with the comprehensive examination score and academic achievement during preclinical studies. By setting the NLE1S as a dependent variable in multivariate regression analyses, models of significant interactions were observed by setting behavioral factors, the comprehensive examination score, and academic achievement during a regular class as independent variables. Thus exhibiting good study habits and showing good academic performance throughout preclinical studies should be encouraged among students to achieve a good NLE1S.

[A new internationally harmonized Swedish basic medical education]

The Swedish Parliament and Government has recently sanctioned a new 6 year undergraduate medical degree leading directly to license, followed by a 12 month introduction to work as a certified doctor. The undergraduate education is internationally harmonized and the 23 learning outcomes address competence needs in future Swedish and international health-care. Particular attention is given to professional competence, critical thinking, team-work and health promotion. All 23 learning outcomes must be documented before the University can issue a MD degree. After license, the doctor applies for a 12 month introduction to employment as doctor in the Swedish health-care system. This introduction is mandatory for all, regardless of the country where the license was issued. The introduction comprises clinical service in general medicine, emergency medicine and elective disciplines. Assessment of 10 learning outcomes is carried out in a formative wav by experienced and trained clinicians during the clinical service. After the introduction, the doctor is duly qualified to apply for specialist training.

USMLE Step 2 CK: Best Predictor of Multimodal Performance in an Internal Medicine Residency

BACKGROUND

Internal medicine (IM) residency programs receive information about applicants via academic transcripts, but studies demonstrate wide variability in satisfaction with and usefulness of this information. In addition, many studies compare application materials to only 1 or 2 assessment metrics, usually standardized test scores and work-based observational faculty assessments.

OBJECTIVE

We sought to determine which application materials best predict performance across a broad array of residency assessment outcomes generated by standardized testing and a yearlong IM residency ambulatory long block.

METHODS

In 2019, we analyzed available Electronic Residency Application Service data for 167 categorical IM residents, including advanced degree status, research experience, failures during medical school, undergraduate medical education award status, and United States Medical Licensing Examination (USMLE) scores. We compared these with post-match residency multimodal performance, including standardized test scores and faculty member, peer, allied health professional, and patient-level assessment measures.

RESULTS

In multivariate analyses, USMLE Step 2 Clinical Knowledge (CK) scores were most predictive of performance across all residency performance domains measured. Having an advanced degree was associated with higher patient-level assessments (eg, physician listens, physician explains, etc). USMLE Step 1 scores were associated with in-training examination scores only. None of the other measured application materials predicted performance.

CONCLUSIONS

USMLE Step 2 CK scores were the highest predictors of residency performance across a broad array of performance measurements generated by standardized testing and an IM residency ambulatory long block.

Challenges Associated With Moving the United States Medical Licensing Examination (USMLE) Step 1 to After the Core Clerkships and How to Approach Them

An increasing number of medical schools have moved away from traditional 2 + 2 curricular structures toward curricula that intentionally integrate basic, clinical, and health systems science, with the goal of graduating physicians who consistently apply their foundational knowledge to clinical practice to improve the care of patients and populations. These curricular reforms often include a shortened preclerkship phase with earlier introduction of learners into clinical environments. This has led schools to reconsider the optimal timing of United States Medical Licensing Examination Step 1. A number of schools have shifted the exam to the period immediately after core clerkships. Although this shift can provide pedagogical advantages, there are potential challenges that must be anticipated and proactively addressed. As more institutions consider making this change, key educational leaders from five schools that repositioned the Step 1 exam after core clerkships share strategies for mitigating some of the potential challenges associated with this approach. The authors describe six possible challenges: lack of readiness without consolidation of basic science knowledge prior to clerkships; risk that weaker students will not be identified and provided academic support early; clerkship or clinical shelf exam performance weaknesses; extension of Step 1 study time; an increase in student anxiety about residency specialty choices; and/or a reduced time frame to take and pass board exams. These potential challenges may be addressed using three main strategies: effective communication with all stakeholders; curricular design and assessments that facilitate integration of basic and clinical sciences; and proactive student coaching and advising.

Pre-medical majors in the humanities and social sciences: impact on communication skills and specialty choice

CONTEXT

Medical school admissions committees use a variety of criteria to determine which candidates to admit to their programmes. Effective communication is increasingly considered a key requisite to the practice of effective medicine. Medical students with pre-medical backgrounds in the humanities and social sciences may be more likely to acquire skills relevant to patient-centred communication, either prior to or during medical school.

OBJECTIVES

The purpose of this study was to investigate the relationship between pre-medical backgrounds in the humanities and social sciences and outcomes in medical school, including in communication and interpersonal skills (CIS), licensure examination results and postgraduate specialty choice (primary care versus non-primary care specialties).

METHODS

The American Medical College Application Service database was used to identify pre-medical college majors, demographic characteristics, Medical College Admission Test scores and college grade point averages for medical students at a large, midwestern medical school. Data were obtained for 465 medical students across three cohorts (classes of 2014-2016). Correlation and regression analyses were used to examine relationships between pre-medical background, performance on graduation competency examination standardised patient encounter CIS scores and on United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge scores, and postgraduate specialty choice.

RESULTS

Graduating medical students with pre-medical humanities or social sciences majors performed significantly better in terms of CIS than those with natural science majors (Cohen's d = 0.28, p = 0.011). There were no significant associations between pre-medical majors and USMLE Step 1 and Step 2 Clinical Knowledge scores or postgraduate specialty choice.

CONCLUSIONS

These results suggest that considering humanistic factors as part of admissions criteria may promote the selection and training of physicians with good communication skills.

Examining Demographics, Prior Academic Performance, and United States Medical Licensing Examination Scores

PURPOSE

To examine whether demographic differences exist in United States Medical Licensing Examination (USMLE) scores and the extent to which any differences are explained by students' prior academic achievement.

METHOD

The authors completed hierarchical linear modeling of data for U.S. and Canadian allopathic and osteopathic medical graduates testing on USMLE Step 1 during or after 2010, and completing USMLE Step 3 by 2015. Main outcome measures were computer-based USMLE examinations: Step 1, Step 2 Clinical Knowledge, and Step 3. Test-taker characteristics included sex, self-identified race, U.S. citizenship status, English as a second language, and age at first Step 1 attempt. Covariates included composite Medical College Admission Test (MCAT) scores, undergraduate grade point average (GPA), and previous USMLE scores.

RESULTS

A total of 45,154 examinees from 172 medical schools met the inclusion criteria. The sample was 67% white and 48% female; 3.7% non-U.S. citizens; and 7.4% with English as a second language. Hierarchical linear models examined demographic variables with and without covariates including MCAT scores and GPA. All Step examinations showed significant differences by gender after adding covariates, varying by Step. Racial differences were observed for each Step, attenuated by the addition of covariates.

CONCLUSIONS

Demographic differences in USMLE performance were tempered by previous examination performance and undergraduate performance. Additional research is required to identify factors that contribute to demographic differences, can aid educators' identification of students who would benefit from assistance preparing for USMLE, and can assist residency program directors in assessing performance measures while meeting diversity goals.

Liner programming method to construct equated item sets for implementation of periodical computer-based testing of Korean medical licensing examination

PURPOSE

This study aimed to find the best way of developing equivalent item sets and to propose a stable and effective management plan for the periodical licensing examinations.

METHODS

Five pre-equated item sets were developed based on the predicted correct answer rate of each item by using linear programming. These pre-equated item sets were compared to the ones that were developed with random item selection method based on the actual answer rate and difficulty from item response theory (IRT). Also, the results with and without common items were compared in the same way. ACAR and the IRT difficulty was used to determine whether there is a significant difference between pre-equating conditions.

RESULTS

There was a statistically significant difference in IRT difficulty among the results from different pre-equated conditions. As predicted correct answer rate was divided into 2 or 3 difficulty boundaries, the actual answer rate and IRT difficulty parameters of the 5 item sets were equally constructed. Comparing item sets conditions with common items and without common items, including common items did not contribute much for the equating of 5 item sets.

CONCLUSION

This study suggested the linear programming method is applicable to construct equated-item sets that reflect each content area. The best method to construct equated item sets suggested is to divide the predicted correct answer rate into 2 or 3 difficulty boundaries regardless of common items. If pre-equated item sets are required to construct a test based on the actual data, several optimal methods should be considered by simulation studies before administrating a real test.

The International Landscape of Medical Licensing Examinations: A Typology Derived From a Systematic Review

BACKGROUND

National licensing examinations (NLEs) are large-scale examinations usually taken by medical doctors close to the point of graduation from medical school. Where NLEs are used, success is usually required to obtain a license for full practice. Approaches to national licensing, and the evidence that supports their use, varies significantly across the globe. This paper aims to develop a typology of NLEs, based on candidacy, to explore the implications of different examination types for workforce planning.

METHODS

A systematic review of the published literature and medical licensing body websites, an electronic survey of all medical licensing bodies in highly developed nations, and a survey of medical regulators.

RESULTS

The evidence gleaned through this systematic review highlights four approaches to NLEs: where graduating medical students wishing to practice in their national jurisdiction must pass a national licensing exam before they are granted a license to practice; where all prospective doctors, whether from the national jurisdiction or international medical graduates, are required to pass a national licensing exam in order to practice within that jurisdiction; where international medical graduates are required to pass a licensing exam if their qualifications are not acknowledged to be comparable with those students from the national jurisdiction; and where there are no NLEs in operation. This typology facilitates comparison across systems and highlights the implications of different licensing systems for workforce planning.

CONCLUSION

The issue of national licensing cannot be viewed in isolation from workforce planning; future research on the efficacy of national licensing systems to drive up standards should be integrated with research on the implications of such systems for the mobility of doctors to cross borders.

Do State Continuing Medical Education Requirements for Physicians Improve Clinical Knowledge?

OBJECTIVE

To evaluate the effect of state continuing medical education (CME) requirements on physician clinical knowledge.

DATA SOURCES

Secondary data for 19,563 general internists who took the Internal Medicine Maintenance of Certification (MOC) examination between 2006 and 2013.

STUDY DESIGN

We took advantage of a natural experiment resulting from variations in CME requirements across states over time and applied a difference-in-differences methodology to measure associations between changes in CME requirements and physician clinical knowledge. We measured changes in clinical knowledge by comparing initial and MOC examination performance 10 years apart. We constructed difference-in-differences estimates by regressing examination performance changes against physician demographics, county and year fixed effects, trend-state indicators, and state CME change indicators.

DATA COLLECTION

Physician data were compiled by the American Board of Internal Medicine. State CME policies were compiled from American Medical Association reports.

PRINCIPAL FINDINGS

More rigorous CME credit-hour requirements (mostly implementing a new requirement) were associated with an increase in examination performance equivalent to a shift in examination score from the 50th to 54th percentile.

CONCLUSIONS

Among physicians required to engage in a summative assessment of their clinical knowledge, CME requirements were associated with an improvement in physician clinical knowledge.

Step Up-Not On-The Step 2 Clinical Skills Exam: Directors of Clinical Skills Courses (DOCS) Oppose Ending Step 2 CS

Recently, a student-initiated movement to end the United States Medical Licensing Examination Step 2 Clinical Skills and the Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation has gained momentum. These are the only national licensing examinations designed to assess clinical skills competence in the stepwise process through which physicians gain licensure and certification. Therefore, the movement to end these examinations and the ensuing debate merit careful consideration. The authors, elected representatives of the Directors of Clinical Skills Courses, an organization comprising clinical skills educators in the United States and beyond, believe abolishing the national clinical skills examinations would have a major negative impact on the clinical skills training of medical students, and that forfeiting a national clinical skills competency standard has the potential to diminish the quality of care provided to patients. In this Perspective, the authors offer important additional background information, outline key concerns regarding the consequences of ending these national clinical skills examinations, and provide recommendations for moving forward: reducing the costs for students, exploring alternatives, increasing the value and transparency of the current examinations, recognizing and enhancing the strengths of the current examinations, and engaging in a national dialogue about the issue.

Non-physician clinicians in rural Africa: lessons from the Medical Licentiate programme in Zambia

BACKGROUND

Most sub-Saharan African countries struggle to make safe surgery accessible to rural populations due to a shortage of qualified surgeons and the unlikelihood of retaining them in district hospitals. In 2002, Zambia introduced a new cadre of non-physician clinicians (NPCs), medical licentiates (MLs), trained initially to the level of a higher diploma and from 2013 up to a BSc degree. MLs have advanced clinical skills, including training in elective and emergency surgery, designed as a sustainable response to the surgical needs of rural populations.

METHODS

This qualitative study aimed to describe the role, contributions and challenges surgically active MLs have experienced. Based on 43 interviewees, it includes the perspective of MLs, their district hospital colleagues-medical officers (MOs), nurses and managers; and surgeon-supervisors and national stakeholders.

RESULTS

In Zambia, MLs play a crucial role in delivering surgical services at the district level, providing emergency surgery and often increasing the range of elective surgical cases that would otherwise not be available for rural dwellers. They work hand in hand with MOs, often giving them informal surgical training and reducing the need for hospitals to refer surgical cases. However, MLs often face professional recognition problems and tensions around relationships with MOs that impact their ability to utilise their surgical skills.

CONCLUSIONS

The paper provides new evidence concerning the benefits of 'task shifting' and identifies challenges that need to be addressed if MLs are to be a sustainable response to the surgical needs of rural populations in Zambia. Policy lessons for other countries in the region that also use NPCs to deliver essential surgery include the need for career paths and opportunities, professional recognition, and suitable employment options for this important cadre of healthcare professionals.

The evolution of the national licensing system of health care professionals: a qualitative descriptive case study in Lao People's Democratic Republic

BACKGROUND

The introduction of a systematic framework for the licensing of health care professions, which is a crucial step in ensuring the quality of human resources for health (HRH), is still evolving in Lao People's Democraic Republic. The aim of this study was to review and document the evolution of Lao HRH policies and the development of its national licensing system.

CASE PRESENTATION

A qualitative descriptive case study methodology was applied to document and describe how Lao People's Democratic Republic laid the foundation for the development of a licensing system. The results demonstrate that Lao People's Democratic Republic is currently in the process of transitioning the focus of its HRH policies from the quantity and deployment of services to remote areas to improvements in the quality of services. The key events in the process of developing the licensing system are as follows: (1) the systematic development of relevant policies and legislation, (2) the establishment of responsible organizations and the assignment of responsible leaders, (3) the acceleration of development efforts in response to the Association of Southeast Asian Nations Mutual Recognition Arrangement for standard qualifications, (4) the strengthening of educational systems for fostering competent health care professionals, (5) the introduction of a 3-year compulsory service component in rural areas for newly recruited government servants, and (6) the introduction of a requirement to obtain a professional health care certificate to work in a private hospital. The Lao Ministry of Health (MOH) has endorsed a specific strategy for licensing to realize this system.

CONCLUSION

The need for licensing systems has increased in recent years due to regional economic integration and a shift in policy toward achieving universal health coverage. A national licensing system would be a significant milestone in health system development, helping to ensure the competency of health care professionals by means of a national examination, continuing professional development, and the revoking of licenses when appropriate.

Justice and care: decision making by medical school student promotions committees

CONTEXT

The function of medical school entities that determine student advancement or dismissal has gone largely unexplored. The decision making of 'academic progress' or student promotions committees is examined using a theoretical framework contrasting ethics of justice and care, with roots in the moral development work of theorists Kohlberg and Gilligan.

OBJECTIVES

To ascertain promotions committee members' conceptualisation of the role of their committee, ethical orientations used in member decision making, and student characteristics most influential in that decision making.

METHODS

An electronic survey was distributed to voting members of promotions committees at 143 accredited allopathic medical schools in the USA. Descriptive statistics were calculated and data were analysed by gender, role, institution type and class size.

RESULTS

Respondents included 241 voting members of promotions committees at 55 medical schools. Respondents endorsed various promotions committee roles, including acting in the best interest of learners' future patients and graduating highly qualified learners. Implementing policy was assigned lower importance. The overall pattern of responses did not indicate a predominant orientation toward an ethic of justice or care. Respondents indicated that committees have discretion to take individual student characteristics into consideration during deliberations, and that they do so in practice. Among the student characteristics with the greatest influence on decision making, professionalism and academic performance were paramount. Eighty-five per cent of participants indicated that they received no training.

CONCLUSIONS

Promotions committee members do not regard orientations of justice and care as being mutually exclusive and endorse an array of statements regarding the committee's purpose that may conflict with one another. The considerable variance in the influence of student characteristics and the general absence of committee member training indicate a need for clear delineation of the medical profession's priorities in terms of justice and care, and of the specific student characteristics that should factor into deliberations.

Perceptions of Unprofessional Social Media Behavior Among Emergency Medicine Physicians

BACKGROUND

Use of social media (SM) by physicians has exposed issues of privacy and professionalism. While guidelines have been created for SM use, details regarding specific SM behaviors that could lead to disciplinary action presently do not exist.

OBJECTIVE

To compare State Medical Board (SMB) directors' perceptions of investigation for specific SM behaviors with those of emergency medicine (EM) physicians.

METHODS

A multicenter anonymous survey was administered to physicians at 3 academic EM residency programs. Surveys consisted of case vignettes, asking, "If the SMB were informed of the content, how likely would they be to initiate an investigation, possibly leading to disciplinary action?" (1, very unlikely, to 4, very likely). Results were compared to published probabilities using exact binomial testing.

RESULTS

Of 205 eligible physicians, 119 (58%) completed the survey. Compared to SMB directors, EM physicians indicated similar probabilities of investigation for themes involving identifying patient images, inappropriate communication, and discriminatory speech. Participants indicated lower probabilities of investigation for themes including derogatory speech (32%, 95% confidence interval [CI] 24-41 versus 46%, P < .05); alcohol intoxication (41%, 95% CI 32-51 versus 73%, P < .05); and holding alcohol without intoxication (7%, 95% CI 3-13 versus 40%, P < .05). There were no significant associations with position, hospital site, years since medical school, or prior SM professionalism training.

CONCLUSIONS

Physicians reported a lower likelihood of investigation for themes that intersect with social identity, compared to SMB directors, particularly for images of alcohol and derogatory speech.

The association of USMLE Step 1 and Step 2 CK scores with residency match specialty and location

BACKGROUND

For future physicians, residency programs offer necessary extended training in specific medical specialties. Medical schools benefit from an understanding of factors that lead their students to match into certain residency specialties. One such factor, often used during the residency application process, is scores on the USA Medical Licensing Exam (USMLE).

OBJECTIVES

To determine the relationship between USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores and students' residency specialty match, and the association between both USMLE scores and state of legal residency (Minnesota) at the time of admission with students staying in-state or leaving the state for residency program.

DESIGN

USMLE scores and residency match data were analyzed from five graduating classes of students at the University of Minnesota Medical School (N = 1054).

RESULTS

A MANOVA found significant differences (p < 0.001) between residency specialties and both USMLE Step 1 and Step 2 CK scores, as well as the combination of the two. Students who matched in Dermatology had the highest mean USMLE scores overall, while students who matched in Family Medicine had the lowest mean scores. Students who went out of state for residency had significantly higher Step 1 scores (p = 0.027) than students who stayed in-state for residency, while there was no significant difference between the groups for Step 2 scores. A significant positive association was found between a student who applied as a legal resident of Minnesota and whether the student stayed in Minnesota for their residency program.

CONCLUSIONS

Residency specialty match was significantly associated with USMLE Step 1 and USMLE Step 2 CK scores, as was staying in-state or leaving the state for residency. Students who were legal residents of the state at the time of application were more likely to stay in-state for residency, regardless of USMLE score.

ABBREVIATIONS

CK: Clinical knowledge; COMLEX: Comprehensive Osteopathic Medical Licensing Examination; GME: Graduate medical education; NRMP: National Resident Matching Program; UME: Undergraduate medical education; USMLE: United States Medical Licensing Examination.

The Predictive Validity of the National Board of Osteopathic Medical Examiners' COMLEX-USA Examinations With Regard to Outcomes on American Board of Family Medicine Examinations

PURPOSE

To examine the predictive validity of the National Board of Osteopathic Medical Examiners' Comprehensive Osteopathic Medical Licensing Examination of the United States of America (COMLEX-USA) series with regard to the American Board of Family Medicine's (ABFM's) In-Training Examination (ITE) and Maintenance of Certification for Family Physicians (MC-FP) Examination.

METHOD

A repeated-measures design was employed, using test scores across seven levels of training for 1,023 DOs who took the MC-FP for the first time between April 2012 and November 2014 and for whom the ABFM had ITE scores for each of their residency years. Pearson and disattenuated correlations were calculated; Fisher r to z transformation was performed; and sensitivity, specificity, and positive and negative predictive values for the COMLEX-USA Level 2-Cognitive Evaluation (CE) with regard to the MC-FP were computed.

RESULTS

The Pearson and disattenuated correlations ranged from 0.55 to 0.69 and from 0.61 to 0.80, respectively. For MC-FP scores, only the correlation increase from the COMLEX-USA Level 2-CE to Level 3 was statistically significant (for Pearson correlations: z = 2.41, P = .008; for disattenuated correlations: z = 3.16, P < .001). The sensitivity, specificity, and positive and negative predictive values of the COMLEX-USA Level 2-CE with the MC-FP were 0.90, 0.39, 0.96, and 0.19, respectively.

CONCLUSIONS

Evidence was found that the COMLEX-USA can assist family medicine residency program directors in predicting later resident performance on the ABFM's ITE and MC-FP, which is becoming increasingly important as graduate medical education accreditation moves toward a single aligned model.

Japanese Medical Education Reforms during the Allied Forces Occupancy; Roles Played by the Public Health and Welfare Section of the Supreme Commander for the Allied Powers

The health and welfare of the Japanese people were of a lower standard compared to other developed countries at the end of the World War II in 1945. Crawford F. Sams, Chief, the Public Health and Welfare Section of the Supreme Commander for the Allied Powers thought that medical care in a wartom country could be improved not by building new hospitals and providing more medical equipment, but through professional education and training. He founded the Council on Medical Education to reform the Japanese medical education. The Council shaped Japanese medical education by establishing the standards for medical school education and initiating internship and a national medical licensure examination. In the early 1950s, the Unitarian Service Committee Medical Mission was invited to teach to medical school professors and students American medicine. This medical mission was also a contribution of the Public Health and Welfare Section to Japanese medical education. This article explores how Public Health and Welfare Section played vital roles in transforming Japanese medical education and postgraduate training during the occupation.

The impact of large scale licensing examinations in highly developed countries: a systematic review

BACKGROUND

To investigate the existing evidence base for the validity of large-scale licensing examinations including their impact.

METHODS

Systematic review against a validity framework exploring: Embase (Ovid Medline); Medline (EBSCO); PubMed; Wiley Online; ScienceDirect; and PsychINFO from 2005 to April 2015. All papers were included when they discussed national or large regional (State level) examinations for clinical professionals, linked to examinations in early careers or near the point of graduation, and where success was required to subsequently be able to practice. Using a standardized data extraction form, two independent reviewers extracted study characteristics, with the rest of the team resolving any disagreement. A validity framework was used as developed by the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education to evaluate each paper's evidence to support or refute the validity of national licensing examinations.

RESULTS

24 published articles provided evidence of validity across the five domains of the validity framework. Most papers (n = 22) provided evidence of national licensing examinations relationships to other variables and their consequential validity. Overall there was evidence that those who do well on earlier or on subsequent examinations also do well on national testing. There is a correlation between NLE performance and some patient outcomes and rates of complaints, but no causal evidence has been established.

CONCLUSIONS

The debate around licensure examinations is strong on opinion but weak on validity evidence. This is especially true of the wider claims that licensure examinations improve patient safety and practitioner competence.

[Internship-test reveals increased knowledge gaps]

We present the results of the medical knowledge test after fulfilled internship for Swedish medical authorization during the years 2009 to the spring of 2015. A total of 7,613 tests were analyzed. Interns graduated from Swedish universities failed in 2.7% to 3.8% of the test moments. Interns who graduated from countries within the European Union (EU) failed in 21.2% and interns graduated from a non-EU country failed in 41.6%. The results from those who graduated from EU and non-EU countries have worsened compared to an earlier study in 2009. Proper measures have now to be implemented for doctors graduated from a non-Swedish university to improve the outcome and introduction to the Swedish health care system.