Association Between Board Certification, Maintenance of Certification, and Surgical Complications in the United States

Re-examining single-moment-in-time high-stakes examinations in specialist training: A critical narrative review

In this critical narrative review, we challenge the belief that single-moment-in-time high-stakes examinations (SMITHSEx) are an essential component of contemporary specialist training. We explore the arguments both for and against SMITHSEx, examine potential alternatives, and discuss the barriers to change.SMITHSEx are viewed as the "gold standard" assessment of competence but focus excessively on knowledge assessment rather than capturing essential competencies required for safe and competent workplace performance. Contrary to popular belief, regulatory bodies do not mandate SMITHSEx in specialist training. Though acting as significant drivers of learning and professional identity formation, these attributes are not exclusive to SMITHSEx.Skills such as crisis management, procedural skills, professionalism, communication, collaboration, lifelong learning, reflection on practice, and judgement are often overlooked by SMITHSEx. Their inherent design raises questions about the validity and objectivity of SMITHSEx as a measure of workplace competence. They have a detrimental impact on trainee well-being, contributing to burnout and differential attainment.Alternatives to SMITHSEx include continuous low-stakes assessments throughout training, ongoing evaluation of competence in the workplace, and competency-based medical education (CBME) concepts. These aim to provide a more comprehensive and context-specific assessment of trainees' competence while also improving trainee welfare.Specialist training colleges should evolve from exam providers to holistic education sources. Assessments should emphasise essential practical knowledge over trivia, align with clinical practice, aid learning, and be part of a diverse toolkit. Eliminating SMITHSEx from specialist training will foster a competency-based approach, benefiting future medical professionals' well-being and success.

Association between the American Board of Emergency Medicine Oral Certifying Examination and Future State Medical Board Disciplinary Actions

Objectives

The American Board of Emergency Medicine (ABEM) requires a written examination (the Qualifying Examination) followed by the Oral Certifying Examination (OCE) to obtain ABEM certification. Maintaining ABEM certification is associated with fewer state medical board (SMB) disciplinary actions. We sought to determine the association between poor initial performance on the OCE and subsequent severe SMB disciplinary action.

Methods

We included physicians who completed US categorical emergency medicine residencies in 2016 and earlier. We classified OCE performance as good (passed on first attempt) and poor (never passed or required > 1 attempt to pass). We obtained data on physician SMB disciplinary actions from the National Practitioner Data Bank that were limited to actions that denied licensure or altered the status of a medical license (eg, suspension). We determined the association between poor OCE performance and subsequent severe SMB disciplinary action.

Results

Of 34,871, 93.5% passed the OCE on the first attempt, 6.1% required multiple attempts, and 0.3% never passed. Of the physicians (93.5%) with good OCE performance, 1.0% received a severe SMB action. Among physicians with poor OCE performance, 2.3% received a severe action; and of those who never passed, 1.7% received a severe action (Table 1). Poor OCE performance was associated with an increased odds of severe SMB disciplinary action (OR 2.21, 95% CI: 1.57-3.12).

Conclusion

Physicians with poor OCE performance exhibited higher odds of experiencing a subsequent severe SMB disciplinary action. The OCE may have utility as a predictor of future professionalism or clinical performance.

The Doctors' Effect on Patients' Physical Health Outcomes Beyond the Intervention: A Methodological Review

Background

Previous research suggests that when a treatment is delivered, patients' outcomes may vary systematically by medical practitioner.

Objective

To conduct a methodological review of studies reporting on the effect of doctors on patients' physical health outcomes and to provide recommendations on how this effect could be measured and reported in a consistent and appropriate way.

Methods

The data source was 79 included studies and randomized controlled trials from a systematic review of doctors' effects on patients' physical health. We qualitatively assessed the studies and summarized how the doctors' effect was measured and reported.

Results

The doctors' effects on patients' physical health outcomes were reported as fixed effects, identifying high and low outliers, or random effects, which estimate the variation in patient health outcomes due to the doctor after accounting for all available variables via the intra-class correlation coefficient. Multivariable multilevel regression is commonly used to adjust for patient risk, doctor experience and other demographics, and also to account for the clustering effect of hospitals in estimating both fixed and random effects.

Conclusion

This methodological review identified inconsistencies in how the doctor's effect on patients' physical health outcomes is measured and reported. For grading doctors from worst to best performances and estimating random effects, specific recommendations are given along with the specific data points to report.

Governance for Patient Safety: A Framework of Strategy Domains for Risk Management

BACKGROUND

Adverse events in healthcare are primarily due to system failures rather than individuals. Risk reduction strategies should therefore focus on strengthening systems, bringing about improvements in governance, and targeting individual practices or products. The purpose of this study was to conduct a scoping review to develop a global framework of management strategies for sustaining a safety-oriented culture in healthcare organizations, focusing on patient safety and the adoption of good safety-related practices.

METHODS

We conducted a search on safety-related strategies in 2 steps. The first involved a search in the PubMed database to identify effective, broadly framed, cross-sector domains relevant to clinical risk management strategies in healthcare systems. In the second step, we then examined the strategies adopted by running a scoping review for each domain.

RESULTS

Our search identified 8 strategy domains relevant to patient safety: transformational leadership, patient engagement, human resources management quality, innovation technology, skills certification, education in patient safety, teamwork, and effective communication.

CONCLUSIONS

This scoping review explores management strategies key to healthcare systems' efforts to create safety-oriented organizations. Improvement efforts should focus particularly on the domains identified: combined together, they would nurture an overall safety-oriented culture and have an impact on preventable adverse events.

Is There a Surgeons’ Effect on Patients’ Physical Health, Beyond the Intervention, That Requires Further Investigation? A Systematic Review

Objective

To find and review published papers researching surgeons’ effects on patients’ physical health. Clinical outcomes of surgery patients with similar prognoses cannot be fully explained by surgeon skill or experience. Just as there are “hospital” and “psychotherapist” effects, there may be “surgeons” effects that persist after controlling for known variables like patient health and operation riskiness.

Methods

Cohort studies and randomized controlled trials (RCTs) of any surgical intervention, which, after multivariate adjustment, either showed proportion of variance in patients’ physical health outcomes due to surgeons (random effects) or graded surgeons from best to worst (fixed effects). Studies with <15 surgeons or only ascribing surgeons’ effects to known variables excluded. Medline, PubMed, Embase, and PsycINFO were used for search until June 2020. Manual search for papers referring/referred by resulting studies. Risk of bias assessed by Cochrane risk-of-bias tool and Newcastle–Ottawa Scale.

Results

Included studies: 52 cohort studies and three RCTs of 52,436+ surgeons covering 102 outcomes (33 unique). Studies either graded surgeons from best to worst or calculated the intra-class correlation coefficient (ICC), the percentage of patients’ variation due to surgeons, in diverse ways. Sixteen studies showed exceptionally good and/or bad performers with confidence intervals wholly above or below the average performance. ICCs ranged from 0 to 47%, median 4.0%. There are no well-established reporting standards; highly heterogeneous reporting, therefore no meta-analysis.

Discussion

Interpretation: There is a surgeons' effect on patients’ physical health for many types of surgeries and outcomes, ranging from small to substantial. Surgeons with exceptional patient outcomes appear regularly even after accounting for all known confounding variables. Many existing cohort studies and RCTs could be reanalyzed for surgeons’ effects especially after methodological reporting guidelines are published.

Conclusion

In terms of patient outcomes, it can matter which surgeon is chosen. Surgeons with exceptional patient outcomes are worth studying further.

Point your SmartPhone at the code above. If you have a QR code reader the video abstract will appear. Or use:

https://youtu.be/pL-eGyAGhSk

Predictive Factors of First Time Pass Rate on the American Board of Surgery Certification in General Surgery Exams: A Systematic Review

OBJECTIVE

General Surgery residency programs are evaluated on their American Board of Surgery (ABS) Qualifying examination (QE) and Certifying examination (CE) pass rates. This systematic review aims to evaluate predictive factors of ABS QE and CE first time pass rates.

DESIGN

Using the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines, the following electronic databases were searched: PubMed, Embase, JAMA Network, and Google Scholar. Studies available in the English language published between January 2000 and July 2020 were deemed eligible. Articles that did not assess either of the ABS board examinations performance and pass-rates as outcomes were excluded. The Oxford Centre for Evidence-Based Medicine was used to determine the quality and risk of bias of each study.

RESULTS

A total of 31 publications were included for analysis. Undergraduate medical education variables associated with first-time pass rates on the QE and CE include USMLE score, AOA membership, and class rank. Program factors affecting pass rates include program size, geographic location, and Program Director turnover. There is strong correlation between ABSITE and QE. Evidence supports the utility of mock oral examinations (MOEs) in predicting CE success.

CONCLUSIONS

ABSITE scores demonstrated higher correlation with QE pass rate than CE pass rate. MOEs have a positive association with first-time CE pass rates. Nonmodifiable factors such as race/ethnicity, marital status, and geographic location were also found to be predictors. Delaying board certification examination beyond 1 year after residency graduation significantly reduces first-time pass rate.

US Medical Licensing Exam Performance and American Board of Surgery Qualifying and Certifying Examinations

BACKGROUND

Program directors use United States Medical Licensing Exam (USMLE) scores as criteria for ranking applicants. First-time pass rates of the American Board of Surgery (ABS) Qualifying (QE) and Certifying (CE) Examinations have become important measures of residency program quality. USMLE Step 1 will become pass/fail in 2022.

STUDY DESIGN

American Board of Surgery QE and CE success rates were assessed considering multiple characteristics of highly ranked (top 20) applicants to 22 general surgery programs in 2011. Chi-square, t-test, Wilcoxon Rank sum, linear and logistic regression were used, as appropriate.

RESULTS

The QE and CE first attempt pass rates were 96% (235/244) and 86% (190/221), respectively. QE/CE success was not significantly associated with sex, race, research experience, or publications. Alpha Omega Alpha (AΩA) status was associated with success on the index CE (98% vs 83%; p = 0.008). Step 1 and Step 2 Clinical Knowledge (CK) scores of surgeons who passed QE on their first attempt were higher than scores of those who failed (Step 1: 233 vs 218; p = 0.016); (Step 2CK: 244 vs 228, p = 0.009). For every 10-point increase in Step 1 and 2CK scores, the odds of passing CE on the first attempt increased 1.5 times (95% CI 1.12, 2.0; p = 0.006) and 1.5 times (95% CI 1.11, 2.02, p = 0.008), respectively. For every 10-point increase in Steps 1 and 2CK scores, the odds of passing the QE on the first attempt increased 1.85 times (95% CI 1.11, 3.09; p = 0.018) and 1.86 times (95% CI 1.14, 3.06, p = 0.013), respectively.

CONCLUSIONS

USMLE Step 1 and Step 2 CK examination scores correlate with American Board of Surgery QE and CE performance and success. The USMLE decision to transition Step 1 to a pass/fail examination will require program directors to identify other factors that predict ABS performance for ranking applicants.

The Estimated Cost of Maintenance of Certification

BACKGROUND

Maintenance of Certification (MOC) is a controversial topic in medicine for many different reasons. Studies have suggested that there may be associations between fewer negative outcomes and participation in MOC. However, MOC still remains controversial because of its cost. We sought to determine the estimated cost of MOC to the average orthopaedic surgeon, including fees and time cost, defined as the market value of the physician's time.

METHODS

We calculated the total cost of MOC to be the sum of the fees required for applications, examinations, and other miscellaneous fees as well as the time cost to the physician and staff. Costs were calculated for the oral, written, and American Board of Orthopaedic Surgery Web-based Longitudinal Assessment (ABOS WLA) MOC pathways based on the responses of 33 orthopaedic surgeons to a survey sent to a state orthopaedic society.

RESULTS

We calculated the average orthopaedic surgeon's total cost in time and fees over the decade-long period to be $71,440.61 ($7,144.06 per year) for the oral examination MOC pathway and $80,391.55 ($8,039.16 per year) for the written examination pathway. We calculated the cost of the American Board of Orthopaedic Surgery web-based examination pathway to be $69,721.04 ($6,972.10 per year).

CONCLUSIONS

The actual cost of MOC is much higher than just the fees paid to organizations providing services. The majority of the cost comes in the form of time cost to the physician. The ABOS WLA was implemented to alleviate the anxiety of a high-stakes examination and to encourage efficient longitudinal learning. We found that the ABOS WLA pathway does save time and money when compared with the written examination pathway when review courses and study periods are taken. We believe that future policy changes should focus on decreasing physician time spent completing MOC requirements, and decreasing the cost of these requirements, while preserving the model of continued evidence-based medical education.

Hanging up the surgical cap: Assessing the competence of aging surgeons

BACKGROUND

As the average age of surgeons continues to rise, determining when a surgeon should retire is an important public safety concern.

AIM

To investigate strategies used to determine competency in the industrial workplace that could be transferrable in the assessment of aging surgeons and to identify existing competency assessments of practicing surgeons.

METHODS

We searched websites describing non-medical professions within the United States where cognitive and physical competency are necessary for public safety. The mandatory age and certification process, including cognitive and physical requirements, were reported for each profession. Methods for determining surgical competency currently in use, and those existing in the literature, were also identified.

RESULTS

Four non-medical professions requiring mental and physical aptitude that involve public safety and have mandatory testing and/or retirement were identified: Airline pilots, air traffic controllers, firefighters, and United States State Judges. Nine late career practitioner policies designed to evaluate the ageing physician, including surgeons, were described. Six of these policies included subjective performance testing, 4 using peer assessment and 2 using dexterity testing. Six objective testing methods for evaluation of surgeon technical skill were identified in the literature. All were validated for surgical trainees. Only Objective Structured Assessment of Technical Skills (OSATS) was capable of distinguishing between surgeons of different skill level and showing a relationship between skill level and post-operative outcomes.

CONCLUSION

A surgeon should not be forced to hang up his/her surgical cap at a predetermined age, but should be able to practice for as long as his/her surgical skills are objectively maintained at the appropriate level of competency. The strategy of using skill-based simulations in evaluating non-medical professionals can be similarly used as part of the assessment of the ageing surgeons' surgical competency, showing who may require remediation or retirement.

Learning to Read: Successful Program-Based Remediation Using the Surgical Council on Resident Education (SCORE) Curriculum

BACKGROUND

The Surgical Council on Resident Education (SCORE) curriculum is aligned with the American Board of Surgery (ABS) objectives. Our program adopted the SCORE curriculum in 2015 after poor ABS In-Training Examination (ABSITE) performance and lowest quartile ABS Certifying Exam (CE) and Qualifying Exam (QE) first-time pass rates. We examined the association of SCORE use with ABSITE performance and ABS board exam first-time pass rate.

STUDY DESIGN

At a single institution, a retrospective review of surgery residents' SCORE metrics and ABSITE percentile was conducted for academic years 2015 to 2019. Metrics analyzed on the SCORE web portal were mean total minutes and total visits per resident for all residents using SCORE that year. First-time pass rates of the ABS QE and CE were examined from 2013 to 2019. Chi-square and linear regression analysis were performed, and a 95% level of confidence was assumed (alpha = 0.05).

RESULTS

Yearly data from categorical general surgery residents showed a significant increase in total minutes, total visits, and ABSITE percentile. Combined first time pass rates for the ABS QE and CE significant increased from 70.8% in 2013 to 2015 to 93.9% in 2016 to 2019 (p = 0.018).

CONCLUSIONS

Increased longitudinal use of the SCORE curriculum was associated with programmatic improvements in ABSITE performance and ABS board exam first-time pass rate.