Development and Pilot Testing of a Programmatic System for Competency Assessment in US Anesthesiology Residency Training

BACKGROUND

In 2018, a set of entrustable professional activities (EPAs) and procedural skills assessments were developed for anesthesiology training, but they did not assess all the Accreditation Council for Graduate Medical Education (ACGME) milestones. The aims of this study were to (1) remap the 2018 EPA and procedural skills assessments to the revised ACGME Anesthesiology Milestones 2.0, (2) develop new assessments that combined with the original assessments to create a system of assessment that addresses all level 1 to 4 milestones, and (3) provide evidence for the validity of the assessments.

METHODS

Using a modified Delphi process, a panel of anesthesiology education experts remapped the original assessments developed in 2018 to the Anesthesiology Milestones 2.0 and developed new assessments to create a system that assessed all level 1 through 4 milestones. Following a 24-month pilot at 7 institutions, the number of EPA and procedural skill assessments and mean scores were computed at the end of the academic year. Milestone achievement and subcompetency data for assessments from a single institution were compared to scores assigned by the institution's clinical competency committee (CCC).

RESULTS

New assessment development, 2 months of testing and feedback, and revisions resulted in 5 new EPAs, 11 nontechnical skills assessments (NTSAs), and 6 objective structured clinical examinations (OSCEs). Combined with the original 20 EPAs and procedural skills assessments, the new system of assessment addresses 99% of level 1 to 4 Anesthesiology Milestones 2.0. During the 24-month pilot, aggregate mean EPA and procedural skill scores significantly increased with year in training. System subcompetency scores correlated significantly with 15 of 23 (65.2%) corresponding CCC scores at a single institution, but 8 correlations (36.4%) were <30.0, illustrating poor correlation.

CONCLUSIONS

A panel of experts developed a set of EPAs, procedural skill assessment, NTSAs, and OSCEs to form a programmatic system of assessment for anesthesiology residency training in the United States. The method used to develop and pilot test the assessments, the progression of assessment scores with time in training, and the correlation of assessment scores with CCC scoring of milestone achievement provide evidence for the validity of the assessments.

Competency-Based Time-Variable Anesthesiology Residency Training: Identification of Problems and Solutions

BACKGROUND

Global medical education is gradually moving toward more comprehensive implementations of a competency-based education (CBE) model. Elimination of standard time-based training and adoption of time-variable training (competency-based time-variable training [CB-TVT]) is one of the final stages of implementation of CBE. While CB-TVT has been implemented in some programs outside the United States, residency programs in the United States are still exploring this approach to training. The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS) are encouraging member boards and residency review committees to consider innovative ways programs could implement CB-TVT. The goals of this study were to (1) identify potential problems with the implementation of CB-TVT in anesthesiology residency training, (2) rank the importance of the problems and the perceived difficulty of solving them, and (3) develop proposed solutions to the identified problems.

METHODS

Study participants were recruited from key stakeholder groups in anesthesiology education, including current or former program directors, department chairs, residents, fellows, American Board of Anesthesiology (ABA) board members, ACGME residency review committee members or ACGME leaders, designated institutional officials, residency program coordinators, clinical operations directors, and leaders of large anesthesiology community practice groups. This study was conducted in 2 phases. In phase 1, survey questionnaires were iteratively distributed to participants to identify problems with the implementation of CB-TVT. Participants were also asked to rank the perceived importance and difficulty of each problem and to identify relevant stakeholder groups that would be responsible for solving each problem. In phase 2, surveys focused on identifying potential solutions for problems identified in phase 1.

RESULTS

A total of 36 stakeholders identified 39 potential problems, grouped into 7 major categories, with the implementation of CB-TVT in anesthesiology residency training. Of the 39 problems, 19 (48.7%) were marked as important or very important on a 5-point scale and 12 of 19 (63.2%) of the important problems were marked as difficult or very difficult to solve on a 5-point scale. Stakeholders proposed 165 total solutions to the identified problems.

CONCLUSIONS

CB-TVT is a promising educational model for anesthesiology residency, which potentially results in learner flexibility, individualization of curricula, and utilization of competencies to determine learner advancement. Because of the potential problems with the implementation of CB-TVT, it is important for future pilot implementations of CB-TVT to document realized problems, efficacy of solutions, and effects on educational outcomes to justify the burden of implementing CB-TVT.

Navigating the Landscape of Precision Education: Insights From On-the-Ground Initiatives

ABSTRACT

A central goal of precision education (PE) is efficiently delivering the right educational intervention to the right learner at the right time. This can be achieved through a PE cycle that involves gathering inputs, using analytics to generate insights, planning and implementing interventions, learning and assessing outcomes, and then using lessons learned to inform modifications to the cycle. In this paper, the authors describe 3 PE initiatives utilizing this cycle. The Graduate Medical Education Laboratory (GEL) uses longitudinal data on graduate trainee behavior, clinical skills, and wellness to improve clinical performance and professional fulfillment. The Transition to Residency Advantage (TRA) program uses learner data from medical school coupled with individualized coaching to improve the transition to residency. The Anesthesia Research Group for Educational Technology (TARGET) is developing an automated tool to deliver individualized education to anesthesia residents based on a longitudinal digital representation of the learner. The authors discuss strengths of the PE cycle and transferrable learnings for future PE innovations. Common challenges are identified, including related to data (e.g., volume, variety, sharing across institutions, using the electronic health record), analytics (e.g., validating augmented intelligence models), and interventions (e.g., scaling up learner assessments with limited resources). PE developers need to share their experiences in order to overcome these challenges, develop best practices, and ensure ethical development of future systems. Adapting a common framework to develop and assess PE initiatives will lead to a clearer understanding of their impact, help to mitigate potential risks, and allow deployment of successful practices on a larger scale.

Pro and Con: How Important Is the Exact Location of Adductor Canal and Femoral Triangle Blocks?

In this Pro-Con commentary article, we debate the importance of anterior thigh block locations for analgesia following total knee arthroplasty. The debate is based on the current literature, our understanding of the relevant anatomy, and a clinical perspective. We review the anatomy of the different fascial compartments, the course of different nerves with respect to the fascia, and the anatomy of the nerve supply to the knee joint. The Pro side of the debate supports the view that more distal block locations in the anterior thigh increase the risk of excluding the medial and intermediate cutaneous nerves of the thigh and the nerve to the vastus medialis, while increasing the risk of spread to the popliteal fossa, making distal femoral triangle block the preferred location. The Con side of the debate adopts the view that while the exact location of local anesthetic injection appears anatomically important, it has not been proven to be clinically relevant.

Perioperative Breast Analgesia: a Systematic Review of the Evidence for Perioperative Analgesic Medications

PURPOSE OF REVIEW

Breast surgery is common and may result in significant acute as well as chronic pain. A wide range of pharmacologic interventions is available including opioids, non-steroidal anti-inflammatory drugs (NSAIDs), N-methyl-D-aspartate (NMDA) receptor antagonists, anticonvulsants, and other non-opioids with analgesic properties. We present a review of the evidence for these pharmacologic interventions. A literature search of the MEDLINE database was performed via PubMed with combined terms related to breast surgery, anesthesia, and analgesia. Articles were limited to randomized controlled trial (RCT) design, adult patients undergoing elective surgery on the breast (not including biopsy), and pharmacologic interventions only. Article titles and abstracts were screened, and risk of bias assessments were performed.

RECENT FINDINGS

The search strategy initially captured 7254 articles of which 60 articles met the full inclusion criteria. Articles were organized according to intervention: 6 opioid agonists, 14 NSAIDs and acetaminophen, 4 alpha-2 agonists, 7 NMDA receptor antagonists, 6 local anesthetics, 7 steroids, 15 anticonvulsants (one of which also discussed an NMDA antagonist), 1 antiarrhythmic, and 2 serotonin reuptake inhibitors (one of which also studied an anticonvulsant). A wide variety of medications is effective for perioperative breast analgesia, but results vary by agent and dose. The most efficacious are likely NSAIDs and anticonvulsants. Some agents may also decrease the incidence of chronic postoperative pain, including flurbiprofen, gabapentin, venlafaxine, and memantine. While many individual agents are well studied, optimal combinations of analgesic medications remain unclear.

The Content Quality of YouTube Videos for Professional Medical Education: A Systematic Review

PURPOSE

To evaluate the content quality of YouTube videos intended for professional medical education based on quality rating tool (QRT) scores and determine if video characteristics, engagement metrics, or author type are associated with quality.

METHOD

The authors searched 7 databases for English-language studies about the quality of YouTube videos intended for professional medical education from each database's inception through April 2019. To be included, studies had to be published in 2005 (when YouTube was created) or later. Studies were classified according to the type of QRT used: externally validated, internally validated, or limited global. Study information and video characteristics and engagement metrics were extracted. Videos were classified by video author type.

RESULTS

Thirty-one studies were included in this review. Three studies used externally validated QRTs, 20 used internally validated QRTs, and 13 used limited global QRTs. Studies using externally validated QRTs had average scores/total possible scores of 1.3/4, 26/80, and 1.7/5. Among the 18 studies using internally validated QRTs, from which an average percentage of total possible QRT score could be computed or extracted, the average score was 44% (range: 9%-71%). Videos with academic-physician authors had higher internally validated QRT mean scores (46%) than those with nonacademic-physician or other authors (26%; P < .05).

CONCLUSIONS

The authors found a wide variation in QRT scores of videos, with many low QRT scores. While videos authored by academic-physicians were of higher quality on average, their quality still varied significantly. Video characteristics and engagement metrics were found to be unreliable surrogate measures of video quality. A lack of unifying grading criteria for video content quality, poor search algorithm optimization, and insufficient peer review or controls on submitted videos likely contributed to the overall poor quality of YouTube videos that could be used for professional medical education.

Development and Pilot Testing of Entrustable Professional Activities for US Anesthesiology Residency Training

BACKGROUND

Modern medical education requires frequent competency assessment. The Accreditation Council for Graduate Medical Education (ACGME) provides a descriptive framework of competencies and milestones but does not provide standardized instruments to assess and track trainee competency over time. Entrustable professional activities (EPAs) represent a workplace-based method to assess the achievement of competency milestones at the point-of-care that can be applied to anesthesiology training in the United States.

METHODS

Experts in education and competency assessment were recruited to participate in a 6-step process using a modified Delphi method with iterative rounds to reach consensus on an entrustment scale, a list of EPAs and procedural skills, detailed definitions for each EPA, a mapping of the EPAs to the ACGME milestones, and a target level of entrustment for graduating US anesthesiology residents for each EPA and procedural skill. The defined EPAs and procedural skills were implemented using a website and mobile app. The assessment system was piloted at 7 anesthesiology residency programs. After 2 months, faculty were surveyed on their attitudes on usability and utility of the assessment system. The number of evaluations submitted per month was collected for 1 year.

RESULTS

Participants in EPA development included 18 education experts from 11 different programs. The Delphi rounds produced a final list of 20 EPAs, each differentiated as simple or complex, a defined entrustment scale, mapping of the EPAs to milestones, and graduation entrustment targets. A list of 159 procedural skills was similarly developed. Results of the faculty survey demonstrated favorable ratings on all questions regarding app usability as well as the utility of the app and EPA assessments. Over the 2-month pilot period, 1636 EPA and 1427 procedure assessments were submitted. All programs continued to use the app for the remainder of the academic year resulting in 12,641 submitted assessments.

CONCLUSIONS

A list of 20 anesthesiology EPAs and 159 procedural skills assessments were developed using a rigorous methodology to reach consensus among education experts. The assessments were pilot tested at 7 US anesthesiology residency programs demonstrating the feasibility of implementation using a mobile app and the ability to collect assessment data. Adoption at the pilot sites was variable; however, the use of the system was not mandatory for faculty or trainees at any site.

Anesthesiology Residency Training and the Anesthesia Toolbox: Five Years of Experience With a Collaboratively Developed Peer-Reviewed e-Learning System

The Internet is a source of professional self-education for medical students and residents. Unfortunately, much of the content discovered through search engines is of insufficient quality for professional education. The Anesthesia Toolbox (AT) was developed to provide online peer-reviewed educational resources for anesthesiology trainees and faculty. Since 2014, AT has developed 24 curricula, 822 content items, and 3238 quiz questions. As of March 2020, 64 anesthesiology residency programs in the United States subscribed to the AT (41% of total). Since the onset of the pandemic in March, AT has added 25 programs (28% increase) and gained 1156 users (26% increase).

Spinal anesthesia in a designated block bay for total joint arthroplasty: improving operating room efficiency

Background

Creating highly efficient operating room (OR) protocols for total joint arthroplasty (TJA) is a challenging and multifactorial process. We evaluated whether spinal anesthesia in a designated block bay (BBSA) would reduce time to incision, improve first case start time and decrease conversion to general anesthesia (GA).

Methods

Retrospective cohort study on the first 86 TJA cases with BBSA from April to December 2018, compared with 344 TJA cases with spinal anesthesia performed in the OR (ORSA) during the same period. All TJA cases were included if the anesthetic plan was for spinal anesthesia. Patients were excluded if circumstances delayed start time or time to incision (advanced vascular access, pacemaker interrogation, surgeon availability). Data were extracted and analyzed via a linear mixed effects model to compare time to incision, via a Wilcoxon rank-sum test to compare first case start time, and via a Fisher’s exact test to compare conversion to GA between the groups.

Results

In the mixed effect model, the BBSA group time to incision was 5.37 min less than the ORSA group (p=0.018). The BBSA group had improved median first case start time (30.0 min) versus the ORSA group (40.5 min, p<0.0001). There was lower conversion to GA 2/86 (2.33%) in the BBSA group versus 36/344 (10.47%) in the ORSA group (p=0.018). No serious adverse events were noted in either group.

Conclusions

BBSA had limited impact on time to incision for TJA, with a small decrease for single OR days and no improvement on OR days with two rooms. BBSA was associated with improved first case start time and decreased rate of conversion to GA. Further research is needed to identify how BBSA affects the efficiency of TJA.

The effect of obesity on the anatomical relationship of the popliteal artery and tibial nerve in the proximal and distal popliteal fossa: relevance to popliteal sciatic nerve block and a traceback technique using the popliteal artery

STUDY OBJECTIVE

To determine the effect of body mass index (BMI) on the relationship of the popliteal artery to the sciatic and tibial nerves in the popliteal fossa.

DESIGN

Prospective, observational study.

SETTING

University medical center.

SUBJECTS

One hundred patients scheduled for magnetic resonance imaging scans of the knee.

MEASUREMENTS

BMI was recorded and magnetic resonance imaging scans were assessed at 3 different measurement points along the femur for the distance and angle between the popliteal artery and tibial nerve, or sciatic nerve if the sciatic nerve had not bifurcated at the measurement point.

MAIN RESULTS

At the distal femur, the tibial nerve was a mean of 2.9 mm from the popliteal artery. The nerve was consistently posterior to the artery; however, it was variably located medial or lateral to the artery. At the 5- and 8-cm measurement points, the nerve was 10.0 and 16.1 mm (SD, 4.1 and 5.2 mm), and 31° and 44° (SD, 15° and 16°) lateral to the popliteal artery, respectively. Zero degree was defined as directly posterior to the artery. Increasing BMI was correlated with increasing distance between the nerve and the artery at the 5- and 8-cm measurement points (r= 0.36 P> |t| .000 and .45 P> |t| .002).

CONCLUSIONS

At 5 cm proximal to the distal femoral condyles, the popliteal artery is a reliable sonographic landmark to locate the tibial nerve due to the close proximity and consistent location of the nerve 1 cm posterolateral to the artery, with only a moderate effect of BMI.

The effect of cimetidine and ranitidine on the duration of action of succinylcholine

A recent study suggested that cimetidine prolongs the duration of action of succinylcholine (SCh). We examined this reported interaction, and investigated whether it is the result of inhibition of plasma cholinesterase (PChE). We also studied the effect of ranitidine on the duration of action of succinylcholine. Thirty patients were randomly assigned to three groups to receive oral cimetidine 300 mg or ranitidine 150 mg or no H2-antagonist the night before surgery and again 1-2 hours prior to induction of anesthesia. Succinylcholine (1.5 mg/kg i.v.) was given to facilitate tracheal intubation after induction of anesthesia; neuromuscular function was monitored by evoked compound electromyogram. Pretreatment blood samples were analyzed for PChE activity. PChE activity and levels of appropriate H2-antagonists were measured in preinduction blood samples. No significant effect of either cimetidine or ranitidine on the duration of action of SCh or PChE activity was demonstrated. We conclude that H2-antagonists administered preoperatively do not prolong the duration of action of succinylcholine or alter PChE activity.