Validation of a Web-Based Curriculum for Resident Education in Orthopedic Surgery

An asynchronous web-based intervention for neurosurgery residents to improve education on cost-effective care

OBJECTIVE

To gauge resident knowledge in the socioeconomic aspects of neurosurgery and assess the efficacy of an asynchronous, longitudinal, web-based, socioeconomics educational program tailored for neurosurgery residents.

METHODS

Trainees completed a 20-question pre- and post-intervention knowledge examination including four educational categories: billing/coding, procedure-specific concepts, material costs, and operating room protocols. Structured data from 12 index cranial neurosurgical operations were organized into 5 online, case-based modules sent to residents within a single training program via weekly e-mail. Content from each educational category was integrated into the weekly modules for resident review.

RESULTS

Twenty-seven neurosurgical residents completed the survey. Overall, there was no statistically significant difference between pre- vs post-intervention resident knowledge of billing/coding (79.2 % vs 88.2 %, p = 0.33), procedure-specific concepts (34.3 % vs 39.2 %, p = 0.11), material costs (31.7 % vs 21.6 %, p = 0.75), or operating room protocols (51.7 % vs 35.3 %, p = 0.61). However, respondents' accuracy increased significantly by 40.8 % on questions containing content presented more than 3 times during the 5-week study period, compared to an increased accuracy of only 2.2 % on questions containing content presented less often during the same time period (p = 0.05).

CONCLUSIONS

Baseline resident knowledge in socioeconomic aspects of neurosurgery is relatively lacking outside of billing/coding. Our socioeconomic educational intervention demonstrates some promise in improving socioeconomic knowledge among neurosurgery trainees, particularly when content is presented frequently. This decentralized, web-based approach to resident education may serve as a future model for self-driven learning initiatives among neurosurgical residents with minimal disruption to existing workflows.

Update on Educational Resources and Evaluation Tools for Orthopaedic Surgery Residents

Innovations in orthopaedic resident educational resources and evaluation tools are essential to ensuring appropriate training and ultimately the graduation of competent orthopaedic surgeons. In recent years, there have been several advancements in comprehensive educational platforms within orthopaedic surgery. Orthobullets PASS, Journal of Bone and Joint Surgery Clinical Classroom, and American Academy of Orthopaedic Surgery Resident Orthopaedic Core Knowledge each have their own unique advantages in preparation for the Orthopaedic In-Training Examination and American Board of Orthopaedic Surgery board certification examinations. In addition, the Accreditation Council for Graduate Medical Education Milestones 2.0 and the American Board of Orthopaedic Surgery Knowledge Skills Behavior program each provide objective assessment of resident core competencies. Understanding and using these new platforms will help orthopaedic residents, faculty, residency programs, and program leadership to best train and evaluate their residents.

Does the Preferred Study Source Impact Orthopedic In-Training Examination Performance?

OBJECTIVE

This study examines the role of electronic learning platforms for medical knowledge acquisition in orthopedic surgery residency training. This study hypothesizes that all methods of medical knowledge acquisition will achieve similar levels of improvement in medical knowledge as measured by change in orthopedic in-training examination (OITE) percentile scores. Our secondary hypothesis is that residents will equally value all study resources for usefulness in acquisition of medical knowledge, preparation for the OITE, and preparation for surgical practice.

DESIGN

9 ACGME accredited orthopedic surgery programs participated with 95% survey completion rate. Survey ranked sources of medical knowledge acquisition and study habits for OITE preparation. Survey results were compared to OITE percentile rank scores.

PARTICIPANTS

386 orthopedic surgery residents SETTING: 9 ACGME accredited orthopaedic surgery residency programs RESULTS: 82% of participants were utilizing online learning resources (Orthobullets, ResStudy, or JBJS Clinical Classroom) as primary sources of learning. All primary resources showed a primary positive change in OITE score from 2018 to 2019. No specific primary source improved performance more than any other sources. JBJS clinical classroom rated highest for improved medical knowledge and becoming a better surgeon while journal reading was rated highest for OITE preparation. Orthopedic surgery residents' expectation for OITE performance on the 2019 examination was a statistically significant predictor of their change (decrease, stay the same, improve) in OITE percentile scores (p<0.001).

CONCLUSIONS

Our results showed that no specific preferred study source outperformed other sources. Significantly 82% of residents listed an online learning platform as their primary source which is a significant shift over the last decade. Further investigation into effectiveness of methodologies for electronic learning platforms in medical knowledge acquisition and in improving surgical competency is warranted.

Orthopaedic In-training Exam Preparation among Orthopaedic Surgery Residency Programs

OBJECTIVE

The Orthopaedic In-Training Examination (OITE) is given annually to residents to assess their knowledge of clinical orthopaedics and orthopaedic basic science. This study sought to determine what resources orthopaedic residency programs recommend and/or require for residents as preparative tools for the OITE and to understand which resources are most beneficial.

DESIGN

An anonymous electronic survey was distributed to program directors of orthopaedic surgery residency programs. It included questions about resources that program directors recommended or required residents to use when preparing for the OITE.

SETTING

The survey was prepared and the results analyzed at two academic medical institutions in Louisiana. The survey was available to respondents in December 2019 and January 2020.

PARTICIPANTS

The survey was delivered to program directors of 148 ACGME accredited orthopaedic surgery residency programs in the United States.

RESULTS

148 orthopaedic surgery residency program directors received the survey and 44 responded (response rate: 29.7%). Twenty-nine programs (73.2%) reported that practice tests/review of old OITE exams positively correlated with better scores. The most recommended resources for OITE preparation were Orthobullets (35 programs; 85%) and Res Study (AAOS) (27 programs; 67.5%). Programs having formal/required preparation programs had lower mean overall scores on the OITE than programs that did not. Those without a formal/required program scored in the 60th percentile, while those with a formal/required program scored in the 53.3^rd percentile, a difference of 6.7 percentile points (p = 0.049). The mean overall percentiles were significantly higher for programs that reported having a threshold/goal OITE score for residents compared to those that didn't. Programs with a threshold/goal score scored in the 60.6^th percentile on average, while those without a threshold/goal score scored in the 51.7^th percentile on average, a difference of 8.9 percentile points (p = 0.0095).

CONCLUSIONS

There is substantial variability in the resources that are utilized in preparation for the OITE, with the most commonly recommended resource being Orthobullets. The study method most cited as having a positive impact on scores was practice tests/review of old OITE exams. Having a formal/required program resulted in programs having lower OITE scores. Setting a goal/threshold score correlates positively with an increase in OITE score. Encouraging residents to use practice questions/old OITE tests, setting a goal/threshold score and avoiding formal/required preparation programs may improve resident performance on the OITE.

Orthopaedic In-Training Examination: History, Perspective, and Tips for Residents

Introduced in 1963, the orthopaedic in-training examination (OITE) is a standardized, national test administered annually to orthopaedic residents by the American Academy of Orthopaedic Surgeons. The examination consists of 275 multiple-choice questions that cover 11 domains of orthopaedic knowledge, including basic science, foot and ankle, hand, hip and knee, oncology, pediatrics, shoulder and elbow, spine, sports medicine, trauma, and practice management. The OITE has been validated and is considered predictive of success in both orthopaedic surgery residency and on the American Board of Orthopaedic Surgery part I examination. This article provides a historical overview of the OITE, details its current structure and scoring system, and reviews currently available study materials. For examination preparation, the residents are encouraged to (1) start the examination preparation early, (2) practice on old OITE or self-assessment examination questions, (3) focus on the questions answered incorrectly, (4) focus on comprehension over memorization, and (5) recognize and avoid burnout. Finally, the residents should have a systemic way of approaching each multiple-choice question, both during practice and on the actual examination.

When Is an Orthopedic Intern Ready to Take Call?

OBJECTIVE

While orthopedic residency training varies among programs, an inevitable phenomenon is a transition for interns from consistent oversight to independent call with indirect supervision. It is therefore crucial to reliably assess trainees' ability to perform basic procedures. The objective of the study was to evaluate the utility of a novel Orthopaedic Intern Skills Assessment (OISA) to assess skill level.

DESIGN

In a cohort study, participants were evaluated on their ability to complete eleven skills in a simulated environment. Using a standardized patient and/or cadaveric specimen, we assessed skeletal traction, joint aspiration, joint injection, laceration repair, ankle brachial index measurement, compartment pressure monitoring, upper and lower extremity splinting, informed consent, spine trauma exam, and cervical spine clearance abilities.

SETTING

The assessment took place in a medical simulation lab at the Oregon Health & Science University in Portland, Oregon.

PARTICIPANTS

Third- or fourth-year medical students interested in orthopedics, incoming interns (preinterns), and residents within 1 month of completing their intern year (postinterns) were invited to participate in the study. All interested individuals were included. Of the 20 individuals contacted, 14 (70%) consented: 4 medical students (40%), 5 preinterns (100%), and 5 postinterns (100%). All consenting individuals finished their participation in the research.

RESULTS

Postintern skill completion rate was significantly higher than preinterns (p = 0.006) and medical students (p < 0.001). Completion rates for preinterns were also significantly higher rate than medical students (p = 0.035).

CONCLUSIONS

Our OISA found that postinterns had reached a basic orthopedic skill level expected of a junior resident without in-house supervision. Our OISA also highlighted areas of training that needed further attention, which will help orthopedic training programs verify an appropriate level of skill as trainees' progress from a supervised intern year to taking indirectly supervised junior resident call, and help identify areas where increased training is needed.

Innovation in Orthopaedic Surgery Education: Novel Tools for Modern Times

Orthopaedic residency training has and will continue to evolve with a wide variety of changes. Hands-on surgical simulation "boot camps," computerized simulation of surgical process, and even virtual reality simulators, all can help trainees acquire surgical experience without compromising patient care. Low-cost training modules help remedy the rising costs associated with teaching complex orthopaedic surgery skills. Motion tracking and checklists help refine standardization of assessment. As technology and healthcare systems continue to grow, we encourage training programs to keep pace by considering engagement of these tools.

Analysis of the Sports Medicine Section of the Orthopaedic In-Training Examination: Improvements in Levels of Evidence and Question Taxonomy Over a 12-Year Period

There has been a recent shift within the orthopedic literature to publish articles with higher levels of evidence. In this investigation, the trends in question taxonomy and the levels of evidence of the references for sports medicine questions on the Orthopaedic In-Training Examination (OITE) during a 12-year period were evaluated. Sports medicine questions were obtained from the OITEs administered between 2005 and 2007 and between 2014 and 2016. The taxonomy of each question was characterized, and levels of evidence for all references were assigned using American Academy of Orthopaedic Surgeons guidelines. Question taxonomy and article levels of evidence from 2005 to 2007 were compared with those from 2014 to 2016. Sports medicine questions comprised 8% of the OITEs in both examination groups. The questions from 2014 to 2016 had a higher mean taxonomic level (2.26 vs 1.52, P=.0001) and a greater proportion of studies with high levels of evidence (levels 1 and 2) (21% vs 10%, P=.027). However, references with low levels of evidence or nonprimary resources made up 82.2% and 68.5% of the total references on the older and more recent examinations, respectively. References from 2014 to 2016 were, on average, 2 years older than those from earlier examinations. This study indicated that sports medicine questions on recent OITEs cite references of higher levels of evidence and contain higher taxonomic question structure than examinations 10 to 12 years ago. However, the majority of questions still cite articles with low levels of evidence or nonprimary sources. These findings can be used to guide resident education and continue improvements in the selection of references for questions on the Sports Medicine section of the OITE. [Orthopedics. 2020;43(5):e460-e464.].

Measuring change in knowledge acquisition of Rwandan residents: using the American Board of Pediatrics International In-Training Examination (I-ITE) as an independent tool to monitor individual and departmental improvements during the Human Resources for Health program: an observational study

BACKGROUND

Rwanda is the only African country to use the pediatric International In-Training Examination (I-ITE). The objectives of this study were to use the scores from the I-ITE to outline the baseline level of knowledge of Rwandan residents entering the pediatric residency and the trends in knowledge acquisition from 2012 to 2018, during the Human Resources for Health (HRH) Program, an education partnership between the Rwanda Ministry of Health and a consortium of US universities.

METHODS

A retrospective descriptive analysis of the I-ITE exam scores, taken by all Rwandan pediatric residents for five of the six academic years of the study period. Individual resident scores were weighted using the non-Rwandan I-ITE sites to minimise confounding from annual variations in exam difficulty. Statistical analysis included descriptives with ANOVA to compare variation in annual mean scores.

RESULTS

Eighty-four residents took 213 I-ITE exam sittings over the five exam cycles. The mean weighted I-ITE score of all residents increased from 34% in 2013 to 49% (p < 0.001) in 2018. The 32-point gap between the mean US-ITE and Rwandan I-ITE score in 2012-2013 was reduced to a 16-point gap in 2017-2018. First year resident (PG1) scores, which likely reflect the knowledge level of undergraduate medical students entering the residency program, increased from 34.8 to 44.3% (p = 0.002) between 2013 and 2018.

CONCLUSIONS

The I-ITE is an independent, robust tool, measuring both learners and the institutional factors supporting residents. This is the first study to demonstrate that the I-ITE can be used to monitor resident knowledge acquisition in resource-limited settings, where assessment of resident knowledge can be a major challenge facing the academic medicine community. The significant increase in I-ITE scores between 2012 and 18 reflects the substantial curricular reorganisation accomplished through collaboration between Rwandan and US embedded faculty and supports the theory that programs such as HRH are highly effective at improving the quality of residency programs and undergraduate medical education.