The Kolb learning cycle in American Board of Surgery In-Training Exam remediation: the Accelerated Clinical Education in Surgery course

Stress, Grit, Satisfaction With Life, and Remediation of Prelicensure Nursing Students

BACKGROUND

Stress and poor academic performance often lead to high levels of nursing school attrition.

PURPOSE

To explore nursing students' perceived stress, grit, and satisfaction with life, and assess the effectiveness of a remediation program for students at risk for poor academic performance.

METHODS

Perceived stress, grit, and satisfaction with life were measured, and the effect of remediation given to at-risk students was studied. Examination and psychometric scores were compared between remediation and nonremediation groups.

RESULTS

The remediation group had higher levels of perceived stress and a lower satisfaction with life compared with the rest of the cohort. Remediation significantly improved examination scores of at-risk students although scores remained lower than those among nonremediation students.

CONCLUSIONS

Perceived stress and grit scores were high in nursing students, and satisfaction varied among age groups. Remediation based on metacognitive theory significantly improved at-risk students' examination scores.

Uses of Electronic Health Record Data to Measure the Clinical Learning Environment of Graduate Medical Education Trainees: A Systematic Review

PURPOSE

This study systematically reviews the uses of electronic health record (EHR) data to measure graduate medical education (GME) trainee competencies.

METHOD

In January 2022, the authors conducted a systematic review of original research in MEDLINE from database start to December 31, 2021. The authors searched for articles that used the EHR as their data source and in which the individual GME trainee was the unit of observation and/or unit of analysis. The database query was intentionally broad because an initial survey of pertinent articles identified no unifying Medical Subject Heading terms. Articles were coded and clustered by theme and Accreditation Council for Graduate Medical Education (ACGME) core competency.

RESULTS

The database search yielded 3,540 articles, of which 86 met the study inclusion criteria. Articles clustered into 16 themes, the largest of which were trainee condition experience (17 articles), work patterns (16 articles), and continuity of care (12 articles). Five of the ACGME core competencies were represented (patient care and procedural skills, practice-based learning and improvement, systems-based practice, medical knowledge, and professionalism). In addition, 25 articles assessed the clinical learning environment.

CONCLUSIONS

This review identified 86 articles that used EHR data to measure individual GME trainee competencies, spanning 16 themes and 6 competencies and revealing marked between-trainee variation. The authors propose a digital learning cycle framework that arranges sequentially the uses of EHR data within the cycle of clinical experiential learning central to GME. Three technical components necessary to unlock the potential of EHR data to improve GME are described: measures, attribution, and visualization. Partnerships between GME programs and informatics departments will be pivotal in realizing this opportunity.

We Tabulated and Organized American Board of Neurological Surgeons Primary Exam Keywords (2015-2023) so You Don't Have to

Background Passing the American Board of Neurological Surgeons (ABNS) Primary Exam is required for residents in training. Both the program directors and residents are given keywords of the exam afterward in the hope to help program directors determine their relative strengths and weakness. We have organized and tabulated these keywords for neurosurgery residents' benefit. Methodology We collected and analyzed ABNS Primary Exam keywords (2015-2023) in each of the exam's main categories for trends and recurrences. We examined the overall passing rates among first-time credit test takers. The frequency of each subcategory was calculated as a percentage within its corresponding category. Recurrent keywords were grouped together with their corresponding years and categorized as once, twice, or thrice and greater occurrences; the last category was considered to be high-yield keywords. Results The number of questions in Neurosciences and Neurology has decreased over the years while Neurosurgery and Critical Care questions have increased. Similarly, there are fewer keyword repeats in Neurosciences and Neurology. The most repeated keywords are in Neuroimaging. The most common keywords are presented and listed along with the years of occurrences. Overall, the passing rate among first-time credit test takers is over 90%. Conclusions Neurosurgery residents can consider the common keywords as a guide in preparation for the ABNS Primary Exam.

Just Do It: Participation in Structured Online Curricula Reliably Improves Low ABSITE Scores

OBJECTIVE

We hypothesized residents enrolled in an Accelerated Clinical Education in Surgery (ACES) program would improve their scores to above the 30^th percentile. We analyzed which components of ACES correlated with improvement.

DESIGN AND SETTING

A retrospective review of three academic cycles (2018-2021) at an academic general surgery residency.

PARTICIPANTS

Residents scoring ≤30^th percentile on the ABSITE were enrolled in ACES. Baseline demographics including STEP scores were collected. ACES included: (1) SCORE and DeckerMed assignments (2) Weekly faculty review sessions and (3) Monthly meeting with assigned mentor. Data were analyzed by Student's t-test, one-way ANOVA and Fisher's exact test.

RESULTS

Twenty-six surgical residents enrolled in ACES. Compared to residents not in ACES, there was no significant difference females (15 vs. 15; p = 0.19) and STEP 2 scores (241 vs. 246; p = 0.06). Residents in ACES had significantly lower STEP 1 (225 vs. 237; p < 0.001) and STEP 3 (212 vs. 223; p < 0.001) scores. Demographics of ACES residents who subsequently scored >30^th percentile were similar to those who didn't, except for STEP 3 scores (216 vs. 204; p = 0.008). For residents in ACES, the completion of assignments between July and January was significantly higher for those who subsequently achieved an ABSITE score >30^th percentile: TWIS, 77% vs. 53% (p = 0.022), Decker WC, 80% vs. 49% (p = 0.009) and Decker MR, 53% vs. 29% (p = 0.016). Completion of an online practice exam prior to ABSITE also correlated with score >30^th percentile (57% vs. 13%, p = 0.007). There was also no correlation between the number of faculty review sessions and ABSITE (11.5 vs.11.9, p = 0.931).

CONCLUSIONS

Participation in a structured online program of reading and quizzes was durably effective in improving ABSITE scores >30^th percentile. Completion of online assignments, rather than scores on practice tests or review sessions, appeared to be the most important factor for success.

Effectiveness of interprofessional shared decision-making training: A mixed-method study

OBJECTIVE

This study evaluated the learning effects and examined the participants' perceptions of an interprofessional shared decision-making (IP-SDM) training program.

METHODS

This mixed-method study used a quasi-experimental pretest-posttest design in the quantitative phase and semi-structured interviews in the qualitative phase. The 6-week curriculum design, based on Kolb's experiential learning cycle, consisted of two simulated objective structured clinical examinations with standardized patients and blended teaching methods through various course modules.

RESULTS

A total of 39 multidisciplinary healthcare personnel completed the 6-week training program, and 32 of them participated in qualitative interviews. The IP-SDM training program effectively improved the SDM process competency of the participants from the perspectives of the participants, standardized patients, and clinical teachers. The interviews illustrated how the curriculum design enhanced learning; the effectiveness results indicated improvements in learners' attitude, knowledge, skills, and teamwork.

CONCLUSION

This IP-SDM training program improved multidisciplinary healthcare personnel's competency, self-efficacy, and intention to engage in IP-SDM.

PRACTICE IMPLICATIONS

Applying Kolb's experiential learning cycle and blended teaching methods to develop and implement the IP-SDM training program can improve multidisciplinary healthcare personnel's knowledge, attitude, skills, and teamwork in IP-SDM.

Association of Gender and Race/Ethnicity with Internal Medicine In-Training Examination Performance in Graduate Medical Education

BACKGROUND

Disparities in objective assessments in graduate medical education such as the In-Training Examination (ITE) that disadvantage women and those self-identifying with race/ethnicities underrepresented in medicine (URiM) are of concern.

OBJECTIVE

Examine ITE trends longitudinally across post-graduate year (PGY) with gender and race/ethnicity.

DESIGN

Longitudinal analysis of resident ITE metrics at 7 internal medicine residency programs, 2014-2019. ITE trends across PGY of women and URiM residents compared to non-URiM men assessed via ANOVA. Those with ITE scores associated with less than 90% probability of passing the American Board of Internal Medicine certification exam (ABIM-CE) were identified and odds of being identified as at-risk between groups were assessed with chi square.

PARTICIPANTS

A total of 689 IM residents, including 330 women and URiM residents (48%).

MAIN MEASURES

ITE score KEY RESULTS: There was a significant difference in ITE score across PGY for women and URiM residents compared to non-URiM men (F_{(2, 1321)} 4.46, p=0.011). Adjusting for program, calendar year, and baseline ITE, women and URiM residents had smaller ITE score gains (adjusted mean change in score between PGY1 and PGY3 (se), non-URiM men 13.1 (0.25) vs women and URiM residents 11.4 (0.28), p<0.001). Women and URiM residents had greater odds of being at potential risk for not passing the ABIM-CE (OR 1.75, 95% CI 1.10 to 2.78) with greatest odds in PGY3 (OR 3.13, 95% CI 1.54 to 6.37).

CONCLUSION

Differences in ITE over training were associated with resident gender and race/ethnicity. Women and URiM residents had smaller ITE score gains across PGY translating into greater odds of potentially being seen as at-risk for not passing the ABIM-CE. Differences in ITE over training may reflect differences in experiences of women and URiM residents during training and may lead to further disparities.

Post graduate remediation programs in medicine: a scoping review

BACKGROUND

Recognizing that physicians may struggle to achieve knowledge, skills, attitudes and or conduct at one or more stages during their training has highlighted the importance of the 'deliberate practice of improving performance through practising beyond one's comfort level under guidance'. However, variations in physician, program, contextual and healthcare and educational systems complicate efforts to create a consistent approach to remediation. Balancing the inevitable disparities in approaches and settings with the need for continuity and effective oversight of the remediation process, as well as the context and population specific nature of remediation, this review will scrutinise the remediation of physicians in training to better guide the design, structuring and oversight of new remediation programs.

METHODS

Krishna's Systematic Evidence Based Approach is adopted to guide this Systematic Scoping Review (SSR in SEBA) to enhance the transparency and reproducibility of this review. A structured search for articles on remediation programs for licenced physicians who have completed their pre-registration postings and who are in training positions published between 1st January 1990 and 31st December 2021 in PubMed, Scopus, ERIC, Google Scholar, PsycINFO, ASSIA, HMIC, DARE and Web of Science databases was carried out. The included articles were concurrently thematically and content analysed using SEBA's Split Approach. Similarities in the identified themes and categories were combined in the Jigsaw Perspective and compared with the tabulated summaries of included articles in the Funnelling Process to create the domains that will guide discussions.

RESULTS

The research team retrieved 5512 abstracts, reviewed 304 full-text articles and included 101 articles. The domains identified were characteristics, indications, frameworks, domains, enablers and barriers and unique features of remediation in licenced physicians in training programs.

CONCLUSION

Building upon our findings and guided by Hauer et al. approach to remediation and Taylor and Hamdy's Multi-theories Model, we proffer a theoretically grounded 7-stage evidence-based remediation framework to enhance understanding of remediation in licenced physicians in training programs. We believe this framework can guide program design and reframe remediation's role as an integral part of training programs and a source of support and professional, academic, research, interprofessional and personal development.

Emotion in remediation: A scoping review of the medical education literature

OBJECTIVES

Remediation can be crucial and high stakes for medical learners, and experts agree it is often not optimally conducted. Research from other fields indicates that explicit incorporation of emotion improves education because of emotion's documented impacts on learning. Because this could present an important opportunity for improving remediation, we aimed to investigate how the literature on remediation interventions in medical education discusses emotion.

METHODS

The authors used Arksey and O'Malley's framework to conduct a scoping literature review of records describing remediation interventions in medical education, using PubMed, CINAHL Complete, ERIC, Web of Science and APA PsycInfo databases, including all English-language publications through 1 May 2020 meeting search criteria. They included publications discussing remediation interventions either empirically or theoretically, pertaining to physicians or physician trainees of any level. Two independent reviewers used a standardised data extraction form to report descriptive information; they reviewed included records for the presence of mentions of emotion, described the mentions and analysed results thematically.

RESULTS

Of 1644 records, 199 met inclusion criteria and were reviewed in full. Of those, 112 (56%) mentioned emotion in some way; others focused solely on cognitive aspects of remediation. The mentions of emotion fell into three themes based on when the emotion was cited as present: during regular coursework or practice, upon referral for remediation and during remediation. One-quarter of records (50) indicated potential intentional incorporation of emotion into remediation programme design, but they were non-specific as to how emotions related to the learning process itself.

CONCLUSION

Even though emotion is omnipresent in remediation, medical educators frequently do not factor emotion into the design of remediation approaches and rarely explicitly utilise emotion to improve the learning process. Applications from other fields may help medical educators leverage emotion to improve learning in remediation, including strategies to frame and design remediation.

Optimising the delivery of remediation programmes for doctors: A realist review

CONTEXT

Medical underperformance puts patient safety at risk. Remediation, the process that seeks to 'remedy' underperformance and return a doctor to safe practice, is therefore a crucially important area of medical education. However, although remediation is used in health care systems globally, there is limited evidence for the particular models or strategies employed. The purpose of this study was to conduct a realist review to ascertain why, how, in what contexts, for whom and to what extent remediation programmes for practising doctors work to restore patient safety.

METHOD

We conducted a realist literature review consistent with RAMESES standards. We developed a programme theory of remediation by carrying out a systematic search of the literature and through regular engagement with a stakeholder group. We searched bibliographic databases (MEDLINE, EMBASE, PsycINFO, HMIC, CINAHL, ERIC, ASSIA and DARE) and conducted purposive supplementary searches. Relevant sections of text relating to the programme theory were extracted and synthesised using a realist logic of analysis to identify context-mechanism-outcome configurations (CMOcs).

RESULTS

A 141 records were included. The majority of the studies were from North America (64%). 29 CMOcs were identified. Remediation programmes are effective when a doctor's insight and motivation are developed and behaviour change reinforced. Insight can be developed by providing safe spaces, using advocacy to promote trust and framing feedback sensitively. Motivation can be enhanced by involving the doctor in remediation planning, correcting causal attribution, goal setting and destigmatising remediation. Sustained change can be achieved by practising new behaviours and skills, and through guided reflection.

CONCLUSION

Remediation can work when it creates environments that trigger behaviour change mechanisms. Our evidence synthesis provides detailed recommendations on tailoring implementation and design strategies to improve remediation interventions for doctors.

Remediation of Underperformance in Surgical Trainees - A Scoping Review

OBJECTIVE

Surgical trainees with significant underperformance require formal support to return to an expected standard, termed remediation. The aim of this scoping review was to define remediation interventions, approaches, and contexts.

DESIGN

Following scoping review protocols, we set out to identify the evidence-base for remediation of surgical trainees, outline key concepts and uncover areas to stimulate further research.

RESULTS

From a screen of 80 articles, 24 reported on remediation of surgical trainees. Most were from medical journals (n = 21, 88%) and published in the United States (n = 20, 83%). Ten articles (41%) reported outcomes of remediation of a trainee cohort and 7 (19%) were survey reports from surgical directors. The remainder were a mix of commentaries, editorials or reviews. Thirteen articles (54%) described trainees with deficiencies in multiple competencies, 8 articles (33%) had a singular focus on academic performance, and 1 article (3%) on technical skills. All articles used targeted individualized remediation strategies, a range of intervention methods (some multimodal) and recommended a 6- to 12-month period of remediation (n = 7, 29%). The program director was often the only supervisor (n = 12, 50%). One article reported trainees' perspective of the process and one used educational theory to inform remediation.

CONCLUSIONS

Data with clearly reported outcomes were limited, but we found that targeted, individualized, multimodal and long-term remediation covering a range of competencies have been reported in the literature for surgical trainees. There is a need for development of explicit frameworks, strengthen the support for supervisors and trainees and further apply educational theory to develop better interventions that remediate deficiencies for all competencies.

Remediation programmes for practising doctors to restore patient safety: the RESTORE realist review

Background

An underperforming doctor puts patient safety at risk. Remediation is an intervention intended to address underperformance and return a doctor to safe practice. Used in health-care systems all over the world, it has clear implications for both patient safety and doctor retention in the workforce. However, there is limited evidence underpinning remediation programmes, particularly a lack of knowledge as to why and how a remedial intervention may work to change a doctor’s practice.

Objectives

To (1) conduct a realist review of the literature to ascertain why, how, in what contexts, for whom and to what extent remediation programmes for practising doctors work to restore patient safety; and (2) provide recommendations on tailoring, implementation and design strategies to improve remediation interventions for doctors.

Design

A realist review of the literature underpinned by the Realist And MEta-narrative Evidence Syntheses: Evolving Standards quality and reporting standards.

Data sources

Searches of bibliographic databases were conducted in June 2018 using the following databases: EMBASE, MEDLINE, Cumulative Index to Nursing and Allied Health Literature, PsycINFO, Education Resources Information Center, Database of Abstracts of Reviews of Effects, Applied Social Sciences Index and Abstracts, and Health Management Information Consortium. Grey literature searches were conducted in June 2019 using the following: Google Scholar (Google Inc., Mountain View, CA, USA), OpenGrey, NHS England, North Grey Literature Collection, National Institute for Health and Care Excellence Evidence, Electronic Theses Online Service, Health Systems Evidence and Turning Research into Practice. Further relevant studies were identified via backward citation searching, searching the libraries of the core research team and through a stakeholder group.

Review methods

Realist review is a theory-orientated and explanatory approach to the synthesis of evidence that seeks to develop programme theories about how an intervention produces its effects. We developed a programme theory of remediation by convening a stakeholder group and undertaking a systematic search of the literature. We included all studies in the English language on the remediation of practising doctors, all study designs, all health-care settings and all outcome measures. We extracted relevant sections of text relating to the programme theory. Extracted data were then synthesised using a realist logic of analysis to identify context–mechanism–outcome configurations.

Results

A total of 141 records were included. Of the 141 studies included in the review, 64% related to North America and 14% were from the UK. The majority of studies (72%) were published between 2008 and 2018. A total of 33% of articles were commentaries, 30% were research papers, 25% were case studies and 12% were other types of articles. Among the research papers, 64% were quantitative, 19% were literature reviews, 14% were qualitative and 3% were mixed methods. A total of 40% of the articles were about junior doctors/residents, 31% were about practicing physicians, 17% were about a mixture of both (with some including medical students) and 12% were not applicable. A total of 40% of studies focused on remediating all areas of clinical practice, including medical knowledge, clinical skills and professionalism. A total of 27% of studies focused on professionalism only, 19% focused on knowledge and/or clinical skills and 14% did not specify. A total of 32% of studies described a remediation intervention, 16% outlined strategies for designing remediation programmes, 11% outlined remediation models and 41% were not applicable. Twenty-nine context–mechanism–outcome configurations were identified. Remediation programmes work when they develop doctors’ insight and motivation, and reinforce behaviour change. Strategies such as providing safe spaces, using advocacy to develop trust in the remediation process and carefully framing feedback create contexts in which psychological safety and professional dissonance lead to the development of insight. Involving the remediating doctor in remediation planning can provide a perceived sense of control in the process and this, alongside correcting causal attribution, goal-setting, destigmatising remediation and clarity of consequences, helps motivate doctors to change. Sustained change may be facilitated by practising new behaviours and skills and through guided reflection.

Limitations

Limitations were the low quality of included literature and limited number of UK-based studies.

Future work

Future work should use the recommendations to optimise the delivery of existing remediation programmes for doctors in the NHS.

Study registration

This study is registered as PROSPERO CRD42018088779.

Funding

This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 9, No. 11. See the NIHR Journals Library website for further project information.

The Surgical Knowledge "Growth Curve": Predicting ABSITE Scores and Identifying "At-Risk" Residents

OBJECTIVE

Resident performance on the American Board of Surgery In-Training Examination (ABSITE) is used for evaluation of surgical knowledge and guides resident selection for institutional remediation programs. Remediation thresholds have historically been based on ABSITE percentile scores; however, this does not account for predictors that can impact a resident's exam performance. We sought to identify predictors of yearly ABSITE performance to help identify residents "at-risk" for performing below their expected growth trajectory.

DESIGN

The knowledge of the residents, as measured by standardized ABSITE scores, was modeled as a function of the corresponding postgraduate year via a linear mixed effects regression model. Additional model covariates included written USMLE-1-3 examination scores, gender, number of practice questions completed, and percentage correct of practice questions. For each resident, the predicted ABSITE standard score along with a 95% bootstrap prediction interval was obtained. Both resident-specific and population-level predictions for ABSITE standard scores were also estimated.

SETTING

The study was conducted at a single, large academic medical center (Massachusetts General Hospital, Boston, MA).

PARTICIPANTS

Six years of general surgery resident score reports at a single institution between 2014 and 2019 were deidentified and analyzed.

RESULTS

A total of 376 score reports from 130 residents were analyzed. Covariates that had a significant effect on the model included USMLE-1 score (PGY1: p = 0.013; PGY2: p = 0.007; PGY3: p = 0.011), USMLE-2 score (PGY1: p < 0.001; PGY2: p < 0.001; PGY3: p < 0.001; PGY4: p < 0.001; PGY5: p = 0.032), male gender (PGY1: p = 0.003; PGY2: p < 0.001; PGY3: p < 0.001; PGY4: p = 0.008), and number of practice questions completed (p=0.003). Five residents were identified as having "fallen off" their predicted knowledge curve, including a single resident on 2 occasions. Population prediction curves were obtained at 7 different covariate percentile levels (5%, 10%, 25%, 50%, 75%, 90%, and 95%) that could be used to plot predicted resident knowledge progress.

CONCLUSION

Performance on USMLE-1 and -2 examinations, male gender, and number of practice questions completed were positive predictors of ABSITE performance. Creating residency-wide knowledge growth curves as well as individualized predictive ABSITE performance models allows for more efficient identification of residents potentially at risk for poor ABSITE performance and structured monitoring of surgical knowledge progression.

Learning to Read: Successful Program-Based Remediation Using the Surgical Council on Resident Education (SCORE) Curriculum

BACKGROUND

The Surgical Council on Resident Education (SCORE) curriculum is aligned with the American Board of Surgery (ABS) objectives. Our program adopted the SCORE curriculum in 2015 after poor ABS In-Training Examination (ABSITE) performance and lowest quartile ABS Certifying Exam (CE) and Qualifying Exam (QE) first-time pass rates. We examined the association of SCORE use with ABSITE performance and ABS board exam first-time pass rate.

STUDY DESIGN

At a single institution, a retrospective review of surgery residents' SCORE metrics and ABSITE percentile was conducted for academic years 2015 to 2019. Metrics analyzed on the SCORE web portal were mean total minutes and total visits per resident for all residents using SCORE that year. First-time pass rates of the ABS QE and CE were examined from 2013 to 2019. Chi-square and linear regression analysis were performed, and a 95% level of confidence was assumed (alpha = 0.05).

RESULTS

Yearly data from categorical general surgery residents showed a significant increase in total minutes, total visits, and ABSITE percentile. Combined first time pass rates for the ABS QE and CE significant increased from 70.8% in 2013 to 2015 to 93.9% in 2016 to 2019 (p = 0.018).

CONCLUSIONS

Increased longitudinal use of the SCORE curriculum was associated with programmatic improvements in ABSITE performance and ABS board exam first-time pass rate.

Design and implementation of an academic enrichment program to improve performance on the internal medicine in-training exam

The internal medicine In-Training Exam (ITE) is administered at residency training programs to assess medical knowledge. Our internal medicine residency program witnessed a performance decline on the ITE between 2011 and 2014. The goal of this quality improvement project was to improve medical knowledge among residents as measured by an improvement in performance on the ITE, through the design and implementation of an Academic Enrichment Program (AEP). The AEP was designed in 2014-2015, and entailed a multipronged approach, including strengthening and tailoring of the didactic curriculum, establishment of a minimum conference attendance rate, and adoption of the New England Journal of Medicine Knowledge-Plus Internal Medicine Board Review platform. Residents performing below a pre-specified percentile rank cutoff on the previous year's ITE in any of the 12 content areas were required to complete a pre-specified percentage of the question bank in that specific topic. We examined a total of 164 residents enrolled in our program under the categorical training track. The mean (± SEM) ITE percentile for the 12 content areas increased significantly from calendar years 2011-2014 to 2015-2018, reflecting implementation of the AEP (p < 0.001). In brief, compared to the AEP-unexposed graduating classes of residents, the AEP-exposed graduating classes of residents displayed a significant improvement in the mean ITE percentile rank. This quality improvement project was carried out at a single institution. The implementation of a structured academic enrichment program significantly improves performance on the ITE.

Remediation of learners struggling with communication skills: a systematic review

BACKGROUND

Communication skills is a core area of competency for healthcare practitioners. However, trainees deficient in those skills are not identified early enough to address the deficiency. Furthermore, faculty often struggle to identify effective remediation strategies for those who fail to meet expectations. We undertook a systematic review to determine which assessment methods are appropriate to identify learners that struggle with communication skills and the strategies used to remediate them.

METHODS

The literature was searched from January 1998 through to May 2019 using academic databases and grey literature. Trainees were defined as healthcare practitioners in undergraduate, graduate and continuing education. Characteristics of studies, assessment and intervention strategies and outcomes were synthesized qualitatively and summarized in tables.

RESULTS

From an initial 1636 records, 16 (1%) studies met the review criteria. Majority of the learners were medical students. A few studies (44%) included students from other disciplines, residents and physicians in practice. The remediation programs, in the studies, ranged from 1 week to 1 year. Around half of the studies focused solely on learners struggling with communication skills. The majority of studies used a format of a clinical OSCE to identify struggling learners. None of the studies had a single intervention strategy with the majority including an experiential component with feedback.

CONCLUSIONS

A few studies collectively described the diagnosis, remediation intervention and the assessment of the outcomes of remediation of communication skills. For a remediation strategy to be successful it is important to ensure: (i) early identification and diagnosis, (ii) the development of an individualized plan and (iii) providing reassessment with feedback to the learner.

Interventions for undergraduate and postgraduate medical learners with academic difficulties: A BEME systematic review: BEME Guide No. 56

Background: Clinical teachers often struggle to report unsatisfactory trainee performance, partly because of a lack of evidence-based remediation options. Objectives: To identify interventions for undergraduate (UG) and postgraduate (PG) medical learners experiencing academic difficulties, link them to a theory-based framework and provide literature-based recommendations around their use. Methods: This systematic review searched MEDLINE, CINAHL, EMBASE, ERIC, Education Source and PsycINFO (1990-2016) combining these concepts: medical education, professional competence/difficulty and educational support. Original research/innovation reports describing intervention(s) for UG/PG medical learners with academic difficulties were included. Data extraction employed Michie's Behavior Change Techniques (BCT) Taxonomy and program evaluation models from Stufflebeam and Kirkpatrick. Quality appraisal used the Mixed Methods Appraisal Tool (MMAT). The authors synthesized extracted evidence by adapting the GRADE approach to formulate recommendations. Results: Sixty-eight articles met the inclusion criteria, most commonly addressing knowledge (66.2%), skills (53.9%) and attitudinal problems (26.2%), or learner personal issues (41.5%). The most common BCTs were Shaping knowledge, Feedback/monitoring, and Repetition/substitution. Quality appraisal was variable (MMAT 0-100%). A thematic content analysis identified 109 interventions (UG: n = 84, PG: n = 58), providing 24 strong, 48 moderate, 26 weak and 11 very weak recommendations. Conclusion: This review provides a repertoire of literature-based interventions for teaching/learning, faculty development, and research purposes.

A Structured Remediation Program Results in Durable Improvement of American Board of Surgery In-Training Examination (ABSITE®) Performance

The objective of this study was to assess the results of a structured remediation program for surgical residents scoring poorly on the American Board of Surgery In-Training Examination®. Residents scoring below the threshold (<20th percentile) were placed in a remediation program consisting of textbook-based review and mandatory weekly quizzes overseen by the program director. The effects of the program were assessed over a five-year period (2014-2018). Of 168 tests taken, 23 (13.7%) instances of scoring below the threshold occurred (average percentile 11, range 1–19). Postremediation scores in the subsequent year improved with a median (IQR) percentile increase of 29 [23.5, 46] ( P < 0.001). Two residents scored below the threshold twice; all others continued to score above the threshold. Participation in a structured remediation program results in dramatic improvement in American Board of Surgery In-Training Examination® performance, and a single occurrence of poor performance does not necessarily lead to a permanent handicap in academic performance.

Patient-Centered Tablet Application for Improving Medication Adherence after a Drug-Eluting Stent

BACKGROUND/AIMS

This study's objective was to evaluate a patient-centered educational electronic tablet application, "My Interventional Drug-Eluting Stent Educational App" (MyIDEA) to see if there was an increase in patient knowledge about dual antiplatelet therapy (DAPT) and medication possession ratio (MPR) compared to treatment as usual.

METHODS

In a pilot project, 24 elderly (≥50 years old) research participants were recruited after a drug-eluting stent. Eleven were randomized to the control arm and 13 to the interventional arm. All the participants completed psychological and knowledge questionnaires. Adherence was assessed through MPR, which was calculated at 3 months for all participants who were scheduled for second and third follow-up visits.

RESULTS

Relative to control, the interventional group had a 10% average increase in MPR. As compared to the interventional group, more patients in the control group had poor adherence (<80% MPR). The psychological data revealed a single imbalance in anxiety between the control and interventional groups. On average, interventional participants spent 21 min using MyIDEA.

DISCUSSION

Consumer health informatics has enabled us to engage patients with their health data using novel methods. Consumer health technology needs to focus more on patient knowledge and engagement to improve long-term health. MyIDEA takes a unique approach in targeting DAPT from the onset.

CONCLUSION

MyIDEA leverages patient-centered information with clinical care and the electronic health record highlighting the patients' role as a team member in their own health care. The patients think critically about adverse events and how to solve issues before leaving the hospital.

Using an Accountability Program to Improve Psychiatry Resident Scores on In-Service Examinations

BACKGROUND

The Psychiatry Resident-In-Training Examination (PRITE) is a standardized examination that measures residents' educational progress during residency training. It also serves as a moderate-to-strong predictor of later performance on the board certification examination.

OBJECTIVE

This study evaluated the effectiveness of an accountability program used by a public psychiatric hospital to increase its residents' PRITE scores.

METHODS

A series of consequences and incentives were developed based on levels of PRITE performance. Poor performance resulted in consequences, including additional academic assignments. Higher performance led to residents earning external moonlighting privileges. Standardized PRITE scores for all residents (N = 67) over a 10-year period were collected and analyzed. The PRITE examination consists of 2 subscales-psychiatry and neurology. Change in the overall level of PRITE scores following the implementation of the accountability program was estimated using a discontinuous growth curve model for each subscale.

RESULTS

Standardized scores on the psychiatry subscale were 51.09 points, approximately 0.50 SD change, which was higher after the accountability program was implemented. Standardized scores on the neurology subscale did not change.

CONCLUSIONS

An accountability program that assigns consequences based on examination performance may be moderately successful in improving scores on the psychiatry subscale scores of the PRITE. This likely has longer-term benefits for residents due to the relationship between PRITE and board certification examination performance.

My Interventional Drug-Eluting Stent Educational App (MyIDEA): Patient-Centered Design Methodology

Background

Patient adherence to medication regimens is critical in most chronic disease treatment plans. This study uses a patient-centered tablet app, “My Interventional Drug-Eluting Stent Educational App (MyIDEA).” This is an educational program designed to improve patient medication adherence.

Objective

Our goal is to describe the design, methodology, limitations, and results of the MyIDEA tablet app. We created a mobile technology-based patient education app to improve dual antiplatelet therapy adherence in patients who underwent a percutaneous coronary intervention and received a drug-eluting stent.

Methods

Patient advisers were involved in the development process of MyIDEA from the initial wireframe to the final launch of the product. The program was restructured and redesigned based on the patient advisers’ suggestions as well as those from multidisciplinary team members. To accommodate those with low health literacy, we modified the language and employed attractive color schemes to improve ease of use. We assumed that the target patient population may have little to no experience with electronic tablets, and therefore, we designed the interface to be as intuitive as possible.

Results

The MyIDEA app has been successfully deployed to a low-health-literate elderly patient population in the hospital setting. A total of 6 patients have interacted with MyIDEA for an average of 17.6 minutes/session.

Conclusions

Including patient advisers in the early phases of a mobile patient education development process is critical. A number of changes in text order, language, and color schemes occurred to improve ease of use. The MyIDEA program has been successfully deployed to a low-health-literate elderly patient population. Leveraging patient advisers throughout the development process helps to ensure implementation success.

Interventions that affect resident performance on the American Board of Surgery In-Training Examination: a systematic review

OBJECTIVE

To systematically review the published literature on the effectiveness of interventions intended to improve residents' American Board of Surgery In-Training Examination (ABSITE) performances.

DESIGN

A systematic review was conducted by 2 independent investigators to identify all publications that examined the effect of specific interventions on residents' ABSITE performances from 1975 to 2013.

RESULTS

Overall, 26 published articles met study criteria. Structured reading programs and setting clear expectations with mandatory remedial programs were consistently effective in improving ABSITE performance, whereas the effect of didactic teaching conferences and problem-based learning groups was mixed. There was marked heterogeneity in the usage of study designs and reporting of results.

CONCLUSIONS

Structured reading programs and mandatory remedial programs appear to be consistently effective measures that can improve residents' ABSITE performances. There is a need for improved study design and reporting in future research conducted in this field.

Surgery residency curriculum examination scores predict future American Board of Surgery in-training examination performance

IMPORTANCE

A protected block curriculum (PBC) with postcurriculum examinations for all surgical residents has been provided to assure coverage of core curricular topics. Biannual assessment of resident competency will soon be required by the Next Accreditation System.

OBJECTIVE

To identify opportunities for early medical knowledge assessment and interventions, we examined whether performance in postcurriculum multiple-choice examinations (PCEs) is predictive of performance in the American Board of Surgery In-Training Examination (ABSITE) and clinical service competency assessments.

DESIGN

Retrospective single-institutional education research study.

SETTING

Academic general surgery residency program.

PARTICIPANTS

A total of 49 surgical residents.

INTERVENTION

Data for PGY1 and PGY2 residents participating in the 2008 to 2012 PBC are included. Each resident completed 6 PCEs during each year.

MAIN OUTCOME MEASURES

The results of 6 examinations were correlated to percentage-correct ABSITE scores and clinical assessments based on the 6 Accreditation Council for Graduate Medical Education core competencies. Individual ABSITE performance was compared between PGY1 and PGY2. Statistical analysis included multivariate linear regression and bivariate Pearson correlations.

RESULTS

A total of 49 residents completed the PGY1 PBC and 36 completed the PGY2 curriculum. Linear regression analysis of percentage-correct ABSITE and PCE scores demonstrated a statistically significant correlation between the PGY1 PCE 1 score and the subsequent PGY1 ABSITE score (p = 0.037, β = 0.299). Similarly, the PGY2 PCE 1 score predicted performance in the PGY2 ABSITE (p = 0.015, β = 0.383). The ABSITE scores correlated between PGY1 and PGY2 with statistical significance, r = 0.675, p = 0.001. Performance on the 6 Accreditation Council for Graduate Medical Education core competencies correlated between PGY1 and PGY2, r = 0.729, p = 0.001, but did not correlate with PCE scores during either years.

CONCLUSIONS AND RELEVANCE

Within a mature PBC, early performance in a PGY1 and PGY2 PCE is predictive of performance in the respective ABSITE. This information can be used for formative assessment and early remediation of residents who are predicted to be at risk for poor performance in the ABSITE.

Focused Board Intervention (FBI): A Remediation Program for Written Board Preparation and the Medical Knowledge Core Competency

BACKGROUND

Residents deemed at risk for low performance on standardized examinations require focused attention and remediation.

OBJECTIVE

To determine whether a remediation program for residents identified as at risk for failure on the Emergency Medicine (EM) Written Board Examination is associated with improved outcomes.

INTERVENTION

All residents in 8 classes of an EM 1-3 program were assessed using the In-Training Examination. Residents enrolled in the Focused Board Intervention (FBI) remediation program based on an absolute score on the EM 3 examination of <70% or a score more than 1 SD below the national mean on the EM 1 or 2 examination. Individualized education plans (IEPs) were created for residents in the FBI program, combining self-study audio review lectures with short-answer examinations. The association between first-time pass rate for the American Board of Emergency Medicine (ABEM) Written Qualifying Examination (WQE) and completion of all IEPs was examined using the χ(2) test.

RESULTS

Of the 64 residents graduating and sitting for the ABEM examination between 2000 and 2008, 26 (41%) were eligible for the program. Of these, 10 (38%) residents were compliant and had a first-time pass rate of 100%. The control group (12 residents who matched criteria but graduated before the FBI program was in place and 4 who were enrolled but failed to complete the program) had a 44% pass rate (7 of 16), which was significantly lower (χ(2)  =  8.6, P  =  .003).

CONCLUSIONS

The probability of passing the ABEM WQE on the first attempt was improved through the completion of a structured IEP.

The remediation challenge: theoretical and methodological insights from a systematic review

OBJECTIVES

Remediation is usually offered to medical students and doctors in training who underperform on written or clinical examinations. However, there is uncertainty and conflicting evidence about the effectiveness of remediation. The aim of this systematic review was to synthesise the available evidence to clarify how and why remediation interventions may have worked in order to progress knowledge on this topic.

METHODS

The MEDLINE, EMBASE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), ERIC (Educational Resources Information Centre), Web of Science and Scopus databases were searched for papers published from 1984 to April 2012, using the search terms 'remedial teaching', 'education', 'medical', 'undergraduate'/or 'clinical clerkship'/or 'internship and residency', 'at risk' and 'struggling'. Only studies that included an intervention, then provided retest data, and reported at least one outcome measure of satisfaction, knowledge, skills or effects on patients were eligible for inclusion. Studies of practising doctors were excluded. Data were abstracted independently in duplicate for all items. Coding differences were resolved through discussion.

RESULTS

Thirty-one of 2113 studies met the review criteria. Most studies were published after 2000 (n=24, of which 12 were published from 2009 onwards), targeted medical students (n=22) and were designed to improve performance on an immediately subsequent examination (n=22). Control or comparison groups, conceptual frameworks, adequate sample sizes and long-term follow-up measures were rare. In studies that included long-term follow-up, improvements were not sustained. Intervention designs tended to be highly complex, but their design or reporting did not enable the identification of the active components of the remedial process.

CONCLUSIONS

Most remediation interventions in medical education focus on improving performance to pass a re-sit of an examination or assessment and provide no insight into what types of extra support work, or how much extra teaching is critical, in terms of developing learning. More recent studies are generally of better quality. Rigorous approaches to developing and evaluating remediation interventions are required.

Internet-based learning and applications for critical care medicine

PURPOSE

Recent changes in duty hour allowances and economic constraints are forcing a paradigm shift in graduate medical education in the United States. Internet-based learning is a rapidly growing component of postgraduate medical education, including the field of critical care medicine. Here, we define the key concepts of Internet-based learning, summarize the current literature, and describe how Internet-based learning may be uniquely suited for the critical care provider.

METHODS

A MEDLINE/PubMed search from January 2000 to July 2011 using the search terms: "e-learning," "Web-based learning," "computer-aided instruction," "adult learning," "knowledge retention," "intensive care," and "critical care."

RESULTS

The growth of the Internet is marked by the development of new technologies, including more user-derived tools. Nonmedical fields have embraced Internet-based learning as a valuable teaching tool. A recent meta-analysis described Internet-based learning in the medical field as being more effective than no intervention and likely as efficacious as traditional teaching methods. Web sites containing interactive features are aptly suited for the adult learner, complementing the paradigm shift to more learner-centered education. Interactive cases, simulators, and games may allow for improvement in clinical care. The total time spent utilizing Internet-based resources, as well as the frequency of returning to those sites, may influence educational gains.

CONCLUSION

Internet-based learning may provide an opportunity for assistance in the transformation of medical education. Many features of Web-based learning, including interactivity, make it advantageous for the adult medical learner, especially in the field of critical care medicine, and further work is necessary to develop a robust learning platform incorporating a variety of learning modalities for critical care providers.

Pediatric board review course for residents "at risk"

OBJECTIVE

There is a correlation between American Board of Pediatrics In-Training and General Pediatrics Certifying Examination scores. A course targeted mainly for residents "at-risk" based on in-training scores may improve the outcome of the certification exam. The objective of the study was to evaluate the impact of a board review course on pediatric board certification scores, particularly for residents at-risk.

METHODS

Residents "at-risk" taking the course (Yes) were compared to residents not taking the course who were not at-risk (No) as well as to a cohort of residents predating the onset of the course with mixed risk (Comparison). Analyses compared board exam scores and passing rates in the three groups and in the subcategory of residents who scored </=300 on the in-training exam in each group.

RESULTS

Standard scores and pass rates were higher for the residents not at risk, middle for residents of mixed risk and lowest for residents at risk. When analyzing the subset of scores for those with in-training score of <300, the scores were higher for the residents in the course than those who never took it. Also, residents with in-training score of <300 achieved higher average point score gains when comparing the certification exam to their last in-training score. Linear regression analysis showed that not delaying board certification, third year in-training scores and taking the board review course were significantly associated with higher board certification scores.

CONCLUSIONS

Pediatric residents at risk for not passing their Pediatric board certification exams benefited from a board review course.

Correlation between an email based board review program and American Board of Pediatrics General Pediatrics Certifying Examination scores

OBJECTIVE

To investigate the impact of a weekly email based board review course on individual resident performance on the American Board of Pediatrics (ABP) General Pediatrics Certifying Examination for pediatric residents and, specifically, residents with low ABP In-training Examination (ITE) scores.

METHODS

Weekly board-type questions were emailed to all pediatric residents from 2004-2007. Responses to board-type questions were tracked, recorded, and correlated with ITE scores and ABP General Pediatrics Certifying Examination Scores.

RESULTS

With regard to total number of questions answered, only total number of questions answered correctly had a significant positive correlation with standard board scores (n = 71, r = 0.24, p = 0.047). For "at risk" residents with ITE scores <or= 200 (n = 21), number of questions answered in PL 3 year (r = 0.51, p = 0.018) and number of questions answered correctly for all PL years (r = 0.59, p = 0.005) had significant positive correlations with standard board scores.

CONCLUSIONS

Participating regularly in the email-based board review course, answering board style questions, and answering correctly to board style questions were associated with higher standard board scores. This benefit existed for all but was especially prominent among those with poor in-training examination scores.