Development of the objective structured system-interaction examination

Where do we go from here? Moving from systems-based practice process measures to true competency via developmental milestones

For many educators it has been challenging to meet the Accreditation Council for Graduate Medical Education's requirements for teaching systems-based practice (SBP). An additional layer of complexity for educators is evaluating competency in SBP, despite milestones and entrustable professional activities (EPAs). In order to address this challenge, the authors present the results of a literature review for how SBP is currently being taught and a series of recommendations on how to achieve competency in SBP for graduate medical trainees with the use of milestones. The literature review included 29 articles and demonstrated that only 28% of the articles taught more than one of the six core principles of SBP in a meaningful way. Only 7% of the articles received the highest grade of A. The authors summarize four guiding principles for creating a competency-based curriculum that is in alignment with the Next Accreditation System (NAS): 1) the curriculum needs to include all of the core principles in that competency, 2) the objectives of the curriculum should be driven by clinical outcomes, 3) the teaching modalities need to be interactive and clinically relevant, and 4) the evaluation process should be able to measure competency and be directly reflective of pertinent milestones and/or EPAs. This literature review and the provided guiding principles can guide other residency educators in their development of competency-based curricula that meets the standards of the NAS.

Internal medicine postgraduate training and assessment of patient handoff skills

BACKGROUND

Effective communication during patient care transitions is essential for high-quality patient care.

OBJECTIVE

The purpose of this study was (1) to objectively assess patient handoff skills of internal medicine residents, and (2) to evaluate correlations between clinical experience and patient handoff skill self-assessment with directly observed skill.

METHODS

We studied simulated patient handoffs in postgraduate year (PGY)-1 and PGY-2 residents between July 2011 and September 2011, using a standardized scenario in an observed structured handoff exam (OSHE). Our design was a posttest-only, with nonequivalent groups. Assessment used a previously published checklist for evaluating handoff skills. Residents were asked about clinical experience with patient handoffs and about their self-confidence in performing a patient handoff independently. We evaluated between-group differences on OSHE checklist performance, patient handoff experience, and self-confidence and used multiple regression analyses to assess the association between performance, experience, and confidence.

RESULTS

Forty-seven PGY-1 residents and 38 PGY-2 residents completed the study. Interrater reliability was substantial (intraclass correlation  =  0.68). There was no significant difference in OSHE performance by PGY-1 residents (mean  =  79%, SD  =  4.6) and PGY-2 residents (mean  =  82%; SD  =  7.6; P  =  .07). The PGY-2 residents were significantly more experienced (P < .001) and confident (P < .001) than PGY-1 residents were, yet clinical experience and self-confidence did not significantly predict OSHE performance.

CONCLUSIONS

Clinical experience and self-assessment do not predict skills in simulated patient handoffs, and residents with substantial clinical experience still benefit from further skills development.

Assessing systems-based practice

The conceptual definition of systems-based practice (SBP) does not easily translate into directly observable actions or behaviors that can be easily assessed. At the Academic Emergency Medicine consensus conference on education research in emergency medicine (EM), a breakout group presented a review of the literature on existing assessment tools for SBP, discussed the recommendations for research tool development during breakout sessions, and developed a research agenda based on this discussion.

Systems-based practice assessed with a performance-based examination simulated and scored by standardized participants in the health care system: feasibility and psychometric properties

BACKGROUND

Systems-based practice is one of the six general competencies proposed by the Accreditation Council for Graduate Medical Education in their Outcome Project. However, little has been published on its assessment--possibly because the systems-based practice competency has been viewed as difficult to define and measure.

PURPOSE

The purpose of this study was to determine whether a full performance-based examination of systems-based practice cases simulated and scored by standardized participants in the health care system could feasibly be constructed and implemented that would provide reliable and valid measurements.

METHODS

In the 1st year of the project (2008), four systems-based practice cases were developed and pilot tested with 13 residents. Videotapes of residents were studied to develop an instrument for subsequent assessment of performance by standardized participants. In the 2nd year (2009), the examination was expanded to a full 12 cases, which were completed by 11 second-year residents, and psychometric analyses were performed on the scores.

RESULTS

The generalizability coefficient for the full 12-case examination based on scoring by standardized participants was .71, which is nearly equal to that based on scoring by faculty physician observers, which was .78. The correlation between total scores obtained with standardized participants and physician observers was .78.

CONCLUSIONS

A performance-based examination can provide a feasible and reliable assessment of systems-based practice. However, attempts to evaluate convergent validity and discriminant validity-by correlating systems-based practice performance assessments with mean global ratings of residents on the 6 competencies by faculty throughout training-were unsuccessful, due to a lack of independence between the rated dimensions.