Development and pilot testing of an OSCE for difficult conversations in surgical intensive care

Validation of a Surgical Objective Structured Clinical Examination (S-OSCE) Using Convergent, Divergent, And Trainee-Based Assessments of Fidelity

OBJECTIVE

Describe the validation of a surgical objective structured clinical examination (S-OSCE) for the purpose of competency assessment based on the Royal College of Canada's Can-MEDS framework.

DESIGN

A surgical OSCE was developed to evaluate the management of common orthopedic surgical problems. The scores derived from this S-OSCE were compared to Ottawa Surgical Competency Operating Room Evaluation (O-SCORE), a validated entrustability assessment, to establish convergent validity. The S-OSCE scores were compared to Orthopedic In-Training Examination (OITE) scores to evaluate divergent validity. Resident evaluations of the clinical encounter with a standardized patient and the operative procedure were scored on a 10-point Likert scale for fidelity.

SETTING

A tertiary level academic teaching hospital.

PARTICIPANTS

21 postgraduate year 2 to 5 trainees of a 5-year Canadian orthopedic residency program creating 160 operative case performances for review.

RESULTS

There were 5 S-OSCE days, over a 4-year period (2016-2019) encompassing a variety of surgical procedures. Performance on the S-OSCE correlated strongly with the O-SCORE (Pearson correlation coefficient 0.88), and a linear regression analysis correlated moderately with year of training (R² = 0.5345). The Pearson correlation coefficient between the S-OSCE and OITE scores was 0.57. There was a significant increase in the average OITE score after the introduction of the surgical OSCE. Resident fidelity ratings were available from 16 residents encompassing 8 different surgical cases. The average score for the overall simulation (8.0±1.6) was significantly higher than the cadaveric surgical simulation (6.5 ± 0.8) (p < 0.001) CONCLUSIONS: The S-OSCE scores correlate strongly with an established form of assessment demonstrating convergent validity. The correlation between the S-OSCE and OITE scores was less, demonstrating divergent validity. Although residents rank the overall simulation highly, the fidelity of the cadaveric simulation may need improvement. Administration of a surgical OSCE can be used to evaluate preoperative and intraoperative decision making and complement other forms of assessment.

Efficacy of an Online Curriculum for Perioperative Goals of Care and Code Status Discussions: A Randomized Controlled Trial

BACKGROUND

Preoperative goals of care (GOC) and code status (CS) discussions are important in achieving an in-depth understanding of the patient's care goals in the setting of a serious illness, enabling the clinician to ensure patient autonomy and shared decision making. Past studies have shown that anesthesiologists are not formally trained in leading these discussions and may lack the necessary skill set. We created an innovative online video curriculum designed to teach these skills. This curriculum was compared to a traditional method of learning from reading the medical literature.

METHODS

In this bi-institutional randomized controlled trial at 2 major academic medical centers, 60 anesthesiology trainees were randomized to receive the educational content in 1 of 2 formats: (1) the novel video curriculum (video group) or (2) journal articles (reading group). Thirty residents were assigned to the experimental video curriculum group, and 30 were assigned to the reading group. The content incorporated into the 2 formats focused on general preoperative evaluation of patients and communication strategies pertaining to GOC and CS discussions. Residents in both groups underwent a pre- and postintervention objective structured clinical examination (OSCE) with standardized patients. Both OSCEs were scored using the same 24-point rubric. Score changes between the 2 OSCEs were examined using linear regression, and interrater reliability was assessed using weighted Cohen's kappa.

RESULTS

Residents receiving the video curriculum performed significantly better overall on the OSCE encounter, with a mean score of 4.19 compared to 3.79 in the reading group. The video curriculum group also demonstrated statistically significant increased scores on 8 of 24 rubric categories when compared to the reading group.

CONCLUSIONS

Our novel video curriculum led to significant increases in resident performance during simulated GOC discussions and modest increases during CS discussions. Further development and refinement of this curriculum are warranted.

Postgraduate Assessment Approaches Across Surgical Specialties: A Systematic Review of the Published Evidence

PURPOSE

Ensuring competence for surgical trainees requires holistic assessment of the qualities and competencies necessary to practice safely and effectively. To determine the next steps toward achieving this aim, the authors conducted a systematic review to summarize and appraise the available evidence related to any assessment approach to postgraduate surgical training and to identify the dominant themes for assessment approaches across different specialties or countries.

METHOD

Medline and Embase were searched on January 10, 2019, without language or time restrictions. Any peer-reviewed study that described an assessment framework (in practice or novel) throughout postgraduate surgical training globally was included. An iterative review and thematic analysis were performed on full-text articles to determine assessment themes. Studies were then grouped by assessment themes. A tailored quality assessment of the studies included in the final analysis was conducted. Assessment themes and validity were compared across surgical specialties and countries.

RESULTS

From an initial 7,059 articles, 91 studies (evaluating 6,563 surgical trainees) were included in the final analysis. Ten defined assessment themes were extracted. Ten studies (11.0%) were deemed low risk of bias based on the quality assessment tool used and thus were determined to be high quality. Minor differences in assessment themes were observed between specialties and countries. Assessment themes neglected by individual surgical specialties and assessment themes that need validated assessment tools were identified.

CONCLUSIONS

This review highlights the low quality of evidence and fragmented efforts to develop and optimize surgical assessments. The minor differences observed demonstrate a common approach, globally and across specialties, related to surgical assessments. A paradigm shift in assessment approaches, which will require national and international collaboration, is required to optimize design and validation so that a comprehensive assessment of surgical competence can be implemented.

"Who wants me to do what?" varied expectations from key stakeholder groups in the surgical intensive care unit creates a challenging learning environment

BACKGROUND

Surgical intensive care units (SICU) require complex care from a multi-disciplinary team. Frequent changes in team members can lead to shifting expectations for junior general surgical trainees, which creates a challenging working and learning environment. We aim to identify expectations of junior surgery trainee's medical knowledge and technical/non-technical skills at the start of their SICU rotation. We hypothesize that expectations will not be consistent across SICU stakeholder groups.

METHODS

Twenty-eight individual semi-structured interviews were conducted with six SICU stakeholder groups at a medium-sized academic hospital. Expectations were identified from interview transcripts. Frequency counts were analyzed.

RESULTS

Forty-one expectations were identified. 4 expectations were identified by a majority of interviewees. Most expectations were identified by 7 or fewer interviewees. 23 (53%) expectations were shared by at least one stakeholder group. 2 (8%) expectations were shared by all groups.

CONCLUSIONS

SICU stakeholder groups identified ten medical knowledge, ten technical skill, and three non-technical skill expectations. Yet, few expectations were shared among the groups. Thus, SICU stakeholder groups have disparate expectations for surgery trainees in our SICU.

Professionalism and inter-communication skills (ICS): a multi-site validity study assessing proficiency in core competencies and milestones in medical learners

BACKGROUND

Interpersonal and Communication Skills (ICS) and Professionalism milestones are challenging to evaluate during medical training. Paucity in proficiency, direction and validity evidence of assessment tools of these milestones warrants further research. We validated the reliability of the previously-piloted Instrument for Communication skills and Professionalism Assessment (InCoPrA) in medical learners.

METHODS

This validity approach was guided by the rigorous Kane's Framework. Faculty-raters and standardized patients (SPs) used their respective InCoPrA sub-component to assess distinctive domains pertinent to ICS and Professionalism through multiple expert-built simulated-scenarios comparable to usual care. Evaluations included; inter-rater reliability of the faculty total score; the correlation between the total score by the SPs; and the average of the total score by two-faculty members. Participants were surveyed regarding acceptability, realism, and applicability of this experience.

RESULTS

Eighty trainees and 25 faculty-raters from five medical residency training sites participated. ICC of the total score between faculty-raters was generally moderate (ICC range 0.44-0.58). There was on average a moderate linear relationship between the SPs and faculty total scores (Pearson correlations range 0.23-0.44). Majority of participants ascertained receiving a meaningful, immediate, and comprehensive patient-faculty feedback.

CONCLUSIONS

This work substantiated that InCoPrA was a reliable, standardized, evidence-based, and user-friendly assessment tool for ICS and Professionalism milestones. Validating InCoPrA showed generally-moderate agreeability and high acceptability. Using InCoPrA also promoted engaging all stakeholders in medical education and training-faculty, learners, and SPs-using simulation-media as pathway for comprehensive feedback of milestones growth.

Resident Education in End-of-Life Communication and Management: Assessing Comfort Level to Enhance Competence and Confidence

OBJECTIVES

Our primary objective was to understand residents' baseline comfort with end-of-life (EOL) communication and management and to compare this with their comfort after completion of their surgical intensive care unit (SICU) rotation. We also evaluated the association between prior training with perceived level of comfort with EOL issues, and whether the resident believed in the concept of a "better death."

DESIGN, SETTING, PARTICIPANTS

As a quality improvement initiative, we conducted surveys of trainees before and after their rotation in the Yale New Haven Hospital SICU. Prerotation and postrotation surveys were administered to all residents who rotated during the 2016-2017 academic year and the first half of 2017-2018. The survey consisted of 34 questions querying residents on their level of training in EOL care, their comfort with management and discussions in different EOL domains, and their beliefs about what measures would have improved their ability to provide EOL care. Residents surveyed were from general surgery, emergency medicine, or anesthesia departments.

RESULTS AND CONCLUSIONS

Our study demonstrates that there is a significant correlation between resident comfort with EOL communication and experience providing EOL care. However, concepts in medicolegal aspects of palliative care could be taught through formal didactics, and structured training may allow residents the opportunity to reflect on the importance of a "better death."

Development of an Objective Structured Clinical Examination as a Component of Assessment for Initial Board Certification in Anesthesiology

With its first administration of an Objective Structured Clinical Examination (OSCE) in 2018, the American Board of Anesthesiology (ABA) became the first US medical specialty certifying board to incorporate this type of assessment into its high-stakes certification examination system. The fundamental rationale for the ABA's introduction of the OSCE is to include an assessment that allows candidates for board certification to demonstrate what they actually "do" in domains relevant to clinical practice. Inherent in this rationale is that the OSCE will capture competencies not well assessed in the current written and oral examinations-competencies that will allow the ABA to judge whether a candidate meets the standards expected for board certification more properly. This special article describes the ABA's journey from initial conceptualization through first administration of the OSCE, including the format of the OSCE, the process for scenario development, the standardized patient program that supports OSCE administration, examiner training, scoring, and future assessment of reliability, validity, and impact of the OSCE. This information will be beneficial to both those involved in the initial certification process, such as residency graduate candidates and program directors, and others contemplating the use of high-stakes summative OSCE assessments.

Effects of a System Thinking-Based Simulation Program for Congestive Heart Failure

This study evaluated a system thinking-based simulation program for the care of patients with congestive heart failure. Participants were 67 undergraduate nursing students from a nursing college in Seoul, South Korea. The experimental group was given a 4-hour system-thinking program and a 2-hour simulation program, whereas the control group had a 4-hour case study and a 2-hour simulation program. There were significant improvements in critical thinking in both groups, but no significant group differences between educational methods (F = 3.26, P = .076). Problem-solving ability in the experimental group was significantly higher than in the control group (F = 5.04, P = .028). Clinical competency skills in the experimental group were higher than in the control group (t = 2.12, P = .038). A system thinking-based simulation program is a more effective learning method in terms of problem-solving ability and clinical competency skills compared to the existing simulation program. Further research using a longitudinal study is needed to test the long-term effect of the intervention and apply it to the nursing curriculum.

Training Surgeons and Anesthesiologists to Facilitate End-of-Life Conversations With Patients and Families: A Systematic Review of Existing Educational Models

OBJECTIVE

Despite caring for patients near the end-of-life (EOL), surgeons and anesthesiologists report low confidence in their ability to facilitate EOL conversations. This discrepancy exists despite competency requirements and professional medical society recommendations. The objective of this systematic review is to identify articles describing EOL communication training available to surgeons and anesthesiologists, and to assess their methodological rigor to inform future curricular design and evaluation.

METHODS

This PRISMA-concordant systematic review identified English-language articles from PubMed, EMBASE, and manual review. Eligible articles included viewpoint pieces, and observational, qualitative, or case studies that featured an educational intervention for surgeons or anesthesiologists on EOL communication skills. Data on the study objective, setting, design, participants, intervention, and results were extracted and analyzed. The Newcastle-Ottawa Scale was used to assess methodological quality.

RESULTS

Database and manual search returned 2710 articles. A total of 2268 studies were screened by title and abstract, 46 reviewed in full-text, and 16 included in the final analysis. Fifteen studies were conducted exclusively in academic hospitals. Two studies included attending surgeons as participants; all others featured residents, fellows, or a mix thereof. Fifteen studies used simulated role-playing to teach and assess EOL communication skills. Measured outcomes included knowledge, attitudes, confidence, self-rated or observer-rated communication skills, and curriculum feedback; significance of results varied widely. Most studies lacked adequate methodological quality and appropriate control groups to be confident about the significance and applicability of their results.

CONCLUSIONS

There are few quality studies evaluating EOL communication training for surgeons and anesthesiologists. These programs frequently use role-playing to teach and assess EOL communication skills. More studies are needed to evaluate the effect of these interventions on patient outcomes. However, evaluating the effectiveness of these initiatives poses methodological challenges.

Development and Validation of a Family Meeting Assessment Tool (FMAT)

CONTEXT

A cornerstone procedure in Palliative Medicine is to perform family meetings. Learning how to lead a family meeting is an important skill for physicians and others who care for patients with serious illnesses and their families. There is limited evidence on how to assess best practice behaviors during end-of-life family meetings.

OBJECTIVES

Our aim was to develop and validate an observational tool to assess trainees' ability to lead a simulated end-of-life family meeting.

METHODS

Building on evidence from published studies and accrediting agency guidelines, an expert panel at our institution developed the Family Meeting Assessment Tool. All fourth-year medical students (MS4) and eight geriatric and palliative medicine fellows (GPFs) were invited to participate in a Family Meeting Objective Structured Clinical Examination, where each trainee assumed the physician role leading a complex family meeting. Two evaluators observed and rated randomly chosen students' performances using the Family Meeting Assessment Tool during the examination. Inter-rater reliability was measured using percent agreement. Internal consistency was measured using Cronbach α.

RESULTS

A total of 141 trainees (MS4 = 133 and GPF = 8) and 26 interdisciplinary evaluators participated in the study. Internal reliability (Cronbach α) of the tool was 0.85. Number of trainees rated by two evaluators was 210 (MS4 = 202 and GPF = 8). Rater agreement was 84%. Composite scores, on average, were significantly higher for fellows than for medical students (P < 0.001).

CONCLUSION

Expert-based content, high inter-rater reliability, good internal consistency, and ability to predict educational level provided initial evidence for construct validity for this novel assessment tool.

Developing and testing a comprehensive tool to assess family meetings: Empirical distinctions between high- and low-quality meetings

BACKGROUND

The heterogeneity with regard to findings on family meetings (or conferences) suggests a need to better understand factors that influence family meetings. While earlier studies have explored frequency or timing of family meetings, little is known about how factors (such as what is said during meetings, how it is said, and by whom) influence family meeting quality.

OBJECTIVES

(1) To develop an evaluation tool to assess family meetings (Phase 1); (2) to identify factors that influence meeting quality by evaluating 34 family meetings (Phase 2).

MATERIALS AND METHODS

For Phase 1, methods included developing a framework, cognitive testing, and finalizing the evaluation tool. The tool consisted of Facilitator Characteristics (i.e., gender, experience, and specialty of the person leading the meeting), and 22 items across 6 Meeting Elements (i.e., Introductions, Information Exchanges, Decisions, Closings, Communication Styles, and Emotional Support) and sub-elements. For Phase 2, methods included training evaluators, assessing family meetings, and analyzing data. We used Spearman's rank-order correlations to calculate meeting quality. Qualitative techniques were used to analyze free-text.

RESULTS

No Facilitator Characteristic had a significant correlation with meeting quality. Sub-elements related to communication style and emotional support most strongly correlated with high-quality family meetings, as well as whether "next steps" were outlined (89.66%) and whether "family understanding" was elicited (86.21%). We also found a significant and strong positive association between overall proportion scores and evaluators' ratings (r_s=0.731, p<0.001).

CONCLUSIONS

We filled a gap by developing an evaluation tool to assess family meetings, and we identified how what is said during meetings impacts quality.

An Agenda for Improving Perioperative Code Status Discussion

Code status discussions (CSDs) clarify patient preferences for cardiopulmonary resuscitation in the event of cardiac or respiratory arrest. CSDs are a key component of perioperative care, particularly at the end of life, and must be both patient-centered and shared. Physicians at all levels of training are insufficiently trained in and inappropriately perform CSD; this may be particularly true of perioperative physicians. In this article, we describe the difficulty of achieving a patient-centered, shared perioperative CSD in the case of a medical professional with a do-not-resuscitate order. We provide a brief background in cardiopulmonary resuscitation, do-not-resuscitate, and CSD before proposing an agenda for improving perioperative CSD.

Evaluation of palliative care training and skills retention by medical students

BACKGROUND

Training in palliative and end-of-life care has been introduced in medical education; however, the impact of such training and the retention of skills and knowledge have not been studied in detail. This survey study examines long-term follow-up on end-of-life communication skills training, evaluation, and skills retention in medical students.

MATERIALS AND METHODS

During the surgical clerkship, all third-year medical students received communication skills training in palliative care using simulated patients. The training involved three scenarios involving diverse surgical patients with conditions commonly encountered during the surgical clerkship. The students used web-based best practice guidelines to prepare for the patient encounters. The following communication abilities were evaluated: (1) giving bad news clearly and with empathy, (2) initiating death and dying conversations with patients and/or their family members, (3) discussing do not resuscitate status and exploring preferences for end-of-life care, and (4) initiating conversations regarding religious or spiritual values and practices. All students were surveyed after 1 year (12-24 mo) to ascertain: (1) the retention of skills and/or knowledge gained during this training, (2) application of these skills during subsequent clinical rotations, and (3) overall perception of the value added by the training to their undergraduate medical education. These results were correlated with residency specialty choice.

RESULTS

The survey was sent to all graduating fourth-year medical students (n = 105) in our program, of which 69 students responded to the survey (66% response rate). All respondents agreed that palliative care training is essential in medical school training. Seventy percent of the respondents agreed that the simulated encounters allowed development of crucial conversation skills needed for palliative/end-of-life care communications. The most useful part of the training was the deliberate practice of "giving bad news" (85%). Most of the respondents (80%) indicated retention of overall communication skills with regard to approach and useful phrases. Forty-five percent claimed retention of communication skills surrounding death and dying, and 44% claimed retention of end-of-life preferences/advance directives/do not resuscitate. Relatively few respondents (16%) retained skills regarding religious or spiritual values. There was no correlation between training evaluation/skill retention and the area of residency specialty the students pursued on graduation.

CONCLUSIONS

Early training in palliative and end-of-life care communication is feasible and effective during the surgical clerkship. Students highly valued the simulated patient and/or family discussions and retained most of the skills and knowledge from the experiential simulated encounters. However, students felt the skills developed could be reinforced with opportunities to observe their attending physicians or residents leading such discussions and involving students in such discussions as and when appropriate.

Cleft Lip Standardized Patient Examinations: The Role in Plastic Surgery Resident Education

BACKGROUND

Our institution has incorporated the use of objective structured clinical examinations (OSCE) in our residency curriculum. The OSCE provides trainee education and evaluation while addressing the six Accreditation Council for Graduate Medical Education (ACGME) core competencies required within training programs. We report our program's experience with the first cleft OSCE ever conducted.

METHODS

A validated method for administration of OSCEs currently used at our medical school was utilized for residents in postgraduate years (PGYs) 3 through 6. The video-recorded patient encounter involved a 1-month-old newborn with a unilateral cleft lip and palate and used standardized patient actors as parents. A post-encounter written exam assessed medical knowledge. A questionnaire regarding the utility of the exercise was administered to residents after the OSCE. Results were evaluated using analysis of variance (P < .05).

RESULTS

There was a positive correlation with increasing level of training in terms of medical knowledge (P < .04). Residents in PGY-3 and PGY-4 demonstrated lower understanding of the surgical markings and details of the lip repair compared with those in PGY-5 and PGY-6 (P < .03). All residents performed similarly on evaluation of the remaining ACGME core competencies. All residents agreed that this was a realistic and useful encounter.

CONCLUSION

Results of our cleft OSCE demonstrate that medical knowledge regarding the evaluation, management, and surgical repair of patients is less in midlevel residents. All residents expressed an interest in earlier exposure to pediatric patients in the training period. Although a cleft OSCE does not replace clinical rotations, it is a valuable adjunct to training and evaluation of trainees, particularly for junior residents.

Tough Conversations: Development of a Curriculum for Medical Students to Lead Family Meetings

Background:

Few educational interventions have been developed to teach Family Meeting (FM) communication skills at the undergraduate level. We developed an innovative curriculum to address this gap.

Methods:

Fourth year medical students during 2011-2013 (n = 674) completed training for conducting a FM. To assess the effectiveness of this training, students completed a FM Objective Structured Clinical Exam (OSCE) that included 15 domains rated on a 1-5 point Likert scale. Tasks included discussing prognosis, establishing goals of care and demonstrating conflict resolution skills. Students received one-to-one feedback from standardized family members and faculty observers. Group debriefings with faculty were held after the OSCE.

Results:

Analysis of faculty feedback narratives revealed four themes in which students required improvement: 1) Discussing prognosis, 2) Explaining palliative care/hospice, 3) Avoiding medical jargon, and 4) Discussing cultural/religious preferences. Evaluation total mean score was 28.2 (Min 15, Max 63; SD 7.57), and identified student’s need to; 1) Ask more about the degree of knowledge family members want, 2) Ask religious beliefs, and 3) Assess family members’ level of education ( p < 0.001). Qualitative analysis of group debriefings suggested that student perception of the OSCE experience was positive overall. Students found the case to be realistic and immediate feedback to be helpful.

Conclusions:

Conducting a FM is an advanced skill. This study shows that it is possible to train fourth year students to lead FMs and identify their strengths, needs using a FM OSCE.

An objective structured clinical exam to measure intrinsic CanMEDS roles

BACKGROUND

The CanMEDS roles provide a comprehensive framework to organize competency-based curricula; however, there is a challenge in finding feasible, valid, and reliable assessment methods to measure intrinsic roles such as Communicator and Collaborator. The objective structured clinical exam (OSCE) is more commonly used in postgraduate medical education for the assessment of clinical skills beyond medical expertise.

METHOD

We developed the CanMEDS In-Training Exam (CITE), a six-station OSCE designed to assess two different CanMEDS roles (one primary and one secondary) and general communication skills at each station. Correlation coefficients were computed for CanMEDS roles within and between stations, and for general communication, global rating, and total scores. One-way analysis of variance (ANOVA) was used to investigate differences between year of residency, sex, and the type of residency program.

RESULTS

In total, 63 residents participated in the CITE; 40 residents (63%) were from internal medicine programs, whereas the remaining 23 (37%) were pursuing other specialties. There was satisfactory internal consistency for all stations, and the total scores of the stations were strongly correlated with the global scores r=0.86, p<0.05. Noninternal medicine residents scored higher in terms of the Professional competency overall, whereas internal medicine residents scored significantly higher in the Collaborator competency overall.

DISCUSSION

The OSCE checklists developed for the assessment of intrinsic CanMEDS roles were functional, but the specific items within stations required more uniformity to be used between stations. More generic types of checklists may also improve correlations across stations.

CONCLUSION

An OSCE measuring intrinsic competence is feasible; however, further development of our cases and checklists is needed. We provide a model of how to develop an OSCE to measure intrinsic CanMEDS roles that educators may adopt as residency programs move into competency-based medical education.

Proficiency in identifying, managing and communicating medical errors: feasibility and validity study assessing two core competencies

BACKGROUND

Communication skills and professionalism are two competencies in graduate medical education that are challenging to evaluate. We aimed to develop, test and validate a de novo instrument to evaluate these two competencies.

METHODS

Using an Objective Standardized Clinical Examination (OSCE) based on a medication error scenario, we developed an assessment instrument that focuses on distinctive domains [context of discussion, communication and detection of error, management of error, empathy, use of electronic medical record (EMR) and electronic medical information resources (EMIR), and global rating]. The aim was to test feasibility, acceptability, and reliability of the method.

RESULTS

Faculty and standardized patients (SPs) evaluated 56 trainees using the instrument. The inter-rater reliability of agreement between faculty was substantial (Fleiss k = 0.71) and intraclass correlation efficient was excellent (ICC = 0.80). The measured agreement between faculty and SPs evaluation of resident was lower (Fleiss k = 0.36). The instrument showed good conformity (ICC = 0.74). The majority of the trainees (75 %) had satisfactory or higher performance in all six assessed domains and 86 % found the OSCE to be realistic. Sixty percent reported not receiving feedback on EMR use and asked for subsequent training.

CONCLUSION

An OSCE-based instrument using a medical error scenario can be used to assess competency in professionalism, communication, using EMRs and managing medical errors.

Evaluation of ethics education in obstetrics and gynecology residency programs

OBJECTIVE

The objective of the study was to assess the current status of ethics education in obstetrics-gynecology residency programs.

STUDY DESIGN

A cross-sectional, web-based survey was designed in conjunction with a professional survey laboratory at the University of Chicago. The survey was piloted with a convenience sample of clinical medical ethics fellows to assess question content and clarity. The survey was deployed by e-mail to all obstetrics-gynecology residency program directors. Descriptive statistics were used to analyze participant responses. The University of Chicago's Institutional Review Board deemed this study exempt from institutional review board formal review.

RESULTS

Of 242 eligible obstetrics-gynecology residency program directors, 118 (49%) completed the survey. Most respondents were from university-based programs (n = 78, 66%) that were not religiously affiliated (n = 98, 83%) and trained 4-6 residents per postgraduate year (n = 64, 70%). Although 50% of program directors (n = 60) reported having ethics as part of their core curriculum, most programs teach ethics in an unstructured manner. Fifty-seven percent of respondents (n = 66) stated their program dedicated 5 or fewer hours per year to ethics. The majority of program directors (n = 80, 73%) responded they would like more to a lot more ethics education and believed that ethics education should be required (n = 93, 85%) for residents to complete their training. Respondents identified that crowding in the curriculum was a significant barrier to increased ethics training (n = 50, 45%) and two-thirds (n = 74, 67%) reported a lack of faculty expertise as a moderate barrier to providing ethics education in the residency curriculum.

CONCLUSION

This study found that a lack of structured curricula, inadequate faculty expertise, and limited time were important barriers for ethics education in obstetrics-gynecology programs across the nation. Despite these existing challenges, program directors have a strong interest in increasing ethics education in residency training. Therefore, additional resources are needed to assist program directors in enhancing resident ethics education.

Communication skills training in surgical residency: a needs assessment and metacognition analysis of a difficult conversation objective structured clinical examination

BACKGROUND

The objective structured clinical examination (OSCE) can be used to evaluate the Accreditation Council for Graduate Medical Education Core Competencies of Professionalism and Interpersonal and Communication Skills. The aim of this study was to describe general surgery resident performance on a "difficult conversation" OSCE.

METHODS

In this prospective study, junior and senior residents participated in a 2-station OSCE. Junior stations involved discussing operative risks and benefits and breaking bad news. Senior stations involved discussing goals of care and discussing transition to comfort measures only status. Residents completed post-OSCE checklist and Likert-based self-evaluations of experience, comfort, and confidence. Trained standardized patients (SPs) evaluated residents using communication skill-based checklists and Likert-based assessments. Pearson correlation coefficients were determined between self-assessment and SP assessment. Mann-Whitney U tests were conducted between junior and senior resident variables, using α = 0.05.

RESULTS

There were 27 junior residents (age 28.1 ± 1.9 years [29.6% female]) and 27 senior residents (age 32.1 ± 2.5 years [26.9% female]). The correlation of self-assessment and SP assessment of overall communication skills by junior residents was -0.32 on the risks and benefits case and 0.07 on the breaking bad news case. The correlation of self-assessment and SP assessment of overall communication skills by senior residents was 0.30 on the goals of care case and 0.26 on the comfort measures only case. SP assessments showed that junior residents had higher overall communication skills than senior residents (p = 0.03). Senior residents perceived that having difficult conversations was more level appropriate (p < 0.001), and they were less nervous having difficult conversations (p < 0.01) than junior residents.

CONCLUSIONS

We found that residents perform difficult conversations well, that subjective and objective skills are correlated, and that skills-based training is needed across all residency levels. This well-received method may be used to observe, document, and provide resident feedback for these important skills.

Designing and Implementing the Objective Structured Clinical Examination in Anesthesiology

Since its description in 1974, the Objective Structured Clinical Examination (OSCE) has gained popularity as an objective assessment tool of medical students, residents, and trainees. With the development of the anesthesiology residents’ milestones and the preparation for the Next Accreditation System, there is an increased interest in OSCE as an evaluation tool of the six core competencies and the corresponding milestones proposed by the Accreditation Council for Graduate Medical Education.

In this article the authors review the history of OSCE and its current application in medical education and in different medical and surgical specialties. They also review the use of OSCE by anesthesiology programs and certification boards in the United States and internationally. In addition, they discuss the psychometrics of test design and implementation with emphasis on reliability and validity measures as they relate to OSCE.

Pilot test of a three-station palliative care observed structured clinical examination for multidisciplinary trainees

Developing effective communication and symptom assessment skills is an important component of palliative care training for advance practice nurses (APNs) and other health care providers. The purpose of this project was to develop and pilot test a three-station palliative care Observed Structured Clinical Examination (OSCE) for APN students and physician fellows. Three stations included discussing goals of care, breaking bad news, and assessing delirium. Measures included the Interpersonal Skills Tool, Station Checklists, the OSCE Evaluation Tool, and a focus group to solicit learners' perspectives about the experience. Findings showed that learners evaluated the exercise as appropriate for their level of training and that standardized patients were convincing and provided helpful feedback. Learner self-evaluation means were significantly lower than those of standardized patient or faculty, and faculty raters demonstrated low interrater reliability. Initial evaluation suggests a three-station palliative care OSCE exercise is effective for multidisciplinary learners, although additional refinement is necessary.

Design, validity, and reliability of a pediatric resident JumpSTART disaster triage scoring instrument

OBJECTIVE

To design an instrument for scoring residents learning pediatric disaster triage (PDT), and to test the validity and reliability of the instrument.

METHODS

We designed a checklist-based scoring instrument including PDT knowledge and skills and performance, as well as a global assessment. Learners' performance in a 10-patient school bus crash simulation was video recorded and scored with the instrument. Learners triaged the patients with a color-coded algorithm (JumpSTART, Simple Triage and Rapid Treatment). Three evaluators observed the recordings and scored triage performance for each learner. Internal and construct validity of the instrument were established via comparison of resident performance by postgraduate year (PGY) and correlating instrument items with overall score. Validity was assessed with analysis of variance and the D statistic. We calculated evaluators' intraclass correlation coefficient (ICC) for each patient, skill, triage decision, and global assessment.

RESULTS

There were 37 learners and 111 observations. There was no difference in total scores by PGY (P = .77), establishing internal validity. Regarding construct validity, most instrument items had a D statistic of >0.5. The overall ICC among scores was 0.83 (95% confidence interval [CI] 0.74-0.89). Individual patient score reliability was high and was greatest among patients with head injury (ICC 0.86; 95% CI 0.79-0.91). Reliability was low for an ambulatory patient (ICC 0.29; 95% CI 0.07-0.48). Triage skills evaluation showed excellent reliability, including airway management (ICC 0.91; 95% CI 0.86-0.94) and triage speed (ICC 0.81; 95% CI 0.72-0.88). The global assessment had moderate reliability for skills (ICC 0.63; 95% CI 0.47-0.75) and knowledge (ICC 0.64; 95% CI 0.49-0.76).

CONCLUSIONS

We report the validity and reliability testing of a PDT-scoring instrument. Validity was confirmed with no performance differential by PGY. Reliability of the scoring instrument for most patient-level triage, knowledge, and specific skills was high.

Simulation in surgical education

The pedagogical approach to surgical training has changed significantly over the past few decades. No longer are surgical skills solely acquired through a traditional apprenticeship model of training. The acquisition of many technical and nontechnical skills is moving from the operating room to the surgical skills laboratory through the use of simulation. Many platforms exist for the learning and assessment of surgical skills. In this article, the authors provide a broad overview of some of the currently available surgical simulation modalities including bench-top models, laparoscopic simulators, simulation for new surgical technologies, and simulation for nontechnical surgical skills.

Geriatric Education for Surgical Residents: Identifying a Major Need

This study evaluated a program designed to test and enhance residents’ knowledge of geriatrics. A 2-year prospective interventional trial was conducted. Surgical residents underwent pretesting (pre) in three areas: polypharmacy, delirium, and end of life. They then received educational materials and completed a posttest within 1 month and a patient simulation examination graded by a physician observer and the patient on his or her satisfaction. Forty-nine residents (51% interns, 55% general surgery residents) participated. Seventy per cent had no prior geriatrics education. Test scores significantly improved from pretest to posttest (12.9 ± 3.1 vs 13.78 ± 3.12, P = 0.01). The scores were consistently better on poly topics and consistently worse on end-of-life topics: pretest per cent correct: polypharmacy 60, end of life 46, P = 0.007; posttest percent correct: polypharmacy 63, end of life 49, P = 0.0014. By Pearson correlation, the pretest and posttest scores did not correlate with either the observer ( R = -0.16, P = 0.27 pre, R = -0.08, P = 0.59 post) or subscores ( R = -0.27, P = 0.11 pre, R = -0.13, P = 0.45 post), although the observer and subscore correlated with each other ( R = 0.35, P = 0.036). Performance was poor and did not correlate with better patient care by simulation. Other options for geriatric education need to be considered and evaluated.

Development and evaluation of a program to strengthen first year residents' proficiency in leading end-of-life discussions

PURPOSE

Multiple interventions have been developed to teach and improve internal medicine residents' end-of-life communication skills, but have not been easily adaptable to other institutions. The purpose of this study was to develop and evaluate a program to enhance physicians' end-of-life communication with families of dying patients using a format that could be incorporated into an existing curriculum for first-year internal medicine residents.

METHODS

An end-of-life educational program was developed and evaluated in the context of educating first-year residents at an urban academic medical center during the 2008-2009 academic year. The program consisted of three sessions including an interactive workshop flanked by pre- and post-workshop evaluations in simulated encounter and clinical vignette formats. Simulated encounters were recorded on video and residents' performances were rated by two independent observers using a 23 point checklist.

RESULTS

Complete data were available for 24 (73%) of 33 residents who participated in the program. The residents' checklist scores increased significantly from a mean of 48.1 at baseline to 73.9 at follow-up. The increase in the scores on the clinical vignettes was also statistically significant, but of lesser magnitude.

CONCLUSIONS

A short, focused intervention can have significant impact on residents' communication skills in the setting of an end-of-life objective structured clinical examination (OSCE).

A multi-institutional study of the Family Conference Objective Structured Clinical Exam: a reliable assessment of professional communication

BACKGROUND

To test the value of a simulated Family Conference Objective Structured Clinical Exam (OSCE) for resident assessment purposes, we examined the generalizability and construct validity of its scores in a multi-institutional study.

METHODS

Thirty-four first-year (PG1) and 27 third-year (PG3) surgery residents (n = 61) from 6 training programs were tested. The OSCE consisted of 2 cases (End-of-Life [EOL] and Disclosure of Complications [DOC]). At each program, 2 clinicians and 2 standardized family members rated residents using case-specific tools. Performance was measured as the percentage of possible score obtained. We examined the generalizability of scores for each case separately. To assess construct validity, we compared PG1 with PG3 performance using repeated measures multivariate analysis of variance (MANOVA).

RESULTS

The relative G-coefficient for EOL was .890. For DOC, the relative G-coefficient was .716. There were no significant performance differences between PG1 and PG3 residents.

CONCLUSIONS

This OSCE provides reliable assessments suitable for formative evaluation of residents' interpersonal communication skills and professionalism.

Resident education and management of end-of-life care: the resident's perspective

BACKGROUND

Twenty percent of Americans die in the intensive care unit of our nation's hospitals. Many of those individuals die after life-sustaining therapy has been withdrawn or withheld. Surgeons should be competent in discussing the withholding and withdrawal of life sustaining therapy (WWLST) with their patients. We surveyed surgical residents to learn their perspectives and training experience with discussing end-of-life care and WWLST with patients.

METHODS

We mailed a survey to residents in all accredited surgical residency programs in New England. Nonresponders were contacted by mail at 3 and 6 weeks after the initial mailing.

RESULTS

Nineteen of 20 (95%) programs participated in this study. Three hundred thirty-five residents were surveyed and 141 residents responded (response rate, 42%). Ninety-two percent (n = 129) of respondents had cared for patients where WWLST had occurred, and 74% (n = 104) had initiated a discussion about WWLST themselves. Most (n = 81, 60%) respondents felt competent to discuss WWLST, whereas 14% rarely (n = 13) or never (n = 6) felt comfortable discussing WWLST. Most (n = 119, 85%) respondents believed that they would be adequately trained at the end of their residencies; however, 39% (n = 53) felt they were inadequately trained in this area. Graduates before 2002 were significantly more likely to agree strongly or generally that they would be well trained in managing WWLST when they completed residency (p = 0.006).

CONCLUSION

Almost all surgical residents will have to discuss WWLST with patients and their families, yet a significant number feel inadequately trained to do so. Steps should be taken to ensure that surgical residents can discuss WWLST as part of their core competencies, and this training should be reinforced throughout residency.

Skills of internal medicine residents in disclosing medical errors: a study using standardized patients

PURPOSE

To determine internal medicine (IM) residents' ability to disclose a medical error using standardized patients (SPs) and to survey residents' experiences of disclosure.

METHOD

In 2005, 42 second-year IM residents at the University of Toronto participated in the study. Each resident disclosed one medical error (insulin overdose) to an SP. The SP and a physician observer scored performance using a rating scale (1 = not performed, 2 = performed somewhat, and 3 = performed well) that measures error disclosure on five specific component skills and that provides an overall assessment score (scored on a five-point scale, 5 = high). Residents also completed a questionnaire.

RESULTS

The mean scores on the five components were explanation of medical facts (2.60), honesty (2.31), empathy (2.47), future error prevention (1.99), and general communication skills (2.47). The residents' mean overall disclosure score was 3.53. Although 27 of 42 residents (64%) reported previous experience in disclosing an error to a patient during their training, only 7 (27%) of these residents reported receiving any feedback about their performance. Of 41 residents, 21 (51%) had received some prior training in disclosure, and 38 (93%) thought additional training would be useful and relevant.

CONCLUSIONS

Disclosing medical error is now a standard practice. Experience with medical error begins early in training, and preparing trainees to discuss these errors is essential. Areas exist for improvement in residents' disclosure abilities, particularly regarding the prevention of future errors. Curricula to increase residents' skills and comfort in disclosure need to be implemented. Most residents would welcome further training.

Professionalism and communication in the intensive care unit: reliability and validity of a simulated family conference

OBJECTIVE

An Objective Structured Clinical Exam was designed to assess physician's ability to discuss end-of-life (EOL) and disclose iatrogenic complications (DOC) with family members of intensive care unit patients. The study explores reliability and validity based on scores from contrasting rater groups (clinicians, SPs, and examinees).

METHODS

Two 20-minute stations were administered to 17 surgical residents and 2 critical fellows at a university-based training program. The exam was conducted, videotaped, and scored in a standardized setting by 8 clinical raters (MD and RN) and 8 standardized families using separate rating tools (EOL and DOC). Examinees assessed themselves using the same tools. We analyzed the internal consistency, inter-rater agreement, and discriminant validity of both cases using data from each rater group. Cross-rater group comparisons were also made.

RESULTS

The internal consistency reliability correlations were above 0.90 regardless of case or rater group. Within rater groups, raters were within 1 point of agreement (5-pt and 6-pt scales) on 81% of the DOC and between 74% and 79% of the EOL items. Family raters were more favorable than clinical raters in scoring DOC, but not EOL cases. Large raw differences in performance by training level favored more experienced trainees (3rd year residents and fellows). These differences were statistically significant when based on residents own self-ratings, but not when they were based on clinical or family ratings.

DISCUSSION

The Family Conference Objective Structured Clinical Exam is a reliable exam with high content validity. It seems unique in the literature for assessing surgical trainees' ability to discuss "bad news" with family members in intensive care.

Ethics education in surgical residency programs: a review of the literature

Although ethics education in undergraduate medical school curricula has become essentially universal, and because ethics and professionalism have become integral parts of competency-based curricula for residents in nearly all specialties of medicine and surgery, it is not clear to what extent postgraduate surgical training routinely includes ethics education or skills training. Most general surgery residencies do not routinely integrate in-depth ethics skills training and assessment into their didactic curricula. In this article, we review the literature concerning ethics education in postgraduate surgical training programs. The few studies in the literature suggest that ethics education, when integrated in surgical residency curricula, can lead to measurable improvements in resident-centered outcomes, which include knowledge and confidence in handling ethical dilemmas. These curricula may lead to improvements in patient care. These findings suggest that ethics education for surgical residents is valuable but that questions regarding the optimal "dose" of ethics education and training as well as the optimal teaching methods would benefit from extended systematic evaluation and inquiry.

Core Competencies in Palliative Care for Surgeons: Interpersonal and Communication Skills

Surgeons are an important part of the multidisciplinary approach to the care of terminally ill and dying patients. Some surgical residencies have recognized the need to incorporate palliative care—related topics into graduate surgical education. One core competency of utmost importance to palliative care is the effective use of interpersonal and communication skills. Four areas of surgical practice are identified where these communication skills are important: during preoperative counseling, when presenting a devastating diagnosis or poor prognosis, when discussing error, and when discussing death. Case examples and recommendations for the appropriate words and actions to use in these scenarios are offered. It is important for both surgeons in practice and those in training to achieve proficiency with these communication skills.