One program's experience of OSCE vs. written board certification results: a pilot study

Subtle skills: Using objective structured clinical examinations to assess gastroenterology fellow performance in system based practice milestones

BACKGROUND

System based practice (SBP) milestones require trainees to effectively navigate the larger health care system for optimal patient care. In gastroenterology training programs, the assessment of SBP is difficult due to high volume, high acuity inpatient care, as well as inconsistent direct supervision. Nevertheless, structured assessment is required for training programs. We hypothesized that objective structured clinical examination (OSCE) would be an effective tool for assessment of SBP.

AIM

To develop a novel method for SBP milestone assessment of gastroenterology fellows using the OSCE.

METHODS

For this observational study, we created 4 OSCE stations: Counseling an impaired colleague, handoff after overnight call, a feeding tube placement discussion, and giving feedback to a medical student on a progress note. Twenty-six first year fellows from 7 programs participated. All fellows encountered identical case presentations. Checklists were completed by trained standardized patients who interacted with each fellow participant. A report with individual and composite scores was generated and forwarded to program directors to utilize in formative assessment. Fellows also received immediate feedback from a faculty observer and completed a post-session program evaluation survey.

RESULTS

Survey response rate was 100%. The average composite score across SBP milestones for all cases were 6.22 (SBP1), 4.34 (SBP2), 3.35 (SBP3), and 6.42 (SBP4) out of 9. The lowest composite score was in SBP 3, which asks fellows to advocate for cost effective care. This highest score was in patient care 2, which asks fellows to develop comprehensive management plans. Discrepancies were identified between the fellows’ perceived performance in their self-assessments and Standardized Patient checklist evaluations for each case. Eighty-seven percent of fellows agreed that OSCEs are an important component of their clinical training, and 83% stated that the cases were similar to actual clinical encounters. All participating fellows stated that the immediate feedback was “very useful.” One hundred percent of the fellows stated they would incorporate OSCE learning into their clinical practice.

CONCLUSION

OSCEs may be used for standardized evaluation of SBP milestones. Trainees scored lower on SBP milestones than other more concrete milestones. Training programs should consider OSCEs for assessment of SBP.

Assessment of Abilities of Gastroenterology Fellows to Provide Information to Patients With Liver Disease

BACKGROUND & AIMS

Patient education is critical in ensuring patient compliance and good health outcomes. Fellows must be able to effectively communicate with their patients, delivering enough information for the patient to understand their medical problem and maximize patient compliance. We created an objective structured clinical examination (OSCE) with 4 liver disease cases to assess fellows' knowledge and ability to inform standardized patients (SPs) about their clinical condition.

METHODS

We developed 4 cases highlighting different aspects of liver disease and created a 4-station OSCE: hepatitis B, acute hepatitis C, new diagnosis of cirrhosis, and an end-stage cirrhotic nontransplant candidate. The SP with hepatitis B was minimizing the fact that she could not read English. The acute hepatitis C SP was a nursing student who is afraid that having hepatitis C might jeopardize her career. The SP with the new diagnosis of alcoholic cirrhosis needed to stop drinking, and the end-stage liver disease patient had to grapple with his advanced directives. Twelve fellows from 4 GI training programs participated. Our focus was to assess the fellows' knowledge about liver diseases and the Accreditation Council for Graduate Medical Education competencies of health literacy, shared decision making, advanced directives, and goals of care. The goal for the fellows was to communicate effectively with the SPs, and acknowledge that each patient had an emotionally charged issue to overcome. The SPs used a checklist to rate fellows' performance. Faculty and the SPs observed the cases and provided feedback. The fellows were surveyed on their performance regarding the case.

RESULTS

The majority of fellows were able to successfully summarize findings and discuss a plan with the patient in the new diagnosis of cirrhosis (76.92%) and hepatitis C case (100%), but were less successful in the hepatitis B case (30.77%) and the end-of-life case (41.67%). Overall, a small percentage of fellows reflected that they did a good job (22%-33%), except at the end-of-life case (67%). The fellows' greatest challenge was trying to cover a lot of information in a single outpatient visit.

CONCLUSIONS

Caring for patients with liver diseases can be complex and time consuming. The patients and fellows' observations were discordant in several areas: for example, the fellows believed they excelled in the end-of-life case, but the SP thought only a small percentage of fellows were able to successfully summarize and discuss the plan. This discrepancy and others highlight important areas of focus in training programs. OSCEs are important to help the fellows facilitate striking the right balance of information delivery and empathy, and this will lead to better patient education, compliance, rapport, and satisfaction.

Objective structured clinical examination as a novel tool in inflammatory bowel disease fellowship education

BACKGROUND

Experiential learning in medical education, as exemplified by objective structured clinical examinations (OSCEs), is a well-validated approach for improving trainee performance. Furthermore, the Accreditation Council for Graduate Medical Education has identified OSCEs as an ideal method for assessing the core competency of interpersonal and communication skills. Here, we describe a novel educational tool, the inflammatory bowel disease OSCE (IBD OSCE), to assess and improve this clinical skill set in Gastroenterology fellows.

METHODS

We developed a 4-station IBD OSCE that assessed shared decision making, physician-physician communication, and physician-patient consultative skills specifically related to the care of patients with IBD. Each station was videotaped and observed live by faculty gastroenterologists. Behaviorally anchored checklists were scored independently by a faculty observer and the standardized patient/physician, who both provided feedback to the fellow immediately after each case. Post-OSCE, fellows attended a debriefing session on patient communication and were surveyed to assess their perspective on the examination's educational value.

RESULTS

Twelve second-year gastroenterology fellows from 5 fellowship programs participated in the IBD OSCE. Fellows performed well in all measured domains and rated the experience highly for its educational value. Fellows cited IBD as an area of relative deficiency in their education compared with other knowledge areas within gastroenterology.

CONCLUSIONS

To our knowledge, this is the first OSCE designed specifically for the evaluation of skills as they relate to IBD management. Using OSCEs for IBD education provides an opportunity to robustly assess core competencies and the role of the physician as an educator.

Informed consent in the older adult: OSCEs for assessing fellows' ACGME and geriatric gastroenterology competencies

OBJECTIVES

The American Gastroenterological Association fellowship curriculum identifies geriatric components for gastroenterology (GI) training; however, few tools are available for this purpose. Using an objective structured clinical examination (OSCE), we aimed to assess ACGME competencies of communication, professionalism, and geriatric-specific patient care among GI fellows.

METHODS

We developed an informed-consent case involving a geriatric patient who needs surveillance colonoscopy. We used a validated faculty skills checklist to rate fellows across three competency domains. Fifteen fellows from four GI training programs participated.

RESULTS

Although the fellows excelled at communication and professionalism, only 51% excelled at geriatric-specific patient-care skills. Fellows were least likely to demonstrate collaboration with the patient, to assess patient understanding, and to explain the limits of the test. Communication and geriatric-specific skills were correlated.

CONCLUSIONS

OSCEs are a feasible method for assessing geriatric-related ACGME competencies for fellows. The results highlight the need for curriculum development.

Observing handoffs and telephone management in GI fellowship training

OBJECTIVES

Gastroenterology (GI) training programs are mandated to teach fellows interpersonal communication and professionalism as basic competencies. We sought to assess important skill sets used by our fellows but not formally observed or measured: handoffs, telephone management, and note writing. We designed an Observed Standardized Clinical Examination (OSCE) form and provided the faculty with checklists to rate fellows' performance on specific criteria.

METHODS

We created two new scenarios: a handoff between a tired overnight senior fellow on call and a more junior fellow, and a telephone management case of an ulcerative colitis flare. Fellows wrote a progress notes documenting the encounters. To add educational value, we gave the participants references about handoff communication. Four OSCE stations-handoff communication, telephone management, informed consent, and delivering bad news-were completed by fellows and observed by faculty.

RESULTS

Eight faculty members and eight fellows from four GI training programs participated. All the fellows agreed that handoffs can be important learning opportunities and can be improved if they are structured, and that handoff skills can improve with practice.

CONCLUSIONS

OSCEs can serve as practicums for assessing complex skill sets such as handoff communication and telephone management.

Teaching the competencies: using observed structured clinical examinations for faculty development

OBJECTIVES

Gastroenterology (GI) training programs must develop the teaching skills of their faculty and provide feedback to their fellows. Many faculty feel uncomfortable offering feedback or identifying specific areas for improvement to the fellows. We developed an Observed Structured Clinical Exam (OSCE) to assess fellows' skills and provided faculty with specific criteria to rate the fellows' performance. We propose that OSCEs can serve as tools for faculty development in delivering effective feedback.

METHODS

Faculty completed a Web-based training module and received written guidelines on giving feedback. Four OSCE stations were completed by each fellow with faculty using standardized checklists to assess the fellows' skills. Afterwards, faculty rated each program component and assessed their comfort level with feedback.

RESULTS

Eight faculty members and 10 fellows from 5 GI training programs in NYC participated. 100% of the faculty agreed that feedback is an important learning tool, should include the learner's self-assessment, and that feedback skills could improve with practice. Compared to faculty skills prior to the program, 87.5% of the faculty agreed that they focused more on specific behaviors and 75% agreed that giving negative feedback was now easier.

CONCLUSIONS

OSCEs can serve as practicums for faculty development in giving constructive feedback.

Development, impact, and measurement of enhanced physical diagnosis skills

Evidence suggests that the quality and frequency of bedside clinical examination have declined. We undertook the study to (1) determine whether intensive instruction in physical examination enhances medical student skills and (2) develop a tool to evaluate those skills using a modified observed structured clinical examination (OSCE). This was a randomized, blinded, prospective, year-long study involving 3rd year students at the Albert Einstein College of Medicine. Students were randomized to receive intensive instruction in physical examination [study group (n = 46)] or usual instruction [control group (n = 75)] and evaluated by a modified OSCE. The OSCE consisted of 6 real patient stations: Head, ears, eyes, neck, throat; pulmonary; cardiovascular; gastrointestinal; neurology; musculoskeletal; and 2 computer imaging stations: genitourinary and dermatology. A faculty member present at each patient station evaluated student performance. Data were analyzed using t-tests for comparison of the mean scores between the two groups for each station and for average scores across stations. A total of 121 students were tested. The study group performed significantly better than the control group in the gastrointestinal station (p = 0.0004), the combined average score across the six real patient stations (p = 0.0001), and the combined average score across all eight stations (p = 0.0014). Intensive physical diagnosis instruction enhances physical examination skills of 3rd year medical students. The modified OSCE is a useful tool to evaluate these skills.

Teaching the competencies: using objective structured clinical encounters for gastroenterology fellows

BACKGROUND AIMS

Objective structured clinical encounters (OSCEs) are used widely to educate and assess the competence of medical students and residents; they generally are absent from fellowship training. The Accreditation Council for Graduate Education has cited OSCEs as a best practice for assessing the 6 core competencies. This article reports on the use of an OSCE to assess the competence of second-year gastroenterology fellows in the difficult-to-assess core competencies: interpersonal and communication skills and professionalism.

METHODS

We developed a 4-station, faculty-observed OSCE with 4 standardized patients. Information gathering, relationship development, patient education, and counseling skills were assessed. Professionalism skills assessed included obtaining informed consent, delivering bad news, managing difficult situations, and showing interdisciplinary respect. In each station, faculty and standardized patients completed an 18- to 24-item checklist evaluating fellows' performance and provided feedback to the fellows. Nine fellows and 5 faculty from 4 gastroenterology training programs in NYC participated.

RESULTS

Fellows and faculty generally highly rated the realism of the OSCE and favorably rated the OSCE for its difficulty and their overall experience. Across all cases, fellows were rated as receiving "well dones" for 56.4% of the communication items (SD, 18.3%) and for 79.1% of the professionalism items (SD, 16.4%).

CONCLUSIONS

Integrating OSCEs into gastroenterology fellowship training may help enhance communication skills and prepare fellows for dealing with difficult clinical situations and provides mechanisms for constructive feedback. OSCEs developed collaboratively can assist in program self-evaluation and reduce costs by sharing resources, in addition to fulfilling Accreditation Council for Graduate Education mandates.

Perception of pharmacy students in Malaysia on the use of objective structured clinical examinations to evaluate competence

OBJECTIVES

To assess bachelor of pharmacy students' overall perception and acceptance of an objective structured clinical examination (OSCE), a new method of clinical competence assessment in pharmacy undergraduate curriculum at our Faculty, and to explore its strengths and weaknesses through feedback.

METHODS

A cross-sectional survey was conducted via a validated 49-item questionnaire, administered immediately after all students completed the examination. The questionnaire comprised of questions to evaluate the content and structure of the examination, perception of OSCE validity and reliability, and rating of OSCE in relation to other assessment methods. Open-ended follow-up questions were included to generate qualitative data.

RESULTS

Over 80% of the students found the OSCE to be helpful in highlighting areas of weaknesses in their clinical competencies. Seventy-eight percent agreed that it was comprehensive and 66% believed it was fair. About 46% felt that the 15 minutes allocated per station was inadequate. Most importantly, about half of the students raised concerns that personality, ethnicity, and/or gender, as well as interpatient and inter-assessor variability were potential sources of bias that could affect their scores. However, an overwhelming proportion of the students (90%) agreed that the OSCE provided a useful and practical learning experience.

CONCLUSIONS

Students' perceptions and acceptance of the new method of assessment were positive. The survey further highlighted for future refinement the strengths and weaknesses associated with the development and implementation of an OSCE in the International Islamic University Malaysia's pharmacy curriculum.

Design and Implementation of a Video-based Clinical Examination

OBJECTIVES

To (1) describe the development of a Video-based Clinical Examination (VCE) as a formal testing format to evaluate student ability to make an accurate pharmaceutical assessment and recommendation, and (2) determine student perception of the VCE testing format.

METHODS

Descriptive study of first-year pharmacy students.

RESULTS

One hundred and twenty-nine students were included in the study. Students perceived that the VCE testing format provided a real life/interactive environment but felt rushed as the video segments of the patient/pharmacist interaction occurred quickly.

IMPLICATIONS

Based on the findings of this project, we will continue to pursue further research related to validity, reliability and application of VCEs. However, the University of Colorado will continue to incorporate VCEs in the performance based evaluations in the Professional Skills Development 1 course, as it appears to be an effective stepping-stone for first-year students to begin developing their active listening, higher level learning and problem-solving skills. Results of this project will be shared with the faculty and curriculum committee at the University of Colorado School of Pharmacy to encourage further use and research of VCEs in other courses.

Creating an ideal objective structured clinical exam for an obstetrics and gynecology medical student clerkship

OBJECTIVE

The objective of the study was to describe our experience with the Objective Structured Clinical Exam during the third-year obstetrics/gynecology clerkship.

STUDY DESIGN

Our 8-station Objective Structured Clinical Exam is described in detail. We analyzed grades from the Objective Structured Clinical Exam for overall distribution, comparing among and between groups by year and among exam sites and correlating with the National Board of Medical Examiners subject exam and the final clerkship grades.

RESULTS

The Objective Structured Clinical Exam is provided at 3 sites, 1 at a technically advanced simulation center. Most students (54.6%) received a grade of A; however, the mean grade from the Objective Structured Clinical Exam at the simulation center (3.41 on a 4-point grade-point average scale) was lower than that at the clinic sites (3.60, P = .005; 3.73, P < .001). Effect of rotation order was identified only in the first year. Grades from the Objective Structured Clinical Exam were positively but weakly correlated with National Board of Medical Examiners subject exam and final clerkship grades.

CONCLUSION

An Objective Structured Clinical Exam in the obstetrics/gynecology clerkship can be administered with limited resources and provides a valuable assessment method.

Surgery residents' perception of the Objective Structured Clinical Examination (OSCE)

PURPOSE

Beginning in July 2003, residency programs will be required to incorporate new educational assessment methods as defined by the Accreditation Council for Graduate Medical Education (ACGME) outcome initiative. The Objective Structured Clinical Examination (OSCE) is an assessment tool that is favorably viewed by the ACGME. Our institution has utilized the OSCE for evaluation of surgery trainees since 1996. Despite the positive acceptance of the OSCE by students, residents expressed dissatisfaction with the examination. This study was therefore undertaken to specifically evaluate resident perception of the OSCE.

METHODS

Two sequential surveys were administered to surgery residents at the Medical College of Ohio. Response of medical students to a standard survey following completion of the OSCE was tabulated.

RESULTS

On the first, general survey, residents felt that the OSCE was not an adequate measure of either clinical (15 of 17 residents) or technical (15 of 18 residents) skills; 14 of 16 residents felt that the OSCE should not be used when considering promotion. When specifically queried in a follow-up survey, residents indicated that the OSCE was an adequate measure of clinical knowledge (2.2 +/- 0.3); however, most still felt that the OSCE should not be used when considering promotion (4.3 +/- 0.3). (Scores = mean +/- SEM on a Likert scale where 1 = strongly agree and 5 = strongly disagree). By contrast, 97.6% of 663 medical students surveyed (September 1996 through February 2002) felt the OSCE was useful.

CONCLUSIONS

The OSCE has been shown to be a reliable and valid measure of basic clinical and technical competence. Despite our residents' current perception, we believe that the OSCE is an important method for resident evaluation, particularly within the context of the current ACGME outcome initiative.

Evaluating medical students' skills in obtaining informed consent for HIV testing

OBJECTIVE

To evaluate fourth-year medical students' abilities to obtain informed consent or refusal for HIV testing through a performance-based evaluation method.

DESIGN

Student competence was assessed in a standardized patient interaction in which the student obtained informed consent or refusal for HIV testing. A previously validated 16-item checklist was completed by the standardized patient. A subset was independently reviewed and scored by a faculty member to calculate interrater reliability for this report. Student feedback on the assessment was elicited.

SETTING

School of Medicine at the University of New Mexico.

PATIENTS/PARTICIPANTS

All senior medical students in the class of 2000 were included.

INTERVENTIONS

A 10-minute standardized patient interaction was administered within the context of a formal comprehensive performance assessment.

MEASUREMENTS AND MAIN RESULTS

Seventy-nine students participated, and most (96%) demonstrated competence on the station. For the 15 specific items, the mean score was 25.5 out of 30 possible points (range, 13 to 30; SD, 3.5) on the checklist. A strong positive correlation (rs =.79) was found between the total score on the 15 Likert-scaled items and the score in response to the global item, "I would return to this clinician" (mean, 3.5; SD, 1.0). Scores given by the standardized patients and the faculty rater were well correlated. The station was generally well received by students, many of whom were stimulated to pursue further learning.

CONCLUSIONS

This method of assessing medical students' abilities to obtain informed consent or refusal for HIV testing can be translated to a variety of clinical settings. Such efforts may help in demonstrating competence in performing key ethics skills and may help ensure ethically sound clinical care for people at risk for HIV infection.