1
|
Badawi MA, Hussain HO, Badawi MA, Al Shawwa LA, Jamjoom RA, Park YS, Tekian A. Learning transfusion medicine through scoring objective structured clinical examination. Transfusion 2023; 63 Suppl 1:S20-S27. [PMID: 36748666 DOI: 10.1111/trf.17199] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2021] [Revised: 12/04/2021] [Accepted: 12/29/2021] [Indexed: 02/08/2023]
Abstract
BACKGROUND Transfusion medicine education at the undergraduate level is typically limited in duration. In view of limitations of traditional teaching methods, we explore effectiveness of scoring (Objective Structured Clinical Examination) OSCE as an educational method. MATERIALS AND METHODS The study was of a randomized interventional three group pre-test-post-test design. Participants were undergraduate medical students in their two final years. The intervention was watching and scoring 2 videotaped OSCE stations about obtaining consent for blood transfusions and assessing the ability to explain risks, benefits, and alternatives of blood transfusion. Participants were asked to assess the performance of the videotaped actor using checklists. Participants were randomized to watch and evaluate one set of videos at either the highest, intermediate, or lowest compliance with required consent elements. Main measure was performance in a knowledge test containing multiple-choice and true/false questions. This was given before (pre-test), immediately after the intervention (post-test 1), and after 8 weeks (post-test 2). Student perceptions regarding the intervention was assessed immediately after the session. RESULTS Sixty-nine students were randomized. Post-test 1 results (mean 16.52, SD 1.88) were significantly greater than pre-test results (mean 11.83, SD 2.13) by group and across all groups (p < 0.001). Post-test 2 results for the complete cohort showed maintenance of significant improvement in comparison with the pre-test. The majority of students agreed that learning through scoring OSCE was an effective educational experience. CONCLUSIONS In the undergraduate medical setting, scoring OSCE stations may enhance learning of content discussed and evaluated in the stations.
Collapse
Affiliation(s)
- Maha Abdulrazak Badawi
- Department of Hematology, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia.,King Abdulaziz University Hospital, Jeddah, Saudi Arabia
| | | | - Mazen Abdulrazak Badawi
- King Abdulaziz University Hospital, Jeddah, Saudi Arabia.,Department of Internal Medicine, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia
| | - Lana Adey Al Shawwa
- Department of Medical Education, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia
| | - Reda Abdullah Jamjoom
- Department of Medical Education, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia
| | - Yoon Soo Park
- Department of Medical Education, College of Medicine, University of Illinois at Chicago, Chicago, Illinois, USA.,Harvard Medical School, Boston, Massachusetts, USA
| | - Ara Tekian
- Department of Medical Education, College of Medicine, University of Illinois at Chicago, Chicago, Illinois, USA
| |
Collapse
|
2
|
Hsu YD, Lin HC, Hsieh CH, Chang FW, Lee JT, Lin CS, Huang HK, Lin C. Is it an impact factor for standardized patients with actor background to perform in the taiwan high-stake objective structured clinical examination? JOURNAL OF MEDICAL SCIENCES 2020. [DOI: 10.4103/1011-4564.282529] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022] Open
|
3
|
Hsu YD, Lin HC, Hsieh CH, Chang FW, Lee JT, Lin CS, Huang HK, Lin C. Is it an impact factor for standardized patients with actor background to perform in the taiwan high-stake objective structured clinical examination? JOURNAL OF MEDICAL SCIENCES 2020. [DOI: 10.4103/jmedsci.jmedsci_203_19] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022] Open
|
4
|
Beck DE. Performance-Based Assessment: Using Pre-Established Criteria and Continuous Feedback to Enhance a Student’s Ability to Perform Practice Tasks. J Pharm Pract 2016. [DOI: 10.1106/lgr5-3c3n-nteg-b9vh] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/14/2022]
Abstract
Performance-based assessment is a process in which pre-established performance criteria and continuous feedback are used by both the student and instructor to improve learning with an endpoint of students demonstrating their achievement of learning by performing realistic practice tasks. This manuscript describes the issues limiting widespread use of performance-based assessment, based on findings from the pharmacy, medical, and general education literature, and then proposes a model for how to successfully implement it. Performance-based assessment methods such as direct observation, the Mini-Clinical Evaluation Exercise, simulations involving standardized patients and computers, projects, presentations, learning portfolios, and the Triple Jump Exam are discussed. Successful implementation of performance-based assessment involves: 1) inculcating a culture for assessment, 2) establishing pre-specified performance criteria, 3) implementing use of the pre-established performance criteria and formative assessment methods to provide continuous feedback during the learning period, 4) using summative performance-based assessment methods, and 5) providing mechanisms for student remediation. Successful inclusion of performance-based assessment in Doctor of Pharmacy curricula should enable instructors to better prepare graduates that can provide pharmaceutical care in a quality manner.
Collapse
Affiliation(s)
- Diane E. Beck
- Department of Pharmacy Practice, School of Pharmacy, Auburn University, AL 36849-5502,
| |
Collapse
|
5
|
Hecimovich M, Volet S. Simulated learning in musculoskeletal assessment and rehabilitation education: comparing the effect of a simulation-based learning activity with a peer-based learning activity. BMC MEDICAL EDUCATION 2014; 14:253. [PMID: 25471306 PMCID: PMC4258291 DOI: 10.1186/s12909-014-0253-6] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/19/2014] [Accepted: 11/17/2014] [Indexed: 05/07/2023]
Abstract
BACKGROUND Musculoskeletal disorders and diseases are leading causes of pain, physical disability, and doctor visits throughout the world. Health professionals must be trained to assess, treat through rehabilitation and monitor patients with these disorders. Yet, due to overcrowded curricula, some health education programs struggle to accommodate more than minimal training in musculoskeletal conditions. Consequently, educators in these professions must consider how traditional instruction could be complemented effectively to enhance students' preparation for the diverse musculoskeletal disorders and pathologies they may encounter. The purpose of this study was to explore the benefits that can be obtained from laboratory practice in musculoskeletal conditions with a standardised patient, rather than a peer patient, in a condensed time frame. METHODS Two groups of students were assigned to either a standardised or a peer patient condition for 2 × 2 hours musculoskeletal assessment and rehabilitation lab sessions. All students completed a pre-post matched questionnaire measuring their clinical knowledge, confidence in clinical skills and motivation for further learning. Their clinical skills were tested at the end. Students and standardised patients' perceptions of the simulated learning environment to practise musculoskeletal assessment and rehabilitation were also elicited. RESULTS A t-test for independent samples revealed that students working with standardised patients displayed significantly higher standards of practical clinical skills than those working with peer patients (p=0.018). Using MANOVAs with repeated measures, no interaction effect for clinical knowledge, confidence in clinical skills, and motivation for future learning were found, both groups displaying significantly enhanced cognition and motivation. Three positive and two negative themes emerged from the analysis of students' perceptions of the simulated learning environments. These were consistent with the simulated patients' perceptions. CONCLUSIONS The findings of this study provide support for the value of using standardised patients to enhance clinical skills in musculoskeletal assessment and rehabilitation when the timeframe for laboratory practice is limited. Students' perceptions of their experience contributed to explain why confidence in clinical skills might not necessarily improve when practising with standardised patients. Suggestions are made for optimising learning with standardised patients and for addressing the economic challenge on health education programs of hiring standardised patients.
Collapse
Affiliation(s)
- Mark Hecimovich
- />School of Psychology and Exercise Science, Murdoch University, Murdoch, Western Australia
| | - Simone Volet
- />School of Education, Murdoch University, Murdoch, Western Australia
| |
Collapse
|
6
|
Hurwitz J, Sands S, Davis E, Nielsen J, Warholak T. Patient knowledge and use of acetaminophen in over-the-counter medications. J Am Pharm Assoc (2003) 2014; 54:19-26. [PMID: 24362497 DOI: 10.1331/japha.2014.13077] [Citation(s) in RCA: 16] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/31/2022]
Abstract
OBJECTIVES To evaluate patient knowledge of over-the-counter (OTC) products containing acetaminophen and to determine patients' accuracy in dosing adult, child, and infant formulations. DESIGN Cross-sectional study. SETTING Six community pharmacies in Tucson, AZ, between February and May 2011. PARTICIPANTS 88 adults aged 19 to 89 years. INTERVENTION Investigator-administered, semistructured interviews. MAIN OUTCOME MEASURES Patient knowledge of and ability to safely use OTC products containing acetaminophen, including understanding risks, identifying products, and dosing different formulations. RESULTS Although most (86%) participants heard of acetaminophen, only 68% understood at least one of its uses and only 9% knew the abbreviation APAP. Virtually all knew that consuming too much acetaminophen in 1 day could be harmful, but only 17% and 35% knew that overdoses could result in death or liver damage, respectively. On average, participants correctly identified 80% (range 27-100%) of products with and without acetaminophen from a lineup of 11 OTC products. Although 38% (n = 84) of participants correctly measured both the child and infant doses of acetaminophen, doses ranged from one-half to twice the amount of the labeled child dose and one-third of the labeled infant dose. Findings from the regression analysis suggested that on average, women and those with college degrees had higher overall scores, while participants' age or parent status were nonsignificant predictors. CONCLUSION Many patients remain confused about using acetaminophen safely, signaling the need for greater patient education to prevent unintentional harm. The results further specify common misunderstandings to address during patient contact, which also includes replacing "APAP" with "acetaminophen" on any prescription bottle labels or patient-directed information.
Collapse
|
7
|
Harrington AR, Desselle SP, Apgar DA, Hesselbacher E, Pié A, Quesnel A, Warholak TL. Pharmacy students' opinions of direct-to-consumer advertising: a pilot study at one university. Res Social Adm Pharm 2012; 9:458-66. [PMID: 22999802 DOI: 10.1016/j.sapharm.2012.07.005] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/09/2012] [Revised: 07/12/2012] [Accepted: 07/12/2012] [Indexed: 10/27/2022]
Abstract
BACKGROUND Direct-to-consumer advertisement (DTCA) of prescription medications has become an important informational source for health care consumers. As future health care professionals on the front line of potential communication and dispensing of products emerging from DTCA, it is important to elicit the attitudes of student-pharmacists. OBJECTIVES This study aims to (1) evaluate the validity of the DTCA attitudinal questionnaire using Rasch rating scale analysis and (2) investigate the attitudes of pharmacy students toward DTCA and determine whether these attitudes were associated with years of pharmacy education and demographic characteristics. METHODS This investigation used a cross-sectional print-based questionnaire to evaluate the attitudes of pharmacy students toward DTCA of prescription medications. The 16-item questionnaire included items addressing the attitudes of pharmacy students toward DTCA with respect to patients' knowledge of medications, pharmacists' interaction with patients, and overall consumer judgment of medical prescriptions. Analyses included Rasch analysis and a multiple linear regression. RESULTS A total of 243 students submitted usable questionnaires (85% response rate). Item response categories were collapsed from 5 categories to 3, and 4 items were removed to achieve acceptable Rasch model fit. Pharmacy students demonstrated little difficulty in agreeing with the statements suggesting that DTCA helps patients take a more active role in health care and had the most difficulty in agreeing with items suggesting that DTCA may lead to inappropriate prescribing to satisfy patient requests. Students' overall support for DTCA was the only variable that predicted the questionnaire score (P<.001). CONCLUSIONS In conclusion, the Rasch analysis evaluated the psychometric properties of the instrument and identified the necessity to adapt the questionnaire from previous iterations to adequately fit the student population. Future research should examine factors that contribute to the variance in attitudes toward DTCA among a larger and more heterogeneous population.
Collapse
Affiliation(s)
- Amanda R Harrington
- Department of Pharmacy Practice and Science, University of Arizona, 1295 North Martin Avenue, Tucson, AZ 85721, USA.
| | | | | | | | | | | | | |
Collapse
|
8
|
Relationship between standardized patient checklist item accuracy and performing arts experience. Simul Healthc 2011; 6:150-4. [PMID: 21646984 DOI: 10.1097/sih.0b013e31821687b3] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
INTRODUCTION : It is not known whether a Standardized Patient's (SP's) performing arts background could affect his or her accuracy in recording candidate performance on a high-stakes clinical skills examination, such as the Comprehensive Osteopathic Medical Licensing Examination Level 2 Performance Evaluation. The purpose of this study is to investigate the differences in recording accuracy of history and physical checklist items between SPs who identify themselves as performing artists and SPs with no performance arts experience. METHODS : Forty SPs identified themselves as being performing artists or nonperforming artists. A sample of SP live examination ratings were compared with a second set of ratings obtained after video review (N = 1972 SP encounters) over 40 cases from the 2008-2009 testing cycle. Differences in SP checklist recording accuracy were tested as a function of performing arts experience. RESULTS : Mean overall agreement rates, both uncorrected and corrected for chance agreement, were very high (0.94 and 0.79, respectively, at the overall examination level). There was no statistically significant difference between the two groups with respect to any of the mean accuracy measures: history taking (z = -0.422, P = 0.678), physical examination (z = -1.453, P = 0.072), and overall data gathering (z = -0.812, P = 0.417) checklist items. CONCLUSION : Results suggest that SPs with or without a performing arts background complete history taking and physical examination checklist items with high levels of precision. Therefore, SPs with and without performing arts experience can be recruited for high-stakes SP-based clinical skills examinations without sacrificing examination integrity or scoring accuracy.
Collapse
|
9
|
|
10
|
|
11
|
Truncali A, Lee JD, Ark TK, Gillespie C, Triola M, Hanley K, Gourevitch MN, Kalet AL. Teaching physicians to address unhealthy alcohol use: A randomized controlled trial assessing the effect of a Web-based module on medical student performance. J Subst Abuse Treat 2011; 40:203-13. [DOI: 10.1016/j.jsat.2010.09.002] [Citation(s) in RCA: 16] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2010] [Revised: 09/10/2010] [Accepted: 09/15/2010] [Indexed: 11/24/2022]
|
12
|
A drug-drug interaction knowledge assessment instrument for health professional students: a Rasch analysis of validity evidence. Res Social Adm Pharm 2010; 7:16-26. [PMID: 21397878 DOI: 10.1016/j.sapharm.2010.01.001] [Citation(s) in RCA: 12] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2009] [Revised: 01/08/2010] [Accepted: 01/09/2010] [Indexed: 11/23/2022]
Abstract
BACKGROUND It is essential that current and future health professionals be able to evaluate for possible clinically significant drug-drug interactions (DDIs) and when detected, determine appropriate management strategies to prevent patient harm. OBJECTIVE Assess the validity of a DDI knowledge assessment instrument in a health professional student population. METHODS This study recruited health professional students (medical, nurse practitioner, and pharmacy) beginning experiential training at the University of Arizona. Students were given a knowledge assessment instrument that included 15 medication pairings selected on the basis of clinical importance and were asked to select the most appropriate DDI management strategy for each pair by selecting "avoid combination," "usually avoid combination," "take precautions," or "no special precautions." Data were analyzed in 2 ways because of the subjective nature of classifying DDIs into specific management categories. In the first analysis, respondents were given credit for a correct item only if they selected the management strategy deemed appropriate (management strategy analysis). In another analysis, students were given credit for an item only if they correctly identified specific DDIs (DDI recognition analysis). Rasch analysis was used to assess the validity of the knowledge instrument. RESULTS A total of 165 of the 226 eligible health professional students completed the DDI knowledge assessment (73% response rate). The mean score for management strategy analysis was 3.82 out of 15, whereas DDI recognition analysis produced a higher average (mean=6.55). Good reliability was demonstrated in both strategies, and no ceiling or floor effects were observed. Some construct underrepresentation occurred with both scoring strategies, and some mistargeting was identified when analyzing the management strategy. CONCLUSIONS Although improvements in construct representation may be beneficial, the instrument used demonstrated good reliability and validity and could be used by educators to assess and improve DDI knowledge. The ability of the participants to identify DDIs and select an appropriate management strategy was low. These results support the need for additional DDI education in this institution's health curricula.
Collapse
|
13
|
Breast cancer risk communication: assessment of primary care physicians by standardized patients. Genet Med 2009; 11:735-41. [PMID: 19661809 DOI: 10.1097/gim.0b013e3181b2e5eb] [Citation(s) in RCA: 21] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022] Open
Abstract
PURPOSE To assess primary care providers' communication about breast cancer risk. METHODS We evaluated 86 primary care providers' communication of risk using unannounced standardized (simulated) patients. Physicians were randomly assigned to receive one of three cases: (1) moderate risk case (n = 25), presenting with a breast lump and mother with postmenopausal breast cancer; (2) high-risk (maternal side) case (n = 28), presenting with concern about breast cancer risk; and (3) high-risk (paternal side) case (n = 33), presenting with an unrelated problem. After the appointment, three qualitative parameters were assessed by standardized patients on a 3-point scale (3 = highest satisfaction, 1 = lowest): whether the physician took adequate time; acknowledged her concerns; and offered reassurance. RESULTS Mean satisfaction with physician communication was higher for the moderate risk case (2.92) than for the high-risk paternal case (2.25) or high-risk maternal case (2.42) (P < 0.0001). The score was not influenced by session length, medical specialty, or physician gender. CONCLUSION Physicians more consistently provided a moderate risk standardized patients with reassurance and support compared with the high-risk cases. Primary care physicians may be more unprepared or uneasy addressing the issues raised by more complex scenarios and may benefit from training in the assessment and communication of breast cancer risk.
Collapse
|
14
|
Furman GE. The role of standardized patient and trainer training in quality assurance for a high-stakes clinical skills examination. Kaohsiung J Med Sci 2009; 24:651-5. [PMID: 19251561 DOI: 10.1016/s1607-551x(09)70031-2] [Citation(s) in RCA: 24] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022] Open
Abstract
For over 30 years, medical educators have used standardized patients (SPs), laypersons trained to portray a patient case in a realistic manner, to teach and to assess clinical skills. All medical schools in the US have SP programs in place, and the US and Canada require national examinations using SPs to assess the competency of those wishing to obtain licensure to practice medicine in these countries. To ensure a valid and reliable examination, unwanted variance that can be introduced by SP performance must be addressed. The goal of SP training is to imbue the SP with the characteristics, mannerisms and history of a real patient so that the portrayal is consistent and accurate. The challenge is to ensure consistent portrayal of each case with sufficient realism to elicit the expected clinical performance and to ensure standardized SP performance across multiple examinees. This paper considers the quality assurance methods applied to training the SP trainers and the protocols used to train the SPs, to ensure that the SP performances are sufficiently accurate and standardized, and that the evaluators completing the checklists and scales used for scoring do so correctly and consistently.
Collapse
Affiliation(s)
- Gail E Furman
- Clinical Skills Evaluation Collaboration, National Board of Medical Examiners, Philadelphia 19147, USA.
| |
Collapse
|
15
|
Seung KJ, Bitalabeho A, Buzaalirwa LE, Diggle E, Downing M, Bhatt Shah M, Tumwebaze B, Gove S. Standardized patients for HIV/AIDS training in resource-poor settings: the expert patient-trainer. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2008; 83:1204-1209. [PMID: 19202501 DOI: 10.1097/acm.0b013e31818c72ac] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/27/2023]
Abstract
This article presents a unique approach to HIV/AIDS training in resource-poor settings that incorporates the use of standardized patients (SPs). Integrated Management of Adolescent and Adult Illness (IMAI) is a World Health Organization health systems strengthening initiative with a strong emphasis on training health workers in the management of common diseases and conditions. In IMAI, SPs are called Expert Patient-Trainers (EPTs) to emphasize their role in the training of health workers. EPTs were first used in IMAI training in Uganda in 2004. Since then, the method has been adopted by a number of other countries in Africa, Latin America, and Asia. EPTs are usually recruited from groups of people living with HIV/AIDS. In the classroom, EPTs discuss living with HIV and help participants understand HIV as it affects patients. Course participants spend approximately two hours per day in "skill stations," multiple-station assessments consisting of one-on-one encounters with EPTs. In each encounter, the health worker interacts with an EPT portraying a standardized case. Instructions on how to portray each case provide only broad outlines of the major clinical and counseling points; the EPT is expected to use his or her own life experiences to fill in emotional details. Course facilitators noted that health workers were often initially skeptical about EPTs, but this generally turned to enthusiasm after participating in the skill stations. EPTs benefited from the sense of being part of the training team, the satisfaction of improving the skills of health workers, and learning more about their illness.
Collapse
Affiliation(s)
- Kwonjune J Seung
- Harvard Medical School, and Division of Global Health Equity, Brigham and Women's Hospital, Boston, Massachusetts 02115, USA.
| | | | | | | | | | | | | | | |
Collapse
|
16
|
Rinker B, Donnelly M, Vasconez HC. Teaching patient selection in aesthetic surgery: use of the standardized patient. Ann Plast Surg 2008; 61:127-31; discussion 132. [PMID: 18650601 DOI: 10.1097/sap.0b013e31815b07b2] [Citation(s) in RCA: 12] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
This study evaluates the effectiveness of standardized patients for teaching patient selection in esthetic surgery. Six actors received detailed character descriptions. Each was interviewed by a resident for 30 minutes in a conference setting. Participants completed a questionnaire, and a discussion was held. Written simulations were administered before and after the sessions, from which a faculty standard was developed using hierarchical cluster analysis. Resident responses were compared with the standard using a squared euclidean metric. Residents' pre- and posttest scores were compared with the faculty standard using 2-way analysis of variance. Accuracy scores were found to be significantly lower (more accurate) after the training than before (P < 0.001). Upon a 6-item questionnaire, both faculty and residents agreed that it was a worthwhile exercise (faculty mean, 6.2 out of 7; resident, 6.3) and that the standardized patients were believable. Standardized patients can provide effective instruction in traditionally difficult-to-teach areas such as communication and patient selection.
Collapse
Affiliation(s)
- Brian Rinker
- Division of Plastic Surgery, University of Kentucky, Lexington, KY 40536-0284, USA.
| | | | | |
Collapse
|
17
|
Yudkowsky R, Downing SM, Wirth S. Simpler standards for local performance examinations: the Yes/No Angoff and whole-test Ebel. TEACHING AND LEARNING IN MEDICINE 2008; 20:212-7. [PMID: 18615294 DOI: 10.1080/10401330802199450] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/13/2023]
Abstract
BACKGROUND The whole-test Ebel and Yes/No Angoff methods offer the possibility of faster and easier standard-setting exercises for local, medium-stakes performance exams. PURPOSE We sought to establish if these less demanding methods might be substituted for the traditional but complex case-level Ebel. METHODS Six faculty judges each used all three methods to set standards for six standardized-patient checklists used for 178 fourth-year medical students. We compared the cut scores, passing rates, reliability, ease of use, and credibility of the three methods. RESULTS The three methods produced roughly equivalent cut scores. Generalizability of judgments was .94 and .96 for the case-level and whole-test Ebel, and .76 for the Yes/No Angoff. Judges found the simplified methods more credible than the case-level Ebel. The Yes/No Angoff method was preferred by five of six judges. CONCLUSIONS The whole-test Ebel and the Yes/No Angoff may be simple and realistic options for setting standards for local performance exams.
Collapse
Affiliation(s)
- Rachel Yudkowsky
- Department of Medical Education, University of Illinois at Chicago, Chicago, Illinois 60612-7309, USA.
| | | | | |
Collapse
|
18
|
Howley L, Szauter K, Perkowski L, Clifton M, McNaughton N. Quality of standardised patient research reports in the medical education literature: review and recommendations. MEDICAL EDUCATION 2008; 42:350-8. [PMID: 18298448 DOI: 10.1111/j.1365-2923.2007.02999.x] [Citation(s) in RCA: 25] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/17/2023]
Abstract
CONTEXT In order to assess or replicate the research findings of published reports, authors must provide adequate and transparent descriptions of their methods. We conducted 2 consecutive studies, the first to define reporting standards relating to the use of standardised patients (SPs) in research, and the second to evaluate the current literature according to these standards. METHODS Standards for reporting SPs in research were established by representatives of the Grants and Research Committee of the Association of Standardized Patient Educators (ASPE). An extensive literature search yielded 177 relevant English-language articles published between 1993 and 2005. Search terms included: 'standardised patient(s)'; 'simulated patient(s)'; 'objective structured clinical examination (OSCE)', and 'clinical skills assessment'. Articles were limited to those reporting the use of SPs as an outcome measure and published in 1 of 5 prominent health sciences education journals. Data regarding the SP encounter, SP characteristics, training and behavioural measure(s) were gathered. RESULTS A random selection of 121 articles was evaluated according to 29 standards. Reviewers judged that few authors provided sufficient details regarding the encounter (21%, n = 25), SPs (16%, n = 19), training (15%, n = 15), and behavioural measures (38%, n = 44). Authors rarely reported SP gender (27%, n = 33) and age range (22%, n = 26), whether training was provided for the SPs (39%, n = 47) or other raters (24%, n = 29), and psychometric evidence to support the behavioural measure (23%, n = 25). CONCLUSIONS The findings suggest that there is a need for increased rigor in reporting research involving SPs. In order to support the validity of research findings, journal editors, reviewers and authors are encouraged to provide adequate detail when describing SP methodology.
Collapse
Affiliation(s)
- Lisa Howley
- Department of Educational Leadership, University of North Carolina at Charlotte, Charlotte, North Carolina, USA.
| | | | | | | | | |
Collapse
|
19
|
Handley LI, Warholak Jackson TL, Jackson TR. An evaluation of the validity of inferences made from 3 diabetes assessment instruments: A Rasch analysis. Res Social Adm Pharm 2008; 4:67-81. [PMID: 18342824 DOI: 10.1016/j.sapharm.2007.04.002] [Citation(s) in RCA: 12] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2006] [Revised: 04/07/2007] [Accepted: 04/07/2007] [Indexed: 11/18/2022]
Affiliation(s)
- Lisa Izzo Handley
- Department of Pharmacy Practice, Midwestern University, Chicago College of Pharmacy, Downers Grove, IL, USA.
| | | | | |
Collapse
|
20
|
Huang YM, Pliego JF, Henrichs B, Bowyer MW, Siddall VJ, McGaghie WC, Raemer DB. 2007 Simulation Education Summit. Simul Healthc 2008; 3:186-91. [DOI: 10.1097/sih.0b013e3181828156] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/22/2023]
|
21
|
Hodgson JL, Lamson AL, Feldhousen EB. Use of simulated clients in marriage and family therapy education. JOURNAL OF MARITAL AND FAMILY THERAPY 2007; 33:35-50. [PMID: 17257379 DOI: 10.1111/j.1752-0606.2007.00003.x] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/13/2023]
Abstract
Knowledge of how one should manage suicidal, homicidal, child maltreatment, and domestic violence situations is paramount in the training of marriage and family therapists (MFTs). Simulated patient modules were created to help clinical faculty address these crisis situations in a protected learning environment. The modules were implemented by the MFT faculty in collaboration with the Office of Clinical Skills Assessment and Education at East Carolina University's Brody School of Medicine. Qualitative data over the course of 2 years revealed six thematic domains regarding therapists' performance, therapists' emotions, the simulation experiences, and lessons learned. Educational, clinical, and research recommendations include tools to implement simulation exercises into marriage and family therapy programs as well as suggestions to assess for teaching effectiveness.
Collapse
Affiliation(s)
- Jennifer L Hodgson
- Department of Child Development and Family Relations, East Carolina University, Greenville, NC 27858-4353, USA.
| | | | | |
Collapse
|
22
|
Abstract
Teaching nursing students therapeutic communication skills begins in the classroom and extends to the clinical environment. The usual method of instruction consists of random patient encounters observed by faculty and measures of competence that rely on paper-and-pencil tests. Using standardized patients (SPs) offers an alternative approach to the traditional method of teaching. Standardized patients are individuals who have been carefully trained to present an illness or scenario in a standardized, unvarying manner. This pilot study compared use of SPs with the usual method of instruction in a class of undergraduate nursing students. Results indicated that students who participated in the SP method overwhelmingly described the experience as positive, creative, and meaningful. No significant differences were found between the two groups on measures of interpersonal skills, therapeutic communication skills, and knowledge of depression
Collapse
Affiliation(s)
- Kathleen L Becker
- Adult Nurse Practitioner Program, Johns Hopkins University School of Nursing, Baltimore, MD 21201, USA.
| | | | | | | | | |
Collapse
|
23
|
Sidhu RS, McIlroy JH, Regehr G. Using a Comprehensive Examination to Assess Multiple Competencies in Surgical Residents: Does the Oral Examination Still Have a Role? J Am Coll Surg 2005; 201:754-8. [PMID: 16256920 DOI: 10.1016/j.jamcollsurg.2005.06.264] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2005] [Revised: 06/20/2005] [Accepted: 06/22/2005] [Indexed: 10/25/2022]
Abstract
BACKGROUND While specialty-level evaluations evolve from traditional examinations to objective structured clinical examination-like assessments, a broader range of competencies are tested; consequently, examiners are forced to integrate results when making a determination of competency. The aim of this study was to describe how experts weigh relative performances on specific components of a comprehensive examination to make decisions of overall competency. STUDY DESIGN The Patient Assessment and Management Examination is a standardized patient-based assessment of general surgery in which each 25-minute station encompasses four components: history and physical examination, investigation interpretation, diagnosis and treatment discussion with the patient, and a structured oral examination (SOE). A six-station Patient Assessment and Management Examination was administered to 21 senior surgery residents. Surgeons marked each station with global rating scales and, in addition, provided an end-of-station overall global assessment of performance. A "gold-standard" examination pass-or-fail decision was determined through videotape review of each candidate's performance across six stations by two blinded surgeons. Multiple linear regression analysis was used to determine which components were associated with the end-of-station overall global assessments. Multivariable logistic regression was used to determine which components were associated with the final "gold-standard" pass-or-fail assessment. RESULTS The only component notably (p < 0.005) associated with end-of-station global assessment for all six stations was the SOE. Mean SOE score was the only notable independent variable associated with the gold-standard pass-or-fail decision (R(2) = 0.63, p < 0.001). CONCLUSIONS Performance on the SOE section of a multicompetency examination is markedly associated with the final determination of competency. These results have implications for the design and implementation of comprehensive specialty-level assessments.
Collapse
Affiliation(s)
- Ravi S Sidhu
- Department of Surgery, University of British Columbia, Vancouver, British Columbia, Canada
| | | | | |
Collapse
|
24
|
Roberts MC, Borden KA, Christiansen MD, Lopez SJ. Fostering a Culture Shift: Assessment of Competence in the Education and Careers of Professional Psychologists. ACTA ACUST UNITED AC 2005. [DOI: 10.1037/0735-7028.36.4.355] [Citation(s) in RCA: 90] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
25
|
Auewarakul C, Downing SM, Praditsuwan R, Jaturatamrong U. Item analysis to improve reliability for an internal medicine undergraduate OSCE. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2005; 10:105-13. [PMID: 16078095 DOI: 10.1007/s10459-005-2315-3] [Citation(s) in RCA: 10] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/13/2004] [Accepted: 02/16/2005] [Indexed: 05/03/2023]
Abstract
Utilization of objective structured clinical examinations (OSCEs) for final assessment of medical students in Internal Medicine requires a representative sample of OSCE stations. The reliability and generalizability of OSCE scores provides validity evidence for OSCE scores and supports its contribution to the final clinical grade of medical students. The objective of this study was to perform item analysis using OSCE stations as the unit of analysis and evaluate the extent to which OSCE score reliability can be improved using item analysis data. OSCE scores from eight cohorts of fourth-year medical students (n = 435) in a 6-year undergraduate program were analyzed. Generalizability (G) coefficients of OSCE scores were computed for each cohort. Item analysis was performed by considering each OSCE station as an item and computing the corrected item-total correlation. OSCE stations which negatively impacted the reliability were deleted and the G-coefficient was recalculated. The G-coefficients of OSCE scores from the eight cohorts ranged from 0.48 to 0.80 (median 0.62). The median number of OSCE stations that negatively impacted the G-coefficient was 3.5 (out of a median of 25 total stations). When the ''problem stations'' were deleted, the median G-coefficient across eight cohorts increased to 0.62--0.72. In conclusion, item analysis of OSCE stations is useful and should be performed to improve the reliability of total OSCE scores. Problem stations can then be identified and improved.
Collapse
Affiliation(s)
- Chirayu Auewarakul
- Department of Medicine, Faculty of Medicine Siriraj Hospital, Mahidol University, Bangkok, Thailand.
| | | | | | | |
Collapse
|
26
|
Battles JB, Wilkinson SL, Lee SJ. Using standardised patients in an objective structured clinical examination as a patient safety tool. Qual Saf Health Care 2004. [PMID: 15465955 DOI: 10.1136/qshc.2004.009803] [Citation(s) in RCA: 22] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/03/2022]
Abstract
Standardised patients (SPs) are a powerful form of simulation that has now become commonplace in training and assessment in medical education throughout the world. Standardised patients are individuals, with or without actual disease, who have been trained to portray a medical case in a consistent manner. They are now the gold standard for measuring the competence of physicians and other health professionals, and the quality of their practice. A common way in which SPs are used in performance assessment has been as part of an objective structured clinical examination (OSCE). The use of an SP based OSCE can be a powerful tool in measuring continued competence in human reliability and skill performance where such skills are a critical attribute to maintaining patient safety. This article will describe how an OSCE could be used as a patient safety tool based on cases derived from actual events related to postdonation information in the blood collection process. The OSCE was developed as a competency examination for health history takers. Postdonation information events in the blood collection process account for the majority of errors reported to the US Food and Drug Administration. SP based assessment is an important patient safety tool that could be applied to a variety of patient safety settings and situations, and should be considered an important weapon in the war on medical error and patient harm.
Collapse
Affiliation(s)
- J B Battles
- University of Texas Southwestern Medical Center at Dallas, USA.
| | | | | |
Collapse
|
27
|
Battles JB, Wilkinson SL, Lee SJ. Using standardised patients in an objective structured clinical examination as a patient safety tool. Qual Saf Health Care 2004; 13 Suppl 1:i46-50. [PMID: 15465955 PMCID: PMC1765798 DOI: 10.1136/qhc.13.suppl_1.i46] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Standardised patients (SPs) are a powerful form of simulation that has now become commonplace in training and assessment in medical education throughout the world. Standardised patients are individuals, with or without actual disease, who have been trained to portray a medical case in a consistent manner. They are now the gold standard for measuring the competence of physicians and other health professionals, and the quality of their practice. A common way in which SPs are used in performance assessment has been as part of an objective structured clinical examination (OSCE). The use of an SP based OSCE can be a powerful tool in measuring continued competence in human reliability and skill performance where such skills are a critical attribute to maintaining patient safety. This article will describe how an OSCE could be used as a patient safety tool based on cases derived from actual events related to postdonation information in the blood collection process. The OSCE was developed as a competency examination for health history takers. Postdonation information events in the blood collection process account for the majority of errors reported to the US Food and Drug Administration. SP based assessment is an important patient safety tool that could be applied to a variety of patient safety settings and situations, and should be considered an important weapon in the war on medical error and patient harm.
Collapse
Affiliation(s)
- J B Battles
- University of Texas Southwestern Medical Center at Dallas, USA.
| | | | | |
Collapse
|
28
|
Pedregal González M, Molina Fernández E, Prados Castillejo JA, Quesada Jiménez F, Bonal Pitz P, Iglesias Arrabal C. [Evaluation of clinical competence of family medicine trainer]. Aten Primaria 2004; 34:68-72. [PMID: 15225526 PMCID: PMC7668705 DOI: 10.1016/s0212-6567(04)79461-5] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/27/2022] Open
Abstract
OBJECTIVES We show the first experience of the application of an objective and structured clinical evaluation (OSCE) procedure to family medicine trainers, that has been carried out in Andalucia. The objective is to use a competence evaluation instrument that, in the short term, will be used not only for trainer accreditation but also for other public sanitary professionals. PARTICIPANTS Tutors of family and community medicine residents. DESIGN Observational descriptive. SETTING Educational unity of family medicine. PRINCIPAL MEASUREMENTS The competencial components to be assessed are the following: anamnesis, physical exploration, communication, technical skill, management, family attention y preventive activities. The clinical situations were selected using the following priority criteria: prevalence, clinical gravity, prevention and early diagnosis importance, case complexity, doctor's capacity of evaluation and simplicity. RESULTS Thirteen family medicine trainers took part in the OSCE. Their average age was 42.8 +/- 3.6 years. The test had an overall reliability coefficient (Cronbach's alpha) of 0.73. The overall mean score of the participants was 73 +/- 6.2. The best results about the competencial components were family attention, communication and technical skill. CONCLUSIONS The OSCE can be a convenient tool for family medical trainer evaluation, helping to orientate their education in the weak points and, in the near future, it can also be used as an instrument do accredit family medicine trainers.
Collapse
|
29
|
Petrusa ER. Taking standardized patient-based examinations to the next level. TEACHING AND LEARNING IN MEDICINE 2004; 16:98-110. [PMID: 14987183 DOI: 10.1207/s15328015tlm1601_19] [Citation(s) in RCA: 16] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/24/2023]
Abstract
PURPOSE The purpose of this article is to summarize this author's view on "where we are" with standardized patient-based assessments of clinical performance and to offer three directions for further research and development. SUMMARY The push for more objective outcome data has fueled proliferation of the most researched innovation in the history of medical education. Near-random clinical experiences of students do not provide consistent, repeated practice with important clinical cases to achieve minimally adequate performance on these objective performance examinations, leading to scoring "psychogymnastics" to titrate fail rates. The second area is to modify these examinations to reflect features at higher levels of professional development such as situational awareness. Theories of professional development should guide changes. The third area incorporates multiperson scenarios; a clinician with a family or a team in the operating room. Simulation of complex situations, especially those requiring rapid, accurate communication and action can reduce medical errors and improve patient safety. CONCLUSIONS Standardized patient-based examinations provide objective outcome data but require artificial adjustments in scoring due to inconsistent learning opportunities. Theoretical research on professional development, acquisition of expertise and team functioning provides fertile, new directions to take standardized patient-based examinations to the next level.
Collapse
Affiliation(s)
- Emil R Petrusa
- Associate Dean for Curriculum Assessment, Duke University School of Medicine, Durham, North Carolina 27710, USA.
| |
Collapse
|
30
|
Williams RG. Have standardized patient examinations stood the test of time and experience? TEACHING AND LEARNING IN MEDICINE 2004; 16:215-222. [PMID: 15446298 DOI: 10.1207/s15328015tlm1602_16] [Citation(s) in RCA: 25] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/24/2023]
Affiliation(s)
- Reed G Williams
- Department of Surgery, Southern Illinois University School of Medicine, Springfield, Illinois, USA
| |
Collapse
|
31
|
Lee SJ, Wilkinson SL, Battles JB, Hynan LS. An objective structured clinical examination to evaluate health historian competencies. Transfusion 2003; 43:34-41. [PMID: 12519428 DOI: 10.1046/j.1537-2995.2003.00275.x] [Citation(s) in RCA: 12] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
Abstract
BACKGROUND Post-donation information events in the blood-collection process account for the majority of errors reported to the FDA. An eight-station objective structured clinical examination (OSCE) based on information reported after donation was developed as a competency examination for health historians. STUDY DESIGN AND METHODS The OSCE measured two individual skill components: history-taking technique (HXE) and ability to record and interpret a medical history (HXI). Donor historians at the Hoxworth Blood Center, University of Cincinnati Medical Center, were examined (n = 56). RESULTS In general, staff performance was acceptable, but several areas for improvement were identified. Of particular concern were the deferrals associated with the malaria scenarios. The overall reliability score was greater than 0.8 for the total OSCE and HXI. Using the cut score of 0.9, or 90 percent, 10 subjects would not have passed the exam based on the HXE score and one person would not have demonstrated competency based on the HXI score. CONCLUSION An OSCE is a reliable, valid, and practical method for assessing continued competency in health historians. This form of competency assessment and subsequent retraining may reduce the incidence of errors in information reported after donation and should be further studied as a suitable national standard for assessing competency.
Collapse
Affiliation(s)
- Stacy J Lee
- Blood Center of Southeastern Wisconsin, Milwaukee, 53233-2121, USA.
| | | | | | | |
Collapse
|
32
|
Williams RG, Klamen DA, McGaghie WC. Cognitive, social and environmental sources of bias in clinical performance ratings. TEACHING AND LEARNING IN MEDICINE 2003; 15:270-92. [PMID: 14612262 DOI: 10.1207/s15328015tlm1504_11] [Citation(s) in RCA: 272] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
BACKGROUND Global ratings based on observing convenience samples of clinical performance form the primary basis for appraising the clinical competence of medical students, residents, and practicing physicians. This review explores cognitive, social, and environmental factors that contribute unwanted sources of score variation (bias) to clinical performance evaluations. SUMMARY Raters have a 1 or 2-dimensional conception of clinical performance and do not recall details. Good news is reported more quickly and fully than bad news, leading to overly generous performance evaluations. Training has little impact on accuracy and reproducibility of clinical performance ratings. CONCLUSIONS Clinical performance evaluation systems should assure broad, systematic sampling of clinical situations; keep rating instruments short; encourage immediate feedback for teaching and learning purposes; encourage maintenance of written performance notes to support delayed clinical performance ratings; give raters feedback about their ratings; supplement formal with unobtrusive observation; make promotion decisions via group review; supplement traditional observation with other clinical skills measures (e.g., Objective Structured Clinical Examination); encourage rating of specific performances rather than global ratings; and establish the meaning of ratings in the manner used to set normal limits for clinical diagnostic investigations.
Collapse
Affiliation(s)
- Reed G Williams
- Department of Surgery, Southern Illinois University School of Medicine, PO Box 19638, Springfield, IL 62794-9638, USA.
| | | | | |
Collapse
|
33
|
Morrison EH, Boker JR, Hollingshead J, Prislin MD, Hitchcock MA, Litzelman DK. Reliability and validity of an objective structured teaching examination for generalist resident teachers. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2002; 77:S29-S32. [PMID: 12377697 DOI: 10.1097/00001888-200210001-00010] [Citation(s) in RCA: 24] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/23/2023]
Affiliation(s)
- Elizabeth H Morrison
- Department of Family Medicine, Unviersity of California, 101 City Drive South, Orange, CA 92868-3298, USA
| | | | | | | | | | | |
Collapse
|
34
|
Grant C, Nicholas R, Moore L, Salisbury C. An observational study comparing quality of care in walk-in centres with general practice and NHS Direct using standardised patients. BMJ 2002; 324:1556. [PMID: 12089094 PMCID: PMC116613 DOI: 10.1136/bmj.324.7353.1556] [Citation(s) in RCA: 36] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 04/04/2002] [Indexed: 11/04/2022]
Abstract
OBJECTIVES To compare the quality of clinical care in walk-in centres with that provided in general practice and by NHS Direct. DESIGN Observational study involving assessment of clinicians by standardised patients. SETTING 20 walk-in centres, 20 general practices, and 11 NHS Direct sites. PARTICIPANTS 297 consultations with standardised patients, 99 in each setting, carried out by professional role players trained to play five clinical scenarios (postcoital contraception, chest pain, sinusitis, headache, and asthma). MAIN OUTCOME MEASURES Primary outcomes were mean scores on consensus derived checklists of essential items for the management of the clinical scenarios. Data were also collected on access to and referral by walk-in centres, general practices, and NHS Direct. RESULTS Walk-in centres achieved a significantly greater mean score for all scenarios combined than general practices (difference between groups 8.2, 95% confidence interval 1.7 to 14.6) and NHS Direct (10.8, 5.5 to 16.1). There was considerable between scenario variation, with walk-in centres performing particularly well on postcoital contraception and asthma scenarios. In contrast to general practices, walk-in centres and NHS Direct referred a higher proportion of patients (26% and 82%, respectively). CONCLUSION Walk-in centres perform adequately and safely compared with general practices and NHS Direct for the range of conditions under study, but the impact of referrals on workload of other healthcare providers requires further research.
Collapse
Affiliation(s)
- Clare Grant
- Division of Primary Health Care, University of Bristol, Bristol BS6 6JL
| | | | | | | |
Collapse
|
35
|
Ruiz I Gil E, Florensa I Claramunt E, Cots I Yago J, Sellarès Salas J, Iruela López A, Blay Pueyo C, Morera Castell R, Martínez Carretero J. [First experiences in evaluating the clinical competence of family doctors in Catalonia]. Aten Primaria 2001; 28:105-9. [PMID: 11440647 PMCID: PMC7675932 DOI: 10.1016/s0212-6567(01)78909-3] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/19/2001] [Indexed: 11/23/2022] Open
Abstract
OBJECTIVES To construct a valid instrument to evaluate the clinical competence of family doctors and establish professional standards. DESIGN Construction of an objective and structured clinical evaluation test by the Test Committee of the Catalan Society of Family and Community Medicine (SCMFiC) with 27 representative clinical situations of primary care practice. This used various evaluative instruments and lasted 5 hours and 30 minutes. SETTING 2 Barcelona health districts for the two runnings of the test. PARTICIPANTS Family doctors, SCMFiC members, who took part voluntarily after having been invited by letter. MEASUREMENTS AND MAIN RESULTS 23 doctors took part in the first running of the test. Their average age was 36.6 (SD 6.9) and 52.2% of them were women. 46 doctors took part in the second test. Their average age was 30.9 (SD 2.1) and 76.1 were women. The test had an overall reliability coefficient (Cronbach's alpha) of 0.83 the first time and 0.65 the second time. The overall mean score of the participants the first time was 56.9 (SD 19.2), with 100 the top score possible, and 62.1 (SD 4.1) the second time. CONCLUSIONS This SCMFiC experiment is the first time the clinical competence of family doctors has been assessed in our area. It was a reliable and valid test to assess the competence of our health professionals.
Collapse
Affiliation(s)
- E Ruiz I Gil
- Societat Catalana de Medicina Familiar i Comunitària e Institut d'Estudis de la Salut. Barcelona.
| | | | | | | | | | | | | | | |
Collapse
|
36
|
Humphris GM, Kaney S. Examiner fatigue in communication skills objective structured clinical examinations. MEDICAL EDUCATION 2001; 35:444-9. [PMID: 11328514 DOI: 10.1046/j.1365-2923.2001.00893.x] [Citation(s) in RCA: 15] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/17/2023]
Abstract
CONTEXT The assessment of undergraduates' communication skills by means of objective structured clinical examinations (OSCEs) is a demanding task for examiners. Tiredness over the course of an examining session may introduce systematic error. In addition, unsystematic error may also be present which changes over the duration of the OSCE session. AIM To determine the strength of some sources of systematic and unsystematic error in the assessment of communication skills over the duration of an examination schedule. METHODS Undergraduate first-year medical students completing their initial summative assessment of communication skills (a four-station OSCE) comprised the study population. Students from three cohorts were included (1996-98 intake). In all 3 years the OSCE was carried out identically. All stations lasted 5 minutes with a simulated patient. Students were assessed using an examiner (content expert) and a simulated-patient evaluation tool, the Liverpool Communication Skills Assessment Scale (LCSAS) and the Global Simulated-patient Rating Scale (GSPRS), respectively. Each student was assigned a time slot ranging from 1 to 24, where 1, for example, would denote that the student entered the exam first and 24 indicates the final slot for entry into the examination. The number of students who failed this exam was noted for each of the 24 time slots. A control set of marks from a communication skills written exam was also adopted for exploring a possible link with the time slot. Analysis was conducted using graphical display, covariate analysis and logistic regression. RESULTS No significant relationship was found between the schedule point that the student entered the OSCE exam and their performance. The reliability of the content expert and simulated-patient assessments was stable throughout the session. CONCLUSION No evidence could be found that duration of examining in a communication OSCE influenced examiners and the marks they awarded. Checks of this nature are recommended for routine inspection to confirm a lack of bias.
Collapse
Affiliation(s)
- G M Humphris
- Department of Clinical Psychology, Whelan Building, The University of Liverpool, Liverpool L69 3GB, UK
| | | |
Collapse
|
37
|
Jain SS, DeLisa JA, Nadler S, Kirshblum S, Banerjee SN, Eyles M, Johnston M, Smith AC. One program's experience of OSCE vs. written board certification results: a pilot study. Am J Phys Med Rehabil 2000; 79:462-7. [PMID: 10994889 DOI: 10.1097/00002060-200009000-00012] [Citation(s) in RCA: 16] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/11/2023]
Abstract
The objective structured clinical examination (OSCE) has been the focus of a lot debate with respect to reliability and validity. Much of the controversy surrounding these components lies in a lack of comparison with a "gold standard." Further work is needed to improve the evaluation of clinical skills to the point that a gold standard can truly be said to exist.
Collapse
Affiliation(s)
- S S Jain
- Department of Physical Medicine and Rehabilitation, UMDNJ-New Jersey Medical School, Newark, USA
| | | | | | | | | | | | | | | |
Collapse
|
38
|
Duerson MC, Romrell LJ, Stevens CB. Impacting faculty teaching and student performance: nine years' experience with the Objective Structured Clinical Examination. TEACHING AND LEARNING IN MEDICINE 2000; 12:176-82. [PMID: 11273366 DOI: 10.1207/s15328015tlm1204_3] [Citation(s) in RCA: 24] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/11/2023]
Abstract
BACKGROUND The impetus for administering the 2nd-year Objective Structured Clinical Examination (OSCE) came from the great variability in student performance observed by 3rd-year clerkship directors. PURPOSE To document the effects of the OSCE on faculty teaching, student performance, and the curriculum over 9 years of administration of the examinations to more than 1,000 second-year medical students. METHOD A 20-station OSCE was administered to all medical students at the end of their 2nd year. Using predetermined criteria, clinical faculty served as evaluators in each station. A mix of 1st-, 3rd-, and 4th-year medical students were recruited to serve as simulated patients. Faculty evaluators and examinees completed a questionnaire evaluating their experience with the OSCE. Students received a report card of their performance. Small-group leaders of the Introduction to Clinical Medicine course received feedback on their group's performance on each station compared to the class mean. Summative data on class performance was reported to the curriculum committee. The academic status committee received data on students who performed unsatisfactorily. RESULTS Faculty and examinee ratings of the OSCE experience were very positive. Over the 9-year period, student performance improved showing less variability and significantly fewer failed stations. CONCLUSION The OSCE has proven to be a technically feasible, authentic evaluation method yielding valuable information for decisions regarding student performance, faculty teaching, and curriculum planning.
Collapse
Affiliation(s)
- M C Duerson
- Harrell Professional Development and Assessment Center, University of Florida, P.O. Box 100281, Gainesville, FL 32610-0281, USA.
| | | | | |
Collapse
|
39
|
O'Connor FW, Albert ML, Thomas MD. Incorporating standardized patients into a psychosocial nurse practitioner program. Arch Psychiatr Nurs 1999; 13:240-7. [PMID: 10565057 DOI: 10.1016/s0883-9417(99)80034-x] [Citation(s) in RCA: 30] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
Interactions with standardized patients (SPs) provide customized, immediate clinical learning for graduate nursing students. SPs are lay persons trained to portray a role in a faculty-designed health care scenario for a series of students. Student learning results from the experience of the encounter, performance feedback from the SP, subsequent classroom discussion with peers, faculty critique, and review of the videotaped interaction. Design and implementation of an SP experience involves clarification of goals for the encounter, development of the clinical scenario, recruitment and training of SPs, preparation of students, and management of the logistics of the exercise. Although the process is labor-intensive, we have found SP experiences versatile, valuable, and popular with students.
Collapse
|
40
|
Hampl JS, Herbold NH, Schneider MA, Sheeley AE. Using standardized patients to train and evaluate dietetics students. JOURNAL OF THE AMERICAN DIETETIC ASSOCIATION 1999; 99:1094-7. [PMID: 10491679 DOI: 10.1016/s0002-8223(99)00261-8] [Citation(s) in RCA: 23] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
Faced with a dwindling number of clinical sites for training dietetics students, we incorporated a standardized patient scenario of a 17-year-old pregnant woman into a nutrition counseling course. A young woman was hired to be the patient and was trained extensively. Each student signed up for a 20-minute block of time to assess her and provide nutrition instruction, after which the standardized patient and course instructor provided individualized feedback to each student. Students then completed a brief questionnaire related to the standardized patient session. Overall, student response to the standardized patient session was positive, and students believed standardized patients should be included formally in undergraduate training. The students noted that the standardized patient session allowed them to be trained and evaluated objectively. Because of their ease in building rapport with the standardized patient, students reported that they were able to provide more in-depth education. They also appreciated the immediate and detailed performance review provided by the standardized patient and the instructor. We conclude that standardized patient sessions provide a great service to dietetics students, and plans to develop another standardized patient scenario that students can follow up throughout the continuum of care are underway.
Collapse
Affiliation(s)
- J S Hampl
- Department of Family Resources and Human Development, Arizona State University, Tempe 85287-2502, USA
| | | | | | | |
Collapse
|
41
|
McGraw RC, O'Connor HM. Standardized patients in the early acquisition of clinical skills. MEDICAL EDUCATION 1999; 33:572-578. [PMID: 10447842 DOI: 10.1046/j.1365-2923.1999.00381.x] [Citation(s) in RCA: 33] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/23/2023]
Abstract
OBJECTIVES Facing dramatic reductions of inpatient beds and fewer faculty tutors, the Queen's University medical school has had to consider alternatives to the traditional inpatient encounter for the early acquisition of clinical skills. The purpose of this study was to investigate the feasibility and impact on students of a model for a first-year clinical skills course, using exclusively standardized patients in place of inpatients, and using a smaller ratio of faculty to students. DESIGN Twenty volunteers were selected from the first-year class of 75 students to participate as the experimental group. The remaining 55 students formed the control group. SETTING Queen's University medical school, Ontario, Canada. SUBJECTS First-year medical students. RESULTS In measures of student satisfaction with the amount of performance feedback received during the course, there was a non-significant trend towards greater satisfaction among the experimental group compared to the control group. This occurred despite a tutor to student ratio of 1.5:10 in the experimental group compared to 3:10 in the control group. In the student evaluation of their tutors, mean scores for the two groups were 4.28 for the experimental group and 4. 06 for the control group (P = 0.10). The mean OSCE scores for the two groups were 76.4 for the experimental group and 76.5 for the control group (P = 0.93). There is no practical or statistical difference in either of these two scores. CONCLUSIONS We conclude that this new model is feasible, was well received by the students and that the reduced tutor to student ratio and lack of inpatient encounters did not appear to adversely affect their learning.
Collapse
Affiliation(s)
- R C McGraw
- Department of Emergency Medicine, Queen's University, Kingston, Ontario, Canada
| | | |
Collapse
|
42
|
Gispert R, Rué M, Roma J, Martinez-Carretero JM. Gender, sequence of cases and day effects on clinical skills assessment with standardized patients. MEDICAL EDUCATION 1999; 33:499-503. [PMID: 10354333 DOI: 10.1046/j.1365-2923.1999.00349.x] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/17/2023]
Abstract
OBJECTIVE Clinical skills assessment using standardized patients (SPs) is an accepted method of professional testing, but some factors can threaten the validity of the scores obtained. OBJECTIVE The main objective of this work was to test whether the gender of the patients, the sequence of cases and the day of the examination have any impact on the scores of the examinees. MATERIALS Data from examinations conducted at three Medical Schools are used. Each student encountered 10 different SPs (5 men and 5 women). METHODS Analysis of variance was used to test the existence of association between the variables. RESULTS The results showed no significant differences or association between the scores and the three variables analysed. CONCLUSIONS The results are coherent with the studies that show that performance based examination using SPs can be used without introducing biases into students score.
Collapse
Affiliation(s)
- R Gispert
- Institut d'Estudis de la Salut, Barcelona, Spain
| | | | | | | |
Collapse
|
43
|
Jacobs DS. Teaching doctors about the eye: trends in the education of medical students and primary care residents. Surv Ophthalmol 1998; 42:383-9. [PMID: 9493282 DOI: 10.1016/s0039-6257(97)00121-5] [Citation(s) in RCA: 34] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
Abstract
Ophthalmologists play a relatively peripheral role in medical student and resident education. A review of the evolution, funding, and administration of medical education in the United States lends insight into why this is so. The author reviews the current status of education in ophthalmology for medical students and residents; the development of an ophthalmology curriculum; alternatives to the traditional medical school curriculum and how these have been incorporated into ophthalmic education; and the effect of new trends in medical education and pressures created by health care reform on the education of medical students and residents. The role of ophthalmologists in the general practice of medicine and in teaching doctors about the eye is discussed. Finally, the opportunity that the current climate presents for ophthalmologists to define their role in health care through education is considered.
Collapse
Affiliation(s)
- D S Jacobs
- Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
44
|
MacRae HM, Cohen R, Regehr G, Reznick R, Burnstein M. A new assessment tool: the patient assessment and management examination. Surgery 1997; 122:335-43; discussion 343-4. [PMID: 9288139 DOI: 10.1016/s0039-6060(97)90025-8] [Citation(s) in RCA: 42] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
Abstract
BACKGROUND The major goal of certification is to assure the public that the candidate is competent in all facets required of the position. The patient assessment and management examination (PAME) was developed to enable a more comprehensive assessment of competence in the practice of surgery. METHODS A six-station, 3-hour, standardized-patient-based evaluation was developed. Each station was scored using a set of five-point global rating scales. PAME results were compared to the last two in training evaluation reports (ITER), the clinical knowledge component of the ITER (ITER-CK), an in-house oral examination (OE), and the Canadian Association of General Surgeons' multiple-choice examination (CAGS). RESULTS Eighteen senior general surgery residents were evaluated. Overall reliability was 0.70 (Cronbach's alpha). Fifth-year residents scored significantly better than fourth-year residents (t = 3.062; p = 0.0074), with 1 year of training accounting for 37% of the variance in scores. Correlations between the PAME and each of the other measures were ITER, 0.24; ITER-CK, 0.38; OE, -0.13; and CAGS, 0.061, with the PAME demonstrating better reliability and stronger evidence of validity than any other. CONCLUSIONS The PAME had better psychometric properties than other measures and assessed areas often not evaluated. This type of evaluation may be useful for feedback, remediation, or certification decisions.
Collapse
Affiliation(s)
- H M MacRae
- Department of Surgery, University of Toronto, Ontario, Canada
| | | | | | | | | |
Collapse
|
45
|
Amiel GE, Tann M, Krausz MM, Bitterman A, Cohen R. Increasing examiner involvement in an objective structured clinical examination by integrating a structured oral examination. Am J Surg 1997; 173:546-9. [PMID: 9207172 DOI: 10.1016/s0002-9610(97)00010-x] [Citation(s) in RCA: 12] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
Abstract
INTRODUCTION The role of physician examiners in an objective structured clinical examination (OSCE) is relatively passive. In our institution examiners criticized the passive nature of their role. This study evaluates the reliability and viability of adding a structured oral examination to an OSCE. METHOD Ten 24-minute stations consisted of three parts. Part I: 12 minutes-patient encounter. Part II: 6 minutes-oral presentation covering findings, differential diagnosis, and management plan. Part III: 6 minutes-structural oral examination (SOE), containing 5 predetermined questions. RESULTS Over 6 consecutive days, 72 graduates were assessed. Overall average score: 72.02 (SD 5.05); reliability 0.84. Part I of the OSCE average score: 69.2 (SD 7.4); reliability 0.69. Part II oral presentation average score 64 (SD 5.8) reliability 0.87. SOE average score 77.7 (SD 6.3); reliability 0.64. Eighty-nine percent of the examiners indicated satisfaction with the new format. CONCLUSIONS The SOE was a reliable component of an OSCE and contributed to the overall reliability. Examiners reported a higher degree of satisfaction with the examination.
Collapse
Affiliation(s)
- G E Amiel
- Department of Cardiothoracic Surgery, Carmel Medical Center, Haifa, Israel
| | | | | | | | | |
Collapse
|
46
|
Jansen JJ, Scherpbier AJ, Metz JC, Grol RP, van der Vleuten CP, Rethans JJ. Performance-based assessment in continuing medical education for general practitioners: construct validity. MEDICAL EDUCATION 1996; 30:339-344. [PMID: 8949472 DOI: 10.1111/j.1365-2923.1996.tb00844.x] [Citation(s) in RCA: 11] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/22/2023]
Abstract
The use of performance-based assessment has been extended to postgraduate education and practising doctors, despite criticism of validity. While differences in expertise at this level are easily reflected in scores on a written test, these differences are relatively small on performance-based tests. However, scores on written tests and performance-based tests of clinical competence generally show moderate correlations. A study was designed to evaluate construct validity of a performance-based test for technical clinical skills in continuing medical education for general practitioners, and to explore the correlation between performance and knowledge of specific skills. A 1-day skills training was given to 71 general practitioners, covering four different technical clinical skills. The effect of the training on performance was measured with a performance-based test using a randomized controlled trial design, while the effect on knowledge was measured with a written test administered 1 month before and directly after the training. A training effect could be shown by the performance-based test for all four clinical skills. The written test also demonstrated a training effect for all but one skill. However, correlations between scores on the written test and on the performance-based test were low for all skills. It is concluded that construct validity of a performance-based test for technical clinical skills of general practitioners was demonstrated, while the knowledge test score was shown to be a poor predictor of competence for specific technical skills.
Collapse
Affiliation(s)
- J J Jansen
- Centre for Research on Quality in Health Care, Universities of Nijmegen and Limburg, Masstricht, Netherlands
| | | | | | | | | | | |
Collapse
|
47
|
Pitts J, Coles C. The development of a 'Standardised Learner' in researching teaching behaviours. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 1996; 1:119-123. [PMID: 24179001 DOI: 10.1007/bf00159276] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/02/2023]
Abstract
Standardised patients are increasingly used in medical education and the reliability and validity of their use as an assessment method are supported by published research. This paper details the application of this methodology to the educational interaction between a teacher and a learner using a standardised learner.In contrast with standardised patients, a standardised learner must use a level of medical knowledge and experience appropriate for the role assumed. Some aspects of reliability and validity have been considered and appear supportive of this methodology.The feasibility and value of this method for researching actual teacher behaviour are discussed.
Collapse
Affiliation(s)
- J Pitts
- Institute of Health and Community Studies, Bournemouth University, Royal London House, Christchurch Road, BH1 3LT, Bournemouth, U.K
| | | |
Collapse
|
48
|
Affiliation(s)
- L H Pololi
- Office of Academic Affairs, East Carolina University School of Medicine, Greenville, North Carolina 27858-4354, USA
| |
Collapse
|
49
|
Abstract
BACKGROUND Standardized patient (SP) exercises are expensive and time consuming. We hypothesized that self-evaluation would further the goals of teaching and evaluation for a group of medical students. METHODS Students were given self-evaluation checklists (essentially identical to those filled out by the SPs) and completed them prior to reviewing the SP checklists. Agreement between student and SP checklists (whether each item was checked or not) and the extent of agreement on the interactional skills rating scales were assessed. RESULTS Overall agreement was 93%, with 98% agreement on interactional items, 92% on physical examination items, and 89% on history items. Disagreements tended to be clustered on a few items in each scale. Scores on the interactional skills rating scales also showed strong agreement. CONCLUSION The use of checklists and rating scales in this context is well established. At virtually no cost, a self-evaluation tool adds a valuable dimension to the exercise.
Collapse
Affiliation(s)
- S Kaiser
- Department of Surgery, Mount Sinai School of Medicine, New York, New York 10029-6574
| | | |
Collapse
|
50
|
Murphy SA, Scott CS, Mandel LS. Evaluating students' substance abuse assessment skills. JOURNAL OF SUBSTANCE ABUSE 1995; 7:357-64. [PMID: 8749794 DOI: 10.1016/0899-3289(95)90028-4] [Citation(s) in RCA: 13] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/02/2023]
Abstract
Graduating nursing students' abilities to assess clients for substance use were evaluated using two standardized patient (SP) cases. In the comprehensive health history case (unprompted), 55% of the study participants assessed alcohol use and 20% assessed drug use. When prompted to conduct a substance use history, at least 90% of the subjects asked about the quantity and frequency of alcohol use and 80% asked about drug use. However, in the prompted case, fewer than 10% of the subjects linked current alcohol and drug (AOD) use with risks and consequences, nor did they ask the SP if he had considered decreasing substance use. The mean score for communication skills used in the two assessment interviews were 3.65 and 3.56 on a 1 to 6 Likert scale. These data point out the need for additional emphasis on instructing students to include AOD questions as part of routine assessments, what questions to ask in a substance use assessment, and how to act on the information received. The use of SPs is one way to obtain valid information about students' assessment competencies.
Collapse
Affiliation(s)
- S A Murphy
- Department of Psychosocial and Community Health, University of Washington, Seattle 98195-7263, USA
| | | | | |
Collapse
|