Technical issues: test application. AAMC

Learning transfusion medicine through scoring objective structured clinical examination

BACKGROUND

Transfusion medicine education at the undergraduate level is typically limited in duration. In view of limitations of traditional teaching methods, we explore effectiveness of scoring (Objective Structured Clinical Examination) OSCE as an educational method.

MATERIALS AND METHODS

The study was of a randomized interventional three group pre-test-post-test design. Participants were undergraduate medical students in their two final years. The intervention was watching and scoring 2 videotaped OSCE stations about obtaining consent for blood transfusions and assessing the ability to explain risks, benefits, and alternatives of blood transfusion. Participants were asked to assess the performance of the videotaped actor using checklists. Participants were randomized to watch and evaluate one set of videos at either the highest, intermediate, or lowest compliance with required consent elements. Main measure was performance in a knowledge test containing multiple-choice and true/false questions. This was given before (pre-test), immediately after the intervention (post-test 1), and after 8 weeks (post-test 2). Student perceptions regarding the intervention was assessed immediately after the session.

RESULTS

Sixty-nine students were randomized. Post-test 1 results (mean 16.52, SD 1.88) were significantly greater than pre-test results (mean 11.83, SD 2.13) by group and across all groups (p < 0.001). Post-test 2 results for the complete cohort showed maintenance of significant improvement in comparison with the pre-test. The majority of students agreed that learning through scoring OSCE was an effective educational experience.

CONCLUSIONS

In the undergraduate medical setting, scoring OSCE stations may enhance learning of content discussed and evaluated in the stations.

Performance-Based Assessment: Using Pre-Established Criteria and Continuous Feedback to Enhance a Student’s Ability to Perform Practice Tasks

Performance-based assessment is a process in which pre-established performance criteria and continuous feedback are used by both the student and instructor to improve learning with an endpoint of students demonstrating their achievement of learning by performing realistic practice tasks. This manuscript describes the issues limiting widespread use of performance-based assessment, based on findings from the pharmacy, medical, and general education literature, and then proposes a model for how to successfully implement it. Performance-based assessment methods such as direct observation, the Mini-Clinical Evaluation Exercise, simulations involving standardized patients and computers, projects, presentations, learning portfolios, and the Triple Jump Exam are discussed. Successful implementation of performance-based assessment involves: 1) inculcating a culture for assessment, 2) establishing pre-specified performance criteria, 3) implementing use of the pre-established performance criteria and formative assessment methods to provide continuous feedback during the learning period, 4) using summative performance-based assessment methods, and 5) providing mechanisms for student remediation. Successful inclusion of performance-based assessment in Doctor of Pharmacy curricula should enable instructors to better prepare graduates that can provide pharmaceutical care in a quality manner.

Simulated learning in musculoskeletal assessment and rehabilitation education: comparing the effect of a simulation-based learning activity with a peer-based learning activity

BACKGROUND

Musculoskeletal disorders and diseases are leading causes of pain, physical disability, and doctor visits throughout the world. Health professionals must be trained to assess, treat through rehabilitation and monitor patients with these disorders. Yet, due to overcrowded curricula, some health education programs struggle to accommodate more than minimal training in musculoskeletal conditions. Consequently, educators in these professions must consider how traditional instruction could be complemented effectively to enhance students' preparation for the diverse musculoskeletal disorders and pathologies they may encounter. The purpose of this study was to explore the benefits that can be obtained from laboratory practice in musculoskeletal conditions with a standardised patient, rather than a peer patient, in a condensed time frame.

METHODS

Two groups of students were assigned to either a standardised or a peer patient condition for 2 × 2 hours musculoskeletal assessment and rehabilitation lab sessions. All students completed a pre-post matched questionnaire measuring their clinical knowledge, confidence in clinical skills and motivation for further learning. Their clinical skills were tested at the end. Students and standardised patients' perceptions of the simulated learning environment to practise musculoskeletal assessment and rehabilitation were also elicited.

RESULTS

A t-test for independent samples revealed that students working with standardised patients displayed significantly higher standards of practical clinical skills than those working with peer patients (p=0.018). Using MANOVAs with repeated measures, no interaction effect for clinical knowledge, confidence in clinical skills, and motivation for future learning were found, both groups displaying significantly enhanced cognition and motivation. Three positive and two negative themes emerged from the analysis of students' perceptions of the simulated learning environments. These were consistent with the simulated patients' perceptions.

CONCLUSIONS

The findings of this study provide support for the value of using standardised patients to enhance clinical skills in musculoskeletal assessment and rehabilitation when the timeframe for laboratory practice is limited. Students' perceptions of their experience contributed to explain why confidence in clinical skills might not necessarily improve when practising with standardised patients. Suggestions are made for optimising learning with standardised patients and for addressing the economic challenge on health education programs of hiring standardised patients.

Patient knowledge and use of acetaminophen in over-the-counter medications

OBJECTIVES To evaluate patient knowledge of over-the-counter (OTC) products containing acetaminophen and to determine patients' accuracy in dosing adult, child, and infant formulations. DESIGN Cross-sectional study. SETTING Six community pharmacies in Tucson, AZ, between February and May 2011. PARTICIPANTS 88 adults aged 19 to 89 years. INTERVENTION Investigator-administered, semistructured interviews. MAIN OUTCOME MEASURES Patient knowledge of and ability to safely use OTC products containing acetaminophen, including understanding risks, identifying products, and dosing different formulations. RESULTS Although most (86%) participants heard of acetaminophen, only 68% understood at least one of its uses and only 9% knew the abbreviation APAP. Virtually all knew that consuming too much acetaminophen in 1 day could be harmful, but only 17% and 35% knew that overdoses could result in death or liver damage, respectively. On average, participants correctly identified 80% (range 27-100%) of products with and without acetaminophen from a lineup of 11 OTC products. Although 38% (n = 84) of participants correctly measured both the child and infant doses of acetaminophen, doses ranged from one-half to twice the amount of the labeled child dose and one-third of the labeled infant dose. Findings from the regression analysis suggested that on average, women and those with college degrees had higher overall scores, while participants' age or parent status were nonsignificant predictors. CONCLUSION Many patients remain confused about using acetaminophen safely, signaling the need for greater patient education to prevent unintentional harm. The results further specify common misunderstandings to address during patient contact, which also includes replacing "APAP" with "acetaminophen" on any prescription bottle labels or patient-directed information.

Pharmacy students' opinions of direct-to-consumer advertising: a pilot study at one university

BACKGROUND

Direct-to-consumer advertisement (DTCA) of prescription medications has become an important informational source for health care consumers. As future health care professionals on the front line of potential communication and dispensing of products emerging from DTCA, it is important to elicit the attitudes of student-pharmacists.

OBJECTIVES

This study aims to (1) evaluate the validity of the DTCA attitudinal questionnaire using Rasch rating scale analysis and (2) investigate the attitudes of pharmacy students toward DTCA and determine whether these attitudes were associated with years of pharmacy education and demographic characteristics.

METHODS

This investigation used a cross-sectional print-based questionnaire to evaluate the attitudes of pharmacy students toward DTCA of prescription medications. The 16-item questionnaire included items addressing the attitudes of pharmacy students toward DTCA with respect to patients' knowledge of medications, pharmacists' interaction with patients, and overall consumer judgment of medical prescriptions. Analyses included Rasch analysis and a multiple linear regression.

RESULTS

A total of 243 students submitted usable questionnaires (85% response rate). Item response categories were collapsed from 5 categories to 3, and 4 items were removed to achieve acceptable Rasch model fit. Pharmacy students demonstrated little difficulty in agreeing with the statements suggesting that DTCA helps patients take a more active role in health care and had the most difficulty in agreeing with items suggesting that DTCA may lead to inappropriate prescribing to satisfy patient requests. Students' overall support for DTCA was the only variable that predicted the questionnaire score (P<.001).

CONCLUSIONS

In conclusion, the Rasch analysis evaluated the psychometric properties of the instrument and identified the necessity to adapt the questionnaire from previous iterations to adequately fit the student population. Future research should examine factors that contribute to the variance in attitudes toward DTCA among a larger and more heterogeneous population.

Relationship between standardized patient checklist item accuracy and performing arts experience

INTRODUCTION

: It is not known whether a Standardized Patient's (SP's) performing arts background could affect his or her accuracy in recording candidate performance on a high-stakes clinical skills examination, such as the Comprehensive Osteopathic Medical Licensing Examination Level 2 Performance Evaluation. The purpose of this study is to investigate the differences in recording accuracy of history and physical checklist items between SPs who identify themselves as performing artists and SPs with no performance arts experience.

METHODS

: Forty SPs identified themselves as being performing artists or nonperforming artists. A sample of SP live examination ratings were compared with a second set of ratings obtained after video review (N = 1972 SP encounters) over 40 cases from the 2008-2009 testing cycle. Differences in SP checklist recording accuracy were tested as a function of performing arts experience.

RESULTS

: Mean overall agreement rates, both uncorrected and corrected for chance agreement, were very high (0.94 and 0.79, respectively, at the overall examination level). There was no statistically significant difference between the two groups with respect to any of the mean accuracy measures: history taking (z = -0.422, P = 0.678), physical examination (z = -1.453, P = 0.072), and overall data gathering (z = -0.812, P = 0.417) checklist items.

CONCLUSION

: Results suggest that SPs with or without a performing arts background complete history taking and physical examination checklist items with high levels of precision. Therefore, SPs with and without performing arts experience can be recruited for high-stakes SP-based clinical skills examinations without sacrificing examination integrity or scoring accuracy.

A drug-drug interaction knowledge assessment instrument for health professional students: a Rasch analysis of validity evidence

BACKGROUND

It is essential that current and future health professionals be able to evaluate for possible clinically significant drug-drug interactions (DDIs) and when detected, determine appropriate management strategies to prevent patient harm.

OBJECTIVE

Assess the validity of a DDI knowledge assessment instrument in a health professional student population.

METHODS

This study recruited health professional students (medical, nurse practitioner, and pharmacy) beginning experiential training at the University of Arizona. Students were given a knowledge assessment instrument that included 15 medication pairings selected on the basis of clinical importance and were asked to select the most appropriate DDI management strategy for each pair by selecting "avoid combination," "usually avoid combination," "take precautions," or "no special precautions." Data were analyzed in 2 ways because of the subjective nature of classifying DDIs into specific management categories. In the first analysis, respondents were given credit for a correct item only if they selected the management strategy deemed appropriate (management strategy analysis). In another analysis, students were given credit for an item only if they correctly identified specific DDIs (DDI recognition analysis). Rasch analysis was used to assess the validity of the knowledge instrument.

RESULTS

A total of 165 of the 226 eligible health professional students completed the DDI knowledge assessment (73% response rate). The mean score for management strategy analysis was 3.82 out of 15, whereas DDI recognition analysis produced a higher average (mean=6.55). Good reliability was demonstrated in both strategies, and no ceiling or floor effects were observed. Some construct underrepresentation occurred with both scoring strategies, and some mistargeting was identified when analyzing the management strategy.

CONCLUSIONS

Although improvements in construct representation may be beneficial, the instrument used demonstrated good reliability and validity and could be used by educators to assess and improve DDI knowledge. The ability of the participants to identify DDIs and select an appropriate management strategy was low. These results support the need for additional DDI education in this institution's health curricula.

Breast cancer risk communication: assessment of primary care physicians by standardized patients

PURPOSE

To assess primary care providers' communication about breast cancer risk.

METHODS

We evaluated 86 primary care providers' communication of risk using unannounced standardized (simulated) patients. Physicians were randomly assigned to receive one of three cases: (1) moderate risk case (n = 25), presenting with a breast lump and mother with postmenopausal breast cancer; (2) high-risk (maternal side) case (n = 28), presenting with concern about breast cancer risk; and (3) high-risk (paternal side) case (n = 33), presenting with an unrelated problem. After the appointment, three qualitative parameters were assessed by standardized patients on a 3-point scale (3 = highest satisfaction, 1 = lowest): whether the physician took adequate time; acknowledged her concerns; and offered reassurance.

RESULTS

Mean satisfaction with physician communication was higher for the moderate risk case (2.92) than for the high-risk paternal case (2.25) or high-risk maternal case (2.42) (P < 0.0001). The score was not influenced by session length, medical specialty, or physician gender.

CONCLUSION

Physicians more consistently provided a moderate risk standardized patients with reassurance and support compared with the high-risk cases. Primary care physicians may be more unprepared or uneasy addressing the issues raised by more complex scenarios and may benefit from training in the assessment and communication of breast cancer risk.

The role of standardized patient and trainer training in quality assurance for a high-stakes clinical skills examination

For over 30 years, medical educators have used standardized patients (SPs), laypersons trained to portray a patient case in a realistic manner, to teach and to assess clinical skills. All medical schools in the US have SP programs in place, and the US and Canada require national examinations using SPs to assess the competency of those wishing to obtain licensure to practice medicine in these countries. To ensure a valid and reliable examination, unwanted variance that can be introduced by SP performance must be addressed. The goal of SP training is to imbue the SP with the characteristics, mannerisms and history of a real patient so that the portrayal is consistent and accurate. The challenge is to ensure consistent portrayal of each case with sufficient realism to elicit the expected clinical performance and to ensure standardized SP performance across multiple examinees. This paper considers the quality assurance methods applied to training the SP trainers and the protocols used to train the SPs, to ensure that the SP performances are sufficiently accurate and standardized, and that the evaluators completing the checklists and scales used for scoring do so correctly and consistently.

Standardized patients for HIV/AIDS training in resource-poor settings: the expert patient-trainer

This article presents a unique approach to HIV/AIDS training in resource-poor settings that incorporates the use of standardized patients (SPs). Integrated Management of Adolescent and Adult Illness (IMAI) is a World Health Organization health systems strengthening initiative with a strong emphasis on training health workers in the management of common diseases and conditions. In IMAI, SPs are called Expert Patient-Trainers (EPTs) to emphasize their role in the training of health workers. EPTs were first used in IMAI training in Uganda in 2004. Since then, the method has been adopted by a number of other countries in Africa, Latin America, and Asia. EPTs are usually recruited from groups of people living with HIV/AIDS. In the classroom, EPTs discuss living with HIV and help participants understand HIV as it affects patients. Course participants spend approximately two hours per day in "skill stations," multiple-station assessments consisting of one-on-one encounters with EPTs. In each encounter, the health worker interacts with an EPT portraying a standardized case. Instructions on how to portray each case provide only broad outlines of the major clinical and counseling points; the EPT is expected to use his or her own life experiences to fill in emotional details. Course facilitators noted that health workers were often initially skeptical about EPTs, but this generally turned to enthusiasm after participating in the skill stations. EPTs benefited from the sense of being part of the training team, the satisfaction of improving the skills of health workers, and learning more about their illness.

Teaching patient selection in aesthetic surgery: use of the standardized patient

This study evaluates the effectiveness of standardized patients for teaching patient selection in esthetic surgery. Six actors received detailed character descriptions. Each was interviewed by a resident for 30 minutes in a conference setting. Participants completed a questionnaire, and a discussion was held. Written simulations were administered before and after the sessions, from which a faculty standard was developed using hierarchical cluster analysis. Resident responses were compared with the standard using a squared euclidean metric. Residents' pre- and posttest scores were compared with the faculty standard using 2-way analysis of variance. Accuracy scores were found to be significantly lower (more accurate) after the training than before (P < 0.001). Upon a 6-item questionnaire, both faculty and residents agreed that it was a worthwhile exercise (faculty mean, 6.2 out of 7; resident, 6.3) and that the standardized patients were believable. Standardized patients can provide effective instruction in traditionally difficult-to-teach areas such as communication and patient selection.

Simpler standards for local performance examinations: the Yes/No Angoff and whole-test Ebel

BACKGROUND

The whole-test Ebel and Yes/No Angoff methods offer the possibility of faster and easier standard-setting exercises for local, medium-stakes performance exams.

PURPOSE

We sought to establish if these less demanding methods might be substituted for the traditional but complex case-level Ebel.

METHODS

Six faculty judges each used all three methods to set standards for six standardized-patient checklists used for 178 fourth-year medical students. We compared the cut scores, passing rates, reliability, ease of use, and credibility of the three methods.

RESULTS

The three methods produced roughly equivalent cut scores. Generalizability of judgments was .94 and .96 for the case-level and whole-test Ebel, and .76 for the Yes/No Angoff. Judges found the simplified methods more credible than the case-level Ebel. The Yes/No Angoff method was preferred by five of six judges.

CONCLUSIONS

The whole-test Ebel and the Yes/No Angoff may be simple and realistic options for setting standards for local performance exams.

Quality of standardised patient research reports in the medical education literature: review and recommendations

CONTEXT

In order to assess or replicate the research findings of published reports, authors must provide adequate and transparent descriptions of their methods. We conducted 2 consecutive studies, the first to define reporting standards relating to the use of standardised patients (SPs) in research, and the second to evaluate the current literature according to these standards.

METHODS

Standards for reporting SPs in research were established by representatives of the Grants and Research Committee of the Association of Standardized Patient Educators (ASPE). An extensive literature search yielded 177 relevant English-language articles published between 1993 and 2005. Search terms included: 'standardised patient(s)'; 'simulated patient(s)'; 'objective structured clinical examination (OSCE)', and 'clinical skills assessment'. Articles were limited to those reporting the use of SPs as an outcome measure and published in 1 of 5 prominent health sciences education journals. Data regarding the SP encounter, SP characteristics, training and behavioural measure(s) were gathered.

RESULTS

A random selection of 121 articles was evaluated according to 29 standards. Reviewers judged that few authors provided sufficient details regarding the encounter (21%, n = 25), SPs (16%, n = 19), training (15%, n = 15), and behavioural measures (38%, n = 44). Authors rarely reported SP gender (27%, n = 33) and age range (22%, n = 26), whether training was provided for the SPs (39%, n = 47) or other raters (24%, n = 29), and psychometric evidence to support the behavioural measure (23%, n = 25).

CONCLUSIONS

The findings suggest that there is a need for increased rigor in reporting research involving SPs. In order to support the validity of research findings, journal editors, reviewers and authors are encouraged to provide adequate detail when describing SP methodology.

Use of simulated clients in marriage and family therapy education

Knowledge of how one should manage suicidal, homicidal, child maltreatment, and domestic violence situations is paramount in the training of marriage and family therapists (MFTs). Simulated patient modules were created to help clinical faculty address these crisis situations in a protected learning environment. The modules were implemented by the MFT faculty in collaboration with the Office of Clinical Skills Assessment and Education at East Carolina University's Brody School of Medicine. Qualitative data over the course of 2 years revealed six thematic domains regarding therapists' performance, therapists' emotions, the simulation experiences, and lessons learned. Educational, clinical, and research recommendations include tools to implement simulation exercises into marriage and family therapy programs as well as suggestions to assess for teaching effectiveness.

The Teaching Effectiveness of Standardized Patients

Teaching nursing students therapeutic communication skills begins in the classroom and extends to the clinical environment. The usual method of instruction consists of random patient encounters observed by faculty and measures of competence that rely on paper-and-pencil tests. Using standardized patients (SPs) offers an alternative approach to the traditional method of teaching. Standardized patients are individuals who have been carefully trained to present an illness or scenario in a standardized, unvarying manner. This pilot study compared use of SPs with the usual method of instruction in a class of undergraduate nursing students. Results indicated that students who participated in the SP method overwhelmingly described the experience as positive, creative, and meaningful. No significant differences were found between the two groups on measures of interpersonal skills, therapeutic communication skills, and knowledge of depression

Using a Comprehensive Examination to Assess Multiple Competencies in Surgical Residents: Does the Oral Examination Still Have a Role?

BACKGROUND

While specialty-level evaluations evolve from traditional examinations to objective structured clinical examination-like assessments, a broader range of competencies are tested; consequently, examiners are forced to integrate results when making a determination of competency. The aim of this study was to describe how experts weigh relative performances on specific components of a comprehensive examination to make decisions of overall competency.

STUDY DESIGN

The Patient Assessment and Management Examination is a standardized patient-based assessment of general surgery in which each 25-minute station encompasses four components: history and physical examination, investigation interpretation, diagnosis and treatment discussion with the patient, and a structured oral examination (SOE). A six-station Patient Assessment and Management Examination was administered to 21 senior surgery residents. Surgeons marked each station with global rating scales and, in addition, provided an end-of-station overall global assessment of performance. A "gold-standard" examination pass-or-fail decision was determined through videotape review of each candidate's performance across six stations by two blinded surgeons. Multiple linear regression analysis was used to determine which components were associated with the end-of-station overall global assessments. Multivariable logistic regression was used to determine which components were associated with the final "gold-standard" pass-or-fail assessment.

RESULTS

The only component notably (p < 0.005) associated with end-of-station global assessment for all six stations was the SOE. Mean SOE score was the only notable independent variable associated with the gold-standard pass-or-fail decision (R(2) = 0.63, p < 0.001).

CONCLUSIONS

Performance on the SOE section of a multicompetency examination is markedly associated with the final determination of competency. These results have implications for the design and implementation of comprehensive specialty-level assessments.

Item analysis to improve reliability for an internal medicine undergraduate OSCE

Utilization of objective structured clinical examinations (OSCEs) for final assessment of medical students in Internal Medicine requires a representative sample of OSCE stations. The reliability and generalizability of OSCE scores provides validity evidence for OSCE scores and supports its contribution to the final clinical grade of medical students. The objective of this study was to perform item analysis using OSCE stations as the unit of analysis and evaluate the extent to which OSCE score reliability can be improved using item analysis data. OSCE scores from eight cohorts of fourth-year medical students (n = 435) in a 6-year undergraduate program were analyzed. Generalizability (G) coefficients of OSCE scores were computed for each cohort. Item analysis was performed by considering each OSCE station as an item and computing the corrected item-total correlation. OSCE stations which negatively impacted the reliability were deleted and the G-coefficient was recalculated. The G-coefficients of OSCE scores from the eight cohorts ranged from 0.48 to 0.80 (median 0.62). The median number of OSCE stations that negatively impacted the G-coefficient was 3.5 (out of a median of 25 total stations). When the ''problem stations'' were deleted, the median G-coefficient across eight cohorts increased to 0.62--0.72. In conclusion, item analysis of OSCE stations is useful and should be performed to improve the reliability of total OSCE scores. Problem stations can then be identified and improved.

Using standardised patients in an objective structured clinical examination as a patient safety tool

Standardised patients (SPs) are a powerful form of simulation that has now become commonplace in training and assessment in medical education throughout the world. Standardised patients are individuals, with or without actual disease, who have been trained to portray a medical case in a consistent manner. They are now the gold standard for measuring the competence of physicians and other health professionals, and the quality of their practice. A common way in which SPs are used in performance assessment has been as part of an objective structured clinical examination (OSCE). The use of an SP based OSCE can be a powerful tool in measuring continued competence in human reliability and skill performance where such skills are a critical attribute to maintaining patient safety. This article will describe how an OSCE could be used as a patient safety tool based on cases derived from actual events related to postdonation information in the blood collection process. The OSCE was developed as a competency examination for health history takers. Postdonation information events in the blood collection process account for the majority of errors reported to the US Food and Drug Administration. SP based assessment is an important patient safety tool that could be applied to a variety of patient safety settings and situations, and should be considered an important weapon in the war on medical error and patient harm.

Using standardised patients in an objective structured clinical examination as a patient safety tool

Standardised patients (SPs) are a powerful form of simulation that has now become commonplace in training and assessment in medical education throughout the world. Standardised patients are individuals, with or without actual disease, who have been trained to portray a medical case in a consistent manner. They are now the gold standard for measuring the competence of physicians and other health professionals, and the quality of their practice. A common way in which SPs are used in performance assessment has been as part of an objective structured clinical examination (OSCE). The use of an SP based OSCE can be a powerful tool in measuring continued competence in human reliability and skill performance where such skills are a critical attribute to maintaining patient safety. This article will describe how an OSCE could be used as a patient safety tool based on cases derived from actual events related to postdonation information in the blood collection process. The OSCE was developed as a competency examination for health history takers. Postdonation information events in the blood collection process account for the majority of errors reported to the US Food and Drug Administration. SP based assessment is an important patient safety tool that could be applied to a variety of patient safety settings and situations, and should be considered an important weapon in the war on medical error and patient harm.

[Evaluation of clinical competence of family medicine trainer]

OBJECTIVES

We show the first experience of the application of an objective and structured clinical evaluation (OSCE) procedure to family medicine trainers, that has been carried out in Andalucia. The objective is to use a competence evaluation instrument that, in the short term, will be used not only for trainer accreditation but also for other public sanitary professionals.

PARTICIPANTS

Tutors of family and community medicine residents.

DESIGN

Observational descriptive.

SETTING

Educational unity of family medicine.

PRINCIPAL MEASUREMENTS

The competencial components to be assessed are the following: anamnesis, physical exploration, communication, technical skill, management, family attention y preventive activities. The clinical situations were selected using the following priority criteria: prevalence, clinical gravity, prevention and early diagnosis importance, case complexity, doctor's capacity of evaluation and simplicity.

RESULTS

Thirteen family medicine trainers took part in the OSCE. Their average age was 42.8 +/- 3.6 years. The test had an overall reliability coefficient (Cronbach's alpha) of 0.73. The overall mean score of the participants was 73 +/- 6.2. The best results about the competencial components were family attention, communication and technical skill.

CONCLUSIONS

The OSCE can be a convenient tool for family medical trainer evaluation, helping to orientate their education in the weak points and, in the near future, it can also be used as an instrument do accredit family medicine trainers.

Taking standardized patient-based examinations to the next level

PURPOSE

The purpose of this article is to summarize this author's view on "where we are" with standardized patient-based assessments of clinical performance and to offer three directions for further research and development.

SUMMARY

The push for more objective outcome data has fueled proliferation of the most researched innovation in the history of medical education. Near-random clinical experiences of students do not provide consistent, repeated practice with important clinical cases to achieve minimally adequate performance on these objective performance examinations, leading to scoring "psychogymnastics" to titrate fail rates. The second area is to modify these examinations to reflect features at higher levels of professional development such as situational awareness. Theories of professional development should guide changes. The third area incorporates multiperson scenarios; a clinician with a family or a team in the operating room. Simulation of complex situations, especially those requiring rapid, accurate communication and action can reduce medical errors and improve patient safety.

CONCLUSIONS

Standardized patient-based examinations provide objective outcome data but require artificial adjustments in scoring due to inconsistent learning opportunities. Theoretical research on professional development, acquisition of expertise and team functioning provides fertile, new directions to take standardized patient-based examinations to the next level.

An objective structured clinical examination to evaluate health historian competencies

BACKGROUND

Post-donation information events in the blood-collection process account for the majority of errors reported to the FDA. An eight-station objective structured clinical examination (OSCE) based on information reported after donation was developed as a competency examination for health historians.

STUDY DESIGN AND METHODS

The OSCE measured two individual skill components: history-taking technique (HXE) and ability to record and interpret a medical history (HXI). Donor historians at the Hoxworth Blood Center, University of Cincinnati Medical Center, were examined (n = 56).

RESULTS

In general, staff performance was acceptable, but several areas for improvement were identified. Of particular concern were the deferrals associated with the malaria scenarios. The overall reliability score was greater than 0.8 for the total OSCE and HXI. Using the cut score of 0.9, or 90 percent, 10 subjects would not have passed the exam based on the HXE score and one person would not have demonstrated competency based on the HXI score.

CONCLUSION

An OSCE is a reliable, valid, and practical method for assessing continued competency in health historians. This form of competency assessment and subsequent retraining may reduce the incidence of errors in information reported after donation and should be further studied as a suitable national standard for assessing competency.

Cognitive, social and environmental sources of bias in clinical performance ratings

BACKGROUND

Global ratings based on observing convenience samples of clinical performance form the primary basis for appraising the clinical competence of medical students, residents, and practicing physicians. This review explores cognitive, social, and environmental factors that contribute unwanted sources of score variation (bias) to clinical performance evaluations.

SUMMARY

Raters have a 1 or 2-dimensional conception of clinical performance and do not recall details. Good news is reported more quickly and fully than bad news, leading to overly generous performance evaluations. Training has little impact on accuracy and reproducibility of clinical performance ratings.

CONCLUSIONS

Clinical performance evaluation systems should assure broad, systematic sampling of clinical situations; keep rating instruments short; encourage immediate feedback for teaching and learning purposes; encourage maintenance of written performance notes to support delayed clinical performance ratings; give raters feedback about their ratings; supplement formal with unobtrusive observation; make promotion decisions via group review; supplement traditional observation with other clinical skills measures (e.g., Objective Structured Clinical Examination); encourage rating of specific performances rather than global ratings; and establish the meaning of ratings in the manner used to set normal limits for clinical diagnostic investigations.

An observational study comparing quality of care in walk-in centres with general practice and NHS Direct using standardised patients

OBJECTIVES

To compare the quality of clinical care in walk-in centres with that provided in general practice and by NHS Direct.

DESIGN

Observational study involving assessment of clinicians by standardised patients.

SETTING

20 walk-in centres, 20 general practices, and 11 NHS Direct sites.

PARTICIPANTS

297 consultations with standardised patients, 99 in each setting, carried out by professional role players trained to play five clinical scenarios (postcoital contraception, chest pain, sinusitis, headache, and asthma).

MAIN OUTCOME MEASURES

Primary outcomes were mean scores on consensus derived checklists of essential items for the management of the clinical scenarios. Data were also collected on access to and referral by walk-in centres, general practices, and NHS Direct.

RESULTS

Walk-in centres achieved a significantly greater mean score for all scenarios combined than general practices (difference between groups 8.2, 95% confidence interval 1.7 to 14.6) and NHS Direct (10.8, 5.5 to 16.1). There was considerable between scenario variation, with walk-in centres performing particularly well on postcoital contraception and asthma scenarios. In contrast to general practices, walk-in centres and NHS Direct referred a higher proportion of patients (26% and 82%, respectively).

CONCLUSION

Walk-in centres perform adequately and safely compared with general practices and NHS Direct for the range of conditions under study, but the impact of referrals on workload of other healthcare providers requires further research.

[First experiences in evaluating the clinical competence of family doctors in Catalonia]

OBJECTIVES

To construct a valid instrument to evaluate the clinical competence of family doctors and establish professional standards.

DESIGN

Construction of an objective and structured clinical evaluation test by the Test Committee of the Catalan Society of Family and Community Medicine (SCMFiC) with 27 representative clinical situations of primary care practice. This used various evaluative instruments and lasted 5 hours and 30 minutes.

SETTING

2 Barcelona health districts for the two runnings of the test.

PARTICIPANTS

Family doctors, SCMFiC members, who took part voluntarily after having been invited by letter.

MEASUREMENTS AND MAIN RESULTS

23 doctors took part in the first running of the test. Their average age was 36.6 (SD 6.9) and 52.2% of them were women. 46 doctors took part in the second test. Their average age was 30.9 (SD 2.1) and 76.1 were women. The test had an overall reliability coefficient (Cronbach's alpha) of 0.83 the first time and 0.65 the second time. The overall mean score of the participants the first time was 56.9 (SD 19.2), with 100 the top score possible, and 62.1 (SD 4.1) the second time.

CONCLUSIONS

This SCMFiC experiment is the first time the clinical competence of family doctors has been assessed in our area. It was a reliable and valid test to assess the competence of our health professionals.

Examiner fatigue in communication skills objective structured clinical examinations

CONTEXT

The assessment of undergraduates' communication skills by means of objective structured clinical examinations (OSCEs) is a demanding task for examiners. Tiredness over the course of an examining session may introduce systematic error. In addition, unsystematic error may also be present which changes over the duration of the OSCE session.

AIM

To determine the strength of some sources of systematic and unsystematic error in the assessment of communication skills over the duration of an examination schedule.

METHODS

Undergraduate first-year medical students completing their initial summative assessment of communication skills (a four-station OSCE) comprised the study population. Students from three cohorts were included (1996-98 intake). In all 3 years the OSCE was carried out identically. All stations lasted 5 minutes with a simulated patient. Students were assessed using an examiner (content expert) and a simulated-patient evaluation tool, the Liverpool Communication Skills Assessment Scale (LCSAS) and the Global Simulated-patient Rating Scale (GSPRS), respectively. Each student was assigned a time slot ranging from 1 to 24, where 1, for example, would denote that the student entered the exam first and 24 indicates the final slot for entry into the examination. The number of students who failed this exam was noted for each of the 24 time slots. A control set of marks from a communication skills written exam was also adopted for exploring a possible link with the time slot. Analysis was conducted using graphical display, covariate analysis and logistic regression.

RESULTS

No significant relationship was found between the schedule point that the student entered the OSCE exam and their performance. The reliability of the content expert and simulated-patient assessments was stable throughout the session.

CONCLUSION

No evidence could be found that duration of examining in a communication OSCE influenced examiners and the marks they awarded. Checks of this nature are recommended for routine inspection to confirm a lack of bias.

One program's experience of OSCE vs. written board certification results: a pilot study

The objective structured clinical examination (OSCE) has been the focus of a lot debate with respect to reliability and validity. Much of the controversy surrounding these components lies in a lack of comparison with a "gold standard." Further work is needed to improve the evaluation of clinical skills to the point that a gold standard can truly be said to exist.

Impacting faculty teaching and student performance: nine years' experience with the Objective Structured Clinical Examination

BACKGROUND

The impetus for administering the 2nd-year Objective Structured Clinical Examination (OSCE) came from the great variability in student performance observed by 3rd-year clerkship directors.

PURPOSE

To document the effects of the OSCE on faculty teaching, student performance, and the curriculum over 9 years of administration of the examinations to more than 1,000 second-year medical students.

METHOD

A 20-station OSCE was administered to all medical students at the end of their 2nd year. Using predetermined criteria, clinical faculty served as evaluators in each station. A mix of 1st-, 3rd-, and 4th-year medical students were recruited to serve as simulated patients. Faculty evaluators and examinees completed a questionnaire evaluating their experience with the OSCE. Students received a report card of their performance. Small-group leaders of the Introduction to Clinical Medicine course received feedback on their group's performance on each station compared to the class mean. Summative data on class performance was reported to the curriculum committee. The academic status committee received data on students who performed unsatisfactorily.

RESULTS

Faculty and examinee ratings of the OSCE experience were very positive. Over the 9-year period, student performance improved showing less variability and significantly fewer failed stations.

CONCLUSION

The OSCE has proven to be a technically feasible, authentic evaluation method yielding valuable information for decisions regarding student performance, faculty teaching, and curriculum planning.

Incorporating standardized patients into a psychosocial nurse practitioner program

Interactions with standardized patients (SPs) provide customized, immediate clinical learning for graduate nursing students. SPs are lay persons trained to portray a role in a faculty-designed health care scenario for a series of students. Student learning results from the experience of the encounter, performance feedback from the SP, subsequent classroom discussion with peers, faculty critique, and review of the videotaped interaction. Design and implementation of an SP experience involves clarification of goals for the encounter, development of the clinical scenario, recruitment and training of SPs, preparation of students, and management of the logistics of the exercise. Although the process is labor-intensive, we have found SP experiences versatile, valuable, and popular with students.

Using standardized patients to train and evaluate dietetics students

Faced with a dwindling number of clinical sites for training dietetics students, we incorporated a standardized patient scenario of a 17-year-old pregnant woman into a nutrition counseling course. A young woman was hired to be the patient and was trained extensively. Each student signed up for a 20-minute block of time to assess her and provide nutrition instruction, after which the standardized patient and course instructor provided individualized feedback to each student. Students then completed a brief questionnaire related to the standardized patient session. Overall, student response to the standardized patient session was positive, and students believed standardized patients should be included formally in undergraduate training. The students noted that the standardized patient session allowed them to be trained and evaluated objectively. Because of their ease in building rapport with the standardized patient, students reported that they were able to provide more in-depth education. They also appreciated the immediate and detailed performance review provided by the standardized patient and the instructor. We conclude that standardized patient sessions provide a great service to dietetics students, and plans to develop another standardized patient scenario that students can follow up throughout the continuum of care are underway.

Standardized patients in the early acquisition of clinical skills

OBJECTIVES

Facing dramatic reductions of inpatient beds and fewer faculty tutors, the Queen's University medical school has had to consider alternatives to the traditional inpatient encounter for the early acquisition of clinical skills. The purpose of this study was to investigate the feasibility and impact on students of a model for a first-year clinical skills course, using exclusively standardized patients in place of inpatients, and using a smaller ratio of faculty to students.

DESIGN

Twenty volunteers were selected from the first-year class of 75 students to participate as the experimental group. The remaining 55 students formed the control group.

SETTING

Queen's University medical school, Ontario, Canada.

SUBJECTS

First-year medical students.

RESULTS

In measures of student satisfaction with the amount of performance feedback received during the course, there was a non-significant trend towards greater satisfaction among the experimental group compared to the control group. This occurred despite a tutor to student ratio of 1.5:10 in the experimental group compared to 3:10 in the control group. In the student evaluation of their tutors, mean scores for the two groups were 4.28 for the experimental group and 4. 06 for the control group (P = 0.10). The mean OSCE scores for the two groups were 76.4 for the experimental group and 76.5 for the control group (P = 0.93). There is no practical or statistical difference in either of these two scores.

CONCLUSIONS

We conclude that this new model is feasible, was well received by the students and that the reduced tutor to student ratio and lack of inpatient encounters did not appear to adversely affect their learning.

Gender, sequence of cases and day effects on clinical skills assessment with standardized patients

OBJECTIVE

Clinical skills assessment using standardized patients (SPs) is an accepted method of professional testing, but some factors can threaten the validity of the scores obtained.

OBJECTIVE

The main objective of this work was to test whether the gender of the patients, the sequence of cases and the day of the examination have any impact on the scores of the examinees.

MATERIALS

Data from examinations conducted at three Medical Schools are used. Each student encountered 10 different SPs (5 men and 5 women).

METHODS

Analysis of variance was used to test the existence of association between the variables.

RESULTS

The results showed no significant differences or association between the scores and the three variables analysed.

CONCLUSIONS

The results are coherent with the studies that show that performance based examination using SPs can be used without introducing biases into students score.

Teaching doctors about the eye: trends in the education of medical students and primary care residents

Ophthalmologists play a relatively peripheral role in medical student and resident education. A review of the evolution, funding, and administration of medical education in the United States lends insight into why this is so. The author reviews the current status of education in ophthalmology for medical students and residents; the development of an ophthalmology curriculum; alternatives to the traditional medical school curriculum and how these have been incorporated into ophthalmic education; and the effect of new trends in medical education and pressures created by health care reform on the education of medical students and residents. The role of ophthalmologists in the general practice of medicine and in teaching doctors about the eye is discussed. Finally, the opportunity that the current climate presents for ophthalmologists to define their role in health care through education is considered.

A new assessment tool: the patient assessment and management examination

BACKGROUND

The major goal of certification is to assure the public that the candidate is competent in all facets required of the position. The patient assessment and management examination (PAME) was developed to enable a more comprehensive assessment of competence in the practice of surgery.

METHODS

A six-station, 3-hour, standardized-patient-based evaluation was developed. Each station was scored using a set of five-point global rating scales. PAME results were compared to the last two in training evaluation reports (ITER), the clinical knowledge component of the ITER (ITER-CK), an in-house oral examination (OE), and the Canadian Association of General Surgeons' multiple-choice examination (CAGS).

RESULTS

Eighteen senior general surgery residents were evaluated. Overall reliability was 0.70 (Cronbach's alpha). Fifth-year residents scored significantly better than fourth-year residents (t = 3.062; p = 0.0074), with 1 year of training accounting for 37% of the variance in scores. Correlations between the PAME and each of the other measures were ITER, 0.24; ITER-CK, 0.38; OE, -0.13; and CAGS, 0.061, with the PAME demonstrating better reliability and stronger evidence of validity than any other.

CONCLUSIONS

The PAME had better psychometric properties than other measures and assessed areas often not evaluated. This type of evaluation may be useful for feedback, remediation, or certification decisions.

Increasing examiner involvement in an objective structured clinical examination by integrating a structured oral examination

INTRODUCTION

The role of physician examiners in an objective structured clinical examination (OSCE) is relatively passive. In our institution examiners criticized the passive nature of their role. This study evaluates the reliability and viability of adding a structured oral examination to an OSCE.

METHOD

Ten 24-minute stations consisted of three parts. Part I: 12 minutes-patient encounter. Part II: 6 minutes-oral presentation covering findings, differential diagnosis, and management plan. Part III: 6 minutes-structural oral examination (SOE), containing 5 predetermined questions.

RESULTS

Over 6 consecutive days, 72 graduates were assessed. Overall average score: 72.02 (SD 5.05); reliability 0.84. Part I of the OSCE average score: 69.2 (SD 7.4); reliability 0.69. Part II oral presentation average score 64 (SD 5.8) reliability 0.87. SOE average score 77.7 (SD 6.3); reliability 0.64. Eighty-nine percent of the examiners indicated satisfaction with the new format.

CONCLUSIONS

The SOE was a reliable component of an OSCE and contributed to the overall reliability. Examiners reported a higher degree of satisfaction with the examination.

Performance-based assessment in continuing medical education for general practitioners: construct validity

The use of performance-based assessment has been extended to postgraduate education and practising doctors, despite criticism of validity. While differences in expertise at this level are easily reflected in scores on a written test, these differences are relatively small on performance-based tests. However, scores on written tests and performance-based tests of clinical competence generally show moderate correlations. A study was designed to evaluate construct validity of a performance-based test for technical clinical skills in continuing medical education for general practitioners, and to explore the correlation between performance and knowledge of specific skills. A 1-day skills training was given to 71 general practitioners, covering four different technical clinical skills. The effect of the training on performance was measured with a performance-based test using a randomized controlled trial design, while the effect on knowledge was measured with a written test administered 1 month before and directly after the training. A training effect could be shown by the performance-based test for all four clinical skills. The written test also demonstrated a training effect for all but one skill. However, correlations between scores on the written test and on the performance-based test were low for all skills. It is concluded that construct validity of a performance-based test for technical clinical skills of general practitioners was demonstrated, while the knowledge test score was shown to be a poor predictor of competence for specific technical skills.

The development of a 'Standardised Learner' in researching teaching behaviours

Standardised patients are increasingly used in medical education and the reliability and validity of their use as an assessment method are supported by published research. This paper details the application of this methodology to the educational interaction between a teacher and a learner using a standardised learner.In contrast with standardised patients, a standardised learner must use a level of medical knowledge and experience appropriate for the role assumed. Some aspects of reliability and validity have been considered and appear supportive of this methodology.The feasibility and value of this method for researching actual teacher behaviour are discussed.

Checklist self-evaluation in a standardized patient exercise

BACKGROUND

Standardized patient (SP) exercises are expensive and time consuming. We hypothesized that self-evaluation would further the goals of teaching and evaluation for a group of medical students.

METHODS

Students were given self-evaluation checklists (essentially identical to those filled out by the SPs) and completed them prior to reviewing the SP checklists. Agreement between student and SP checklists (whether each item was checked or not) and the extent of agreement on the interactional skills rating scales were assessed.

RESULTS

Overall agreement was 93%, with 98% agreement on interactional items, 92% on physical examination items, and 89% on history items. Disagreements tended to be clustered on a few items in each scale. Scores on the interactional skills rating scales also showed strong agreement.

CONCLUSION

The use of checklists and rating scales in this context is well established. At virtually no cost, a self-evaluation tool adds a valuable dimension to the exercise.

Evaluating students' substance abuse assessment skills

Graduating nursing students' abilities to assess clients for substance use were evaluated using two standardized patient (SP) cases. In the comprehensive health history case (unprompted), 55% of the study participants assessed alcohol use and 20% assessed drug use. When prompted to conduct a substance use history, at least 90% of the subjects asked about the quantity and frequency of alcohol use and 80% asked about drug use. However, in the prompted case, fewer than 10% of the subjects linked current alcohol and drug (AOD) use with risks and consequences, nor did they ask the SP if he had considered decreasing substance use. The mean score for communication skills used in the two assessment interviews were 3.65 and 3.56 on a 1 to 6 Likert scale. These data point out the need for additional emphasis on instructing students to include AOD questions as part of routine assessments, what questions to ask in a substance use assessment, and how to act on the information received. The use of SPs is one way to obtain valid information about students' assessment competencies.