Process-rating forms versus task-specific checklists in an OSCE for medical licensure. Medical Council of Canada

Evaluating the validity evidence of an OSCE: results from a new medical school

BACKGROUND

To prevent the problems of traditional clinical evaluation, the "Objective Structured Clinical Examination (OSCE)" was presented by Harden as a more valid and reliable assessment instrument. However, an essential condition to guarantee a high-quality and effective OSCE is the assurance of evidence to support the validity of its scores. This study examines the psychometric properties of OSCE scores, with an emphasis on consequential and internal structure validity evidence.

METHODS

Fifty-three first year medical students took part in a summative OSCE at the Lebanese American University-School of Medicine. Evidence to support consequential validity was gathered by using criterion-based standard setting methods. Internal structure validity evidence was gathered by examining various psychometric measures both at the station level and across the complete OSCE.

RESULTS

Compared to our actual method of computing results, the introduction of standard setting resulted in lower students' average grades and a higher cut score. Across stations, Cronbach's alpha was moderately low.

CONCLUSION

Gathering consequential and internal structure validity evidence by multiple metrics provides support for or against the quality of an OSCE. It is critical that this analysis be performed routinely on local iterations of given tests, and the results used to enhance the quality of assessment.

Large-scale multi-site OSCEs for national competency examination of medical doctors in Indonesia

BACKGROUND

Implementing large-scale multi-site objective structured clinical examination (OSCEs) for national competency examination in a low resource country is challenging.

AIMS

To describe the first national OSCE for national competency examination of medical doctors in Indonesia and evaluate the reliability, validity, feasibility, acceptability, and educational impact.

METHODS

We collected electronically the OSCE scores from 49 out of 73 medical schools that participated to assess reliability and validity. We conducted electronic survey to examiners, examinees, SP trainers, and OSCE coordinators to assess feasibility, acceptability, and educational impact.

RESULTS

The Cronbach's alpha coefficient across station was 0.79. There was strong correlation between rubric and global rating scores in each station (coefficient correlation ranges from 0.705 to 0.82). The content validity ratio was 0.97. The coefficient correlation between OSCE and MCQ was 0.335 (p = 0.00). All 49 medical schools were able to conduct OSCE simultaneously. Examiners, examinees, SP trainers, and OSCE coordinators had good perception regarding feasibility and acceptability of OSCE. Both examiners and examinees indicated good educational impact of OSCE application. The cutting score based on the borderline regression method was 61.96%. There were 67.39% of the examinees achieved similar or above the cutting score.

CONCLUSION

With 12 stations 15 min each, the reliability coefficient across station is intermediate. Content validity is good. It is feasible and acceptable to implement large-scale multi-site OSCEs in Indonesia. Examiners and examinees perceive good educational impact on OSCE implementation.

Practitioner research and formative assessment

BACKGROUND

Several workplace-based formative assessment (WFA) tools exist; however, text-based feedback is minimal but valuable because residents have difficulty extracting meaning from numerical scores or rankings. Our programme lacked a formal WFA programme, so we aimed to develop and assess a primarily text-based tool, named formative assessment of skills in training (FAST), using action research.

METHODS

Action research (AR) methods, including iterative plan-act-observe-reflect cycles that included the FAST tool and our clinical context. Thirteen residents and 11 faculty members performed 133 assessments during three study cycles, and responded to post-use surveys.

RESULTS

Overall, 83 per cent of participants indicated that FAST should be added to the resident curriculum. Time was perceived to be a barrier; however, time studies found that it did not prolong resident wait or patient visits. Changes were made to increase space for comments on the FAST form, and subsequently comment specificity and length increased.

DISCUSSION

The FAST facilitates WFA in our programme and allows for specific written feedback with a copy retained by the resident. On average, it adds about 3 minutes to a clinic visit. The AR method facilitated WFA use, stakeholder buy-in, and FAST tool improvements.

Reliability and validity of an extended clinical examination

INTRODUCTION

An extended clinical examination (ECE) was administered to 85 final year medical students at the Radboud University Medical Centre in the Netherlands. The aim of the study was to determine the psychometric quality and the suitability of the ECE as a measurement tool to assess the clinical proficiency of eight separate clinical skills.

METHODS

Generalizability studies were conducted to determine the generalizability coefficient and the sources of variance of the ECE. An additional D-study was performed to estimate the generalizability coefficients with altering numbers of stations.

RESULTS

The largest sources of variance were found in skill difficulties (36.18%), the general error term (26.76%) and in the rank ordering of skill difficulties across the stations (21.89%). The generalizability coefficient of the entire ECE was above the 0.70 lower bound (G = 0.74). D studies showed that the separate skills could yield sufficient G coefficients in seven out of eight skills, if the ECE was lengthened from 8 to 14 stations.

DISCUSSION

The ECE proved to be a reliable clinical assessment that enables examinees to compose a clinical reasoning path through self-obtained data. The ECE can also be used as an assessment tool for separate clinical skills.

Assessing the assessment in emergency care training

Objective

Each year over 1.5 million health care professionals attend emergency care courses. Despite high stakes for patients and extensive resources involved, little evidence exists on the quality of assessment. The aim of this study was to evaluate the validity and reliability of commonly used formats in assessing emergency care skills.

Methods

Residents were assessed at the end of a 2-week emergency course; a subgroup was videotaped. Psychometric analyses were conducted to assess the validity and inter-rater reliability of the assessment instrument, which included a checklist, a 9-item competency scale and a global performance scale.

Results

A group of 144 residents and 12 raters participated in the study; 22 residents were videotaped and re-assessed by 8 raters. The checklists showed limited validity and poor inter-rater reliability for the dimensions “correct” and “timely” (ICC = .30 and.39 resp.). The competency scale had good construct validity, consisting of a clinical and a communication subscale. The internal consistency of the (sub)scales was high (α = .93/.91/.86). The inter-rater reliability was moderate for the clinical competency subscale (.49) and the global performance scale (.50), but poor for the communication subscale (.27). A generalizability study showed that for a reliable assessment 5–13 raters are needed when using checklists, and four when using the clinical competency scale or the global performance scale.

Conclusions

This study shows poor validity and reliability for assessing emergency skills with checklists but good validity and moderate reliability with clinical competency or global performance scales. Involving more raters can improve the reliability substantially. Recommendations are made to improve this high stakes skill assessment.

Validity and reliability of the Standardized Orthopedic Assessment Tool (SOAT): a variation of the traditional objective structured clinical examination

CONTEXT

Health care professions have replaced traditional multiple choice tests or essays with structured and practical, performance-based examinations with the hope of eliminating rater bias and measuring clinical competence.

OBJECTIVE

To establish the validity and reliability of the Standardized Orthopedic Assessment Tool (SOAT) as a measure of clinical competence of orthopaedic injury evaluation.

DESIGN

Descriptive laboratory study.

SETTING

University.

PATIENTS OR OTHER PARTICIPANTS

A total of 60 undergraduate students and 11 raters from 3 Canadian universities and 1 standardized patient.

INTERVENTION(S)

Students were required to complete a 30-minute musculoskeletal evaluation in 1 of 2 randomly assigned mock scenarios involving the knee (second-degree medial collateral ligament sprain) or the shoulder (third-degree supraspinatus muscle strain).

MAIN OUTCOME MEASURE(S)

We measured interreliability with an intraclass correlation coefficient (ICC) (2,k) and stability of the tool with standard error of measurement and confidence intervals. Agreement was measured using Bland-Altman plots. Concurrent validity was measured using a Pearson product moment correlation coefficient whereby the raters' global rating of a student was matched to the cumulative mean grade score.

RESULTS

The ICCs were 0.75 and 0.82 for the shoulder and knee cases, respectively. Bland-Altman plots indicated no systematic bias between raters. In addition, Pearson product moment correlation analysis demonstrated a strong relationship between the overall cumulative mean grade score and the global rating score of the examinees' performances.

CONCLUSIONS

This study demonstrated good interrater reliability of the SOAT with a standard error of measurement that indicated very modest stability, strong agreement between raters, and correlation indicative of concurrent validity.

The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part II: organisation & administration

The organisation, administration and running of a successful OSCE programme need considerable knowledge, experience and planning. Different teams looking after various aspects of OSCE need to work collaboratively for an effective question bank development, examiner training and standardised patients' training. Quality assurance is an ongoing process taking place throughout the OSCE cycle. In order for the OSCE to generate reliable results it is essential to pay attention to each and every element of quality assurance, as poorly standardised patients, untrained examiners, poor quality questions and inappropriate scoring rubrics each will affect the reliability of the OSCE. The validity will also be influenced if the questions are not realistic and mapped against the learning outcomes of the teaching programme. This part of the Guide addresses all these important issues in order to help the reader setup and quality assure their new or existing OSCE programmes.

A predictive and construct validity study of a high-stakes objective clinical examination for assessing the clinical competence of international medical graduates

BACKGROUND

The purpose of this study was to investigate the predictive and construct validity of a high-stakes objective structured clinical examination (OSCE) used to select candidates for a 3-month clinical rotation to assess practice-readiness status.

SUMMARY

Analyses were undertaken to establish the reliability and validity of the OSCE. The generalizability coefficient (Ep(2)) for the assessment scores (checklist, global, and total) were all high, ranging from 0.73 to 0.84. Two discriminant analyses (promotion to the 3-month rotation and pass/fail status on the rotation) provided evidence of predictive validity with a 100% correct classification rate in the pass/fail rotation results. Factor analysis results provided evidence of construct validity with four factors identified: Clinical Skills, Internal Medicine, General Medical Knowledge, and Counseling. The known group differences between licensing status and residency experience also provided evidence of construct validity.

CONCLUSIONS

The results are encouraging for the predictive and construct validity of the OSCE as an assessment of clinical competence.

GULiVER--travelling into the heart of good doctor-patient communication from a patient perspective: study protocol of an international multicentre study

BACKGROUND

The project GULiVer explores how lay people in Belgium (Gent), the Netherlands (Utrecht), the UK (Liverpool) and Italy (Verona) evaluate physicians' communicative skills. The aims are to present the study design and to assess the quality of collected data.

METHODS

In each centre one out of two sets of four videotaped consultations involving medical students with varying communication skills were shown to eight lay panels of six to nine participants each (n = 259). The selection of lay participants was stratified by gender and age in order to obtain a heterogeneous sample. Background characteristics included socio-demographics, participants' general physical (COOP-WONCA) and mental health (GHQ), communication preferences (QUOTE-com) and trust in doctors (TMP). Participants were asked to give quantitative and qualitative evaluations of the student doctors' performance in a mixed-methods design. Quality assessment of the collected data and protocol adherence of the four centres was carried out by Generalized Linear Model (GLM).

RESULTS

The overall sample comprised 259 participants. Participants were equally distributed among the centres and balanced in terms of age, gender and OSCE scenario, confirming the quality of collected data.

CONCLUSION

The study design and the applied procedures will ensure a great richness of data allowing a wider European perspective on lay persons' views, assessed both individually and through focus group discussion.

Numeracy for nursing, report of a pilot study to compare outcomes of two practical simulation tools--an online medication dosage assessment and practical assessment in the style of objective structured clinical examination

This pilot study compares the results of medications calculations carried out by student nurses using an online assessment tool with the results of the same calculations carried out within simulated practice. The numeracy project, of which this is part, is funded by NHS Education for Scotland (NES).

OBJECTIVE

To test the efficacy of a realistic computer-based assessment of pre-registration nurses' medication calculations skills by comparing outcomes from using an online assessment tool with a practical assessment tool in the style of an Objective Structured Clinical Examination OSCE. Both assessment methods used medicine calculations usually presenting authentically in the practice setting.

DESIGN

A multi-stage quantitative study using a cross-over design.

SAMPLE

Fifty early 3rd year students on the adult branch of a pre-registration nursing programme at a large school of nursing in England.

RESULTS

Results showed that, for assessing accuracy of calculation, there was a high level of congruence between the two methods.

CONCLUSIONS

Computerised assessment of medications calculations using this particular platform is likely to closely mirror assessment of medication calculations done in a practical setting. As such it could be a useful adjunct to current assessment methods.

Competency-based training: objective structured clinical exercises (OSCE) in marriage and family therapy

The field of marriage and family therapy (MFT) has recently engaged in the process of defining core competencies for the profession. Many MFT training programs are adapting their curriculum to develop more competency-based training strategies. The Objective Structured Clinical Examination (OSCE) is widely used in the medical profession to assess clinical competence. These examinations involve using simulated clinical situations as a tool in conducting summative evaluations of trainee competence. This article describes an adaptation of the OSCE procedures for competency-based training of MFT students. Instead of using the procedures as a summative examination as is typical in medical education, this article proposes how to use them as formative exercises in the development of student competence. The development of the OSCE is discussed, including "blueprinting," focused competencies, procedures, and feedback protocols. The article concludes with suggestions of how to continue the development of the OSCE for evaluation in MFT education.

Transatlantic comparison of the competence of surgeons at the start of their professional career

Background

Although the objective in European Union and North American surgical residency programmes is similar—to train competent surgeons—residents' working hours are different. It was hypothesized that practice-ready surgeons with more working hours would perform significantly better than those being educated within shorter working week curricula.

Methods

At each test site, 21 practice-ready candidate surgeons were recruited. Twenty qualified Canadian and 19 qualified Dutch surgeons served as examiners. At both sites, three validated outcome instruments assessing multiple aspects of surgical competency were used.

Results

No significant differences were found in performance on the integrative and cognitive examination (Comprehensive Integrative Puzzle) or the technical skills test (Objective Structured Assessment of Technical Skill; OSATS). A significant difference in outcome was observed only on the Patient Assessment and Management Examination, which focuses on skills needed to manage patients with complex problems (P < 0·001). A significant interaction was observed between examiner and candidate origins for both task-specific OSATS checklist (P = 0·001) and OSATS global rating scale (P < 0·001) scores.

Conclusion

Canadian residents, serving many more working hours, perform equivalently to Dutch residents when assessed on technical skills and cognitive knowledge, but outperformed Dutch residents in skills for patient management. Secondary analyses suggested that cultural differences influence the assessment process significantly.

Development of a content-valid standardized orthopedic assessment tool (SOAT)

INTRODUCTION

Content validation of an instrument that measures student performance in OSCE-type practical examinations is a critical step in a tool's overall validity and reliability [Hopkins (1998), Educational and Psychological Measurement and Evaluation (8th ed.). Toronto: Allyn & Bacon].

PURPOSE

The purpose of the paper is to outline the process employed to create a content-valid Standardized Orthopedic Assessment Tool (SOAT). Orthopedic assessment skills are employed by athletic therapists, physiotherapists and physicians. All follow very similar diagnostic algorithms and that system was used to develop the original SOAT [Cyriax (1982). Textbook of Orthopaedic Medicine, (vol. 1) Bailliere Tindall].

METHODS

To validate the tool, the study followed procedures outlined by Violato, Salami, and Muiznieks (2002), Journal of Manipulative Physiological Therapeutics, 25, 111-115, and Butterwick, Paskevich, Vallevand and Lafave (2006), Journal of Allied Health: a modified Ebel procedure. An expert group of athletic therapists from across Canada were chosen to create the content valid tool. Representation from all accredited athletic therapy programs in Canada was sought. Experts participated in three stages of content validation: Stage one consisted of individuals grading tasks on difficulty (hard, medium, easy) and importance (essential, important, not important) for 8 body regions (cervical spine, lumbar spine, shoulder, elbow, wrist/hand/thumb, hip, knee and lower leg/foot/ankle) and three diagnoses for each body region (24 total). If 80% consensus was not achieved in the first stage, then in stage two, face to face discussion is meant to clarify positions and achieved consensus, if possible.

RESULTS

The history component, the observation component, scanning exams, clearing joints above and below the lesion site and active range of motion, passive range of motion and isometric resisted testing all yielded 80% consensus in the first two stages of development. A third stage was added to this process because a new model of measurement was conceptualized during the second stage due to a lack of consensus on two orthopedic assessment categories: special testing and palpation. The new model consisted of a ;;clinical reasoning" metric that tied each section of an orthopedic assessment (history; observation; scanning and clearing; examination, special testing; palpation; conclusion) together and would permit the expert rater to evaluate the student performance based on the student's rationale for tests chosen rather than the traditionally rigid checklists. At least 80% consensus was achieved with the new ;;clinical reasoning" metric and the originally contentious special testing and palpation categories.

CONCLUSIONS

The new SOAT that underwent content validation may be a hybrid between the traditional OSCE-type checklists and global scales that provide a more realistic and thus more valid depiction of orthopedic assessment clinical competence. Now that content validation has been established, the next steps include inter-rater reliability testing.

Objective structured clinical examination (OSCE): review of literature and implications for nursing education

Objective structured clinical examination (OSCE) has been used to assess medical students since the mid 1970s, and in more recent years has been increasingly utilised to assess students from nursing and the allied health professions. This growing utilisation has led to considerable debate within the literature pertaining to the optimal use of OSCE as an assessment process. The purpose of this paper is to present a narrative review regarding some of the key issues affecting the utilisation of OSCE within the assessment of nursing students. The paper briefly reviews the historical development of OSCE within health professional assessment, and summarises some of its key strengths and limitations. It then offers a more 'in depth' consideration of the research literature pertaining to the reliability and validity of the OSCE process, which is then used as a basis for exploring some of the particular issues that need to be considered when OSCE is used to assess nursing students. Key issues identified include the need to carefully prepare and pilot new OSCE examinations and marking tools in order to ensure reliability and validity is optimised, and also the need to carefully consider the length, number and interdependence of OSCE stations to ensure that the potentially competing requirements of validity and reliability are balanced. The paper also recognises that whilst the evidence base regarding OSCE is extensive, the evidence base specific to nursing is more limited. There is therefore scope for further research in this area, as well as the need for careful debate regarding how national guidance may be a way of enhancing and standardising future OSCE examinations. The paper concludes that whilst caution must be applied in relying on OSCE as a sole means of practitioner assessment, used carefully it can make a helpful and meaningful contribution to health professional education.

The Objective Structured Clinical Examination: three decades of development

The Objective Structured Clinical Examination (OSCE) has become a standard for performance-based assessment in many health professions. Originally developed for assessment in medical schools, the OSCE has been widely adopted for teaching, assessment, and certification purposes in most health professions. This article outlines the reasons for the rapid uptake of OSCEs and explores some of the key features of OSCE development that have implications for use in veterinary medicine.

OSCE! Variations on a theme by Harden

BACKGROUND

In 1979, Harden described the first objective structured clinical examination (OSCE). Harden's OSCE dramatically changed the assessment of professional competence because it used actors and choreographed scenarios to evaluate the performance of professional behaviours.

ANALYSIS

Because of the intense focus on performance, OSCEs have had a powerful influence on doctor training and practice. However, the immediate psychometric characteristics of OSCEs rather than their performance effects have been the subject of most research.

CONCLUSION

The time has come to undertake a sophisticated sociological investigation of how OSCEs affect medical practice, including the ways in which they shape doctor interaction with patients, families and other health professionals.

Analytic global OSCE ratings are sensitive to level of training

PURPOSE

There are several reasons for using global ratings in addition to checklists for scoring objective structured clinical examination (OSCE) stations. However, there has been little evidence collected regarding the validity of these scales. This study assessed the construct validity of an analytic global rating with 4 component subscales: empathy, coherence, verbal and non-verbal expression.

METHODS

A total of 19 Year 3 and 38 Year 4 clinical clerks were scored on content checklists and these global ratings during a 10-station OSCE. T-tests were used to assess differences between groups for overall checklist and global scores, and for each of the 4 subscales.

RESULTS

The mean global rating was significantly higher for senior clerks (75.5% versus 71.3%, t55 = 2.12, P < 0.05) and there were significant differences by level of training for the coherence (t55 = 3.33, P < 0.01) and verbal communication (t55 = 2.33, P < 0.05) subscales. Interstation reliability was 0.70 for the global rating and ranged from 0.58 to 0.65 for the subscales. Checklist reliability was 0.54.

CONCLUSION

In this study, a summated analytic global rating demonstrated construct validity, as did 2 of the 4 scales measuring specific traits. In addition, the analytic global rating showed substantially higher internal consistency than did the checklists, a finding consistent with that seen in previous studies cited in the literature. Global ratings are an important element of OSCE measurement and can have good psychometric properties. However, OSCE researchers should clearly describe the type of global ratings they use. Further research is needed to define the most effective global rating scales.

The challenge of creating new OSCE measures to capture the characteristics of expertise

PURPOSE

Although expert clinicians approach interviewing in a different manner than novices, OSCE measures have not traditionally been designed to take into account levels of expertise. Creating better OSCE measures requires an understanding of how the interviewing style of experts differs objectively from novices.

METHODS

Fourteen clinical clerks, 14 family practice residents and 14 family physicians were videotaped during 2 15-minute standardized patient interviews. Videotapes were reviewed and every utterance coded by type including questions, empathic comments, giving information, summary statements and articulated transitions. Utterances were plotted over time and examined for characteristic patterns related to level of expertise.

RESULTS

The mean number of utterances exceeded one every 10 s for all groups. The largest proportion was questions, ranging from 76% of utterances for clerks to 67% for experts. One third of total utterances consisted of a group of 'low frequency' types, including empathic comments, information giving and summary statements. The topic was changed often by all groups. While utterance type over time appeared to show characteristic patterns reflective of expertise, the differences were not robust. Only the pattern of use of summary statements was statistically different between groups (P < 0.05).

CONCLUSIONS

Measures that are sensitive to the nature of expertise, including the sequence and organisation of questions, should be used to supplement OSCE checklists that simply count questions. Specifically, information giving, empathic comments and summary statements that occupy a third of expert interviews should be credited. However, while there appear to be patterns of utterances that characterise levels of expertise, in this study these patterns were subtle and not amenable to counting and classification.

Assessment of medical students' communicative behaviour and attitudes: estimating the reliability of the use of the Amsterdam attitudes and communication scale through generalisability coefficients

It is widely accepted that adequate attitudes and communicative skills are among the essential objectives in medical education. The Amsterdam attitude and communication scale (AACS) was developed to assess communicative skills and professional attitudes of medical students. More specifically, it was designed to evaluate the clinical behaviour of clerks to establish their suitability for the medical profession. The AACS covers nine dimensions. Moreover, an overall judgement of the student's performance is included. The present paper reports first results on the reliability of the use of the AACS. Data were collected in the course of an AACS training programme for future judges: senior medical and nursing staff members (N=98). Participants judged three videotapes of clerks interviewing patients at the bedside. For the assessment of videotapes, the first four dimensions of the AACS and the overall judgement are relevant. By applying Generalisability Theory to the training data we can forecast the reliability of the AACS in practice and gain insight in the number of raters that is needed to achieve sufficient reliability in clinical practice. If clerk behaviour is rated by six judges, summative assessment is sufficiently precise, i.e. <0.25. When using the full AACS, covering 10 items, the same number of judges is needed. Scores on individual AACS items are not sufficiently reliable. In conclusion, the results indicate that students' behaviour can be evaluated in a reliable manner using the AACS as long as enough judges and items are involved.

Assessment of clinical competence

Tests of clinical competence, which allow decisions to be made about medical qualification and fitness to practise, must be designed with respect to key issues including blueprinting, validity, reliability, and standard setting, as well as clarity about their formative or summative function. Multiple choice questions, essays, and oral examinations could be used to test factual recall and applied knowledge, but more sophisticated methods are needed to assess clincial performance, including directly observed long and short cases, objective structured clinical examinations, and the use of standardised patients. The goal of assessment in medical education remains the development of reliable measurements of student performance which, as well as having predictive value for subsequent clinical competence, also have a formative, educational role.

In-training assessment - its potential in enhancing clinical teaching

CONTEXT

In-training assessment (ITA) has established its place alongside formative and summative assessment at both the undergraduate and postgraduate level. In this paper the authors aimed to identify those characteristics of ITA that could enhance clinical teaching.

METHODS

A literature review and discussions by an expert working group at the Ninth Cambridge Conference identified the aspects of ITA that could enhance clinical teaching.

RESULTS

The features of ITA identified included defining the specific benefits to the learner, teacher and institution, and highlighting the patient as the context for ITA and clinical teaching. The 'mapping' of a learner's progress towards the clinical teaching objectives by using multiple assessments over time, by multiple observers in both a systematic and opportunistic way correlates with the incremental nature of reaching clinical competence.

CONCLUSIONS

The importance of ITA based on both direct and indirect evidence of what the learner actually does in the real clinical setting is emphasized. Particular attention is given to addressing concerns in the more controversial areas of assessor training, ratings and documentation for ITA. Areas for future research are also identified.