Expert panel consensus on assessment checklists for a rheumatology objective structured clinical examination

Virtual Learning and Assessment in Rheumatology Fellowship Training: Objective Structured Clinical Examination Revisited

OBJECTIVE

With the onset of the COVID-19 pandemic, an annual multi-institutional face-to-face rheumatology objective structured clinical examination (ROSCE) was transformed into a virtual format. The educational goals of the virtual ROSCE (vROSCE) were to reproduce the educational value of the previous in-person ROSCE, providing a valuable formative assessment of rheumatology training activities encompassing the 6 Accreditation Council for Graduate Medical Education (ACGME) core competencies for fellows-in-training (FITs). This article describes the novel design, feasibility, and stakeholder value of a vROSCE.

METHODS

Through an established collaboration of 5 rheumatology fellowship training programs, in February 2021, a vROSCE was created and conducted using a Zoom platform. Station development included learning objectives, FIT instructions, faculty proctor instructions, and a checklist by which to provide structured formative feedback. An anonymous, optional web-based survey was sent to FIT participants to evaluate the experience.

RESULTS

Twenty-three rheumatology FITs from 5 institutions successfully rotated through 6 stations in the vROSCE. Immediate feedback was given to each FIT using standardized rubrics structured around ACGME core competencies. A total of 65% of FITs (15 of 23) responded to the survey, and 93% of survey respondents agreed or strongly agreed that the vROSCE was a helpful educational activity and identified individualized opportunities for improvement.

CONCLUSION

A vROSCE is an innovative, feasible, valuable, and well-received educational technology tool. The vROSCE enriched rheumatology FITs' education and offered collaborative learning experiences across institutions.

Development of an Objective Structured Clinical Examination Checklist and a Post-Education Questionnaire for Musculoskeletal Ultrasound Training Focusing on Volar Wrist and Carpal Tunnel Syndrome

Background: The objective structured clinical examination (OSCE) is used in the colleges of Korean Medicine, but few studies have validated the OSCE evaluation criteria or post-education questionnaires. Diagnostic ultrasound is used in Korean medicine treatment including acupuncture, acupotomy, and pharmacopuncture to increase the safety and efficacy of treatment. We aimed to develop and validate a OSCE checklist and questionnaire for diagnostic musculoskeletal ultrasound training.Methods: A OSCE checklist and rubric for diagnostic ultrasound training, and questionnaire was developed using literature research. Eight expert panelists verified each draft item in a single-round survey. Items with a content validity ratio (CVR) < 0.75 were excluded or modified to reflect the experts’ opinions.Results: The OSCE checklist and rubric for diagnostic ultrasound training focusing on volar wrist and carpal tunnel syndrome included: 15 items revised according to CVR and expert opinions, the pre-examination procedure, structures to be identified by ultrasound, scans with 2 diagnostic criteria for carpal tunnel syndrome, an explanation of the exam results, and the post exam procedure. The questionnaire consisted of 15 items, including the overall evaluation of training, the effect of the OSCE, and the perception of the ultrasound. All 6 self-evaluation items were not revised, as they had a CVR of ≥ 0.75.Conclusion: An ultrasound OSCE for scanning the volar wrist and diagnosing carpal tunnel syndrome was developed using 15 validated tasks, 15 survey questions about ultrasound training, and 6 questions for selfevaluation. These results may be used in the future for education in diagnostic ultrasound.

Development, validation and application of clinical simulation scenarios for assessment of stomatherapy specialists

OBJECTIVES

to build and validate three clinical simulation scenarios and report the application with candidates for the specialist's degree in stomatherapy.

METHODS

methodological study, building three scenarios and evaluation checklists; content validation with judges, using content validity index and Modified Kappa Coefficient; pre-test and application.

RESULTS

scenarios built based on nursing care for: 1. insufficiency and venous ulcer; 2. demarcation of intestinal stomia; and 3. Clean intermittent catheterization. In the content validation of the 24 items appreciated, 83%, 80%, and 92% were validated without change. In the pre-test, the objectives and checklists were adjusted. In the application, to standardize the evaluation, actors and evaluators were trained previously, and each candidate passed the three stations.

CONCLUSIONS

scenarios built and with validated content, based on evidence and covering the three areas of stomatherapy. The pre-test allowed for adjustments in the scenarios, and the candidates achieved the expected objectives.

Designing and Executing a Checklist for Evaluating Midwifery Students' Practical Skills

Background

This study aimed to design and execute a reliable checklist for evaluating midwifery students' clinical skills.

Materials and Methods

In this cross-sectional study, 8 checklists were designed and validated for midwifery procedures. The students' performances were compared using this method and two other evaluation methods.

Results

The face and content validity of all questions were approved (content validity ratio and content validity index >0.80, inter-rater reliability >0.50, and Cronbach's alpha >0.70). The rate of the students' satisfaction was 19%, 14%, and 76% with the nonchecklist method, the general checklist, and the designed checklist, respectively. Furthermore, both students (F_2,60= 107.07, p < 0.004) and evaluators (F_2,9 = 152.23, p < 0.001) gained significantly higher attitude scores towards the new checklist compared to the two other methods.

Conclusions

The designed checklist was quite reliable and valid for evaluating the midwifery students' clinical skills.

Turning Objective Structured Clinical Examinations into Reality

Objective structured clinical examinations assess learners "showing how" to perform complex clinical tasks. Devised as summative evaluations, these examinations with immediate feedback are useful formative evaluations to improve learner performance. This review describes how objective structured clinical examinations have been used in rheumatology education. Steps for creating an objective structured clinical examination are discussed. Validity and reproducibility are important considerations, especially for high-stakes summative objective structured clinical examinations. Consideration of the potential benefits in clinical education and their hazards are reviewed. When well-designed, formative objective structured clinical examinations have high educational value for learners and medical educators.

The method used to set the pass mark in an objective structured clinical examination defines the performance of candidates for certification as rheumatologists

BACKGROUND

The Mexican Accreditation Council for Rheumatology certifies trainees (TR) on an annual basis using both a multiple-choice question (MCQ) test and an objective structured clinical examination (OSCE). For 2013 and 2014, the OSCE pass mark (PM) was set by criterion referencing as ≥6 (CPM), whereas overall rating of borderline performance method (BPM) was added for 2015 and 2016 accreditations. We compared OSCE TR performance according to CPM and BPM, and examined whether correlations between MCQ and OSCE were affected by PM.

METHODS

Forty-three (2015) and 37 (2016) candidates underwent both tests. Altogether, OSCE were integrated by 15 validated stations; one evaluator per station scored TR performance according to a station-tailored check-list and a Likert scale (fail, borderline, above range) of overall performance. A composite OSCE score was derived for each candidate. Appropriate statistics were used.

RESULTS

Mean (±standard derivation [SD]) MCQ test scores were 6.6±0.6 (2015) and 6.4±0.6 (2016) with 5 candidates receiving a failing score each year. Mean (±SD) OSCE scores were 7.4±0.6 (2015) and 7.3±0.6 (2016); no candidate received a failing CPM score in either 2015 or 2016 OSCE, although 21 (49%) and 19 (51%) TR, respectively, received a failing BPM score (calculated as 7.3 and 7.4, respectively). Stations for BPM ranged from 4.5 to 9.5; overall, candidates showed better performance in CPM. In all, MCQ correlated with composite OSCE, r=0.67 (2015) and r=0.53 (2016); P≤.001. Trainees with a passing BPM score in OSCE had higher MCQ scores than those with a failing score.

CONCLUSIONS

Overall, OSCE-PM selection impacted candidates' performance but had a limited affect on correlation between clinical and practical examinations.

Validity evidence for two objective structured clinical examination stations to evaluate core skills of the shoulder and knee assessment

BACKGROUND

We developed two objective structured clinical examinations (OSCEs) to educate and evaluate trainees in the evaluation and management of shoulder and knee pain. Our objective was to examine the evidence for validity of these OSCEs.

METHODS

A multidisciplinary team of content experts developed checklists of exam maneuvers and criteria to guide rater observations. Content was proposed by faculty, supplemented by literature review, and finalized using a Delphi process. One faculty simulated the patient, another rated examinee performance. Two faculty independently rated a portion of cases. Percent agreement was calculated and Cohen's kappa corrected for chance agreement on binary outcomes. Examinees' self-assessment was explored by written surveys. Responses were stratified into 3 categories and compared with similarly stratified OSCE scores using Pearson's coefficient.

RESULTS

A multi-disciplinary cohort of 69 examinees participated. Examinees correctly identified rotator cuff and meniscal disease 88% and 89% of the time, respectively. Inter-rater agreement was moderate for the knee (87%; k = 0.61) and near perfect for the shoulder (97%; k = 0.88). No correlation between stratified self-assessment and OSCE scores were found for either shoulder (0.02) or knee (-0.07).

CONCLUSIONS

Validity evidence supports the continuing use of these OSCEs in educational programs addressing the evaluation and management of shoulder and knee pain. Evidence for validity includes the systematic development of content, rigorous control of the response process, and demonstration of acceptable interrater agreement. Lack of correlation with self-assessment suggests that these OSCEs measure a construct different from learners' self-confidence.

Establishing cross-discipline consensus on contraception, pregnancy and breast feeding-related educational messages and clinical practices to support women with rheumatoid arthritis: an Australian Delphi study

OBJECTIVE

Recognising the need for a best-practice and consistent approach in providing care to women with rheumatoid arthritis (RA) in relation to (1) general health, (2) contraception, (3) conception and pregnancy, (4) breast feeding and (5) early parenting, we sought to achieve cross-discipline, clinical consensus on key messages and clinical practice behaviours in these 5 areas.

DESIGN

3-round eDelphi study. In round 1, panellists provided free-text responses to open-ended questions about care for women with RA across the 5 areas. Subsequently, panellists refined and scored the synthesised responses, presented as metathemes, themes and detailed elements. Where ≥5% of panellists did not support a theme in a given round, it was removed.

SETTING

Panel of practicing Australian rheumatologists (n=22), obstetricians/obstetric medicine physicians (n=9) and pharmacists (n=5).

RESULTS

34 (94.4%) panellists participated in all 3 rounds. The panel supported 18 themes across the 5 areas (support/strongly support: 88.2-100%) underpinned by 5 metathemes. Metathemes focused on coordination in information delivery, the mode and timing of information delivery, evidence underpinning information, engagement of the right health professionals at the right time and a non-judgemental approach to infant feeding. Themes included practices for primary prevention of chronic disease and their sequelae, the importance of contraception and planning pregnancy and breast feeding, close monitoring of medications, supporting mental well-being, managing disease activity and providing practical support for early parenting.

CONCLUSIONS

A cross-disciplinary clinical panel highly supported key information and clinical practices in the care for women with RA across the continuum of contraception to early parenting within a whole-person, chronic disease management approach.