1
|
Wolfe RM, Hant FN, Ishizawar RC, Criscione-Schreiber LG, Jonas BL, O'Rourke KS, Bolster MB. Virtual Learning and Assessment in Rheumatology Fellowship Training: Objective Structured Clinical Examination Revisited. Arthritis Care Res (Hoboken) 2023; 75:2435-2441. [PMID: 37291752 DOI: 10.1002/acr.25170] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2022] [Revised: 04/27/2023] [Accepted: 06/06/2023] [Indexed: 06/10/2023]
Abstract
OBJECTIVE With the onset of the COVID-19 pandemic, an annual multi-institutional face-to-face rheumatology objective structured clinical examination (ROSCE) was transformed into a virtual format. The educational goals of the virtual ROSCE (vROSCE) were to reproduce the educational value of the previous in-person ROSCE, providing a valuable formative assessment of rheumatology training activities encompassing the 6 Accreditation Council for Graduate Medical Education (ACGME) core competencies for fellows-in-training (FITs). This article describes the novel design, feasibility, and stakeholder value of a vROSCE. METHODS Through an established collaboration of 5 rheumatology fellowship training programs, in February 2021, a vROSCE was created and conducted using a Zoom platform. Station development included learning objectives, FIT instructions, faculty proctor instructions, and a checklist by which to provide structured formative feedback. An anonymous, optional web-based survey was sent to FIT participants to evaluate the experience. RESULTS Twenty-three rheumatology FITs from 5 institutions successfully rotated through 6 stations in the vROSCE. Immediate feedback was given to each FIT using standardized rubrics structured around ACGME core competencies. A total of 65% of FITs (15 of 23) responded to the survey, and 93% of survey respondents agreed or strongly agreed that the vROSCE was a helpful educational activity and identified individualized opportunities for improvement. CONCLUSION A vROSCE is an innovative, feasible, valuable, and well-received educational technology tool. The vROSCE enriched rheumatology FITs' education and offered collaborative learning experiences across institutions.
Collapse
Affiliation(s)
- Rachel M Wolfe
- Wake Forest School of Medicine, Winston-Salem, North Carolina
| | - Faye N Hant
- Medical University of South Carolina, Charleston
| | | | | | - Beth L Jonas
- University of North Carolina at Chapel Hill, Chapel Hill
| | | | - Marcy B Bolster
- Harvard Medical School and Massachusetts General Hospital, Boston
| |
Collapse
|
2
|
Development of an Objective Structured Clinical Examination Checklist and a Post-Education Questionnaire for Musculoskeletal Ultrasound Training Focusing on Volar Wrist and Carpal Tunnel Syndrome. JOURNAL OF ACUPUNCTURE RESEARCH 2022. [DOI: 10.13045/jar.2022.00038] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022] Open
Abstract
Background: The objective structured clinical examination (OSCE) is used in the colleges of Korean Medicine, but few studies have validated the OSCE evaluation criteria or post-education questionnaires. Diagnostic ultrasound is used in Korean medicine treatment including acupuncture, acupotomy, and pharmacopuncture to increase the safety and efficacy of treatment. We aimed to develop and validate a OSCE checklist and questionnaire for diagnostic musculoskeletal ultrasound training.Methods: A OSCE checklist and rubric for diagnostic ultrasound training, and questionnaire was developed using literature research. Eight expert panelists verified each draft item in a single-round survey. Items with a content validity ratio (CVR) < 0.75 were excluded or modified to reflect the experts’ opinions.Results: The OSCE checklist and rubric for diagnostic ultrasound training focusing on volar wrist and carpal tunnel syndrome included: 15 items revised according to CVR and expert opinions, the pre-examination procedure, structures to be identified by ultrasound, scans with 2 diagnostic criteria for carpal tunnel syndrome, an explanation of the exam results, and the post exam procedure. The questionnaire consisted of 15 items, including the overall evaluation of training, the effect of the OSCE, and the perception of the ultrasound. All 6 self-evaluation items were not revised, as they had a CVR of ≥ 0.75.Conclusion: An ultrasound OSCE for scanning the volar wrist and diagnosing carpal tunnel syndrome was developed using 15 validated tasks, 15 survey questions about ultrasound training, and 6 questions for selfevaluation. These results may be used in the future for education in diagnostic ultrasound.
Collapse
|
3
|
Almeida AOD, Dantas SRPE, Paula MABD, Silva JLG, Franck EM, Oliveira-Kumakura ARDS. Development, validation and application of clinical simulation scenarios for assessment of stomatherapy specialists. Rev Bras Enferm 2021; 74:e20200360. [PMID: 33787798 DOI: 10.1590/0034-7167-2020-0360] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2020] [Accepted: 06/28/2020] [Indexed: 11/21/2022] Open
Abstract
OBJECTIVES to build and validate three clinical simulation scenarios and report the application with candidates for the specialist's degree in stomatherapy. METHODS methodological study, building three scenarios and evaluation checklists; content validation with judges, using content validity index and Modified Kappa Coefficient; pre-test and application. RESULTS scenarios built based on nursing care for: 1. insufficiency and venous ulcer; 2. demarcation of intestinal stomia; and 3. Clean intermittent catheterization. In the content validation of the 24 items appreciated, 83%, 80%, and 92% were validated without change. In the pre-test, the objectives and checklists were adjusted. In the application, to standardize the evaluation, actors and evaluators were trained previously, and each candidate passed the three stations. CONCLUSIONS scenarios built and with validated content, based on evidence and covering the three areas of stomatherapy. The pre-test allowed for adjustments in the scenarios, and the candidates achieved the expected objectives.
Collapse
|
4
|
Refaei M, Ahmadini H, Masoumi SZ, Shayan A. Designing and Executing a Checklist for Evaluating Midwifery Students' Practical Skills. IRANIAN JOURNAL OF NURSING AND MIDWIFERY RESEARCH 2020; 25:356-358. [PMID: 33014749 PMCID: PMC7494170 DOI: 10.4103/ijnmr.ijnmr_255_19] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 11/01/2019] [Revised: 11/09/2019] [Accepted: 05/12/2020] [Indexed: 11/04/2022]
Abstract
Background This study aimed to design and execute a reliable checklist for evaluating midwifery students' clinical skills. Materials and Methods In this cross-sectional study, 8 checklists were designed and validated for midwifery procedures. The students' performances were compared using this method and two other evaluation methods. Results The face and content validity of all questions were approved (content validity ratio and content validity index >0.80, inter-rater reliability >0.50, and Cronbach's alpha >0.70). The rate of the students' satisfaction was 19%, 14%, and 76% with the nonchecklist method, the general checklist, and the designed checklist, respectively. Furthermore, both students (F2,60= 107.07, p < 0.004) and evaluators (F2,9 = 152.23, p < 0.001) gained significantly higher attitude scores towards the new checklist compared to the two other methods. Conclusions The designed checklist was quite reliable and valid for evaluating the midwifery students' clinical skills.
Collapse
Affiliation(s)
- Mansoureh Refaei
- Department of Midwifery, Mother and Child Care Research Center, School of Nursing and Midwifery, Hamadan University of Medical Sciences, Hamadan, Iran
| | - Hassan Ahmadini
- PhD Student of Biostatistics, Department of Biostatistics, School of Public Health, Hamadan University of Medical Sciences, Hamadan, Iran
| | - Seyedeh-Zahra Masoumi
- Department of Midwifery, Mother and Child Care Research Center, School of Nursing and Midwifery, Hamadan University of Medical Sciences, Hamadan, Iran
| | - Arezoo Shayan
- Department of Midwifery, Mother and Child Care Research Center, School of Nursing and Midwifery, Hamadan University of Medical Sciences, Hamadan, Iran
| |
Collapse
|
5
|
Criscione-Schreiber L. Turning Objective Structured Clinical Examinations into Reality. Rheum Dis Clin North Am 2019; 46:21-35. [PMID: 31757285 DOI: 10.1016/j.rdc.2019.09.010] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
Abstract
Objective structured clinical examinations assess learners "showing how" to perform complex clinical tasks. Devised as summative evaluations, these examinations with immediate feedback are useful formative evaluations to improve learner performance. This review describes how objective structured clinical examinations have been used in rheumatology education. Steps for creating an objective structured clinical examination are discussed. Validity and reproducibility are important considerations, especially for high-stakes summative objective structured clinical examinations. Consideration of the potential benefits in clinical education and their hazards are reviewed. When well-designed, formative objective structured clinical examinations have high educational value for learners and medical educators.
Collapse
Affiliation(s)
- Lisa Criscione-Schreiber
- Division of Rheumatology and Immunology, Program for Women in Internal Medicine, Department of Medicine, Duke University School of Medicine, Box 3490 DUMC, 40 Duke Medicine Circle, Durham, NC 27710, USA.
| |
Collapse
|
6
|
Pascual-Ramos V, Guilaisne Bernard-Medina A, Flores-Alvarado DE, Portela-Hernández M, Maldonado-Velázquez MDR, Jara-Quezada LJ, Amezcua-Guerra LM, Rubio-Judith López-Zepeda NE, Álvarez-Hernandez E, Saavedra MÁ, Arce-Salinas CA. The method used to set the pass mark in an objective structured clinical examination defines the performance of candidates for certification as rheumatologists. ACTA ACUST UNITED AC 2017; 14:137-141. [PMID: 28161396 DOI: 10.1016/j.reuma.2016.11.007] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2016] [Revised: 10/28/2016] [Accepted: 11/25/2016] [Indexed: 10/20/2022]
Abstract
BACKGROUND The Mexican Accreditation Council for Rheumatology certifies trainees (TR) on an annual basis using both a multiple-choice question (MCQ) test and an objective structured clinical examination (OSCE). For 2013 and 2014, the OSCE pass mark (PM) was set by criterion referencing as ≥6 (CPM), whereas overall rating of borderline performance method (BPM) was added for 2015 and 2016 accreditations. We compared OSCE TR performance according to CPM and BPM, and examined whether correlations between MCQ and OSCE were affected by PM. METHODS Forty-three (2015) and 37 (2016) candidates underwent both tests. Altogether, OSCE were integrated by 15 validated stations; one evaluator per station scored TR performance according to a station-tailored check-list and a Likert scale (fail, borderline, above range) of overall performance. A composite OSCE score was derived for each candidate. Appropriate statistics were used. RESULTS Mean (±standard derivation [SD]) MCQ test scores were 6.6±0.6 (2015) and 6.4±0.6 (2016) with 5 candidates receiving a failing score each year. Mean (±SD) OSCE scores were 7.4±0.6 (2015) and 7.3±0.6 (2016); no candidate received a failing CPM score in either 2015 or 2016 OSCE, although 21 (49%) and 19 (51%) TR, respectively, received a failing BPM score (calculated as 7.3 and 7.4, respectively). Stations for BPM ranged from 4.5 to 9.5; overall, candidates showed better performance in CPM. In all, MCQ correlated with composite OSCE, r=0.67 (2015) and r=0.53 (2016); P≤.001. Trainees with a passing BPM score in OSCE had higher MCQ scores than those with a failing score. CONCLUSIONS Overall, OSCE-PM selection impacted candidates' performance but had a limited affect on correlation between clinical and practical examinations.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | | | | | | | | | -
- Consejo Mexicano de Reumatología, Ciudad de México, México
| |
Collapse
|
7
|
Battistone MJ, Barker AM, Beck JP, Tashjian RZ, Cannon GW. Validity evidence for two objective structured clinical examination stations to evaluate core skills of the shoulder and knee assessment. BMC MEDICAL EDUCATION 2017; 17:13. [PMID: 28086879 PMCID: PMC5237332 DOI: 10.1186/s12909-016-0850-7] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/22/2016] [Accepted: 12/20/2016] [Indexed: 05/24/2023]
Abstract
BACKGROUND We developed two objective structured clinical examinations (OSCEs) to educate and evaluate trainees in the evaluation and management of shoulder and knee pain. Our objective was to examine the evidence for validity of these OSCEs. METHODS A multidisciplinary team of content experts developed checklists of exam maneuvers and criteria to guide rater observations. Content was proposed by faculty, supplemented by literature review, and finalized using a Delphi process. One faculty simulated the patient, another rated examinee performance. Two faculty independently rated a portion of cases. Percent agreement was calculated and Cohen's kappa corrected for chance agreement on binary outcomes. Examinees' self-assessment was explored by written surveys. Responses were stratified into 3 categories and compared with similarly stratified OSCE scores using Pearson's coefficient. RESULTS A multi-disciplinary cohort of 69 examinees participated. Examinees correctly identified rotator cuff and meniscal disease 88% and 89% of the time, respectively. Inter-rater agreement was moderate for the knee (87%; k = 0.61) and near perfect for the shoulder (97%; k = 0.88). No correlation between stratified self-assessment and OSCE scores were found for either shoulder (0.02) or knee (-0.07). CONCLUSIONS Validity evidence supports the continuing use of these OSCEs in educational programs addressing the evaluation and management of shoulder and knee pain. Evidence for validity includes the systematic development of content, rigorous control of the response process, and demonstration of acceptable interrater agreement. Lack of correlation with self-assessment suggests that these OSCEs measure a construct different from learners' self-confidence.
Collapse
Affiliation(s)
- Michael J. Battistone
- Salt Lake City Veterans Affairs Medical Center (SLC VAMC), 500 Foothill Drive, Salt Lake City, UT 84148 USA
- Department of Medicine, Division of Rheumatology, University of Utah, Salt Lake City, USA
| | - Andrea M. Barker
- Salt Lake City Veterans Affairs Medical Center (SLC VAMC), 500 Foothill Drive, Salt Lake City, UT 84148 USA
- Department of Family and Preventive Medicine, University of Utah, Salt Lake City, USA
| | - J. Peter Beck
- Salt Lake City Veterans Affairs Medical Center (SLC VAMC), 500 Foothill Drive, Salt Lake City, UT 84148 USA
- Department of Orthopaedics, University of Utah, Salt Lake City, USA
| | - Robert Z. Tashjian
- Salt Lake City Veterans Affairs Medical Center (SLC VAMC), 500 Foothill Drive, Salt Lake City, UT 84148 USA
- Department of Orthopaedics, University of Utah, Salt Lake City, USA
| | - Grant W. Cannon
- Salt Lake City Veterans Affairs Medical Center (SLC VAMC), 500 Foothill Drive, Salt Lake City, UT 84148 USA
- Department of Medicine, Division of Rheumatology, University of Utah, Salt Lake City, USA
| |
Collapse
|
8
|
Briggs AM, Jordan JE, Ackerman IN, Van Doornum S. Establishing cross-discipline consensus on contraception, pregnancy and breast feeding-related educational messages and clinical practices to support women with rheumatoid arthritis: an Australian Delphi study. BMJ Open 2016; 6:e012139. [PMID: 27633637 PMCID: PMC5030591 DOI: 10.1136/bmjopen-2016-012139] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/11/2022] Open
Abstract
OBJECTIVE Recognising the need for a best-practice and consistent approach in providing care to women with rheumatoid arthritis (RA) in relation to (1) general health, (2) contraception, (3) conception and pregnancy, (4) breast feeding and (5) early parenting, we sought to achieve cross-discipline, clinical consensus on key messages and clinical practice behaviours in these 5 areas. DESIGN 3-round eDelphi study. In round 1, panellists provided free-text responses to open-ended questions about care for women with RA across the 5 areas. Subsequently, panellists refined and scored the synthesised responses, presented as metathemes, themes and detailed elements. Where ≥5% of panellists did not support a theme in a given round, it was removed. SETTING Panel of practicing Australian rheumatologists (n=22), obstetricians/obstetric medicine physicians (n=9) and pharmacists (n=5). RESULTS 34 (94.4%) panellists participated in all 3 rounds. The panel supported 18 themes across the 5 areas (support/strongly support: 88.2-100%) underpinned by 5 metathemes. Metathemes focused on coordination in information delivery, the mode and timing of information delivery, evidence underpinning information, engagement of the right health professionals at the right time and a non-judgemental approach to infant feeding. Themes included practices for primary prevention of chronic disease and their sequelae, the importance of contraception and planning pregnancy and breast feeding, close monitoring of medications, supporting mental well-being, managing disease activity and providing practical support for early parenting. CONCLUSIONS A cross-disciplinary clinical panel highly supported key information and clinical practices in the care for women with RA across the continuum of contraception to early parenting within a whole-person, chronic disease management approach.
Collapse
Affiliation(s)
- Andrew M Briggs
- School of Physiotherapy and Exercise Science, Curtin University, Perth, Western Australia, Australia
- Move: Muscle, Bone and Joint Health, Melbourne, Victoria, Australia
| | | | - Ilana N Ackerman
- Department of Epidemiology and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
- Department of Medicine, Melbourne EpiCentre, University of Melbourne, Royal Melbourne Hospital, Melbourne, Victoria, Australia
| | - Sharon Van Doornum
- Department of Medicine, Melbourne EpiCentre, University of Melbourne, Royal Melbourne Hospital, Melbourne, Victoria, Australia
- Department of Rheumatology, Royal Melbourne Hospital, Melbourne, Victoria, Australia
| |
Collapse
|