1
|
Rosenberg I, Thomas L, Ceccolini G, Feinn R. 'Early identification of struggling pre-clerkship learners using formative clinical skills OSCEs: an assessment for learning program.'. MEDICAL EDUCATION ONLINE 2022; 27:2028333. [PMID: 35048773 PMCID: PMC8786239 DOI: 10.1080/10872981.2022.2028333] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/17/2021] [Revised: 11/01/2021] [Accepted: 01/08/2022] [Indexed: 06/14/2023]
Abstract
Multiple experts in clinical skills remediation recommend early identification to support struggling learners, but there is minimal documentation on implementation of these programs. We share one school's outcomes-based research utilizing the formative assessment for learning model to early-identify pre-clerkship students struggling with clinical skills using formative OSCEs (F-OSCE). Student scores were monitored over longitudinal F-OSCE experiences as part of a curricular innovation. Points towards early identification accumulated when a student's score fell below the 80% threshold for each section of an OSCE. Students who accumulated enough points were advised of the need for intervention, and coaching was recommended. Students were surveyed about their experiences with the program. The objective was to explore whether this early identification program and coaching intervention had a positive impact on subsequent OSCE performance. Of 184 students in 2 cohorts who completed F-OSCEs, 38 (20.7%) were flagged for early identification. Of these, 17 (44.7%) sought additional help by voluntarily participating in the coaching program. Students who participated in extra clinical skills coaching demonstrated statistically significant improvements in performance on subsequent FOSCEs, as did the early identified students who did not participate in extra coaching. The greatest impact of coaching intervention was noted in the physical examination domain. This program was effective in identifying students struggling with clinical skills on formative OSCEs. Early identified students demonstrated improvements in subsequent OSCE performance, with those who sought coaching faring slightly better. Development of robust early identification programs as formative assessments of clinical skills and follow-up coaching programs to guide skills development are important implications of this work. Monitoring short- and long-term results for students identified through this approach to see if improvement is sustained is planned.
Collapse
Affiliation(s)
- Ilene Rosenberg
- Clinical Skills Remediation, Department of Medical Sciences, Frank H. Netter Md School of Medicine at Quinnipiac University, Hamden, CT, USA
| | - Listy Thomas
- Department of Medical Sciences, Clinical Arts and Sciences Course, the Frank H. Netter Md School of Medicine at Quinnipiac University, Hamden, CT, USA
| | - Gabbriel Ceccolini
- Standardized Patient & Assessment Center, Frank H. Netter Md School of Medicine, at Quinnipiac University, Hamden, CT, USA
| | - Richard Feinn
- Department of Medical Sciences, Frank H Netter Md School of Medicine, at Quinnipiac University, Hamden, CT, USA
| |
Collapse
|
2
|
Wu JW, Cheng HM, Huang SS, Liang JF, Huang CC, Shulruf B, Yang YY, Chen CH, Hou MC, Huey-Herng Sheu W. Medical school grades may predict future clinical competence. J Chin Med Assoc 2022; 85:909-914. [PMID: 36150103 DOI: 10.1097/jcma.0000000000000782] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/26/2022] Open
Abstract
BACKGROUND In real-world medical education, there is a lack of reliable predictors of future clinical competencies. Hence, we aim to identify the factors associated with clinical competencies and construct a prediction model to identify "improvement required" trainees. METHODS We analyzed data from medical students who graduated from National Yang-Ming University with clerkship training and participated in the postgraduate year (PGY) interview at Taipei Veterans General Hospital. Clinical competencies were evaluated using grades of national objective structured clinical examination (OSCEs). This study used data from medical students who graduated in July 2018 as the derivation cohort (N = 50) and those who graduated in July 2020 (n = 56) for validation. RESULTS Medical school grades were associated with the performance of national OSCEs (Pearson r = 0.34, p = 0.017), but the grades of the structured PGY interviews were marginally associated with the national OSCE (Pearson r = 0.268, p = 0.06). A prediction model was constructed to identify "improvement required" trainees, defined: trainees with the lowest 25% of scores in the national OSCEs. According to this model, trainees with the lowest 25% medical school grades predicted a higher risk of the "improvement required" clinical performance (Q1-Q3 vs Q4 = 15% vs 60%, odds ratio = 8.5 [95% confidence interval = 1.8-39.4], p = 0.029). In the validation cohort, our prediction model could accurately classify 76.7% "improvement required" and "nonimprovement required" students. CONCLUSION Our study suggests that interventions for students with unsatisfactory medical school grades are warranted to improve their clinical competencies.
Collapse
Affiliation(s)
- Jr-Wei Wu
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan, ROC
- College of Medicine, National Yang Ming Tung University, Taipei, Taiwan, ROC
- Clinical Innovation Center, Taipei Veterans General Hospital, Taipei, Taiwan, ROC
| | - Hao-Min Cheng
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan, ROC
- College of Medicine, National Yang Ming Tung University, Taipei, Taiwan, ROC
- Center for Evidence-based Medicine, Taipei Veterans General Hospital, ROC
| | - Shiau-Shian Huang
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan, ROC
- College of Medicine, National Yang Ming Tung University, Taipei, Taiwan, ROC
| | - Jen-Feng Liang
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan, ROC
- College of Medicine, National Yang Ming Tung University, Taipei, Taiwan, ROC
| | - Chia-Chang Huang
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan, ROC
- College of Medicine, National Yang Ming Tung University, Taipei, Taiwan, ROC
- Division of Clinical Skills Training Center, Taipei Veterans General Hospital, Taipei, Taiwan, ROC
| | - Boaz Shulruf
- University of New South Wales, Sydney, Australia
| | - Ying-Ying Yang
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan, ROC
- College of Medicine, National Yang Ming Tung University, Taipei, Taiwan, ROC
- Clinical Innovation Center, Taipei Veterans General Hospital, Taipei, Taiwan, ROC
- Division of Clinical Skills Training Center, Taipei Veterans General Hospital, Taipei, Taiwan, ROC
| | - Chen-Huan Chen
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan, ROC
- College of Medicine, National Yang Ming Tung University, Taipei, Taiwan, ROC
| | - Ming-Chih Hou
- College of Medicine, National Yang Ming Tung University, Taipei, Taiwan, ROC
- Division of Gastroenterology and Hepatology, Department of Medicine, Taipei Veterans General Hospital, Taipei, ROC
| | - Wayne Huey-Herng Sheu
- College of Medicine, National Yang Ming Tung University, Taipei, Taiwan, ROC
- Section of Endocrinology and Metabolism, Department of Medicine, Taipei Veterans General Hospital, Taipei, Taiwan, ROC
- Institute of Medical Technology, College of Life Science, National Chung-Hsing University, Taichung, Taiwan, ROC
| |
Collapse
|
3
|
Roslan MHB, Chen CJ. Predicting students' performance in English and Mathematics using data mining techniques. EDUCATION AND INFORMATION TECHNOLOGIES 2022; 28:1427-1453. [PMID: 35919875 PMCID: PMC9334550 DOI: 10.1007/s10639-022-11259-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/26/2022] [Accepted: 07/20/2022] [Indexed: 06/15/2023]
Abstract
This study attempts to predict secondary school students' performance in English and Mathematics subjects using data mining (DM) techniques. It aims to provide insights into predictors of students' performance in English and Mathematics, characteristics of students with different levels of performance, the most effective DM technique for students' performance prediction, and the relationship between these two subjects. The study employed the archival data of students who were 16 years old in 2019 and sat for the Malaysian Certificate of Examination (MCE) in 2021. The learning of English and Mathematics is a concern in many countries. Three main factors, namely students' past academic performance, demographics, and psychological attributes were scrutinized to identify their impact on the prediction. This study utilized the Orange software for the DM process. It employed Decision Tree (DT) rules to determine the characteristics of students with low, moderate, and high performance in English and Mathematics subjects. DT and Naïve Bayes (NB) techniques show the best predictive performance for English and Mathematics subjects, respectively. Such characteristics and predictions may cue appropriate interventions to improve students' performance in these subjects. This study revealed students' past academic performance as the most critical predictor, as well as a few demographics and psychological attributes. By examining top predictors derived using four different classifier types, this study found that students' past Mathematics performance predicts their MCE English performance and students' past English performance predicts their MCE Mathematics performance. This finding shows students' performances in both subjects are interrelated.
Collapse
Affiliation(s)
- Muhammad Haziq Bin Roslan
- Faculty of Cognitive Sciences and Human Development, Universiti Malaysia Sarawak, 94300 Kota Samarahan, Malaysia
| | - Chwen Jen Chen
- Faculty of Cognitive Sciences and Human Development, Universiti Malaysia Sarawak, 94300 Kota Samarahan, Malaysia
| |
Collapse
|
4
|
Desy J, Coderre S, Veale P, Busche K, Woloschuk W, McLaughlin K. Can we predict failure in licensure exams from medical students' undergraduate academic performance? CANADIAN MEDICAL EDUCATION JOURNAL 2021; 12:6-13. [PMID: 35003426 PMCID: PMC8740250 DOI: 10.36834/cmej.68172] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
BACKGROUND In 2015, the Medical Council of Canada increased the minimum pass level for the Medical Council of Canada Qualifying Examination Part I, and students had a higher rate of failure than in previous years. The purpose of this study was to predict students at an increased odds of examination failure to allow for early, targeted interventions. METHODS We divided our dataset into a derivation cohort and two validation cohorts and used multiple logistic regression to predict licensing examination failure. We then performed receiver operating characteristics and a sensitivity analysis using different cutoffs for explanatory variables to identify the cutoff threshold with the best predictive value at identifying students at increased odds of failure. RESULTS After multivariate analysis, only pre-clerkship GPA was a significant independent predictor of failure (OR 0.76, 95% CI [0.66, 0.88], p < 0.001). The probability of failure increased steeply when the pre-clerkship GPA fell below 80% and 76% was found to be the most efficient cutoff for predicting failure (OR 9.37, 95% CI [3.08, 38.41]). CONCLUSIONS Pre-clerkship performance can predict students at increased odds of licensing examination failure. Further studies are needed to explore whether early interventions for at-risk students alter their examination performance.
Collapse
Affiliation(s)
- Janeve Desy
- Office of Undergraduate Medical Education, Cumming School of Medicine, University of Calgary, Alberta, Canada
| | - Sylvain Coderre
- Office of Undergraduate Medical Education, Cumming School of Medicine, University of Calgary, Alberta, Canada
| | - Pamela Veale
- Office of Undergraduate Medical Education, Cumming School of Medicine, University of Calgary, Alberta, Canada
| | - Kevin Busche
- Office of Undergraduate Medical Education, Cumming School of Medicine, University of Calgary, Alberta, Canada
| | - Wayne Woloschuk
- Office of Undergraduate Medical Education, Cumming School of Medicine, University of Calgary, Alberta, Canada
| | - Kevin McLaughlin
- Office of Undergraduate Medical Education, Cumming School of Medicine, University of Calgary, Alberta, Canada
| |
Collapse
|
5
|
Rashid H, Coppola KM, Lebeau R. Three Decades Later: A Scoping Review of the Literature Related to the United States Medical Licensing Examination. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2020; 95:S114-S121. [PMID: 33105189 DOI: 10.1097/acm.0000000000003639] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
Abstract
PURPOSE To conduct a scoping review of the timing, scope, and purpose of literature related to the United States Medical Licensing Examination (USMLE) given the recent impetus to revise USMLE scoring. METHOD The authors searched PubMed, PsycInfo, and ERIC for relevant articles published from 1990 to 2019. Articles selected for review were labeled as research or commentaries and coded by USMLE Step level, sample characteristics (e.g., year(s), single/multiple institutions), how scores were used (e.g., predictor/outcome/descriptor), and purpose (e.g., clarification/justification/description). RESULTS Of the 741 articles meeting inclusion criteria, 636 were research and 105 were commentaries. Publication totals in the past 5 years exceeded those of the first 20 years.Step 1 was the sole focus of 38%, and included in 84%, of all publications. Approximately half of all research articles used scores as a predictor or outcome measure related to other curricular/assessment efforts, with a marked increase in the use of scores as predictors in the past 10 years. The overwhelming majority of studies were classified as descriptive in purpose. CONCLUSIONS Nearly 30 years after the inception of the USMLE, aspirations for its predictive utility are rising faster than evidence supporting the manner in which the scores are used. A closer look is warranted to systematically review and analyze the contexts and purposes for which USMLE scores can productively be used. Future research should explore cognitive and noncognitive factors that can be used in conjunction with constrained use of USMLE results to inform evaluation of medical students and schools and to support the residency selection process.
Collapse
Affiliation(s)
- Hanin Rashid
- H. Rashid is associate director, Office for Advancing Learning, Teaching, and Assessment, and assistant professor, Cognitive Skills Program, Rutgers Robert Wood Johnson Medical School, Piscataway, New Jersey
| | - Kristen M Coppola
- K.M. Coppola is assistant professor, Cognitive Skills Program, Rutgers Robert Wood Johnson Medical School, Piscataway, New Jersey
| | - Robert Lebeau
- R. Lebeau is director, Office for Advancing Learning, Teaching, and Assessment, and Cognitive Skills Program, Rutgers Robert Wood Johnson Medical School, Piscataway, New Jersey
| |
Collapse
|
6
|
Kam J, Khadra S, Tran QH, Ainsworth H, Louie-Johnsun M, Winter M. Portable Video Media Versus Standard Verbal Communication in Surgical Teaching: A Prospective, Multicenter, and Randomized Controlled Crossover Trial. JOURNAL OF SURGICAL EDUCATION 2019; 76:440-445. [PMID: 30253985 DOI: 10.1016/j.jsurg.2018.08.013] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/03/2018] [Revised: 08/02/2018] [Accepted: 08/15/2018] [Indexed: 06/08/2023]
Abstract
OBJECTIVE This study aims to evaluate the effectiveness of portable video media (PVM) compared to standard verbal communication (SVC) as a novel adjunct for surgical education of junior medical officers and medical students, in terms of knowledge acquisition and participant satisfaction. BACKGROUND The effective, continued education of final year medical students and junior doctors is the foundation of quality healthcare. The development of new media technologies and rapid internet streaming has resulted in an opportunity for the integration of PVM into medical education. PVM is an educational platform with the advantage of being standardized, efficient, and readily available. DESIGN This is a multicenter, prospective, and randomized controlled crossover study. Participants completed a preintervention knowledge test and were then randomized in an allocation ratio of 1:1 to receive surgical education regarding cystoscopy and ureteric stenting for acute renal colic via either PVM or SVC. A 32-point knowledge test and a modified Client Satisfaction Questionnaire-8 were then administered and the participants were then crossed over to the other educational method. The knowledge and satisfaction tests were then readministered. RESULTS Fifty-four participants were recruited for this study with 27 participants in each group. Both groups had a 18% to 20% increase in knowledge scores following the first intervention p < 0.001 and on crossover there was a further 4% increase in knowledge scores, p < 0.01. There was no significant difference between the groups in knowledge scores before intervention, p = 0.23 after first intervention p = 0.74 or following crossover p = 0.09. After first intervention, participants in the group receiving PVM education first had a significant 8% higher satisfaction score compared to the SVC group, p = 0.023. CONCLUSIONS Our study has shown that PVM shows similar efficacy in information uptake to traditional forms of education. Furthermore, PVM was shown to have higher satisfaction scores compared to SVC. Further studies will need to evaluate the use of PVM for education in other surgical and medical domains and assess the long-term knowledge retention.
Collapse
Affiliation(s)
- Jonathan Kam
- Nepean Urology Research Group (NURG), Kingswood, NSW, Australia; Faculty of Medicine, University of Newcastle, Callaghan, NSW, Australia.
| | - Sam Khadra
- Nepean Urology Research Group (NURG), Kingswood, NSW, Australia.
| | - Quoc Hoai Tran
- Nepean Urology Research Group (NURG), Kingswood, NSW, Australia.
| | | | - Mark Louie-Johnsun
- Faculty of Medicine, University of Newcastle, Callaghan, NSW, Australia; Gosford District Hospital, Gosford, NSW, Australia.
| | - Matthew Winter
- Nepean Urology Research Group (NURG), Kingswood, NSW, Australia.
| |
Collapse
|
7
|
Yen-Ju Lin B, Liu PC, Ku KT, Lee CC. Adaptation of Medical Students During Clinical Training: Effects of Holistic Preclinical Education on Clerkship Performance. TEACHING AND LEARNING IN MEDICINE 2019; 31:65-75. [PMID: 30526084 DOI: 10.1080/10401334.2018.1489815] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/10/2017] [Revised: 05/15/2018] [Accepted: 05/23/2018] [Indexed: 06/09/2023]
Abstract
THEORY Taiwan's medical undergraduate program at a university or medical center is a continuation of 12 years of compulsory citizenship education rooted in holistic philosophies. Students acquire both technical knowledge and nontechnical attributes, which are necessary for success in further work and life. The early clinical learning experiences of medical students are primarily acquired through clerkships. These clerkships require medical students to apply and extend what they learned during their preclinical education; however, previous studies have explored this issue through examining fragmentary factors such as preclinical course grades and traits but not undertaking comprehensive, whole-person investigations. HYPOTHESES To account for the potential benefits of a holistic approach in medical students' learning, we propose three hypotheses: Medical students' preclinical performance on Taiwan's technical and nontechnical higher education assessments are positively associated with their clinical competence (Hypothesis 1) and psychological well-being (Hypothesis 2) during clerkships, and medical students' psychological well-being during clerkships is positively associated with their clinical competence (Hypothesis 3). METHOD We studied a cohort of 65 medical students engaged in clerkships from September 2013 to April 2015. Their preclinical technical knowledge scores-formal curricular grades received from course instructors-were obtained from their medical school's archival dataset. Their nontechnical attributes-moral and social performance scores received from student mentors and physical performance scores received from course instructors-were also obtained from the school's archival data set. The medical students' competence in their 2-year clinical clerkships was measured using the objective structured clinical examination scores from the end of both clerkship years. The medical students' psychological well-being during their 2-year clerkships was measured according to burnout level, which was determined using routine online surveys that employed validated, structured, and self-administered questionnaires at each specialty rotation. Multiple regressions and linear mixed-effects model were employed for statistical analysis. RESULTS Our study revealed that higher preclinical technical knowledge predicted superior clinical competence and a higher level of burnout during clerkships. By contrast, higher preclinical nontechnical attributes (i.e., higher preclinical moral, social, and physical performance) predicted lower level of burnout. However, no relationship was discovered between clerkship burnout and the clinical competence of the medical students. CONCLUSIONS Our study verified the value of a holistic education that encompasses both technical knowledge and nontechnical attributes during the preclinical learning stage for medical students. Our findings can serve as a reference for medical educators designing preclinical educational programs for medical students.
Collapse
Affiliation(s)
- Blossom Yen-Ju Lin
- a Department of Medical Humanities and Social Sciences , College of Medicine, Chang Gung University , Taoyuan , Taiwan, ROC
- b Gynecologic Cancer Research Center , Chang Gung Memorial Hospital , Taoyuan , Taiwan, ROC
| | - Pei-Chun Liu
- c School of Medicine , China Medical University , Taichung , Taiwan, ROC
| | - Kuo-Tung Ku
- c School of Medicine , China Medical University , Taichung , Taiwan, ROC
| | - Cheng-Chun Lee
- d College of Health Care , China Medical University , Taichung , Taiwan, ROC
| |
Collapse
|