Effect of online formative assessment on summative performance in integrated musculoskeletal system module

Assessment in Undergraduate Competency-Based Medical Education: A Systematic Review

BACKGROUND

Studies that have methodically compiled the body of research on the competency-based medical education (CBME) assessment procedure and pinpointed knowledge gaps about the structure of the assessment process are few. Thus, the goals of the study were to create a model assessment framework for competency-based medical education that would be applicable in the Indian setting as well as to thoroughly examine the competency-based medical education assessment framework.

METHODS

PubMed, MEDLINE (Ovid), EMBASE (Ovid), Scopus, Web of Science, and Google Scholar were the databases that were searched. The search parameters were restricted to English language publications about competency-based education and assessment methods, which were published between January 2006 and December 2020. A descriptive overview of the included research (in tabular form) served as the foundation for the data synthesis.

RESULTS

Databases provided 732 records; out of which 36 fulfilled the inclusion and exclusion criteria. Thirty-six studies comprised a mix of randomized controlled trials, focus group interviews, and questionnaire studies, including cross-sectional studies, qualitative studies (03), mixed-method studies, etc. The papers were published in 10 different journals. The greatest number was published in BMC Medical Education (18). The average quality score for included studies was 62.53% (range: 35.71-83.33%). Most authors are from the UK (07), followed by the USA (05). The included studies were grouped into seven categories based on their dominant focus: moving away from a behavioristic approach to a constructive approach of assessment (01 studies), formative assessment (FA) and feedback (10 studies), the hurdles in the implementation of feedback (04 studies), utilization of computer or online based formative test with automated feedback (05 studies), video feedback (02 studies), e-learning platforms for formative assessment (04 studies), studies related to workplace-based assessment (WBA)/mini-clinical evaluation exercise (mini-CEX)/direct observation of procedural skills (DOPS) (10 studies).

CONCLUSIONS

Various constructivist techniques, such as concept maps, portfolios, and rubrics, can be used for assessments. Self-regulated learning, peer feedback, online formative assessment, an online computer-based formative test with automated feedback, the use of a computerized web-based objective structured clinical examination (OSCE) evaluation system, and the use of narrative feedback instead of numerical scores in mini-CEX are all ways to increase student involvement in the design and implementation of the formative assessment.

The Role of E-Content Development in Medical Teaching: How Far Have We Come?

With the advancements in technology, medical educators are now able to create and deliver content to students through digital platforms. Electronic content (e-content) development has allowed educators to incorporate multimedia, animations, simulations, and interactive elements which support verbal instruction, such as improved expression and comprehension, into their teaching materials. E-content development is a relatively new field, but it is growing very rapidly. Recent findings have indicated that the e-learning sector will likely experience a huge surge in the upcoming years. The Indian government has launched various initiatives for e-content development in medical education. E-content development has great potential and can be used in various learning scenarios. While it initially gained popularity in higher education, it has since been applied to many other sectors, including healthcare. It allows educators to create highly engaging learning experiences that are accessible by all students. Challenges in e-content development include availability of the internet, creating content that is engaging and relevant to a wide range of learners, and access. Still, it is expected that the use of e-content in medical teaching will continue to increase in the future. The future of e-content development in medical teaching is likely to see continued growth and innovation as technology advances and more educators and learners recognize the benefits of online and digital resources.

Evaluating TESLA-G, a gamified, telegram-delivered, quizzing platform for surgical education in medical students: protocol for a pilot randomised controlled trial

INTRODUCTION

Online multiple-choice question (MCQ) quizzes are popular in medical education due to their ease of access and ability for test-enhanced learning. However, a general lack of motivation among students often results in decreasing usage over time. We aim to address this limitation by developing Telegram Education for Surgical Learning and Application Gamified (TESLA-G), an online platform for surgical education that incorporates game elements into conventional MCQ quizzes.

METHODS AND ANALYSIS

This online, pilot randomised control trial will be conducted over 2 weeks. Fifty full-time undergraduate medical students from a medical school in Singapore will be recruited and randomised into an intervention group (TESLA-G) and an active control group (non-gamified quizzing platform) with a 1:1 allocation ratio, stratified by year of study.We will evaluate TESLA-G in the area of endocrine surgery education. Our platform is designed based on Bloom's taxonomy of learning domains: questions are created in blocks of five questions per endocrine surgery topic, with each question corresponding to one level on Bloom's taxonomy. This structure promotes mastery while boosting student engagement and motivation. All questions are created by two board-certified general surgeons and one endocrinologist, and validated by the research team. The feasibility of this pilot study will be determined quantitatively by participant enrolment, participant retention and degree of completion of the quizzes. The acceptability of the intervention will be assessed quantitatively by a postintervention learner satisfaction survey consisting of a system satisfaction questionnaire and a content satisfaction questionnaire. The improvement of surgical knowledge will be assessed by comparing the scores of preintervention and postintervention knowledge tests, which consist of separately created questions on endocrine surgery. Retention of surgical knowledge will be measured using a follow-up knowledge test administered 2 weeks postintervention. Finally, qualitative feedback from participants regarding their experience will be obtained and thematically analysed.

ETHICS AND DISSEMINATION

This research is approved by Singapore Nanyang Technological University (NTU) Institutional Review Boards (Reference Number: IRB-2021-732). All participants will be expected to read and sign a letter of informed consent before they are considered as recruited into the study. This study poses minimal risk to participants. Study results will be published in peer-reviewed open-access journals and presented in conference presentations.

TRIAL REGISTRATION NUMBER

NCT05520671.

Effective Feedback Strategy for Formative Assessment in an Integrated Medical Neuroscience Course

Purpose

Despite the different benefits of formative assessments in an integrated medical curriculum, the effective strategies to provide feedback to medical students to benefit from the different merits of formative assessment are not fully understood. This study aims to determine the effect of different strategies of formative feedback on students' outcomes in a medical neuroscience course.

Method

We compared medical students' performance in summative examinations in the academic year that formative feedback was provided using in-person discussion and compared such performances with the academic year when the feedback was provided by written rationales or a combination of written rationales and in-person discussion. We also surveyed medical students' preferences for whether written or in-person formative feedback is a better strategy to provide feedback at the end of each course.

Results

ANOVA found a significant difference in summative performance scores for those scoring ≥ 70% when formative feedback was provided by providing a rationale, in-person, and a combination of both ([F (2,80) = 247.60, P < 0.001]. Post hoc analysis revealed a significant and highest performance when feedback was provided using the written rationale approach (***P < 0.05), followed by in-person (**P < 0.05). In contrast, the least performance was recorded when formative feedback was provided using a combination of providing a written rationale for the answers to the questions and in-person discussion of the questions (*P < 0.05). Students' preferred approach for receiving formative feedback for their formative assessment was highest for written rationale (***P < 0.05), followed by in-person or a combination of in-person and written rationale (**P < 0.05).

Conclusion

Our results found that medical students preferred a written formative feedback approach, which was associated with better student performance on the summative examination. This study reveals the importance of developing effective strategies to provide formative feedback to medical students for medical students to fully benefit from the merits of formative assessment in an integrated medical school curriculum.

Musculoskeletal education in undergraduate medical curricula-A systematic review

Rheumatic and musculoskeletal diseases encompass a vast spectrum of up to 200 conditions that are increasingly prevalent, with significant associated disability and socioeconomic burden. Their impact is pervasive, with musculoskeletal conditions being the second leading cause of years lived with disability worldwide, in addition to the 9th most common cause of disability-adjusted life years. It is therefore imperative that all graduating medical physicians are competent in their management, and that the quality of undergraduate musculoskeletal education is commensurate with patient and societal needs. A systematic literature review was conducted between April 1, 2021 and June 1, 2021 assessing the quality of undergraduate musculoskeletal education in medical schools. Educational interventions in musculoskeletal medicine were also included. Quality assessment appraisal of the studies was done using a Mixed Methods Appraisal Tool. One thousand and thirty-three titles were screened, and 44 studies were included in the final analysis. Our analysis of these studies showed that the quality of undergraduate musculoskeletal education, as determined by the cognitive mastery and clinical confidence of undergraduate medical students remains inadequate. Multiple educational interventions were assessed with mixed results. Despite the prevalence, and burden associated with rheumatic and musculoskeletal diseases, the musculoskeletal education of undergraduate medical students remains inadequate. Urgent international collaboration is required to devise teaching strategies and curriculum initiatives that are globally and reproducibly applicable and effective. Further research into educational interventions and teaching strategies is also required.

A structured multimodal teaching approach enhancing musculoskeletal physical examination skills among undergraduate medical students

Current evidence indicates that undergraduate medical students display deficits in musculoskeletal physical examination skills (MPES). While various instructional methods are recommended for teaching clinical skills, effective methods for teaching MPES have not been established. This study compared the effectiveness of a multimodal teaching approach incorporating video-based learning, interactive small-group teaching, hands-on practicing, peer-assisted learning, formative assessment, and constructive feedback with traditional bedside teaching in developing undergraduate orthopedic MPES. Participants were 151 fifth-year medical students divided into two groups. One group received multimodal teaching, and the other received traditional bedside teaching. In both groups, the participants learned how to physically examine the knee and shoulder. The primary outcome was objective structured clinical examination (OSCE) scores, while the secondary outcomes included teaching sessions' total durations, facilitator's demonstration time, participants' practice time, and proportion of students with passing checklist scores and global ratings-based assessments for the two teaching approaches. The multimodal teaching group had significantly higher OSCE scores (checklist scores, global ratings, and passing rates; p = 0.02, 0.02, 0.01, respectively) than the comparison group. Individual OSCE component assessments showed significant improvements in the special musculoskeletal physical examination test. The overall duration and amount of participants' hands-on time were significantly longer for the multimodal than for the traditional bedside teaching group (p = 0.01 and 0.01, respectively), and the facilitator's demonstration time was significantly shorter (p = 0.01). The multimodal learner-centered teaching approach evaluated in this study was effective for teaching MPES. It appeared to maximize learner engagement through enhancing interactions and providing increased time to engage in hands-on practice. This teaching approach improved MPES levels, maximized teaching efficiency for scenarios with limited instruction time and resources, and enhanced competency of undergraduate medical students in performing special musculoskeletal physical examinations compared to traditional bedside teaching.

The Practice of Feedback in Health Professions Education in the Hierarchical and Collectivistic Culture: a Scoping Review

The global trend in student assessment is moving towards outcome-based assessment that requires multiple systematic data points and continuous feedback. However, in hierarchical and collectivistic cultures, one-way communication is practised more often, leading to less dialogical feedback. This scoping review explored feedback practice in Asian educational setting. Based on the 17 articles selected, the findings were categorised into four themes, i.e. inhibiting or facilitating factors of feedback, influences of cultural factors on feedback, discrepancies between students' and teachers' perceptions of feedback and impact of feedback. Hierarchical and collectivist cultural aspects, such as preference for group feedback, are pertinent to feedback practice, which likely influence the readiness for programmatic assessment.

Efficacy of an intensive 2-week musculoskeletal anatomy course in a medical education setting

There is little consistency in student contact hours and teaching activities used in medical curriculums across North America, and controversy exists regarding the most efficient or effective way to teach musculoskeletal (MSK) anatomy in medical school. The purpose of this investigation was to retrospectively examine academic performance and levels of course satisfaction among first year medical students who had completed an intensive 2-week MSK anatomy course during the pre-clerkship portion of their Association of Faculties of Medicine in Canada accredited undergraduate medical program. Course data from formative and summative student evaluations, as well as end-of-course satisfaction surveys were analyzed for a homogenous group of 549 first-year medical students over a 5-year period. Data indicated that student's academic performance on both formative and summative evaluations for the 2-week MSK anatomy course were consistently high, and not significantly different from the results for other system-based 1^st year anatomy courses delivered over longer durations. Likert data from end-of-course surveys also suggested that student feedback regarding the course was overwhelmingly positive, and that the 2-week MSK anatomy course had the highest level of overall satisfaction when compared to all other first-year anatomy courses. These results can be used to help in the design and implementation of MSK anatomy courses within medical curriculums, and contribute to a growing body of literature which is focused on examining the effect of curricular design on MSK anatomy learning in a medical education setting.

Web-based formative assessment through clinical cases: role in pathophysiology teaching

BACKGROUND

Active learning strategies such as formative assessment through clinical cases may help to get a deeper learning. We have studied the effect of this kind of online formative assessment in pathophysiology teaching.

METHODS

Seven brief clinical cases were used to give formative assessment in the first semester of a pathophysiology course. To evaluate its effect on learning, we analyzed the proportion of students that passed the end of semester exam with a score above 60 over 100. We also analyzed the effect of the intervention according to the students' previous academic performance.

RESULTS

Ninety-six students participated in the study and sat the exam. Sixty-five of them passed it. Students that passed the exam had a higher previous academic performance and had done a higher number of exercises of formative assessment, both in univariate and multivariate analysis. The participants were divided in three groups, according to their previous academic performance. In the intermediate group, the number of cases done by the students who passed the exam was significantly higher than in those who did not pass it (median: 4 versus 0; P = 0.009).

CONCLUSION

Formative assessment through web-based clinical cases was followed by an improvement of the academic results in pathophysiology, mainly in students with intermediate performance.

Use of an Adaptive e-Learning Platform as a Formative Assessment Tool in the Cardiovascular System Course Component of an MBBS Programme

BACKGROUND

Technology-enhanced learning includes the adaptive e-learning platform, a data-driven method with computer algorithms, providing customised learning enhancing critical thinking of individual learners. "Firecracker" - an online adaptive e-learning platform, and assessment software, promotes critical thinking, helps prepare students for courses and high-stakes examinations, and evaluates progress relative to co-learners. The objectives of this study were to determine the usage rates of Firecracker, examine the performance of Firecracker formative quizzes, identify the correlation between Firecracker use and performance with that of performance at summative course assessments, and assess students' satisfaction with Firecracker usage.

METHODS

Study participants were Year-2 MBBS (Bachelor of Medicine, Bachelor of Surgery) students (n=91) of the Faculty of Medical Sciences, The University of the West Indies, Cave Hill Campus, Barbados. The Firecracker Administrator uploaded quizzes covering basic science content in the Cardiovascular System course. Access, usage, and performance on Firecracker formative quizzes were retrieved from the Firecracker dashboard. A questionnaire sought the views of study participants.

RESULTS

Seven sets of quizzes were administered over nine weeks, with weekly student completion rates ranging from 53% to 73%. Mean quiz scores ranged from 52% to 72%. Students completing >4 quiz sessions compared to those completing ≤4 demonstrated significantly better performance in Firecracker quizzes (P<0.01), final examinations (P<0.01) and in-course assessment plus final examination (P<0.05) scores. Correlations between overall Firecracker performance and in-course assessment marks (P<0.05); between total overall Firecracker performance and final examination (P<0.01); and overall Firecracker performance and total course marks (P<0.01) were all significant. Most students (70%) were happy using Firecracker and felt it complemented coursework (78%) and prepared them for course exams (58%) (P<0.01).

CONCLUSION

Overall, Firecracker was perceived very positively and welcomed by the students. Students were satisfied with the Firecracker as a formative assessment tool, and its use correlated with improved performance in the course examinations.

Do students achieve the desired learning goals using open-book formative assessments?

OBJECTIVES

The present study aimed to examine whether medical students benefit from an open-book online formative assessment as a preparation for a practical course.

METHODS

A between-subjects experimental design was used: participants - a whole cohort of second-year medical students (N=232) - were randomly assigned to either a formative assessment that covered the topic of a subsequent practical course (treatment condition) or a formative assessment that did not cover the topic of the subsequent course (control condition). Course-script-knowledge, as well as additional in-depth-knowledge, was assessed.

RESULTS

Students in the treatment condition had better course-script knowledge, both at the beginning, t₍₂₁₂₎ = 4.96, p < .01, d = 0.72., and in the end of the practical course , t₍₂₀₈₎ = 4.80, p < .01, d = 0.68. Analyses of covariance show that this effect is stronger for those students who understood the feedback that was presented within the formative assessment, F_{(1, 213)}=10.17, p<.01. Additionally, the gain of in-depth-knowledge was significantly higher for students in the treatment condition compared to students in the control condition, t₍₂₀₈₎ = 3.68., p < .05, d = 0.72 (0.51).

CONCLUSIONS

Students benefit from a formative assessment that is related to and takes place before a subsequent practical course. They have a better understanding of the topic and gain more in-depth-knowledge that goes beyond the content of the script. Moreover, the study points out the importance of feedback pages in formative assessments.

Twelve tips for efficient procedural simulation

Procedural simulation (PS) is increasingly being used worldwide in healthcare for training caregivers in psychomotor competencies. It has been demonstrated to improve learners' confidence and competence in technical procedures, with consequent positive impacts on patient outcomes and safety. Several frameworks can guide healthcare educators in using PS as an educational tool. However, no theory-informed practical framework exists to guide them in including PS in their training programs. We present 12 practical tips for efficient PS training that translates educational concepts from theory to practice, based on the existing literature. In doing this, we aim to help healthcare educators to adequately incorporate and use PS both for optimal learning and for transfer into professional practice.

Use of planning metrics software for automated feedback to radiotherapy students

Background and purpose

Pre-registration teaching of radiotherapy planning in a non-clinical setting should allow students the opportunity to develop clinical decision-making skills. Students frequently struggle with their ability to prioritise and optimise multiple objectives when producing a clinically acceptable plan. Emerging software applications providing quantitative assessment of plan quality are designed for clinical use but may have value for teaching these skills. This project aimed to evaluate the potential value of automated feedback to second year BSc (Hons) Radiotherapy students.

Materials and methods

All 26 students studying a pre-registration radiotherapy planning module were provided with automated prediction of relative feasibility for left lung tumour planning targets by planning metrics software. Students were also provided with interim quantitative reports during the development of their plan. Student perceptions of the software were gathered using an anonymous questionnaire. Independent blinded marking of plans was performed after module completion and analysed for correlation with software-assigned marks.

Results

In total, 25 plans were utilised for marking comparison and 16 students submitted feedback relating to the software. Overall, student feedback was positive regarding the software. A ‘strong’ Spearman’s rank-order correlation (rs=0·7165) was evident between human and computer marks (p=0·000055).

Conclusions

Automated software is capable of providing useful feedback to students as a teaching aid, in particular with regard to relative feasibility of goals. The strong correlation between human and computer marks suggests a role in benchmarking or moderation; however, the narrow scope of assessment parameters suggests value as an adjunct and not a replacement to human marking.