Test-enhanced learning in medical education

Establishing TELLme - an online platform for self-directed, formative assessment of knowledge acquisition across an undergraduate medical curriculum

PURPOSE

The purpose of this study is to report on the design, implementation, use and evaluation of TELLme, an online platform with annotated multiple-choice questions (MCQs) for formative assessment of knowledge acquisition covering an entire undergraduate medical curriculum.

MATERIALS AND METHODS

We used the Educational Design Research (EDR) framework to develop TELLme as an online platform with 24/7 access for students in a co-creation process between educators, faculty and students. EDR cycle 1 focused on prototyping, while EDR cycle 2 focused on upscaling, usage analysis and evaluation. Database entries were analysed for TELLme usage. Online surveys evaluated platform usability and support for student learning.

RESULTS

At the end of EDR cycle 2, TELLme contained 6,713 fully annotated MCQs aligned with the cognitive learning objectives of the curriculum. Up to 600 students per day used TELLme to self-assess their knowledge, answering 3,168,622 MCQs in two semesters. Surveys indicated good usability of TELLme, 75% of students indicated that TELLme helped them to identify knowledge gaps, and 72% agreed that TELLme supported their learning.

CONCLUSIONS

Medical students use TELLme extensively to formatively assess their learning in a self-directed manner across the entire curriculum. TELLme aligns with the existing summative and formative system of assessment of the study programme.

Impact of Simulation-based Mastery Learning on Management of Massive Hemoptysis

Background

The management of massive hemoptysis is a high-risk, low-volume procedure that is associated with high mortality rates, and pulmonary and critical care medicine (PCCM) fellows often lack training. Simulation-based mastery learning (SBML) is an educational strategy that improves skill but has not been applied to massive hemoptysis management.

Objective

This pilot study aimed to develop a high-fidelity simulator, implement an SBML curriculum, and evaluate the impact on PCCM fellows managing massive hemoptysis.

Methods

We modified a simulator to bleed from segmental airways. Next, we developed an SBML curriculum and a validated 26-item checklist and set a minimum passing standard (MPS) to assess massive hemoptysis management. A cohort of traditionally trained providers was assessed using the checklist. First-year PCCM fellows reviewed a lecture before a pretest on the simulator using the skills checklist and underwent rapid-cycle deliberate practice with feedback. Subsequently, fellows were posttested on the simulator, with additional training as necessary until the MPS was met. We compared pretest and posttest performance and also compared SBML-trained fellows versus traditionally trained providers.

Results

The MPS on the checklist was set at 88%. All first-year PCCM fellows (N = 5) completed SBML training. Mean checklist scores for SBML participants improved from 67.7 ± 8.4% (standard deviation) at pretest to 84.6 ± 6.7% at the initial posttest and 92.3 ± 5.4% at the final (mastery) posttest. Traditionally trained participants had a mean test score of 60.6 ± 13.1%.

Conclusion

The creation and implementation of a massive hemoptysis simulator and SBML curriculum was feasible and may address gaps in massive hemoptysis management training.

The Relation Between Theoretical and Practical Exams for Health Sciences Students at King Saud Bin Abdulaziz University for Health Sciences- Jeddah

Introduction

A comprehensive approach to assessment is essential to ensure that all students' learning competencies are measured accurately. Therefore, multiple methods of assessment have been developed to address this matter. This Study aims to assess the correlation between health sciences students' performance on theoretical and practical exams.

Methods

A correlational study design was conducted. The academic performance of 352 students across theoretical/practical courses was tested. SPSS version 29.0 was used for analysis. Spearman's rho correlation (Rs), Wilcoxon, and Mann Whitney were computed at p<0.05.

Results

The theoretical performance was strongly correlated with the practical performance of all programs pooled together (Rs (352) = 0.67, p<0.001). Also, there was a strong correlation between theoretical and practical performance for male students (Rs (181) = 0.72, p<0.001), while a moderate correlation for female students (Rs (171) = 0.53, p<0.001). Mann-Whitney test revealed significant mean performance difference by gender both at theoretical (U = 9284, p<0.0001) and practical (U = 11,373, p < 0.0001) levels.

Conclusion

There were significant correlations between theoretical knowledge and practical skills across the selected four programs.; The mean student's performance was better in the practical skills than in the theoretical knowledge assessment, and female students surpassed male students in both practical and theoretical assessments in the four programs offered to both genders.

Test-Only Learning via Virtual Patients to Improve Surgical Illness Scripts of Preclinical Medical Students as a Preparation for Clinical Clerkships: An Experimental Study

OBJECTIVE

The aim was to determine whether preclinical medical students can develop their illness scripts to a level comparable to that of clerkship students through test-only learning using repeated formative online testlets.

METHODS

In this experimental study, participants were 52 preclinical and 53 clinical medical students. The intervention group consisted of preclinical medical students, and the control group consisted of clinical medical students. The intervention group responded to online testlets containing feedback, an innovative formative assessment method called ContExtended Questions, on general surgery for 8 days by spending no more than 30 minutes each day. The control group completed the general surgery clerkship. The performances were assessed using 20 Key-Feature Question items. The intervention group was assessed twice: immediately after the intervention (the immediate test), and again 1 month later (the delayed test). The control group was assessed once, immediately after the clerkship. All performance tests were identical.

RESULTS

The preclinical students had a significantly higher mean score on the immediate test (83.1 ± 9.6) compared to the clinical students (75.4 ± 8.9), P < .001. The effect size (Cohen's d) was .83. However, the mean score in the delayed test (76.9 ± 13.6) was not significantly different from clinical students' mean score (75.4 ± 8.9), P > .05.

CONCLUSIONS

Test-only learning as a spaced repetition of online formative testlets is effective in preparing preclinical medical students to the clinical clerkship. Through using this approach in preclinical period, they can prepare themselves for the clinical environment to optimize the benefits derived from clerkships.

Improving Health Professional Recognition and Response to Child Maltreatment and Intimate Partner Violence: Protocol for Two Mixed Methods Pilot Randomized Controlled Trials

BACKGROUND

The optimal educational approach for preparing health professionals with the knowledge and skills to effectively recognize and respond to family violence, including child maltreatment and intimate partner violence, remains unclear. The Violence, Evidence, Guidance, and Action (VEGA) Family Violence Education Resources is a novel intervention that can be completed via self-directed learning or in a workshop format; both approaches focus on improving health professional preparedness to address family violence.

OBJECTIVE

Our studies aim to determine the acceptability and feasibility of conducting a randomized controlled trial to evaluate the effectiveness of the self-directed (experimental intervention) and workshop (active control) modalities of VEGA, as an adjunct to standard education, to improve learner (Researching the Impact of Service provider Education [RISE] with Residents) and independent practice (RISE with Veterans) health professional preparedness, knowledge, and skills related to recognizing family violence in their health care encounters.

METHODS

The RISE with Residents and RISE with Veterans research studies use embedded experimental mixed methods research designs. The quantitative strand for each study follows the principles of a pilot randomized controlled trial. For RISE with Residents, we aimed to recruit 80 postgraduate medical trainees; for RISE with Veterans, we intended to recruit 80 health professionals who work or have worked with Veterans (or their family members) of the Canadian military or the Royal Canadian Mounted Police in a direct service capacity. Participants complete quantitative assessments at baseline, after intervention, and at 3-month follow-up. A subset of participants from each arm also undergoes a qualitative semistructured interview with the aim of describing participants' perceptions of the value and impact of each VEGA modality, as well as research burden. Scores on potential outcome measures will be mapped to excerpts of qualitative data via a mixed methods joint display to aid in the interpretation of findings.

RESULTS

We consented 71 individuals to participate in the RISE with Residents study. Data collection was completed on August 31, 2023, and data are currently being cleaned and prepared for analysis. As of January 15, 2024, we consented 34 individuals in the RISE with Veterans study; data collection will be completed in March 2024. For both studies, no data analysis had taken place at the time of manuscript submission. Results will be disseminated through peer-reviewed publications; academic conferences; and posting and sharing of study summaries and infographics on social media, the project website, and via professional network listserves.

CONCLUSIONS

Reducing the impacts of family violence remains a pressing public health challenge. Both research studies will provide a valuable methodological contribution about the feasibility of trial methods in health professions education focused on family violence. They will also contribute to education science about the differences in the effectiveness of self-directed versus facilitator-led learning strategies.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05490121, https://clinicaltrials.gov/study/NCT05490121; ClinicalTrials.gov NCT05490004, https://clinicaltrials.gov/study/NCT05490004.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/50864.

Educators' consideration of learner motivation in ophthalmology education in medical school: Influences on teaching practice and course design

BACKGROUND

Ophthalmology education in medical school has historically neglected the impact of autonomous motivation on student learning and wellbeing. This study aimed to understand ophthalmology educators' consideration and application of student motivation in ophthalmology medical education.

MATERIAL AND METHODS

Lead ophthalmology educators from Australian and New Zealand medical schools participated in an online semi-structured in-depth interview. Interview transcripts were analysed using thematic analysis. Codes were generated and aligned into overarching themes.

FINDINGS

Six educators participated in the study. Five main themes arose from the transcripts: the lack of explicit consideration of student motivation, implicit consideration of motivation in curriculum design and in teaching practices, the impact of innovation on motivation and the relationship between teacher and student motivation. Participants also commented on trends in ophthalmology education including generalists' confidence in managing ophthalmic disease, the role of fundoscopy in medical education and time pressure on ophthalmology in medical schools.

CONCLUSION

There has only been an implicit instead of explicit consideration of motivation in ophthalmology education in medical school, which leaves an unfulfilled potential for teaching practices to impact the affective along with cognitive and metacognitive aspects of learning. This study highlights the need for motivation to be explicitly incorporated into the development of teaching practices and curriculum reform.

What have we learned about constructed response short-answer questions from students and faculty? A multi-institutional study

PURPOSE

The purpose of this study was to enrich understanding about the perceived benefits and drawbacks of constructed response short-answer questions (CR-SAQs) in preclerkship assessment using Norcini's criteria for good assessment as a framework.

METHODS

This multi-institutional study surveyed students and faculty at three institutions. A survey using Likert scale and open-ended questions was developed to evaluate faculty and student perceptions of CR-SAQs using the criteria of good assessment to determine the benefits and drawbacks. Descriptive statistics and Chi-square analyses are presented, and open responses were analyzed using directed content analysis to describe benefits and drawbacks of CR-SAQs.

RESULTS

A total of 260 students (19%) and 57 faculty (48%) completed the survey. Students and faculty report that the benefits of CR-SAQs are authenticity, deeper learning (educational effect), and receiving feedback (catalytic effect). Drawbacks included feasibility, construct validity, and scoring reproducibility. Students and faculty found CR-SAQs to be both acceptable (can show your reasoning, partial credit) and unacceptable (stressful, not USMLE format).

CONCLUSIONS

CR-SAQs are a method of aligning innovative curricula with assessment and could enrich the assessment toolkit for medical educators.

The spacing effect: Improving electrocardiogram interpretation

BACKGROUND

Accurate electrocardiogram (ECG) interpretation is critical for safe patient care, making this skill a necessary competency for medical school graduation. Improved long-term memory retention with repeated exposure to material is one of the most evidenced-based components of adult learning science. This curricular innovation aimed to determine if implementing spaced repetition and retrieval practice using ECG quizzes during the principal clinical year would improve ECG interpretation skills among medical students enrolled in a Longitudinal Integrated Clerkship (LIC).

APPROACH

The curricular innovation applied the spacing effect and retrieval practice. Cognitive science demonstrates enhanced long-term retention through repeated interval exposure to learned material. Studies of spaced retrieval indicate that memory retention is enhanced through tests involving effortful recall. LIC students in an intervention group were exposed to the spacing effect with periodic ECG quizzes throughout their clinical clerkship year.

EVALUATION

The results of the 17-item post-test for 140 students were analysed: LIC intervention, N = 54; block control, N = 62; and LIC control, N = 24. The ANOVA test was significant (p < 0.001). Games-Howell post hoc testing showed that the mean score in the LIC intervention group was significantly higher compared with the LIC control group (p < 0.001) and the block control group (p < 0.001). There was no significant difference between the LIC control and block control groups (p = 0.59).

IMPLICATIONS

Spaced repetition of material through ECG quizzes improved ECG interpretation skills on an ECG post-test and mitigates the forgetting curve, maintaining student competency in ECG interpretation.

Self-determination theory in ophthalmology education: factors influencing autonomy, competence and relatedness in medical students

BACKGROUND

The affective components of learning, including student motivation, has yet to be thoroughly investigated in undergraduate ophthalmology education. This study aims to use Self-Determination Theory (SDT) as a framework to describe the variations in student perceptions of motivation in studying ophthalmology through their satisfactions of autonomy, competence and relatedness, and to highlight factors that stimulate or hinder this.

METHODS

Penultimate year medical students from a single tertiary educational institution undertaking a clinical placement in ophthalmology participated in in-depth interviews to explore factors affecting their perceptions of motivation in studying ophthalmology. Interviews were transcribed and analysed according to the principles of interpretive phenomenography through the theoretical framework of SDT.

RESULTS

Of the 39 students invited, 10 agreed to participate. Variations in perceptions of experiences generated the outcome space. Participants experienced either amotivation, external locus extrinsic motivation, internal locus extrinsic motivation and intrinsic motivation (conceptions of the outcome space). This was described with respect to their satisfaction of autonomy, competence and relatedness (dimensions of the outcome space). Additionally, 21 factors that impacted on motivation were identified, of which five over-arching factors impacted all three basic psychological needs - guidance, growth mindset, assessment, curricular pressure and extracurricular pressure.

CONCLUSIONS

The findings of this study provide a unique insight into the motivation of medical students studying ophthalmology. This provides an exciting opportunity for medical educators to address the affective aspect of learning.

Feedback, fairness, and validity: effects of disclosing and reusing multiple-choice questions in medical schools

BACKGROUND

Disclosure of items used in multiple-choice-question (MCQ) exams may decrease student anxiety and improve transparency, feedback, and test-enhanced learning but potentially compromises the reliability and fairness of exams if items are eventually reused. Evidence regarding whether disclosure and reuse of test items change item psychometrics is scarce and inconclusive.

METHODS

We retrospectively analysed difficulty and discrimination coefficients of 10,148 MCQ items used between fall 2017 and fall 2019 in a large European medical school in which items were disclosed from fall 2017 onwards. We categorised items as 'new'; 'reused, not disclosed'; or 'reused, disclosed'. For reused items, we calculated the difference from their first ever use, that is, when they were new. Differences between categories and terms were analysed with one-way analyses of variance and independent-samples t tests.

RESULTS

The proportion of reused, disclosed items grew from 0% to 48.4%; mean difficulty coefficients increased from 0.70 to 0.76; that is, items became easier, P < .001, η_p² = 0.011. On average, reused, disclosed items were significantly easier (M = 0.83) than reused, not disclosed items (M = 0.71) and entirely new items (M = 0.66), P < .001, η_p² = 0.087. Mean discrimination coefficients increased from 0.21 to 0.23; that is, item became slightly more discriminating, P = .002, η_p² = 0.002.

CONCLUSIONS

Disclosing test items provides the opportunity to enhance feedback and transparency in MCQ exams but potentially at the expense of decreased item reliability. Discrimination was positively affected. Our study may help weigh advantages and disadvantages of using previously disclosed items.

When Feedback is Not Perceived as Feedback: Challenges for Regulatory Body-Mandated Peer Review

PURPOSE

Safe and competent patient care depends on physicians recognizing and correcting performance deficiencies. Generating effective insight depends on feedback from credible sources. Unfortunately, physicians often have limited access to meaningful guidance. To facilitate quality improvement, many regulatory authorities have designed peer-facilitated practice enhancement programs. Their mandate to ensure practice quality, however, can create tension between formative intentions and risk (perceived or otherwise) of summative repercussions. This study explored how physicians engage with feedback when required to undergo review.

METHOD

Between October 2018 and May 2020, 30 physicians representing various specialties and career stages were interviewed about their experiences with peer review in the context of regulatory body-mandated programs. Twenty had been reviewees and reviewers and, hence, spoke from both vantage points. Interview transcripts were analyzed using a 3-stage coding process informed by constructivist grounded theory.

RESULTS

Perceptions about the learning value of mandated peer review were mixed. Most saw value but felt anxiety about being selected due to being wary of regulatory bodies. Recognizing barriers such perceptions could create, reviewers described techniques for optimizing the value of interactions with reviewees. Their strategies aligned well with the R2C2 feedback and coaching model with which they had been trained but did not always overcome reviewees' concerns. Reasons included that most feedback was "validating," aimed at "tweaks" rather than substantial change.

CONCLUSIONS

This study establishes an intriguing and challenging paradox: feedback appears often to not be recognized as feedback when it poses no threat, yet feedback that carries such threat is known to be suboptimal for inducing performance improvement. In efforts to reconcile that tension, the authors suggest that peer review for individuals with a high likelihood of strong performance may be more effective if expectations are managed through feedforward rather than feedback.

Implementation of Spaced Repetition by First-Year Medical Students: a Retrospective Comparison Based on Summative Exam Performance

Many medical students use spaced repetition as a study strategy to improve knowledge retention, and there has been growing interest from medical students in using flashcard software, such as Anki, to implement spaced repetition. Previous studies have provided insights into the relationship between medical students' use of spaced repetition and exam performance, but most of these studies have relied on self-reports. Novel insights about how medical students use spaced repetition can be gleaned from research that takes advantage of the ability of digital interfaces to log detailed data about how students use software. This study is unique in its use of data extracted from students' digital Anki data files, and those data are used to compare study patterns over the first year of medical school. Implementation of spaced repetition was compared between two groups of students who were retrospectively grouped based on average performance on three exams throughout the first year of medical school. Results indicate that students in the higher scoring group studied more total flashcards and implemented spaced repetition via Anki earlier in the year compared to the lower scoring group. These findings raise the possibility that implementing spaced repetition as a study strategy early in medical school may be related to improved knowledge retention and exam performance. Additional research should be performed at more sites to further examine the relationship between spaced repetition implementation and exam performance.

Constructing validity evidence from a pilot key-features assessment of clinical decision-making in cerebral palsy diagnosis: application of Kane's validity framework to implementation evaluations

BACKGROUND

Physician decision-making skills training is a priority to improve adoption of the cerebral palsy (CP) clinical guideline and, through this, lower the age of CP diagnosis. Clinical guideline implementation aims to improve physician practice, but evaluating meaningful change is complex. Limitations in the validity evidence of evaluation instruments impact the evidence base. Validity frameworks, such as Kane's, enable a targeted process to gather evidence for instrument scores, congruent to context and purpose. Yet, application of argument-based methodology to implementation validation is rare. Key-features examination methodology has established validity evidence supporting its use to measure decision-making skills, with potential to predict performance. We aimed to apply Kane's framework to evaluate a pilot key-features examination on physician decision-making in early CP diagnosis.

METHODS

Following Kane's framework, we evaluated evidence across inferences of scoring, generalisation, extrapolation and implications in a study design describing the development and pilot of a CP diagnosis key-features examination for practising physicians. If found to be valid, we proposed to use the key-feature scores as an outcome measure of decision-making post education intervention to expedite CP diagnosis and to correlate with real-world performance data to predict physician practice.

RESULTS

Supporting evidence for acceptance of scoring inferences was achieved through examination development with an expert group (n = 10) and pilot results (n = 10): (1) high internal consistency (0.82); (2) acceptable mean item-discrimination (0.34); and (3) acceptable reliability of examination scorers (95.2% congruence). Decreased physician acceptance of examination time (70%) was identified as a threat and prioritised in case reduction processes. Partial acceptance of generalisation, extrapolation and implications inferences were defensible with: (1) accumulated development evidence following established key-features methodology; (2) high pilot acceptance for authenticity (90%); and (3) plausibility of assumptions of score correlation with population register data.

CONCLUSIONS

Kane's approach is beneficial for prioritising sources of validity evidence alongside the iterative development of a key-features examination in the CP field. The validity argument supports scoring assumptions and use of scores as an outcome measure of physician decision-making for CP guideline education implementation interventions. Scoring evidence provides the foundation to direct future studies exploring association of key-feature scores with real-world performance.

If we assess, will they learn? Students' perspectives on the complexities of assessment-for-learning

Introduction

Assessment can positively influence learning, however designing effective assessment-for-learning interventions has proved challenging. We implemented a mandatory assessment-for-learning system comprising a workplace-based assessment of non-medical expert competencies and a progress test in undergraduate medical education and evaluated its impact.

Methods

We conducted semi-structured interviews with year-3 and 4 medical students at McGill University to explore how the assessment system had influenced their learning in year 3. We conducted theory-informed thematic analysis of the data.

Results

Eleven students participated, revealing that the assessment influenced learning through several mechanisms. Some required little student engagement (i.e., feed-up, test-enhanced learning, looking things up after an exam). Others required substantial engagement (e.g., studying for tests, selecting raters for quality feedback, using feedback). Student engagement was moderated by the perceived credibility of the system and of the costs and benefits of engagement. Credibility was shaped by students' goals-in-context: becoming a good doctor, contributing to the healthcare team, succeeding in assessments.

Discussion

Our assessment system failed to engage students enough to leverage its full potential. We discuss the inherent flaws and external factors that hindered student engagement. Assessment designers should leverage easy-to-control mechanisms to support assessment-for-learning and anticipate significant collaborative work to modify learning cultures.

Interactive computer-assisted learning as an educational method for learning pediatric interproximal dental caries identification

OBJECTIVE

We developed a web-based tool to measure the amount and rate of skill acquisition in pediatric interproximal caries diagnosis among pre- and postdoctoral dental students and identified variables predictive for greater image interpretation difficulty.

STUDY DESIGN

In this multicenter prospective cohort study, a convenience sample of pre- and postdoctoral dental students participated in computer-assisted learning in the interpretation of bitewing radiographs of 193 children. Participants were asked to identify the presence or absence of interproximal caries and, where applicable, locate the lesions. After every case, participants received specific visual and text feedback on their diagnostic performance. They were requested to complete the 193-case set but could complete enough cases to achieve a competency performance standard of 75% accuracy, sensitivity, and specificity.

RESULTS

Of 130 participants, 62 (47.7%) completed all cases. The mean change from initial to maximal diagnostic accuracy was +15.3% (95% CI, 13.0-17.7), sensitivity was +10.8% (95% CI, 9.0-12.7), and specificity was +15.5% (95% CI, 12.9-18.1). The median number of cases completed to achieve competency was 173 (interquartile range, 82-363). Of these 62 participants, 45 (72.6%) showed overall improvement in diagnostic accuracy. Greater numbers of interproximal lesions (P < .001) and the presence of noninterproximal caries (P < .001) predicted greater interpretation difficulty.

CONCLUSIONS

Computer-assisted learning led to improved diagnosis of interproximal caries on bitewing radiographs among pre- and postdoctoral dental students.

The Problematic Persistence of Tiered Grading in Medical School

Issue: The evaluation of medical students is a critical, complex, and controversial process. It is tightly woven into the medical school curriculum, beginning at the inception of the medical student's professional journey. In this respect, medical student evaluation is among the first in a series of ongoing, lifelong assessments that influence the interpersonal, ethical, and socioeconomic dimensions necessary for an effective physician workforce. Yet, tiered grading has a questionable historic pedagogic basis in American medical education, and evidence suggests that tiered grading itself is a source of student burnout, anxiety, depression, increased competitiveness, reduced group cohesion, and racial biases. Evidence: In its most basic form, medical student evaluation is an assessment of the initial cognitive and technical competencies ultimately needed for the safe and effective practice of contemporary medicine. At many American medical schools, such evaluation relies largely on norm-based comparisons, such as tiered grading. Yet, tiered grading can cause student distress, is considered unfair by most students, is associated with biases against under-represented minorities, and demonstrates inconsistent correlation with residency performance. While arguments that tiered grading motivates student performance have enjoyed historic precedence in academia, such arguments are not supported by robust data or theories of motivation. Implications: Given the evolving recognition of the deleterious effects on medical student mental health, cohesiveness, and diversity, the use of tiered grading in medical schools to measure or stimulate academic performance, or by residency program directors to distinguish residency applicants, remains questionable. Examination of tiered grading in its historical, psychometric, psychosocial, and moral dimensions and the various arguments used to maintain it reveals a need for investigation of, if not transition to, alternative and non-tiered assessments of our medical students.

Use of very short answer questions compared to multiple choice questions in undergraduate medical students: An external validation study

Multiple choice questions (MCQs) offer high reliability and easy machine-marking, but allow for cueing and stimulate recognition-based learning. Very short answer questions (VSAQs), which are open-ended questions requiring a very short answer, may circumvent these limitations. Although VSAQ use in medical assessment increases, almost all research on reliability and validity of VSAQs in medical education has been performed by a single research group with extensive experience in the development of VSAQs. Therefore, we aimed to validate previous findings about VSAQ reliability, discrimination, and acceptability in undergraduate medical students and teachers with limited experience in VSAQs development. To validate the results presented in previous studies, we partially replicated a previous study and extended results on student experiences. Dutch undergraduate medical students (n = 375) were randomized to VSAQs first and MCQs second or vice versa in a formative exam in two courses, to determine reliability, discrimination, and cueing. Acceptability for teachers (i.e., VSAQ review time) was determined in the summative exam. Reliability (Cronbach's α) was 0.74 for VSAQs and 0.57 for MCQs in one course. In the other course, Cronbach's α was 0.87 for VSAQs and 0.83 for MCQs. Discrimination (average Rir) was 0.27 vs. 0.17 and 0.43 vs. 0.39 for VSAQs vs. MCQs, respectively. Reviewing time of one VSAQ for the entire student cohort was ±2 minutes on average. Positive cueing occurred more in MCQs than in VSAQs (20% vs. 4% and 20.8% vs. 8.3% of questions per person in both courses). This study validates the positive results regarding VSAQs reliability, discrimination, and acceptability in undergraduate medical students. Furthermore, we demonstrate that VSAQ use is reliable among teachers with limited experience in writing and marking VSAQs. The short learning curve for teachers, favourable marking time and applicability regardless of the topic suggest that VSAQs might also be valuable beyond medical assessment.

Educational initiatives in headache medicine: A 20-year scoping review

BACKGROUND

Headache disorders are among the most common and disabling medical conditions worldwide, have a great societal impact and are a common reason to seek medical care. Headache disorders are often misdiagnosed and undertreated, and the number of headache fellowship-trained physicians cannot meet patient demand. Educational initiatives for non-headache-specialist clinicians may be an avenue to increase clinician competency and patient access to appropriate management.

OBJECTIVE

To undertake a scoping review of the educational initiatives in headache medicine for medical students, trainees, general practitioners/primary care physicians, and neurologists.

METHODS

Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines for scoping reviews, an author (M.D.), with the help of a medical librarian, conducted a search of the Embase, Ovid Medline, and PsychInfo databases for articles related to medical educational initiatives on headache medicine in medical students, residents, and physicians over the last 20 years.

RESULTS

A total of 17 articles met the inclusion criteria for this scoping review. Six articles were identified for medical students, seven for general practitioners/primary care physicians, one for emergency medicine residents, two for neurology residents, and one for neurologists. Certain educational initiatives were headache-focused while others had headaches as one of the educational topics. Educational content was delivered and assessed via diverse and innovative means, such as flipped classroom, simulation, theatrical performance, repeated quizzing and study, and a formalized headache elective.

CONCLUSION

Education initiatives in headache medicine are important to improve competency and patient access to appropriate management of various headache disorders. Future research should focus on using innovative and evidence-based methods of content delivery, knowledge, and procedural assessment, and evaluating changes in practice behaviors.

Evaluating TESLA-G, a gamified, telegram-delivered, quizzing platform for surgical education in medical students: protocol for a pilot randomised controlled trial

INTRODUCTION

Online multiple-choice question (MCQ) quizzes are popular in medical education due to their ease of access and ability for test-enhanced learning. However, a general lack of motivation among students often results in decreasing usage over time. We aim to address this limitation by developing Telegram Education for Surgical Learning and Application Gamified (TESLA-G), an online platform for surgical education that incorporates game elements into conventional MCQ quizzes.

METHODS AND ANALYSIS

This online, pilot randomised control trial will be conducted over 2 weeks. Fifty full-time undergraduate medical students from a medical school in Singapore will be recruited and randomised into an intervention group (TESLA-G) and an active control group (non-gamified quizzing platform) with a 1:1 allocation ratio, stratified by year of study.We will evaluate TESLA-G in the area of endocrine surgery education. Our platform is designed based on Bloom's taxonomy of learning domains: questions are created in blocks of five questions per endocrine surgery topic, with each question corresponding to one level on Bloom's taxonomy. This structure promotes mastery while boosting student engagement and motivation. All questions are created by two board-certified general surgeons and one endocrinologist, and validated by the research team. The feasibility of this pilot study will be determined quantitatively by participant enrolment, participant retention and degree of completion of the quizzes. The acceptability of the intervention will be assessed quantitatively by a postintervention learner satisfaction survey consisting of a system satisfaction questionnaire and a content satisfaction questionnaire. The improvement of surgical knowledge will be assessed by comparing the scores of preintervention and postintervention knowledge tests, which consist of separately created questions on endocrine surgery. Retention of surgical knowledge will be measured using a follow-up knowledge test administered 2 weeks postintervention. Finally, qualitative feedback from participants regarding their experience will be obtained and thematically analysed.

ETHICS AND DISSEMINATION

This research is approved by Singapore Nanyang Technological University (NTU) Institutional Review Boards (Reference Number: IRB-2021-732). All participants will be expected to read and sign a letter of informed consent before they are considered as recruited into the study. This study poses minimal risk to participants. Study results will be published in peer-reviewed open-access journals and presented in conference presentations.

TRIAL REGISTRATION NUMBER

NCT05520671.

Information overdose: Student performance and perceptions of pharmacology resources on exams

Physicians of the future will be expected to synthesize new knowledge and appropriately apply it in patient care. Here, we report on the effects of and student attitudes towards resource-enhanced exams by comparing student performance on closed-book exams with or without access to pharmacology flashcards. Setting: the University of California, San Francisco (UCSF) School of Medicine (SOM), class of 2021 (N = 149), followed over 4 years. We provided pharmacology flashcards for studying purposes in all blocks; flashcards were only accessible during closed-book exams in 2 of 5 blocks. We collected pharmacology open-ended question (OEQ) scores and analyzed results using repeated measures ANOVA (SPSS). We collected MS4 survey data using Qualtrics and conducted a thematic content analysis. Performance on pharmacology questions on exams was not higher with access to pharmacology flashcards during exams. The number of students who passed pharmacology questions without flashcards on exams was as follows: 137 ± 3.7, 132 ± 5.0, and 134 ± 7.9 (average ± SEM). The number of students who passed pharmacology questions with flashcards on exams was as follows: 132 ± 6.6 and 120 ± 7.5. Survey comments revealed several themes. Access to pharmacology flashcards during exams allowed learners to focus on understanding the bigger picture and reduced stress. A subset of students reported having access to flashcards on pre-clerkship exams hurt their preparation for clerkships. Flashcards as exam resources were received well by approximately half the class, who reported benefits including more time to focus on understanding bigger picture concepts and reduced stress.

Effective bedside teaching as a foundation doctor

Bedside teaching is a key component of education at medical school, particularly in the clinical years. It provides an invaluable opportunity for students to practise examining patients, and further develops their communication and professional skills. Doctors who have newly graduated from medical school are often expected to provide such teaching to medical students placed on their wards. However, foundation doctors often receive little to no training for bedside teaching and there is limited literature on practical tips on how to enhance bedside teaching. Here we consider the three stages of effective bedside teaching: preparation, teaching and evaluation.

A workers' health surveillance online training programme for occupational physicians

BACKGROUND

To support occupational physicians (OPs) in the implementation of workers' health surveillance (WHS), a training programme was developed.

AIMS

(i) To evaluate the effects of a WHS training programme for OPs on knowledge, self-efficacy and skills to implement WHS. (ii) To evaluate to what extent a WHS training programme is acceptable and feasible for implementation in practice.

METHODS

A single-blinded randomized controlled trial with waiting-list control group was used. The WHS training programme consisted of an e-learning and a 4.5-h online training session. OPs completed a knowledge test (0-8), self-efficacy questionnaires on knowledge and skills (6-60), and vignette assignments (0-16) to measure skills. OPs completed the questionnaires, either before and after the WHS training programme (intervention group), or before the training programme (control group) while receiving the training programme after the waiting period. All OPs completed questionnaires about the training's acceptability, and feasibility for implementation in practice. ANCOVA and Poisson regression analyses were conducted.

RESULTS

The self-efficacy score (M = 44.1 versus M = 37.2) (P < 0.001) and skills score (M = 9.6 versus M = 8.3) (P < 0.05) of OPs in the training group (N = 16) were higher than the control group (N = 23). No effect was found on knowledge. Evaluation of acceptability and feasibility showed that 21 (58%) OPs were very satisfied with the training part on initiating WHS, and 29 (85%) would recommend the WHS training programme to colleagues.

CONCLUSIONS

This WHS training programme has a positive effect on self-efficacy and skills of OPs to implement WHS, and may be acceptable and feasible to implement in practice.

Evidence-based teaching in contact lenses education: Teaching and learning strategies

INTRODUCTION

Contact lens (CL) practice is an ever-changing field with clinical knowledge, techniques and equipment continuously evolving. These new developments are backed with clinical trials and research to ensure that practitioners feel confident that there is an evidence base to support these advances. Evidence-based practice is now a crucial part of CL practice, and its importance also filters down to CL education. For example, lectures are one of the most popular tools for an educator but, is standing at the front of a lecture theatre full of students a more effective way of teaching than providing the same material for students to read by themselves? What evidence exists specific to CL education?

METHOD

An expert panel of educators completed a comprehensive literature review of current evidence of teaching methods in CL training, or if not available then what can be learnt from other health care professional training that could be potentially applicable to CL education.

RESULTS

Due to the amount of evidence available in the overall subject area relating to healthcare education, the initial plan of compiling evidence into one narrative review paper was discarded in favour of producing two linked papers. Here, the first paper details definitions of terminology, and also teaching methods. The second paper focuses on assessment and specific clinical training required to attain CL practice competency. In this first paper, no direct evidence of the spreading and benefit of new education strategies evidence such as flipped classrooms, spaced learning, test-enhanced learning, group work, CBL, PBL, TBL, and reflective practice in CL education was found. The only technique that was widely used in the CL field was case reports and the group discussion of them. Nevertheless, the authors found a consensus of opinion from other disciplines that are transferable to CL teaching and could help students meet the intended learning outcomes.

CONCLUSION

There is a small amount of evidence supporting CL education, but most of this seems to be related to the practical element of the training. However, there is a lot of evidence in the field of healthcare education from related disciplines which provides additional but important learning tools that may be effectively implemented in CL education.

Educational Videos Versus Question Banks: Maximizing Medical Student Performance on the United States Medical Licensing Examination Step 1 Exam

Objective The aim of this research was to determine if medical students' use of the active study strategy of working practice questions is associated with improved performance on the United States Medical Licensing Examination (USMLE) Step 1 exam when compared to students who used the passive study strategy of watching educational videos. Methods The study used a correlational design. Participants were students from two cohorts in a United States medical school (N=164 and N=163) who completed their first two years and took the USMLE Step 1 exam. Data collected retrospectively included the number of practice questions completed, educational videos watched, Step 1 exam scores, average scores on in-class exams, and scores on the Medical College Admission Test (MCAT). Results The number of videos watched was negatively and significantly correlated with the Step 1 score for cohort 2022 (r= -0.294, α=0.01) and cohort 2023 (r= -0.175, α=0.05). The number of practice questions worked was positively and significantly correlated with the Step 1 score for cohort 2022 (r=0.176, α=0.05) and cohort 2023 (r=0.143 though not significant). The number of practice questions was a significant positive predictor of Step 1 score for cohort 2022 (β=0.141, p=0.017) and cohort 2023 (β=0.133, p=0.015). Videos were significant negative predictors for cohort 2023 (β= -0.118, p=0.034). Conclusions Answering practice questions appears to be a more effective study method than passively watching videos. Though other studies have supported the use of active learning methods, this study is unique in finding a negative correlation between test scores and the number of educational videos watched. Medical students should be urged to make the most effective use of study time by incorporating working practice questions and limiting watching educational videos.

Twelve tips for introducing very short answer questions (VSAQs) into your medical curriculum

Most undergraduate written examinations use multiple-choice questions, such as single best answer questions (SBAQs) to assess medical knowledge. In recent years, a strong evidence base has emerged for the use of very short answer questions (VSAQs). VSAQs have been shown to be an acceptable, reliable, discriminatory, and cost-effective assessment tool in both formative and summative undergraduate assessments. VSAQs address many of the concerns raised by educators using SBAQs including inauthentic clinical scenarios, cueing and test-taking behaviours by students, as well as the limited feedback SBAQs provide for both students and teachers. The widespread use of VSAQs in medical assessment has yet to be adopted, possibly due to lack of familiarity and experience with this assessment method. The following twelve tips have been constructed using our own practical experience of VSAQs alongside supporting evidence from the literature to help medical educators successfully plan, construct and implement VSAQs within medical curricula.

Evidence for commonly used teaching, learning and assessment methods in contact lens clinical skills education

INTRODUCTION

Evidence based practice is now an important part of healthcare education. The aim of this narrative literature review was to determine what evidence exists on the efficacy of commonly used teaching and learning and assessment methods in the realm of contact lens skills education (CLE) in order to provide insights into best practice. A summary of the global regulation and provision of postgraduate learning and continuing professional development in CLE is included.

METHOD

An expert panel of educators was recruited and completed a literature review of current evidence of teaching and learning and assessment methods in healthcare training, with an emphasis on health care, general optometry and CLE.

RESULTS

No direct evidence of benefit of teaching and learning and assessment methods in CLE were found. There was evidence for the benefit of some teaching and learning and assessment methods in other disciplines that could be transferable to CLE and could help students meet the intended learning outcomes. There was evidence that the following teaching and learning methods helped health-care and general optometry students meet the intended learning outcomes; clinical teaching and learning, flipped classrooms, clinical skills videos and clerkships. For assessment these methods were; essays, case presentations, objective structured clinical examinations, self-assessment and formative assessment. There was no evidence that the following teaching and learning methods helped health-care and general optometry students meet the intended learning outcomes; journal clubs and case discussions. Nor was any evidence found for the following assessment methods; multiple-choice questions, oral examinations, objective structured practical examinations, holistic assessment, and summative assessment.

CONCLUSION

Investigation into the efficacy of common teaching and learning and assessment methods in CLE are required and would be beneficial for the entire community of contact lens educators, and other disciplines that wish to adapt this approach of evidence-based teaching.

Differences in Physician Performance and Self-rated Confidence on High- and Low-Stakes Knowledge Assessments in Board Certification

INTRODUCTION

Evidence links assessment to optimal learning, affirming that physicians are more likely to study, learn, and practice skills when some form of consequence ("stakes") may result from an assessment. We lack evidence, however, on how physicians' confidence in their knowledge relates to performance on assessments, and whether this varies based on the stakes of the assessment.

METHODS

Our retrospective repeated-measures design compared differences in patterns of physician answer accuracy and answer confidence among physicians participating in both a high-stakes and a low-stakes longitudinal assessment of the American Board of Family Medicine.

RESULTS

After 1 and 2 years, participants were more often correct but less confident in their accuracy on a higher-stakes longitudinal knowledge assessment compared with a lower-stakes assessment. There were no differences in question difficulty between the two platforms. Variation existed between platforms in time spent answering questions, use of resources to answer questions, and perceived question relevance to practice.

DISCUSSION

This novel study of physician certification suggests that the accuracy of physician performance increases with higher stakes, even as self-reported confidence in their knowledge declines. It suggests that physicians may be more engaged in higher-stakes compared with lower-stakes assessments. With medical knowledge growing exponentially, these analyses provide an example of the complementary roles of higher- and lower-stakes knowledge assessment in supporting physician learning during continuing specialty board certification.

Does Professional Development Effectively Support the Implementation of Inclusive Education? A Meta-Analysis

Inclusive education is a reform aimed at educating all students in general classrooms, independent of diversity features such as special educational needs, giftedness, or migration. Its successful implementation requires teachers with professional knowledge about inclusive education, skills to address the diverse needs in the classroom, and positive beliefs toward inclusive education. Teachers are provided with professional development opportunities, but are these effective in improving their learning process and positively impacting students’ behavior? We conducted a meta-analysis to address this question. The screening of 12,050 search results revealed 342 eligible studies with more than 155,000 participants and 1123 effects from four outcome categories: teachers’ knowledge, skills, and beliefs and students’ behavior. We observed positive, though varying, effects on all four outcome categories: large effects on teachers' knowledge regarding inclusive education (g = 0.93), moderate effects on teachers’ skills (g = 0.49), small effects on teachers’ beliefs (g = 0.23), and small-to-moderate effects on student behavior (g = 0.37). We also examined factors that might explain the differences in the strength of training effects. The data suggest that long-term training with high practical relevance and active learning opportunities facilitates transfer to schools.

Study Based on Gamification of Tests through Kahoot!™ and Reward Game Cards as an Innovative Tool in Physiotherapy Students: A Preliminary Study

BACKGROUND

Kahoot! is an educational tool allowing teachers to create a series of gamified tests with the aim of reinforcing educational content, thus improving the teaching-learning process. The objective of this project is to evaluate the acquisition of content through gamified tests with Kahoot! and reward cards compared to the traditional teaching methodology (contents not reinforced).

METHODS

This Physiotherapy Teaching Innovation Project (PTIP) was carried out in four subjects of the Degree in Physiotherapy at the University of Jaén (Spain). The teachers responsible for each subject were instructed in the use of Kahoot! and reward cards. These teachers randomly selected the contents that were going to be reinforced with Kahoot! while the other 50% of the contents would not be reinforced. In the final exam of each subject, the results related to the reinforced contents were compared with those non-reinforced and the degree of satisfaction of the students with the experience was evaluated.

RESULTS

A total of 313 students participated in this PTIP. In all subjects, we determined a significant increase in the number of correct answers in an improvement range from 7% (95% CI 3.85 to 9.38) to more than 20% (95% CI 17.61 to 26.86) in favor of the questions that alluded to reinforced content using Kahoot! compared to the non-reinforced contents. More than 90% of the participants considered the use of Kahoot! useful and motivating. Our findings showed that Kahoot! motivated more than 65% of students to study daily.

CONCLUSIONS

The students obtained better academic results in the questions related to contents reinforced with tests through Kahoot! and reward cards compared to those non-reinforced, showing that this methodology can be an effective tool to promote retention and content assimilation.

Building a student learning-focused assessment and grading system in dental school: One school's experience

PURPOSE/OBJECTIVES

As health professions education moves toward competency-based education, there has been increased focus on the structure of assessment systems that support student competency development and learning. This has been buoyed by a growing body of research supporting assessment for learning processes to promote student growth and learning rather than relying on assessment systems primarily to measure performance. This paper presents the rationale and evidence for moving to an assessment for learning system and the results of a quasi-experimental interrupted time series study using data from 2015 to 2022 to evaluate the impacts of these changes.

METHODS

Columbia University College of Dental Medicine faculty voted to implement assessment for learning system changes beginning in 2017 with the graduating class of 2021. These changes included moving from using a grading system for didactic courses with Honors, Pass, and Fail as available grades to a grading system with only Pass and Fail as available grades, as well as creating synthesis and assessment weeks, weekly problem sets, post-exam review sessions, exam remediation opportunities, and formative progress exams throughout the curriculum. The revised assessment and grading system changes were communicated to residency program directors, and programmatic competency data about student performance across the curriculum were shared with programs in Dean's Letters.

RESULTS

Once assessment system changes were implemented, it was found that student exam failure rates were lower, course exam scores were the same or higher, and performance on board exams improved compared to the national average. Students reported positive perceptions with regard to well-being and learning climate that they associated with the adoption of Pass/Fail grading. Match outcomes, including student satisfaction and program director ratings, have remained consistently positive.

CONCLUSION

As dental educators, our goal is to nurture students to become life-long learners. Adopting a grading structure that is Pass/Fail and an assessment system that fosters learning allows students to shape learning practices that favor long-term retention and application of information, also enhancing the learning environment and student well-being. These system changes may also facilitate the inclusion and support of students whose backgrounds are underrepresented in dentistry.

Online educational interventions in pediatric intensive care medicine

Background

Online education has experienced explosive growth, particularly in the wake of the COVID-19 pandemic. We explored the current state of the evidence base for online education targeted towards healthcare professionals working in pediatric intensive care units (PICUs), to report how we are using online education in our field.

Materials and Methods

We performed a literature review by systematically generating a list of publications indexed in PubMed describing online educational interventions in the PICU, using Medical Subject Header (MeSH)-based search terms and the following inclusion criteria: studies published after 2005 that describe online educational interventions aimed at healthcare professional working in the PICU. We reviewed the full text of all included articles, and summarized the study aims, design, and results.

Results

Our initial search yielded 1,071 unique articles. After screening abstracts and titles, then full texts, eight articles were included in the review. Many online learning modalities are represented, including websites, self-study modules, videos, videoconferencing, online self-assessment with feedback, virtual patient cases, screen-based simulation, and podcasts. Three studies focused on residents, two studies on nurses, two studies on a multidisciplinary team, and one study on transport nurses and paramedics. Most studies utilized participant surveys to assess satisfaction, and half included pre- and post-intervention multiple-choice question tests. Only one study included a patient-related outcome measure.

Conclusions

Despite growth in online medical educational intervention research, there are relatively few published studies in pediatric critical care, and only one study evaluated the impact of online learning on patient outcomes. There remain significant opportunities for PICU educators to assess the impact of online educational interventions, especially related to clinician behaviors and patient outcomes.

Quantifying research on anatomical structures, a potential new metric for assessing clinical relevance

Human anatomy remains an integral part of medical education, and recent studies have documented an emerging consensus on the key anatomical learning objectives for physicians and other health professionals in training, both at the graduate and postgraduate levels. Despite this progress, less attention has been given to assessing the clinical relevance of individual anatomical structures, and which structures students should master to achieve these learning objectives. In this study we hypothesized that published research involving individual anatomical structures is largely driven by the clinical relevance of these structures, and that tabulating the number of such publications can provide an up-to-date, evolving metric of clinical relevance. To test this hypothesis, we developed a semi-automated search routine that uses the PubMed database to quantify the publication frequency of anatomical structures and compared that to a previous study that assessed the importance of structures of the head and neck using the Delphi method, a formal procedure of generating expert consensus. Using our new approach, we were able to rank the research intensity of 2182 anatomical structures included in Grant's Dissector, a widely used textbook for anatomical dissection. Furthermore, a sample of these PubMed-derived ranks had a highly significant, positive correlation with ranks derived from a consensus of experts. Similar results were obtained when PubMed searches were restricted to journals that focus on applying knowledge in a clinical setting. Our study provides a potential new tool for anatomical educators who are aligning their basic science curricula with the clinical knowledge expected of medical graduates.

Simulation-based summative assessment in healthcare: an overview of key principles for practice

BACKGROUND

Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.

METHODS

First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee.

RESULTS

Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted.

CONCLUSION

Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.

Impact of test-enhanced learning as a study strategy: An exploratory study with first-year dental students

PURPOSE

Research has demonstrated that learners who practice self-testing have superior long-term retention compared to those rereading the material alone, a phenomenon called test-enhanced learning. This testing effect can be leveraged by spacing out the testing practice over time, a technique called spaced repetition. In 2017, we provided dental students at the school with access to Osmosis, a web-based platform that supports test-enhanced learning and spaced repetition through flashcards. This exploratory study examined students' adoption of self-testing with flashcards and its impact on learning performance in basic sciences.

METHODS

Participants were 143 first-year predoctoral students at a dental school in the US. The platform analytics revealed the number of flashcards students answered throughout the first academic year (2019-2020). Regression analyses examined how self-testing with flashcards impacted students' exam scores in basic sciences. Analysis of variance (ANOVA) tests examined the difference in students' exam performance among the non, minimal, occasional, and regular flashcard users who answered 0, 1-99, 100-499, and over 500 flashcards, respectively.

RESULTS

Students answered 82,766 flashcards during the year. Additionally, they created 17,973 flashcards using the platform's flashcard authoring tool. Regression analyses showed that self-testing with flashcards correlated positively with students' exam performance in anatomy, biochemistry, nutrition, and physiology. ANOVA results revealed a statistically significant difference in students' exam performance in anatomy, biochemistry, and nutrition among the four groups.

CONCLUSIONS

This study is the first in dental education to examine students' self-testing on the Osmosis platform. Results revealed that there was widespread adoption of self-testing with flashcards. The study provided additional evidence to support the value of self-testing for dental students. It has practical implications of how test-enhanced learning can be incorporated into dental education to support student learning. The study contributed to the test-enhanced learning literature in dental education, an area that has been underexplored.

Patients with Obstructive Sleep Apnea Symptoms Referral to Otolaryngologists in Iranian Residency Entrance Examination Volunteers: The Effect of Source of Education and Level of Knowledge

Obstructive sleep apnea (OSA) has various complications for individuals' health. This study aimed to evaluate the factors for referring obstructive sleep apnea patients to otolaryngologists in Iranian residency entrance examination volunteers. This cross-sectional study recruited volunteer participants from the Iranian Residency Entrance Examination on March 2, 2018. The Obstructive Sleep Apnea Knowledge and Attitudes (OSAKA)/the Obstructive Sleep Apnea Knowledge and Attitudes in Children (OSAKA-KIDS) Questionnaires were distributed among residents attending exam preparation. Number of years working as a physician, number of adult and pediatric patients visited during the academic rank in residency examination, and the main specialty during general physician education in which they get familiar with obstructive sleep apnea were documented. The effect of the mentioned variables evaluated on patient referral preference specialty. Of the 95 volunteers, 57.9% were female; mean age was 29.6 ± 3.3 years. The overall knowledge score of the OSAKA questionnaire was 9.85 ± 3.9 and for the OSAKA-KIDS questionnaire was 9.2 ± 4.9. In patients under 18 years of age, the most frequent referrals were from otolaryngology specialists (51.2%). The only factor which had positive significant effect on adult patient referral preference was source of obstructive sleep apnea knowledge during general physician taring (p < 0.001). Given the low awareness about OSA in candidates for the residency entrance exam and the high importance of this treatable disease, the need for comprehensive training courses during residency is warranted and the adequacy of sleep apnea education through all involved specialty during general physician education should be improved.

Beyond the tensions within transfer theories: implications for adaptive expertise in the health professions

Ensuring trainees develop the flexibility with their knowledge to address novel problems, and to efficiently build upon prior knowledge to learn new knowledge is a common goal in health profession education. How trainees come to develop this capacity to transfer and transform knowledge across contexts can be described by adaptive expertise, which focuses on the ability of some experts to innovate upon their existing knowledge to develop novel solutions to novel problems. While adaptive expertise is often presented as an alternative framework to more traditional cognitivist and constructivist expertise models, it is unclear whether the non-routine and routine forms of transfer it describes are distinct from those described by other accounts of transfer. Furthermore, whether what (e.g., knowledge) is transferred and how (e.g., cognitive processes) differs between these views is still debated. In this review, we describe various theories of transfer and present a synthesis clarifying the relationship between transfer and adaptive expertise. Informed by our analysis, we argue that the mechanisms of transfer in adaptive expertise share important commonalities with traditional accounts of transfer, which when understood, can complement efforts by educators and researchers to foster and study adaptive expertise. We present three instructional principles that may better support transfer and adaptive expertise in trainees: i) identifying and incorporating meaningful variability in practice, ii) integrating conceptual knowledge during practice iii) using assessments of trainees' transfer. Taken together, we offer an integrative perspective to how educational systems and experiences can be designed to develop and encourage adaptive expertise and transfer.

Competency and confidence in ECG interpretation among medical students

OBJECTIVES

To assess competency and confidence in ECG interpretation in medical students across years of medical school and evaluate the associations of various factors, a curriculum change, and student confidence with ECG competency.

METHODS

Four hundred and fourteen (414) third- to sixth-year medical students participated in this cross-sectional study conducted in 2019 in the Hebrew University of Jerusalem, Israel. A voluntary response sample of participants answered a validated, web-based questionnaire, composed of eight ECG strips. Participants were also asked about confidence and sources for ECG education and exposure. Competency and confidence across medical school years were compared using the ANOVA and chi-square tests.

RESULTS

Competency was low overall (mean score, SD (standard deviation) 3.23±1.81 out of 8), and higher in sixth-year students compared to third-, fourth- and fifth-year students (4.37±1.69 vs. 2.90±1.82, 2.90±1.54, 2.50±1.56, respectively, F_(3,337)=24.425, p<0.0001). There was no difference between students before and after the curriculum change. Work experience in medicine was associated with competency (odds ratio (OR), 7.97; 95% confidence interval (CI), 4.03-15.77, p<0.0001). The reported confidence level was low (median 2 out of 5) and was found to be correlated with the total score achieved (r₍₃₃₂₎=0.5, p<0.0001).

CONCLUSIONS

Student competency was shown to be insufficient throughout medical school. Competency and confidence in ECG interpretation seem to be significantly improved by increased and repetitive exposure to ECG. Thus, strategies to facilitate better ECG skills should involve an extended focus on ECG in the undergraduate and graduate curricula and include competency-based educational programs.

Tablet-assisted objective structured spotter practical examination (TOSSPE): Advantages of an innovative anatomy spotter examination method for medical student assessment

The affordances of technology-based assessments, like the objectively structured practical examination, have become an integral part of gross anatomy courses. The Department of Anatomy Faculty of Medicine at the University of Warmia and Mazury developed and introduced an application for tablet devices which has been implemented in student examinations and assessments, called the tablet-assisted objective structured spotter practical examination. It was created to simplify the educational process and to build a rich learning environment, facilitating deep learning for students through examination and feedback data. The method consists of cadaver stations with traditional corresponding pin spotters in an expanded tablet application. It not only provides instant feedback on various observations of teaching-learning skills but has also positively affected the entire process of education. The method provides an unbiased evaluation of knowledge and understanding of the anatomy course, ensuring objectivity and standardization. The current study was performed on a total of 608 first-year medical students in Polish and English divisions and focused on the observed advantages since the new method was introduced. Outcomes indicate that after the implementation of the method for both the Polish and English divisions' first-year medical students, the mean score of examinations significantly increased compared to other teaching-learning methods. The study highlights that students were excited about the implementation of the new method and identified its many benefits. It is recognized that technological development and the digital environment offer a range of opportunities and added value versus traditional assessment activities, methods, and processes.

Advances in e-learning in undergraduate clinical medicine: a systematic review

BACKGROUND

E-learning is recognised as a useful educational tool and is becoming more common in undergraduate medical education. This review aims to examine the scope and impact of e-learning interventions on medical student learning in clinical medicine, in order to aid medical educators when implementing e-learning strategies in programme curricula.

METHODS

A systematic review compliant with PRISMA guidelines that appraises study design, setting and population, context and type of evaluations. Specific search terms were used to locate articles across nine databases: MEDLINE/PubMed, ScienceDirect, EMBASE, Cochrane Library, ERIC, Academic Search Complete, CINAHL, Scopus and Google Scholar. Only studies evaluating e-learning interventions in undergraduate clinical medical education between January 1990 and August 2021 were selected. Of the 4,829 papers identified by the search, 42 studies met the inclusion criteria.

RESULTS

The 42 studies included varied in scope, cognitive domain, subject matter, design, quality and evaluation. The most popular approaches involved multimedia platforms (33%) and case-based approaches (26%), were interactive (83%), asynchronous (71%) and accessible from home (83%). Twelve studies (29%) evaluated usability, all of which reported positive feedback. Competence in use of technology, high motivation and an open attitude were key characteristics of successful students and preceptors.

CONCLUSIONS

Medical education is evolving consistently to accommodate rapid changes in therapies and procedures. In today's technologically adept world, e-learning is an effective and convenient pedagogical approach for the teaching of undergraduate clinical medicine.

Attitudes of an international student cohort to the Quizlet study system employed in an advanced clinical health care review course

Computer-based learning applications and mobile technology have transformed many aspects of the educational experience over the last decade, producing software aimed at improving learning efficiency and streamlining the presentation of course materials. One such class of software, purpose-created to take advantage of spaced learning and spaced testing principles, are electronic flashcard applications. We provide a perspective on the novel use of the Quizlet flashcard application in a tertiary educational setting. To reduce cognitive load for international graduate dental students taking a pharmacology review course, we implemented Quizlet, which integrates both spaced learning and self-testing, to improve the student learning experience. This study assessed students' perceptions of the Quizlet flashcard system in a student cohort comprised of two consecutive years' classes (n = 51 students in total). Results indicated broad acceptance of Quizlet based on ease of use of the software and ease of study of the material. Our data provide insight into the use of this common software in a professional healthcare tertiary education setting and further demonstrate the successful application of electronic flashcards for a mixed international student cohort. Further research should include an assessment of the impact of flashcard on long-term knowledge retention in this setting.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s10639-022-11371-3.

Team-based learning (TBL): Each phase matters! An empirical study to explore the importance of each phase of TBL

CONTEXT

In Team Based Learning (TBL), it is, based on theory, assumed that knowledge development in each phase contributes to the subsequent phase and to learning performance. However, there is no empirical evidence for this assumption.

AIM

In order to find support for the relation between TBL and the underlying theory, we determined to what extent each phase of TBL is associated with the knowledge development in the next phase and with the total learning performance.

METHODS

We measured the scientific concepts recalled by 56 second-year undergraduate medical students before TBL, after each of the three phases and after TBL. We used multivariate regression analysis to determine the statistical association between the phases as well as the total learning performance.

RESULTS

Results showed that in each phase, students produced new concepts in addition to those previously recalled. Regression models showed statistically significant explained variance ranging from 0.19 to 0.26, between the three phases and the total learning performance.

DISCUSSION

Each phase of the TBL is significantly associated with knowledge development in the subsequent phase and with the total learning performance, and therefore matters. This study contributes to the scientific underpinning of TBL and offers leads to more elaborate research and interventions to improve TBL.

Enhancing Examination Success: the Cumulative Benefits of Self-Assessment Questions and Virtual Patient Cases

PURPOSE

Research on the learning benefits of the feedback-rich formative assessment environment of virtual patient cases (VPCs) has largely been limited to single institutions and focused on discrete clinical skills or topical knowledge. To augment current understanding, we designed a multi-institutional study to explore the distinct and cumulative effects of VPC formative assessments and optional self-assessment questions (SAQs) on exam performance.

METHOD

In this correlational study, we examined the records of 1,692 students on their family medicine (FM) clerkship at 20 medical schools during the 2014-2015 academic year. Schools utilized an established online curriculum, which included family medicine VPCs, embedded formative assessments, context-rich SAQs corresponding with each VPC, and an associated comprehensive family medicine exam. We used mixed-effects modeling to relate the student VPC composite formative assessment score, SAQ completion, and SAQ performance to students' scores on the FM final examination.

RESULTS

Students scored higher on the final exam when they performed better on the VPC formative assessments, completed associated SAQs, and scored higher on those SAQs. Students' SAQ completion enhanced examination performance above that explained by engagement with the VPC formative assessments alone.

CONCLUSIONS

This large-scale, multi-institutional study furthers the body of research on the effect of formative assessments associated with VPCs on exam performance and demonstrates the added benefit of optional associated SAQs. Findings highlight opportunities for future work on the broader impact of formative assessments for learning, exploring the benefits of integrating VPCs and SAQs, and documenting effects on clinical performance and summative exam scores.

The Use of a Modified Script Concordance Test in Clinical Rounds to Foster and Assess Clinical Reasoning Skills

The development of clinical reasoning skills is a high priority during clinical service, but an unpredictable case load and limited time for formal instruction makes it challenging for faculty to foster and assess students' individual clinical reasoning skills. We developed an assessment for learning activity that helps students build their clinical reasoning skills based on a modified version of the script concordance test (SCT). To modify the standard SCT, we simplified it by limiting students to a 3-point Likert scale instead of a 5-point scale and added a free-text box for students to provide justification for their answer. Students completed the modified SCT during clinical rounds to prompt a group discussion with the instructor. Student feedback was positive, and the instructor gained valuable insight into the students' thought process. A modified SCT can be adopted as part of a multimodal approach to teaching on the clinic floor. The purpose of this article is to describe our modifications to the standard SCT and findings from implementation in a clinical rounds setting as a method of formative assessment for learning and developing clinical reasoning skills.

Do Physician Assistant Students Use Retrieval-Based Strategies While Studying?

INTRODUCTION

Research on learning indicates that active retrieval of information (ie, testing) enhances student retention of knowledge, yet, it is underutilized by learners. This research investigated physician assistant (PA) students' study strategies and the extent to which retrieval-based strategies (RBS) are used.

METHODS

A survey instrument adapting items from Hartwig and Dunlosky's Study Habits Survey was administered to first-year PA students to investigate their study behaviors over a 4-week time frame in preparation for multiple-choice exams.

RESULTS

Seventy-eight students (88%) completed the survey. The most common study strategies were reviewing lecture slide decks (83%), reading notes (78%), and taking notes (73%). Retrieval-based study strategies most often used were practicing test questions (38%) and creating questions and responding to them (12%).

DISCUSSION

Many PA students are underutilizing retrieval-based study strategies and might benefit from more awareness of this learning approach as well as faculty guidance about how to incorporate it into their exam preparation.

Cancel culture: exploring the unintended consequences of cancelling the Canadian national licensing clinical examination

Assessment drives learning. However, when it comes to high-stakes examinations (e.g., for licensure or certification), these assessments of learning may be seen as unnecessary hurdles by some. Licensing clinical skills assessment in particular have come under fire over the years. Recently, assessments such as the Medical Council of Canada Qualifying Examination Part II, a clinical skills objective structured clinical examination, have been permanently cancelled. The authors explore potential consequences of this cancellation including those that are inadvertent and undesirable. Future next steps for clinical skills assessment are explored.

Flipped spotters learning model: An innovative student activity-based strategy. A preparation tool for anatomy practical examinations in medical education

The flipped spotters learning model is a modern student activity-based and learner-centered method in medical education. The aim of the study was to determine if the flipped spotters learning model improves students' learning. Participants were 1214 medical students of Polish (PD) and English (ED) divisions between 2013 and 2019 academic years at the University of Warmia and Mazury in Olsztyn, Poland. They were divided into a traditional group (control group) and a flipped spotters learning group (treatment group). Each flipped spotters learning group was asked to label anatomical structures on various specimens according to the structures name list prepared by the teacher on the multiple stations. The flipped spotters learning group leaders were instructed to take pictures with the appropriately marked structures on each of the human body prosections. After completion of the class, each flipped spotters team received photos for evaluation. In the flipped spotters learning model, the students strengthened their skills and knowledge by matching specimens independently as a form of practical laboratory activities. Students' performance in gross anatomy practical examinations between the group utilizing the flipped spotters learning model, and the group with the traditional teaching model was compared. Students participating in the treatment group achieved, on average 9.9 percentage points higher among PD students, and 13.0 percentage points higher among ED students than the control group in all nine practical examinations (the effect size ranging from 0.47 to 0.95). The results suggest the positive impact of flipped spotters model on improving student's performance in the practical examinations.

Application of test-enhanced learning (TEL) in obstetrics and gynecology: a prospective study

OBJECTIVE

Clinical reasoning is an essential skill, the foundations of which should be acquired during medical school. Within the format of test-based learning, such examinations can also be used to support the long-term retention of procedural knowledge necessary for clinical reasoning. The aim was to investigate whether repeated exposure to clinical cases in obstetrics and gynecology (OBGYN) with built-in questions leads to higher learning outcome than pure reading cases and what influence the delay between the intervention and the final test has on the retention of the respective content.

METHODS

In this non-randomised crossover study, 5th-year medical students (duration of the study is 6 years) taking a 1-week clinical attachment in OBGYN participated in computer-based case seminars in winter term 2020/2021, in which different case histories on gynecological-obstetric diseases were presented. Case content was identical for all groups, but the presentation format (cases with key feature questions vs read-only cases) of individual case vignettes changed weekly. The also intervention was repeated after 2 weeks for each group. Knowledge was assessed in an entry and an exit exam consisting of 40 short-answer questions.

RESULTS

A total of 94 out of 118 eligible students participated in the study (response rate: 79.7%). Learning outcome was significantly higher for items presented in the key feature format compared to items presented as read-only cases (74.2 ± 8.6% vs. 71.0 ± 9.2%; p = 0.017). Furthermore, the analysis showed that the temporal distance of the intervention package from the final examination had no influence on retention.

CONCLUSION

This is the first study to demonstrate an effect of test-enhanced learning on clinical reasoning in the subject of OGBYN. In this cross-over study, repeated testing was more effective than repeated case-based learning alone. Curricular implementation of longitudinal key feature testing can thus improve learning outcomes for OBGYN.