Pattern recognition as a concept for multiple-choice questions in a national licensing exam

"Fit for the finals" - project report on a telemedical training with simulated patients, peers, and assessors for the licensing exam

Background

Undergraduate medical students take the licensing exam (M3) as a two-day oral-practical examination. The main requirements are to demonstrate history taking skills and coherent case presentations. The aim of this project was to establish a training in which students can test their communication skills during history taking and their clinical reasoning skills in focused case presentations.

Methods

In the newly developed training, final-year students took four telemedical histories in the role of physicians from simulated patients (SP). They received further findings for two SPs and presented these in a handover, in which they also received a handover of two SPs which they had not seen themselves. Each student presented one of the two received SPs in a case discussion with a senior physician. Feedback was given to the participants on their communication and interpersonal skills by the SPs with the ComCare questionnaire and on the case presentation by the senior physician. Sixty-two students from the universities of Hamburg and Freiburg in their final year participated in September 2022 and evaluated the training.

Results

Participants felt that the training was very appropriate for exam preparation. The SPs' feedback on communication and the senior physician's feedback on clinical reasoning skills received the highest ratings in importance to the students. Participants highly valued the practice opportunity for structured history taking and case presentation and would like to have more such opportunities in the curriculum.

Conclusion

Essential elements of the medical licensing exam can be represented, including feedback, in this telemedical training and it can be offered independent of location.

The Evolving Importance of Artificial Intelligence and Radiology in Medical Trainee Education

Radiology education is understood to be an important component of medical school and resident training, yet lacks a standardization of instruction. The lack of uniformity in both how radiology is taught and learned has afforded opportunities for new technologies to intervene. Now with the integration of artificial intelligence within medicine, it is likely that the current medical trainee curricula will experience the impact it has to offer both for education and medical practice. In this paper, we seek to investigate the landscape of radiologic education within the current medical trainee curricula, and also to understand how artificial intelligence may potentially impact the current and future radiologic education model.

What faculty write versus what students see? Perspectives on multiple-choice questions using Bloom's taxonomy

BACKGROUND

Using revised Bloom's taxonomy, some medical educators assume they can write multiple choice questions (MCQs) that specifically assess higher (analyze, apply) versus lower-order (recall) learning. The purpose of this study was to determine whether three key stakeholder groups (students, faculty, and education assessment experts) assign MCQs the same higher- or lower-order level.

METHODS

In Phase 1, stakeholders' groups assigned 90 MCQs to Bloom's levels. In Phase 2, faculty wrote 25 MCQs specifically intended as higher- or lower-order. Then, 10 students assigned these questions to Bloom's levels.

RESULTS

In Phase 1, there was low interrater reliability within the student group (Krippendorf's alpha = 0.37), the faculty group (alpha = 0.37), and among three groups (alpha = 0.34) when assigning questions as higher- or lower-order. The assessment team alone had high interrater reliability (alpha = 0.90). In Phase 2, 63% of students agreed with the faculty as to whether the MCQs were higher- or lower-order. There was low agreement between paired faculty and student ratings (Cohen's Kappa range .098-.448, mean .256).

DISCUSSION

For many questions, faculty and students did not agree whether the questions were lower- or higher-order. While faculty may try to target specific levels of knowledge or clinical reasoning, students may approach the questions differently than intended.

Mastering Electrocardiogram Interpretation Skills Through a Perceptual and Adaptive Learning Module

Although accurate interpretation of the standard 12-lead electrocardiogram (ECG) is fundamental to diagnosing heart disease, several prior studies report low accuracy rates among medical students, residents, and practicing physicians. The objective of this study was to determine if an online ECG Perceptual and Adaptive Learning Module (ECG PALM) is an efficient instrument to teach ECG interpretation. The ECG PALM consists of 415 unique ECG tracings with associated pretest, posttest, and delayed tests, each using 30 additional ECGs to gauge the effectiveness and durability of training. Between 2013 and 2015, a total of 113 third-year and 156 fourth-year medical students and 34 first-year, 41 second-year, and 37 third-year emergency medicine residents completed the PALM and associated tests. We measured two mastery criteria: accuracy, the percentage of correct interpretations, and fluency, the percentage of images interpreted accurately within 15 seconds. The ECG PALM produced statistically significant improvements (0.0001 < p < 0.0045) in student and resident performance for both accuracy (effect size = 0.9 to 3.2) and fluency (effect size = 2.5 to 3.1) following training ranging from 46 ± 24 minutes (R3s) to 88 ± 32 minutes (third-year medical students). Medical students and residents performed significantly better on a test the year following training (delayed test) than those without prior ECG PALM training (pretest). The fluency of R3 residents in classifying the 15 diagnostic categories was less than 60% for nine of the 15 diagnoses and greater than 80% for only one. Following PALM training, fluency was higher than 80% for seven of the 15 categories and less than 60% for only two categories. Accuracy in recognizing ST-elevation myocardial infarctions (STEMIs) was high both before and after PALM training for R3s, but fluency was only 64% for anterior STEMIs on the pretest, increasing to 93% following PALM training. These observations suggest that the ECG PALM is an effective and durable supplemental tool for developing mastery in interpreting common ECG abnormalities.

Five decades of research and theorization on clinical reasoning: a critical review

Clinical reasoning is a complex cognitive process that is essential to evaluate and manage a patient's medical problem. The aim of this paper was to provide a critical review of the research literature on clinical reasoning theories and models. To conduct our study, we applied the process of conducting a literature review in four stages in accordance with the approach of Carnwell and Daly. First, we defined the scope of the review as being limited to clinical reasoning theories and models in medical education. In the second stage, we conducted a search based on related words in PubMed, Google Scholar, PsycINFO, ERIC, ScienceDirect and Web of Science databases. In the third stage, we classified the results of the review into three categories, and in the fourth stage, we concluded and informed further studies. Based on the inclusion and exclusion criteria, 31 articles were eligible to be reviewed. Three theories and two models were recognized and classified into three categories. Several theories and models have been proposed in relation to clinical reasoning, but it seems that these theories and models could only explain part of this complex process and not the whole process. Therefore, to fulfill this gap, it may be helpful to build a Meta-model or Meta-theory, which unified all the models, and theories of clinical reasoning.

[Competence-based assessment in the national licensing examination in Germany]

In Germany, future physicians have to pass a national licensing examination at the end of their medical studies. Passing this examination is the requirement for the license to practice medicine. The Masterplan Medizinstudium 2020 with its 41 measures aims to shift the paradigm in medical education and medical licensing examinations.The main goals of the Masterplan include the development towards competency-based and practical medical education and examination as well as the strengthening of general medicine. The healthcare policy takes into account social developments, which are very important for the medical education and licensing examination.Seven measures of the Masterplan relate to the realignment of the licensing examinations. Their function to drive learning should better support students in achieving the study goal defined in the German Medical Licensure Act: to educate a medical doctor scientifically and practically who is qualified for autonomous and independent professional practice, postgraduate education and continuous training.

Assessing core competences of medical students with a test for flight school applicants

BACKGROUND

Important competences of physicians regarding patient safety include communication, leadership, stress resistance, adherence to procedures, awareness, and teamwork. Similarly, while selected, prospective flight school applicants are tested for the same set of skills. The aim of our study was to assess these core competences in advanced undergraduate medical students from different medical schools.

METHODS

In 2017, 67 medical students (year 5 and 6) from the universities of Hamburg, Oldenburg, and TU Munich, Germany, participated in the verified Group Assessment Performance (GAP)-Test at the German Aerospace Center (DLR) in Hamburg. All participants were rated by DLR assessment observers with a set of empirically derived behavioural checklists. This lists consisted of 6-point rating scales (1: very low occurrence to 6: very high occurrence) and included the competences leadership, teamwork, stress resistance, communication, awareness, and adherence to procedures. Medical students' scores were compared with the results of 117 admitted flight school applicants.

RESULTS

Medical students showed significantly higher scores than admitted flight school applicants for adherence to procedures (p < .001, d = .63) and communication (p < .01, d = .62). They reached significantly lower ratings for teamwork (p < .001, d = .77), stress resistance (p < 0.001, d = .70), and awareness (p < .001, d = 1.31). Students in semester 10 showed significantly (p < .02, d = .58) higher scores in domain awareness compared to the final year students. On average, flight school entrance level was not reached by either group for this domain.

CONCLUSIONS

Advanced medical students' low results for awareness are alarming as awareness is essential and integrative for clinical reasoning and patient safety. Further studies should elucidate and discuss whether awareness needs to be included in medical student selection or integrated into the curriculum in training units.

Pushing Critical Thinking Skills With Multiple-Choice Questions: Does Bloom's Taxonomy Work?

Medical school assessments should foster the development of higher-order thinking skills to support clinical reasoning and a solid foundation of knowledge. Multiple-choice questions (MCQs) are commonly used to assess student learning, and well-written MCQs can support learner engagement in higher levels of cognitive reasoning such as application or synthesis of knowledge. Bloom's taxonomy has been used to identify MCQs that assess students' critical thinking skills, with evidence suggesting that higher-order MCQs support a deeper conceptual understanding of scientific process skills. Similarly, clinical practice also requires learners to develop higher-order thinking skills that include all of Bloom's levels. Faculty question writers and examinees may approach the same material differently based on varying levels of knowledge and expertise, and these differences can influence the cognitive levels being measured by MCQs. Consequently, faculty question writers may perceive that certain MCQs require higher-order thinking skills to process the question, whereas examinees may only need to employ lower-order thinking skills to render a correct response. Likewise, seemingly lower-order questions may actually require higher-order thinking skills to respond correctly. In this Perspective, the authors describe some of the cognitive processes examinees use to respond to MCQs. The authors propose that various factors affect both the question writer and examinee's interaction with test material and subsequent cognitive processes necessary to answer a question.

Implementation of a Clinical Reasoning Course in the Internal Medicine trimester of the final year of undergraduate medical training and its effect on students' case presentation and differential diagnostic skills

Background: Clinical reasoning, comprising the processes of clinical thinking, which form the basis of medical decisions, constitutes a central competence in the clinical routine on which diagnostic and therapeutic steps are based. In medical curricula in Germany, clinical reasoning is currently taught explicitly only to a small extend. Therefore, the aim of this project was to develop and implement a clinical reasoning course in the final year of undergraduate medical training. Project description: A clinical reasoning course with six learning units and 18 learning objectives was developed, which was taught by two to four instructors on the basis of 32 paper cases from the clinical practice of the instructors. In the years 2011 to 2013, the course of eight weeks with two hours per week was taught seven times. Before the first and after the last seminar, the participating students filled out a self-assessment questionnaire with a 6-point Likert scale regarding eight different clinical reasoning skills. At the same times, they received a patient case with the assignment to prepare a case presentation and differential diagnoses. Results: From 128 participating students altogether, 42 complete data sets were available. After the course, participants assessed themselves significantly better than before the course in all eight clinical reasoning skills, for example in "Summarizing and presentation of a paper case" or in the "Skill to enumerate differential diagnoses" (p<0.05). The greatest increase occurred in the skill to recognize typical cognitive errors in medicine and to identify risk situations for their occurrence (pre: 2.98±0.92 and retro-pre: 2.64±1.01, respectively, versus post: 4.38±0.88). Based on the ratio of number of words used per keywords used the problem presentation of the paper case was significantly more focused after the course (p=0.011). A significant increase in the number of gathered differential diagnoses was not detected after the course. Conclusion: The newly developed and established Clinical Reasoning Course leads to a gain in the desired skills from the students' self-assessment perspective and to a more structured case presentation. To establish better options to exercise clinical reasoning, a longitudinal implementation in the medical curriculum seems to be desirable. Faculty training would be useful to implement the concept as standardized as possible.

Technical flaws in multiple-choice questions in the access exam to medical specialties ("examen MIR") in Spain (2009-2013)

BACKGROUND

The main factor that determines the selection of a medical specialty in Spain after obtaining a medical degree is the MIR ("médico interno residente", internal medical resident) exam. This exam consists of 235 multiple-choice questions with five options, some of which include images provided in a separate booklet. The aim of this study was to analyze the technical quality of the multiple-choice questions included in the MIR exam over the last five years.

METHODS

All the questions included in the exams from 2009 to 2013 were analyzed. We studied the proportion of questions including clinical vignettes, the number of items related to an image and the presence of technical flaws in the questions. For the analysis of technical flaws, we adapted the National Board of Medical Examiners (NBME) guidelines. We looked for 18 different issues included in the manual, grouped into two categories: issues related to testwiseness and issues related to irrelevant difficulties.

RESULTS

The final number of questions analyzed was 1,143. The percentage of items based on clinical vignettes increased from 50% in 2009 to 56-58% in the following years (2010-2013). The percentage of items based on an image increased progressively from 10% in 2009 to 15% in 2012 and 2013. The percentage of items with at least one technical flaw varied between 68 and 72%. We observed a decrease in the percentage of items with flaws related to testwiseness, from 30% in 2009 to 20% in 2012 and 2013. While most of these issues decreased dramatically or even disappeared (such as the imbalance in the correct option numbers), the presence of non-plausible options remained frequent. With regard to technical flaws related to irrelevant difficulties, no improvement was observed; this is especially true with respect to negative stem questions and "hinged" questions.

CONCLUSION

The formal quality of the MIR exam items has improved over the last five years with regard to testwiseness. A more detailed revision of the items submitted, checking systematically for the presence of technical flaws, could improve the validity and discriminatory power of the exam, without increasing its difficulty.