Informing the uninformed: a multitier approach to uncover students' misconceptions on cardiovascular physiology

Dissecting knowledge, guessing, and blunder in multiple choice assessments

Multiple choice results are inherently probabilistic outcomes, as correct responses reflect a combination of knowledge and guessing, while incorrect responses additionally reflect blunder, a confidently committed mistake. To objectively resolve knowledge from responses in an MC test structure, we evaluated probabilistic models that explicitly account for guessing, knowledge and blunder using eight assessments (>9,000 responses) from an undergraduate biotechnology curriculum. A Bayesian implementation of the models, aimed at assessing their robustness to prior beliefs in examinee knowledge, showed that explicit estimators of knowledge are markedly sensitive to prior beliefs with scores as sole input. To overcome this limitation, we examined self-ranked confidence as a proxy knowledge indicator. For our test set, three levels of confidence resolved test performance. Responses rated as least confident were correct more frequently than expected from random selection, reflecting partial knowledge, but were balanced by blunder among the most confident responses. By translating evidence-based guessing and blunder rates to pass marks that statistically qualify a desired level of examinee knowledge, our approach finds practical utility in test analysis and design.

Intuitive concepts in internal medicine and their occurrence in undergraduate medical students in different semesters

Background: Dealing with errors in medical practice is of great importance for patient safety. In the natural sciences, intuitive concepts, so-called misconceptions, are increasingly coming into focus of teaching because they lead to a faulty understanding of contexts and thus to faulty scientific reasoning. In medicine, intuitive concepts still play a subordinate role. However, once intuitive concepts have been memorized, they can become firmly established and, under certain circumstances, lead to diagnostic and treatment errors in medical practice. The aim of this study was to identify potential intuitive concepts in internal medicine and to analyze their occurrence in medical students from different semesters. Methods: Eight internists from different subspecialties were asked about intuitive concepts by means of a structured interview. A total of 17 intuitive concepts were identified. Using these concepts, a multiple-choice test was created with 17 patient cases. For each case, there were four possible answers: the correct answer, an incorrect answer that included the intuitive concept, the answer "both are incorrect", and the answer "I am not sure", which is to be understood in the sense of "I do not know whether one of the three answers is correct". As an online multiple-choice test, these 17 cases were made available to all 2^nd, 6^th, and 12^th semester students (N=1170, n=418 from the 2nd semester, n=425 from the 6^th semester, and n=327 from the 12^th semester, i.e., the final year) for four weeks in June 2015. The test had to be answered within nine minutes. A mixed logistic regression model was used for evaluation. Results: Of the N=317 participating students (n=97 from the 2^nd semester, n=124 from the 6^th semester, and n=96 from the internship year, overall response rate 27.1%), on average, students from all three groups chose the intuitive concept most often, approximately 40%, although the correct answer increased toward the final year with simultaneously decreasing uncertainty and decreasing feeling of not knowing, respectively. In the final year, compared to the 2^nd semester, the intuitive concept was selected significantly more often for two questions (p<0.01). For four questions, the intuitive concept was selected significantly less frequently in the final year (p<0.01). Conclusion: Intuitive concepts can be identified in internal medicine and do not appear to be significantly reduced in students during the course of their studies. This suggests that this could also be the case for other medical subjects. Therefore, similar studies should be conducted for other medical subjects in order to identify potential sources of error in clinical work. In addition, suitable didactic methods should be developed and tested with which students learn not to succumb to intuitive concepts as far as possible in order to prevent diagnostic or therapeutic errors in later medical practice.

What Were You Thinking? Medical Students' Metacognition and Perceptions of Self-Regulated Learning

PHENOMENON

As a component of self-regulated learning, metacognition is gaining attention in the medical education research community. Metacognition, simply put, is thinking about one's thinking. Having a metacognitive habit of mind is essential for healthcare professionals. This study identified the metacognitive competencies of medical students as they completed a conceptual learning task, and provided insight into students' perceptions of self-regulated learning in their curriculum. Approach: Eleven third-year medical students from a Dutch University were purposively sampled to participate in this qualitative study. The study design included a think-aloud assignment followed by a semi-structured interview. During the assignment, participants were instructed to think aloud while solving questions about medical physiological concepts such as blood flow, pressure, and resistance. Think-aloud data were collected through audiotaping and used to identify participants' metacognitive competencies. The assignment also served as a prompt for an interview in which participants were questioned about metacognitive knowledge, monitoring, experiences, and perceptions of self-regulated learning in their curriculum. All data were transcribed verbatim and analyzed iteratively using a template analysis. Findings: Students differed in their use of metacognitive skills, with an overall focus on monitoring and, to a lesser extent, on planning and evaluation. Additionally, differences were found in students' metacognitive knowledge and metacognitive experiences. There was apparent use of inefficient, superficial predictive cues. Regarding perceptions of self-regulated learning skills, some students felt no need to develop such skills as they perceived medical education as an exercise in memorizing facts. Others emphasized the need for more insight into their actual level of knowledge and competence. Insights: Pre-clinical medical students require explicit teaching of metacognitive skills to facilitate self-regulated learning. Educators should aim to integrate metacognition in the everyday discourse of the classroom to foster an environment in which students discuss their own learning.

Supplemental data for this article is available online at https://doi.org/10.1080/10401334.2021.1889559.

Putting post-decision wagering to the test: a measure of self-perceived knowledge in basic sciences?

INTRODUCTION

Students learn more effectively when they know what they do not know. Gaining insight into students' metacognitive awareness is needed as misalignment between actual and self-perceived knowledge impedes their learning process. The optimal method of measuring self-perceived knowledge is still under debate. In this study, we evaluate the use of psychology-derived post-decision wagering for mapping students self-perceived knowledge.

METHODS

Students (n = 71) performed a pre-test on medical physiology, followed by a teacher-moderated discussion and a post-test with isomorph questions. Half of the students rated their self-perceived knowledge on each question using post-decision wagering, i. e. betting 1-5 points on the correctness of their answer, whereas the other half used a 5-point Likert scale to rate their confidence.

RESULTS

Self-perceived knowledge scores were higher for post-decision wagering (pre: 3.75 ± 0.14, post: 4.60 ± 0.07) compared with Likert scales (pre: 3.13 ± 0.08, post: 3.92 ± 0.08) despite similar actual knowledge scores. Furthermore, Likert ratings showed a near-normal distribution, whereas wagers were placed preferentially using the outer ends of the scale. Correlations between mean actual and self-perceived knowledge scores were low in both groups. On average, 8.5% of responses were classified as misconceptions, defined as highly confident incorrect answers.

DISCUSSION

Despite the presumed reliability of post-decision wagering, our findings suggest that we should adhere to the use of Likert scales as a balanced measure for self-perceived knowledge in medical education. Moreover, the prevalence of misconceptions did not alter after instruction, indicating a need for instructional designs that enhance students' conceptual understanding in basic sciences.