What we measure … and what we should measure in medical education

Post competency training in standardized training of resident physicians and integrated postgraduates

We focus on the importance and necessity of post-competency-based training in medical education alongside its current challenges and opportunities. Integrating post competency training into standardized educational frameworks is increasingly recognized as a critical component of preparing residents and postgraduates for real-world clinical practice. Post competency training represents a commitment to excellence in medical education, striving to produce competent, skilled practitioners to meet the challenges of modern healthcare.

Primary care physicians' knowledge and confidence in providing cancer survivorship care: a systematic review

PURPOSE

To systematically review existing literature on knowledge and confidence of primary care physicians (PCPs) in cancer survivorship care.

METHODS

PubMed, Ovid MEDLINE, CINAHL, Embase, and PsycINFO were searched from inception to July 2022 for quantitative and qualitative studies. Two reviewers independently assessed studies for eligibility and quality. Outcomes were characterized by domains of quality cancer survivorship care.

RESULTS

Thirty-three papers were included, representing 28 unique studies; 22 cross-sectional surveys, 8 qualitative, and 3 mixed-methods studies. Most studies were conducted in North America (n = 23) and Europe (n = 8). For surveys, sample sizes ranged between 29 and 1124 PCPs. Knowledge and confidence in management of physical (n = 19) and psychosocial effects (n = 12), and surveillance for recurrences (n = 14) were described most often. Generally, a greater proportion of PCPs reported confidence in managing psychosocial effects (24-47% of PCPs, n= 5 studies) than physical effects (10-37%, n = 8). PCPs generally thought they had the necessary knowledge to detect recurrences (62-78%, n = 5), but reported limited confidence to do so (6-40%, n = 5). There was a commonly perceived need for education on long-term and late physical effects (n = 6), and cancer surveillance guidelines (n = 9).

CONCLUSIONS

PCPs' knowledge and confidence in cancer survivorship care varies across care domains. Suboptimal outcomes were identified in managing physical effects and recurrences after cancer.

IMPLICATIONS FOR CANCER SURVIVORS

These results provide insights into the potential role of PCPs in cancer survivorship care, medical education, and development of targeted interventions.

Knowledge Mapping and Global Trends in the Field of the Objective Structured Clinical Examination: Bibliometric and Visual Analysis (2004-2023)

BACKGROUND

The Objective Structured Clinical Examination (OSCE) is a pivotal tool for assessing health care professionals and plays an integral role in medical education.

OBJECTIVE

This study aims to map the bibliometric landscape of OSCE research, highlighting trends and key influencers.

METHODS

A comprehensive literature search was conducted for materials related to OSCE from January 2004 to December 2023, using the Web of Science Core Collection database. Bibliometric analysis and visualization were performed with VOSviewer and CiteSpace software tools.

RESULTS

Our analysis indicates a consistent increase in OSCE-related publications over the study period, with a notable surge after 2019, culminating in a peak of activity in 2021. The United States emerged as a significant contributor, responsible for 30.86% (1626/5268) of total publications and amassing 44,051 citations. Coauthorship network analysis highlighted robust collaborations, particularly between the United States and the United Kingdom. Leading journals in this domain-BMC Medical Education, Medical Education, Academic Medicine, and Medical Teacher-featured the highest volume of papers, while The Lancet garnered substantial citations, reflecting its high impact factor (to be verified for accuracy). Prominent authors in the field include Sondra Zabar, Debra Pugh, Timothy J Wood, and Susan Humphrey-Murto, with Ronaldo M Harden, Brian D Hodges, and George E Miller being the most cited. The analysis of key research terms revealed a focus on "education," "performance," "competence," and "skills," indicating these are central themes in OSCE research.

CONCLUSIONS

The study underscores a dynamic expansion in OSCE research and international collaboration, spotlighting influential countries, institutions, authors, and journals. These elements are instrumental in steering the evolution of medical education assessment practices and suggest a trajectory for future research endeavors. Future work should consider the implications of these findings for medical education and the potential areas for further investigation, particularly in underrepresented regions or emerging competencies in health care training.

Authentic assessment in medical education: exploring AI integration and student-as-partners collaboration

BACKGROUND

Traditional assessments often lack flexibility, personalized feedback, real-world applicability, and the ability to measure skills beyond rote memorization. These may not adequately accommodate diverse learning styles and preferences, nor do they always foster critical thinking or creativity. The inclusion of Artificial Intelligence (AI), especially Generative Pre-trained Transformers, in medical education marks a significant shift, offering both exciting opportunities and notable challenges for authentic assessment practices. Various fields, including anatomy, physiology, pharmacy, dentistry, and pathology, are anticipated to employ the metaverse for authentic assessments increasingly. This innovative approach will likely enable students to engage in immersive, project-based learning experiences, facilitating interdisciplinary collaboration and providing a platform for real-world application of knowledge and skills.

METHODS

This commentary paper explores how AI, authentic assessment, and Student-as-Partners (SaP) methodologies can work together to reshape assessment practices in medical education.

RESULTS

The paper provides practical insights into effectively utilizing AI tools to create authentic assessments, offering educators actionable guidance to enhance their teaching practices. It also addresses the challenges and ethical considerations inherent in implementing AI-driven assessments, emphasizing the need for responsible and inclusive practices within medical education. Advocating for a collaborative approach between AI and SaP methodologies, the commentary proposes a robust plan to ensure ethical use while upholding academic integrity.

CONCLUSION

Through navigating emerging assessment paradigms and promoting genuine evaluation of medical knowledge and proficiency, this collaborative effort aims to elevate the quality of medical education and better prepare learners for the complexities of clinical practice.

Physical and biophysical markers of assessment in medical training: A scoping review of the literature

PURPOSE

Assessment in medical education has changed over time to measure the evolving skills required of current medical practice. Physical and biophysical markers of assessment attempt to use technology to gain insight into medical trainees' knowledge, skills, and attitudes. The authors conducted a scoping review to map the literature on the use of physical and biophysical markers of assessment in medical training.

MATERIALS AND METHODS

The authors searched seven databases on 1 August 2022, for publications that utilized physical or biophysical markers in the assessment of medical trainees (medical students, residents, fellows, and synonymous terms used in other countries). Physical or biophysical markers included: heart rate and heart rate variability, visual tracking and attention, pupillometry, hand motion analysis, skin conductivity, salivary cortisol, functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS). The authors mapped the relevant literature using Bloom's taxonomy of knowledge, skills, and attitudes and extracted additional data including study design, study environment, and novice vs. expert differentiation from February to June 2023.

RESULTS

Of 6,069 unique articles, 443 met inclusion criteria. The majority of studies assessed trainees using heart rate variability (n = 160, 36%) followed by visual attention (n = 143, 32%), hand motion analysis (n = 67, 15%), salivary cortisol (n = 67, 15%), fMRI (n = 29, 7%), skin conductivity (n = 26, 6%), fNIRs (n = 19, 4%), and pupillometry (n = 16, 4%). The majority of studies (n = 167, 38%) analyzed non-technical skills, followed by studies that analyzed technical skills (n = 155, 35%), knowledge (n = 114, 26%), and attitudinal skills (n = 61, 14%). 169 studies (38%) attempted to use physical or biophysical markers to differentiate between novice and expert.

CONCLUSION

This review provides a comprehensive description of the current use of physical and biophysical markers in medical education training, including the current technology and skills assessed. Additionally, while physical and biophysical markers have the potential to augment current assessment in medical education, there remains significant gaps in research surrounding reliability, validity, cost, practicality, and educational impact of implementing these markers of assessment.

A "Red Flag" system adds value to medical school admissions interviews

INTRODUCTION

Non-cognitive traits should be considered when selecting candidates to study medicine. However, evaluating these traits remains difficult. We explored whether measuring undesirable non-cognitive behaviour ('Red Flags') added value to a medical school admissions system. Red Flags included rudeness, ignoring the contributions of others, disrespectful behaviour, or poor communication.

METHODS

Following an admissions interview testing non-cognitive attributes in 648 applicants to a UK medical school, we measured the association between interview score and Red Flag frequency. We tested linear and polynomial regression models to evaluate whether the association was linear or non-linear.

RESULTS

In total, 1126 Red Flags were observed. While Red Flags were concentrated among low-scorers, candidates in the highest- and second-highest deciles for interview score still received Red Flags (six and twenty-two, respectively). The polynomial regression model indicated candidates with higher scores received fewer Red Flags, but the association was not linear (F(3644) = 159.8, p = .001, adjusted R2 = 0.42).

CONCLUSIONS

The non-linear association between interview score and Red Flag frequency shows some candidates with desirable non-cognitive attributes will still display undesirable-or even exclusionary-non-cognitive attributes. Recording Red Flag behaviour reduces the likelihood such candidates will be offered a place at medical school.

Feedback, fairness, and validity: effects of disclosing and reusing multiple-choice questions in medical schools

BACKGROUND

Disclosure of items used in multiple-choice-question (MCQ) exams may decrease student anxiety and improve transparency, feedback, and test-enhanced learning but potentially compromises the reliability and fairness of exams if items are eventually reused. Evidence regarding whether disclosure and reuse of test items change item psychometrics is scarce and inconclusive.

METHODS

We retrospectively analysed difficulty and discrimination coefficients of 10,148 MCQ items used between fall 2017 and fall 2019 in a large European medical school in which items were disclosed from fall 2017 onwards. We categorised items as 'new'; 'reused, not disclosed'; or 'reused, disclosed'. For reused items, we calculated the difference from their first ever use, that is, when they were new. Differences between categories and terms were analysed with one-way analyses of variance and independent-samples t tests.

RESULTS

The proportion of reused, disclosed items grew from 0% to 48.4%; mean difficulty coefficients increased from 0.70 to 0.76; that is, items became easier, P < .001, η_p² = 0.011. On average, reused, disclosed items were significantly easier (M = 0.83) than reused, not disclosed items (M = 0.71) and entirely new items (M = 0.66), P < .001, η_p² = 0.087. Mean discrimination coefficients increased from 0.21 to 0.23; that is, item became slightly more discriminating, P = .002, η_p² = 0.002.

CONCLUSIONS

Disclosing test items provides the opportunity to enhance feedback and transparency in MCQ exams but potentially at the expense of decreased item reliability. Discrimination was positively affected. Our study may help weigh advantages and disadvantages of using previously disclosed items.

Diagnosing virtual patients: the interplay between knowledge and diagnostic activities

Clinical reasoning theories agree that knowledge and the diagnostic process are associated with diagnostic success. However, the exact contributions of these components of clinical reasoning to diagnostic success remain unclear. This is particularly the case when operationalizing the diagnostic process with diagnostic activities (i.e., teachable practices that generate knowledge). Therefore, we conducted a study investigating to what extent knowledge and diagnostic activities uniquely explain variance in diagnostic success with virtual patients among medical students. The sample consisted of N = 106 medical students in their third to fifth year of university studies in Germany (6-years curriculum). Participants completed professional knowledge tests before diagnosing virtual patients. Diagnostic success with the virtual patients was assessed with diagnostic accuracy as well as a comprehensive diagnostic score to answer the call for more extensive measurement of clinical reasoning outcomes. The three diagnostic activities hypothesis generation, evidence generation, and evidence evaluation were tracked. Professional knowledge predicted performance in terms of the comprehensive diagnostic score and displayed a small association with diagnostic accuracy. Diagnostic activities predicted comprehensive diagnostic score and diagnostic accuracy. Hierarchical regressions showed that the diagnostic activities made a unique contribution to diagnostic success, even when knowledge was taken into account. Our results support the argument that the diagnostic process is more than an embodiment of knowledge and explains variance in diagnostic success over and above knowledge. We discuss possible mechanisms explaining this finding.

Equivalent Performance of Exam Items Associated with Case-Based Learning, Flipped Classroom, and Lecture in a Pre-clerkship Medical Curriculum

The purpose of our study was to determine if knowledge acquisition, as measured by exam item performance, differed for active or passive learning activities in our medical curriculum. Additionally, we looked for differences in exam item performance in one second-year course that varies the method of an active learning activity, case-based collaborative learning (CBCL). Finally, we assessed whether item performance was impacted when small group activities were conducted online due to the COVID-19 pandemic. Exam item difficulty values were collected for several years of lectures, flipped classroom, and CBCL. Statistical analysis and modeling of data were performed to identify differences in difficulty of exam items that assess content delivered by different learning activities. Our analysis revealed no differences in difficulty of exam items that assess content delivered by different learning activities. Similarly, we determined that varying the execution of CBCL in one course did not impact exam item performance. Finally, moving CBCL small group sessions online did not impact exam item difficulty. However, we did detect a minor reduction in overall exam scores for the period of online instruction. Our results indicate that knowledge acquisition, as assessed by our multiple-choice summative exams, was equivalent regardless of learning activity modality.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-023-01842-8.

Impact of teaching, learning, and assessment of medical law on cognitive, affective and psychomotor skills of medical students: a systematic review

BACKGROUND

It is necessary to improve medical students' legal cognitive, affective, and psychomotor skills to prevent further legal issues in the medical profession. Choosing the proper teaching and assessment methods is crucial in this matter. This study aimed to investigate the impact of teaching, learning, and assessment of medical law on the cognitive, affective, and psychomotor skills of medical students.

METHODS

A systematic review was conducted in PubMed, Embass, and Web of Science databases, and Google Scholar search engine using MECIR and PRISMA, AMEE Guide 94 for 1980 to 2022.12.30. Nineteen articles met the inclusion criteria. MERSQI checklist was used to assess the quality of the articles, and URSEC (theoretical underpinning of the development, resources required, setting, educational methods employed, and content) used to assess the risk of educational bias.

RESULTS

Internship courses called Medical Education Humanities and Society (MESH), clinical scenario design, seminars and small group discussions, web-based interactive training, legal training courses, PBL, and mind maps have been used to improve the medico-legal knowledge of medical students. MESH clerkship, simulation of a legal event, medico-legal advocacy program based on interdisciplinary education, group discussion, integration, and court-based learning used to improve student attitudes. Multidisciplinary training, small group discussions after the seminar, mock trial competition, and interdisciplinary education are used to teach psychomotor skills. All studies, except one on knowledge, reported positive effects of legal education on students' knowledge, attitudes, and legal performance. Written assessments were used for cognitive and affective domains, while performance was assessed by OSCE, simulated court, and evaluation of patient referrals.

CONCLUSION

There are few studies to examine the cognitive, affective, and legal psychomotor skills of medical students. The texts have not yet fully explored the high level of affective and psychomotor domains, which is evidence of a gap in this sector. Recognizing that medico-legal problems are prevented through proper education and assessment, it is recommended that this area be considered a research priority and that effective educational policies are adopted.

Do Moral Judgments in Moral Dilemmas Make One More Inclined to Choose a Medical Degree?

The role of moral intuitions and moral judgments has become increasingly prominent in educational and academic choices. The present research aims to examine if the moral judgments elicited in sacrificial trolley dilemmas have a distinct pattern for the decisions made by junior medical students, in comparison to those of senior high school students. We work with this sample because it represents the population out of which medical students are recruited in the case of Bucharest, Romania. Our findings show that moral judgments are indeed a significant predictor for a respondent's status as medical students. This result, albeit with limitations, bears multiple practical implications, from developing empirically informed medical ethics courses in medical schools to evidence-based policy designs which consider factors such as morality alongside financial outcomes and incentives.

Improving the resident assessment process: application of App-based e-training platform and lean thinking

BACKGROUND

The assessment system for standardized resident training is crucial for developing competent doctors. However, it is complex, making it difficult to manage. The COVID-19 pandemic has also aggravated the difficulty of assessment. We, therefore, integrated lean thinking with App-based e-training platform to improve the assessment process through Define-Measure-Analyze-Improve-Control (DMAIC) cycles. This was designed to avoid unnecessary activities that generate waste.

METHODS

Panels and online surveys were conducted in 2021-2022 to find the main issues that affect resident assessment and the root causes under the frame of waste. An online app was developed. Activities within the process were improved by brainstorming. Online surveys were used to improve the issues, satisfaction, and time spent on assessment using the app.

RESULTS

A total of 290 clinical educators in 36 departments responded to the survey, and 153 clinical educators used the online app for assessment. Unplanned delay or cancellation was defined as the main issue. Eleven leading causes accounted for 87.5% of the issues. These were examiner time conflict, student time conflict, insufficient examiners, supervisor time conflict, grade statistics, insufficient exam assistants, reporting results, material archiving, unfamiliarity with the process, uncooperative patients, and feedback. The median rate of unplanned delay or cancellation was lower with use of the app (5% vs 0%, P < 0.001), and satisfaction increased (P < 0.001). The median time saved by the app across the whole assessment process was 60 (interquartile range 60-120) minutes.

CONCLUSIONS

Lean thinking integrated with an App-based e-training platform could optimize the process of resident assessment. This could reduce waste and promote teaching and learning in medical education.

Medical Council of Canada Qualifying Examinations and performance in future practice

The purpose of medical licensing examinations is to protect the public from practitioners who do not have adequate knowledge, skills, and abilities to provide acceptable patient care, and therefore evaluating the validity of these examinations is a matter of accountability. Our objective was to discuss the Medical Council of Canada's Qualifying Examinations (MCCQEs) Part I (QE1) and Part II (QE2) in terms of how well they reflect future performance in practice. We examined the supposition that satisfactory performance on the MCCQEs are important determinants of practice performance and, ultimately, patient outcomes. We examined the literature before the implementation of the QE2 (pre-1992), post QE2 but prior to the implementation of the new Blueprint (1992-2018), and post Blueprint (2018-present). The literature suggests that MCCQE performance is predictive of future physician behaviours, that the relationship between examination performance and outcomes did not attenuate with practice experience, and that associations between examination performance and outcomes made sense clinically. While the evidence suggests the MCC qualifying examinations measure the intended constructs and are predictive of future performance, the validity argument is never complete. As new competency requirements emerge, we will need to develop valid and reliable mechanisms for determining practice readiness in these areas.

Four ways to get a grip on making robust decisions from workplace-based assessments

Synthesising the results of workplace-based assessments to inform robust decisions is seen as both important and difficult. Concerns about failing to fail the trainee not ready to proceed has drawn disproportionate attention to assessors. This paper proposes a model for a more systems-based view so that the value of the assessor's judgement is incorporated while preserving the value and robustness of collective decision-making. Our experience has shown it can facilitate robust decisions on some of the more difficult areas, such as professionalism.

Types of clinical reasoning in a summative clerkship oral examination

INTRODUCTION

Dual-process theory characterizes clinical reasoning (CR) as Type 1 (intuitive) and Type 2 (analytical) thinking. This study examined CR on a summative clinical clerkship structured clinical oral examination (SCOE).

METHODS AND SUBJECTS

511 clinical clerks at the University of Toronto underwent SCOEs. Type 1, Type 2, and Global CR performance were compared to other internal medicine clerkship assessments using descriptive statistics and Spearman correlations.

RESULTS

Clinical clerks achieved mean marks >75% on the three clinical reasoning stations, on Type 1 and 2 CR tasks, and the overall SCOE. Performance on the SCOE CR stations correlated with each of the other clerkship assessments: written examination, inpatient, and ambulatory clinic assessments. The correlation of performance between Type 1 and Type 2 clinical reasoning tasks was statistically significant but weak (r_s = 0.28). This suggests that defined measures of Type 1 and Type 2 reasoning were indeed assessing distinct constructs.

CONCLUSION

Clinical clerks used both Type 1 and Type 2 reasoning with success. This study's characterization of Type 1 and Type 2 CR as separate domains, distinct from existing measures on the SCOE as well as the other clerkship assessments, can suggest a further addition to multimodal clerkship assessment.

Validity evidence for summative performance evaluations in postgraduate community pharmacy education

INTRODUCTION

Workplace-based assessment of competencies is complex. In this study, the validity of summative performance evaluations (SPEs) made by supervisors in a two-year longitudinal supervisor-trainee relationship was investigated in a postgraduate community pharmacy specialization program in the Netherlands. The construct of competence was based on an adapted version of the 2005 Canadian Medical Education Directive for Specialists (CanMEDS) framework.

METHODS

The study had a case study design. Both quantitative and qualitative data were collected. The year 1 and year 2 SPE scores of 342 trainees were analyzed using confirmatory factor analysis and generalizability theory. Semi-structured interviews were held with 15 supervisors and the program director to analyze the inferences they made and the impact of SPE scores on the decision-making process.

RESULTS

A good model fit was found for the adapted CanMEDS based seven-factor construct. The reliability/precision of the SPE measurements could not be completely isolated, as every trainee was trained in one pharmacy and evaluated by one supervisor. Qualitative analysis revealed that supervisors varied in their standards for scoring competencies. Some supervisors were reluctant to fail trainees. The competency scores had little impact on the high-stakes decision made by the program director.

CONCLUSIONS

The adapted CanMEDS competency framework provided a valid structure to measure competence. The reliability/precision of SPE measurements could not be established and the SPE measurements provided limited input for the decision-making process. Indications of a shadow assessment system in the pharmacies need further investigation.

The pursuit of fairness in assessment: Looking beyond the objective

Health professions education has undergone significant changes over the last few decades, including the rise of competency-based medical education, a shift to authentic workplace-based assessments, and increased emphasis on programmes of assessment. Despite these changes, there is still a commonly held assumption that objectivity always leads to and is the only way to achieve fairness in assessment. However, there are well-documented limitations to using objectivity as the 'gold standard' to which assessments are judged. Fairness, on the other hand, is a fundamental quality of assessment and a principle that almost no one contests. Taking a step back and changing perspectives to focus on fairness in assessment may help re-set a traditional objective approach and identify an equal role for subjective human judgement in assessment alongside objective methods. This paper explores fairness as a fundamental quality of assessments. This approach legitimises human judgement and shared subjectivity in assessment decisions alongside objective methods. Widening the answer to the question: 'What is fair assessment' to include not only objectivity but also expert human judgement and shared subjectivity can add significant value in ensuring learners are better equipped to be the health professionals required of the 21st century.

The European Examination in Core Cardiology in Focus: Evaluation and Recommendations Using Educational Theory

The European Examination in Core Cardiology (EECC) is a knowledge-based postgraduate examination for cardiology specialists in Europe. It is designed to assess if a trainee has gained sufficient knowledge for independent specialist practice in core cardiology. A critical evaluation of the ECCC was undertaken using current educational theory. Miller’s Pyramid was considered, and the Utility Equation was employed in a mixed methods approach. The utility analysis findings were that the EECC measured well on reliability and validity although improvement could be made in educational impact, cost-effectiveness and accessibility. Recommendations for enhancement were then put forward. No assessment instrument is perfect, and it is important to remember that the EECC is one component of assessment strategy for specialist trainees, complementing other evidence of professional competence. After appraisal, while improvement can be made, the EECC fulfils its ambitions of assessment.

Longitudinal survey of self-reported level of entrustment across the first year of clinical practice

Background. Newly graduated medical students often report that they lack the skills required to care for patients, and feel unprepared for clinical practice. However, little is known about when, and if, they acquire these skills in practice. The aim of this study was to assess self-reported level of entrustment in, and frequency of performance of, the seven Entrustable Professional Activities (EPAs) comprising the EPA framework for interns in Ireland. EPAs describe discrete activities that are essential to a particular profession. Methods. Self-report survey data were collected from doctors in the Republic of Ireland during their first year of clinical practice at four different time points during 2020/21. Results. Response rates to the survey varied from 73.3% (126/172) at Time 1 to 25.6% (44/172) at Time 4. After three months, the respondents reported that they could execute all of the EPAs, inclusive of 12 essential procedural skills, with indirect supervision. As the year progressed there was an increase in the proportion of respondents reporting that they performed the EPAs at least once a week. However, the proportion of respondents performing five of the essential procedural skills (e.g. nasogastric tube insertion) remained low across all time points. Conclusion. Consideration should be given as to how to better prepare medical students to execute these EPAs, how the interns can be better supported during the first quarter of internship. The findings from this research are positive. However, there is an urgent need to carry out formal assessments of entrustability, rather than relying on self-report.

What Were You Thinking? Medical Students' Metacognition and Perceptions of Self-Regulated Learning

PHENOMENON

As a component of self-regulated learning, metacognition is gaining attention in the medical education research community. Metacognition, simply put, is thinking about one's thinking. Having a metacognitive habit of mind is essential for healthcare professionals. This study identified the metacognitive competencies of medical students as they completed a conceptual learning task, and provided insight into students' perceptions of self-regulated learning in their curriculum. Approach: Eleven third-year medical students from a Dutch University were purposively sampled to participate in this qualitative study. The study design included a think-aloud assignment followed by a semi-structured interview. During the assignment, participants were instructed to think aloud while solving questions about medical physiological concepts such as blood flow, pressure, and resistance. Think-aloud data were collected through audiotaping and used to identify participants' metacognitive competencies. The assignment also served as a prompt for an interview in which participants were questioned about metacognitive knowledge, monitoring, experiences, and perceptions of self-regulated learning in their curriculum. All data were transcribed verbatim and analyzed iteratively using a template analysis. Findings: Students differed in their use of metacognitive skills, with an overall focus on monitoring and, to a lesser extent, on planning and evaluation. Additionally, differences were found in students' metacognitive knowledge and metacognitive experiences. There was apparent use of inefficient, superficial predictive cues. Regarding perceptions of self-regulated learning skills, some students felt no need to develop such skills as they perceived medical education as an exercise in memorizing facts. Others emphasized the need for more insight into their actual level of knowledge and competence. Insights: Pre-clinical medical students require explicit teaching of metacognitive skills to facilitate self-regulated learning. Educators should aim to integrate metacognition in the everyday discourse of the classroom to foster an environment in which students discuss their own learning.

Supplemental data for this article is available online at https://doi.org/10.1080/10401334.2021.1889559.

Transcatheter Mitral Valve Repair Simulator Equipped with Eye Tracking Based Performance Assessment Capabilities: A Pilot Study

Background

The increase in cardiovascular disease cases that require minimally invasive treatment is inducing a new need to train physicians to perform them safely and effectively. Nevertheless, adaptation to simulation-based training has been slow, especially for complex procedures.

Objectives

We describe a newly developed mitral valve repair (MVR) simulator, equipped with new objective performance assessment methods, with an emphasis on its use for training the MitraClip™ procedure.

Methods

The MVR contains phantoms of all anatomical structures encountered during mitral valve repair with a transvenous, transseptal approach. In addition, several cameras, line lasers, and ultraviolet lights are used to mimic echocardiographic and fluoroscopic imaging and with a remote eye tracker the cognitive behaviour of the operator is recorded. A pilot study with a total of 9 interventional cardiologists, cardiac surgeons and technical experts was conducted. All participants performed the MitraClip procedure on the MVR simulator using standard interventional tools. Subsequently, each participant completed a structured questionnaire to assess the simulator.

Results

The simulator functioned well, and the implemented objective performance assessment methods worked reliably. Key performance metrics such as x-ray usage were comparable with results from studies assessing these metrics in real interventions. Fluoroscopy imaging is realistic for the transseptal puncture but reaches its limits during the final steps of the procedure.

Conclusion

The functionality and objective performance assessment of the MVR simulator were demonstrated. Especially for complex procedures such as the MitraClip procedure, this simulator offers a suitable platform for risk-free training and education.

Supplementary Information

The online version contains supplementary material available at 10.1007/s13239-021-00549-4.

Fairness in human judgement in assessment: a hermeneutic literature review and conceptual framework

Human judgement is widely used in workplace-based assessment despite criticism that it does not meet standards of objectivity. There is an ongoing push within the literature to better embrace subjective human judgement in assessment not as a 'problem' to be corrected psychometrically but as legitimate perceptions of performance. Taking a step back and changing perspectives to focus on the fundamental underlying value of fairness in assessment may help re-set the traditional objective approach and provide a more relevant way to determine the appropriateness of subjective human judgements. Changing focus to look at what is 'fair' human judgement in assessment, rather than what is 'objective' human judgement in assessment allows for the embracing of many different perspectives, and the legitimising of human judgement in assessment. However, this requires addressing the question: what makes human judgements fair in health professions assessment? This is not a straightforward question with a single unambiguously 'correct' answer. In this hermeneutic literature review we aimed to produce a scholarly knowledge synthesis and understanding of the factors, definitions and key questions associated with fairness in human judgement in assessment and a resulting conceptual framework, with a view to informing ongoing further research. The complex construct of fair human judgement could be conceptualised through values (credibility, fitness for purpose, transparency and defensibility) which are upheld at an individual level by characteristics of fair human judgement (narrative, boundaries, expertise, agility and evidence) and at a systems level by procedures (procedural fairness, documentation, multiple opportunities, multiple assessors, validity evidence) which help translate fairness in human judgement from concepts into practical components.

The clinical learning environment in anaesthesiology in Kerala---Is it good enough?---A web-based survey

Background and Aims

Evaluation of postgraduate (PG) training is important so that necessary modifications can be made. The quality of anaesthesiology teaching in our country has not been explored. We performed a survey among the anaesthesiology PGs of Kerala to assess their perceptions about their learning environment as well as to look for the deficits in the training program.

Methods

An online web-based survey was conducted. The questionnaire was developed after a detailed validation process and circulated via Google forms. The questionnaire had three parts. First part asked for demographic details. Second part had 22 statements in four domains to evaluate the quality of training, which were to be answered based on a five-point Likert scale. The third part asked for the suggestions of the trainees on changes to be made for the improvement in quality of their training.

Results

The response rate was 64%. The social atmosphere was perceived positively in general by our PGs. Majority of them received feedback on their work and had a regular class schedule. Only 34% of the residents got training in non-technical skills. More tutorials and case discussions were the most common change suggested by the residents. Lack of research training and training in simulation labs were the major lacunae identified.

Conclusion

The clinical learning environment (CLE) was perceived to be of medium to high quality by the anaesthesiology PGs of Kerala. Lack of routine assessments and regular academic sessions were reflected in the responses.

The embodiment of practice thresholds: from standardization to stabilization in surgical education

Surgeons practice their own variations on a procedure. Residents experience shifting thresholds between variations that one surgeon holds firmly as principle and another takes more lightly as preference. Such variability has implications for surgical education, but the impact is not well understood. This is a critical problem to investigate as programs seek to define procedures for competency-based medical education (CBME) and improve learning through deliberate practice. Our study analyzes the emergence of procedural variation in an early-adopter CBME program through a situational analysis of tonsillectomy, a foundation level procedure in this otolaryngology, head and neck surgical program. An earlier phase of the study identified frequent variations (n = 12) on tonsillectomy among co-located surgeons who routinely perform this procedure (n = 6). In the phase reported here we interviewed these surgeons (n = 4) and residents at different stages of training (n = 3) about their experiences of these variations to map the relations of contributing social and material actors. Our results show that even a basic procedure resists standardization. This study contributes a sociomaterial grounded theory of surgical practice as an embodied response to conditions materialized by intra-relations of human and more-than-human actors. Shifting root metaphors about practice in surgical education from standardization to stabilization can help residents achieve stable-for-now embodiments of performance as their practice thresholds continue to emerge.

Cognitive levels in testing knowledge in evidence-based medicine: a cross sectional study

BACKGROUND

Knowledge assessment in evidence-based medicine (EBM) is usually performed by the measurement of memorised facts, understanding of EBM concepts and application of learned knowledge in familiar situations, all of which are considered lower-level educational objectives. The aim of this study was to assess EBM knowledge both on higher and lower cognitive levels across EBM topics.

METHODS

In order to assess knowledge on different EBM topics across learning levels, we created a knowledge test (Six Progressive Levels in Testing - SPLIT instrument), which consists of 36 multiple choice items and measures knowledge in EBM at six cognitive levels (Remembering, Understanding, Applying, Analysing, Evaluating and Creating) and addresses six EBM topics (Evidence-based practice, Internal validity, Clinical importance, Study design, Sources of evidence, Diagnostic studies). Three independent assessors defined the minimum passing score (MPS) for the overall test, based on the first-year course content and educational objectives. The instrument was assessed in a sample of first- (n = 119) and third-year medical students (n = 70) and EBM experts (n = 14).

RESULTS

The MPS was 16 correct answers out of total 36 questions, and was achieved by 21 out of 119 first-year students, 14 out of 70 third-year students and 9 out of 14 EBM experts (χ2 = 13.3; P < 0.001, with significantly higher proportion of experts passing compared to students). Although experts had the highest scores overall, none of the groups outperformed others on individual cognitive levels, but the experts outperformed students in EBM topics of Study design and Sources of evidence (P = 0.002 and 0.004, respectively, Kruskal-Wallis test). First- and third-year students performed better on specific course topics taught in that study year (Diagnostic studies and Clinical relevance, respectively).

CONCLUSION

EBM knowledge of students and experts differ according to the specificities of their education/expertise, but neither group had excellent knowledge in all areas. It may be difficult to develop a knowledge test that includes different EBM topics at different cognitive levels to follow the development of specific and general aspects of EBM knowledge.

Evaluating the effectiveness of undergraduate clinical education programs

Medical schools should use a variety of measures to evaluate the effectiveness of their clinical curricula. Both outcome measures and process measures should be included, and these can be organized according to the four-level training evaluation model developed by Donald Kirkpatrick. Managing evaluation data requires the institution to employ deliberate strategies to monitor signals in real-time and aggregate data so that informed decisions can be made. Future steps in program evaluation includes increased emphasis on patient outcomes and multi-source feedback, as well as better integration of existing data sources.

Chest compressions quality during sudden cardiac arrest scenario performed in virtual reality: A crossover study in a training environment

Potential attributes of virtual reality (VR) can be a breakthrough in the improvement of sudden cardiac arrest (SCA) training. However, interference with the virtual world is associated with the need of placing additional equipment on the trainee's body. The primary aim of the study was to evaluate if it does not affect the quality of chest compressions (CCs).91 voluntarily included in the study medical students participated twice in the scenario of SCA - Traditional Scenario (TS) and Virtual Reality Scenario (VRS). In both cases two minutes of resuscitation was performed.If VRS was the first scenario there were significant differences in CCs depth (VRS - Me = 47 mm [IQR 43 - 52] vs TS - Me = 48 mm [IQR 43 - 55]; P = .02) and chest relaxation (VRS - Me = 37% [IQR 5 - 91] vs TS - Me = 97% [IQR 87 - 100]; P < .001). 97.8% of respondents believe that training with the use of VR is more effective than a traditional method (P < .01). Most of the study group (91%, P < .01) denied any negative symptoms during the VR scenario.Virtual reality can be a safe and highly valued by medical students, method of hands-on CPR training. However additional VR equipment placed on the trainee's body may cause chest compressions harder to provide. If it is not preceded by traditional training, the use of VR may have an adverse impact on depth and full chest relaxation during the training. To make the best use of all the potential that virtual reality offers, future studies should focus on finding the most effective way to combine VR with traditional skill training in CPR courses curriculum.

Summative Assessment of Interprofessional "Collaborative Practice" Skills in Graduating Medical Students: A Validity Argument

PURPOSE

To describe how the authors developed an objective structured clinical examination (OSCE) station to assess aspects of collaborative practice competency and how they then assessed validity using Kane's framework.

METHOD

After piloting the collaborative practice OSCE station in 2015 and 2016, this was introduced at the Cumming School of Medicine in 2017. One hundred fifty-five students from the class of 2017 and 22 students from the class of 2018 participated. To create a validity argument, the authors used Kane's framework that views the argument for validity as 4 sequential inferences on the validity of scoring, generalization, extrapolation, and implications, RESULTS: Scoring validity is supported by psychometric analysis of checklist items and the fact that the contribution of rater specificity to students' ratings was similar to OSCE stations assessing clinical skills alone. The claim of validity of generalization is backed by structural equation modeling and confirmatory factor analysis that identified 5 latent variables, including 3 related to collaborative practice ("provides an effective handover," "provides mutual support," and "shares their mental model"). Validity of extrapolation is argued based upon the correlation between the rating for "shares their mental model" and the rating on in-training evaluations for "relationship with other members of the health care team," in addition to the association between performance on the collaborative practice OSCE station and the subsequent rating of performance during residency. Finally, validity of implications is supported by the fact that pass/fail decisions on the collaborative practice station were similar to other stations and by the observation that ratings on different aspects of collaborative practice associate with pass/fail decisions.

CONCLUSIONS

Based upon the validity argument presented, the authors posit that this tool can be used to assess the collaborative practice competence of graduating medical students and the adequacy of training in collaborative practice.

Low-fidelity otoscopy simulation and anatomy training: A randomized controlled trial

OBJECTIVES

To evaluate whether the use of low-fidelity otoscopy simulation improved medical students' theoretical knowledge of middle ear anatomy and pathologies compared to traditional teaching methods.

METHODS

This was a randomized controlled trial. Simulation workshops were conducted in April 2019 in the Lyon Sud University medical faculty, France. Students were randomly assigned to the simulation group (n=105) or to the control group (n=95). The students in the control group answered a questionnaire evaluating theoretical knowledge (25 true-false questions) before the simulation tutorial, while the students in the simulation group answered the same questions after the tutorial. Both groups also filled out a satisfaction questionnaire for feedback.

RESULTS

196 of the 200 students who participated in the study completed the knowledge assessment questionnaire. Scores were 32.0% higher in the simulation group than in the control group (mean scores, 12.0/20 vs. 9.1/20; P<0.0001). 184 of the 191 students who completed the satisfaction questionnaire (96.3%) were satisfied or very satisfied with the workshop, and all but one (99.5%) recommended keeping it in the curriculum. In the free comments fields, students highlighted the educational value of learning without the stress of patient discomfort.

CONCLUSION

Otoscopy simulation is an effective training method, improving theoretical knowledge compared with conventional theoretical training.

Assessing the Validity and Reliability of the Pharmacist Interprofessional Competencies Tool

Objective. To determine the validity and reliability of the Pharmacist Interprofessional Competencies Tool (PICT). Methods. Faculty members at Ferris State University, College of Pharmacy developed the PICT, which has five interprofessional criterion (collaboration, ownership, respect, engagement, and application) and four competency levels (unacceptable, novice, competent, and proficient) to assess the interprofessional competencies of pharmacy students. Fourteen pharmacy faculty members were trained in how to use the PICT and then used it to assess students' behaviors in four to six video-recorded interprofessional education (IPE) learning activities. A subset of these faculty members evaluated the video-recorded IPE learning activities using two other previously validated interprofessional assessment tools. Psychometric analysis of the PICT, including internal consistency and inter-rater reliability, was conducted, along with a correlation analysis and factor analysis, and the results were compared to those from the other validated assessment tools. Results. The overall rating of the internal consistency of the PICT was excellent and item-total correlations of the individual criterion were fair to good, with the exception of the respect criterion. The PICT demonstrated excellent overall inter-rater reliability, and individual criterion rated as fair to excellent with the exception of the respect criterion. Specific dimensions of the PICT showed high convergence with previously validated interprofessional assessment tools. Conclusion. The PICT exhibited overall validity and reliability as an assessment tool for measuring the interprofessional competencies of pharmacy students. In establishing the overall validity and reliability of the assessment tool, the respect criterion was not proved to be reliable or valid. Additional training and slight modifications to the PICT and associated IPE learning activities are planned to assist with longitudinal assessment of student performance across the curriculum.

Artificial Intelligence Education and Tools for Medical and Health Informatics Students: Systematic Review

BACKGROUND

The use of artificial intelligence (AI) in medicine will generate numerous application possibilities to improve patient care, provide real-time data analytics, and enable continuous patient monitoring. Clinicians and health informaticians should become familiar with machine learning and deep learning. Additionally, they should have a strong background in data analytics and data visualization to use, evaluate, and develop AI applications in clinical practice.

OBJECTIVE

The main objective of this study was to evaluate the current state of AI training and the use of AI tools to enhance the learning experience.

METHODS

A comprehensive systematic review was conducted to analyze the use of AI in medical and health informatics education, and to evaluate existing AI training practices. PRISMA-P (Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols) guidelines were followed. The studies that focused on the use of AI tools to enhance medical education and the studies that investigated teaching AI as a new competency were categorized separately to evaluate recent developments.

RESULTS

This systematic review revealed that recent publications recommend the integration of AI training into medical and health informatics curricula.

CONCLUSIONS

To the best of our knowledge, this is the first systematic review exploring the current state of AI education in both medicine and health informatics. Since AI curricula have not been standardized and competencies have not been determined, a framework for specialized AI training in medical and health informatics education is proposed.

Attitudes towards medical artificial intelligence talent cultivation: an online survey study

Background

To investigate the attitude and formal suggestions on talent cultivation in the field of medical artificial intelligence (AI).

Methods

An electronic questionnaire was sent to both medical-related field or non-medical field population using the WenJuanXing web-application via social media. The questionnaire was designed to collect: (I) demographic information; (II) perception of medical AI; (III) willingness to participate in the medical AI related teaching activities; (IV) teaching content of medical AI; (V) the role of medical AI teaching; (VI) future career planning. Respondents’ anonymity was ensured.

Results

A total of 710 respondents provided valid answers to the questionnaire (57.75% medical related, 42.25% non-medical). About 73.8% of respondents acquired related information from network and social platform. More than half the respondents had basic perception of AI applicational scenarios and specialties in medicine, meanwhile were willing to participate in related general science activities (conference and lectures). Respondents from medical healthcare related fields, with high academic qualifications of male ones demonstrated showed significant better understanding and stronger willingness (P<0.05). The majority agreed medical AI courses should be set as major elective (42.82%) during undergraduate stages (89.58%) involving medical and computer science contents. An overwhelming majority of respondents (>80%) acknowledged the potential roles of medical AI teaching. Surgeon, ophthalmologist, physicians and researchers are the top tier considerations for ideal career regardless of AI influence. Radiology and clinical laboratory subjects are more preferred considering the development of medical AI (P>0.05).

Conclusions

The potential role of medical AI talent cultivation is widely acknowledged by public. Medical related professions demonstrated higher level of perception and stronger willingness for medical AI educational events. Merging subjects as radiology and clinical laboratory subjects are preferred with broad talents demands and bright prospects.

Exploring transformative learning when developing medical students' non-technical skills

CONTEXT

Non-technical skills (NTS) training should be incorporated into medical students' education and simulation-based approaches are often utilised to facilitate this. Such experiences have the potential to foster transformative learning by facilitating a reassessment of one's prior assumptions and a significant shift in one's outlook, referred to as the process of perspective transformation. The aim of this research was to explore how NTS training might facilitate transformative learning in final-year medical students.

METHODS

Following ethical approval, medical student volunteers from four medical schools (Aberdeen, Dundee, Edinburgh and Glasgow) participated in simulation sessions, were debriefed with an emphasis on NTS using a behavioural marker system and then took part in focus groups. Focus group discussions were semi-structured and questions were based on the phases of perspective transformation identified by Jack Mezirow. Focus group discussions were audiorecorded, transcribed verbatim, anonymised and analysed using template analysis.

RESULTS

A total of 33 medical students took part in five focus groups. There was evidence of the following stages of perspective transformation: Phase 2 (self-examination with emotional disturbance, including fear, anxiety, guilt, shame and frustration); Phase 3 (critical assessment of assumptions, including the undervaluing of NTS, recognising that technical skills alone are insufficient, and recognising that it is possible to improve one's NTS); Phase 5 (exploring options for new roles, relationships and actions), and Phase 6 (planning a course of action for future simulations, as a medical student and as a doctor).

CONCLUSIONS

This study deepens our understanding of how exposure to NTS training in simulation-based education influences the learning of medical students and shows that such exposure can result in the cognitive phases of transformative learning. It provides us with valuable insights into medical students' perspectives on their learning of NTS at a pivotal stage in training and represents an interesting way of assessing the educational impact of such sessions.

Implementation of a large-scale simulation-based cardiovascular clinical examination course for undergraduate medical students - a pilot study

BACKGROUND

We report the implementation of a large-scale simulation-based cardiovascular diagnostics course for undergraduate medical students.

METHODS

A simulation-based course was integrated into the curriculum of second-year medical students (> 400 students/year). The first session aimed at teaching cardiac auscultation skills on mannequins and the second at teaching blood pressure measurement, peripheral arterial examination, and the clinical examination of heart failure in a technical skill-based manner and in a scenario.

RESULTS

A total of 414 (99.8%) and 402 (98.5%) students, as well as 102 and 104 educators, participated during the 2016-2017 and 2017-2018 academic years across both types of sessions. The number of positive appreciations by students was high and improved from the first to the second year (session 1: 77% vs. 98%, session 2: 89% vs. 98%; p < 0.0001). Similar results were observed for educators (session 1: 84% vs. 98%, p = 0.007; session 2: 82% vs. 98%, p = 0.01). Feedbacks by students were positive regarding the usefulness of the course, fulfillment of pedagogical objectives, quality of the teaching method, time management, and educator-student interactivity. In contrast, 95% of students criticized the quality of the mannequins during the first year leading to the replacement of the simulation material the following year. Students most appreciated the auscultation workshop (25%), the practical aspect of the course (22%), and the availability of educators (21%).

CONCLUSIONS

Despite the need to commit significant human and material resources, the implementation of this large-scale program involving > 400 students/year was feasible, and students and educators reacted favorably.

Legitimizing Continuous Quality Improvement (CQI): Navigating Rationality in Undergraduate Medical Education

In July of 2015, the Liaison Committee on Medical Education (LCME)-the primary accrediting body for North American allopathic medical schools-formally advanced a model of "formative accreditation" by requiring that medical schools engage in "ongoing planning and continuous quality improvement processes that establish short and long-term programmatic goals, result in the achievement of measurable outcomes that are used to improve programmatic quality, and ensure effective monitoring of the medical education program's compliance with accreditation standards."As these and parallel forces redefine undergraduate medical education (UME) in increasingly rationalistic (i.e., operational, measureable, controllable) terms, efforts to implement meaningful continuous quality improvement (CQI) processes may be challenged to overcome perceptions of questionable purpose, worth, and impact often associated with administration mandates. This commentary discusses potential factors underlying the growing rationalism in UME and offers practical strategies to shield CQI from being passively dismissed, excessively routinized, or redirected toward other institutional ends-remaining, instead, purposefully focused on the task at hand: Enhancing teaching and learning in undergraduate medical curricula.

Students' self-assessment of achievement of terminal competency and 4-year trend of student evaluation on outcome-based education

PURPOSE

This study was designed to allow a student at School of Medicine, Kyungpook National University (KNUSOM) to self-assess how well they had achieved competency and to analyze the differences and trends of the scores by year. Furthermore, students are asked to evaluate the need for curriculum improvement by competency, the tendency of the score is analyzed by year, and the results are reflected in the improvement of the curriculum.

METHODS

We conducted a questionnaire survey for fourth-year students of KNUSOM who took medical education classes from 2015 to 2018. Questionnaire items were evaluated on the basis of their current achievement of 30 subordinate competencies of competency and the degree of necessity with respect to revising the curriculum. One-way analysis of variance was performed for the yearly difference analysis.

RESULTS

The students' scores on the graduation competency were 2.03 to 4.06. In the yearly difference analysis, there was no significant difference in the average of 30 total competencies, but 16 of the sub-competencies showed significant year-to-year differences. The scores for the 30 graduation competencies were different for each year, but the competencies showing high scores and low scores were found to be similar each year.

CONCLUSION

We found that the achievement level of the students was approximately 60% to 70%. We were able to confirm the contents of the education that the students continuously demand. The curriculum trend graphs for each year showed that the students' scores improved when the curriculum was being revised. We found that it is necessary to accept the students' self-evaluation reliable as the students indicated that the contents of the curriculum should be added to the areas where the contents were lacking in the present curriculum.