Characterizing the literature on validity and assessment in medical education: a bibliometric study

Interdisciplinary fetal-neonatal neurology training improves brain health across the lifespan

Integrated fetal, neonatal, and pediatric training constitute an interdisciplinary fetal-neonatal neurology (FNN) program. A dynamic neural exposome concept strengthens curriculum content. Trainees participate in mentoring committee selection for guidance during a proposed two-year program. Prenatal to postnatal clinical learning re-enforces early toxic stressor interplay that influences gene-environment interactions. Maternal-placental-fetal triad, neonatal, or childhood diseases require diagnostic and therapeutic decisions during the first 1,000 days when 80 % of neural connections contribute to life-course phenotypic expression. Pediatric follow-up through 3 years adjusts to gestational ages of preterm survivors. Cumulative reproductive, pregnancy, pediatric and adult exposome effects require educational experiences that emphasize a principle-to-practice approach to a brain capital strategy across the lifespan. More rigorous training during fetal, neonatal, and pediatric rotations will be offered to full time trainees. Adult neurology residents, medical students, and trainees from diverse disciplines will learn essential topics during time-limited rotations. Curriculum content will require periodic re-assessments using educational science standards that maintain competence while promoting creative and collaborative problem-solving. Continued career-long learning by FNN graduates will strengthen shared healthcare decisions by all stakeholders. Recognition of adaptive or maladaptive neuroplasticity mechanisms requires analytic skills that identify phenotypes associated with disease pathways. Developmental origins and life-course concepts emphasize brain health across the developmental-aging continuum, applicable to interdisciplinary research collaborations. Social determinants of health recognize diversity, equity, and inclusion priorities with each neurological intervention, particularly for those challenged with disparities. Diagnostic and therapeutic strategies must address resource challenges particularly throughout the Global South to effectively lower the worldwide burden of neurologic disease. Sustainable development goals proposed by the World Health Organization offer universally applicable guidelines in response to ongoing global and regional polycrises. Gender, race, ethnicity, and socio-economic equality promote effective preventive, rescue and reparative neuroprotective interventions. Global synergistic efforts can be enhanced by establishing leadership within academic teaching hubs in FNN training to assist with structure and guidance for smaller healthcare facilities in each community that will improve practice, education and research objectives. Reduced mortality with an improved quality of life must prioritize maternal-pediatric health and well-being to sustain brain health across each lifespan with transgenerational benefits.

Evaluating Student Clerkship Performance Using Multiple Assessment Components

Introduction

Family medicine clerkships utilize a broad set of objectives. The scope of these objectives cannot be measured by one assessment alone. Using multiple assessments aimed at measuring different objectives may provide more holistic evaluation of students. A further concern is to ensure longitudinal accuracy of assessments. In this study, we sought to better understand the relevance and validity of different assessment tools used in family medicine clerkships.

Methods

We retrospectively correlated family medicine clerkship students' scores across different assessments to evaluate the strengths of the correlations, between the different assessment tools. We defined ρ<0.3 as weak, ρ>0.3 to ρ<0.5 as moderate, and ρ>0.5 as high correlation.

Results

We compared individual assessment scores for 267 students for analysis. The correlation of the clinical evaluation was 0.165 (P<.01); with case-based short-answer questions it was 0.153 (P<.01); and with objective structured clinical examinations it was -0.246 (P<0.01).

Conclusion

Overall low levels of correlations between our assessments are expected, as they are each designed to measure different objectives. The relatively higher correlation between component scores supports convergent validity while correlations closer to zero suggest discriminant validity. Unexpectedly, comparing the multiple-choice questions and objective, structured clinical encounter (OSCE) assessments, we found higher correlation, although we believe these should measure disparate objectives. We replaced our in-house multiple-choice questions with a nationally-standardized exam and preliminary analysis shows the expected weaker correlation with the OSCE assessment, suggesting periodic correlations between assessments may be useful.

Bridging Validity Frameworks in Assessment: Beyond Traditional Approaches in Health Professions Education

Construct: High-stakes assessments measure several constructs, such as knowledge, competencies, and skills. In this case, validity evidence for test scores' uses and interpretations is of utmost importance, because of the consequences for everyone involved in their development and implementation. Background: Educational assessment requires an appropriate understanding and use of validity frameworks; however, health professions educators still struggle with the conceptual challenges of validity, and frequently validity analyses have a narrow focus. Important obstacles are the plurality of validity frameworks and the difficulty of grounding these abstract concepts in practice. Approach: We reviewed the validity frameworks literature to identify the main elements of frequently used models (Messick and Kane's) and proposed linking frameworks including Russell's recent overarching proposal. Examples are provided with commonly used assessment instruments in health professions education. Findings: Several elements in these frameworks can be integrated into a common approach, matching and aligning Messick's sources of validity with Kane's four inference types. Conclusions: This proposal to contribute evidence for assessment inferences may provide guidance to understanding the use of validity evidence in applied settings. The evolving field of validity research provides opportunities for its integration and practical use in health professions education.

A Bibliometric Analysis of the 100 Most Cited Articles on Problem-Based Learning in Medical Education

Problem-based learning (PBL) is an instructional approach used in medical education that is characterized by solving problems in small groups with tutor guidance. More than 50 years since PBL's inception, many questions remain to be addressed about its processes and learning outcomes. The purpose of the study was to examine the bibliometric characteristics of the 100 most cited articles on PBL in medical education and to identify landmark papers that have made significant contributions to PBL research. Results were systematically reviewed for citation frequency, publication year, journal, article type, article focus, authors, author collaboration, and country collaboration. The number of citations ranged from 81 to 3531 times cited with 31,041 total citations. The articles were contributed by 211 authors in 23 journals and most articles (68%) were published in Medical Education, Academic Medicine, and Medical Teacher. The majority of the articles (71%) originated from Netherlands, Canada, and the United States and six prolific authors were identified. Almost half of the articles are classified as empirical research. Article foci included theoretical foundations of PBL, curriculum design, learning outcomes and processes, tutors, assessment, guides to PBL implementation, commentaries, and student well-being. The strong author and country collaborations indicate continued global interest in the PBL instructional method, which is likely to remain an active topic of research as the evidence of its effectiveness over traditional instructional methods as well as its most impactful components is inconclusive.

How argumentation theory can inform assessment validity: A critical review

INTRODUCTION

Many health professions education (HPE) scholars frame assessment validity as a form of argumentation in which interpretations and uses of assessment scores must be supported by evidence. However, what are purported to be validity arguments are often merely clusters of evidence without a guiding framework to evaluate, prioritise, or debate their merits. Argumentation theory is a field of study dedicated to understanding the production, analysis, and evaluation of arguments (spoken or written). The aim of this study is to describe argumentation theory, articulating the unique insights it can offer to HPE assessment, and presenting how different argumentation orientations can help reconceptualize the nature of validity in generative ways.

METHODS

The authors followed a five-step critical review process consisting of iterative cycles of focusing, searching, appraising, sampling, and analysing the argumentation theory literature. The authors generated and synthesised a corpus of manuscripts on argumentation orientations deemed to be most applicable to HPE.

RESULTS

We selected two argumentation orientations that we considered particularly constructive for informing HPE assessment validity: New rhetoric and informal logic. In new rhetoric, the goal of argumentation is to persuade, with a focus on an audience's values and standards. Informal logic centres on identifying, structuring, and evaluating arguments in real-world settings, with a variety of normative standards used to evaluate argument validity.

DISCUSSION

Both new rhetoric and informal logic provide philosophical, theoretical, or practical groundings that can advance HPE validity argumentation. New rhetoric's foregrounding of audience aligns with HPE's social imperative to be accountable to specific stakeholders such as the public and learners. Informal logic provides tools for identifying and structuring validity arguments for analysis and evaluation.

Validity as a social imperative: users' and leaders' perceptions

INTRODUCTION

Recently, validity as a social imperative was proposed as an emerging conceptualization of validity in the assessment literature in health professions education (HPE). To further develop our understanding, we explored the perceived acceptability and anticipated feasibility of validity as a social imperative with users and leaders engaged with assessment in HPE in Canada.

METHODS

We conducted a qualitative interpretive description study. Purposeful and snowball sampling were used to recruit participants for semi-structured individual interviews and focus groups. Each transcript was analyzed by two team members and discussed with the team until consensus was reached.

RESULTS

We conducted five focus group and eleven interviews with two different stakeholder groups (users and leaders). Our findings suggest that the participants perceived the concept of validity as a social imperative as acceptable. Regardless of group, participants shared similar considerations regarding: the limits of traditional validity models, the concept's timeliness and relevance, the need to clarify some terms used to characterize the concept, the similarities with modern theories of validity, and the anticipated challenges in applying the concept in practice. In addition, participants discussed some limits with current approaches to validity in the context of workplace-based and programmatic assessment.

CONCLUSION

Validity as a social imperative can be interwoven throughout existing theories of validity and may represent how HPE is adapting traditional models of validity in order to respond to the complexity of assessment in HPE; however, challenges likely remain in operationalizing the concept prior to its implementation.

Factors affecting perceived credibility of assessment in medical education: A scoping review

Assessment is more educationally effective when learners engage with assessment processes and perceive the feedback received as credible. With the goal of optimizing the educational value of assessment in medical education, we mapped the primary literature to identify factors that may affect a learner's perceptions of the credibility of assessment and assessment-generated feedback (i.e., scores or narrative comments). For this scoping review, search strategies were developed and executed in five databases. Eligible articles were primary research studies with medical learners (i.e., medical students to post-graduate fellows) as the focal population, discussed assessment of individual learners, and reported on perceived credibility in the context of assessment or assessment-generated feedback. We identified 4705 articles published between 2000 and November 16, 2020. Abstracts were screened by two reviewers; disagreements were adjudicated by a third reviewer. Full-text review resulted in 80 articles included in this synthesis. We identified three sets of intertwined factors that affect learners' perceived credibility of assessment and assessment-generated feedback: (i) elements of an assessment process, (ii) learners' level of training, and (iii) context of medical education. Medical learners make judgments regarding the credibility of assessments and assessment-generated feedback, which are influenced by a variety of individual, process, and contextual factors. Judgments of credibility appear to influence what information will or will not be used to improve later performance. For assessment to be educationally valuable, design and use of assessment-generated feedback should consider how learners interpret, use, or discount assessment-generated feedback.

The validity argument: Addressing the misconceptions

Assessment of students necessitates validity. Validity provides strong reasoning for decisions made based on students' scores. It is a unitary concept; the extent to which evidence supports and justify the proposed interpretations of test scores. However, the comprehension of the concept of validity has been associated with some observed misconceptions in the literature. With the consequent impact on the research and application being in consideration, this work aims to address some associated misconceptions to clarify the concept of validity. In doing so, we are adopting the current standard framework that is based on the five sources of validity.

Twelve tips for introducing the concept of validity argument in assessment to novice medical teachers in a workshop

This article was migrated. The article was marked as recommended. Background: Misconceptions have been observed in the application of validity by faculty and in the reporting of validity in a significant amount of published work in the field of students' assessment. As a result, actions concerning the dissemination of information about the concept of validity in relation to assessments, especially among novice medical teachers, is needed. Aim: This work aims to guide how the concept of validity argument in assessments is delivered to novice medical teachers in a workshop. Methods: Critical reflection and a careful review of relevant literature were used to develop these tips. Results and Conclusion: Twelve tips were introduced to support instructors conducting workshops on introducing the concept of validity, especially to novice medical teachers.