Validation of learning assessments: A primer

Legitimation Without Argumentation: An Empirical Discourse Analysis of 'Validity as an Argument' in Assessment

Introduction

Validity is frequently conceptualized in health professions education (HPE) assessment as an argument that supports the interpretation and uses of data. However, previous work has shown that many validity scholars believe argument and argumentation are relatively lacking in HPE. To better understand HPE's discourse around argument and argumentation with regard to assessment validity, the authors explored the discourses present in published HPE manuscripts.

Methods

The authors used a bricolage of critical discourse analysis approaches to understand how the language in influential peer reviewed manuscripts has shaped HPE's understanding of validity arguments and argumentation. The authors used multiple search strategies to develop a final corpus of 39 manuscripts that were seen as influential in how validity arguments are conceptualized within HPE. An analytic framework drawing on prior research on Argumentation Theory was used to code manuscripts before developing themes relevant to the research question.

Results

The authors found that the elaboration of argument and argumentation within HPE's validity discourse is scant, with few components of Argumentation Theory (such as intended audience) existing within the discourse. The validity as an argument discourse was legitimized via authorization (reference to authority), rationalization (reference to institutionalized action), and mythopoesis (narrative building). This legitimation has cemented the validity as an argument discourse in HPE despite minimal exploration of what argument and argumentation are.

Discussion

This study corroborates previous work showing the dearth of argument and argumentation present within HPE's validity discourse. An opportunity exists to use Argumentation Theory in HPE to better develop validation practices that support use of argument.

Validity in the Next Era of Assessment: Consequences, Social Impact, and Equity

Validity has long held a venerated place in education, leading some authors to refer to it as the "sine qua non" or "cardinal virtue" of assessment. And yet, validity has not held a fixed meaning; rather it has shifted in its definition and scope over time. In this Eye Opener, the authors explore if and how current conceptualizations of validity fit a next era of assessment that prioritizes patient care and learner equity. They posit that health profession education's conceptualization of validity will change in three related but distinct ways. First, consequences of assessment decisions will play a central role in validity arguments. Second, validity evidence regarding impacts of assessment on patients and society will be prioritized. Third, equity will be seen as part of validity rather than an unrelated concept. The authors argue that health professions education has the agency to change its ideology around validity, and to align with values that predominate the next era of assessment such as high-quality care and equity for learners and patients.

Academic conference posters: Describing visual impression in pharmacy education

Background

Academic conference posters are a key communication before journal articles. Attention to visual attributes can enhance academic poster communication.

Objective

This investigation's purpose was to create a visual impression measurement instrument, and then to describe and compare visual impression among scientific posters from an academic conference.

Methods

A mixed-approach rubric was created to quickly measure visual impression of academic posters. Then, posters from a pharmacy education conference were retrospectively reviewed and scored. Visual impression was compared for traditional versus contemporary poster-formats. Various poster characteristics (poster-format, summary statement presence, abstract presence, wordiness, QR-code presence, logical sequencing, visuals) that might have impacted visual communication were coded. These characteristics were regressed onto visual impression scores.

Results

Three-hundred seventy-eight posters were scored with sound inter-rater reliability. Contemporary poster-format scored significantly higher than traditional. Poster-format, abstract absence, lack of wordiness, QR-code presence, logical sequencing, and number of visuals were significant when regressed.

Conclusion

Posters at one academic conference had varied visual impression. While a contemporary poster-format appeared more helpful, it was not a panacea; variation from poor through exemplary was seen with both poster-formats. Posters are not text-filled articles; displaying a combination of visuals/text clearly and concisely can help effective communication with academic posters.

Interrogating Consequential Validity Evidence in NAPLEX Studies Involving the Use of Demographic Variables

OBJECTIVE

to review the purpose, methods, and discussion of student demographics related to North American Pharmacist Licensure Examination (NAPLEX) performance; demographic characteristics, significant findings, and related text were extracted from each reviewed article as evidence of consequential validity.

FINDINGS

Nine articles met the inclusion criteria. Prior degree attainment (n = 8, 88.9%), age (n = 6, 66.7%), race/ethnicity (n = 6, 66.7%), and sex (n = 5, 55.6%) were the most common demographic variables included. One study found that prior degree attainment was negatively related to NAPLEX performance and no studies using this variable discussed their findings. Three studies found significant relationships between age and NAPLEX performance, attributing differences to older students for "unique psychosocial challenges and competing responsibilities" "greater maturity" and being more "professional." Measures of race/ethnicity differed in every study, with 3 reporting significant findings and 2 discussing their findings. Studies referenced literature suggesting that "minority groups" tended to score lower on standardized examinations while others referenced literature that suggests "standardized testing may not be predictive of the performance of minority students." Sex was not related to NAPLEX performance nor was it discussed in any studies.

SUMMARY

Consequential validity evidence is a critical yet underreported aspect of NAPLEX evaluation in pharmacy education. How demographic variables are selected, utilized, and discussed warrants further exploration and consideration by educators, scholars, and practitioners as these decisions can have important sociocultural and political implications.

Using Rasch measurement for instrument rating scale refinement

OUR SITUATION

Rasch measurement is an analysis tool that can provide validity evidence for instruments that attempt to measure student learning or other psychosocial behaviors, regardless if tools are newly created, modified, or previously developed. Rating scales are exceedingly common among psychosocial instruments and properly functioning rating scales are critical to effective measurement. Rasch measurement can help investigate this.

METHODOLOGICAL LITERATURE REVIEW

Aside from using Rasch measurement from the beginning to help create rigorous new measurement instruments, researchers can also benefit from employing Rasch measurement on previously developed instruments that had not included Rasch measurement during development. This article is focused on Rasch measurement's unique analysis of rating scales. That is, Rasch measurement can uniquely help examine if and how an instrument's rating scale is functioning among newly studied respondents (who will likely differ from the originally researched sample).

OUR RECOMMENDATIONS AND THEIR APPLICATION

After reviewing this article, the reader should be able to describe Rasch measurement, including how it is focused on fundamental measurement and how it differs from classical test theory and item-response theory, and reflect on situations in their own research where a Rasch measurement analysis might be helpful for generating validation evidence with a previously developed instrument.

POTENTIAL IMPACT

In the end, Rasch measurement can offer a helpful, unique, rigorous approach to further developing instruments that scientifically measure, accurately and precisely.

Strategies for Measuring Advances in Pharmacy Student Learning

Objective. To provide guidance to authors and reviewers on how to design and evaluate educational research studies to better capture evidence of pharmacy student learning.Findings. A wide variety of assessment tools are available to measure student learning associated with educational innovations. Each assessment tool is associated with different advantages and disadvantages that must be weighed to determine the appropriateness of the tool for each situation. Additionally, the educational research design must be aligned with the intent of the study to strengthen its impact.Summary. By selecting research methods aligned with Kirkpatrick's levels of training evaluation, researchers can create stronger evidence of student learning when evaluating the effectiveness of teaching innovations.

A Teachable Moment Matters for: Striving for inter-rater reliability with a patient counseling assessment rubric: Lessons learned

The Teachable Moments Matter (TMM) category of articles is designed to offer readers insight into a methodological issue identified within a companion article. Written in collaboration with one of these authors, these articles provide an opportunity to focus on a challenge experienced by the authors and, in the process, provide one or more perspectives as to how to successfully navigate this issue. The current TMM is focused on issues and pitfalls in validation. The Journal hopes this case-based approach will help highlight the nuance of a topic in context, something that might get "lost" in the entirety of a full-length article.

Validation Evidence using Generalizability Theory for an Objective Structured Clinical Examination

OBJECTIVES

Performance-based assessments, including objective structured clinical examinations (OSCEs), are essential learning assessments within pharmacy education. Because important educational decisions can follow from performance-based assessment results, pharmacy colleges/schools should demonstrate acceptable rigor in validation of their learning assessments. Though G-Theory has rarely been reported in pharmacy education, it would behoove pharmacy educators to, using G-Theory, produce evidence demonstrating reliability as a part of their OSCE validation process. This investigation demonstrates the use of G-Theory to describes reliability for an OSCE, as well as to show methods for enhancement of the OSCE's reliability.

INNOVATION

To evaluate practice-readiness in the semester before final-year rotations, third-year PharmD students took an OSCE. This OSCE included 14 stations over three weeks. Each week had four or five stations; one or two stations were scored by faculty-raters while three stations required students' written responses. All stations were scored 1-4. For G-Theory analyses, we used G_Strings and then mGENOVA.

CRITICAL ANALYSIS

Ninety-seven students completed the OSCE; stations were scored independently. First, univariate G-Theory design of students crossed with stations nested in weeks (p × s:w) was used. The total-score g-coefficient (reliability) for this OSCE was 0.72. Variance components for test parameters were identified. Of note, students accounted for only some OSCE score variation. Second, a multivariate G-Theory design of students crossed with stations (p• × s°) was used. This further analysis revealed which week(s) were weakest for the reliability of test-scores from this learning assessment. Moreover, decision-studies showed how reliability could change depending on the number of stations each week. For a g-coefficient >0.80, seven stations per week were needed. Additionally, targets for improvements were identified.

IMPLICATIONS

In test validation, evidence of reliability is vital for the inference of generalization; G-Theory provided this for our OSCE. Results indicated that the reliability of scores was mediocre and could be improved with more stations. Revision of problematic stations could help reliability as well. Within this need for more stations, one practical insight was to administer those stations over multiple weeks/occasions (instead of all stations in one occasion).

Providing Validation Evidence for a Clinical-Science Module: Improving Testing Reliability with Quizzes

DESCRIPTION OF THE PROBLEM

High-stakes decision-making should have sound validation evidence; reliability is vital towards this. A short exam may not be very reliable on its own within didactic courses, and so supplementing it with quizzes might help. But how much? This study's objective was to understand how much reliability (for the overall module-grades) could be gained by adding quiz data to traditional exam data in a clinical-science module.

THE INNOVATION

In didactic coursework, quizzes are a common instructional strategy. However, individual contexts/instructors can vary quiz use formatively and/or summatively. Second-year PharmD students took a clinical-science course, wherein a 5-week module focused on cardiovascular therapeutics. Generalizability Theory (G-Theory) combined seven quizzes leading to an exam into one module-level reliability, based on a model where students were crossed with items nested in eight fixed testing occasions (mGENOVA used). Furthermore, G-Theory decision-studies were planned to illustrate changes in module-grade reliability, where the number of quiz-items and relative-weighting of quizzes were altered.

CRITICAL ANALYSIS

One-hundred students took seven quizzes and one exam. Individually, the exam had 32 multiple-choice questions (MCQ) (KR-20 reliability=0.67), while quizzes had a total of 50MCQ (5-9MCQ each) with most individual quiz KR-20s less than or equal to 0.54. After combining the quizzes and exam using G-Theory, estimated reliability of module-grades was 0.73; improved from the exam alone. Doubling the quiz-weight, from the syllabus' 18% quizzes and 82% exam, increased the composite-reliability of module-grades to 0.77. Reliability of 0.80 was achieved with equal-weight for quizzes and exam.

NEXT STEPS

Expectedly, more items lent to higher reliability. However, using quizzes predominantly formatively had little impact on reliability, while using quizzes more summatively (i.e., increasing their relative-weight in module-grade) improved reliability further. Thus, depending on use, quizzes can add to a course's rigor.

Initial Validation Evidence for Clinical Case Presentations by Student Pharmacists

OBJECTIVE

There is a paucity of validation evidence for assessing clinical case-presentations by Doctor of Pharmacy (PharmD) students. Within Kane's Framework for Validation, evidence for inferences of scoring and generalization should be generated first. Thus, our objectives were to characterize and improve scoring, as well as build initial generalization evidence, in order to provide validation evidence for performance-based assessment of clinical case-presentations.

DESIGN

Third-year PharmD students worked up patient-cases from a local hospital. Students orally presented and defended their therapeutic care-plan to pharmacist preceptors (evaluators) and fellow students. Evaluators scored each presentation using an 11-item instrument with a 6-point rating-scale. In addition, evaluators scored a global-item with a 4-point rating-scale. Rasch Measurement was used for scoring analysis, while Generalizability Theory was used for generalization analysis.

FINDINGS

Thirty students each presented five cases that were evaluated by 15 preceptors using an 11-item instrument. Using Rasch Measurement, the 11-item instrument's 6-point rating-scale did not work; it only worked once collapsed to a 4-point rating-scale. This revised 11-item instrument also showed redundancy. Alternatively, the global-item performed reasonably on its own. Using multivariate Generalizability Theory, the g-coefficient (reliability) for the series of five case-presentations was 0.76 with the 11-item instrument, and 0.78 with the global-item. Reliability was largely dependent on multiple case-presentations and, to a lesser extent, the number of evaluators per case-presentation.

CONCLUSIONS

Our pilot results confirm that scoring should be simple (scale and instrument). More specifically, the longer 11-item instrument measured but had redundancy, whereas the single global-item provided measurement over multiple case-presentations. Further, acceptable reliability can be balanced between more/fewer case-presentations and using more/fewer evaluators.

Moving beyond Cronbach's Alpha and Inter-Rater Reliability: A Primer on Generalizability Theory for Pharmacy Education

BACKGROUND

When available, empirical evidence should help guide decision-making. Following each administration of a learning assessment, data becomes available for analysis. For learning assessments, Kane's Framework for Validation can helpfully categorize evidence by inference (i.e., scoring, generalization, extrapolation, implications). Especially for test-scores used within a high-stakes setting, generalization evidence is critical. While reporting Cronbach's alpha, inter-rater reliability, and other reliability coefficients for a single measurement error are somewhat common in pharmacy education, dealing with multiple concurrent sources of measurement error within complex learning assessments is not. Performance-based assessments (e.g., OSCEs) that use raters, are inherently complex learning assessments.

PRIMER

Generalizability Theory (G-Theory) can account for multiple sources of measurement error. G-Theory is a powerful tool that can provide a composite reliability (i.e., generalization evidence) for more complex learning assessments, including performance-based assessments. It can also help educators explore ways to make a learning assessment more rigorous if needed, as well as suggest ways to better allocate resources (e.g., staffing, space, fiscal). A brief review of G-Theory is discussed herein focused on pharmacy education.

MOVING FORWARD

G-Theory has been common and useful in medical education, though has been used rarely in pharmacy education. Given the similarities in assessment methods among health-professions, G-Theory should prove helpful in pharmacy education as well. Within this Journal and accompanying this Idea Paper, there are multiple reports that demonstrate use of G-Theory in pharmacy education.

Validation Evidence from using Generalizability Theory in a Basic-Science Course: Reliability of Course-Grades from Multiple Examinations

DESCRIPTION OF THE PROBLEM

Reliability is critical validation evidence on which to base high-stakes decision-making. Many times, one exam in a didactic course may not be acceptably reliable on its own. But how much might multiple exams add when combined together?

THE INNOVATION

To improve validation evidence towards high-stakes decision-making, Generalizability Theory (G-Theory) can combine reliabilities from multiple exams into one composite-reliability (G_String IV software). Further, G-Theory decision-studies can illustrate changes in course-grade reliability, depending on the number of exams and exam-items.

CRITICAL ANALYSIS

101 first-year PharmD students took two midterm-exams and one final-exam in a pharmaceutics course. Individually, Exam1 had 50MCQ (KR-20=0.69), Exam2 had 43MCQ (KR-20=0.65), and Exam3 had 67MCQ (KR-20=0.67). After combining exam occasions using G-Theory, the composite-reliability was 0.71 for overall course-grades-better than any exam alone. Remarkably, increased numbers of exam occasions showed fewer items per exam were needed, and fewer items over all exams, to obtain an acceptable composite-reliability. Acceptable reliability could be achieved with different combinations of number of MCQs on each exam and number of exam occasions.

IMPLICATIONS

G-Theory provided reliability critical validation evidence towards high-stakes decision-making. Final course-grades appeared quite reliable after combining multiple course exams-though this reliability could and should be improved. Notably, more exam occasions allowed fewer items per exam and fewer items over all the exams. Thus, one added benefit of more exam occasions for educators is developing fewer items per exam and fewer items over all exams.

Guidance for high-stakes testing within pharmacy educational assessment

BACKGROUND

The term "high-stakes testing" is widely used among pharmacy educators, but the term often seems misused or used incompletely. This Teachable Moments Matter (TMM) focuses on the importance of scientific-rigor when assessing learners' abilities. This article discusses high-stakes testing - what it is and what it is not. This TMM is not meant as an extensive review of the topic.

IMPACT

As imperative for ethically-fair high-stakes testing, we will focus on defining and explaining high-stake testing, to include: evidence for validation, development of cut-scores, magnitudes of reliability coefficients, and other reliability measurement tools such as Generalizability Theory and Item-Response Theory.

TEACHABLE MOMENT

From our perspectives as educational psychometricians, we hope that this discussion will help foster scientifically-rigorous use and reporting of high-stakes testing in pharmacy education and research.

Development and validation of interprofessional learning assessment tool for health professionals in continuing professional development (CPD)

Introduction

Interprofessional learning (IPL) is increasingly recognized as a promising strategy in addressing the burgeoning challenges in healthcare. Its assessment remains to be perplexing and requires accurate measurements. Thus, this study intended to develop a valid and reliable reflective tool in assessing IPL as a Continuing Professional Development (CPD) outcome.

Methods

A one-group post-posttest pre-experimental design with tool development was employed to establish the validity and reliability of the “Inventory of Reflective Vignette–Interprofessional Learning” (IRV-IPL). This tool was developed from an extensive literature review and designed with three segments to assess interprofessional competencies before, after, and what if scenarios using vignettes. After it was validated by education experts (n = 5) and written consent forms were signed by the participants, the IRV-IPL was pilot tested among healthcare professionals (n = 10) for analysis and improvement. During the actual implementation, it was administered to healthcare professionals (n = 45) who participated in a university-provided CPD event. Collected data underwent validity and reliability testing.

Results

IRV-IPL generated excellent internal consistency (α = 0.98), and across all segments of collaboration (α = 0.96), coordination (α = 0.96), cooperation (α = 0.96), communication (α = 0.97), and commendation (α = 0.98). Items exhibited significantly positive large correlations (r > 0.35, p < 0.05) in all segments showing beneficial measures for postdictive validity in recalling prior interprofessional competencies, and predictive validity in estimating interprofessional learning as an outcome of CPD and alternative interventions.

Conclusion

This study provided a piece of groundwork evidence on the use of IRV-IPL as a reflective assessment tool for interprofessional learning in CPD contexts. Further studies are essential to explore the educational utility of IRV framework in crafting relevant assessments and to establish construct validity of IRV-IPL using exploratory and confirmatory factor analyses.

Developing a valid and reliable assessment of knowledge translation (KT) for continuing professional development program of health professionals

Introduction

Knowledge Translation (KT) is expected to be a critical learning outcome of a Continuing Professional Development (CPD) program. It continues to serve as an area of interest among educators and healthcare providers due to its importance to evidence-based practice. This study endeavored to develop a valid and reliable KT learning assessment tool in CPD.

Methods

The Inventory of Reflective Vignettes (IRV), an innovative approach of integrating research vignettes, was utilized in crafting the 20-item IRV-KT tool. This instrument includes knowledge creation and action as essential KT constructs. KT competency was assessed in three segments (i.e., before and after CPD event and if in a lecture) using a one-group post-posttest pre-experimental design. Health professionals who successfully completed a CPD program on a knowledge translation topic were asked to complete the IRV-KT during the pilot study (n = 10) and actual implementation (n = 45). Responses were subjected to Cronbach’s reliability and criterion-validity testing.

Results

The initial test of the IRV-KT tool demonstrated a high internal reliability (α = 0.97) and most items yielded acceptable validity scores. During the actual implementation, a higher reliability score of 0.98 was generated with significant correlations between the before-after segments for both KT constructs of creation (r = 0.33, p < 0.05) and action (r = 0.49, p < 0.05). All items have significant positive validity coefficients (r > 0.35, p < 0.05) in all segments of the tool.

Discussion

The study produced a reflective assessment tool to validly and reliably assess KT learning in a CPD. IRV-KT is seen to guide the curriculum process of CPD programs to bridge learning and healthcare outcomes.

Development of critical thinking in health professions education: A meta-analysis of longitudinal studies

INTRODUCTION

While reports of critical thinking exist in the health professions literature, development of critical thinking across a broad range of health-professions students has not been systematically reviewed.

METHODS

In this meta-analysis, multiple databases and journals were searched through February 2016 to identify longitudinal studies using standardized tests of critical thinking [California Critical Thinking Skills Test (CCTST), Health Science Reasoning Test (HSRT), and Defining Issues Test (DIT)] in any language. Two reviewers extracted information and collected information regarding primary author, publishing journal, health profession, critical thinking test, and time1/time2 means and standard deviations. Standardized mean differences (SMD) with 95% confidence intervals (CI) were reported using a random-effects model.

RESULTS

Four hundred sixty-two studies were screened, and 79 studies (representing 6884 students) were included. Studies contained 37 CCTST, 22 DIT, and 20 HSRT. Health professions comprised nursing, pharmacy, physical therapy, occupational therapy, dentistry, medicine, veterinary medicine, dental hygiene, clinical laboratory sciences, and allied health. Cohen's kappa was strong (0.82) for inter-reviewer agreement. Both the CCTST (SMD = 0.37, 95% CI = 0.23-0.52) and DIT (SMD = 0.28, 95%CI = 0.18-0.39) demonstrated significant increases in total scores, but the HSRT (SMD = 0.03, 95%CI = -0.05-0.12) did not show improvement.

DISCUSSION/CONCLUSIONS

In this meta-analysis, students from the majority of health professions consistently showed improvement in development of critical thinking. In this diverse population, only the CCTST and DIT appeared responsive to change.