The cognitive process of test takers when using the script concordance test rating scale

Development of script concordance test for assessment of clinical reasoning in nursing: Lessons learned regarding construct validity

Background

The script concordance test (SCT) has been shown to be an effective tool to assess the clinical reasoning skills of nursing students. Various nursing studies have demonstrated the construct validity of this test. However, studies on the barriers that may impede construct validity during the development process are limited.

Objective

This evaluation describes the barriers to the development of SCT for Bachelor's nursing students and the lessons learned regarding construct validity.

Methods

We conducted a descriptive evaluation of the SCT development and a validation process was performed. The evaluation was based on written comments during the assessment (N = 327), a Student's Perspective Questionnaire (N = 100), and student feedback during three live review sessions (N = 27).

Results

Despite consideration of the guidelines during SCT development, we encountered three main barriers that may impede construct validity. We undertook the necessary efforts to recruit an appropriate expert panel. We overestimated the experts' and students' understanding of the SCT methodology. Additionally, four potential causes of invalid item construction were identified. These possible causes were 'questionable intervention, hypothesis, or investigation', 'blurred data in new information', 'regression to the middle', and 'misinterpretation of the midpoint'.

Conclusion

The three lessons learned are as follows: 1) The recruitment of an appropriate expert panel must not be underestimated. Besides clinical expertise, experts need training in SCT methodology, including awareness of possible pitfalls; 2) SCT training is a prerequisite for SCT as an assessment; and 3) student feedback may offer a deeper understanding of potential hidden script errors and causes for misinterpretation of SCT. Further studies are necessary to identify additional causes which may impede the construct validity of SCT in nursing education.

Prediction of Graduate Learners' Academic Achievement in an Online Learning Environment Using a Blended Trauma Course

BACKGROUND

The concepts of online and blended education came into the limelight in the 19th century. Over time, the concepts expanded and reached a peak in 2021 in response to the COVID-19 lockdown. One of the challenges is the monitoring of the performance of distant learners. In face-to-face courses, an instructor can easily identify struggling learners during the regular meetings.

AIM OF THE STUDY

This study explored variables that can predict the academic achievement of learners early in online learning environments. Although there was no consensus, the factors were still hypothesized as predictors for academic achievement.

METHODS

A quasi-experimental study was conducted to test the hypothesis. Thirty-three graduate learners were enrolled in a blended trauma course. The learners' age, their previous experiences in online education, pre-test scores, and the number of logs to the online platform were studied. These elements were considered as predictors of academic achievement in the online aspect of the course.

RESULTS

The findings revealed that there was no statistically significant correlation between the age, the previous experience in online education, the pre-test scores, and the number of logs in the first two weeks. However, there was a statistically significant correlation between the number of logs into the online platform in the first three weeks of study and the learners' academic achievement. Additionally, the number of logs in the first three weeks was a statistically significant predictor for academic achievement in online education. This early prediction can help instructors to identify and support struggling learners.

CONCLUSION

The records of the online activity of learners in the first three weeks of study can help in early prediction of their academic achievement. Age, previous online education, and pretest scores were not statistically significant predictors.

Script concordance test acceptability and utility for assessing medical students' clinical reasoning: a user's survey and an institutional prospective evaluation of students' scores

Script Concordance Testing (SCT) is a method for clinical reasoning assessment in the field of health-care training. Our aim was to assess SCT acceptability and utility with a survey and an institutional prospective evaluation of students' scores.With a user's online survey, we collected the opinions and satisfaction data of all graduate students and teachers involved in the SCT setting. We performed a prospective analysis comparing the scores obtained with SCT to those obtained with the national standard evaluation modality.General opinions about SCT were mostly negative. Students tended to express more negative opinions and perceptions. There was a lower proportion of negative responses in the teachers' satisfaction survey. The proportion of neutral responses was higher for teachers. There was a higher proportion of positive positions towards all questions among teachers. PCC scores significantly increased each year, but SCT scores increased only between the first and second tests. PCC scores were found significantly higher than SCT scores for the second and third tests. Medical students' and teachers' global opinion on SCT was negative. At the beginning SCT scores were found quite similar to PCC scores. There was a higher progression for PCC scores through time.

A methodological study on the combination of qualitative and quantitative methods in cognitive interviewing for cross-cultural adaptation

Aim

The aim of this study is to explore the use of the Questionnaire Appraisal System with a combination of qualitative and quantitative methods in cognitive interviewing for cross‐cultural adaptation.

Design

This is a descriptive methodological study.

Methods

Using the Mandarin version of the Post‐Stroke Checklist as an example, cognitive interviews were conducted with 27 stroke survivors in Guangzhou between November 2020 and February 2021. The Questionnaire Appraisal System was applied as a codebook in focus group discussions to perform quantitative data collections and quantitative content analysis.

Results

Thirty‐eight problems were proposed in focus group discussions and identified all but four of the 30 questions that emerged in the cognitive interviews. A new item was added to the Questionnaire Appraisal System for better categorization. Four categories and six subcategories of problems in the checklist were revealed.

Script concordance tests: A call for action in dental education

The Script Concordance Test (SCT) is an educational tool that aims to assess the ability to interpret medical information under conditions of uncertainty. It is widely used and validated in health education, but almost unknown in dentistry. Based on authentic clinical problem-solving situations, it allows to assess clinical reasoning that experienced health workers develop over the years. A specific scoring system, dedicated to SCT, considers the variability of responses of practitioners in the same clinical situations. Finally, the scores generated by SCT reflect the respondents' ability to interpret clinical data compared to experienced clinicians. This article aims to familiarise the dental educators' community with SCT construction, optimisation and its possible applications.

Script Concordance Approach in Nursing Education

BACKGROUND

The script concordance approach aims at triggering judgments in simulated contexts of uncertainty.

PROBLEM

Nursing students need to be prepared to manage the uncertainty of clinical practice.

APPROACH

The purpose of this article is to describe the theoretical foundation and the pedagogical use of the script concordance approach, as well as to present the current state of nursing evidence on the subject. The script concordance approach includes (1) script concordance testing, which is a quantitative examination that evaluates clinical reasoning; (2) a face-to-face script concordance activity; and (3) a digital educational strategy based on script concordance delivered via an online teaching/learning platform that aims to support clinical reasoning development.

CONCLUSIONS

Relying on questioning and experts' modeling, the script concordance offers an innovative pedagogical approach that approximates the uncertainty of clinical practice.

The Impact of Surgeon Experience on Script Concordance Test Scoring

OBJECTIVE

The Script Concordance Test (SCT) is a test of clinical decision-making that relies on an expert panel to create its scoring key. Existing literature demonstrates the value of specialty-specific experts, but the effect of experience among the expert panel is unknown. The purpose of this study was to explore the role of surgeon experience in SCT scoring.

DESIGN

An SCT was administered to 29 general surgery residents and 14 staff surgeons. Staff surgeons were stratified as either junior or senior experts based on years since completing residency training (<15 versus >25 years). The SCT was scored using the full expert panel, the senior panel, the junior panel, and a subgroup junior panel in practice <5 years. A one-way ANOVA was used to compare the scores of first (R1) and fifth (R5) year residents using each scoring scheme. Cognitive interviews were analyzed for differences between junior and senior expert panelist responses.

RESULTS

There was no statistically significant difference between the mean score of six R1s and five R5s using the full expert panel (R1 69.08 versus R5 67.06, F_1,9 = 0.10, P = 0.76), the junior panel (R1 66.73 versus R5 62.50, F_1,9 = 0.35, P = 0.57), or the subgroup panel in practice <5 years (R1 61.07 versus R5 58.79, F_1,9 = 0.18, P = 0.75). However, the average score of R1s was significantly lower than R5s when using the senior faculty panel (R1 52.04 versus R5 63.26, F_1,9 = 26.90, P = 0.001). Cognitive interview data suggests that some responses of junior experts demonstrate less confidence than those of senior experts.

CONCLUSIONS

SCT scores are significantly affected by the responses of the expert panel. Expert differences between first and fifth year residents were only demonstrated when using an expert panel consisting of senior faculty members. Confidence may play a role in the response selections of junior experts. When constructing an SCT expert panel, consideration must be given to the experience of panel members.

Pediatric Hospitalists' Performance and Perceptions of Script Concordance Testing for Self-Assessment

OBJECTIVES

The cognitive expertise of Pediatric Hospitalists (PH) lies not in standard knowledge but in making decisions under conditions of uncertainty. To maintain expertise, PH should engage in deliberate practice via self-assessments that promote higher-level cognitive processes necessary to address problems with missing or ambiguous information. Higher levels of cognition are purported with Script Concordance Test (SCT) questions compared to Multiple Choice Questions (MCQ). To determine if PH use higher levels of cognition when answering SCT versus MCQ questions and to analyze participants' perceptions of the utility of using SCT self-assessment for deliberate practice in addressing clinical problems encountered in daily practice.

METHODS

This is a mixed methods study comparing the cognitive level expressed according to Bloom's Taxonomy by PH answering MCQ versus SCT questions using a "think aloud" (TA) exercise, followed by qualitative analysis of interviews conducted afterward.

RESULTS

A significantly greater percentage of comments were coded as higher cognitive processes (apply, analyze, evaluate, and create) for SCT versus MCQ (74% vs 19%) compared with lower order (remember, understand); chi-square P < .00001. Analysis of interviews revealed 6 themes.

CONCLUSION

SCT questions elicited higher level cognition essential to clinical reasoning compared to MCQ questions. PH-indicated MCQ questions measure standard knowledge, while SCT questions better measure decision-making under conditions of uncertainty. PH-perceived SCT could be useful for deliberate practice in Pediatric Hospital Medicine decision-making if they could compare their rationale in answering questions with that of experts.

How do cognitive processes influence script concordance test responses?

INTRODUCTION

The script concordance test (SCT) is a test of clinical decision-making (CDM) that compares the thought process of learners to that of experts to determine to what extent their cognitive 'scripts' align. Without understanding test-takers' cognitive process, however, it is unclear what influences their responses. The objective of this study was to gather response process validity evidence by studying the cognitive process of test-takers to determine whether the SCT tests CDM and what cognitive processes may influence SCT responses.

METHODS

Cases from an SCT used in a national validation study were administered and semi-structured cognitive interviews were conducted with ten residents and five staff surgeons. A retrospective verbal probing technique was used. Data was independently analysed and coded by two analysts. Themes were identified as factors that influence SCT responses during the cognitive interview.

RESULTS

Cognitive interviews demonstrated variability in CDM among test-takers. Consistent with dual process theory, test-takers relied on scripts formed through past experiences, when available, to make decisions and used conscious deliberation in the absence of experience. However, test-takers' response process was also influenced by their comprehension of specific terms, desire for additional information, disagreement with the planned management, underlying knowledge gaps and desire to demonstrate confidence or humility.

CONCLUSION

The rationale behind SCT answers may be influenced by comprehension, underlying knowledge and social desirability in addition to formed scripts and/or conscious deliberation. Having test-takers verbalise their rationale for responses provides a depth of assessment that is otherwise lost in the SCT's current format. With the improved ability to standardise CDM assessment using the SCT, consideration of test-makers improving the SCT construction process and combining the SCT question format with verbal responses may improve the use of the SCT for CDM assessment.

Using cognitive interviews and think-aloud protocols to understand thought processes

OUR SITUATION

Cognitive interviews (CIs) and think-aloud protocols (TAPs) are two strategies that offer unique approaches to capture participant thought processes when engaging with instruments and interventions. In this paper, we describe the origins of CIs and TAPs, the types of research questions they can be used to address, and strategies to implement these approaches.

METHODOLOGICAL LITERATURE REVIEW

CIs and TAPs are interview methods that make thought processes more explicit. CIs include targeted interview questions that are focused on evaluating participant comprehension during an activity, such as a survey. TAPs request the participant to think-aloud while they solve a problem or complete a task; this is to help understand the problem-solving process.

OUR RECOMMENDATIONS AND THEIR APPLICATIONS

An understanding of the research aim is critical to identify whether a CI or TAP is preferred. Researchers should ensure that participants are selected based on the needs of the research and that they are appropriately prepared for the selected interview type. Individuals can analyze verbal reports from CIs and TAPs using qualitative and quantitative methods, and researchers are encouraged to disseminate their findings to inform design practices of other researchers.

POTENTIAL IMPACT

CIs and TAPs are methods that help pharmacy education researchers explore thought processes; specifically, these methods are useful to study test response processes, improve survey construction, and map cognitive models of complex thought processes. CIs and TAPs are methodologies that should be in the toolbox of pharmacy education researchers to support design, evaluation, and refinement of instruments and educational interventions.