Students' perceptions of the Script Concordance Test and its impact on their learning behavior: a mixed methods study

Test for clinical reasoning evaluation in Speech-Language Pathology: content validity

PURPOSE

To validate the content of the Speech-Language Pathology Concordance Test called FonoTCS.

METHODS

This is a content validation study of the instrument. Five speech-language pathologists, all with doctoral degrees and teaching experience, averaging 24.8 years of professional practice, participated in the development of FonoTCS and reached a consensus during the process. Thirty questions and 120 items were created, covering seven areas of speech-language pathology expertise across three domains. For content validation, FonoTCS was electronically sent to 15 evaluators to respond to a questionnaire with five questions, rated on a five-point scale, regarding the criteria of clarity, ethics, and relevance of the questions. The Corrected Content Validity Coefficient was calculated for all statements to analyze the responses. Questions with agreement percentages equal to or less than 80% were revised.

RESULTS

Thirteen evaluators, all female, with an average age of 39.07 years, including eight with master's degrees and five with doctoral degrees, and an average clinical practice experience of 15.38 years, participated in the analysis. The average Corrected Content Validity Coefficient values for the clarity criterion were 0.93 and 0.95, for the relevance criterion 0.98 and 0.92, and for the ethics criterion 0.99. Two questions received scores of 0.78 and 0.80, both related to the audiology area in the assessment/diagnosis domain, specifically question 2 regarding the relevance criterion. These questions were reviewed and restructured by the judges.

CONCLUSION

FonoTCS is a valid instrument from a content perspective.

Comparison between MCQ, Extended matching questions (EMQ) and Script concordance test (SCT) for assessment among first-year medical students - A pilot study

BACKGROUND

Single-best response types of MCQs are commonly used tools in medical assessment. However, these are not particularly apt for the assessment of higher-order thinking skills (HOTS) among students. Assessment of HOTS and clinical reasoning skills requires unexplored tools like extended matching questions (EMQs) and SCTs. The aim is to assess HOTS and clinical reasoning skills during formative assessment among medical students post a simulation-based education (SBE) using EMQ and SCT on the topic of shock and collect student perceptions regarding new assessment tools.

METHODS AND MATERIAL

The research is an observational descriptive study. Fifty-two first-year medical students were asked to take a formative assessment consisting of 20 MCQs, 6 EMQs, and 2 SCT post a SBE during July 2022 on the topic of shock. MCQs were categorized into themes of aetiology, pathophysiology, and management of shock. These categorized MCQs were compared and analyzed with EMQs and SCTs prepared on the same themes. The data analysis by a one-way analysis of variance (ANOVA) was conducted to compare the difference in per cent mean scores of MCQ with EMQ and MCQ with SCT.

RESULTS

The overall mean scores and also under the theme of aetiology and management, the student's score in MCQ were higher than EMQ and SCT and the difference was statistically significant with the P value (≤0.001).

CONCLUSIONS

Students scored better in familiar assessment tools like MCQ, but majority of the students were of the opinion that EMQ tested the delivered content better and SCT tested the clinical application better.

The Impact of Script Concordance Testing on Clinical Decision-Making in Paramedic Education

This study investigates the effectiveness of the Script Concordance Test (SCT) in enhancing clinical reasoning skills within paramedic education. Focusing on the Medical University of Lublin, we evaluated the SCT's application across two cohorts of paramedic students, aiming to understand its potential to improve decision-making skills in emergency scenarios. Our approach, informed by Van der Vleuten's assessment framework, revealed that while the SCT's correlation with traditional methods like multiple-choice questions (MCQs) was limited, its formative nature significantly contributed to improved performance in summative assessments. These findings suggest that the SCT can be an effective tool in paramedic training, particularly in strengthening cognitive abilities critical for emergency responses. The study underscores the importance of incorporating innovative assessment tools like SCTs in paramedic curricula, not only to enhance clinical reasoning but also to prepare students for effective emergency responses. Our research contributes to the ongoing efforts in refining paramedic education and highlights the need for versatile assessment strategies in preparing future healthcare professionals for diverse clinical challenges.

Prueba de concordancia de guiones para entrenar el razonamiento clínico en estudiantes de fonoaudiología

La prueba de concordancia de guiones (PCG) ha sido utilizada en el entrenamiento y evaluación del razonamiento clínico (RC) como una estrategia innovadora en la formación de profesionales. Sin embargo, no se dispone de evidencia de su aplicación en el pregrado de fonoaudiología. El objetivo de esta investigación fue analizar el desempeño y la percepción de estudiantes de fonoaudiología con respecto al uso de scripts. Se diseñó un piloto pre-experimental y multicéntrico, complementado con tres grupos focales. Las variables cuantitativas continuas fueron resumidas a través de medias y desviación estándar. La comparación entre grupos se ejecutó con Anova one way y la prueba post hoc de Bonferroni, considerando un nivel de significancia p<.05. La fase cualitativa incorporó un análisis de contenido mediante la codificación abierta de textos y la identificación e interpretación de familias de significado emergentes. El rendimiento promedio de los estudiantes fue de 4.03 (DS= 0.35), observándose un incremento en el rendimiento de RC durante el semestre (p= 0.03). La percepción de los estudiantes resulto positiva y se identificó cuatro familias de significado relacionadas con: razonamiento clínico, oportunidades de mejora implementación de la estrategia y retroalimentación docente. A modo de conclusión, la incorporación de scripts en estudiantes de pregrado de fonoaudiología es factible, incrementa el rendimiento y apoya el desarrollo del RC.

The Use of a Modified Script Concordance Test in Clinical Rounds to Foster and Assess Clinical Reasoning Skills

The development of clinical reasoning skills is a high priority during clinical service, but an unpredictable case load and limited time for formal instruction makes it challenging for faculty to foster and assess students' individual clinical reasoning skills. We developed an assessment for learning activity that helps students build their clinical reasoning skills based on a modified version of the script concordance test (SCT). To modify the standard SCT, we simplified it by limiting students to a 3-point Likert scale instead of a 5-point scale and added a free-text box for students to provide justification for their answer. Students completed the modified SCT during clinical rounds to prompt a group discussion with the instructor. Student feedback was positive, and the instructor gained valuable insight into the students' thought process. A modified SCT can be adopted as part of a multimodal approach to teaching on the clinic floor. The purpose of this article is to describe our modifications to the standard SCT and findings from implementation in a clinical rounds setting as a method of formative assessment for learning and developing clinical reasoning skills.

Script concordance tests: A call for action in dental education

The Script Concordance Test (SCT) is an educational tool that aims to assess the ability to interpret medical information under conditions of uncertainty. It is widely used and validated in health education, but almost unknown in dentistry. Based on authentic clinical problem-solving situations, it allows to assess clinical reasoning that experienced health workers develop over the years. A specific scoring system, dedicated to SCT, considers the variability of responses of practitioners in the same clinical situations. Finally, the scores generated by SCT reflect the respondents' ability to interpret clinical data compared to experienced clinicians. This article aims to familiarise the dental educators' community with SCT construction, optimisation and its possible applications.

Assessing the acceptability of script concordance testing: a nationwide study in otolaryngology

Background

Script concordance testing (SCT) is an objective method to evaluate clinical reasoning that assesses the ability to interpret medical information under conditions of uncertainty. Many studies have supported its validity as a tool to assess higher levels of learning, but little is known about its acceptability to major stakeholders. The aim of this study was to determine the acceptability of SCT to residents in otolaryngology – head and neck surgery (OTL-HNS) and a reference group of experts.

Methods

In 2013 and 2016, a set of SCT questions, as well a post-test exit survey, were included in the National In-Training Examination (NITE) for OTL-HNS. This examination is administered to all OTL-HNS residents across Canada who are in the second to fifth year of residency. The same SCT questions and survey were then sent to a group of OTL-HNS surgeons from 4 Canadian universities.

Results

For 64.4% of faculty and residents, the study was their first exposure to SCT. Overall, residents found it difficult to adapt to this form of testing, thought that the clinical scenarios were not clear and believed that SCT was not useful for assessing clinical reasoning. In contrast, the vast majority of experts felt that the test questions reflected real-life clinical situations and would recommend SCT as an evaluation method in OTL-HNS.

Conclusion

Views about the acceptability of SCT as an assessment tool for clinical reasoning differed between OTL-HNS residents and experts. Education about SCT and increased exposure to this testing method are necessary to improve residents’ perceptions of SCT.

Encouraging Critical Clinical Thinking (CCT) Skills in First-Year Veterinary Students

First-year didactic course instructors at the University of Illinois College of Veterinary Medicine leverage earlier clinical rotation experiences with weekly "Clinical Correlations" exercises to provide early exposure to critical clinical thinking (CCT). This study evaluated the efficacy of individual and paired group exercises on CCT development. Before and after instruction, the Cornell Critical Thinking Test (Level Z) (CCTTZ) was administered. Based on the hypothesis that students with higher scores would coach lower-scoring colleagues during group exercises, heterogeneous groups with similar mean scores were established for the year. Students completed 14 individual and paired group exercises over 6 months. Exercises were designed to increase in complexity and decline in scaffolding. Seven of the exercises were cases using the Applied Learning Platform (ALP) at http://www.whenknowingmatters.com . Student analyses were scored according to a six-category critical-thinking rubric using a 5-point scale. Consistent with our hypothesis, individual and group rubric scores increased significantly, plateauing near the end of the year. Contrary to our hypothesis, mean overall CCTTZ scores did not change, but there was a small statistically significant increase in the ability to assess the validity of an argument. Student attitudes were mixed. Positive comments focused on reinforcement of prior didactic instruction, while negative comments focused on preparation time needed to conduct research on clinical concepts, and on a lack of explicit evaluation by summative examinations. Nonetheless, end-of-year GPAs correlated linearly with cumulative individual rubric scores. In summary, the value of early curriculum CCT training was confirmed when discipline-specific criteria were applied.

Scaffolded Active Learning: Nine Pedagogical Principles for Building a Modern Veterinary Curriculum

Veterinary discipline experts unfamiliar with the broader educational literature can find the adoption of an evidence-based approach to curriculum development challenging. However, greater societal and professional demands for achieving and verifying Day One knowledge and skills, together with continued progress in information generation and technology, make it all the more important that the defined period for initial professional training be well used. This article presents and discusses nine pedagogical principles that have been used in modern curricular development in Australia, the United Kingdom, and the United States: (1) outcomes-based curriculum design; (2) valid and reliable assessments; (3) active learning; (4) integrated knowledge for action; (5) tightly controlled core curriculum; (6) "just-in-time" rather than "just-in-case" knowledge; (7) vertical integration, the spiral curriculum, and sequential skills development; (8) learning skills support; and (9) bridges from classroom to workplace. Crucial to effective educational progress is active learning that embraces the skills required by the modern professional, made possible by tight control of curricular content. In this information age, professionals' ability to source information on a "just-in-time" basis to support high quality reasoning and decision making is far more important than the memorization of large bodies of increasingly redundant information on a "just-in-case" basis. It is important that those with responsibility for veterinary curriculum design ensure that their programs fully equip the modern veterinary professional for confident entry into the variety of roles in which society needs their skills.