Accuracy of script concordance tests in fourth-year medical students

The Impact of Script Concordance Testing on Clinical Decision-Making in Paramedic Education

This study investigates the effectiveness of the Script Concordance Test (SCT) in enhancing clinical reasoning skills within paramedic education. Focusing on the Medical University of Lublin, we evaluated the SCT's application across two cohorts of paramedic students, aiming to understand its potential to improve decision-making skills in emergency scenarios. Our approach, informed by Van der Vleuten's assessment framework, revealed that while the SCT's correlation with traditional methods like multiple-choice questions (MCQs) was limited, its formative nature significantly contributed to improved performance in summative assessments. These findings suggest that the SCT can be an effective tool in paramedic training, particularly in strengthening cognitive abilities critical for emergency responses. The study underscores the importance of incorporating innovative assessment tools like SCTs in paramedic curricula, not only to enhance clinical reasoning but also to prepare students for effective emergency responses. Our research contributes to the ongoing efforts in refining paramedic education and highlights the need for versatile assessment strategies in preparing future healthcare professionals for diverse clinical challenges.

Prueba de concordancia de guiones para entrenar el razonamiento clínico en estudiantes de fonoaudiología

La prueba de concordancia de guiones (PCG) ha sido utilizada en el entrenamiento y evaluación del razonamiento clínico (RC) como una estrategia innovadora en la formación de profesionales. Sin embargo, no se dispone de evidencia de su aplicación en el pregrado de fonoaudiología. El objetivo de esta investigación fue analizar el desempeño y la percepción de estudiantes de fonoaudiología con respecto al uso de scripts. Se diseñó un piloto pre-experimental y multicéntrico, complementado con tres grupos focales. Las variables cuantitativas continuas fueron resumidas a través de medias y desviación estándar. La comparación entre grupos se ejecutó con Anova one way y la prueba post hoc de Bonferroni, considerando un nivel de significancia p<.05. La fase cualitativa incorporó un análisis de contenido mediante la codificación abierta de textos y la identificación e interpretación de familias de significado emergentes. El rendimiento promedio de los estudiantes fue de 4.03 (DS= 0.35), observándose un incremento en el rendimiento de RC durante el semestre (p= 0.03). La percepción de los estudiantes resulto positiva y se identificó cuatro familias de significado relacionadas con: razonamiento clínico, oportunidades de mejora implementación de la estrategia y retroalimentación docente. A modo de conclusión, la incorporación de scripts en estudiantes de pregrado de fonoaudiología es factible, incrementa el rendimiento y apoya el desarrollo del RC.

Development of a script concordance test to assess clinical reasoning in a pharmacy curriculum

INTRODUCTION

Clinical reasoning is a vital skill for student pharmacists in the provision of patient-centered care, but these skills are often difficult to assess in the didactic curriculum. A script concordance test (SCT) is an innovative assessment method that can be used to assess clinical reasoning skills. The objective of this study was to develop and refine an SCT to assess clinical reasoning skills of third year student pharmacists (P3s).

METHODS

An SCT was written and administered to P3s. Pharmacy practice faculty members served as the expert group. The SCT was scored and Rasch analysis was performed.

RESULTS

The SCT included 20 case vignettes and 60 questions. Test reliability was 0.34 with mean square values for all items between 0.7 and 1.3. Forty-two questions had a difficulty score between 0 and - 1 logits, indicating there were multiple questions with similar difficulty levels. Two case vignettes and 43.3% of questions (n = 26) were revised to enhance clarity and decrease ambiguity.

CONCLUSIONS

The SCT is a tool to assess clinical reasoning in the didactic curriculum. Faculty can create the SCT and use statistical methods such as Rasch analysis to assess validity and reliability of the SCT.

Are script concordance tests suitable for the assessment of undergraduate students? A multicenter comparative study

INTRODUCTION

Script concordance tests (SCTs) have been developed to assess clinical reasoning in uncertain situations. Their reliability for the evaluation of undergraduate medical students has not been evaluated.

METHODS

Twenty internal medicine SCT cases were implemented in undergraduate students of two programs. The results obtained on the SCTs were compared to those obtained by the same students on clinical-based classical multiple-choice questions (MCQs).

RESULTS

A total of 551/883 students (62%) answered the SCTs. The mean aggregate score (based on a total 20 points) was 11.54 (3.29). The success rate and mean score for each question did not differ depending on the modal response but the discrimination rate did. The results obtained by the students on the SCT test correlated with their scores on the MCQ tests. Among students, 446/517 (86%) considered the SCTs to be more difficult than classical MCQs, although the mean score did not differ between the SCT and MCQ tests.

CONCLUSION

The use of SCTs is a feasible option for the evaluation of undergraduate students. The SCT scores correlated with those obtained on classical MCQ tests.

Development and psychometrics of script concordance test (SCT) in midwifery

Background: Clinical reasoning plays an important role in the accurate diagnosis and treatment of diseases. Script Concordance test (SCT) is one of the tools that assess clinical reasoning skill. This study was conducted to determine the reliability and concurrent and predictive validity of SCT in assessing final lessons and gynecology exams of undergraduate midwifery students.

Methods: At first, 20 clinical scenarios followed by 3 questions were designed by 2 experienced midwives. Then, after examining the content validity, 15 scenarios were selected. The test was used for 55 midwifery students. The correlation of SCT results with grade point average (GPA) was measured. To evaluate the concurrent validity of SCT, the correlation between SCT scores and the final exam of the gynecology course was measured. To measure predictive validity, the correlation of SCT scores with comprehensive exams of midwifery was calculated. Data were analyzed using SPSS software. Descriptive statistics, Pearson correlation, and coefficient Cronbach's alpha were used for analysis. The test’s item difficulty level (IDL) and item discriminative index (IDI) were determined using Whitney and Sabers’ method.

Results: The internal reliability of the test (calculated using Cronbach’s alpha coefficient) was 0.74. All questions were positively correlated with the total score. The highest correlation coefficient was related to GPA and comprehensive test with the score of 0.91. The correlation coefficient between SCT and the final test (concurrent validity) was 0.654, and the correlation coefficient between SCT and comprehensive test (predictive validity) was 0.721. The range of item discriminative index and item difficulty level in this exam was 0.39-0.59 and 0.32-0.66, respectively.

Conclusion: SCT shows a relatively high internal validity and can predict the success rate of students in the comprehensive exams of midwifery. Also, it showed a high concurrent validity in the final test of gynecology course. This test could be a good alternative for formative and summative tests of clinical courses.

Multicentered Study Evaluating Pharmacy Students’ Perception of Palliative Care and Clinical Reasoning Using Script Concordance Testing

Introduction:

As the role of the pharmacist on the transdisciplinary palliative care team grows, the need for adequate instruction on palliative care and clinical reasoning skills in schools of pharmacy grows accordingly.

Methods:

This study evaluates second- and third-year pharmacy students from 6 accredited schools of pharmacy that participated in surveys before and after the delivery of a didactic palliative care elective. The survey collected student demographics, perceptions of the importance of and student skill level in palliative care topics. The script concordance test (SCT) was used to assess clinical decision-making skills on patient cases. Student scores on the SCT were compared to those of a reference panel of experts.

Results:

A total of 89 students completed the pre-/postsurveys and were included in data analysis. There was no statistically significant difference in student perceived importance of palliative care skills before and after the elective. Students from all 6 institutions showed significant increase in confidence in their palliative care skills at the end of the course. There was also a significant improvement across all institutions in clinical reasoning skills in most of the SCT questions used to assess these skills.

Conclusions:

Students choosing an elective in palliative care likely do so because they already have an understanding of the importance of these topics in their future practice settings. Delivery of a palliative care elective in the pharmacy curriculum significantly increases both student confidence in their palliative care skills and their clinical reasoning skills in these areas.