Script concordance testing: from theory to practice: AMEE guide no. 75

Development of script concordance test for assessment of clinical reasoning in nursing: Lessons learned regarding construct validity

Background

The script concordance test (SCT) has been shown to be an effective tool to assess the clinical reasoning skills of nursing students. Various nursing studies have demonstrated the construct validity of this test. However, studies on the barriers that may impede construct validity during the development process are limited.

Objective

This evaluation describes the barriers to the development of SCT for Bachelor's nursing students and the lessons learned regarding construct validity.

Methods

We conducted a descriptive evaluation of the SCT development and a validation process was performed. The evaluation was based on written comments during the assessment (N = 327), a Student's Perspective Questionnaire (N = 100), and student feedback during three live review sessions (N = 27).

Results

Despite consideration of the guidelines during SCT development, we encountered three main barriers that may impede construct validity. We undertook the necessary efforts to recruit an appropriate expert panel. We overestimated the experts' and students' understanding of the SCT methodology. Additionally, four potential causes of invalid item construction were identified. These possible causes were 'questionable intervention, hypothesis, or investigation', 'blurred data in new information', 'regression to the middle', and 'misinterpretation of the midpoint'.

Conclusion

The three lessons learned are as follows: 1) The recruitment of an appropriate expert panel must not be underestimated. Besides clinical expertise, experts need training in SCT methodology, including awareness of possible pitfalls; 2) SCT training is a prerequisite for SCT as an assessment; and 3) student feedback may offer a deeper understanding of potential hidden script errors and causes for misinterpretation of SCT. Further studies are necessary to identify additional causes which may impede the construct validity of SCT in nursing education.

Concordance of judgement: A tool to foster the development of professionalism in dentistry

INTRODUCTION

Developing professionalism notably involves learning how to make professional judgements in ambiguous situations. The Concordance of Judgement Test (CJT) is a learning tool that was proposed to develop professionalism competencies, but it was never performed in dentistry or used with a synchronous methodology. The present study evaluated the feasibility of the use of CJT in the context of dental education, to foster professionalism and stimulate reflexivity and discussion.

MATERIALS AND METHODS

After different steps of optimization, a questionnaire presenting 12 vignettes was submitted to 33 Canadian students. Second, after an additional optimization, a questionnaire of 7 vignettes was submitted to 87 French students. An immediate educational feedback was proposed after each vignette to promote reflexivity and discussions during the experience.

RESULTS

The overall experience of the students was reported as good, thanks to the feedback of real-life situations. This promoted reflexivity and stimulated discussion between students and educators regarding professionalism issues. The students considered CJT as a relevant and well-adapted tool, and reported positive feelings regarding the inter-university aspect of the activity. The mean score of the panel members was close to 80/100 and the mean score of the students was 5 to 10 points lower, which is in agreement with docimological performance.

CONCLUSION

The results suggested that the use of CJT in a synchronous way was a feasible and relevant tool to motivate the students to improve their professionalism, and to stimulate their reflexivity and discussion. The students reported positive experience with CJT, and we believe that this tool can be integrated in the dental curriculum.

[The expert panel for Script Concordance Tests: A truly adequate reference?]

The Script Concordance Tests (SCTs) are an examination modality introduced by decree in the French National Ranking Exam for medical students in 2024. Their objective is to evaluate clinical reasoning in situations of uncertainty. In practice, SCTs assess the impact of new information on the probability of a hypothesis formulated a priori based on an authentic clinical scenario. This approach resembles probabilistic (or Bayesian) reasoning. Due to the uncertainty associated with the explored clinical situation, SCTs do not compare the student's response to an expected one in a theoretical knowledge reference. Instead, the distribution of responses from a panel of experienced physicians is used to establish the question's scoring scale. Literature data suggest that physicians, even experienced ones, like most humans, often exhibit biased intuitive probabilistic reasoning. These biases raise questions about the relevance of using expert panel responses as scoring scales for SCTs.

Reliability of rubrics in the assessment of clinical oral presentation: A prospective controlled study

BACKGROUND

Rubrics are assessment tools customarily designed to judge a student's skills in diagnosis, clinical assessment, presentations. as well as academic performance. The aim of the present study was to assess the reliability of rubrics in clinical oral presentations by students.

MATERIALS AND METHODS

This was a prospective control study that enrolled 300 undergraduate students, students in their internships, and post-graduate students in various clinical streams. Ethical committee approval was obtained before finalizing the study from appropriate regulatory and institutional bodies. A pre-validated questionnaire was designed and distributed to all the participating students. All students were assessed using the traditional examination method as well as by rubric assessment. The responses obtained were compared and statistically analyzed using one-way ANOVA and the Chi-square statistical tool.

RESULTS

Rubric assessment was found to have high statistical significance (P < 0.0001) when compared with the traditional method of examination.

CONCLUSION

In the present study, rubrics were found to serve as an effective method for judging student skills following a clinical oral presentation. However, due to contradictory scientific evidence, there is a requirement for a large sample size to be included for rubric or traditional assessment method studies.

Test for clinical reasoning evaluation in Speech-Language Pathology: content validity

PURPOSE

To validate the content of the Speech-Language Pathology Concordance Test called FonoTCS.

METHODS

This is a content validation study of the instrument. Five speech-language pathologists, all with doctoral degrees and teaching experience, averaging 24.8 years of professional practice, participated in the development of FonoTCS and reached a consensus during the process. Thirty questions and 120 items were created, covering seven areas of speech-language pathology expertise across three domains. For content validation, FonoTCS was electronically sent to 15 evaluators to respond to a questionnaire with five questions, rated on a five-point scale, regarding the criteria of clarity, ethics, and relevance of the questions. The Corrected Content Validity Coefficient was calculated for all statements to analyze the responses. Questions with agreement percentages equal to or less than 80% were revised.

RESULTS

Thirteen evaluators, all female, with an average age of 39.07 years, including eight with master's degrees and five with doctoral degrees, and an average clinical practice experience of 15.38 years, participated in the analysis. The average Corrected Content Validity Coefficient values for the clarity criterion were 0.93 and 0.95, for the relevance criterion 0.98 and 0.92, and for the ethics criterion 0.99. Two questions received scores of 0.78 and 0.80, both related to the audiology area in the assessment/diagnosis domain, specifically question 2 regarding the relevance criterion. These questions were reviewed and restructured by the judges.

CONCLUSION

FonoTCS is a valid instrument from a content perspective.

Evaluating clinical reasoning in first year DPT students using a script concordance test

BACKGROUND

A script concordance test (SCT) provides a series of clinical vignettes to assess clinical reasoning in uncertainty. Appraised throughout health education literature, SCTs are cognitive assessments of clinical reasoning, though their use in Doctor of Physical Therapy (DPT) entry-level education has not been investigated. The purpose of this study was to develop and explore the reliability and validity of a SCT for first year DPT students.

METHODS

The SCT was developed and implemented over four phases. During phases one and two, DPT program faculty consulted on course content from the first-year curriculum. Thirty clinical vignettes with three follow-up questions each were constructed. The SCT was pilot tested with five clinicians in phase three to assess question clarity. During phase four, the SCT was administered to students and a reference panel via Qualtrics. First year DPT students (n = 44) and reference panel physical therapists with at least two years of experience and advanced certification (n = 15) completed the SCT. Internal consistency was analyzed using Cronbach's Alpha. Differences between student and reference panel percent-correct scores were analyzed with a t-test. Relationships between student SCT scores and academic records were explored with Spearman's Rho.

RESULTS

The SCT had an internal consistency of 0.74. A significant difference in scores was found between the students [mean 58.5 (+/-5.31)] and reference panel [65.8 (+/-4.88), p < .01]. No significant correlations between student SCT scores and academic records were found.

CONCLUSIONS

The developed SCT was reliable and demonstrated satisfactory internal consistency among test items. The SCT successfully differentiated between groups, with the reference panel demonstrating statistically significant higher percent-correct scores compared to students. SCTs may provide means to measure clinical reasoning in DPT students and lead to novel pedagogical approaches to enhance clinical reasoning.

The Impact of Script Concordance Testing on Clinical Decision-Making in Paramedic Education

This study investigates the effectiveness of the Script Concordance Test (SCT) in enhancing clinical reasoning skills within paramedic education. Focusing on the Medical University of Lublin, we evaluated the SCT's application across two cohorts of paramedic students, aiming to understand its potential to improve decision-making skills in emergency scenarios. Our approach, informed by Van der Vleuten's assessment framework, revealed that while the SCT's correlation with traditional methods like multiple-choice questions (MCQs) was limited, its formative nature significantly contributed to improved performance in summative assessments. These findings suggest that the SCT can be an effective tool in paramedic training, particularly in strengthening cognitive abilities critical for emergency responses. The study underscores the importance of incorporating innovative assessment tools like SCTs in paramedic curricula, not only to enhance clinical reasoning but also to prepare students for effective emergency responses. Our research contributes to the ongoing efforts in refining paramedic education and highlights the need for versatile assessment strategies in preparing future healthcare professionals for diverse clinical challenges.

Script concordance testing in genetic counseling training: A pilot study

Clinical reasoning is a complex skill that represents a trainee's ability to use their professional knowledge and skills to assess and solve the problems that arise in clinical practice. As an integral tenet of the genetic counseling process, clinical reasoning skills underlie many of the Practice-Based Competencies (2019) across a variety of domains. Despite the long-lasting recognition of the importance of this complex skill in the training of genetic counselors, clinical reasoning has traditionally been difficult to assess in a standardized way in healthcare education. Script concordance testing is a standardized method of assessing clinical reasoning skills in ambiguous clinical situations. The tool has been used to successfully measure the clinical reasoning skills of trainees in various healthcare training programs and has never been used in a genetic counseling training program. We conducted a pilot study to assess the utility of script concordance testing in the field of genetic counseling as an objective measure of clinical reasoning in trainees. The script concordance test was constructed for the field of genetic counseling and administered to 22 second year genetic counseling students in the Joan H. Marks Graduate Program in Human Genetics at Sarah Lawrence College. Twelve genetic counselors served on a panel to provide expert clinical reasoning responses and a scoring grid was developed using the aggregate scores method. The utility of the tool was measured using Cronbach's alpha coefficient, and scores of students and the panel were compared using Hedge's g. Results revealed statistically significant differences between the scores of panelists and students and good reliability. This study shows that script concordance testing can be used to measure clinical reasoning skills in genetic counseling trainees in a way that is reliable, standardized, and easy to use, thereby allowing programs to better assess the clinical reasoning skills of trainees prior to graduation.

Adapting to a Pandemic: Web-Based Residency Training and Script Concordance Testing in Emergency Medicine During COVID-19

OBJECTIVE

The coronavirus disease (COVID-19) pandemic necessitated alternative methods to ensure the continuity of medical education. Our study explores the efficacy and acceptability of a digital continuous medical education initiative for medical residents during this challenging period.

METHODS

From September to December 2020, 47 out of 60 enrolled trainee doctors participated in this innovative digital Continuous Medical Education (CME) approach. We utilized the Script Concordance Test to bolster clinical reasoning skills. Three simulation scenarios, namely Advanced Trauma Life Support (ATLS), Advanced Life Support (ALS), and European Paediatric Life Support (EPLS), were transformed into interactive online sessions via Zoom™. Participant feedback was also collected through a survey.

RESULTS

Consistent Script Concordance Testing (SCT) scores among participants indicated the effectiveness of the online training module. Feedback suggested a broad acceptance of this novel training approach. However, discrepancies observed between formative SCT scores, and summative Multiple-Choice Questions (MCQ) assessments highlighted areas for potential refinement.

CONCLUSIONS

Our findings showcase the resilience and adaptability of medical education amidst challenges like the global pandemic. The success of methodologies such as SCT, endorsed by prestigious bodies like the European Resuscitation Council and the American Heart Association, suggests their potential in preparing health care professionals for emergent situations. This research offers valuable insights for shaping future online CME strategies.

Impact of an online learning by concordance program on reflection

BACKGROUND

Learning by concordance (LbC) is a recent approach that introduces learners to the complexity and uncertainty of clinical practice. Some data on LbC suggest that it stimulates reflection in future clinicians. We developed an online LbC training program on electrocardiogram (ECG) interpretation in general practice at the University of Strasbourg, France, and conducted an exploratory qualitative study to document the impact of this ECG learning-by-concordance training program on reflection in participants.

METHODS

We created 18 clinical vignettes on ECG interpretation based on a review of the literature on frequent and serious cardiovascular diseases that can be identified using an ECG in general practice. The training program was delivered online to postgraduate general practice students in two faculties of medicine. We conducted a qualitative study based on thematic analysis of two focus groups and six individual interviews. Inductive and deductive coding were performed. The five major components of reflection in the Nguyen model were used in the deductive coding: (i) thoughts and actions, (ii) attentive, critical, exploratory, and iterative processes (ACEI), (iii) underlying conceptual frame, (iv) change and (v) self.

RESULTS

Two focus groups and six individual interviews were conducted. The qualitative analysis indicated 203 codes in the focus groups and 206 codes in the individual interviews, which were divided into five groups based on the components of reflection in the Nguyen model: (i) the self; (ii) attentive, critical, exploratory, and iterative interactions with (iii) one's thoughts and actions; and (iv) a view on both the change itself and (v) the underlying conceptual frame. Inductive coding revealed interesting insights into the impact of the identity of the panel members, the absence of a scoring system and the question of uncertainty in ECG reading.

CONCLUSIONS

This study supports the claim that the use of LbC in the context of ECG interpretation could foster reflection in future general practitioners. We discuss future research avenues on instructional design of LbC and reflection.

Scoring Methods in Script Concordance Tests: An Exploratory Psychometric Study

BACKGROUND

Despite the increasingly popular role of script concordance test (SCT) scoring methods in the evaluation of clinical reasoning, studies examining these methods in nursing are relatively scarce. This study explored the psychometric properties of five SCT scoring methods.

METHOD

An SCT was administered to 12 experts and 43 learners. Scores were calculated using five methods and descriptive statistics. Differences in scores were assessed with the Mann-Whitney U test, and Spearman correlation coefficients were calculated for the different methods.

RESULTS

The median scores of both experts and learners differed substantially according to the scoring method used. Learners' scores were statistically different from experts' scores (p < .01) for each method. Spearman coefficients (range, 0.44 to 0.95) were positive for the different methods.

CONCLUSION

Further research is needed to refine the influence of SCT scoring methods for use in certifying assessment of clinical reasoning in nursing. [J Nurs Educ. 2023;62(10):549-555.].

Script concordance test to assess diagnostic and management reasoning in acute medicine

Clinical reasoning, an essential skill for patient care, can be difficult to assess. We created and validated a script concordance test (SCT) to assess clinical reasoning in acute medicine. This tool was used to provide feedback and targeted remediation for Postgraduate-Year-1 (PGY1) doctors, guide teaching and learning, and facilitate programme evaluation.

Use of situational judgment tests for assessing non-cognitive attributes of final year dental students

OBJECTIVE

Situational judgment tests (SJT) have been previously used in the admission process of medical and dental undergraduate programs, but their utilization as an assessment tool is fairly new. The objective of this study was to explore essential non-cognitive attributes of dental graduates. We also aimed to assess the attributes of final year dental students utilizing SJT and later collected participants' reactions on the test.

METHODS

A mixed-method exploratory sequential study was conducted. One-on-one and group discussions were done with subject matter experts (SMEs) to explore the essential attributes of fresh dental graduates. Based on the thematic analysis, 10 SJT items per domain were constructed by one group of SMEs and expert validated by a different group of SMEs and vice versa. The final dental-SJT was piloted online on final year dental students and their perceptions were collected using a five-point Likert scale questionnaire. Descriptive statistics and Cronbach's alpha were calculated for the test.

RESULTS

The consultation with SMEs generated three main themes, namely, Conventional skills, Tribal skills, and Occupational skills. The 70-item dental-SJT piloted on 150 final year dental students revealed data were normally distributed. The internal reliability of test was good (0.738). Students scored highest in "management skills" (>70%) and lowest in "professional ethics" (56%) and "diverse experiences" (58%). The overall student perception about the test was good.

CONCLUSION

SJT is an effective tool to assess non-cognitive attributes of dental students. It has an overall positive perception about perceived predictive validity, perceived fairness, face and content validity.

Clinical reasoning in undergraduate paramedicine: utilisation of a script concordance test

INTRODUCTION

Clinical reasoning is a complex cognitive and metacognitive process paramount to patient care in paramedic practice. While universally recognised as an essential component of practice, clinical reasoning has been historically difficult to assess in health care professions. Is the Script Concordance Test (SCT) an achievable and reliable option to test clinical reasoning in undergraduate paramedic students?

METHODS

This was a single institution observational cohort study designed to use the SCT to measure clinical reasoning in paramedic students. Clinical vignettes were constructed across a range of concepts with varying shades of clinical ambiguity. A reference panel mean scores of the test were compared to that of students. Test responses were graded with the aggregate scoring method with scores awarded for both partially and fully correct responses.

RESULTS

Eighty-three student paramedic participants (mean age: 21.8 (3.5) years, 54 (65%) female, 27 (33%) male and 2 (2%) non-binary) completed the SCT. The difference between the reference group mean score of 80 (5) and student mean of score of 65.6 (8.4) was statistically significant (p < 0.001).

DISCUSSION

Clinical reasoning skills are not easily acquired as they are a culmination of education, experience and the ability to apply this in the context to a specific patient. The SCT has shown to be reliable and effective in measuring clinical reasoning in undergraduate paramedics as it has in other health professions such as nursing and medicine. More investigation is required to establish effective pedogeological techniques to optimise clinical reasoning in student and novice paramedics who are devoid of experience.

Modeling Diagnostic Expertise in Cases of Irreducible Uncertainty: The Decision-Aligned Response Model

PURPOSE

Assessing expertise using psychometric models usually yields a measure of ability that is difficult to generalize to the complexity of diagnoses in clinical practice. However, using an item response modeling framework, it is possible to create a decision-aligned response model that captures a clinician's decision-making behavior on a continuous scale that fully represents competing diagnostic possibilities. In this proof-of-concept study, the authors demonstrate the necessary statistical conceptualization of this model using a specific electrocardiogram (ECG) example.

METHOD

The authors collected a range of ECGs with elevated ST segments due to either ST-elevation myocardial infarction (STEMI) or pericarditis. Based on pilot data, 20 ECGs were chosen to represent a continuum from "definitely STEMI" to "definitely pericarditis," including intermediate cases in which the diagnosis was intentionally unclear. Emergency medicine and cardiology physicians rated these ECGs on a 5-point scale ("definitely STEMI" to "definitely pericarditis"). The authors analyzed these ratings using a graded response model showing the degree to which each participant could separate the ECGs along the diagnostic continuum. The authors compared these metrics with the discharge diagnoses noted on chart review.

RESULTS

Thirty-seven participants rated the ECGs. As desired, the ECGs represented a range of phenotypes, including cases where participants were uncertain in their diagnosis. The response model showed that participants varied both in their propensity to diagnose one condition over another and in where they placed the thresholds between the 5 diagnostic categories. The most capable participants were able to meaningfully use all categories, with precise thresholds between categories.

CONCLUSIONS

The authors present a decision-aligned response model that demonstrates the confusability of a particular ECG and the skill with which a clinician can distinguish 2 diagnoses along a continuum of confusability. These results have broad implications for testing and for learning to manage uncertainty in diagnosis.

Development of a script concordance test to assess clinical reasoning in a pharmacy curriculum

INTRODUCTION

Clinical reasoning is a vital skill for student pharmacists in the provision of patient-centered care, but these skills are often difficult to assess in the didactic curriculum. A script concordance test (SCT) is an innovative assessment method that can be used to assess clinical reasoning skills. The objective of this study was to develop and refine an SCT to assess clinical reasoning skills of third year student pharmacists (P3s).

METHODS

An SCT was written and administered to P3s. Pharmacy practice faculty members served as the expert group. The SCT was scored and Rasch analysis was performed.

RESULTS

The SCT included 20 case vignettes and 60 questions. Test reliability was 0.34 with mean square values for all items between 0.7 and 1.3. Forty-two questions had a difficulty score between 0 and - 1 logits, indicating there were multiple questions with similar difficulty levels. Two case vignettes and 43.3% of questions (n = 26) were revised to enhance clarity and decrease ambiguity.

CONCLUSIONS

The SCT is a tool to assess clinical reasoning in the didactic curriculum. Faculty can create the SCT and use statistical methods such as Rasch analysis to assess validity and reliability of the SCT.

Online student culture as site for negotiating assessment in medical education

Classic studies of medical education have examined how professional socialization reproduces the prevailing professional culture, as well as how students actively negotiate their place in educational processes. However, sociological research has not re-examined student culture in light of structural transformations in medical education, such as the introduction of new assessment types and their use as modes of commensuration. In this paper, we examine data from two studies of online forums where medical trainees and applicants to medical school discuss their experiences preparing for tests of professional skills, including judgment, empathy, and communication. Examining how medical students talk about these tests on such forums allows us to understand the meaning-making processes at work as students negotiate the commensuration processes such tests enable. We examine how these negotiations take place in online forums, where participants confront common challenges, form common perspectives, and share common solutions, all hallmarks of student culture. Through qualitative analysis, we find that online communities are spaces where students grapple with these new forms of commensuration, interrogate the standards and quantifications that underlie them, and collectively negotiate how to approach these assessments. Using the case of online forum communities, our findings advance past work on student culture in medical sociology by theorizing student culture as an extra-organizational phenomenon that spans multiple career stages. In so doing, we highlight the importance of online forum data for studying social processes.

Does Self-Efficacy Affect Clinical Reasoning in Dental Students?

Introduction

Individual ability, motivation, age, and gender are all specific learner-related factors that can affect the academic performance of undergraduate dental students. Clinical reasoning and self-efficacy may potentially play crucial roles in this. This research aims to study the effects of clinical reasoning and self-efficacy on academic performance and to determine the relationship between them.

Materials and methods

This is a cross-sectional quantitative study that was conducted in 2021, and the study participants included 81 (86.19% response rate) final-year dental students who responded to an online questionnaire containing a specially designed clinical reasoning test and a self-efficacy scale.

Results

Although the levels of clinical reasoning skills and self-efficacy were not directly related, they were positively associated with students’ academic achievements. Furthermore, these 2 factors were considered to be predictors of a student's academic performance.

Conclusions

High levels of self-efficacy and clinical reasoning skills were associated with high academic achievement in students. However, having high levels of self-efficacy does not necessarily indicate mastery of clinical reasoning skills. This conclusion reflects the complexity of the clinical reasoning process during which an individual is faced with uncertainty. High levels of confidence might make students rush to a conclusion without considering all the conflicting possibilities or alternatives. In all instances, dental educators should follow recommended measures to improve clinical reasoning and self-efficacy abilities due to their importance in improving learning in students.

Script concordance test acceptability and utility for assessing medical students' clinical reasoning: a user's survey and an institutional prospective evaluation of students' scores

Script Concordance Testing (SCT) is a method for clinical reasoning assessment in the field of health-care training. Our aim was to assess SCT acceptability and utility with a survey and an institutional prospective evaluation of students' scores.With a user's online survey, we collected the opinions and satisfaction data of all graduate students and teachers involved in the SCT setting. We performed a prospective analysis comparing the scores obtained with SCT to those obtained with the national standard evaluation modality.General opinions about SCT were mostly negative. Students tended to express more negative opinions and perceptions. There was a lower proportion of negative responses in the teachers' satisfaction survey. The proportion of neutral responses was higher for teachers. There was a higher proportion of positive positions towards all questions among teachers. PCC scores significantly increased each year, but SCT scores increased only between the first and second tests. PCC scores were found significantly higher than SCT scores for the second and third tests. Medical students' and teachers' global opinion on SCT was negative. At the beginning SCT scores were found quite similar to PCC scores. There was a higher progression for PCC scores through time.

The Student-Generated Reasoning Tool (SGRT): Linking medical knowledge and clinical reasoning in preclinical education

INTRODUCTION

The simultaneous integration of knowledge acquisition and development of clinical reasoning in preclinical medical education remains a challenge. To help address this challenge, the authors developed and implemented the Student-Generated Reasoning Tool (SGRT)-a tool asking students to propose and justify pathophysiological hypotheses, generate findings, and critically appraise information.

METHODS

In 2019, students in a first-year preclinical course (n = 171; SGRT group) were assigned to one of 20 teams. Students used the SGRT individually, then in teams, and faculty provided feedback. The control group (n = 168) consisted of students from 2018 who did not use SGRT. Outcomes included academic performance, effectiveness of collaborative environments using the SGRT, and student feedback.

RESULTS

Students were five times more likely to get questions correct if they were in the SGRT group versus control group. Accuracy of pathophysiological hypotheses was significantly lower for individuals than teams. Qualitative analysis indicated students benefited from generating their own data, justifying their reasoning, and working individually as well as in teams.

CONCLUSIONS

This study introduces the SGRT as a potentially engaging, case-based, and collaborative learning method that may help preclinical medical students become aware of their knowledge gaps and integrate their knowledge in basic and clinical sciences in the context of clinical reasoning.

An Ontology-Driven Learning Assessment Using the Script Concordance Test

Assessing the level of domain-specific reasoning acquired by students is one of the major challenges in education particularly in medical education. Considering the importance of clinical reasoning in preclinical and clinical practice, it is necessary to evaluate students’ learning achievements accordingly. The traditional way of assessing clinical reasoning includes long-case exams, oral exams, and objective structured clinical examinations. However, the traditional assessment techniques are not enough to answer emerging requirements in the new reality due to limited scalability and difficulty for adoption in online education. In recent decades, the script concordance test (SCT) has emerged as a promising tool for assessment, particularly in medical education. The question is whether the usability of SCT could be raised to a level high enough to match the current education requirements by exploiting opportunities that new technologies provide, particularly semantic knowledge graphs (SCGs) and ontologies. In this paper, an ontology-driven learning assessment is proposed using a novel automated SCT generation platform. SCTonto ontology is adopted for knowledge representation in SCT question generation with the focus on using electronic health records data for medical education. Direct and indirect strategies for generating Likert-type scores of SCT are described in detail as well. The proposed automatic question generation was evaluated against the traditional manually created SCT, and the results showed that the time required for tests creation significantly reduced, which confirms significant scalability improvements with respect to traditional approaches.

The Promise of the New Educational Strategy for Curriculum Development (SPICES) Model on the Development of Students' Clinical Reasoning Ability. A Comparative Cross-Sectional Study

INTRODUCTION

Clinical reasoning skills are a core competency that must be taught at all levels of health-care education. In the last decade, several health professional education curricula in Ethiopia have been redesigned with the goal of improving student competence in key health-care delivery skills. Despite the fact that some academic programs followed the conventional educational strategy, a significant number of academic programs adopted a new educational strategy for curriculum development: Student-centered, Problem-based, Integrated, Community-based, Elective, and Systematic (SPICES) model. More empirical evidence, however, is required to determine whether the new curricular approach is effective in improving students' clinical reasoning. The purpose of this study is to determine whether the new educational strategy for curriculum development improves the clinical reasoning ability of midwifery students when compared to a peer institution that follows a traditional curriculum.

METHODS

A comparative cross-sectional study was conducted to compare the clinical reasoning skills of midwifery students who completed the new curricular approach versus students who completed a traditional curriculum. A Script Concordance Test (SCT) was used to collect data. The mean SCT score and an independent two-sample t-test were calculated to see if the two groups differed significantly in terms of clinical reasoning skills in managing Post-Partum hemorrhage (PPH).

RESULTS

A total of 77 final-year midwifery students participated (38 from the new and 39 from the traditional curriculum approach). Midwifery students who completed the new and conventional curriculum approaches had mean clinical reasoning SCT scores of 0.7 (SD = 0.35) and 0.53 (SD = 0.37), respectively. There was a statistically significant difference in the overall mean SCT score between the two study groups in terms of clinical reasoning skills (p = 0.008).

CONCLUSION

Our study found that the new SPICES model curricular approach is promising in fostering the development of clinical reasoning skills of Midwifery students in managing PPH.

Creating Clinical Reasoning Assessment Tools in Different Languages: Adaptation of the Pediatric Emergency Medicine Script Concordance Test to Japanese

Introduction: Clinical reasoning is a crucial skill in the practice of pediatric emergency medicine and a vital element of the various competencies achieved during the clinical training of resident doctors. Pediatric emergency physicians are often required to stabilize patients and make correct diagnoses with limited clinical information, time and resources. The Pediatric Emergency Medicine Script Concordance Test (PEM-SCT) has been developed specifically for assessing physician's reasoning skills in the context of the uncertainties in pediatric emergency practice. In this study, we developed the Japanese version of the PEM-SCT (Jpem-SCT) and confirmed its validity by collecting relevant evidence.

Methods: The Jpem-SCT was developed by translating the PEM-SCT into Japanese using the Translation, Review, Adjudication, Pretest, Documentation team translation model, which follows cross-cultural survey guidelines for proper translation and cross-cultural and linguistic equivalences between the English and Japanese version of the survey. First, 15 experienced pediatricians participated in the pre-test session, serving as a reference panel for modifying the test descriptions, incorporating Japanese context, and establishing the basis for the scoring process. Then, a 1-h test containing 60 questions was administered to 75 trainees from three academic institutions. Following data collection, we calculated the item-total correlations of the scores to optimize selection of the best items in the final version of the Jpem-SCT. The reliability of the finalized Jpem-SCT was calculated using Cronbach's α coefficient for ensuring generalizability of the evidence. We also conducted multiple regression analysis of the test score to collect evidence on validity of the extrapolation.

Results: The final version of the test, based on item-total correlation data analysis, contained 45 questions. The participant's specialties were as follows: Transitional interns 12.0%, pediatric residents 56.0%, emergency medicine residents 25.3%, and PEM fellows 6.7%. The mean score of the final version of the Jpem-SCT was 68.6 (SD 9.8). The reliability of the optimized test (Cronbach's α) was 0.70. Multiple regression analysis showed that being a transitional intern was a negative predictor of test scores, indicating that clinical experience relates to performance on the Jpem-SCT.

Conclusion: This pediatric emergency medicine Script Concordance Test was reliable and valid for assessing the development of clinical reasoning by trainee doctors during residency training.

Educators and practitioners' perspectives in the development of a learning by concordance tool for medical clerkship in the context of the COVID pandemic

BACKGROUND

The COVID-19 pandemic has forced medical schools to create educational material to palliate the anticipated and observed decrease in clinical experiences during clerkships. An online learning by concordance (LbC) tool was developed to overcome the limitation of students' experiences with clinical cases. However, knowledge about the instructional design of an LbC tool is scarce, especially the perspectives of collaborators involved in its design: 1- educators who wrote the vignettes' questions and 2- practitioners who constitute the reference panel by answering the LbC questions. The aim of this study was to describe the key elements that supported the pedagogical design of an LbC tool from the perspectives of educators and practitioners.

METHODS

A descriptive qualitative research design has been used. Online questionnaires were used, and descriptive analysis was conducted.

RESULTS

Six educators and 19 practitioners participated in the study. Important to the educators in designing the LbC tool were prevalent or high-stake situations, theoretical knowledge, professional situations experienced and perceived difficulties among students, and that the previous workshop promoted peer discussion and helped solidify the writing process. Important for practitioners was standards of practice and consensus among experts. However, they were uncertain of the educational value of their feedback, considering the ambiguity of the situations included in the LbC tool.

CONCLUSIONS

The LbC tool is a relatively new training tool in medical education. Further research is needed to refine our understanding of the design of such a tool and ensure its content validity to meet the pedagogical objectives of the clerkship.

Differential effects of team-based learning on clinical reasoning

BACKGROUND

Clinical reasoning (CR) is the ability to integrate information, knowledge and contextual factors for patient care. Few studies have explored effects of team-based learning (TBL) on neurological CR. This study compared simplified TBL (sTBL) against interactive lectures (IL) for teaching CR in neuroanatomical localisation (NL) and neurological emergencies (NE), assessed using a validated Script Concordance Test (SCT).

METHODS

A crossover study was conducted with third- and fifth-year undergraduates, randomly assigned to two groups, from the Yong Loo Lin School of Medicine in Singapore. Group 1 was taught NE with sTBL and NL with IL, whereas Group 2 was taught NL with sTBL and NE with IL. Teaching was conducted sequentially over 3 h followed immediately by the SCT. The primary outcome was the difference in mean SCT scores of NE and NL taught with sTBL versus IL.

FINDINGS

A total of 179 students (Group 1, n = 81; Group 2, n = 98) participated. Mean NL SCT scores for students taught with sTBL were significantly higher compared with IL (64.8% vs. 61.7%, mean difference 3.1%, 95% confidence interval [CI] 0.6%-5.5%, p = 0.013); effect size was 0.38. Mean NE SCT scores were similar between students taught with sTBL or IL (66.6% vs. 67.0%, mean difference -0.4%, 95% CI -2.2% to 3.1%, p = 0.75).

CONCLUSIONS

sTBL was superior to IL for teaching NL, whereas both methods were comparable for teaching NE. TBL may be suitable for teaching more complex neurological topics involving diagnostic reasoning through development of problem representation, hypothesis generation and illness script selection.

A guide for assessing the clinical relevance of findings in small bowel capsule endoscopy: analysis of 8064 answers of international experts to an illustrated script questionnaire

BACKGROUND AND AIM

Although recommended, the P-score used for assessing the pertinence / relevance of findings seen in small bowel (SB) capsule endoscopy (CE) is based on a low level of knowledge. The aim of this study was to evaluate the clinical relevance of the most frequent SBCE findings through an illustrated script questionnaire.

MATERIALS AND METHODS

Sixteen types of SBCE findings were illustrated four times each in three different settings (occult and overt obscure gastrointestinal bleeding and suspected Crohn's disease), and with a variable number (n = 1/n = 2-5/n ≥ 6), thus providing a questionnaire with 192 scenarios and 576 illustrated questions. Fifteen international experts were asked to rate the finding's relevance for each question as very unlikely (-2) / unlikely (-1) / doubtful (0) / likely (+1) / very likely (+2). The median score (≤-0.75, between -0.75 and 0.75, or ≥0.75) obtained for each scenario determined a low (P0), intermediate (P1) or high (P2) relevance, respectively.

RESULTS

8064 answers were analyzed. Participation and completion rates were 93% and 100%, respectively. In overt or occult OGIB, resultant P2 findings were 'typical angiectasia', 'deep ulceration', 'stenosis', and'blood', whatever their numbers, and 'superficial ulcerations' when multiple. While in suspected CD, consensus P2 lesions were 'deep ulceration' and 'stenosis' whatever their numbers, and 'aphthoid erosions' and 'superficial ulcerations' when multiple.

CONCLUSION

This study establishes a guide for the evaluation of relevance of SBCE findings. It represents a step forward for SB-CE interpretation and is intended to be used as a tool for teaching and academic research.

Script concordance tests: A call for action in dental education

The Script Concordance Test (SCT) is an educational tool that aims to assess the ability to interpret medical information under conditions of uncertainty. It is widely used and validated in health education, but almost unknown in dentistry. Based on authentic clinical problem-solving situations, it allows to assess clinical reasoning that experienced health workers develop over the years. A specific scoring system, dedicated to SCT, considers the variability of responses of practitioners in the same clinical situations. Finally, the scores generated by SCT reflect the respondents' ability to interpret clinical data compared to experienced clinicians. This article aims to familiarise the dental educators' community with SCT construction, optimisation and its possible applications.

Script Concordance Approach in Nursing Education

BACKGROUND

The script concordance approach aims at triggering judgments in simulated contexts of uncertainty.

PROBLEM

Nursing students need to be prepared to manage the uncertainty of clinical practice.

APPROACH

The purpose of this article is to describe the theoretical foundation and the pedagogical use of the script concordance approach, as well as to present the current state of nursing evidence on the subject. The script concordance approach includes (1) script concordance testing, which is a quantitative examination that evaluates clinical reasoning; (2) a face-to-face script concordance activity; and (3) a digital educational strategy based on script concordance delivered via an online teaching/learning platform that aims to support clinical reasoning development.

CONCLUSIONS

Relying on questioning and experts' modeling, the script concordance offers an innovative pedagogical approach that approximates the uncertainty of clinical practice.

Use of a script concordance test to evaluate the impact of a targeted educational strategy on clinical reasoning in advanced pharmacy practice experiential students

BACKGROUND AND PURPOSE

It is unclear how clinical reasoning is impacted by a single advanced pharmacy practice experience (APPE) and how preceptors can further develop these skills.

EDUCATIONAL ACTIVITY AND SETTING

Students completing an APPE within four sites were invited to participate. To assess clinical reasoning skills, students completed a 30 item script concordance test (SCT) during week 1 and week 5 of a rotation. Students were divided into control and intervention groups. The intervention group participated in a clinical reasoning discussion, during which students presented a case and led a discussion on how to reason through treatment options.

FINDINGS

Change in mean SCT scores between week 1 and week 5 were 0.84 (2.8%) and 1.23 (4.1%) in the control (n = 15) and intervention groups (n = 28), respectively. There was no significant change in scores in the control group (P = .07, CI -0.34, 2.01). The change in scores was statistically significant in the intervention group (P = .02, CI 0.23, 2.23). An independent samples t-test comparing the SCT score change for the control and intervention group showed no significant difference (P = .62, CI -1.18, 1.96).

SUMMARY

This study demonstrated the feasibility of implementing a SCT in experiential education. SCT scores did not significantly improve beyond the standard APPE in response to the focused educational intervention, but investigators found that the discussion facilitated rich conversations about patient cases and was valuable for assessing a student's thinking pattern.

Learning by concordance (LbC) to develop professional reasoning skills: AMEE Guide No. 141

Developing effective clinical reasoning is central to health professions education. Learning by concordance (LbC) is an on-line educational strategy that makes learners practice reasoning competency in case-based clinical situations. The questions asked are similar to those professionals ask themselves in their practice and participant answers are compared to those of a reference panel. When participants answer the questions, they receive an automated feedback that is two-fold as they see (1) how the panelists respond and (2) justifications each panelist gives for their answer. This provides rich contextual knowledge about the situation, supplemented by a synthesis summarizing crucial points. As many educators in the health sciences are engaging in introducing innovative approaches, many consider building LbC learning modules. Elaborating, designing and implementing a LbC tool remain a challenge. This AMEE Guide describes the steps and elements to be considered when designing a LbC tool, drawing on examples from distinct health professions: medicine, nursing, physiotherapy, and dentistry. Specifically, the following elements will be discussed: (1) LbC theoretical underpinnings; (2) principles of LbC questioning; (3) goals of the concordance-based activity; (4) nature of reasoning tasks; (5) content/levels of complexity; (6) reference panel; (7) feedback/synthesis messages; (8) on-line learning platforms.

Assessing the acceptability of script concordance testing: a nationwide study in otolaryngology

Background

Script concordance testing (SCT) is an objective method to evaluate clinical reasoning that assesses the ability to interpret medical information under conditions of uncertainty. Many studies have supported its validity as a tool to assess higher levels of learning, but little is known about its acceptability to major stakeholders. The aim of this study was to determine the acceptability of SCT to residents in otolaryngology – head and neck surgery (OTL-HNS) and a reference group of experts.

Methods

In 2013 and 2016, a set of SCT questions, as well a post-test exit survey, were included in the National In-Training Examination (NITE) for OTL-HNS. This examination is administered to all OTL-HNS residents across Canada who are in the second to fifth year of residency. The same SCT questions and survey were then sent to a group of OTL-HNS surgeons from 4 Canadian universities.

Results

For 64.4% of faculty and residents, the study was their first exposure to SCT. Overall, residents found it difficult to adapt to this form of testing, thought that the clinical scenarios were not clear and believed that SCT was not useful for assessing clinical reasoning. In contrast, the vast majority of experts felt that the test questions reflected real-life clinical situations and would recommend SCT as an evaluation method in OTL-HNS.

Conclusion

Views about the acceptability of SCT as an assessment tool for clinical reasoning differed between OTL-HNS residents and experts. Education about SCT and increased exposure to this testing method are necessary to improve residents’ perceptions of SCT.

Pediatric Hospitalists' Performance and Perceptions of Script Concordance Testing for Self-Assessment

OBJECTIVES

The cognitive expertise of Pediatric Hospitalists (PH) lies not in standard knowledge but in making decisions under conditions of uncertainty. To maintain expertise, PH should engage in deliberate practice via self-assessments that promote higher-level cognitive processes necessary to address problems with missing or ambiguous information. Higher levels of cognition are purported with Script Concordance Test (SCT) questions compared to Multiple Choice Questions (MCQ). To determine if PH use higher levels of cognition when answering SCT versus MCQ questions and to analyze participants' perceptions of the utility of using SCT self-assessment for deliberate practice in addressing clinical problems encountered in daily practice.

METHODS

This is a mixed methods study comparing the cognitive level expressed according to Bloom's Taxonomy by PH answering MCQ versus SCT questions using a "think aloud" (TA) exercise, followed by qualitative analysis of interviews conducted afterward.

RESULTS

A significantly greater percentage of comments were coded as higher cognitive processes (apply, analyze, evaluate, and create) for SCT versus MCQ (74% vs 19%) compared with lower order (remember, understand); chi-square P < .00001. Analysis of interviews revealed 6 themes.

CONCLUSION

SCT questions elicited higher level cognition essential to clinical reasoning compared to MCQ questions. PH-indicated MCQ questions measure standard knowledge, while SCT questions better measure decision-making under conditions of uncertainty. PH-perceived SCT could be useful for deliberate practice in Pediatric Hospital Medicine decision-making if they could compare their rationale in answering questions with that of experts.

Teaching clinical reasoning to undergraduate medical students by illness script method: a randomized controlled trial

BACKGROUND

The illness script method employs a theoretical outline (e.g., epidemiology, pathophysiology, signs and symptoms, diagnostic tests, interventions) to clarify how clinicians organized medical knowledge for clinical reasoning in the diagnosis domain. We hypothesized that an educational intervention based on the illness script method would improve medical students' clinical reasoning skills in the diagnosis domain.

METHODS

This study is a randomized controlled trial involving 100 fourth-year medical students in Shiraz Medical School, Iran. Fifty students were randomized to the intervention group, who were taught clinical reasoning skills based on the illness script method for three diseases during one clinical scenario. Another 50 students were randomized to the control group, who were taught the clinical presentation based on signs and symptoms of the same three diseases as the intervention group. The outcomes of interest were learner satisfaction with the intervention and posttest scores on both an internally developed knowledge test and a Script Concordance Test (SCT).

RESULTS

Of the hundred participating fourth-year medical students, 47 (47%) were male, and 53 (53%) were female. On the knowledge test, there was no difference in pretest scores between the intervention and control group, which suggested a similar baseline knowledge in both groups; however, posttest scores in the intervention group were (15.74 ± 2.47 out of 20) statistically significantly higher than the control group (14.38 ± 2.59 out of 20, P = 0.009). On the SCT, the mean score for the intervention group (6.12 ± 1.95 out of 10) was significantly higher than the control group (4.54 ± 1.56 out of 10; P = 0.0001). Learner satisfaction data indicated that the intervention was well-received by students.

CONCLUSION

Teaching with the illness script method was an effective way to improve students' clinical reasoning skills in the diagnosis domain suggested by posttest and SCT scores for specific clinical scenarios. Whether this approach translates to improved generalized clinical reasoning skills in real clinical settings merits further study.

Resident competencies before and after short intensive care unit rotations: a multicentre pilot observational study

PURPOSE

Residency programs need to understand the competencies developed by residents during an intensive care unit (ICU) rotation, so that curricula and assessments maximize residents' learning. The primary study objective was to evaluate the feasibility for training programs and acceptability by residents of conducting a multi-competency assessment during a four-week ICU rotation.

METHODS

We conducted a prospective, multicentre observational pilot study in three ICUs. During weeks 1 and 4 of an ICU rotation, we conducted repeated standardized assessments of non-critical care specialty residents' competencies in cognitive reasoning (script concordance test [SCT]), procedural skills (objective structured assessment of technical skills [OSATS]-global rating scale], and communication skills through a written test, two procedural simulations, and a simulated encounter with a "family member". The feasibility outcomes included program costs, the proportion of enrolled residents able to complete at least one three-station assessment during their four-week ICU rotation, and acceptability of the assessment for the trainees.

RESULTS

We enrolled 63 (69%) of 91 eligible residents, with 58 (92%) completing at least one assessment. The total cost to conduct 90 assessments was CAD 33,800. The majority of participants agreed that the assessment was fair and that it measured important clinical abilities. For the 32 residents who completed two assessments, the mean (standard deviation) cognitive reasoning and procedural skill scores increased between weeks 1 and 4 [SCT difference, 3.1 (6.5), P = 0.01; OSATS difference for bag-mask ventilation and central line insertion, 0.4 (0.5) and 0.6 (0.8), respectively; both P ≤ 0.001]. Nevertheless, the communication scores did not change significantly.

CONCLUSIONS

A monthly multi-competency assessment for specialty residents rotating in the ICU is likely feasible for most programs with appropriate resources, and generally acceptable for residents. Specialty residents' cognitive reasoning and procedural skills may improve during a four-week ICU rotation, whereas communication skills may not.

Using the diagnostic thinking inventory in musculoskeletal physiotherapy: a validity and reliability study

BACKGROUND

Development of clinical reasoning is an essential aspect in musculoskeletal physiotherapy practice that is linked to better outcomes. The measurement of clinical reasoning has placed an emphasis on diagnostic reasoning using different types of examinations. The Diagnostic Thinking Inventory (DTI) is a self-assessment tool developed to measure two aspects of diagnostic reasoning: flexibility in thinking (FT) and structure in memory (SM). DTI is valid and reliable that has been used extensively in medical field.

OBJECTIVE

To investigate the validity and reliability of DTI in musculoskeletal physiotherapy practice.

METHODS

Two groups of musculoskeletal physiotherapists completed DTI. Expert musculoskeletal physiotherapists assessed face and content validity. Data from the second group of musculoskeletal physiotherapists were used to assess test-retest reliability. Internal consistency was calculated using Cronbach's alpha. Construct validity was assessed by comparing both groups. Data were analyzed using the IBM SPSS statistics 25.0 version.

RESULTS

The experts agreed that DTI measures diagnostic reasoning. For test-retest reliability, average intraclass correlation coefficient was 0.91, 0.92 and 0.90 (p < 0.001) for DTI, FT and SM scores, respectively. Cronbach's alpha was 0.909, 0.919 and 0.897 (p < 0.001) for DTI, FT and SM, respectively. The independent samples t-test demonstrated that the experts group achieved higher and statistically significant score (p < 0.001).

CONCLUSION

DTI is valid and reliable in measuring diagnostic reasoning in the context of musculoskeletal physiotherapy practice. It can be used to assess the impact of continuing education on musculoskeletal physiotherapists' diagnostic reasoning.

Does burnout affect clinical reasoning? An observational study among residents in general practice

BACKGROUND

Burnout results from excessive demands at work. Caregivers suffering from burnout show a state of emotional exhaustion, leading them to distance themselves from their patients and to become less efficient in their work. While some studies have shown a negative impact of burnout on physicians' clinical reasoning, others have failed to demonstrate any such impacts. To better understand the link between clinical reasoning and burnout, we carried out a study looking for an association between burnout and clinical reasoning in a population of general practice residents.

METHODS

We conducted a cross-sectional observational study among residents in general practice in 2017 and 2019. Clinical reasoning performance was assessed using a script concordance test (SCT). The Maslach Burnout Inventory for Human Services Survey (MBI-HSS) was used to determine burnout status in both original standards of Maslach's burnout inventory manual (conventional approach) and when individuals reported high emotional exhaustion in combination with high depersonalization or low personal accomplishment compared to a norm group ("emotional exhaustion +1" approach).

RESULTS

One hundred ninety-nine residents were included. The participants' mean SCT score was 76.44% (95% CI: 75.77-77.10). In the conventional approach, 126 residents (63.31%) had no burnout, 37 (18.59%) had mild burnout, 23 (11.56%) had moderate burnout, and 13 (6.53%) had severe burnout. In the "exhaustion + 1" approach, 38 residents had a burnout status (19.10%). We found no significant correlation between burnout status and SCT scores either for conventional or "exhaustion + 1" approaches.

CONCLUSIONS

Our data seem to indicate that burnout status has no significant impact on clinical reasoning. However, one speculation is that SCT mostly examines the clinical reasoning process's analytical dimension, whereas emotions are conventionally associated with the intuitive dimension. We think future research might aim to explore the impact of burnout on intuitive clinical reasoning processes.

Are script concordance tests suitable for the assessment of undergraduate students? A multicenter comparative study

INTRODUCTION

Script concordance tests (SCTs) have been developed to assess clinical reasoning in uncertain situations. Their reliability for the evaluation of undergraduate medical students has not been evaluated.

METHODS

Twenty internal medicine SCT cases were implemented in undergraduate students of two programs. The results obtained on the SCTs were compared to those obtained by the same students on clinical-based classical multiple-choice questions (MCQs).

RESULTS

A total of 551/883 students (62%) answered the SCTs. The mean aggregate score (based on a total 20 points) was 11.54 (3.29). The success rate and mean score for each question did not differ depending on the modal response but the discrimination rate did. The results obtained by the students on the SCT test correlated with their scores on the MCQ tests. Among students, 446/517 (86%) considered the SCTs to be more difficult than classical MCQs, although the mean score did not differ between the SCT and MCQ tests.

CONCLUSION

The use of SCTs is a feasible option for the evaluation of undergraduate students. The SCT scores correlated with those obtained on classical MCQ tests.

L’apprentissage du raisonnement clinique infirmier dans le cadre d’un dispositif éducatif numérique basé sur la concordance de scripts

Contexte : Un des défis actuels des responsables de la formation des futurs infirmiers est de concevoir des stratégies éducatives pour favoriser le développement optimal du raisonnement clinique infirmier (RCI).But : L’étude visait à développer, mettre à l’essai et évaluer un dispositif éducatif numérique basé sur la concordance de scripts auprès d’étudiants en sciences infirmières. Méthodes : Un devis de recherche-développement a permis de soutenir les étapes de réalisation du dispositif éducatif. Une validation de contenu a été intégrée au devis à l’aide d’une technique Delphi modifiée. Les choix de réponses des étudiants et des experts aux questions du dispositif ont été compilés. Un questionnaire en ligne et des entrevues de groupe focalisées ont été utilisés auprès d’étudiants afin d’évaluer l’acceptabilité et l’utilisabilité du dispositif éducatif. Résultats : Cinq experts-collaborateurs ont validé le contenu du dispositif éducatif. La version finale comptait 81 questions liées à 22 vignettes. Douze experts-panélistes et quarante-cinq étudiants ont répondu aux questions du dispositif éducatif. Les indices alpha (α) de Cronbach ont oscillé entre 0,87 et 0,90. Globalement, les étudiants ont apprécié l’utilisation du dispositif éducatif, dont la rétroaction variée et formative d’experts à des situations apparentées à la vie professionnelle. Conclusion : Le dispositif éducatif repose sur l’utilisation d’un test de concordance de scripts numérisé auquel des rétroactions d’experts sont intégrées à des fins d’apprentissage du RCI. Essentiel, l’engagement des experts-panélistes inscrit la conception d’un tel dispositif éducatif dans une démarche de co-construction et de validation du matériel pédagogique.

Training and standardization of simulated patients for multicentre studies in clinical pharmacy education

Objective:

To evaluate the training and standardization methods of multiple simulated patients (SPs) performing a single scenario in a multicenter study.

Methods:

A prospective quasi-experimental study, using a multicenter approach, evaluated the performance of five different individuals with the same biotype during a simulation session in a high-fidelity environment. The SPs training and standardization process consisted of four steps and six web or face-to-face mediated: Step 1: simulation scenario design and pilot test. Step 2: SPs selection, recruitment and beginning training (Session 1: performance instructions and memorization request.) Session 2: check the SPs’ performances and adjustments). Step 3 and session 3: training role-play and performance’s evaluation. Step 4: SPs’ standardization and performances’ evaluation (Sessions 4 and 5: first and second rounds of SPs’ standardization assessment. Session 6: Global training and standardization evaluation. SPs performance consistency was estimated using Cronbach’s alpha and ICC.

Results:

In the evaluation of training results, the Maastricht Simulated Patient Assessment dimensions of SPs performances “It seems authentic”, “Can be a real patient” and “Answered questions naturally”, presented “moderate or complete agreement” of all evaluators. The dimensions “Seems to retain information unnecessarily”, “Remains in his/her role all the time”, “Challenges/tests the student”, and “Simulates physical complaints in an unrealistic way” presented “moderate or complete disagreement” in all evaluations. The SPs “Appearance fits the role” showed “moderate or complete agreement” in most evaluations. In the second round of evaluations, the SPs had better performance than the first ones. This could indicate the training process’s had good influence on SPs performances. The Cronbach’s alpha in the second assessment was better than the first (varied from 0.699 to 0.978). The same improvement occurred in the second round of intraclass correlation coefficient that was between 0.424 and 0.978. The SPs were satisfied with the training method and standardization process. They could perceive improvement on their role-play authenticity.

Conclusions:

The SPs training and standardization process revealed good SPs reliability and simulation reproducibility, demonstrating to be a feasible method for SPs standardization in multicenter studies. The Maastricht Simulated Patient Assessment was regarded as missing the assessment of the information consistency between the simulation script and the SPs provision.

Impact of panelists' experience on script concordance test scores of medical students

BACKGROUND

The evaluation process of French medical students will evolve in the next few years in order to improve assessment validity. Script concordance testing (SCT) offers the possibility to assess medical knowledge alongside clinical reasoning under conditions of uncertainty. In this study, we aimed at comparing the SCT scores of a large cohort of undergraduate medical students, according to the experience level of the reference panel.

METHODS

In 2019, the authors developed a 30-item SCT and sent it to experts with varying levels of experience. Data analysis included score comparisons with paired Wilcoxon rank sum tests and concordance analysis with Bland & Altman plots.

RESULTS

A panel of 75 experts was divided into three groups: 31 residents, 21 non-experienced physicians (NEP) and 23 experienced physicians (EP). Among each group, random samples of N = 20, 15 and 10 were selected. A total of 985 students from nine different medical schools participated in the SCT examination. No matter the size of the panel (N = 20, 15 or 10), students' SCT scores were lower with the NEP group when compared to the resident panel (median score 67.1 vs 69.1, p < 0.0001 if N = 20; 67.2 vs 70.1, p < 0.0001 if N = 15 and 67.7 vs 68.4, p < 0.0001 if N = 10) and with EP compared to NEP (65.4 vs 67.1, p < 0.0001 if N = 20; 66.0 vs 67.2, p < 0.0001 if N = 15 and 62.5 vs 67.7, p < 0.0001 if N = 10). Bland & Altman plots showed good concordances between students' SCT scores, whatever the experience level of the expert panel.

CONCLUSIONS

Even though student SCT scores differed statistically according to the expert panels, these differences were rather weak. These results open the possibility of including less-experienced experts in panels for the evaluation of medical students.

Assessment of clinical reasoning: three evolutions of thought

Although assessing clinical reasoning is almost universally considered central to medical education it is not a straightforward issue. In the past decades, our insights into clinical reasoning as a phenomenon, and consequently the best ways to assess it, have undergone significant changes. In this article, we describe how the interplay between fundamental research, practical applications, and evaluative research has pushed the evolution of our thinking and our practices in assessing clinical reasoning.

Empirical comparison of three assessment instruments of clinical reasoning capability in 230 medical students

BACKGROUND

Several instruments intend to measure clinical reasoning capability, yet we lack evidence contextualizing their scores. The authors compared three clinical reasoning instruments [Clinical Reasoning Task (CRT), Patient Note Scoring rubric (PNS), and Summary Statement Assessment Rubric (SSAR)] using Messick's convergent validity framework in pre-clinical medical students. Scores were compared to a validated clinical reasoning instrument, Clinical Data Interpretation (CDI).

METHOD

Authors administered CDI and the first clinical case to 235 students. Sixteen randomly selected students (four from each CDI quartile) wrote a note on a second clinical case. Each note was scored with CRT, PNS, and SSAR. Final scores were compared to CDI.

RESULTS

CDI scores did not significantly correlate with any other instrument. A large, significant correlation between PNS and CRT was seen (r = 0.71; p = 0.002).

CONCLUSIONS

None of the tested instruments outperformed the others when using CDI as a standard measure of clinical reasoning. Differing strengths of association between clinical reasoning instruments suggest they each measure different components of the clinical reasoning construct. The large correlation between CRT and PNS scoring suggests areas of novice clinical reasoning capability, which may not be yet captured in CDI or SSAR, which are weighted toward knowledge synthesis and hypothesis testing.

Medical student education in transfusion medicine, part II: Moving forward to building up a "Know How" education program in transfusion medicine for under-graduate medical students

There is a general trend in changing paradigm in teaching medicine; the emerging concept relies on a competence-based approach. Transfusion is either a discipline or a subsidiary of others depending on the countries and systems; this variability can be explained because transfusion is a medical care that is transdisciplinary. As a collective of professionals in both transfusion medicine practice and education, authors aim to propose a revision of the way education in transfusion medicine is delivered in this era of the 'global competency approach'. They advocate in favor of a Know How on 5 key issues: Diagnosing the patient condition in line with the Patient Blood Management principles; Facing acute blood loss; Addressing compatibility and avoiding immunization; Seeking for maximized benefits and dampening complications; and Inlaying competence within global health care issues, also comprising od economy. The methods used would be those developed for medical education at large, such as assessment tools. The global objective is to deliver the necessary competence to manage patients by an intern/resident. At the end of the curriculum, students should be able to self-evaluate the following items: 1) Do I know why my patient is anemic, thrombocytopenic, bleeding….? 2) Do I know the best approach to treat anemia, thrombocytopenia, bleeding (including the "no treatment" option)? 3) Do I know whether a transfusion approach is appropriate for my patients? 4) Do I know how to evaluate and anticipate benefits from blood transfusion and to avoid side-effects in the patient? 5) Do I know how to avoid unnecessary use of the products?

Examining response process validity of script concordance testing: a think-aloud approach

OBJECTIVES

This study investigated whether medical student responses to Script Concordance Testing (SCT) items represent valid clinical reasoning. Using a think-aloud approach students provided written explanations of the reasoning that underpinned their responses, and these were reviewed for concordance with an expert reference panel.

METHODS

A set of 12, 11 and 15 SCT items were administered online to Year 3 (2018), Year 4 (2018) and Year 3 (2019) medical students respectively. Students' free-text descriptions of the reasoning supporting each item response were analysed, and compared with those of the expert panel. Response process validity was quantified as the rate of true positives (percentage of full and partial credit responses derived through correct clinical reasoning); and true negatives (percentage of responses with no credit derived through faulty clinical reasoning).

RESULTS

Two hundred and nine students completed the online tests (response rate = 68.3%). The majority of students who had chosen the response which attracted full or partial credit also provided justifications which were concordant with the experts (true positive rate of 99.6% for full credit; 99.4% for partial credit responses). Most responses that attracted no credit were based on faulty clinical reasoning (true negative of 99.0%).

CONCLUSIONS

The findings provide support for the response process validity of SCT scores in the setting of undergraduate medicine. The additional written think-aloud component, to assess clinical reasoning, provided useful information to inform student learning. However, SCT scores should be validated on each testing occasion, and in other contexts.