Assessing clinical judgment using the Script Concordance test: the importance of using specialty-specific experts to develop the scoring key

Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study

Background

Undergraduate medical studies represent a wide range of learning opportunities served in the form of various teaching-learning modalities for medical learners. A clinical scenario is frequently used as a modality, followed by multiple-choice and open-ended questions among other learning and teaching methods. As such, script concordance tests (SCTs) can be used to promote a higher level of clinical reasoning. Recent technological developments have made generative artificial intelligence (AI)-based systems such as ChatGPT (OpenAI) available to assist clinician-educators in creating instructional materials.

Objective

The main objective of this project is to explore how SCTs generated by ChatGPT compared to SCTs produced by clinical experts on 3 major elements: the scenario (stem), clinical questions, and expert opinion.

Methods

This mixed method study evaluated 3 ChatGPT-generated SCTs with 3 expert-created SCTs using a predefined framework. Clinician-educators as well as resident doctors in psychiatry involved in undergraduate medical education in Quebec, Canada, evaluated via a web-based survey the 6 SCTs on 3 criteria: the scenario, clinical questions, and expert opinion. They were also asked to describe the strengths and weaknesses of the SCTs.

Results

A total of 102 respondents assessed the SCTs. There were no significant distinctions between the 2 types of SCTs concerning the scenario (P=.84), clinical questions (P=.99), and expert opinion (P=.07), as interpretated by the respondents. Indeed, respondents struggled to differentiate between ChatGPT- and expert-generated SCTs. ChatGPT showcased promise in expediting SCT design, aligning well with Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria, albeit with a tendency toward caricatured scenarios and simplistic content.

Conclusions

This study is the first to concentrate on the design of SCTs supported by AI in a period where medicine is changing swiftly and where technologies generated from AI are expanding much faster. This study suggests that ChatGPT can be a valuable tool in creating educational materials, and further validation is essential to ensure educational efficacy and accuracy.

Script concordance testing in genetic counseling training: A pilot study

Clinical reasoning is a complex skill that represents a trainee's ability to use their professional knowledge and skills to assess and solve the problems that arise in clinical practice. As an integral tenet of the genetic counseling process, clinical reasoning skills underlie many of the Practice-Based Competencies (2019) across a variety of domains. Despite the long-lasting recognition of the importance of this complex skill in the training of genetic counselors, clinical reasoning has traditionally been difficult to assess in a standardized way in healthcare education. Script concordance testing is a standardized method of assessing clinical reasoning skills in ambiguous clinical situations. The tool has been used to successfully measure the clinical reasoning skills of trainees in various healthcare training programs and has never been used in a genetic counseling training program. We conducted a pilot study to assess the utility of script concordance testing in the field of genetic counseling as an objective measure of clinical reasoning in trainees. The script concordance test was constructed for the field of genetic counseling and administered to 22 second year genetic counseling students in the Joan H. Marks Graduate Program in Human Genetics at Sarah Lawrence College. Twelve genetic counselors served on a panel to provide expert clinical reasoning responses and a scoring grid was developed using the aggregate scores method. The utility of the tool was measured using Cronbach's alpha coefficient, and scores of students and the panel were compared using Hedge's g. Results revealed statistically significant differences between the scores of panelists and students and good reliability. This study shows that script concordance testing can be used to measure clinical reasoning skills in genetic counseling trainees in a way that is reliable, standardized, and easy to use, thereby allowing programs to better assess the clinical reasoning skills of trainees prior to graduation.

Educators and practitioners' perspectives in the development of a learning by concordance tool for medical clerkship in the context of the COVID pandemic

BACKGROUND

The COVID-19 pandemic has forced medical schools to create educational material to palliate the anticipated and observed decrease in clinical experiences during clerkships. An online learning by concordance (LbC) tool was developed to overcome the limitation of students' experiences with clinical cases. However, knowledge about the instructional design of an LbC tool is scarce, especially the perspectives of collaborators involved in its design: 1- educators who wrote the vignettes' questions and 2- practitioners who constitute the reference panel by answering the LbC questions. The aim of this study was to describe the key elements that supported the pedagogical design of an LbC tool from the perspectives of educators and practitioners.

METHODS

A descriptive qualitative research design has been used. Online questionnaires were used, and descriptive analysis was conducted.

RESULTS

Six educators and 19 practitioners participated in the study. Important to the educators in designing the LbC tool were prevalent or high-stake situations, theoretical knowledge, professional situations experienced and perceived difficulties among students, and that the previous workshop promoted peer discussion and helped solidify the writing process. Important for practitioners was standards of practice and consensus among experts. However, they were uncertain of the educational value of their feedback, considering the ambiguity of the situations included in the LbC tool.

CONCLUSIONS

The LbC tool is a relatively new training tool in medical education. Further research is needed to refine our understanding of the design of such a tool and ensure its content validity to meet the pedagogical objectives of the clerkship.

The Impact of Surgeon Experience on Script Concordance Test Scoring

OBJECTIVE

The Script Concordance Test (SCT) is a test of clinical decision-making that relies on an expert panel to create its scoring key. Existing literature demonstrates the value of specialty-specific experts, but the effect of experience among the expert panel is unknown. The purpose of this study was to explore the role of surgeon experience in SCT scoring.

DESIGN

An SCT was administered to 29 general surgery residents and 14 staff surgeons. Staff surgeons were stratified as either junior or senior experts based on years since completing residency training (<15 versus >25 years). The SCT was scored using the full expert panel, the senior panel, the junior panel, and a subgroup junior panel in practice <5 years. A one-way ANOVA was used to compare the scores of first (R1) and fifth (R5) year residents using each scoring scheme. Cognitive interviews were analyzed for differences between junior and senior expert panelist responses.

RESULTS

There was no statistically significant difference between the mean score of six R1s and five R5s using the full expert panel (R1 69.08 versus R5 67.06, F_1,9 = 0.10, P = 0.76), the junior panel (R1 66.73 versus R5 62.50, F_1,9 = 0.35, P = 0.57), or the subgroup panel in practice <5 years (R1 61.07 versus R5 58.79, F_1,9 = 0.18, P = 0.75). However, the average score of R1s was significantly lower than R5s when using the senior faculty panel (R1 52.04 versus R5 63.26, F_1,9 = 26.90, P = 0.001). Cognitive interview data suggests that some responses of junior experts demonstrate less confidence than those of senior experts.

CONCLUSIONS

SCT scores are significantly affected by the responses of the expert panel. Expert differences between first and fifth year residents were only demonstrated when using an expert panel consisting of senior faculty members. Confidence may play a role in the response selections of junior experts. When constructing an SCT expert panel, consideration must be given to the experience of panel members.

How do cognitive processes influence script concordance test responses?

INTRODUCTION

The script concordance test (SCT) is a test of clinical decision-making (CDM) that compares the thought process of learners to that of experts to determine to what extent their cognitive 'scripts' align. Without understanding test-takers' cognitive process, however, it is unclear what influences their responses. The objective of this study was to gather response process validity evidence by studying the cognitive process of test-takers to determine whether the SCT tests CDM and what cognitive processes may influence SCT responses.

METHODS

Cases from an SCT used in a national validation study were administered and semi-structured cognitive interviews were conducted with ten residents and five staff surgeons. A retrospective verbal probing technique was used. Data was independently analysed and coded by two analysts. Themes were identified as factors that influence SCT responses during the cognitive interview.

RESULTS

Cognitive interviews demonstrated variability in CDM among test-takers. Consistent with dual process theory, test-takers relied on scripts formed through past experiences, when available, to make decisions and used conscious deliberation in the absence of experience. However, test-takers' response process was also influenced by their comprehension of specific terms, desire for additional information, disagreement with the planned management, underlying knowledge gaps and desire to demonstrate confidence or humility.

CONCLUSION

The rationale behind SCT answers may be influenced by comprehension, underlying knowledge and social desirability in addition to formed scripts and/or conscious deliberation. Having test-takers verbalise their rationale for responses provides a depth of assessment that is otherwise lost in the SCT's current format. With the improved ability to standardise CDM assessment using the SCT, consideration of test-makers improving the SCT construction process and combining the SCT question format with verbal responses may improve the use of the SCT for CDM assessment.

Are script concordance tests suitable for the assessment of undergraduate students? A multicenter comparative study

INTRODUCTION

Script concordance tests (SCTs) have been developed to assess clinical reasoning in uncertain situations. Their reliability for the evaluation of undergraduate medical students has not been evaluated.

METHODS

Twenty internal medicine SCT cases were implemented in undergraduate students of two programs. The results obtained on the SCTs were compared to those obtained by the same students on clinical-based classical multiple-choice questions (MCQs).

RESULTS

A total of 551/883 students (62%) answered the SCTs. The mean aggregate score (based on a total 20 points) was 11.54 (3.29). The success rate and mean score for each question did not differ depending on the modal response but the discrimination rate did. The results obtained by the students on the SCT test correlated with their scores on the MCQ tests. Among students, 446/517 (86%) considered the SCTs to be more difficult than classical MCQs, although the mean score did not differ between the SCT and MCQ tests.

CONCLUSION

The use of SCTs is a feasible option for the evaluation of undergraduate students. The SCT scores correlated with those obtained on classical MCQ tests.

Evaluation of the first year of Dental Health Partnerships: a web-based distance learning partnership between UK dental educators and students from low-resource countries

Aim

To introduce and discuss the results of a newly-founded online, distance learning organisation, Dental Health Partnerships. The motivation behind the project is to bridge the gap between dental educators in the UK and dental students in low-resource communities. The results of the first partnership with Amoud Dental University, Somaliland, are presented.

Method

Sixteen tutorials were conducted between February 2016 and June 2016 for the final year dental students via an online virtual classroom (WizIQ). Sixteen students participated in the weekly tutorials, each of which lasted for approximately 90 minutes. The students were sent online questionnaires at two stages during the study to qualitatively evaluate their experience.

Results

Nine of the sixteen students responded to both questionnaires. One-hundred percent of the participants were 'very satisfied' or 'satisfied' with the programme at the end of the year. The attendance and participation rates for the weekly sessions were positive. The students had theoretical knowledge of the topics, however, the application of theoretical concepts to clinical scenarios was a challenge.

Conclusion

The results of the first year of the partnership indicates the relevance of case-based tutorials to dental practice in Somaliland, equipping students with the clinical-based knowledge to be effective in the field after graduation.

Nurses' Clinical Judgment Development: A Qualitative Research in Iran

Background:

Clinical judgment development is necessary because it leads to appropriate nursing diagnoses, clinical decision-making and health promotion.

Objectives:

In this study we explored the process of Iranian nurses’ development in clinical judgment.

Patients and Methods:

This qualitative study was conducted in 2013 at hospitals of Kurdistan University of Medical Sciences, located in the Sanandaj city of Iran. The data were collected based on semi-structured interviews and the study included 24 participants. Data analysis was carried out concurrently with data collection using the grounded theory method.

Results:

The study participants’ main concern was ‘being non-professional in clinical judgment’. In response to this concern, they were struggling for gaining professional autonomy, striving for integrating clinical judgment skills, scrambling to make effective educational interventions and striving for professional and inter professional collaboration in clinical judgment. The core category was ‘struggling for becoming professional in clinical judgment development’. When nurses were supported professionally, they were able to develop their professional clinical judgment.

Conclusions:

The findings of this study provided critical information about nurses’ professionalization in clinical judgment. Accordingly, the participants adopted different strategies to develop their clinical judgment ability. Integrating these strategies into nursing theory and clinical education can improve nurses’ clinical judgment ability.

Challenging script concordance test reference standard by evidence: do judgments by emergency medicine consultants agree with likelihood ratios?

Background

We aimed to compare the clinical judgments of a reference panel of emergency medicine academic physicians against evidence-based likelihood ratios (LRs) regarding the diagnostic value of selected clinical and paraclinical findings in the context of a script concordance test (SCT).

Findings

A SCT with six scenarios and five questions per scenario was developed. Subsequently, 15 emergency medicine attending physicians (reference panel) took the test and their judgments regarding the diagnostic value of those findings for given diseases were recorded. The LRs of the same findings for the same diseases were extracted from a series of published systematic reviews. Then, the reference panel judgments were compared to evidence-based LRs. To investigate the test-retest reliability, five participants took the test one month later, and the correlation of their first and second judgments were quantified using Spearman rank-order coefficient.

In 22 out of 30 (73.3%) findings, the expert judgments were significantly different from the LRs. The differences included overestimation (30%), underestimation (30%), and judging the diagnostic value in an opposite direction (13.3%). Moreover, the score of a hypothetical test-taker was calculated to be 21.73 out of 30 if his/her answers were based on evidence-based LRs.

The test showed an acceptable test-retest reliability coefficient (Spearman coefficient: 0.83).

Conclusions

Although SCT is an interesting test to evaluate clinical decision-making in emergency medicine, our results raise concerns regarding whether the judgments of an expert panel are sufficiently valid as the reference standard for this test.

A model of cardiopulmonary bypass staged training integrating technical and non-technical skills dedicated to cardiac trainees

OBJECTIVES

To develop a standardized simulation-based curriculum to teach medical knowledge and technical, communication and critical thinking skills necessary to initiate and wean from cardiopulmonary bypass (CPB) to junior cardiac trainees (CTs) in France. Performance on post-curricular tests was compared between CTs who participated in the new curriculum to those who did not.

METHODS

The simulation-based curriculum was developed by content and education experts. Simulations sequentially taught the skills necessary for initiating and weaning from CPB as well as managing crises by adding fidelity and complexity to scenarios. Nine CTs were randomly assigned to the new curriculum (n=5) or the traditional curriculum (n=4). Skills were assessed using tests of medical knowledge and technical, communication (GRS) and critical thinking (SCT) skills. A two-sample Wilcoxon rank-sum test compared average scores between the two groups. Alpha of 0.05 was set to indicate statistically significant differences.

RESULTS

The resutls revealed that CTs in the new curriculum significantly outperformed CTs in the traditional curriculum on technical (18.2 vs 14.8, p=0.05) and communication (3.5 vs 2.2, p=0.013) skills. There was no significant difference between CTs in the new curriculum in the Script Concordance Test (16.5 vs 14.8, p=0.141) and knowledge tests (26.9 vs 24.6, p=0.14) compared to CTs in the traditional curriculum.

CONCLUSION

Our new curriculum teaches communication and technical skills necessary for CPB. The results of this pilot study are encouraging and relevant. They give grounds for future research with a larger panel of trainees. Based on the current distribution of scores, a sample size of 12 CTs per group should yield significant results for all tests.

Evaluation of the first year of the Oxpal Medlink: A web-based partnership designed to address specific challenges facing medical education in the occupied Palestinian territories

Objectives

To (1) evaluate educational needs of clinical students at Al-Quds University Medical School in the West Bank; (2) address these needs where possible using synchronous distance learning, with clinicians in Oxford providing case-based tutorials to undergraduates in the West Bank via an online platform (WizIQ) and (3) assess the impact of this education.

Design

Review of online OxPal Medlink database for tutorials held between March 2012 and April 2013. Needs assessment and evaluation of student and tutor experiences through online questionnaires, focus groups and semi-structured interviews.

Setting

Oxford University Hospitals, Oxford, UK, and Al-Quds University Medical School, Abu Dies, Palestine.

Participants

Doctors at Oxford University Hospitals and fourth-, fifth- and sixth-year medical students and faculty members at Al-Quds Medical School.

Main outcome measures

Number of tutorials, student participation, student-rated satisfaction and qualitative feedback from tutors and students.

Results

Students demonstrated strong theoretical knowledge but struggled to apply this in presentation-based scenarios. Between March 2012 and April 2013, 90 tutorials were delivered to 60 students. Feedback: >95% respondents rated tutorials as ‘Excellent’ or ‘Good’ and ‘Very’ or ‘Fairly’ relevant to their future practice in Palestine. Students reported the programme had modified their approach to patients but requested better synchronization with concurrent attachments and clarification of learning outcomes.

Conclusions

OxPal Medlink is a novel, web-based distance-learning partnership designed to overcome some of the challenges to local medical education in the occupied Palestinian territories. Evaluation of the first year indicates teaching is relevant to local practice and of high quality. This approach may have the potential to strengthen local capacity for medical education.

Construction and utilization of a script concordance test as an assessment tool for DCEM3 (5th year) medical students in rheumatology

BACKGROUND

The script concordance test (SCT) is a method for assessing clinical reasoning of medical students by placing them in a context of uncertainty such as they will encounter in their future daily practice. Script concordance testing is going to be included as part of the computer-based national ranking examination (iNRE).This study was designed to create a script concordance test in rheumatology and use it for DCEM3 (fifth year) medical students administered via the online platform of the Clermont-Ferrand medical school.

METHODS

Our SCT for rheumatology teaching was constructed by a panel of 19 experts in rheumatology (6 hospital-based and 13 community-based). One hundred seventy-nine DCEM3 (fifth year) medical students were invited to take the test. Scores were computed using the scoring key available on the University of Montreal website. Reliability of the test was estimated by the Cronbach alpha coefficient for internal consistency.

RESULTS

The test comprised 60 questions. Among the 26 students who took the test (26/179: 14.5%), 15 completed it in its entirety. The reference panel of rheumatologists obtained a mean score of 76.6 and the 15 students had a mean score of 61.5 (p = 0.001). The Cronbach alpha value was 0.82.

CONCLUSIONS

An online SCT can be used as an assessment tool for medical students in rheumatology. This study also highlights the active participation of community-based rheumatologists, who accounted for the majority of the 19 experts in the reference panel.A script concordance test in rheumatology for 5th year medical students.