Virtual Antenatal Encounter and Standardized Simulation Assessment (VANESSA): Pilot Study

Increasing Realism and Variety of Virtual Patient Dialogues for Prenatal Counseling Education Through a Novel Application of ChatGPT: Exploratory Observational Study

BACKGROUND

Using virtual patients, facilitated by natural language processing, provides a valuable educational experience for learners. Generating a large, varied sample of realistic and appropriate responses for virtual patients is challenging. Artificial intelligence (AI) programs can be a viable source for these responses, but their utility for this purpose has not been explored.

OBJECTIVE

In this study, we explored the effectiveness of generative AI (ChatGPT) in developing realistic virtual standardized patient dialogues to teach prenatal counseling skills.

METHODS

ChatGPT was prompted to generate a list of common areas of concern and questions that families expecting preterm delivery at 24 weeks gestation might ask during prenatal counseling. ChatGPT was then prompted to generate 2 role-plays with dialogues between a parent expecting a potential preterm delivery at 24 weeks and their counseling physician using each of the example questions. The prompt was repeated for 2 unique role-plays: one parent was characterized as anxious and the other as having low trust in the medical system. Role-play scripts were exported verbatim and independently reviewed by 2 neonatologists with experience in prenatal counseling, using a scale of 1-5 on realism, appropriateness, and utility for virtual standardized patient responses.

RESULTS

ChatGPT generated 7 areas of concern, with 35 example questions used to generate role-plays. The 35 role-play transcripts generated 176 unique parent responses (median 5, IQR 4-6, per role-play) with 268 unique sentences. Expert review identified 117 (65%) of the 176 responses as indicating an emotion, either directly or indirectly. Approximately half (98/176, 56%) of the responses had 2 or more sentences, and half (88/176, 50%) included at least 1 question. More than half (104/176, 58%) of the responses from role-played parent characters described a feeling, such as being scared, worried, or concerned. The role-plays of parents with low trust in the medical system generated many unique sentences (n=50). Most of the sentences in the responses were found to be reasonably realistic (214/268, 80%), appropriate for variable prenatal counseling conversation paths (233/268, 87%), and usable without more than a minimal modification in a virtual patient program (169/268, 63%).

CONCLUSIONS

Generative AI programs, such as ChatGPT, may provide a viable source of training materials to expand virtual patient programs, with careful attention to the concerns and questions of patients and families. Given the potential for unrealistic or inappropriate statements and questions, an expert should review AI chat outputs before deploying them in an educational program.

Measuring antenatal counseling skill with a milestone-based assessment tool: a validation study

BACKGROUND

Antenatal counseling for parents in the setting of expected preterm delivery is an important component of pediatric training. However, healthcare professionals receive a variable amount and quality of formal training. This study evaluated and discussed validity of a practical tool to assess antenatal counseling skills and provide evaluative feedback: the Antenatal Counseling Milestones Scale (ACoMS).

METHODS

Experts in antenatal counseling developed an anchored milestone-based tool to evaluate observable skills. Study participants with a range of antenatal counseling skills were recruited to participate in simulation of counseling sessions in person or via video with standardized patient actors presenting with preterm labor at 23 weeks' gestation. Two faculty observers scored each session independently using the ACoMS. Participants completed an ACoMS self-assessment, demographic, and feedback survey. Validity was measured with weighted kappas for inter-rater agreement, Kruskal-Wallis and Dunn's tests for milestone levels between degrees of expertise in counseling, and cronbach's alpha for item consistency.

RESULTS

Forty-two participants completed observed counseling sessions. Of the 17 items included in the tool, 15 items were statistically significant with scores scaling with level of training. A majority of elements had fair-moderate agreement between raters, and there was high internal consistency amongst all items.

CONCLUSION

This study demonstrates that the internal structure of the ACoMS rubric has greater than fair inter-rater reliability and high internal consistency amongst items. Content validity is supported by the scale's ability to discern level of training. Application of the ACoMS to clinical encounters is needed to determine utility in clinical practice.

Artificial intelligence in virtual standardized patients: Combining natural language understanding and rule based dialogue management to improve conversational fidelity

INTRODUCTION

Advances in natural language understanding have facilitated the development of Virtual Standardized Patients (VSPs) that may soon rival human patients in conversational ability. We describe herein the development of an artificial intelligence (AI) system for VSPs enabling students to practice their history taking skills.

METHODS

Our system consists of (1) Automated Speech Recognition (ASR), (2) hybrid AI for question identification, (3) classifier to choose between the two systems, and (4) automated speech generation. We analyzed the accuracy of the ASR, the two AI systems, the classifier, and student feedback with 620 first year medical students from 2018 to 2021.

RESULTS

System accuracy improved from ∼75% in 2018 to ∼90% in 2021 as refinements in algorithms and additional training data were utilized. Student feedback was positive, and most students felt that practicing with the VSPs was a worthwhile experience.

CONCLUSION

We have developed a novel hybrid dialogue system that enables artificially intelligent VSPs to correctly answer student questions at levels comparable with human SPs. This system allows trainees to practice and refine their history-taking skills before interacting with human patients.

Teaching Antenatal Counseling Skills via Video Conference

Neonatologists provide counseling to expectant parents to prepare them for the birth and subsequent medical care that their extremely preterm, or otherwise medically complex newborn may require. The skills required to conduct these sensitive conversations are often taught to neonatology trainees via direct observation or simulated scenarios in advance of counseling actual patients. This technical report details how we taught antenatal counseling skills to junior neonatal-perinatal medicine (NPM) fellows via video conferencing during the coronavirus disease 2019 (COVID-19) pandemic. This approach could be used to effectively prepare future trainees to perform antenatal counseling.

Virtual Patients in Health Professions Education

Health care professionals must not only have knowledge, but also be able to organise, synthesise and apply this knowledge in such a way that it promotes the development of clinical reasoning. Panels of Virtual patients (VPs) are widely being used in health professions education to facilitate the development of clinical reasoning. VPs can also be used to teach wider educational outcomes such as communication skills, resource utilisation and longitudinal patient care. This chapter will define virtual patients and examine the evidence behind their use in health professions learning and teaching. The chapter will discuss virtual patient design, such as gamification. Finally, the chapter will discuss where this pedagogical innovation is best integrated into assessment and potential barriers to implementation into existing curricula.