A Natural Language Processing-Based Virtual Patient Simulator and Intelligent Tutoring System for the Clinical Diagnostic Process: Simulator Development and Case Study

A scoping review of artificial intelligence in medical education: BEME Guide No. 84

BACKGROUND

Artificial Intelligence (AI) is rapidly transforming healthcare, and there is a critical need for a nuanced understanding of how AI is reshaping teaching, learning, and educational practice in medical education. This review aimed to map the literature regarding AI applications in medical education, core areas of findings, potential candidates for formal systematic review and gaps for future research.

METHODS

This rapid scoping review, conducted over 16 weeks, employed Arksey and O'Malley's framework and adhered to STORIES and BEME guidelines. A systematic and comprehensive search across PubMed/MEDLINE, EMBASE, and MedEdPublish was conducted without date or language restrictions. Publications included in the review spanned undergraduate, graduate, and continuing medical education, encompassing both original studies and perspective pieces. Data were charted by multiple author pairs and synthesized into various thematic maps and charts, ensuring a broad and detailed representation of the current landscape.

RESULTS

The review synthesized 278 publications, with a majority (68%) from North American and European regions. The studies covered diverse AI applications in medical education, such as AI for admissions, teaching, assessment, and clinical reasoning. The review highlighted AI's varied roles, from augmenting traditional educational methods to introducing innovative practices, and underscores the urgent need for ethical guidelines in AI's application in medical education.

CONCLUSION

The current literature has been charted. The findings underscore the need for ongoing research to explore uncharted areas and address potential risks associated with AI use in medical education. This work serves as a foundational resource for educators, policymakers, and researchers in navigating AI's evolving role in medical education. A framework to support future high utility reporting is proposed, the FACETS framework.

Development and evaluation of a software system for medical students to teach and practice anamnestic interviews with virtual patient avatars

BACKGROUND AND OBJECTIVES

Taking a medical history is a core competence of the diagnostic process. At the beginning of their study medical students need to learn and practice the necessary techniques, initially focusing on good structuring and completeness. For this purpose, an interactive software system (ViPATalk) was developed in which the student can train to pose questions to virtual patient avatars in free conversation. At the end, the student receives feedback on the completeness of the questioning and an explanation of the essential items. The use of this software was compared to the traditional format of student role play in a randomized trial.

METHODS

The central component of ViPATalk is a chatbot based on the AI language AIML, which generates an appropriate answer based on keywords in the student's question. To enable a realistic use, the student can enter the question via microphone (speech-to-text) and the answer generated by the chatbot is presented as a short video sequence, where the avatar is generated from a real image. Here, the transition between the sequences is seamless, resulting in a continuous movement of the avatar during the conversation.

RESULTS

The learning success by practicing with ViPATalk was tested in an anamnestic interview with actors as simulated patients. The completeness of the conversation was evaluated with regard to numerous aspects and also certain behaviors during the conversation. These results were compared with those after practicing using peer role play.

CONCLUSIONS

It was found that practicing with ViPATalk was mostly equivalent to the students' role play. In the subsequent survey of the students, the wish was expressed that the ViPATalk software should also be used as an online tool for self-study and that there should be more cases for practicing.

Strategies for Digital Clinical Teaching During the COVID Pandemic: A Scoping Review

Widespread "lockdowns" during the COVID pandemic in 2020-2021 restricted medical students' access to patients. We used a scoping review with exploratory thematic synthesis to examine how reports of digital clinical teaching during the first year of the COVID pandemic could inform digital clinical teaching in the post-pandemic world. We looked at strategies used and outcomes reported, lessons learned about how best to use digital methods for clinical teaching, and learning theories used. The eighty-three articles included in the final review fell into four groups. These were telehealth interventions; virtual case-based teaching; multi-modal virtual rotations; and a small group of "other" strategies. Telehealth reports indicated that COVID has probably accelerated the adoption of telehealth, and these skills will be required in future curricula. Engagement with virtual case-based teaching was problematic. Virtual rotations were particularly valued in specialties that relied on visual interpretation such as radiology and dermatology. For general clinical specialties, digital clinical teaching was not a satisfactory substitute for real clinical exposure because it lacked the complexity of usual clinical practice. Sixty-seven articles reported students' reactions only, and 16 articles reported a change in knowledge or skills. Demands on instructors were considerable. Few studies were theorized and none tested theory, which limited their transferability. While telehealth teaching may be a valuable addition to some curricula, digital clinical teaching is unlikely substantially to replace exposure to real patients outside of specialties that rely on visual interpretation. High demands on instructors suggest little potential for new, scalable digital clinical offerings after COVID.

Natural language processing for automatic evaluation of free-text answers - a feasibility study based on the European Diploma in Radiology examination

BACKGROUND

Written medical examinations consist of multiple-choice questions and/or free-text answers. The latter require manual evaluation and rating, which is time-consuming and potentially error-prone. We tested whether natural language processing (NLP) can be used to automatically analyze free-text answers to support the review process.

METHODS

The European Board of Radiology of the European Society of Radiology provided representative datasets comprising sample questions, answer keys, participant answers, and reviewer markings from European Diploma in Radiology examinations. Three free-text questions with the highest number of corresponding answers were selected: Questions 1 and 2 were "unstructured" and required a typical free-text answer whereas question 3 was "structured" and offered a selection of predefined wordings/phrases for participants to use in their free-text answer. The NLP engine was designed using word lists, rule-based synonyms, and decision tree learning based on the answer keys and its performance tested against the gold standard of reviewer markings.

RESULTS

After implementing the NLP approach in Python, F1 scores were calculated as a measure of NLP performance: 0.26 (unstructured question 1, n = 96), 0.33 (unstructured question 2, n = 327), and 0.5 (more structured question, n = 111). The respective precision/recall values were 0.26/0.27, 0.4/0.32, and 0.62/0.55.

CONCLUSION

This study showed the successful design of an NLP-based approach for automatic evaluation of free-text answers in the EDiR examination. Thus, as a future field of application, NLP could work as a decision-support system for reviewers and support the design of examinations being adjusted to the requirements of an automated, NLP-based review process.

CLINICAL RELEVANCE STATEMENT

Natural language processing can be successfully used to automatically evaluate free-text answers, performing better with more structured question-answer formats. Furthermore, this study provides a baseline for further work applying, e.g., more elaborated NLP approaches/large language models.

KEY POINTS

• Free-text answers require manual evaluation, which is time-consuming and potentially error-prone. • We developed a simple NLP-based approach - requiring only minimal effort/modeling - to automatically analyze and mark free-text answers. • Our NLP engine has the potential to support the manual evaluation process. • NLP performance is better on a more structured question-answer format.

Artificial Intelligence Supporting the Training of Communication Skills in the Education of Health Care Professions: Scoping Review

BACKGROUND

Communication is a crucial element of every health care profession, rendering communication skills training in all health care professions as being of great importance. Technological advances such as artificial intelligence (AI) and particularly machine learning (ML) may support this cause: it may provide students with an opportunity for easily accessible and readily available communication training.

OBJECTIVE

This scoping review aimed to summarize the status quo regarding the use of AI or ML in the acquisition of communication skills in academic health care professions.

METHODS

We conducted a comprehensive literature search across the PubMed, Scopus, Cochrane Library, Web of Science Core Collection, and CINAHL databases to identify articles that covered the use of AI or ML in communication skills training of undergraduate students pursuing health care profession education. Using an inductive approach, the included studies were organized into distinct categories. The specific characteristics of the studies, methods and techniques used by AI or ML applications, and main outcomes of the studies were evaluated. Furthermore, supporting and hindering factors in the use of AI and ML for communication skills training of health care professionals were outlined.

RESULTS

The titles and abstracts of 385 studies were identified, of which 29 (7.5%) underwent full-text review. Of the 29 studies, based on the inclusion and exclusion criteria, 12 (3.1%) were included. The studies were organized into 3 distinct categories: studies using AI and ML for text analysis and information extraction, studies using AI and ML and virtual reality, and studies using AI and ML and the simulation of virtual patients, each within the academic training of the communication skills of health care professionals. Within these thematic domains, AI was also used for the provision of feedback. The motivation of the involved agents played a major role in the implementation process. Reported barriers to the use of AI and ML in communication skills training revolved around the lack of authenticity and limited natural flow of language exhibited by the AI- and ML-based virtual patient systems. Furthermore, the use of educational AI- and ML-based systems in communication skills training for health care professionals is currently limited to only a few cases, topics, and clinical domains.

CONCLUSIONS

The use of AI and ML in communication skills training for health care professionals is clearly a growing and promising field with a potential to render training more cost-effective and less time-consuming. Furthermore, it may serve learners as an individualized and readily available exercise method. However, in most cases, the outlined applications and technical solutions are limited in terms of access, possible scenarios, the natural flow of a conversation, and authenticity. These issues still stand in the way of any widespread implementation ambitions.

IvCDS: An End-to-End Driver Simulator for Personal In-Vehicle Conversational Assistant

An advanced driver simulator methodology facilitates a well-connected interaction between the environment and drivers. Multiple traffic information environment language processing aims to help drivers accommodate travel demand: safety prewarning, destination navigation, hotel/restaurant reservation, and so on. Task-oriented dialogue systems generally aim to assist human users in achieving these specific goals by a conversation in the form of natural language. The development of current neural network based dialogue systems relies on relevant datasets, such as KVRET. These datasets are generally used for training and evaluating a dialogue agent (e.g., an in-vehicle assistant). Therefore, a simulator for the human user side is necessarily required for assessing an agent system if no real person is involved. We propose a new end-to-end simulator to operate as a human driver that is capable of understanding and responding to assistant utterances. This proposed driver simulator enables one to interact with an in-vehicle assistant like a real person, and the diversity of conversations can be simply controlled by changing the assigned driver profile. Results of our experiment demonstrate that this proposed simulator achieves the best performance on all tasks compared with other models.

Artificial intelligence in virtual standardized patients: Combining natural language understanding and rule based dialogue management to improve conversational fidelity

INTRODUCTION

Advances in natural language understanding have facilitated the development of Virtual Standardized Patients (VSPs) that may soon rival human patients in conversational ability. We describe herein the development of an artificial intelligence (AI) system for VSPs enabling students to practice their history taking skills.

METHODS

Our system consists of (1) Automated Speech Recognition (ASR), (2) hybrid AI for question identification, (3) classifier to choose between the two systems, and (4) automated speech generation. We analyzed the accuracy of the ASR, the two AI systems, the classifier, and student feedback with 620 first year medical students from 2018 to 2021.

RESULTS

System accuracy improved from ∼75% in 2018 to ∼90% in 2021 as refinements in algorithms and additional training data were utilized. Student feedback was positive, and most students felt that practicing with the VSPs was a worthwhile experience.

CONCLUSION

We have developed a novel hybrid dialogue system that enables artificially intelligent VSPs to correctly answer student questions at levels comparable with human SPs. This system allows trainees to practice and refine their history-taking skills before interacting with human patients.

Adaptations of Clinical Teaching During the COVID-19 Pandemic: Perspectives of Medical Students and Faculty Members

BACKGROUND

The COVID-19 pandemic had serious implications on medical schools' programs that necessitated lots of adaptations of teaching, learning, and assessment to guarantee continuity of education in medical schools. Our study aimed to evaluate perspectives of clerkship students and faculty members regarding clinical teaching adaptations implemented during the COVID-19 pandemic.

METHODS

A descriptive, cross-sectional, survey-based study was conducted and targeted 5^th and 6^th year clerkship students and full- and part-time clinical faculty. The survey explored (1) perception of the degree of contribution of implemented adaptations to student achievement of expected clinical competencies, (2) degree of confidence regarding students' achievement of expected clinical competencies through such adaptations, and (3) perception of the effect of implemented educational adaptations on students' learning. Descriptive statistics were used, and statistical significance level was set at p < 0.05.

RESULTS

The survey exhibited high internal consistency. Both students and faculty members felt that most of the adaptations had moderate to high contribution to student achievement of expected clinical competencies. On a 5-point scale, the highest score was given by faculty members to "Interpretation of investigations" (3.93±0.84) while the lowest scores were given by faculty members (3.10±1.21) and students (2.57±1.36) to "Performing clinical procedures". Students and faculty members agreed that the adaptations had positive effect on students' learning except for the statement "Students were able to easily monitor their academic progress" where students gave less scores than faculty members, with a statistically significance difference (p=0.029).

CONCLUSION

Students and faculty members had similar perspectives regarding the implemented adaptations and their impact and contribution to student learning and achievement of the basic clinical competencies. Both of them agreed on the need for and importance of the implemented adaptations. Our findings recommend such adaptations during the times of crises, which can be conducted through integrating online teaching with face-to-face teaching.

Learning Analytics Applied to Clinical Diagnostic Reasoning Using a Natural Language Processing-Based Virtual Patient Simulator: Case Study

BACKGROUND

Virtual patient simulators (VPSs) log all users' actions, thereby enabling the creation of a multidimensional representation of students' medical knowledge. This representation can be used to create metrics providing teachers with valuable learning information.

OBJECTIVE

The aim of this study is to describe the metrics we developed to analyze the clinical diagnostic reasoning of medical students, provide examples of their application, and preliminarily validate these metrics on a class of undergraduate medical students. The metrics are computed from the data obtained through a novel VPS embedding natural language processing techniques.

METHODS

A total of 2 clinical case simulations (tests) were created to test our metrics. During each simulation, the students' step-by-step actions were logged into the program database for offline analysis. The students' performance was divided into seven dimensions: the identification of relevant information in the given clinical scenario, history taking, physical examination, medical test ordering, diagnostic hypothesis setting, binary analysis fulfillment, and final diagnosis setting. Sensitivity (percentage of relevant information found) and precision (percentage of correct actions performed) metrics were computed for each issue and combined into a harmonic mean (F₁), thereby obtaining a single score evaluating the students' performance. The 7 metrics were further grouped to reflect the students' capability to collect and to analyze information to obtain an overall performance score. A methodological score was computed based on the discordance between the diagnostic pathway followed by students and the reference one previously defined by the teacher. In total, 25 students attending the fifth year of the School of Medicine at Humanitas University underwent test 1, which simulated a patient with dyspnea. Test 2 dealt with abdominal pain and was attended by 36 students on a different day. For validation, we assessed the Spearman rank correlation between the performance on these scores and the score obtained by each student in the hematology curricular examination.

RESULTS

The mean overall scores were consistent between test 1 (mean 0.59, SD 0.05) and test 2 (mean 0.54, SD 0.12). For each student, the overall performance was achieved through a different contribution in collecting and analyzing information. Methodological scores highlighted discordances between the reference diagnostic pattern previously set by the teacher and the one pursued by the student. No significant correlation was found between the VPS scores and hematology examination scores.

CONCLUSIONS

Different components of the students' diagnostic process may be disentangled and quantified by appropriate metrics applied to students' actions recorded while addressing a virtual case. Such an approach may help teachers provide students with individualized feedback aimed at filling competence drawbacks and methodological inconsistencies. There was no correlation between the hematology curricular examination score and any of the proposed scores as these scores address different aspects of students' medical knowledge.

A Framework of AI-Based Approaches to Improving eHealth Literacy and Combating Infodemic

The global COVID-19 pandemic has put everyone in an urgent need of accessing and comprehending health information online. Meanwhile, there has been vast amount of information/misinformation/disinformation generated over the Internet, particularly social media platforms, resulting in an infodemic. This public health crisis of COVID-19 pandemic has put each individual and the entire society in a test: what is the level of eHealth literacy is needed to seek accurate health information from online resources and to combat infodemic during a pandemic? This article aims to summarize the significances and challenges of improving eHealth literacy in both communicable (e.g., COVID-19) and non-communicable diseases [e.g., cancer, Alzheimer's disease, and cardiovascular diseases (CVDs)]. Also, this article will make our recommendations of a general framework of AI-based approaches to improving eHealth literacy and combating infodemic, including AI-augmented lifelong learning, AI-assisted translation, simplification, and summarization, and AI-based content filtering. This general framework of AI-based approaches to improving eHealth literacy and combating infodemic has the general advantage of matching the right online health information to the right people.

A scoping review on adaptations of clinical education for medical students during COVID-19

Rapid advances in clinical education in response to the COVID-19 pandemic are taking place globally. This scoping review updated the educational strategies which could be applied by clinical educators in their practice to effectively maintain clinical attachment programs for medical students amidst public health crises. Almost all elements of clinical teaching were deliverable, whether it was online, onsite, virtual or blended, their educational effectiveness should be further examined. Increase in the number of telemedicine related publications were remarkable, and they could serve as a scalable model for future educational programs to be incorporated into the medical student curricula.