Artificial scholarship: LLMs in health professions education research

Latest developments of generative artificial intelligence and applications in ophthalmology

The emergence of generative artificial intelligence (AI) has revolutionized various fields. In ophthalmology, generative AI has the potential to enhance efficiency, accuracy, personalization and innovation in clinical practice and medical research, through processing data, streamlining medical documentation, facilitating patient-doctor communication, aiding in clinical decision-making, and simulating clinical trials. This review focuses on the development and integration of generative AI models into clinical workflows and scientific research of ophthalmology. It outlines the need for development of a standard framework for comprehensive assessments, robust evidence, and exploration of the potential of multimodal capabilities and intelligent agents. Additionally, the review addresses the risks in AI model development and application in clinical service and research of ophthalmology, including data privacy, data bias, adaptation friction, over interdependence, and job replacement, based on which we summarized a risk management framework to mitigate these concerns. This review highlights the transformative potential of generative AI in enhancing patient care, improving operational efficiency in the clinical service and research in ophthalmology. It also advocates for a balanced approach to its adoption.

Large Language Models in Ophthalmology: Potential and Pitfalls

Large language models (LLMs) show great promise in assisting clinicians in general, and ophthalmology in particular, through knowledge synthesis, decision support, accelerating research, enhancing education, and improving patient interactions. Specifically, LLMs can rapidly summarize the latest literature to keep clinicians up-to-date. They can also analyze patient data to highlight crucial insights and recommend appropriate tests or referrals. LLMs can automate tedious research tasks like data cleaning and literature reviews. As AI tutors, LLMs can fill knowledge gaps and assess competency in trainees. As chatbots, they can provide empathetic, personalized responses to patient inquiries and improve satisfaction. The visual capabilities of LLMs like GPT-4 allow assisting the visually impaired by describing environments. However, there are significant ethical, technical, and legal challenges around the use of LLMs that should be addressed regarding privacy, fairness, robustness, attribution, and regulation. Ongoing oversight and refinement of models is critical to realize benefits while minimizing risks and upholding responsible AI principles. If carefully implemented, LLMs hold immense potential to push the boundaries of care, discovery, and quality of life for ophthalmology patients.

Detecting Artificial Intelligence-Generated Personal Statements in Professional Physical Therapist Education Program Applications: A Lexical Analysis

OBJECTIVE

The objective of this study was to compare the lexical sophistication of personal statements submitted by professional physical therapist education program applicants with those generated by OpenAI's Chat Generative Pretrained Transformer (ChatGPT).

METHODS

Personal statements from 152 applicants and 20 generated by ChatGPT were collected, all in response to a standardized prompt. These statements were coded numerically, then analyzed with recurrence quantification analyses (RQAs). RQA indices including recurrence, determinism, max line, mean line, and entropy were compared with t-tests. A receiver operating characteristic curve analysis was used to examine discriminative validity of RQA indices to distinguish between ChatGPT and human-generated personal statements.

RESULTS

ChatGPT-generated personal statements exhibited higher recurrence, determinism, mean line, and entropy values than did human-generated personal statements. The strongest discriminator was a 13.04% determinism rate, which differentiated ChatGPT from human-generated writing samples with 70% sensitivity and 91.4% specificity (positive likelihood ratio = 8.14). Personal statements with determinism rates exceeding 13% were 8 times more likely to have been ChatGPT than human generated.

CONCLUSION

Although RQA can distinguish artificial intelligence (AI)-generated text from human-generated text, it is not absolute. Thus, AI introduces additional challenges to the authenticity and utility of personal statements. Admissions committees along with organizations providing guidelines in professional physical therapist education program admissions should reevaluate the role of personal statements in applications.

IMPACT

As AI-driven chatbots like ChatGPT complicate the evaluation of personal statements, RQA emerges as a potential tool for admissions committees to detect AI-generated statements.

A scoping review of ChatGPT's role in healthcare education and research

OBJECTIVES

To examine and consolidate literature regarding the advantages and disadvantages of utilizing ChatGPT in healthcare education and research.

DESIGN/METHODS

We searched seven electronic databases (PubMed/Medline, CINAHL, Embase, PsycINFO, Scopus, ProQuest Dissertations and Theses Global, and Web of Science) from November 2022 until September 2023. This scoping review adhered to Arksey and O'Malley's framework and followed reporting guidelines outlined in the PRISMA-ScR checklist. For analysis, we employed Thomas and Harden's thematic synthesis framework.

RESULTS

A total of 100 studies were included. An overarching theme, "Forging the Future: Bridging Theory and Integration of ChatGPT" emerged, accompanied by two main themes (1) Enhancing Healthcare Education, Research, and Writing with ChatGPT, (2) Controversies and Concerns about ChatGPT in Healthcare Education Research and Writing, and seven subthemes.

CONCLUSIONS

Our review underscores the importance of acknowledging legitimate concerns related to the potential misuse of ChatGPT such as 'ChatGPT hallucinations', its limited understanding of specialized healthcare knowledge, its impact on teaching methods and assessments, confidentiality and security risks, and the controversial practice of crediting it as a co-author on scientific papers, among other considerations. Furthermore, our review also recognizes the urgency of establishing timely guidelines and regulations, along with the active engagement of relevant stakeholders, to ensure the responsible and safe implementation of ChatGPT's capabilities. We advocate for the use of cross-verification techniques to enhance the precision and reliability of generated content, the adaptation of higher education curricula to incorporate ChatGPT's potential, educators' need to familiarize themselves with the technology to improve their literacy and teaching approaches, and the development of innovative methods to detect ChatGPT usage. Furthermore, data protection measures should be prioritized when employing ChatGPT, and transparent reporting becomes crucial when integrating ChatGPT into academic writing.

A Systematic Review and Meta-Analysis of Artificial Intelligence Tools in Medicine and Healthcare: Applications, Considerations, Limitations, Motivation and Challenges

Artificial intelligence (AI) has emerged as a transformative force in various sectors, including medicine and healthcare. Large language models like ChatGPT showcase AI's potential by generating human-like text through prompts. ChatGPT's adaptability holds promise for reshaping medical practices, improving patient care, and enhancing interactions among healthcare professionals, patients, and data. In pandemic management, ChatGPT rapidly disseminates vital information. It serves as a virtual assistant in surgical consultations, aids dental practices, simplifies medical education, and aids in disease diagnosis. A total of 82 papers were categorised into eight major areas, which are G1: treatment and medicine, G2: buildings and equipment, G3: parts of the human body and areas of the disease, G4: patients, G5: citizens, G6: cellular imaging, radiology, pulse and medical images, G7: doctors and nurses, and G8: tools, devices and administration. Balancing AI's role with human judgment remains a challenge. A systematic literature review using the PRISMA approach explored AI's transformative potential in healthcare, highlighting ChatGPT's versatile applications, limitations, motivation, and challenges. In conclusion, ChatGPT's diverse medical applications demonstrate its potential for innovation, serving as a valuable resource for students, academics, and researchers in healthcare. Additionally, this study serves as a guide, assisting students, academics, and researchers in the field of medicine and healthcare alike.

Technology-Enhanced Learning in Medical Education Collection: Latest Developments

Technology-enhanced learning (TEL) refers to learning activities and environments that are potentially improved or enhanced with information and communication technologies (Shen and Ho, 2020; Wasson and Kirschner, 2020). TEL may be implemented in face-to-face, distance/remote and blended or hybrid modes; in various environments such as online, classrooms, workplaces, communities, and other built and natural environments; include a range of learning designs and pedagogies/andragogies; involve synchronous and asynchronous interactions amongst students, teachers, workplace staff and clients, and/or community members; and delivered with the support of various technologies (Wasson and Kirschner, 2020). To date, the Technology-Enhanced Learning in Medical Education collection, part of MedEdPublish, has received submissions relating to several technologies to support learning, including web conferencing, web 2.0, e-textbooks, e-portfolios, software, generative artificial intelligence, simulation mannequins and wearables for point-of-view video, often in combination. Learning designs included flipped classroom with interactive case discussions (Imran et al., 2022), e-portfolios (Javed et al., 2023), didactic teaching followed by demonstrations of clinical skills on a simulation mannequin (Zwaiman et al., 2023), interdisciplinary case discussions to promote interprofessional learning (Major et al., 2023), patient panels to share narratives and perspectives (Papanagnou et al., 2023), and team-based learning (Lee & Wong, 2023). In the four papers that included evaluation, participant reaction (feedback on learning activities) and/or learning (self-reported through surveys, with pre- vs post-training comparisons or at different timepoints during learning) were reported, corresponding to levels 1 and 2 of the commonly used outcomes-focused Kirkpatrick model of evaluation (Allen et al., 2022). Two papers focused on the work of health professions educators, including conducting the nominal group technique, a qualitative research method, via web conferencing (Khurshid et al., 2023); and using ChatGPT to assist with various medical education tasks (Peacock et al., 2023).

Opportunities, Challenges, and Future Directions of Generative Artificial Intelligence in Medical Education: Scoping Review

BACKGROUND

Generative artificial intelligence (AI) technologies are increasingly being utilized across various fields, with considerable interest and concern regarding their potential application in medical education. These technologies, such as Chat GPT and Bard, can generate new content and have a wide range of possible applications.

OBJECTIVE

This study aimed to synthesize the potential opportunities and limitations of generative AI in medical education. It sought to identify prevalent themes within recent literature regarding potential applications and challenges of generative AI in medical education and use these to guide future areas for exploration.

METHODS

We conducted a scoping review, following the framework by Arksey and O'Malley, of English language articles published from 2022 onward that discussed generative AI in the context of medical education. A literature search was performed using PubMed, Web of Science, and Google Scholar databases. We screened articles for inclusion, extracted data from relevant studies, and completed a quantitative and qualitative synthesis of the data.

RESULTS

Thematic analysis revealed diverse potential applications for generative AI in medical education, including self-directed learning, simulation scenarios, and writing assistance. However, the literature also highlighted significant challenges, such as issues with academic integrity, data accuracy, and potential detriments to learning. Based on these themes and the current state of the literature, we propose the following 3 key areas for investigation: developing learners' skills to evaluate AI critically, rethinking assessment methodology, and studying human-AI interactions.

CONCLUSIONS

The integration of generative AI in medical education presents exciting opportunities, alongside considerable challenges. There is a need to develop new skills and competencies related to AI as well as thoughtful, nuanced approaches to examine the growing use of generative AI in medical education.