Integrating human expertise & automated methods for a dynamic and multi-parametric evaluation of large language models' feasibility in clinical decision-making

Critical care studies using large language models based on electronic healthcare records: A technical note

The integration of large language models (LLMs) in clinical medicine, particularly in critical care, has introduced transformative capabilities for analyzing and managing complex medical information. This technical note explores the application of LLMs, such as generative pretrained transformer 4 (GPT-4) and Qwen-Chat, in interpreting electronic healthcare records to assist with rapid patient condition assessments, predict sepsis, and automate the generation of discharge summaries. The note emphasizes the significance of LLMs in processing unstructured data from electronic health records (EHRs), extracting meaningful insights, and supporting personalized medicine through nuanced understanding of patient histories. Despite the technical complexity of deploying LLMs in clinical settings, this document provides a comprehensive guide to facilitate the effective integration of LLMs into clinical workflows, focusing on the use of DashScope's application programming interface (API) services for judgment on patient prognosis and organ support recommendations based on natural language in EHRs. By illustrating practical steps and best practices, this work aims to lower the technical barriers for clinicians and researchers, enabling broader adoption of LLMs in clinical research and practice to enhance patient care and outcomes.

Artificial intelligence chatbots in endodontic education-Concepts and potential applications

The integration of artificial intelligence (AI) into education is transforming learning across various domains, including dentistry. Endodontic education can significantly benefit from AI chatbots; however, knowledge gaps regarding their potential and limitations hinder their effective utilization. This narrative review aims to: (A) explain the core functionalities of AI chatbots, including their reliance on natural language processing (NLP), machine learning (ML), and deep learning (DL); (B) explore their applications in endodontic education for personalized learning, interactive training, and clinical decision support; (C) discuss the challenges posed by technical limitations, ethical considerations, and the potential for misinformation. The review highlights that AI chatbots provide learners with immediate access to knowledge, personalized educational experiences, and tools for developing clinical reasoning through case-based learning. Educators benefit from streamlined curriculum development, automated assessment creation, and evidence-based resource integration. Despite these advantages, concerns such as chatbot hallucinations, algorithmic biases, potential for plagiarism, and the spread of misinformation require careful consideration. Analysis of current research reveals limited endodontic-specific studies, emphasizing the need for tailored chatbot solutions validated for accuracy and relevance. Successful integration will require collaborative efforts among educators, developers, and professional organizations to address challenges, ensure ethical use, and establish evaluation frameworks.

Evaluating ChatGPT-4omni in paediatric developmental screening: direct versus sequential prompts

Integrating Large Language Models like ChatGPT-4omni (ChatGPT-4o) into paediatric healthcare could revolutionise developmental screening. This study evaluated ChatGPT-4o's efficacy in paediatric developmental screening using Direct and Sequential Prompting methods compared with the Bayley Scales of Infant Development, Third Edition. Among 106 paediatric cases, Direct Prompting showed a sensitivity of 73.42% and overall accuracy of 69.81%, while Sequential Prompting had a specificity of 62.96% and overall accuracy of 67.92%. Both methods demonstrate potential for improving the efficiency and accessibility of paediatric developmental screening, with Direct Prompts being more sensitive and Sequential Prompts more specific.

Assessing online chat-based artificial intelligence models for weight loss recommendation appropriateness and bias in the presence of guideline incongruence

BACKGROUND AND AIM

Managing obesity requires a comprehensive approach that involves therapeutic lifestyle changes, medications, or metabolic surgery. Many patients seek health information from online sources and artificial intelligence models like ChatGPT, Google Gemini, and Microsoft Copilot before consulting health professionals. This study aims to evaluate the appropriateness of the responses of Google Gemini and Microsoft Copilot to questions on pharmacologic and surgical management of obesity and assess for bias in their responses to either the ADA or AACE guidelines.

METHODS

Ten questions were compiled into a set and posed separately to the free editions of Google Gemini and Microsoft Copilot. Recommendations for the questions were extracted from the ADA and the AACE websites, and the responses were graded by reviewers for appropriateness, completeness, and bias to any of the guidelines.

RESULTS

All responses from Microsoft Copilot and 8/10 (80%) responses from Google Gemini were appropriate. There were no inappropriate responses. Google Gemini refused to respond to two questions and insisted on consulting a physician. Microsoft Copilot (10/10; 100%) provided a higher proportion of complete responses than Google Gemini (5/10; 50%). Of the eight responses from Google Gemini, none were biased towards any of the guidelines, while two of the responses from Microsoft Copilot were biased.

CONCLUSION

The study highlights the role of Microsoft Copilot and Google Gemini in weight loss management. The differences in their responses may be attributed to the variation in the quality and scope of their training data and design.

Accuracy and consistency of publicly available Large Language Models as clinical decision support tools for the management of colon cancer

BACKGROUND

Large Language Models (LLM; e.g., ChatGPT) may be used to assist clinicians and form the basis of future clinical decision support (CDS) for colon cancer. The objectives of this study were to (1) evaluate the response accuracy of two LLM-powered interfaces in identifying guideline-based care in simulated clinical scenarios and (2) define response variation between and within LLMs.

METHODS

Clinical scenarios with "next steps in management" queries were developed based on National Comprehensive Cancer Network guidelines. Prompts were entered into OpenAI ChatGPT and Microsoft Copilot in independent sessions, yielding four responses per scenario. Responses were compared to clinician-developed responses and assessed for accuracy, consistency, and verbosity.

RESULTS

Across 108 responses to 27 prompts, both platforms yielded completely correct responses to 36% of scenarios (n = 39). For ChatGPT, 39% (n = 21) were missing information and 24% (n = 14) contained inaccurate/misleading information. Copilot performed similarly, with 37% (n = 20) having missing information and 28% (n = 15) containing inaccurate/misleading information (p = 0.96). Clinician responses were significantly shorter (34 ± 15.5 words) than both ChatGPT (251 ± 86 words) and Copilot (271 ± 67 words; both p < 0.01).

CONCLUSIONS

Publicly available LLM applications often provide verbose responses with vague or inaccurate information regarding colon cancer management. Significant optimization is required before use in formal CDS.