Exploring the potential of Chat-GPT as a supportive tool for sialendoscopy clinical decision making and patient information support

Generative Pre-trained Transformer for Pediatric Stroke Research: A Pilot Study

BACKGROUND

Pediatric stroke is an important cause of morbidity in children. Although research can be challenging, large amounts of data have been captured through collaborative efforts in the International Pediatric Stroke Study (IPSS). This study explores the use of an advanced artificial intelligence program, the Generative Pre-trained Transformer (GPT), to enter pediatric stroke data into the IPSS.

METHODS

The most recent 50 clinical notes of patients with ischemic stroke or cerebral venous sinus thrombosis at the UTHealth Pediatric Stroke Clinic were deidentified. Domain-specific prompts were engineered for an offline artificial intelligence program (GPT) to answer IPSS questions. Responses from GPT were compared with the human rater. Percent agreement was assessed across 50 patients for each of the 114 queries developed from the IPSS database outcome questionnaire.

RESULTS

GPT demonstrated strong performance on several questions but showed variability overall. In its early iterations it was able to match human judgment occasionally with an accuracy score of 1.00 (n = 20, 17.5%), but it scored as low as 0.26 in some patients. Prompts were adjusted in four subsequent iterations to increase accuracy. In its fourth iteration, agreement was 93.6%, with a maximum agreement of 100% and minimum of 62%. Of 2400 individual items assessed, our model entered 2247 (93.6%) correctly and 153 (6.4%) incorrectly.

CONCLUSIONS

Although our tailored generative model with domain-specific prompt engineering and ontological guidance shows promise for research applications, further refinement is needed to enhance its accuracy. It cannot enter data entirely independently, but it can be employed in tandem with human oversight contributing to a collaborative approach that reduces overall effort.

Assessing the Quality and Readability of Online Patient Information: ENT UK Patient Information e-Leaflets versus Responses by a Generative Artificial Intelligence

BACKGROUND

 The evolution of artificial intelligence has introduced new ways to disseminate health information, including natural language processing models like ChatGPT. However, the quality and readability of such digitally generated information remains understudied. This study is the first to compare the quality and readability of digitally generated health information against leaflets produced by professionals.

METHODOLOGY

 Patient information leaflets from five ENT UK leaflets and their corresponding ChatGPT responses were extracted from the Internet. Assessors with various degrees of medical knowledge evaluated the content using the Ensuring Quality Information for Patients (EQIP) tool and readability tools including the Flesch-Kincaid Grade Level (FKGL). Statistical analysis was performed to identify differences between leaflets, assessors, and sources of information.

RESULTS

 ENT UK leaflets were of moderate quality, scoring a median EQIP of 23. Statistically significant differences in overall EQIP score were identified between ENT UK leaflets, but ChatGPT responses were of uniform quality. Nonspecialist doctors rated the highest EQIP scores, while medical students scored the lowest. The mean readability of ENT UK leaflets was higher than ChatGPT responses. The information metrics of ENT UK leaflets were moderate and varied between topics. Equivalent ChatGPT information provided comparable content quality, but with reduced readability.

CONCLUSION

 ChatGPT patient information and professionally produced leaflets had comparable content, but large language model content required a higher reading age. With the increasing use of online health resources, this study highlights the need for a balanced approach that considers both the quality and readability of patient education materials.

Generative AI and Otolaryngology-Head & Neck Surgery

The increasing development of artificial intelligence (AI) generative models in otolaryngology-head and neck surgery will progressively change our practice. Practitioners and patients have access to AI resources, improving information, knowledge, and practice of patient care. This article summarizes the currently investigated applications of AI generative models, particularly Chatbot Generative Pre-trained Transformer, in otolaryngology-head and neck surgery.

ChatGPT and trainee performances in the management of maxillofacial patients

INTRODUCTION

ChatGPT is an artificial intelligence based large language model with the ability to generate human-like response to text input, its performance has already been the subject of several studies in different fields. The aim of this study was to evaluate the performance of ChatGPT in the management of maxillofacial clinical cases.

MATERIALS AND METHODS

A total of 38 clinical cases consulting at the Stomatology-Maxillofacial Surgery Department were prospectively recruited and presented to ChatGPT, which was interrogated for diagnosis, differential diagnosis, management and treatment. The performance of trainees and ChatGPT was compared by three blinded board-certified maxillofacial surgeons using the AIPI score.

RESULTS

The average total AIPI score assigned to the practitioners was 18.71 and 16.39 to ChatGPT, significantly lower (p < 0.001). According to the experts, ChatGPT was significantly less effective for diagnosis and treatment (p < 0.001). Following two of the three experts, ChatGPT was significantly less effective in considering patient data (p = 0.001) and suggesting additional examinations (p < 0.0001). The primary diagnosis proposed by ChatGPT was judged by the experts as not plausible and /or incomplete in 2.63 % to 18 % of the cases, the additional examinations were associated with inadequate examinations in 2.63 %, to 21.05 % of the cases and proposed an association of pertinent, but incomplete therapeutic findings in 18.42 % to 47.37 % of the cases, while the therapeutic findings were considered pertinent, necessary and inadequate in 18.42 % of cases.

CONCLUSIONS

ChatGPT appears less efficient in diagnosis, the selection of the most adequate additional examination and the proposition of pertinent and necessary therapeutic approaches.

Accuracy and consistency of ChatGPT-3.5 and - 4 in providing differential diagnoses in oral and maxillofacial diseases: a comparative diagnostic performance analysis

OBJECTIVE

To investigate the performance of ChatGPT in the differential diagnosis of oral and maxillofacial diseases.

METHODS

Thirty-seven oral and maxillofacial lesions findings were presented to ChatGPT-3.5 and - 4, 18 dental surgeons trained in oral medicine/pathology (OMP), 23 general dental surgeons (DDS), and 16 dental students (DS) for differential diagnosis. Additionally, a group of 15 general dentists was asked to describe 11 cases to ChatGPT versions. The ChatGPT-3.5, -4, and human primary and alternative diagnoses were rated by 2 independent investigators with a 4 Likert-Scale. The consistency of ChatGPT-3.5 and - 4 was evaluated with regenerated inputs.

RESULTS

Moderate consistency of outputs was observed for ChatGPT-3.5 and - 4 to provide primary (κ = 0.532 and κ = 0.533 respectively) and alternative (κ = 0.337 and κ = 0.367 respectively) hypotheses. The mean of correct diagnoses was 64.86% for ChatGPT-3.5, 80.18% for ChatGPT-4, 86.64% for OMP, 24.32% for DDS, and 16.67% for DS. The mean correct primary hypothesis rates were 45.95% for ChatGPT-3.5, 61.80% for ChatGPT-4, 82.28% for OMP, 22.72% for DDS, and 15.77% for DS. The mean correct diagnosis rate for ChatGPT-3.5 with standard descriptions was 64.86%, compared to 45.95% with participants' descriptions. For ChatGPT-4, the mean was 80.18% with standard descriptions and 61.80% with participant descriptions.

CONCLUSION

ChatGPT-4 demonstrates an accuracy comparable to specialists to provide differential diagnosis for oral and maxillofacial diseases. Consistency of ChatGPT to provide diagnostic hypotheses for oral diseases cases is moderate, representing a weakness for clinical application. The quality of case documentation and descriptions impacts significantly on the performance of ChatGPT.

CLINICAL RELEVANCE

General dentists, dental students and specialists in oral medicine and pathology may benefit from ChatGPT-4 as an auxiliary method to define differential diagnosis for oral and maxillofacial lesions, but its accuracy is dependent on precise case descriptions.

Accuracy assessment of ChatGPT responses to frequently asked questions regarding anterior cruciate ligament surgery

BACKGROUND

The emergence of artificial intelligence (AI) has allowed users to have access to large sources of information in a chat-like manner. Thereby, we sought to evaluate ChatGPT-4 response's accuracy to the 10 patient most frequently asked questions (FAQs) regarding anterior cruciate ligament (ACL) surgery.

METHODS

A list of the top 10 FAQs pertaining to ACL surgery was created after conducting a search through all Sports Medicine Fellowship Institutions listed on the Arthroscopy Association of North America (AANA) and American Orthopaedic Society of Sports Medicine (AOSSM) websites. A Likert scale was used to grade response accuracy by two sports medicine fellowship-trained surgeons. Cohen's kappa was used to assess inter-rater agreement. Reproducibility of the responses over time was also assessed.

RESULTS

Five of the 10 responses received a 'completely accurate' grade by two-fellowship trained surgeons with three additional replies receiving a 'completely accurate' status by at least one. Moreover, inter-rater reliability accuracy assessment revealed a moderate agreement between fellowship-trained attending physicians (weighted kappa = 0.57, 95% confidence interval 0.15-0.99). Additionally, 80% of the responses were reproducible over time.

CONCLUSION

ChatGPT can be considered an accurate additional tool to answer general patient questions regarding ACL surgery. None the less, patient-surgeon interaction should not be deferred and must continue to be the driving force for information retrieval. Thus, the general recommendation is to address any questions in the presence of a qualified specialist.

Evaluating the Accuracy of ChatGPT in Common Patient Questions Regarding HPV+ Oropharyngeal Carcinoma

OBJECTIVES

Large language model (LLM)-based chatbots such as ChatGPT have been publicly available and increasingly utilized by the general public since late 2022. This study sought to investigate ChatGPT responses to common patient questions regarding Human Papilloma Virus (HPV) positive oropharyngeal cancer (OPC).

METHODS

This was a prospective, multi-institutional study, with data collected from high volume institutions that perform >50 transoral robotic surgery cases per year. The 100 most recent discussion threads including the term "HPV" on the American Cancer Society's Cancer Survivors Network's Head and Neck Cancer public discussion board were reviewed. The 11 most common questions were serially queried to ChatGPT 3.5; answers were recorded. A survey was distributed to fellowship trained head and neck oncologic surgeons at 3 institutions to evaluate the responses.

RESULTS

A total of 8 surgeons participated in the study. For questions regarding HPV contraction and transmission, ChatGPT answers were scored as clinically accurate and aligned with consensus in the head and neck surgical oncology community 84.4% and 90.6% of the time, respectively. For questions involving treatment of HPV+ OPC, ChatGPT was clinically accurate and aligned with consensus 87.5% and 91.7% of the time, respectively. For questions regarding the HPV vaccine, ChatGPT was clinically accurate and aligned with consensus 62.5% and 75% of the time, respectively. When asked about circulating tumor DNA testing, only 12.5% of surgeons thought responses were accurate or consistent with consensus.

CONCLUSION

ChatGPT 3.5 performed poorly with questions involving evolving therapies and diagnostics-thus, caution should be used when using a platform like ChatGPT 3.5 to assess use of advanced technology. Patients should be counseled on the importance of consulting their surgeons to receive accurate and up to date recommendations, and use LLM's to augment their understanding of these important health-related topics.

Evaluation of ChatGPT-4's Performance in Therapeutic Decision-Making During Multidisciplinary Oncology Meetings for Head and Neck Squamous Cell Carcinoma

Objectives First reports suggest that artificial intelligence (AI) such as ChatGPT-4 (Open AI, ChatGPT-4, San Francisco, USA) might represent reliable tools for therapeutic decisions in some medical conditions. This study aims to assess the decisional capacity of ChatGPT-4 in patients with head and neck carcinomas, using the multidisciplinary oncology meeting (MOM) and the National Comprehensive Cancer Network (NCCN) decision as references. Methods This retrospective study included 263 patients with squamous cell carcinoma of the oral cavity, oropharynx, hypopharynx, and larynx who were followed at our institution between January 1, 2016, and December 31, 2021. The recommendation of GPT4 for the first- and second-line treatments was compared to the MOM decision and NCCN guidelines. The degrees of agreement were calculated using the Kappa method, which measures the degree of agreement between two evaluators. Results ChatGPT-4 demonstrated a moderate agreement in first-line treatment recommendations (Kappa = 0.48) and a substantial agreement (Kappa = 0.78) in second-line treatment recommendations compared to the decisions from MOM. A substantial agreement with the NCCN guidelines for both first- and second-line treatments was observed (Kappa = 0.72 and 0.66, respectively). The degree of agreement decreased when the decision included gastrostomy, patients over 70, and those with comorbidities. Conclusions The study illustrates that while ChatGPT-4 can significantly support clinical decision-making in oncology by aligning closely with expert recommendations and established guidelines, ongoing enhancements and training are crucial. The findings advocate for the continued evolution of AI tools to better handle the nuanced aspects of patient health profiles, thus broadening their applicability and reliability in clinical practice.

Applications of ChatGPT in Otolaryngology-Head Neck Surgery: A State of the Art Review

OBJECTIVE

To review the current literature on the application, accuracy, and performance of Chatbot Generative Pre-Trained Transformer (ChatGPT) in Otolaryngology-Head and Neck Surgery.

DATA SOURCES

PubMED, Cochrane Library, and Scopus.

REVIEW METHODS

A comprehensive review of the literature on the applications of ChatGPT in otolaryngology was conducted according to Preferred Reporting Items for Systematic Reviews and Meta-analyses statement.

CONCLUSIONS

ChatGPT provides imperfect patient information or general knowledge related to diseases found in Otolaryngology-Head and Neck Surgery. In clinical practice, despite suboptimal performance, studies reported that the model is more accurate in providing diagnoses, than in suggesting the most adequate additional examinations and treatments related to clinical vignettes or real clinical cases. ChatGPT has been used as an adjunct tool to improve scientific reports (referencing, spelling correction), to elaborate study protocols, or to take student or resident exams reporting several levels of accuracy. The stability of ChatGPT responses throughout repeated questions appeared high but many studies reported some hallucination events, particularly in providing scientific references.

IMPLICATIONS FOR PRACTICE

To date, most applications of ChatGPT are limited in generating disease or treatment information, and in the improvement of the management of clinical cases. The lack of comparison of ChatGPT performance with other large language models is the main limitation of the current research. Its ability to analyze clinical images has not yet been investigated in otolaryngology although upper airway tract or ear images are an important step in the diagnosis of most common ear, nose, and throat conditions. This review may help otolaryngologists to conceive new applications in further research.

Artificial Intelligence-Based Applications for Bone Fracture Detection Using Medical Images: A Systematic Review

Artificial intelligence (AI) is making notable advancements in the medical field, particularly in bone fracture detection. This systematic review compiles and assesses existing research on AI applications aimed at identifying bone fractures through medical imaging, encompassing studies from 2010 to 2023. It evaluates the performance of various AI models, such as convolutional neural networks (CNNs), in diagnosing bone fractures, highlighting their superior accuracy, sensitivity, and specificity compared to traditional diagnostic methods. Furthermore, the review explores the integration of advanced imaging techniques like 3D CT and MRI with AI algorithms, which has led to enhanced diagnostic accuracy and improved patient outcomes. The potential of Generative AI and Large Language Models (LLMs), such as OpenAI's GPT, to enhance diagnostic processes through synthetic data generation, comprehensive report creation, and clinical scenario simulation is also discussed. The review underscores the transformative impact of AI on diagnostic workflows and patient care, while also identifying research gaps and suggesting future research directions to enhance data quality, model robustness, and ethical considerations.

Assessing ChatGPT's Potential in HIV Prevention Communication: A Comprehensive Evaluation of Accuracy, Completeness, and Inclusivity

With the advancement of artificial intelligence(AI), platforms like ChatGPT have gained traction in different fields, including Medicine. This study aims to evaluate the potential of ChatGPT in addressing questions related to HIV prevention and to assess its accuracy, completeness, and inclusivity. A team consisting of 15 physicians, six members from HIV communities, and three experts in gender and queer studies designed an assessment of ChatGPT. Queries were categorized into five thematic groups: general HIV information, behaviors increasing HIV acquisition risk, HIV and pregnancy, HIV testing, and the prophylaxis use. A team of medical doctors was in charge of developing questions to be submitted to ChatGPT. The other members critically assessed the generated responses regarding level of expertise, accuracy, completeness, and inclusivity. The median accuracy score was 5.5 out of 6, with 88.4% of responses achieving a score ≥ 5. Completeness had a median of 3 out of 3, while the median for inclusivity was 2 out of 3. Some thematic groups, like behaviors associated with HIV transmission and prophylaxis, exhibited higher accuracy, indicating variable performance across different topics. Issues of inclusivity were identified, notably the use of outdated terms and a lack of representation for some communities. ChatGPT demonstrates significant potential in providing accurate information on HIV-related topics. However, while responses were often scientifically accurate, they sometimes lacked the socio-political context and inclusivity essential for effective health communication. This underlines the importance of aligning AI-driven platforms with contemporary health communication strategies and ensuring the balance of accuracy and inclusivity.

Can chatbots enhance the management of pediatric sialadenitis in clinical practice?

OBJECTIVE

The purpose of this study was to assess how well ChatGPT, an AI-powered chatbot, performed in helping to manage pediatric sialadenitis and identify when sialendoscopy was necessary.

METHODS

49 clinical cases of pediatric sialadenitis were retrospectively reviewed. ChatGPT was given patient data, and it offered differential diagnoses, proposed further tests, and suggested treatments. The decisions made by the treating otolaryngologists were contrasted with the answers provided by ChatGPT. Analysis was done on ChatGPT response consistency and interrater reliability.

RESULTS

ChatGPT showed 78.57% accuracy in primary diagnosis, and 17.35% of cases were considered likely. On the other hand, otolaryngologists recommended fewer further examinations than ChatGPT (111 vs. 60, p < 0.001). For additional exams, poor agreement was found between ChatGPT and otolaryngologists. Only 28.57% of cases received a pertinent and essential treatment plan via ChatGPT, indicating that the platform's treatment recommendations were frequently lacking. For treatment ratings, judges' interrater reliability was greatest (Kendall's tau = 0.824, p < 0.001). For the most part, ChatGPT's response constancy was high.

CONCLUSIONS

Although ChatGPT has the potential to correctly diagnose pediatric sialadenitis, there are a number of noteworthy limitations with regard to its ability to suggest further testing and treatment regimens. Before widespread clinical use, more research and confirmation are required. To guarantee that chatbots are utilized properly and effectively to supplement human expertise rather than to replace it, a critical viewpoint is required.

ChatGPT Responses to Frequently Asked Questions on Ménière's Disease: A Comparison to Clinical Practice Guideline Answers

Objective

Evaluate the quality of responses from Chat Generative Pre-Trained Transformer (ChatGPT) models compared to the answers for "Frequently Asked Questions" (FAQs) from the American Academy of Otolaryngology-Head and Neck Surgery (AAO-HNS) Clinical Practice Guidelines (CPG) for Ménière's disease (MD).

Study Design

Comparative analysis.

Setting

The AAO-HNS CPG for MD includes FAQs that clinicians can give to patients for MD-related questions. The ability of ChatGPT to properly educate patients regarding MD is unknown.

Methods

ChatGPT-3.5 and 4.0 were each prompted with 16 questions from the MD FAQs. Each response was rated in terms of (1) comprehensiveness, (2) extensiveness, (3) presence of misleading information, and (4) quality of resources. Readability was assessed using Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease Score (FRES).

Results

ChatGPT-3.5 was comprehensive in 5 responses whereas ChatGPT-4.0 was comprehensive in 9 (31.3% vs 56.3%, P = .2852). ChatGPT-3.5 and 4.0 were extensive in all responses (P = 1.0000). ChatGPT-3.5 was misleading in 5 responses whereas ChatGPT-4.0 was misleading in 3 (31.3% vs 18.75%, P = .6851). ChatGPT-3.5 had quality resources in 10 responses whereas ChatGPT-4.0 had quality resources in 16 (62.5% vs 100%, P = .0177). AAO-HNS CPG FRES (62.4 ± 16.6) demonstrated an appropriate readability score of at least 60, while both ChatGPT-3.5 (39.1 ± 7.3) and 4.0 (42.8 ± 8.5) failed to meet this standard. All platforms had FKGL means that exceeded the recommended level of 6 or lower.

Conclusion

While ChatGPT-4.0 had significantly better resource reporting, both models have room for improvement in being more comprehensive, more readable, and less misleading for patients.

Assessing Generative Pretrained Transformers (GPT) in Clinical Decision-Making: Comparative Analysis of GPT-3.5 and GPT-4

BACKGROUND

Artificial intelligence, particularly chatbot systems, is becoming an instrumental tool in health care, aiding clinical decision-making and patient engagement.

OBJECTIVE

This study aims to analyze the performance of ChatGPT-3.5 and ChatGPT-4 in addressing complex clinical and ethical dilemmas, and to illustrate their potential role in health care decision-making while comparing seniors' and residents' ratings, and specific question types.

METHODS

A total of 4 specialized physicians formulated 176 real-world clinical questions. A total of 8 senior physicians and residents assessed responses from GPT-3.5 and GPT-4 on a 1-5 scale across 5 categories: accuracy, relevance, clarity, utility, and comprehensiveness. Evaluations were conducted within internal medicine, emergency medicine, and ethics. Comparisons were made globally, between seniors and residents, and across classifications.

RESULTS

Both GPT models received high mean scores (4.4, SD 0.8 for GPT-4 and 4.1, SD 1.0 for GPT-3.5). GPT-4 outperformed GPT-3.5 across all rating dimensions, with seniors consistently rating responses higher than residents for both models. Specifically, seniors rated GPT-4 as more beneficial and complete (mean 4.6 vs 4.0 and 4.6 vs 4.1, respectively; P<.001), and GPT-3.5 similarly (mean 4.1 vs 3.7 and 3.9 vs 3.5, respectively; P<.001). Ethical queries received the highest ratings for both models, with mean scores reflecting consistency across accuracy and completeness criteria. Distinctions among question types were significant, particularly for the GPT-4 mean scores in completeness across emergency, internal, and ethical questions (4.2, SD 1.0; 4.3, SD 0.8; and 4.5, SD 0.7, respectively; P<.001), and for GPT-3.5's accuracy, beneficial, and completeness dimensions.

CONCLUSIONS

ChatGPT's potential to assist physicians with medical issues is promising, with prospects to enhance diagnostics, treatments, and ethics. While integration into clinical workflows may be valuable, it must complement, not replace, human expertise. Continued research is essential to ensure safe and effective implementation in clinical environments.

Evaluating ChatGPT-4's performance as a digital health advisor for otosclerosis surgery

Purpose

This study aims to evaluate the effectiveness of ChatGPT-4, an artificial intelligence (AI) chatbot, in providing accurate and comprehensible information to patients regarding otosclerosis surgery.

Methods

On October 20, 2023, 15 hypothetical questions were posed to ChatGPT-4 to simulate physician-patient interactions about otosclerosis surgery. Responses were evaluated by three independent ENT specialists using the DISCERN scoring system. The readability was evaluated using multiple indices: Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (Gunning FOG), Simple Measure of Gobbledygook (SMOG), Coleman-Liau Index (CLI), and Automated Readability Index (ARI).

Results

The responses from ChatGPT-4 received DISCERN scores ranging from poor to excellent, with an overall score of 50.7 ± 8.2. The readability analysis indicated that the texts were above the 6th-grade level, suggesting they may not be easily comprehensible to the average reader. There was a significant positive correlation between the referees' scores. Despite providing correct information in over 90% of the cases, the study highlights concerns regarding the potential for incomplete or misleading answers and the high readability level of the responses.

Conclusion

While ChatGPT-4 shows potential in delivering health information accurately, its utility is limited by the level of readability of its responses. The study underscores the need for continuous improvement in AI systems to ensure the delivery of information that is both accurate and accessible to patients with varying levels of health literacy. Healthcare professionals should supervise the use of such technologies to enhance patient education and care.

The role of artificial intelligence in training ENT residents: a survey on ChatGPT, a new method of investigation

Objective

The primary focus of this study was to analyze the adoption of ChatGPT among Ear, Nose, and Throat (ENT) trainees, encompassing its role in scientific research and personal study. We examined in which year ENT trainees become involved in clinical research and how many scientific investigations they have been engaged in.

Methods

An online survey was distributed to ENT residents employed in Italian University Hospitals.

Results

Out of 609 Italian ENT trainees, 181 (29.7%) responded to the survey. Among these, 67.4% were familiar with ChatGPT, and 18.9% of them used artificial intelligence as a tool for research and study. In all, 32.6% were not familiar with ChatGPT and its functions. Within our sample, there was an increasing trend of participation by ENT trainees in scientific publications throughout their training.

Conclusions

ChatGPT remains relatively unfamiliar and underutilised in Italy, even though it could be a valuable and efficient tool for ENT trainees, providing quick access for study and research through both personal computers and smartphones.

Evaluating the validity of the nursing statements algorithmically generated based on the International Classifications of Nursing Practice for respiratory nursing care using large language models

OBJECTIVE

This study aims to facilitate the creation of quality standardized nursing statements in South Korea's hospitals using algorithmic generation based on the International Classifications of Nursing Practice (ICNP) and evaluation through Large Language Models.

MATERIALS AND METHODS

We algorithmically generated 15 972 statements related to acute respiratory care using 117 concepts and concept composition models of ICNP. Human reviewers, Generative Pre-trained Transformers 4.0 (GPT-4.0), and Bio_Clinical Bidirectional Encoder Representations from Transformers (BERT) evaluated the generated statements for validity. The evaluation by GPT-4.0 and Bio_ClinicalBERT was conducted with and without contextual information and training.

RESULTS

Of the generated statements, 2207 were deemed valid by expert reviewers. GPT-4.0 showed a zero-shot  AUC of 0.857, which aggravated with contextual information. Bio_ClinicalBERT, after training, significantly improved, reaching an AUC of 0.998.

CONCLUSION

Bio_ClinicalBERT effectively validates auto-generated nursing statements, offering a promising solution to enhance and streamline healthcare documentation processes.

Evaluation of the Impact of ChatGPT on the Selection of Surgical Technique in Bariatric Surgery

PURPOSE

With the growing interest in artificial intelligence (AI) applications in medicine, this study explores ChatGPT's potential to influence surgical technique selection in metabolic and bariatric surgery (MBS), contrasting AI recommendations with established clinical guidelines and expert consensus.

MATERIALS AND METHODS

Conducting a single-center retrospective analysis, the study involved 161 patients who underwent MBS between January 2022 and December 2023. ChatGPT4 was used to analyze patient data, including demographics, pathological history, and BMI, to recommend the most suitable surgical technique. These AI recommendations were then compared with the hospital's algorithm-based decisions.

RESULTS

ChatGPT recommended Roux-en-Y gastric bypass in over half of the cases. However, a significant difference was observed between AI suggestions and actual surgical techniques applied, with only a 34.16% match rate. Further analysis revealed any significant correlation between ChatGPT recommendations and the established surgical algorithm.

CONCLUSION

Despite ChatGPT's ability to process and analyze large datasets, its recommendations for MBS techniques do not align closely with those determined by expert surgical teams using a high success rate algorithm. Consequently, the study concludes that ChatGPT4 should not replace expert consultation in selecting MBS techniques.

Validation of the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool: a new tool to assess the quality of health information provided by AI platforms

BACKGROUND

The widespread diffusion of Artificial Intelligence (AI) platforms is revolutionizing how health-related information is disseminated, thereby highlighting the need for tools to evaluate the quality of such information. This study aimed to propose and validate the Quality Assessment of Medical Artificial Intelligence (QAMAI), a tool specifically designed to assess the quality of health information provided by AI platforms.

METHODS

The QAMAI tool has been developed by a panel of experts following guidelines for the development of new questionnaires. A total of 30 responses from ChatGPT4, addressing patient queries, theoretical questions, and clinical head and neck surgery scenarios were assessed by 27 reviewers from 25 academic centers worldwide. Construct validity, internal consistency, inter-rater and test-retest reliability were assessed to validate the tool.

RESULTS

The validation was conducted on the basis of 792 assessments for the 30 responses given by ChatGPT4. The results of the exploratory factor analysis revealed a unidimensional structure of the QAMAI with a single factor comprising all the items that explained 51.1% of the variance with factor loadings ranging from 0.449 to 0.856. Overall internal consistency was high (Cronbach's alpha = 0.837). The Interclass Correlation Coefficient was 0.983 (95% CI 0.973-0.991; F (29,542) = 68.3; p < 0.001), indicating excellent reliability. Test-retest reliability analysis revealed a moderate-to-strong correlation with a Pearson's coefficient of 0.876 (95% CI 0.859-0.891; p < 0.001).

CONCLUSIONS

The QAMAI tool demonstrated significant reliability and validity in assessing the quality of health information provided by AI platforms. Such a tool might become particularly important/useful for physicians as patients increasingly seek medical information on AI platforms.

ChatGPT-4 accuracy for patient education in laryngopharyngeal reflux

INTRODUCTION

Chatbot Generative Pre-trained Transformer (ChatGPT) is an artificial intelligence-powered language model chatbot able to help otolaryngologists in practice and research. The ability of ChatGPT in generating patient-centered information related to laryngopharyngeal reflux disease (LPRD) was evaluated.

METHODS

Twenty-five questions dedicated to definition, clinical presentation, diagnosis, and treatment of LPRD were developed from the Dubai definition and management of LPRD consensus and recent reviews. Questions about the four aforementioned categories were entered into ChatGPT-4. Four board-certified laryngologists evaluated the accuracy of ChatGPT-4 with a 5-point Likert scale. Interrater reliability was evaluated.

RESULTS

The mean scores (SD) of ChatGPT-4 answers for definition, clinical presentation, additional examination, and treatments were 4.13 (0.52), 4.50 (0.72), 3.75 (0.61), and 4.18 (0.47), respectively. Experts reported high interrater reliability for sub-scores (ICC = 0.973). The lowest performances of ChatGPT-4 were on answers about the most prevalent LPR signs, the most reliable objective tool for the diagnosis (hypopharyngeal-esophageal multichannel intraluminal impedance-pH monitoring (HEMII-pH)), and the criteria for the diagnosis of LPR using HEMII-pH.

CONCLUSION

ChatGPT-4 may provide adequate information on the definition of LPR, differences compared to GERD (gastroesophageal reflux disease), and clinical presentation. Information provided upon extra-laryngeal manifestations and HEMII-pH may need further optimization. Regarding the recent trends identifying increasing patient use of internet sources for self-education, the findings of the present study may help draw attention to ChatGPT-4's accuracy on the topic of LPR.

Randomized Controlled Trial of Clinical Guidelines Versus Interactive Decision-Support for Improving Medical Trainees' Confidence with Latent Tuberculosis Care

BACKGROUND

In order to eliminate tuberculosis (TB) in the USA, primary care providers must take on an expanded role in the diagnosis and management of latent tuberculosis infection (LTBI). Clinical practice guidelines and recommendations exist for LTBI management, but there is a need for innovative tools to improve medical students' and residents' knowledge of evidence-based practices for LTBI testing and treatment.

OBJECTIVE

To assess the impact of LTBI-ASSIST, a free online decision support aid, as a novel educational tool and mechanism of delivering clinical practice guidelines for medical trainees.

DESIGN

A single site, randomized controlled trial of trainees delivered by electronic survey.

INTERVENTIONS

Medical students and Internal Medicine residents at the Johns Hopkins University School of Medicine.

PARTICIPANTS

Participants were randomized in 1:1 ratio to receive the US clinical practice guidelines and recommendations for Latent TB management (control arm) or the guidelines plus an introduction to LTBI-ASSIST (LTBI-ASSIST arm) as they completed a case-based knowledge assessment and reported confidence with domains of LTBI care.

MAIN MEASURES

(1) Proportion of questions answered correctly on a case-based knowledge assessment; (2) change in reported confidence with domains of LTBI care.

KEY RESULTS

One hundred and thirty participants completed the knowledge assessment. Those randomized to receive the LTBI-ASSIST Tool performed better on the case-based knowledge assessment with a mean score of 75.9% (95% CI: 70.6-81.1), compared to 57.4% (52.8-62.0) in the group that received the guidelines only (p <0.001). Similarly, the LTBI-ASSIST group reported a higher change in confidence (measured as post-assessment confidence minus pre-assessment confidence), compared to the control group, in six of the seven domains of LTBI care.

CONCLUSIONS

LTBI-ASSIST can be an effective supplement to existing guidelines in educating medical trainees and helping providers find evidence-based, guideline-supported answers for questions encountered in clinical practice.

TRIAL REGISTRATION

NIH Clinical Trial Registry No. NCT05772065.

The Application of ChatGPT in Medicine: A Scoping Review and Bibliometric Analysis

Purpose

ChatGPT has a wide range of applications in the medical field. Therefore, this review aims to define the key issues and provide a comprehensive view of the literature based on the application of ChatGPT in medicine.

Methods

This scope follows Arksey and O'Malley's five-stage framework. A comprehensive literature search of publications (30 November 2022 to 16 August 2023) was conducted. Six databases were searched and relevant references were systematically catalogued. Attention was focused on the general characteristics of the articles, their fields of application, and the advantages and disadvantages of using ChatGPT. Descriptive statistics and narrative synthesis methods were used for data analysis.

Results

Of the 3426 studies, 247 met the criteria for inclusion in this review. The majority of articles (31.17%) were from the United States. Editorials (43.32%) ranked first, followed by experimental studys (11.74%). The potential applications of ChatGPT in medicine are varied, with the largest number of studies (45.75%) exploring clinical practice, including assisting with clinical decision support and providing disease information and medical advice. This was followed by medical education (27.13%) and scientific research (16.19%). Particularly noteworthy in the discipline statistics were radiology, surgery and dentistry at the top of the list. However, ChatGPT in medicine also faces issues of data privacy, inaccuracy and plagiarism.

Conclusion

The application of ChatGPT in medicine focuses on different disciplines and general application scenarios. ChatGPT has a paradoxical nature: it offers significant advantages, but at the same time raises great concerns about its application in healthcare settings. Therefore, it is imperative to develop theoretical frameworks that not only address its widespread use in healthcare but also facilitate a comprehensive assessment. In addition, these frameworks should contribute to the development of strict and effective guidelines and regulatory measures.

Can ChatGPT Replace an Otolaryngologist in Guiding Parents on Tonsillectomy?

Background: ChatGPT is an artificial intelligence tool, which utilizes machine learning to analyze and generate human-like text. The user-friendly accessibility of this tool enables patients conveniently access medical information without intricate terminology challenges. The objective of this study was to assess the accuracy of ChatGPT in providing insights into indications and management of complications after tonsillectomy, a common pediatric otolaryngology procedure. Methods: The responses generated by ChatGPT were compared to the "Clinical practice guidelines: tonsillectomy in children-executive summary" developed by the American Academy of Otolaryngology-Head and Neck Surgery Foundation (AAO-HNSF). An assessment was carried out by presenting predetermined questions regarding indications and complications post tonsillectomy to ChatGPT, followed by a comparison of its responses with the established guideline by 2 otolaryngology experts. The responses of both parties were reviewed by the senior author. Results: A total of 16 responses generated by ChatGPT were assessed. After a comprehensive review, it was concluded that 15 out of 16 (93.8%) responses demonstrated a high degree of reliability and accuracy, closely adhering to the standard established by the AAO-HNSF guideline. Conclusion: The results validate the potential of using ChatGPT to enhance healthcare delivery making guidelines more accessible to patients while also emphasizing the importance of ensuring the provision of accurate and reliable medical advice to patients.

Reliability of large language models in managing odontogenic sinusitis clinical scenarios: a preliminary multidisciplinary evaluation

PURPOSE

This study aimed to evaluate the utility of large language model (LLM) artificial intelligence tools, Chat Generative Pre-Trained Transformer (ChatGPT) versions 3.5 and 4, in managing complex otolaryngological clinical scenarios, specifically for the multidisciplinary management of odontogenic sinusitis (ODS).

METHODS

A prospective, structured multidisciplinary specialist evaluation was conducted using five ad hoc designed ODS-related clinical scenarios. LLM responses to these scenarios were critically reviewed by a multidisciplinary panel of eight specialist evaluators (2 ODS experts, 2 rhinologists, 2 general otolaryngologists, and 2 maxillofacial surgeons). Based on the level of disagreement from panel members, a Total Disagreement Score (TDS) was calculated for each LLM response, and TDS comparisons were made between ChatGPT3.5 and ChatGPT4, as well as between different evaluators.

RESULTS

While disagreement to some degree was demonstrated in 73/80 evaluator reviews of LLMs' responses, TDSs were significantly lower for ChatGPT4 compared to ChatGPT3.5. Highest TDSs were found in the case of complicated ODS with orbital abscess, presumably due to increased case complexity with dental, rhinologic, and orbital factors affecting diagnostic and therapeutic options. There were no statistically significant differences in TDSs between evaluators' specialties, though ODS experts and maxillofacial surgeons tended to assign higher TDSs.

CONCLUSIONS

LLMs like ChatGPT, especially newer versions, showed potential for complimenting evidence-based clinical decision-making, but substantial disagreement was still demonstrated between LLMs and clinical specialists across most case examples, suggesting they are not yet optimal in aiding clinical management decisions. Future studies will be important to analyze LLMs' performance as they evolve over time.

Chat GPT for the management of obstructive sleep apnea: do we have a polar star?

PURPOSE

This study explores the potential of the Chat-Generative Pre-Trained Transformer (Chat-GPT), a Large Language Model (LLM), in assisting healthcare professionals in the diagnosis of obstructive sleep apnea (OSA). It aims to assess the agreement between Chat-GPT's responses and those of expert otolaryngologists, shedding light on the role of AI-generated content in medical decision-making.

METHODS

A prospective, cross-sectional study was conducted, involving 350 otolaryngologists from 25 countries who responded to a specialized OSA survey. Chat-GPT was tasked with providing answers to the same survey questions. Responses were assessed by both super-experts and statistically analyzed for agreement.

RESULTS

The study revealed that Chat-GPT and expert responses shared a common answer in over 75% of cases for individual questions. However, the overall consensus was achieved in only four questions. Super-expert assessments showed a moderate agreement level, with Chat-GPT scoring slightly lower than experts. Statistically, Chat-GPT's responses differed significantly from experts' opinions (p = 0.0009). Sub-analysis revealed areas of improvement for Chat-GPT, particularly in questions where super-experts rated its responses lower than expert consensus.

CONCLUSIONS

Chat-GPT demonstrates potential as a valuable resource for OSA diagnosis, especially where access to specialists is limited. The study emphasizes the importance of AI-human collaboration, with Chat-GPT serving as a complementary tool rather than a replacement for medical professionals. This research contributes to the discourse in otolaryngology and encourages further exploration of AI-driven healthcare applications. While Chat-GPT exhibits a commendable level of consensus with expert responses, ongoing refinements in AI-based healthcare tools hold significant promise for the future of medicine, addressing the underdiagnosis and undertreatment of OSA and improving patient outcomes.

Validity and reliability of an instrument evaluating the performance of intelligent chatbot: the Artificial Intelligence Performance Instrument (AIPI)

OBJECTIVES

To evaluate the reliability and validity of the Artificial Intelligence Performance Instrument (AIPI).

METHODS

Medical records of patients consulting in otolaryngology were evaluated by physicians and ChatGPT for differential diagnosis, management, and treatment. The ChatGPT performance was rated twice using AIPI within a 7-day period to assess test-retest reliability. Internal consistency was evaluated using Cronbach's α. Internal validity was evaluated by comparing the AIPI scores of the clinical cases rated by ChatGPT and 2 blinded practitioners. Convergent validity was measured by comparing the AIPI score with a modified version of the Ottawa Clinical Assessment Tool (OCAT). Interrater reliability was assessed using Kendall's tau.

RESULTS

Forty-five patients completed the evaluations (28 females). The AIPI Cronbach's alpha analysis suggested an adequate internal consistency (α = 0.754). The test-retest reliability was moderate-to-strong for items and the total score of AIPI (rs = 0.486, p = 0.001). The mean AIPI score of the senior otolaryngologist was significantly higher compared to the score of ChatGPT, supporting adequate internal validity (p = 0.001). Convergent validity reported a moderate and significant correlation between AIPI and modified OCAT (rs = 0.319; p = 0.044). The interrater reliability reported significant positive concordance between both otolaryngologists for the patient feature, diagnostic, additional examination, and treatment subscores as well as for the AIPI total score.

CONCLUSIONS

AIPI is a valid and reliable instrument in assessing the performance of ChatGPT in ear, nose and throat conditions. Future studies are needed to investigate the usefulness of AIPI in medicine and surgery, and to evaluate the psychometric properties in these fields.

ChatGPT vs UpToDate: comparative study of usefulness and reliability of Chatbot in common clinical presentations of otorhinolaryngology-head and neck surgery

PURPOSE

The usage of Chatbots as a kind of Artificial Intelligence in medicine is getting to increase in recent years. UpToDate® is another well-known search tool established on evidence-based knowledge and is used daily by doctors worldwide. In this study, we aimed to investigate the usefulness and reliability of ChatGPT compared to UpToDate in Otorhinolaryngology and Head and Neck Surgery (ORL-HNS).

MATERIALS AND METHODS

ChatGPT-3.5 and UpToDate were interrogated for the management of 25 common clinical case scenarios (13 males/12 females) recruited from literature considering the daily observation at the Department of Otorhinolaryngology of Ege University Faculty of Medicine. Scientific references for the management were requested for each clinical case. The accuracy of the references in the ChatGPT answers was assessed on a 0-2 scale and the usefulness of the ChatGPT and UpToDate answers was assessed with 1-3 scores by reviewers. UpToDate and ChatGPT 3.5 responses were compared.

RESULTS

ChatGPT did not give references in some questions in contrast to UpToDate. Information on the ChatGPT was limited to 2021. UpToDate supported the paper with subheadings, tables, figures, and algorithms. The mean accuracy score of references in ChatGPT answers was 0.25-weak/unrelated. The median (Q1-Q3) was 1.00 (1.25-2.00) for ChatGPT and 2.63 (2.75-3.00) for UpToDate, the difference was statistically significant (p < 0.001). UpToDate was observed more useful and reliable than ChatGPT.

CONCLUSIONS

ChatGPT has the potential to support the physicians to find out the information but our results suggest that ChatGPT needs to be improved to increase the usefulness and reliability of medical evidence-based knowledge.

Exploring the role of ChatGPT in clinical decision-making in otorhinolaryngology: a ChatGPT designed study

PURPOSE

Since the beginning of 2023, ChatGPT emerged as a hot topic in healthcare research. The potential to be a valuable tool in clinical practice is compelling, particularly in improving clinical decision support by helping physicians to make clinical decisions based on the best medical knowledge available. We aim to investigate ChatGPT's ability to identify, diagnose and manage patients with otorhinolaryngology-related symptoms.

METHODS

A prospective, cross-sectional study was designed based on an idea suggested by ChatGPT to assess the level of agreement between ChatGPT and five otorhinolaryngologists (ENTs) in 20 reality-inspired clinical cases. The clinical cases were presented to the chatbot on two different occasions (ChatGPT-1 and ChatGPT-2) to assess its temporal stability.

RESULTS

The mean score of ChatGPT-1 was 4.4 (SD 1.2; min 1, max 5) and of ChatGPT-2 was 4.15 (SD 1.3; min 1, max 5), while the ENTs mean score was 4.91 (SD 0.3; min 3, max 5). The Mann-Whitney U test revealed a statistically significant difference (p < 0.001) between both ChatGPT's and the ENTs's score. ChatGPT-1 and ChatGPT-2 gave different answers in five occasions.

CONCLUSIONS

Artificial intelligence will be an important instrument in clinical decision-making in the near future and ChatGPT is the most promising chatbot so far. Despite needing further development to be used with safety, there is room for improvement and potential to aid otorhinolaryngology residents and specialists in making the most correct decision for the patient.

Accuracy of ChatGPT in head and neck oncological board decisions: preliminary findings

OBJECTIVES

To evaluate the ChatGPT-4 performance in oncological board decisions.

METHODS

Twenty medical records of patients with head and neck cancer were evaluated by ChatGPT-4 for additional examinations, management, and therapeutic approaches. The ChatGPT-4 propositions were assessed with the Artificial Intelligence Performance Instrument. The stability of ChatGPT-4 was evaluated through regenerated answers at 1-day interval.

RESULTS

ChatGPT-4 provided adequate explanations for cTNM staging in 19 cases (95%). ChatGPT-4 proposed a significant higher number of additional examinations than practitioners (72 versus 103; p = 0.001). ChatGPT-4 indications of endoscopy-biopsy, HPV research, ultrasonography, and PET-CT were consistent with the oncological board decisions. The therapeutic propositions of ChatGPT-4 were accurate in 13 cases (65%). Most additional examination and primary treatment propositions were consistent throughout regenerated response process.

CONCLUSIONS

ChatGPT-4 may be an adjunctive theoretical tool in oncological board simple decisions.

The quality and readability of patient information provided by ChatGPT: can AI reliably explain common ENT operations?

PURPOSE

Access to high-quality and comprehensible patient information is crucial. However, information provided by increasingly prevalent Artificial Intelligence tools has not been thoroughly investigated. This study assesses the quality and readability of information from ChatGPT regarding three index ENT operations: tonsillectomy, adenoidectomy, and grommets.

METHODS

We asked ChatGPT standard and simplified questions. Readability was calculated using Flesch-Kincaid Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI) and Simple Measure of Gobbledygook (SMOG) scores. We assessed quality using the DISCERN instrument and compared these with ENT UK patient leaflets.

RESULTS

ChatGPT readability was poor, with mean FRES of 38.9 and 55.1 pre- and post-simplification, respectively. Simplified information from ChatGPT was 43.6% more readable (FRES) but scored 11.6% lower for quality. ENT UK patient information readability and quality was consistently higher.

CONCLUSIONS

ChatGPT can simplify information at the expense of quality, resulting in shorter answers with important omissions. Limitations in knowledge and insight curb its reliability for healthcare information. Patients should use reputable sources from professional organisations alongside clear communication with their clinicians for well-informed consent and making decisions.

How ChatGPT works: a mini review

OBJECTIVE

This paper offers a mini-review of OpenAI's language model, ChatGPT, detailing its mechanisms, applications in healthcare, and comparisons with other large language models (LLMs).

METHODS

The underlying technology of ChatGPT is outlined, focusing on its neural network architecture, training process, and the role of key elements such as input embedding, encoder, decoder, attention mechanism, and output projection. The advancements in GPT-4, including its capacity for internet connection and the integration of plugins for enhanced functionality are discussed.

RESULTS

ChatGPT can generate creative, coherent, and contextually relevant sentences, making it a valuable tool in healthcare for patient engagement, medical education, and clinical decision support. Yet, like other LLMs, it has limitations, including a lack of common sense knowledge, a propensity for hallucination of facts, a restricted context window, and potential privacy concerns.

CONCLUSION

Despite the limitations, LLMs like ChatGPT offer transformative possibilities for healthcare. With ongoing research in model interpretability, common-sense reasoning, and handling of longer context windows, their potential is vast. It is crucial for healthcare professionals to remain informed about these technologies and consider their ethical integration into practice.

ChatGPT in Head and Neck Oncology-Opportunities and Challenges

Head and neck oncology represents a complex and challenging field, encompassing the diagnosis, treatment and management of various malignancies affecting the intricate anatomical structures of the head and neck region. With advancements in artificial intelligence (AI), chatbot applications have emerged as a promising tool to revolutionize the field of Head and Neck oncology. ChatGPT is a cutting-edge language model developed by OpenAI that can help the oncologist in the clinic in scheduling appointments, establishing a clinical diagnosis, making a treatment plan and follow-up. ChatGPT also plays an essential role in telemedicine consultations, medical documentation, scientific writing and research. ChatGPT carries its inherent drawbacks too. ChatGPT raises significant ethical concerns related to authorship, accountability, transparency, bias, and the potential for misinformation. ChatGPT's training data is limited to September 2021; thus, regular updates are required to keep pace with the rapidly evolving medical research and advancements. Therefore, a judicial approach to using ChatGPT is of utmost importance. Head and Neck Oncologists can reap the maximum benefit of this technology in terms of patient care, education and research to improve clinical outcomes.

Is artificial intelligence ready to replace specialist doctors entirely? ENT specialists vs ChatGPT: 1-0, ball at the center

PURPOSE

The purpose of this study is to evaluate ChatGPT's responses to Ear, Nose and Throat (ENT) clinical cases and compare them with the responses of ENT specialists.

METHODS

We have hypothesized 10 scenarios, based on ENT daily experience, with the same primary symptom. We have constructed 20 clinical cases, 2 for each scenario. We described them to 3 ENT specialists and ChatGPT. The difficulty of the clinical cases was assessed by the 5 ENT authors of this article. The responses of ChatGPT were evaluated by the 5 ENT authors of this article for correctness and consistency with the responses of the 3 ENT experts. To verify the stability of ChatGPT's responses, we conducted the searches, always from the same account, for 5 consecutive days.

RESULTS

Among the 20 cases, 8 were rated as low complexity, 6 as moderate complexity and 6 as high complexity. The overall mean correctness and consistency score of ChatGPT responses was 3.80 (SD 1.02) and 2.89 (SD 1.24), respectively. We did not find a statistically significant difference in the average ChatGPT correctness and coherence score according to case complexity. The total intraclass correlation coefficient (ICC) for the stability of the correctness and consistency of ChatGPT was 0.763 (95% confidence interval [CI] 0.553-0.895) and 0.837 (95% CI 0.689-0.927), respectively.

CONCLUSIONS

Our results revealed the potential usefulness of ChatGPT in ENT diagnosis. The instability in responses and the inability to recognise certain clinical elements are its main limitations.

ChatGPT performance in laryngology and head and neck surgery: a clinical case-series

OBJECTIVES

To study the performance of ChatGPT in the management of laryngology and head and neck (LHN) cases.

METHODS

History and clinical examination of patients consulting at the Otolaryngology-Head and Neck Surgery department were presented to ChatGPT, which was interrogated for differential diagnosis, management, and treatment. The ChatGPT performance was assessed by two blinded board-certified otolaryngologists using the following items of a composite score and the Ottawa Clinic Assessment Tool: differential diagnosis; additional examination; and treatment options. The complexity of clinical cases was evaluated with the Amsterdam Clinical Challenge Scale test.

RESULTS

Forty clinical cases were submitted to ChatGPT, accounting for 14 (35%), 12 (30%), and 14 (35%) easy, moderate and difficult cases, respectively. ChatGPT indicated a significant higher number of additional examinations compared to practitioners (p = 0.001). There was a significant agreement between practitioners and ChatGPT for the indication of some common examinations (audiometry, ultrasonography, biopsy, gastrointestinal endoscopy or videofluoroscopy). ChatGPT never indicated some important additional examinations (PET-CT, voice quality assessment, or impedance-pH monitoring). ChatGPT reported highest performance in the proposition of the primary (90%) or the most plausible differential diagnoses (65%), and the therapeutic options (60-68%). The ChatGPT performance in the indication of additional examinations was lowest.

CONCLUSIONS

ChatGPT is a promising adjunctive tool in LHN practice, providing extensive documentation about disease-related additional examinations, differential diagnoses, and treatments. The ChatGPT is more efficient in diagnosis and treatment, rather than in the selection of the most adequate additional examination.

Assessing the accuracy of ChatGPT references in head and neck and ENT disciplines

PURPOSE

ChatGPT has gained popularity as a web application since its release in 2022. While artificial intelligence (AI) systems' potential in scientific writing is widely discussed, their reliability in reviewing literature and providing accurate references remains unexplored. This study examines the reliability of references generated by ChatGPT language models in the Head and Neck field.

METHODS

Twenty clinical questions were generated across different Head and Neck disciplines, to prompt ChatGPT versions 3.5 and 4.0 to produce texts on the assigned topics. The generated references were categorized as "true," "erroneous," or "inexistent" based on congruence with existing records in scientific databases.

RESULTS

ChatGPT 4.0 outperformed version 3.5 in terms of reference reliability. However, both versions displayed a tendency to provide erroneous/non-existent references.

CONCLUSIONS

It is crucial to address this challenge to maintain the reliability of scientific literature. Journals and institutions should establish strategies and good-practice principles in the evolving landscape of AI-assisted scientific writing.

Are artificial intelligence large language models a reliable tool for difficult differential diagnosis? An a posteriori analysis of a peculiar case of necrotizing otitis externa

Key Clinical Message

Large language models have made artificial intelligence readily available to the general public and potentially have a role in healthcare; however, their use in difficult differential diagnosis is still limited, as demonstrated by a case of necrotizing otitis externa.

Abstract

This case report presents a peculiar case of necrotizing otitis externa (NOE) with skull base involvement which proved diagnostically challenging. The initial patient presentation and the imaging performed on the 78-year-old patient suggested a neoplastic rhinopharyngeal lesion and only after several unsuccessful biopsies the patient was transferred to our unit. Upon re-evaluation of the clinical picture, a clinical hypothesis of NOE with skull base erosion was made and confirmed by identifying Pseudomonas aeruginosa in biopsy specimens of skull base bone and external auditory canal skin. Upon diagnosis confirmation, the patient was treated with culture-oriented long-term antibiotics with complete resolution of the disease. Given the complex clinical presentation, we chose to submit a posteriori this NOE case to two large language models (LLM) to test their ability to handle difficult differential diagnoses. LLMs are easily approachable artificial intelligence tools that enable human-like interaction with the user relying upon large information databases for analyzing queries. The LLMs of choice were ChatGPT-3 and ChatGPT-4 and they were requested to analyze the case being provided with only objective clinical and imaging data.