Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma

Assessment of the Responses of the Artificial Intelligence-based Chatbot ChatGPT-4 to Frequently Asked Questions About Amblyopia and Childhood Myopia

PURPOSE

To assess the responses of the ChatGPT-4, the forerunner artificial intelligence-based chatbot, to frequently asked questions regarding two common pediatric ophthalmologic disorders, amblyopia and childhood myopia.

METHODS

Twenty-seven questions about amblyopia and 28 questions about childhood myopia were asked of the ChatGPT twice (totally 110 questions). The responses were evaluated by two pediatric ophthalmologists as acceptable, incomplete, or unacceptable.

RESULTS

There was remarkable agreement (96.4%) between the two pediatric ophthalmologists on their assessment of the responses. Acceptable responses were provided by the ChatGPT to 93 of 110 (84.6%) questions in total (44 of 54 [81.5%] for amblyopia and 49 of 56 [87.5%] questions for childhood myopia). Seven of 54 (12.9%) responses to questions on amblyopia were graded as incomplete compared to 4 of 56 (7.1%) of questions on childhood myopia. The ChatGPT gave inappropriate responses to three questions about amblyopia (5.6%) and childhood myopia (5.4%). The most noticeable inappropriate responses were related to the definition of reverse amblyopia and the threshold of refractive error for prescription of spectacles to children with myopia.

CONCLUSIONS

The ChatGPT has the potential to serve as an adjunct informational tool for pediatric ophthalmology patients and their caregivers by demonstrating a relatively good performance in answering 84.6% of the most frequently asked questions about amblyopia and childhood myopia. [J Pediatr Ophthalmol Strabismus. 2024;61(2):86-89.].

Artificial intelligence compared with human-derived patient educational materials on cirrhosis

BACKGROUND

The study compared the readability, grade level, understandability, actionability, and accuracy of standard patient educational material against artificial intelligence chatbot-derived patient educational material regarding cirrhosis.

METHODS

An identical standardized phrase was used to generate patient educational materials on cirrhosis from 4 large language model-derived chatbots (ChatGPT, DocsGPT, Google Bard, and Bing Chat), and the outputs were compared against a pre-existing human-derived educational material (Epic). Objective scores for readability and grade level were determined using Flesch-Kincaid and Simple Measure of Gobbledygook scoring systems. 14 patients/caregivers and 8 transplant hepatologists were blinded and independently scored the materials on understandability and actionability and indicated whether they believed the material was human or artificial intelligence-generated. Understandability and actionability were determined using the Patient Education Materials Assessment Tool for Printable Materials. Transplant hepatologists also provided medical accuracy scores.

RESULTS

Most educational materials scored similarly in readability and grade level but were above the desired sixth-grade reading level. All educational materials were deemed understandable by both groups, while only the human-derived educational material (Epic) was considered actionable by both groups. No significant difference in perceived actionability or understandability among the educational materials was identified. Both groups poorly identified which materials were human-derived versus artificial intelligence-derived.

CONCLUSIONS

Chatbot-derived patient educational materials have comparable readability, grade level, understandability, and accuracy to human-derived materials. Readability, grade level, and actionability may be appropriate targets for improvement across educational materials on cirrhosis. Chatbot-derived patient educational materials show promise, and further studies should assess their usefulness in clinical practice.

Evaluating ChatGPT's Capabilities on Orthopedic Training Examinations: An Analysis of New Image Processing Features

Introduction The efficacy of integrating artificial intelligence (AI) models like ChatGPT into the medical field, specifically orthopedic surgery, has yet to be fully determined. The most recent adaptation of ChatGPT that has yet to be explored is its image analysis capabilities. This study assesses ChatGPT's performance in answering Orthopedic In-Training Examination (OITE) questions, including those that require image analysis. Methods Questions from the 2014, 2015, 2021, and 2022 AAOS OITE were screened for inclusion. All questions without images were entered into ChatGPT 3.5 and 4.0 twice. Questions that necessitated the use of images were only entered into ChatGPT 4.0 twice, as this is the only version of the system that can analyze images. The responses were recorded and compared to AAOS's correct answers, evaluating the AI's accuracy and precision. Results A total of 940 questions were included in the final analysis (457 questions with images and 483 questions without images). ChatGPT 4.0 performed significantly better on questions that did not require image analysis (67.81% vs 47.59%, p<0.001). Discussion While the use of AI in orthopedics is an intriguing possibility, this evaluation demonstrates how, even with the addition of image processing capabilities, ChatGPT still falls short in terms of its accuracy. As AI technology evolves, ongoing research is vital to harness AI's potential effectively, ensuring it complements rather than attempts to replace the nuanced skills of orthopedic surgeons.

Appropriateness of Artificial Intelligence Chatbots in Diabetic Foot Ulcer Management

Type 2 diabetes is a significant global health concern. It often causes diabetic foot ulcers (DFUs), which affect millions of people and increase amputation and mortality rates. Despite existing guidelines, the complexity of DFU treatment makes clinical decisions challenging. Large language models such as chat generative pretrained transformer (ChatGPT), which are adept at natural language processing, have emerged as valuable resources in the medical field. However, concerns about the accuracy and reliability of the information they provide remain. We aimed to assess the accuracy of various artificial intelligence (AI) chatbots, including ChatGPT, in providing information on DFUs based on established guidelines. Seven AI chatbots were asked clinical questions (CQs) based on the DFU guidelines. Their responses were analyzed for accuracy in terms of answers to CQs, grade of recommendation, level of evidence, and agreement with the reference, including verification of the authenticity of the references provided by the chatbots. The AI chatbots showed a mean accuracy of 91.2% in answers to CQs, with discrepancies noted in grade of recommendation and level of evidence. Claude-2 outperformed other chatbots in the number of verified references (99.6%), whereas ChatGPT had the lowest rate of reference authenticity (66.3%). This study highlights the potential of AI chatbots as tools for disseminating medical information and demonstrates their high degree of accuracy in answering CQs related to DFUs. However, the variability in the accuracy of these chatbots and problems like AI hallucinations necessitate cautious use and further optimization for medical applications. This study underscores the evolving role of AI in healthcare and the importance of refining these technologies for effective use in clinical decision-making and patient education.

Exploring the role of large language models in radiation emergency response

In recent times, the field of artificial intelligence (AI) has been transformed by the introduction of large language models (LLMs). These models, popularized by OpenAI's GPT-3, have demonstrated the emergent capabilities of AI in comprehending and producing text resembling human language, which has helped them transform several industries. But its role has yet to be explored in the nuclear industry, specifically in managing radiation emergencies. The present work explores LLMs' contextual awareness, natural language interaction, and their capacity to comprehend diverse queries in a radiation emergency response setting. In this study we identify different user types and their specific LLM use-cases in radiation emergencies. Their possible interactions with ChatGPT, a popular LLM, has also been simulated and preliminary results are presented. Drawing on the insights gained from this exercise and to address concerns of reliability and misinformation, this study advocates for expert guided and domain-specific LLMs trained on radiation safety protocols and historical data. This study aims to guide radiation emergency management practitioners and decision-makers in effectively incorporating LLMs into their decision support framework.

Evaluation of Informative Content on Cerebral Palsy in the Era of Artificial Intelligence: The Value of ChatGPT

AIMS

In addition to the popular search engines on the Internet, ChatGPT may provide accurate and reliable health information. The aim of this study was to examine whether ChatGPT's responses to frequently asked questions concerning cerebral palsy (CP) by families were reliable and useful.

METHODS

Google trends were used to find the most frequently searched keywords for CP. Five independent physiatrists assessed ChatGPT responses to 10 questions. Seven-point Likert-type scales were used to rate information reliability and usefulness based on whether the answer can be validated and is understandable.

RESULTS

The median ratings for reliability of information for each question varied from 2 (very unsafe) to 5 (relatively very reliable). The median rating was 4 (reliable) for four questions. The median ratings for usefulness of information varied from 2 (very little useful) to 5 (moderately useful). The median rating was 4 (partly useful) for seven questions.

CONCLUSION

Although ChatGPT appears promising as an additional tool for informing family members of individuals with CP about medical information, it should be emphasized that both consumers and health care providers should be aware of the limitations of artificial intelligence-generated information.

Is ChatGPT a trusted source of information for total hip and knee arthroplasty patients?

Aims

While internet search engines have been the primary information source for patients' questions, artificial intelligence large language models like ChatGPT are trending towards becoming the new primary source. The purpose of this study was to determine if ChatGPT can answer patient questions about total hip (THA) and knee arthroplasty (TKA) with consistent accuracy, comprehensiveness, and easy readability.

Methods

We posed the 20 most Google-searched questions about THA and TKA, plus ten additional postoperative questions, to ChatGPT. Each question was asked twice to evaluate for consistency in quality. Following each response, we responded with, "Please explain so it is easier to understand," to evaluate ChatGPT's ability to reduce response reading grade level, measured as Flesch-Kincaid Grade Level (FKGL). Five resident physicians rated the 120 responses on 1 to 5 accuracy and comprehensiveness scales. Additionally, they answered a "yes" or "no" question regarding acceptability. Mean scores were calculated for each question, and responses were deemed acceptable if ≥ four raters answered "yes."

Results

The mean accuracy and comprehensiveness scores were 4.26 (95% confidence interval (CI) 4.19 to 4.33) and 3.79 (95% CI 3.69 to 3.89), respectively. Out of all the responses, 59.2% (71/120; 95% CI 50.0% to 67.7%) were acceptable. ChatGPT was consistent when asked the same question twice, giving no significant difference in accuracy (t = 0.821; p = 0.415), comprehensiveness (t = 1.387; p = 0.171), acceptability (χ2 = 1.832; p = 0.176), and FKGL (t = 0.264; p = 0.793). There was a significantly lower FKGL (t = 2.204; p = 0.029) for easier responses (11.14; 95% CI 10.57 to 11.71) than original responses (12.15; 95% CI 11.45 to 12.85).

Conclusion

ChatGPT answered THA and TKA patient questions with accuracy comparable to previous reports of websites, with adequate comprehensiveness, but with limited acceptability as the sole information source. ChatGPT has potential for answering patient questions about THA and TKA, but needs improvement.

Large Language Models and Artificial Intelligence: A Primer for Plastic Surgeons on the Demonstrated and Potential Applications, Promises, and Limitations of ChatGPT

BACKGROUND

The rapidly evolving field of artificial intelligence (AI) holds great potential for plastic surgeons. ChatGPT, a recently released AI large language model (LLM), promises applications across many disciplines, including healthcare.

OBJECTIVES

The aim of this article was to provide a primer for plastic surgeons on AI, LLM, and ChatGPT, including an analysis of current demonstrated and proposed clinical applications.

METHODS

A systematic review was performed identifying medical and surgical literature on ChatGPT's proposed clinical applications. Variables assessed included applications investigated, command tasks provided, user input information, AI-emulated human skills, output validation, and reported limitations.

RESULTS

The analysis included 175 articles reporting on 13 plastic surgery applications and 116 additional clinical applications, categorized by field and purpose. Thirty-four applications within plastic surgery are thus proposed, with relevance to different target audiences, including attending plastic surgeons (n = 17, 50%), trainees/educators (n = 8, 24.0%), researchers/scholars (n = 7, 21%), and patients (n = 2, 6%). The 15 identified limitations of ChatGPT were categorized by training data, algorithm, and ethical considerations.

CONCLUSIONS

Widespread use of ChatGPT in plastic surgery will depend on rigorous research of proposed applications to validate performance and address limitations. This systemic review aims to guide research, development, and regulation to safely adopt AI in plastic surgery.

Credibility of ChatGPT in the assessment of obesity in type 2 diabetes according to the guidelines

BACKGROUND

The Chat Generative Pre-trained Transformer (ChatGPT) allows students, researchers, and patients in the medical field to access information easily and has gained attention nowadays. We aimed to evaluate the credibility of ChatGPT according to the guidelines for the assessment of obesity in type 2 diabetes (T2D), which is one of the major concerns of this century.

MATERIALS AND METHOD

In this cross-sectional non-human subject study, experienced endocrinologists posed 20 questions to ChatGPT in subsections, which were assessments and different treatment options for obesity according to the American Diabetes Association and American Association of Clinical Endocrinology guidelines. The responses of ChatGPT were classified into four categories: compatible, compatible but insufficient, partially incompatible and incompatible with the guidelines.

RESULTS

ChatGPT demonstrated a systematic approach to answering questions and recommended consulting a healthcare provider to receive personalized advice based on the specific health needs and circumstances of patients. The compatibility of ChatGPT with the guidelines was 100% in the assessment of obesity in type 2 diabetes; however, it was lower in the therapy sections, which included nutritional, medical, and surgical approaches to weight loss. Furthermore, ChatGPT required additional prompts for responses that were evaluated as "compatible but insufficient" to provide all the information in the guidelines.

CONCLUSION

The assessment and management of obesity in T2D are highly individualized. Despite ChatGPT's comprehensive and understandable responses, it should not be used as a substitute for healthcare professionals' patient-centered approach.

Appropriateness and Reliability of an Online Artificial Intelligence Platform's Responses to Common Questions Regarding Distal Radius Fractures

PURPOSE

Chat Generative Pre-Trained Transformer (ChatGPT) is a novel artificial intelligence chatbot that is changing the way humans gather information online. The purpose of this study was to investigate ChatGPT's ability to appropriately and reliably answer common questions regarding distal radius fractures.

METHODS

Thirty common questions regarding distal radius fractures were presented in an identical manner to the online ChatGPT-3.5 interface three separate times, yielding 90 unique responses because ChatGPT produces an original answer with each query. All responses were graded as "appropriate," "appropriate but incomplete," or "inappropriate" by a consensus discussion among three hand surgeon reviewers. The questions were additionally subcategorized into one of four domains based on Bloom's cognitive learning taxonomy, and descriptive statistics were reported.

RESULTS

Seventy of the 90 total responses (78%) produced by ChatGPT were "appropriate," and 29 of the 30 questions (97%) had at least one response considered appropriate (of the three possible). However, only 17 of the 30 questions (57%) were answered appropriately on all three iterations. The test-retest reliability of ChatGPT was poor with an intraclass correlation coefficient of 0.12. Finally, ChatGPT performed best answering questions requiring lower-order thinking skills (Bloom's levels 1-3) and less well on level 4 questions.

CONCLUSIONS

This study found that although ChatGPT has the capability to answer common questions regarding distal radius fractures, caution should be taken before implementing its use, given ChatGPT's inconsistency in providing a complete and accurate response to the same question every time.

CLINICAL RELEVANCE

As the popularity and technology of ChatGPT continue to grow, it is important to understand the potential and limitations of this platform to determine how it may be best implemented to improve patient care.

Assessing the performance of ChatGPT in bioethics: a large language model's moral compass in medicine

Chat Generative Pre-Trained Transformer (ChatGPT) has been a growing point of interest in medical education yet has not been assessed in the field of bioethics. This study evaluated the accuracy of ChatGPT-3.5 (April 2023 version) in answering text-based, multiple choice bioethics questions at the level of US third-year and fourth-year medical students. A total of 114 bioethical questions were identified from the widely utilised question banks UWorld and AMBOSS. Accuracy, bioethical categories, difficulty levels, specialty data, error analysis and character count were analysed. We found that ChatGPT had an accuracy of 59.6%, with greater accuracy in topics surrounding death and patient-physician relationships and performed poorly on questions pertaining to informed consent. Of all the specialties, it performed best in paediatrics. Yet, certain specialties and bioethical categories were under-represented. Among the errors made, it tended towards content errors and application errors. There were no significant associations between character count and accuracy. Nevertheless, this investigation contributes to the ongoing dialogue on artificial intelligence's (AI) role in healthcare and medical education, advocating for further research to fully understand AI systems' capabilities and constraints in the nuanced field of medical bioethics.

May ChatGPT be a tool producing medical information for common inflammatory bowel disease patients' questions? An evidence-controlled analysis

Artificial intelligence is increasingly entering everyday healthcare. Large language model (LLM) systems such as Chat Generative Pre-trained Transformer (ChatGPT) have become potentially accessible to everyone, including patients with inflammatory bowel diseases (IBD). However, significant ethical issues and pitfalls exist in innovative LLM tools. The hype generated by such systems may lead to unweighted patient trust in these systems. Therefore, it is necessary to understand whether LLMs (trendy ones, such as ChatGPT) can produce plausible medical information (MI) for patients. This review examined ChatGPT's potential to provide MI regarding questions commonly addressed by patients with IBD to their gastroenterologists. From the review of the outputs provided by ChatGPT, this tool showed some attractive potential while having significant limitations in updating and detailing information and providing inaccurate information in some cases. Further studies and refinement of the ChatGPT, possibly aligning the outputs with the leading medical evidence provided by reliable databases, are needed.

Evaluating the performance of ChatGPT in answering questions related to urolithiasis

PURPOSE

ChatGPT is an artificial intelligence (AI) program with natural language processing. We analyzed ChatGPT's knowledge about urolithiasis whether it can be used to inform patients about urolithiasis.

METHODS

Frequently asked questions (FAQs) about urolithiasis on the websites of urological associations and hospitals were analyzed. Also, strong recommendation-level information was gathered from the urolithiasis section of the European Association of Urology (EAU) 2022 Guidelines. All questions were asked in order in ChatGPT August 3rd version. All answers were evaluated separately by two specialist urologists and scored between 1 and 4, where 1: completely correct, 2: correct but inadequate, 3: a mix of correct and misleading information, and 4: completely incorrect.

RESULTS

Of the FAQs, 94.6% were answered completely correctly. No question was answered completely incorrectly. All questions about general, diagnosis, and ureteral stones were graded as 1. Of the 60 questions prepared according to the EAU guideline recommendations, 50 (83.3%) were evaluated as grade 1, and 8 (13.3%) and 2 (3.3%) as grade 3. All questions related to general, diagnostic, renal calculi, ureteral calculi, and metabolic evaluation received the same answer the second time they were asked.

CONCLUSION

Our findings demonstrated that ChatGPT accurately and satisfactorily answered more than 95% of the questions about urolithiasis. We conclude that applying ChatGPT in urology clinics under the supervision of urologists can help patients and their families to have better understanding on urolithiasis diagnosis and treatment.

Assessing the performance of large language models (LLMs) in answering medical questions regarding breast cancer in the Chinese context

Purpose

Large language models (LLMs) are deep learning models designed to comprehend and generate meaningful responses, which have gained public attention in recent years. The purpose of this study is to evaluate and compare the performance of LLMs in answering questions regarding breast cancer in the Chinese context.

Material and Methods

ChatGPT, ERNIE Bot, and ChatGLM were chosen to answer 60 questions related to breast cancer posed by two oncologists. Responses were scored as comprehensive, correct but inadequate, mixed with correct and incorrect data, completely incorrect, or unanswered. The accuracy, length, and readability among answers from different models were evaluated using statistical software.

Results

ChatGPT answered 60 questions, with 40 (66.7%) comprehensive answers and six (10.0%) correct but inadequate answers. ERNIE Bot answered 60 questions, with 34 (56.7%) comprehensive answers and seven (11.7%) correct but inadequate answers. ChatGLM generated 60 answers, with 35 (58.3%) comprehensive answers and six (10.0%) correct but inadequate answers. The differences for chosen accuracy metrics among the three LLMs did not reach statistical significance, but only ChatGPT demonstrated a sense of human compassion. The accuracy of the three models in answering questions regarding breast cancer treatment was the lowest, with an average of 44.4%. ERNIE Bot's responses were significantly shorter compared to ChatGPT and ChatGLM (p < .001 for both). The readability scores of the three models showed no statistical significance.

Conclusions

In the Chinese context, the capabilities of ChatGPT, ERNIE Bot, and ChatGLM are similar in answering breast cancer-related questions at present. These three LLMs may serve as adjunct informational tools for breast cancer patients in the Chinese context, offering guidance for general inquiries. However, for highly specialized issues, particularly in the realm of breast cancer treatment, LLMs cannot deliver reliable performance. It is necessary to utilize them under the supervision of healthcare professionals.

Assessing the potential of ChatGPT for psychodynamic formulations in psychiatry: An exploratory study

Although there were several attempts to apply ChatGPT (Generative Pre-Trained Transformer) to medicine, little is known about therapeutic applications in psychiatry. In this exploratory study, we aimed to evaluate the characteristics and appropriateness of the psychodynamic formulations created by ChatGPT. Along with a case selected from the psychoanalytic literature, input prompts were designed to include different levels of background knowledge. These included naïve prompts, keywords created by ChatGPT, keywords created by psychiatrists, and psychodynamic concepts from the literature. The psychodynamic formulations generated from the different prompts were evaluated by five psychiatrists from different institutions. We next conducted further tests in which instructions on the use of different psychodynamic models were added to the input prompts. The models used were ego psychology, self-psychology, and object relations. The results from naïve prompts and psychodynamic concepts were rated as appropriate by most raters. The psychodynamic concept prompt output was rated the highest. Interrater agreement was statistically significant. The results from the tests using instructions in different psychoanalytic theories were also rated as appropriate by most raters. They included key elements of the psychodynamic formulation and suggested interpretations similar to the literature. These findings suggest potential of ChatGPT for use in psychiatry.

Evaluating the comprehension and accuracy of ChatGPT's responses to diabetes-related questions in Urdu compared to English

Introduction

Patients with diabetes require healthcare and information that are accurate and extensive. Large language models (LLMs) like ChatGPT herald the capacity to provide such exhaustive data. To determine (a) the comprehensiveness of ChatGPT's responses in Urdu to diabetes-related questions and (b) the accuracy of ChatGPT's Urdu responses when compared to its English responses.

Methods

A cross-sectional observational study was conducted. Two reviewers experienced in internal medicine and endocrinology graded 53 Urdu and English responses on diabetes knowledge, lifestyle, and prevention. A senior reviewer resolved discrepancies. Responses were assessed for comprehension and accuracy, then compared to English.

Results

Among the Urdu responses generated, only two of 53 (3.8%) questions were graded as comprehensive, and five of 53 (9.4%) were graded as correct but inadequate. We found that 25 of 53 (47.2%) questions were graded as mixed with correct and incorrect/outdated data, the most significant proportion of responses being graded as such. When considering the comparison of response scale grading the comparative accuracy of Urdu and English responses, no Urdu response (0.0%) was considered to have more accuracy than English. Most of the Urdu responses were found to have an accuracy less than that of English, an overwhelming majority of 49 of 53 (92.5%) responses.

Conclusion

We found that although the ability to retrieve such information about diabetes is impressive, it can merely be used as an adjunct instead of a solitary source of information. Further work must be done to optimize Urdu responses in medical contexts to approximate the boundless potential it heralds.

What Does ChatGPT Know About Dementia? A Comparative Analysis of Information Quality

The quality of information about dementia retrieved using ChatGPT is unknown. Content was evaluated for length, readability, and quality using the QUEST, a validated tool, and compared against online material from three North American organizations. Both sources of information avoided conflicts of interest, supported the patient-physician relationship, and used a balanced tone. Official bodies but not ChatGPT referenced identifiable research and pointed to local resources. Users of ChatGPT are likely to encounter accurate but shallow information about dementia. Recommendations are made for information creators and providers who counsel patients around digital health practices.

Unveiling the ChatGPT phenomenon: Evaluating the consistency and accuracy of endodontic question answers

AIM

Chatbot Generative Pre-trained Transformer (ChatGPT) is a generative artificial intelligence (AI) software based on large language models (LLMs), designed to simulate human conversations and generate novel content based on the training data it has been exposed to. The aim of this study was to evaluate the consistency and accuracy of ChatGPT-generated answers to clinical questions in endodontics, compared to answers provided by human experts.

METHODOLOGY

Ninety-one dichotomous (yes/no) questions were designed and categorized into three levels of difficulty. Twenty questions were randomly selected from each difficulty level. Sixty answers were generated by ChatGPT for each question. Two endodontic experts independently answered the 60 questions. Statistical analysis was performed using the SPSS program to calculate the consistency and accuracy of the answers generated by ChatGPT compared to the experts. Confidence intervals (95%) and standard deviations were used to estimate variability.

RESULTS

The answers generated by ChatGPT showed high consistency (85.44%). No significant differences in consistency were found based on question difficulty. In terms of answer accuracy, ChatGPT achieved an average accuracy of 57.33%. However, significant differences in accuracy were observed based on question difficulty, with lower accuracy for easier questions.

CONCLUSIONS

Currently, ChatGPT is not capable of replacing dentists in clinical decision-making. As ChatGPT's performance improves through deep learning, it is expected to become more useful and effective in the field of endodontics. However, careful attention and ongoing evaluation are needed to ensure its accuracy, reliability and safety in endodontics.

The Genie Is Out of the Bottle: What ChatGPT Can and Cannot Do for Medical Professionals

ChatGPT is a cutting-edge artificial intelligence technology that was released for public use in November 2022. Its rapid adoption has raised questions about capabilities, limitations, and risks. This article presents an overview of ChatGPT, and it highlights the current state of this technology for the medical field. The article seeks to provide a balanced perspective on what the model can and cannot do in three specific domains: clinical practice, research, and medical education. It also provides suggestions on how to optimize the use of this tool.

Large Language Models in Neurosurgery

A large language model (LLM), in the context of natural language processing and artificial intelligence, refers to a sophisticated neural network that has been trained on a massive amount of text data to understand and generate human-like language. These models are typically built on architectures like transformers. The term "large" indicates that the neural network has a significant number of parameters, making it more powerful and capable of capturing complex patterns in language. One notable example of a large language model is ChatGPT. ChatGPT is a large language model developed by OpenAI that uses deep learning techniques to generate human-like text. It can be trained on a variety of tasks, such as language translation, question answering, and text completion. One of the key features of ChatGPT is its ability to understand and respond to natural language inputs. This makes it a powerful tool for generating a wide range of text, including medical reports, surgical notes, and even poetry. Additionally, the model has been trained on a large corpus of text, which allows it to generate text that is both grammatically correct and semantically meaningful. In terms of applications in neurosurgery, ChatGPT can be used to generate detailed and accurate surgical reports, which can be very useful for sharing information about a patient's case with other members of the medical team. Additionally, the model can be used to generate detailed surgical notes, which can be very useful for training and educating residents and medical students. Overall, LLMs have the potential to be a valuable tool in the field of neurosurgery. Indeed, this abstract has been generated by ChatGPT within few seconds. Potential applications and pitfalls of the applications of LLMs are discussed in this paper.

Evaluating the performance of ChatGPT in clinical pharmacy: A comparative study of ChatGPT and clinical pharmacists

AIMS

To evaluate the performance of chat generative pretrained transformer (ChatGPT) in key domains of clinical pharmacy practice, including prescription review, patient medication education, adverse drug reaction (ADR) recognition, ADR causality assessment and drug counselling.

METHODS

Questions and clinical pharmacist's answers were collected from real clinical cases and clinical pharmacist competency assessment. ChatGPT's responses were generated by inputting the same question into the 'New Chat' box of ChatGPT Mar 23 Version. Five licensed clinical pharmacists independently rated these answers on a scale of 0 (Completely incorrect) to 10 (Completely correct). The mean scores of ChatGPT and clinical pharmacists were compared using a paired 2-tailed Student's t-test. The text content of the answers was also descriptively summarized together.

RESULTS

The quantitative results indicated that ChatGPT was excellent in drug counselling (ChatGPT: 8.77 vs. clinical pharmacist: 9.50, P = .0791) and weak in prescription review (5.23 vs. 9.90, P = .0089), patient medication education (6.20 vs. 9.07, P = .0032), ADR recognition (5.07 vs. 9.70, P = .0483) and ADR causality assessment (4.03 vs. 9.73, P = .023). The capabilities and limitations of ChatGPT in clinical pharmacy practice were summarized based on the completeness and accuracy of the answers. ChatGPT revealed robust retrieval, information integration and dialogue capabilities. It lacked medicine-specific datasets as well as the ability for handling advanced reasoning and complex instructions.

CONCLUSIONS

While ChatGPT holds promise in clinical pharmacy practice as a supplementary tool, the ability of ChatGPT to handle complex problems needs further improvement and refinement.

ChatGPT utilization within the building blocks of the healthcare services: A mixed-methods study

Introduction

ChatGPT, as an AI tool, has been introduced in healthcare for various purposes. The objective of the study was to investigate the principal benefits of ChatGPT utilization in healthcare services and to identify potential domains for its expansion within the building blocks of the healthcare industry.

Methods

A comprehensive three-phase study was conducted employing mixed methods. The initial phase comprised a systematic review and thematic analysis of the data. In the subsequent phases, a questionnaire, developed based on the findings from the first phase, was distributed to a sample of eight experts. The objective was to prioritize the benefits and potential expansion domains of ChatGPT in healthcare building blocks, utilizing gray SWARA (Stepwise Weight Assessment Ratio Analysis) and gray MABAC (Multi-Attributive Border Approximation Area Comparison), respectively.

Results

The systematic review yielded 74 studies. A thematic analysis of the data from these studies identified 11 unique themes. In the second phase, employing the gray SWARA method, clinical decision-making (weight: 0.135), medical diagnosis (weight: 0.098), medical procedures (weight: 0.070), and patient-centered care (weight: 0.053) emerged as the most significant benefit of ChatGPT in the healthcare sector. Subsequently, it was determined that ChatGPT demonstrated the highest level of usefulness in the information and infrastructure, information and communication technologies blocks.

Conclusion

The study concluded that, despite the significant benefits of ChatGPT in the clinical domains of healthcare, it exhibits a more pronounced potential for growth within the informational domains of the healthcare industry's building blocks, rather than within the domains of intervention and clinical services.

Harnessing ChatGPT and GPT-4 for evaluating the rheumatology questions of the Spanish access exam to specialized medical training

The emergence of large language models (LLM) with remarkable performance such as ChatGPT and GPT-4, has led to an unprecedented uptake in the population. One of their most promising and studied applications concerns education due to their ability to understand and generate human-like text, creating a multitude of opportunities for enhancing educational practices and outcomes. The objective of this study is twofold: to assess the accuracy of ChatGPT/GPT-4 in answering rheumatology questions from the access exam to specialized medical training in Spain (MIR), and to evaluate the medical reasoning followed by these LLM to answer those questions. A dataset, RheumaMIR, of 145 rheumatology-related questions, extracted from the exams held between 2010 and 2023, was created for that purpose, used as a prompt for the LLM, and was publicly distributed. Six rheumatologists with clinical and teaching experience evaluated the clinical reasoning of the chatbots using a 5-point Likert scale and their degree of agreement was analyzed. The association between variables that could influence the models' accuracy (i.e., year of the exam question, disease addressed, type of question and genre) was studied. ChatGPT demonstrated a high level of performance in both accuracy, 66.43%, and clinical reasoning, median (Q1-Q3), 4.5 (2.33-4.67). However, GPT-4 showed better performance with an accuracy score of 93.71% and a median clinical reasoning value of 4.67 (4.5-4.83). These findings suggest that LLM may serve as valuable tools in rheumatology education, aiding in exam preparation and supplementing traditional teaching methods.

Innovating Personalized Nephrology Care: Exploring the Potential Utilization of ChatGPT

The rapid advancement of artificial intelligence (AI) technologies, particularly machine learning, has brought substantial progress to the field of nephrology, enabling significant improvements in the management of kidney diseases. ChatGPT, a revolutionary language model developed by OpenAI, is a versatile AI model designed to engage in meaningful and informative conversations. Its applications in healthcare have been notable, with demonstrated proficiency in various medical knowledge assessments. However, ChatGPT's performance varies across different medical subfields, posing challenges in nephrology-related queries. At present, comprehensive reviews regarding ChatGPT's potential applications in nephrology remain lacking despite the surge of interest in its role in various domains. This article seeks to fill this gap by presenting an overview of the integration of ChatGPT in nephrology. It discusses the potential benefits of ChatGPT in nephrology, encompassing dataset management, diagnostics, treatment planning, and patient communication and education, as well as medical research and education. It also explores ethical and legal concerns regarding the utilization of AI in medical practice. The continuous development of AI models like ChatGPT holds promise for the healthcare realm but also underscores the necessity of thorough evaluation and validation before implementing AI in real-world medical scenarios. This review serves as a valuable resource for nephrologists and healthcare professionals interested in fully utilizing the potential of AI in innovating personalized nephrology care.

Let's chat about cervical cancer: Assessing the accuracy of ChatGPT responses to cervical cancer questions

OBJECTIVE

To quantify the accuracy of ChatGPT in answering commonly asked questions pertaining to cervical cancer prevention, diagnosis, treatment, and survivorship/quality-of-life (QOL).

METHODS

ChatGPT was queried with 64 questions adapted from professional society websites and the authors' clinical experiences. The answers were scored by two attending Gynecologic Oncologists according to the following scale: 1) correct and comprehensive, 2) correct but not comprehensive, 3) some correct, some incorrect, and 4) completely incorrect. Scoring discrepancies were resolved by additional reviewers as needed. The proportion of responses earning each score were calculated overall and within each question category.

RESULTS

ChatGPT provided correct and comprehensive answers to 34 (53.1%) questions, correct but not comprehensive answers to 19 (29.7%) questions, partially incorrect answers to 10 (15.6%) questions, and completely incorrect answers to 1 (1.6%) question. Prevention and survivorship/QOL had the highest proportion of "correct" scores (scores of 1 or 2) at 22/24 (91.7%) and 15/16 (93.8%), respectively. ChatGPT performed less well in the treatment category, with 15/21 (71.4%) correct scores. It performed the worst in the diagnosis category with only 1/3 (33.3%) correct scores.

CONCLUSION

ChatGPT accurately answers questions about cervical cancer prevention, survivorship, and QOL. It performs less accurately for cervical cancer diagnosis and treatment. Further development of this immensely popular large language model should include physician input before it can be utilized as a tool for Gynecologists or recommended as a patient resource for information on cervical cancer diagnosis and treatment.

Auxiliary use of ChatGPT in surgical diagnosis and treatment

ChatGPT can be used as an auxiliary tool in surgical diagnosis and treatment in several ways. One of the most incredible values of using ChatGPT is its ability to quickly process and handle large amounts of data and provide relatively accurate information to healthcare workers. Due to its high accuracy and ability to process big data, ChatGPT has been widely used in the healthcare industry for tasks such as assisting medical diagnosis, giving predictions of some diseases, and analyzing some medical cases. Surgical diagnosis and treatment can serve as an auxiliary tool to help healthcare professionals. Process large amounts of medical data, provide real-time guidance and feedback, and increase healthcare's overall speed and quality. Although it has great acceptance, it still faces issues such as ethics, patient privacy, data security, law, trustworthiness, and accuracy. This study aimed to explore the auxiliary use of ChatGPT in surgical diagnosis and treatment.

A systematic review and meta-analysis on ChatGPT and its utilization in medical and dental research

Since its release, ChatGPT has taken the world by storm with its utilization in various fields of life. This review's main goal was to offer a thorough and fact-based evaluation of ChatGPT's potential as a tool for medical and dental research, which could direct subsequent research and influence clinical practices.

METHODS

Different online databases were scoured for relevant articles that were in accordance with the study objectives. A team of reviewers was assembled to devise a proper methodological framework for inclusion of articles and meta-analysis.

RESULTS

11 descriptive studies were considered for this review that evaluated the accuracy of ChatGPT in answering medical queries related to different domains such as systematic reviews, cancer, liver diseases, diagnostic imaging, education, and COVID-19 vaccination. The studies reported different accuracy ranges, from 18.3 % to 100 %, across various datasets and specialties. The meta-analysis showed an odds ratio (OR) of 2.25 and a relative risk (RR) of 1.47 with a 95 % confidence interval (CI), indicating that the accuracy of ChatGPT in providing correct responses was significantly higher compared to the total responses for queries. However, significant heterogeneity was present among the studies, suggesting considerable variability in the effect sizes across the included studies.

CONCLUSION

The observations indicate that ChatGPT has the ability to provide appropriate solutions to questions in the medical and dentistry areas, but researchers and doctors should cautiously assess its responses because they might not always be dependable. Overall, the importance of this study rests in shedding light on ChatGPT's accuracy in the medical and dentistry fields and emphasizing the need for additional investigation to enhance its performance. © 2017 Elsevier Inc. All rights reserved.

Evaluating the performance of ChatGPT in answering questions related to benign prostate hyperplasia and prostate cancer

BACKGROUND

The aim of this study was to evaluate the accuracy and reproducibility of ChatGPT's answers to frequently asked questions about benign prostate hyperplasia (BPH) and prostate cancer.

METHODS

Frequently asked questions on the websites of urology associations, hospitals, and social media about prostate cancer and BPH were evaluated. Also, strong recommendation-level data were noted in the recommendations tables of the European Urology Association (EAU) 2022 Guidelines on Prostate Cancer and Management of Non-neurogenic Male Lower Urinary Tract Symptoms sections. All questions were asked in order in ChatGPT Mar 23 Version. All answers were evaluated separately by two specialist urologists and scored between 1-4.

RESULTS

Forty questions about BPH and 86 questions about prostate cancer were included in the study. The answers to all BPH-related questions resulted in 90.0% completely correct. This rate for questions about prostate cancer was 94.2%. The completely correct rate in the questions prepared according to the strong recommendations of the EAU guideline was 77.8% for BPH and 76.2% for prostate cancer. The similarity rates of the answers to the repeated questions were 90.0% and 93% for questions related to BPH and prostate cancer, respectively.

CONCLUSIONS

ChatGPT has given satisfactory answers to questions about BPH and prostate cancer. Although it has limitations, it can be predicted that it will take an important place in the health sector in the future, as it is a constantly evolving platform. ChatGPT was able to provide helpful information about BPH and prostate cancer, although it is not perfect. It is constantly getting better, and may become an important resource in the healthcare field in the future.

Editorial Commentary: Recognizing and Avoiding Medical Misinformation Across Digital Platforms: Smoke, Mirrors (and Streaming)

The evolution of social media and related online sources has substantially increased the ability of patients to query and access publicly available information that may have relevance to a potential musculoskeletal condition of interest. Although increased accessibility to information has several purported benefits, including encouragement of patients to become more invested in their care through self-teaching, a downside to the existence of a vast number of unregulated resources remains the risk of misinformation. As health care providers, we have a moral and ethical obligation to mitigate this risk by directing patients to high-quality resources for medical information and to be aware of resources that are unreliable. To this end, a growing body of evidence has suggested that YouTube lacks reliability and quality in terms of medical information concerning a variety of musculoskeletal conditions.

ChatGPT: promise and challenges for deployment in low- and middle-income countries

In low- and middle-income countries (LMICs), the fields of medicine and public health grapple with numerous challenges that continue to hinder patients' access to healthcare services. ChatGPT, a publicly accessible chatbot, has emerged as a potential tool in aiding public health efforts in LMICs. This viewpoint details the potential benefits of employing ChatGPT in LMICs to improve medicine and public health encompassing a broad spectrum of domains ranging from health literacy, screening, triaging, remote healthcare support, mental health support, multilingual capabilities, healthcare communication and documentation, medical training and education, and support for healthcare professionals. Additionally, we also share potential concerns and limitations associated with the use of ChatGPT and provide a balanced discussion on the opportunities and challenges of using ChatGPT in LMICs.

Novel research and future prospects of artificial intelligence in cancer diagnosis and treatment

Research into the potential benefits of artificial intelligence for comprehending the intricate biology of cancer has grown as a result of the widespread use of deep learning and machine learning in the healthcare sector and the availability of highly specialized cancer datasets. Here, we review new artificial intelligence approaches and how they are being used in oncology. We describe how artificial intelligence might be used in the detection, prognosis, and administration of cancer treatments and introduce the use of the latest large language models such as ChatGPT in oncology clinics. We highlight artificial intelligence applications for omics data types, and we offer perspectives on how the various data types might be combined to create decision-support tools. We also evaluate the present constraints and challenges to applying artificial intelligence in precision oncology. Finally, we discuss how current challenges may be surmounted to make artificial intelligence useful in clinical settings in the future.

Evaluating the Performance of Different Large Language Models on Health Consultation and Patient Education in Urolithiasis

OBJECTIVES

To evaluate the effectiveness of four large language models (LLMs) (Claude, Bard, ChatGPT4, and New Bing) that have large user bases and significant social attention, in the context of medical consultation and patient education in urolithiasis.

MATERIALS AND METHODS

In this study, we developed a questionnaire consisting of 21 questions and 2 clinical scenarios related to urolithiasis. Subsequently, clinical consultations were simulated for each of the four models to assess their responses to the questions. Urolithiasis experts then evaluated the model responses in terms of accuracy, comprehensiveness, ease of understanding, human care, and clinical case analysis ability based on a predesigned 5-point Likert scale. Visualization and statistical analyses were then employed to compare the four models and evaluate their performance.

RESULTS

All models yielded satisfying performance, except for Bard, who failed to provide a valid response to Question 13. Claude consistently scored the highest in all dimensions compared with the other three models. ChatGPT4 ranked second in accuracy, with a relatively stable output across multiple tests, but shortcomings were observed in empathy and human caring. Bard exhibited the lowest accuracy and overall performance. Claude and ChatGPT4 both had a high capacity to analyze clinical cases of urolithiasis. Overall, Claude emerged as the best performer in urolithiasis consultations and education.

CONCLUSION

Claude demonstrated superior performance compared with the other three in urolithiasis consultation and education. This study highlights the remarkable potential of LLMs in medical health consultations and patient education, although professional review, further evaluation, and modifications are still required.

A SWOT (Strengths, Weaknesses, Opportunities, and Threats) Analysis of ChatGPT in the Medical Literature: Concise Review

BACKGROUND

ChatGPT is a 175-billion-parameter natural language processing model that is already involved in scientific content and publications. Its influence ranges from providing quick access to information on medical topics, assisting in generating medical and scientific articles and papers, performing medical data analyses, and even interpreting complex data sets.

OBJECTIVE

The future role of ChatGPT remains uncertain and a matter of debate already shortly after its release. This review aimed to analyze the role of ChatGPT in the medical literature during the first 3 months after its release.

METHODS

We performed a concise review of literature published in PubMed from December 1, 2022, to March 31, 2023. To find all publications related to ChatGPT or considering ChatGPT, the search term was kept simple ("ChatGPT" in AllFields). All publications available as full text in German or English were included. All accessible publications were evaluated according to specifications by the author team (eg, impact factor, publication modus, article type, publication speed, and type of ChatGPT integration or content). The conclusions of the articles were used for later SWOT (strengths, weaknesses, opportunities, and threats) analysis. All data were analyzed on a descriptive basis.

RESULTS

Of 178 studies in total, 160 met the inclusion criteria and were evaluated. The average impact factor was 4.423 (range 0-96.216), and the average publication speed was 16 (range 0-83) days. Among the articles, there were 77 editorials (48,1%), 43 essays (26.9%), 21 studies (13.1%), 6 reviews (3.8%), 6 case reports (3.8%), 6 news (3.8%), and 1 meta-analysis (0.6%). Of those, 54.4% (n=87) were published as open access, with 5% (n=8) provided on preprint servers. Over 400 quotes with information on strengths, weaknesses, opportunities, and threats were detected. By far, most (n=142, 34.8%) were related to weaknesses. ChatGPT excels in its ability to express ideas clearly and formulate general contexts comprehensibly. It performs so well that even experts in the field have difficulty identifying abstracts generated by ChatGPT. However, the time-limited scope and the need for corrections by experts were mentioned as weaknesses and threats of ChatGPT. Opportunities include assistance in formulating medical issues for nonnative English speakers, as well as the possibility of timely participation in the development of such artificial intelligence tools since it is in its early stages and can therefore still be influenced.

CONCLUSIONS

Artificial intelligence tools such as ChatGPT are already part of the medical publishing landscape. Despite their apparent opportunities, policies and guidelines must be implemented to ensure benefits in education, clinical practice, and research and protect against threats such as scientific misconduct, plagiarism, and inaccuracy.

Assessing the Accuracy and Comprehensiveness of ChatGPT in Offering Clinical Guidance for Atopic Dermatitis and Acne Vulgaris

Collapse

Development of a Liver Disease-Specific Large Language Model Chat Interface using Retrieval Augmented Generation

Background

Large language models (LLMs) have significant capabilities in clinical information processing tasks. Commercially available LLMs, however, are not optimized for clinical uses and are prone to generating incorrect or hallucinatory information. Retrieval-augmented generation (RAG) is an enterprise architecture that allows embedding of customized data into LLMs. This approach "specializes" the LLMs and is thought to reduce hallucinations.

Methods

We developed "LiVersa," a liver disease-specific LLM, by using our institution's protected health information (PHI)-complaint text embedding and LLM platform, "Versa." We conducted RAG on 30 publicly available American Association for the Study of Liver Diseases (AASLD) guidelines and guidance documents to be incorporated into LiVersa. We evaluated LiVersa's performance by comparing its responses versus those of trainees from a previously published knowledge assessment study regarding hepatitis B (HBV) treatment and hepatocellular carcinoma (HCC) surveillance.

Results

LiVersa answered all 10 questions correctly when forced to provide a "yes" or "no" answer. Full detailed responses with justifications and rationales, however, were not completely correct for three of the questions.

Discussions

In this study, we demonstrated the ability to build disease-specific and PHI-compliant LLMs using RAG. While our LLM, LiVersa, demonstrated more specificity in answering questions related to clinical hepatology - there were some knowledge deficiencies due to limitations set by the number and types of documents used for RAG. The LiVersa prototype, however, is a proof of concept for utilizing RAG to customize LLMs for clinical uses and a potential strategy to realize personalized medicine in the future.

The Role of ChatGPT in the Advancement of Diagnosis, Management, and Prognosis of Cardiovascular and Cerebrovascular Disease

Cardiovascular and cerebrovascular disease incidence has risen mainly due to poor control of preventable risk factors and still constitutes a significant financial and health burden worldwide. ChatGPT is an artificial intelligence language-based model developed by OpenAI. Due to the model's unique cognitive capabilities beyond data processing and the production of high-quality text, there has been a surge of research interest concerning its role in the scientific community and contemporary clinical practice. To fully exploit ChatGPT's potential benefits and reduce its possible misuse, extreme caution must be taken to ensure its implications ethically and equitably. In this narrative review, we explore the language model's possible applications and limitations while emphasizing its potential value for diagnosing, managing, and prognosis of cardiovascular and cerebrovascular disease.

AI Chatbots in Clinical Laboratory Medicine: Foundations and Trends

BACKGROUND

Artificial intelligence (AI) conversational agents, or chatbots, are computer programs designed to simulate human conversations using natural language processing. They offer diverse functions and applications across an expanding range of healthcare domains. However, their roles in laboratory medicine remain unclear, as their accuracy, repeatability, and ability to interpret complex laboratory data have yet to be rigorously evaluated.

CONTENT

This review provides an overview of the history of chatbots, two major chatbot development approaches, and their respective advantages and limitations. We discuss the capabilities and potential applications of chatbots in healthcare, focusing on the laboratory medicine field. Recent evaluations of chatbot performance are presented, with a special emphasis on large language models such as the Chat Generative Pre-trained Transformer in response to laboratory medicine questions across different categories, such as medical knowledge, laboratory operations, regulations, and interpretation of laboratory results as related to clinical context. We analyze the causes of chatbots' limitations and suggest research directions for developing more accurate, reliable, and manageable chatbots for applications in laboratory medicine.

SUMMARY

Chatbots, which are rapidly evolving AI applications, hold tremendous potential to improve medical education, provide timely responses to clinical inquiries concerning laboratory tests, assist in interpreting laboratory results, and facilitate communication among patients, physicians, and laboratorians. Nevertheless, users should be vigilant of existing chatbots' limitations, such as misinformation, inconsistencies, and lack of human-like reasoning abilities. To be effectively used in laboratory medicine, chatbots must undergo extensive training on rigorously validated medical knowledge and be thoroughly evaluated against standard clinical practice.

Exploring the potential of ChatGPT as a supplementary tool for providing orthopaedic information

PURPOSE

To investigate the potential use of large language models (LLMs) in orthopaedics by presenting queries pertinent to anterior cruciate ligament (ACL) surgery to generative pre-trained transformer (ChatGPT, specifically using its GPT-4 model of March 14th 2023). Additionally, this study aimed to evaluate the depth of the LLM's knowledge and investigate its adaptability to different user groups. It was hypothesized that the ChatGPT would be able to adapt to different target groups due to its strong language understanding and processing capabilities.

METHODS

ChatGPT was presented with 20 questions and response was requested for two distinct target audiences: patients and non-orthopaedic medical doctors. Two board-certified orthopaedic sports medicine surgeons and two expert orthopaedic sports medicine surgeons independently evaluated the responses generated by ChatGPT. Mean correctness, completeness, and adaptability to the target audiences (patients and non-orthopaedic medical doctors) were determined. A three-point response scale facilitated nuanced assessment.

RESULTS

ChatGPT exhibited fair accuracy, with average correctness scores of 1.69 and 1.66 (on a scale from 0, incorrect, 1, partially correct, to 2, correct) for patients and medical doctors, respectively. Three of the 20 questions (15.0%) were deemed incorrect by any of the four orthopaedic sports medicine surgeon assessors. Moreover, overall completeness was calculated to be 1.51 and 1.64 for patients and medical doctors, respectively, while overall adaptiveness was determined to be 1.75 and 1.73 for patients and doctors, respectively.

CONCLUSION

Overall, ChatGPT was successful in generating correct responses in approximately 65% of the cases related to ACL surgery. The findings of this study imply that LLMs offer potential as a supplementary tool for acquiring orthopaedic knowledge. However, although ChatGPT can provide guidance and effectively adapt to diverse target audiences, it cannot supplant the expertise of orthopaedic sports medicine surgeons in diagnostic and treatment planning endeavours due to its limited understanding of orthopaedic domains and its potential for erroneous responses.

LEVEL OF EVIDENCE

V.

Breaking barriers: can ChatGPT compete with a shoulder and elbow specialist in diagnosis and management?

Background

ChatGPT is an artificial intelligence (AI) language processing model that uses deep learning to generate human-like responses to natural language inputs. Its potential use in health care has raised questions and several studies have assessed its effectiveness in writing articles, clinical reasoning, and solving complex questions. This study aims to investigate ChatGPT's capabilities and implications in diagnosing and managing patients with new shoulder and elbow complaints in a private clinical setting to provide insights into its potential use as a diagnostic tool for patients and a first consultation resource for primary physicians.

Methods

In a private clinical setting, patients were assessed by ChatGPT after being seen by a shoulder and elbow specialist for shoulder and elbow symptoms. To be assessed by the AI model, a research fellow filled out a standardized form (including age, gender, major comorbidities, symptoms and the localization, natural history, and duration, any associated symptoms or movement deficit, aggravating/relieving factors, and x-ray/imaging report if present). This form was submitted through the ChatGPT portal and the AI model was asked for a diagnosis and best management modality.

Results

A total of 29 patients with 15 males and 14 females, were included in this study. The AI model was able to correctly choose the diagnosis and management in 93% (27/29) and 83% (24/29) of the patients, respectively. Furthermore, of the remaining 24 patients that were managed correctly, ChatGPT did not specify the appropriate management in 6 patients and chose only one management in 5 patients, where both were applicable and dependent on the patient's choice. Therefore, 55% of ChatGPT's management was poor.

Conclusion

ChatGPT made a worthy opponent; however, it will not be able to replace in its current form a shoulder and elbow specialist in diagnosing and treating patients for many reasons such as misdiagnosis, poor management, lack of empathy and interactions with patients, its dependence on magnetic resonance imaging reports, and its lack of new knowledge.

Assessing the Performance of Chat Generative Pretrained Transformer (ChatGPT) in Answering Andrology-Related Questions

OBJECTIVE

The internet and social media have become primary sources of health information, with men frequently turning to these platforms before seeking professional help. Chat generative pretrained transformer (ChatGPT), an artificial intelligence model developed by OpenAI, has gained popularity as a natural language processing program. The present study evaluated the accuracy and reproducibility of ChatGPT's responses to andrology-related questions.

METHODS

The study analyzed frequently asked andrology questions from health forums, hospital websites, and social media platforms like YouTube and Instagram. Questions were categorized into topics like male hypogonadism, erectile dysfunction, etc. The European Association of Urology (EAU) guideline recommendations were also included. These questions were input into ChatGPT, and responses were evaluated by 3 experienced urologists who scored them on a scale of 1 to 4.

RESULTS

Out of 136 evaluated questions, 108 met the criteria. Of these, 87.9% received correct and adequate answers, 9.3% were correct but insufficient, and 3 responses contained both correct and incorrect information. No question was answered completely wrong. The highest correct answer rates were for disorders of ejaculation, penile curvature, and male hypogonadism. The EAU guideline-based questions achieved a correctness rate of 86.3%. The reproducibility of the answers was over 90%.

CONCLUSION

The study found that ChatGPT provided accurate and reliable answers to over 80% of andrology-related questions. While limitations exist, such as potential outdated data and inability to understand emotional aspects, ChatGPT's potential in the health-care sector is promising. Collaborating with health-care professionals during artificial intelligence model development could enhance its reliability.

Overview of Chatbots with special emphasis on artificial intelligence-enabled ChatGPT in medical science

The release of ChatGPT has initiated new thinking about AI-based Chatbot and its application and has drawn huge public attention worldwide. Researchers and doctors have started thinking about the promise and application of AI-related large language models in medicine during the past few months. Here, the comprehensive review highlighted the overview of Chatbot and ChatGPT and their current role in medicine. Firstly, the general idea of Chatbots, their evolution, architecture, and medical use are discussed. Secondly, ChatGPT is discussed with special emphasis of its application in medicine, architecture and training methods, medical diagnosis and treatment, research ethical issues, and a comparison of ChatGPT with other NLP models are illustrated. The article also discussed the limitations and prospects of ChatGPT. In the future, these large language models and ChatGPT will have immense promise in healthcare. However, more research is needed in this direction.

ChatGPT and mycosis- a new weapon in the knowledge battlefield

As current trend for physician tools, ChatGPT can sift through massive amounts of information and solve problems through easy-to-understand conversations, ultimately improving efficiency. Mycosis is currently facing great challenges, including high fungal burdens, high mortality, limited choice of antifungal drugs and increasing drug resistance. To address these challenges, We asked ChatGPT for fungal infection scenario-based questions and assessed its appropriateness, consistency, and potential pitfalls. We concluded ChatGPT can provide compelling responses to most prompts, including diagnosis, recommendations for examination, treatment and rational drug use. Moreover, we summarized exciting future applications in mycosis, such as clinical work, scientific research, education and healthcare. However, the largest barriers to implementation are deficits in indiviudal advice, timely literature updates, consistency, accuracy and data safety. To fully embrace the opportunity, we need to address these barriers and manage the risks. We expect that ChatGPT will become a new weapon in in the battlefield of mycosis.

Appropriateness and Readability of ChatGPT-4-Generated Responses for Surgical Treatment of Retinal Diseases

OBJECTIVE

To evaluate the appropriateness and readability of the medical knowledge provided by ChatGPT-4, an artificial intelligence-powered conversational search engine, regarding common vitreoretinal surgeries for retinal detachments (RDs), macular holes (MHs), and epiretinal membranes (ERMs).

DESIGN

Retrospective cross-sectional study.

SUBJECTS

This study did not involve any human participants.

METHODS

We created lists of common questions about the definition, prevalence, visual impact, diagnostic methods, surgical and nonsurgical treatment options, postoperative information, surgery-related complications, and visual prognosis of RD, MH, and ERM, and asked each question 3 times on the online ChatGPT-4 platform. The data for this cross-sectional study were recorded on April 25, 2023. Two independent retina specialists graded the appropriateness of the responses. Readability was assessed using Readable, an online readability tool.

MAIN OUTCOME MEASURES

The "appropriateness" and "readability" of the answers generated by ChatGPT-4 bot.

RESULTS

Responses were consistently appropriate in 84.6% (33/39), 92% (23/25), and 91.7% (22/24) of the questions related to RD, MH, and ERM, respectively. Answers were inappropriate at least once in 5.1% (2/39), 8% (2/25), and 8.3% (2/24) of the respective questions. The average Flesch Kincaid Grade Level and Flesch Reading Ease Score were 14.1 ± 2.6 and 32.3 ± 10.8 for RD, 14 ± 1.3 and 34.4 ± 7.7 for MH, and 14.8 ± 1.3 and 28.1 ± 7.5 for ERM. These scores indicate that the answers are difficult or very difficult to read for the average lay person and college graduation would be required to understand the material.

CONCLUSIONS

Most of the answers provided by ChatGPT-4 were consistently appropriate. However, ChatGPT and other natural language models in their current form are not a source of factual information. Improving the credibility and readability of responses, especially in specialized fields, such as medicine, is a critical focus of research. Patients, physicians, and laypersons should be advised of the limitations of these tools for eye- and health-related counseling.

FINANCIAL DISCLOSURE(S)

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Evaluating the Efficacy of ChatGPT as a Valuable Resource for Pharmacology Studies in Traditional and Complementary Medicine (T&CM) Education

Artificial intelligence (AI) is gaining increasing prominence in the field of education, yet comprehensive investigations into its underlying patterns, research limitations, and potential applications remain scarce. ChatGPT, an AI-powered platform developed by the AI research and deployment company OpenAI, allows users to input text instructions and receive prompt textual responses based on its machine learning-driven interactions with online information sources. This study aims to assess the efficacy of ChatGPT in addressing student-centered medical inquiries pertaining to pharmacology, thereby examining its relevance as a self-study resource to enhance the learning experiences of students. Specifically, the study encompasses various domains of pharmacology, such as pharmacokinetics, mechanism of action, clinical uses, adverse effects, contraindications, and drug-drug interactions. The findings demonstrate that ChatGPT provides pertinent and accurate answers to these questions.

Expanding horizons and navigating challenges for enhanced clinical workflows: ChatGPT in urology

Purpose of review

ChatGPT has emerged as a potential tool for facilitating doctors' workflows. However, when it comes to applying these findings within a urological context, there have not been many studies. Thus, our objective was rooted in analyzing the pros and cons of ChatGPT use and how it can be exploited and used by urologists.

Recent findings

ChatGPT can facilitate clinical documentation and note-taking, patient communication and support, medical education, and research. In urology, it was proven that ChatGPT has the potential as a virtual healthcare aide for benign prostatic hyperplasia, an educational and prevention tool on prostate cancer, educational support for urological residents, and as an assistant in writing urological papers and academic work. However, several concerns about its exploitation are presented, such as lack of web crawling, risk of accidental plagiarism, and concerns about patients-data privacy.

Summary

The existing limitations mediate the need for further improvement of ChatGPT, such as ensuring the privacy of patient data and expanding the learning dataset to include medical databases, and developing guidance on its appropriate use. Urologists can also help by conducting studies to determine the effectiveness of ChatGPT in urology in clinical scenarios and nosologies other than those previously listed.