1
|
Nwachukwu BU, Varady NH, Allen AA, Dines JS, Altchek DW, Williams RJ, Kunze KN. Currently Available Large Language Models Do Not Provide Musculoskeletal Treatment Recommendations That Are Concordant With Evidence-Based Clinical Practice Guidelines. Arthroscopy 2024:S0749-8063(24)00575-9. [PMID: 39173690 DOI: 10.1016/j.arthro.2024.07.040] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/09/2024] [Revised: 07/26/2024] [Accepted: 07/29/2024] [Indexed: 08/24/2024]
Abstract
PURPOSE To determine whether several leading, commercially available large language models (LLMs) provide treatment recommendations concordant with evidence-based clinical practice guidelines (CPGs) developed by the American Academy of Orthopaedic Surgeons (AAOS). METHODS All CPGs concerning the management of rotator cuff tears (n = 33) and anterior cruciate ligament injuries (n = 15) were extracted from the AAOS. Treatment recommendations from Chat-Generative Pretrained Transformer version 4 (ChatGPT-4), Gemini, Mistral-7B, and Claude-3 were graded by 2 blinded physicians as being concordant, discordant, or indeterminate (i.e., neutral response without definitive recommendation) with respect to AAOS CPGs. The overall concordance between LLM and AAOS recommendations was quantified, and the comparative overall concordance of recommendations among the 4 LLMs was evaluated through the Fisher exact test. RESULTS Overall, 135 responses (70.3%) were concordant, 43 (22.4%) were indeterminate, and 14 (7.3%) were discordant. Inter-rater reliability for concordance classification was excellent (κ = 0.92). Concordance with AAOS CPGs was most frequently observed with ChatGPT-4 (n = 38, 79.2%) and least frequently observed with Mistral-7B (n = 28, 58.3%). Indeterminate recommendations were most frequently observed with Mistral-7B (n = 17, 35.4%) and least frequently observed with Claude-3 (n = 8, 6.7%). Discordant recommendations were most frequently observed with Gemini (n = 6, 12.5%) and least frequently observed with ChatGPT-4 (n = 1, 2.1%). Overall, no statistically significant difference in concordant recommendations was observed across LLMs (P = .12). Of all recommendations, only 20 (10.4%) were transparent and provided references with full bibliographic details or links to specific peer-reviewed content to support recommendations. CONCLUSIONS Among leading commercially available LLMs, more than 1-in-4 recommendations concerning the evaluation and management of rotator cuff and anterior cruciate ligament injuries do not reflect current evidence-based CPGs. Although ChatGPT-4 showed the highest performance, clinically significant rates of recommendations without concordance or supporting evidence were observed. Only 10% of responses by LLMs were transparent, precluding users from fully interpreting the sources from which recommendations were provided. CLINICAL RELEVANCE Although leading LLMs generally provide recommendations concordant with CPGs, a substantial error rate exists, and the proportion of recommendations that do not align with these CPGs suggests that LLMs are not trustworthy clinical support tools at this time. Each off-the-shelf, closed-source LLM has strengths and weaknesses. Future research should evaluate and compare multiple LLMs to avoid bias associated with narrow evaluation of few models as observed in the current literature.
Collapse
Affiliation(s)
- Benedict U Nwachukwu
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A
| | - Nathan H Varady
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A
| | - Answorth A Allen
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A
| | - Joshua S Dines
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A
| | - David W Altchek
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A
| | - Riley J Williams
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A
| | - Kyle N Kunze
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A..
| |
Collapse
|
2
|
Laterza V, Marchegiani F, Aisoni F, Ammendola M, Schena CA, Lavazza L, Ravaioli C, Carra MC, Costa V, De Franceschi A, De Simone B, de’Angelis N. Smart Operating Room in Digestive Surgery: A Narrative Review. Healthcare (Basel) 2024; 12:1530. [PMID: 39120233 PMCID: PMC11311806 DOI: 10.3390/healthcare12151530] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2024] [Revised: 07/24/2024] [Accepted: 07/29/2024] [Indexed: 08/10/2024] Open
Abstract
The introduction of new technologies in current digestive surgical practice is progressively reshaping the operating room, defining the fourth surgical revolution. The implementation of black boxes and control towers aims at streamlining workflow and reducing surgical error by early identification and analysis, while augmented reality and artificial intelligence augment surgeons' perceptual and technical skills by superimposing three-dimensional models to real-time surgical images. Moreover, the operating room architecture is transitioning toward an integrated digital environment to improve efficiency and, ultimately, patients' outcomes. This narrative review describes the most recent evidence regarding the role of these technologies in transforming the current digestive surgical practice, underlining their potential benefits and drawbacks in terms of efficiency and patients' outcomes, as an attempt to foresee the digestive surgical practice of tomorrow.
Collapse
Affiliation(s)
- Vito Laterza
- Department of Digestive Surgical Oncology and Liver Transplantation, University Hospital of Besançon, 3 Boulevard Alexandre Fleming, 25000 Besancon, France;
| | - Francesco Marchegiani
- Unit of Colorectal and Digestive Surgery, DIGEST Department, Beaujon University Hospital, AP-HP, University of Paris Cité, Clichy, 92110 Paris, France
| | - Filippo Aisoni
- Unit of Emergency Surgery, Department of Surgery, Ferrara University Hospital, 44124 Ferrara, Italy;
| | - Michele Ammendola
- Digestive Surgery Unit, Health of Science Department, University Hospital “R.Dulbecco”, 88100 Catanzaro, Italy;
| | - Carlo Alberto Schena
- Unit of Robotic and Minimally Invasive Surgery, Department of Surgery, Ferrara University Hospital, 44124 Ferrara, Italy; (C.A.S.); (N.d.)
| | - Luca Lavazza
- Hospital Network Coordinator of Azienda Ospedaliero, Universitaria and Azienda USL di Ferrara, 44121 Ferrara, Italy;
| | - Cinzia Ravaioli
- Azienda Ospedaliero, Universitaria di Ferrara, 44121 Ferrara, Italy;
| | - Maria Clotilde Carra
- Rothschild Hospital (AP-HP), 75012 Paris, France;
- INSERM-Sorbonne Paris Cité, Epidemiology and Statistics Research Centre, 75004 Paris, France
| | - Vittore Costa
- Unit of Orthopedics, Humanitas Hospital, 24125 Bergamo, Italy;
| | | | - Belinda De Simone
- Department of Emergency Surgery, Academic Hospital of Villeneuve St Georges, 91560 Villeneuve St. Georges, France;
| | - Nicola de’Angelis
- Unit of Robotic and Minimally Invasive Surgery, Department of Surgery, Ferrara University Hospital, 44124 Ferrara, Italy; (C.A.S.); (N.d.)
- Department of Translational Medicine, University of Ferrara, 44121 Ferrara, Italy
| |
Collapse
|
3
|
Jacobs T, Shaari A, Gazonas CB, Ziccardi VB. Is ChatGPT an Accurate and Readable Patient Aid for Third Molar Extractions? J Oral Maxillofac Surg 2024:S0278-2391(24)00587-1. [PMID: 39019079 DOI: 10.1016/j.joms.2024.06.177] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2024] [Revised: 06/24/2024] [Accepted: 06/25/2024] [Indexed: 07/19/2024]
Abstract
BACKGROUND Artificial intelligence (AI) platforms such as Chat Generative Pre-Trained Transformer (ChatGPT) (Open AI, San Francisco, California, USA) have the capacity to answer health-related questions. It remains unknown whether AI can be a patient-friendly and accurate resource regarding third molar extraction. PURPOSE The purpose was to determine the accuracy and readability of AI responses to common patient questions regarding third molar extraction. STUDY DESIGN, SETTING, SAMPLE This is a cross sectional in-silico assessment of readability and soundness of a computer-generated report. INDEPENDENT VARIABLE Not applicable. MAIN OUTCOME VARIABLES Accuracy, or the ability to provide clinically correct and relevant information, was determined subjectively by 2 reviewers using a 5-point Likert scale, and objectively by comparing responses to American Association of Oral and Maxillofacial Surgeons (AAOMS) clinical consensus papers. Readability, or how easy a piece of text is to read, was assessed using the Flesch Kincaid Reading Ease (FKRE) and Flesch Kincaid Grade Level (FKGL). Both assess readability based on mean number of syllables per word, and words per sentence. To be deemed readable, FKRE should be >60 and FKGL should be <8. COVARIATES Not applicable. ANALYSES Descriptive statistics were used to analyze the findings of this study. RESULTS AI-generated responses above the recommended level for the average patient (FKRE: 52; FKGL: 10). The average Likert score was 4.36, suggesting that most responses were accurate with minor inaccuracies or missing information. AI correctly deferred to the provider in instances where no definitive answer exists. Of the responses that addressed content in AAOMS consensus papers, 18/19 responses closely aligned with them. All prompts did not provide citations or references. CONCLUSION AND RELEVANCE AI was able to provide mostly accurate responses, and content was closely aligned with AAOMS guidelines. However, responses were too complex for the average third molar extraction patient, and were deficient in citations and references. It is important for providers to educate patients on the utility of AI, and to decide whether to recommend using it for information. Ultimately, the best resource for answers is from the practitioners themselves because the AI platform lacks clinical experience.
Collapse
Affiliation(s)
- Tyler Jacobs
- Resident, Department of Oral and Maxillofacial Surgery, Rutgers School of Dental Medicine, Newark, NJ.
| | - Ariana Shaari
- Candidate, Rutgers New Jersey Medical School, Newark, NJ
| | | | - Vincent B Ziccardi
- Professor, Chair, and Associate Dean for Hospital Affairs, Department of Oral and Maxillofacial Surgery, Rutgers School of Dental Medicine, Newark, NJ
| |
Collapse
|
4
|
Johns WL, Martinazzi BJ, Miltenberg B, Nam HH, Hammoud S. ChatGPT Provides Unsatisfactory Responses to Frequently Asked Questions Regarding Anterior Cruciate Ligament Reconstruction. Arthroscopy 2024; 40:2067-2079.e1. [PMID: 38311261 DOI: 10.1016/j.arthro.2024.01.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/23/2023] [Revised: 01/01/2024] [Accepted: 01/08/2024] [Indexed: 02/10/2024]
Abstract
PURPOSE To determine whether the free online artificial intelligence platform ChatGPT could accurately, adequately, and appropriately answer questions regarding anterior cruciate ligament (ACL) reconstruction surgery. METHODS A list of 10 questions about ACL surgery was created based on a review of frequently asked questions that appeared on websites of various orthopaedic institutions. Each question was separately entered into ChatGPT (version 3.5), and responses were recorded, scored, and graded independently by 3 authors. The reading level of the ChatGPT response was calculated using the WordCalc software package, and readability was assessed using the Flesch-Kincaid grade level, Simple Measure of Gobbledygook index, Coleman-Liau index, Gunning fog index, and automated readability index. RESULTS Of the 10 frequently asked questions entered into ChatGPT, 6 were deemed as unsatisfactory and requiring substantial clarification; 1, as adequate and requiring moderate clarification; 1, as adequate and requiring minor clarification; and 2, as satisfactory and requiring minimal clarification. The mean DISCERN score was 41 (inter-rater reliability, 0.721), indicating the responses to the questions were average. According to the readability assessments, a full understanding of the ChatGPT responses required 13.4 years of education, which corresponds to the reading level of a college sophomore. CONCLUSIONS Most of the ChatGPT-generated responses were outdated and failed to provide an adequate foundation for patients' understanding regarding their injury and treatment options. The reading level required to understand the responses was too advanced for some patients, leading to potential misunderstanding and misinterpretation of information. ChatGPT lacks the ability to differentiate and prioritize information that is presented to patients. CLINICAL RELEVANCE Recognizing the shortcomings in artificial intelligence platforms may equip surgeons to better set expectations and provide support for patients considering and preparing for ACL reconstruction.
Collapse
Affiliation(s)
- William L Johns
- Rothman Orthopaedic Institute, Thomas Jefferson University Hospital, Philadelphia, Pennsylvania, U.S.A
| | - Brandon J Martinazzi
- Rothman Orthopaedic Institute, Thomas Jefferson University Hospital, Philadelphia, Pennsylvania, U.S.A..
| | - Benjamin Miltenberg
- Rothman Orthopaedic Institute, Thomas Jefferson University Hospital, Philadelphia, Pennsylvania, U.S.A
| | - Hannah H Nam
- Penn State College of Medicine, Hershey, Pennsylvania, U.S.A
| | - Sommer Hammoud
- Rothman Orthopaedic Institute, Thomas Jefferson University Hospital, Philadelphia, Pennsylvania, U.S.A
| |
Collapse
|
5
|
Levin JM, Lorentz SG, Hurley ET, Lee J, Throckmorton TW, Garrigues GE, MacDonald P, Anakwenze O, Schoch BS, Klifto C. Artificial intelligence in shoulder and elbow surgery: overview of current and future applications. J Shoulder Elbow Surg 2024; 33:1633-1641. [PMID: 38430978 DOI: 10.1016/j.jse.2024.01.033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/22/2023] [Revised: 01/10/2024] [Accepted: 01/14/2024] [Indexed: 03/05/2024]
Abstract
Artificial intelligence (AI) is amongst the most rapidly growing technologies in orthopedic surgery. With the exponential growth in healthcare data, computing power, and complex predictive algorithms, this technology is poised to aid providers in data processing and clinical decision support throughout the continuum of orthopedic care. Understanding the utility and limitations of this technology is vital to practicing orthopedic surgeons, as these applications will become more common place in everyday practice. AI has already demonstrated its utility in shoulder and elbow surgery for imaging-based diagnosis, predictive modeling of clinical outcomes, implant identification, and automated image segmentation. The future integration of AI and robotic surgery represents the largest potential application of AI in shoulder and elbow surgery with the potential for significant clinical and financial impact. This editorial's purpose is to summarize common AI terms, provide a framework to understand and interpret AI model results, and discuss current applications and future directions within shoulder and elbow surgery.
Collapse
Affiliation(s)
- Jay M Levin
- Department of Orthopaedic Surgery, Duke University Medical Center, Durham, NC, USA.
| | - Samuel G Lorentz
- Department of Orthopaedic Surgery, Duke University Medical Center, Durham, NC, USA
| | - Eoghan T Hurley
- Department of Orthopaedic Surgery, Duke University Medical Center, Durham, NC, USA
| | - Julia Lee
- Department of Orthopedic Surgery, Sierra Pacific Orthopedics, Fresno, CA, USA
| | - Thomas W Throckmorton
- Department of Orthopaedic Surgery, University of Tennessee-Campbell Clinic, Germantown, TN, USA
| | | | - Peter MacDonald
- Section of Orthopaedic Surgery & The Pan Am Clinic, University of Manitoba, Winnipeg, MB, Canada
| | - Oke Anakwenze
- Department of Orthopaedic Surgery, Duke University Medical Center, Durham, NC, USA
| | - Bradley S Schoch
- Department of Orthopedic Surgery, Mayo Clinic, Jacksonville, FL, USA
| | - Christopher Klifto
- Department of Orthopaedic Surgery, Duke University Medical Center, Durham, NC, USA
| |
Collapse
|
6
|
Hurley ET, Crook BS, Dickens JF. Editorial Commentary: At Present, ChatGPT Cannot Be Relied Upon to Answer Patient Questions and Requires Physician Expertise to Interpret Answers for Patients. Arthroscopy 2024; 40:2080-2082. [PMID: 38484923 DOI: 10.1016/j.arthro.2024.02.039] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/25/2024] [Accepted: 02/28/2024] [Indexed: 04/14/2024]
Abstract
ChatGPT is designed to provide accurate and reliable information to the best of its abilities based on the data input and knowledge available. Thus, ChatGPT is being studied as a patient information tool. This artificial intelligence (AI) tool has been shown to frequently provide technically correct information but with limitations. ChatGPT provides different answers to similar questions based on the prompts, and patients may not have expertise in prompting ChatGPT to elicit a best answer. (Prompting large language models has been shown to be a skill that can improve.) Of greater concern, ChatGPT fails to provide sources or references for its answers. At present, ChatGPT cannot be relied upon to address patient questions; in the future, ChatGPT will improve. Today, AI requires physician expertise to interpret AI answers for patients.
Collapse
|
7
|
Kunze KN, Varady NH, Mazzucco M, Lu AZ, Chahla J, Martin RK, Ranawat AS, Pearle AD, Williams RJ. The Large Language Model ChatGPT-4 Exhibits Excellent Triage Capabilities and Diagnostic Performance for Patients Presenting With Various Causes of Knee Pain. Arthroscopy 2024:S0749-8063(24)00456-0. [PMID: 38925234 DOI: 10.1016/j.arthro.2024.06.021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/25/2024] [Revised: 05/28/2024] [Accepted: 06/14/2024] [Indexed: 06/28/2024]
Abstract
PURPOSE To provide a proof-of-concept analysis of the appropriateness and performance of ChatGPT-4 to triage, synthesize differential diagnoses, and generate treatment plans concerning common presentations of knee pain. METHODS Twenty knee complaints warranting triage and expanded scenarios were input into ChatGPT-4, with memory cleared prior to each new input to mitigate bias. For the 10 triage complaints, ChatGPT-4 was asked to generate a differential diagnosis that was graded for accuracy and suitability in comparison to a differential created by 2 orthopaedic sports medicine physicians. For the 10 clinical scenarios, ChatGPT-4 was prompted to provide treatment guidance for the patient, which was again graded. To test the higher-order capabilities of ChatGPT-4, further inquiry into these specific management recommendations was performed and graded. RESULTS All ChatGPT-4 diagnoses were deemed appropriate within the spectrum of potential pathologies on a differential. The top diagnosis on the differential was identical between surgeons and ChatGPT-4 for 70% of scenarios, and the top diagnosis provided by the surgeon appeared as either the first or second diagnosis in 90% of scenarios. Overall, 16 of 30 diagnoses (53.3%) in the differential were identical. When provided with 10 expanded vignettes with a single diagnosis, the accuracy of ChatGPT-4 increased to 100%, with the suitability of management graded as appropriate in 90% of cases. Specific information pertaining to conservative management, surgical approaches, and related treatments was appropriate and accurate in 100% of cases. CONCLUSIONS ChatGPT-4 provided clinically reasonable diagnoses to triage patient complaints of knee pain due to various underlying conditions that were generally consistent with differentials provided by sports medicine physicians. Diagnostic performance was enhanced when providing additional information, allowing ChatGPT-4 to reach high predictive accuracy for recommendations concerning management and treatment options. However, ChatGPT-4 may show clinically important error rates for diagnosis depending on prompting strategy and information provided; therefore, further refinements are necessary prior to implementation into clinical workflows. CLINICAL RELEVANCE Although ChatGPT-4 is increasingly being used by patients for health information, the potential for ChatGPT-4 to serve as a clinical support tool is unclear. In this study, we found that ChatGPT-4 was frequently able to diagnose and triage knee complaints appropriately as rated by sports medicine surgeons, suggesting that it may eventually be a useful clinical support tool.
Collapse
Affiliation(s)
- Kyle N Kunze
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A.; Sports Medicine and Shoulder Service, Hospital for Special Surgery, New York, New York, U.S.A..
| | - Nathan H Varady
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A.; Sports Medicine and Shoulder Service, Hospital for Special Surgery, New York, New York, U.S.A
| | | | - Amy Z Lu
- Weill Cornell College of Medicine, New York, New York, U.S.A
| | - Jorge Chahla
- Department of Orthopaedic Surgery, Rush University Medical Center, Chicago, Illinois, U.S.A
| | - R Kyle Martin
- Department of Orthopedic Surgery, University of Minnesota, Minneapolis, Minnesota, U.S.A
| | - Anil S Ranawat
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A.; Sports Medicine and Shoulder Service, Hospital for Special Surgery, New York, New York, U.S.A
| | - Andrew D Pearle
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A.; Sports Medicine and Shoulder Service, Hospital for Special Surgery, New York, New York, U.S.A
| | - Riley J Williams
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A.; Sports Medicine and Shoulder Service, Hospital for Special Surgery, New York, New York, U.S.A
| |
Collapse
|
8
|
Şahin Ş, Tekin MS, Yigit YE, Erkmen B, Duymaz YK, Bahşi İ. Evaluating the Success of ChatGPT in Addressing Patient Questions Concerning Thyroid Surgery. J Craniofac Surg 2024:00001665-990000000-01698. [PMID: 38861337 DOI: 10.1097/scs.0000000000010395] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2024] [Accepted: 05/15/2024] [Indexed: 06/13/2024] Open
Abstract
OBJECTIVE This study aimed to evaluate the utility and efficacy of ChatGPT in addressing questions related to thyroid surgery, taking into account accuracy, readability, and relevance. METHODS A simulated physician-patient consultation on thyroidectomy surgery was conducted by posing 21 hypothetical questions to ChatGPT. Responses were evaluated using the DISCERN score by 3 independent ear, nose and throat specialists. Readability measures including Flesch Reading Ease), Flesch-Kincaid Grade Level, Gunning Fog Index, Simple Measure of Gobbledygook, Coleman-Liau Index, and Automated Readability Index were also applied. RESULTS The majority of ChatGPT responses were rated fair or above using the DISCERN system, with an average score of 45.44 ± 11.24. However, the readability scores were consistently higher than the recommended grade 6 level, indicating the information may not be easily comprehensible to the general public. CONCLUSION While ChatGPT exhibits potential in answering patient queries related to thyroid surgery, its current formulation is not yet optimally tailored for patient comprehension. Further refinements are necessary for its efficient application in the medical domain.
Collapse
Affiliation(s)
- Şamil Şahin
- Ear Nose and Throat Specialist, Private Practice
| | | | - Yesim Esen Yigit
- Department of Otolaryngology, Umraniye Training and Research Hospital, University of Health Sciences, Istanbul
| | - Burak Erkmen
- Ear Nose and Throat Specialist, Private Practice
| | - Yasar Kemal Duymaz
- Department of Otolaryngology, Umraniye Training and Research Hospital, University of Health Sciences, Istanbul
| | - İlhan Bahşi
- Department of Anatomy, Faculty of Medicine, Gaziantep University, Gaziantep, Turkey
| |
Collapse
|
9
|
Gaudiani MA, Castle JP, Abbas MJ, Pratt BA, Myles MD, Moutzouros V, Lynch TS. ChatGPT-4 Generates More Accurate and Complete Responses to Common Patient Questions About Anterior Cruciate Ligament Reconstruction Than Google's Search Engine. Arthrosc Sports Med Rehabil 2024; 6:100939. [PMID: 39006779 PMCID: PMC11240040 DOI: 10.1016/j.asmr.2024.100939] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2023] [Accepted: 03/27/2024] [Indexed: 07/16/2024] Open
Abstract
Purpose To replicate a patient's internet search to evaluate ChatGPT's appropriateness in answering common patient questions about anterior cruciate ligament reconstruction compared with a Google web search. Methods A Google web search was performed by searching the term "anterior cruciate ligament reconstruction." The top 20 frequently asked questions and responses were recorded. The prompt "What are the 20 most popular patient questions related to 'anterior cruciate ligament reconstruction?'" was input into ChatGPT and questions and responses were recorded. Questions were classified based on the Rothwell system and responses assessed via Flesch-Kincaid Grade Level, correctness, and completeness were for both Google web search and ChatGPT. Results Three of 20 (15%) questions were similar between Google web search and ChatGPT. The most common question types among the Google web search were value (8/20, 40%), fact (7/20, 35%), and policy (5/20, 25%). The most common question types amongst the ChatGPT search were fact (12/20, 60%), policy (6/20, 30%), and value (2/20, 10%). Mean Flesch-Kincaid Grade Level for Google web search responses was significantly lower (11.8 ± 3.8 vs 14.3 ± 2.2; P = .003) than for ChatGPT responses. The mean correctness for Google web search question answers was 1.47 ± 0.5, and mean completeness was 1.36 ± 0.5. Mean correctness for ChatGPT answers was 1.8 ± 0.4 and mean completeness was 1.9 ± 0.3, which were both significantly greater than Google web search answers (P = .03 and P = .0003). Conclusions ChatGPT-4 generated more accurate and complete responses to common patient questions about anterior cruciate ligament reconstruction than Google's search engine. Clinical Relevance The use of artificial intelligence such as ChatGPT is expanding. It is important to understand the quality of information as well as how the results of ChatGPT queries compare with those from Google web searches.
Collapse
Affiliation(s)
- Michael A. Gaudiani
- Department of Orthopedic Surgery, Henry Ford Health, Detroit, Michigan, U.S.A
| | - Joshua P. Castle
- Department of Orthopedic Surgery, Henry Ford Health, Detroit, Michigan, U.S.A
| | - Muhammad J. Abbas
- Department of Orthopedic Surgery, Henry Ford Health, Detroit, Michigan, U.S.A
| | - Brittaney A. Pratt
- Department of Orthopedic Surgery, Henry Ford Health, Detroit, Michigan, U.S.A
| | - Marquisha D. Myles
- Michigan State University College of Human Medicine, Detroit, Michigan, U.S.A
| | - Vasilios Moutzouros
- Department of Orthopedic Surgery, Henry Ford Health, Detroit, Michigan, U.S.A
| | - T. Sean Lynch
- Department of Orthopedic Surgery, Henry Ford Health, Detroit, Michigan, U.S.A
| |
Collapse
|
10
|
Rodler S, Ganjavi C, De Backer P, Magoulianitis V, Ramacciotti LS, De Castro Abreu AL, Gill IS, Cacciamani GE. Generative artificial intelligence in surgery. Surgery 2024; 175:1496-1502. [PMID: 38582732 DOI: 10.1016/j.surg.2024.02.019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2023] [Revised: 02/18/2024] [Accepted: 02/23/2024] [Indexed: 04/08/2024]
Abstract
Generative artificial intelligence is able to collect, extract, digest, and generate information in an understandable way for humans. As the first surgical applications of generative artificial intelligence are applied, this perspective paper aims to provide a comprehensive overview of current applications and future perspectives for the application of generative artificial intelligence in surgery, from preoperative planning to training. Generative artificial intelligence can be used before surgery for planning and decision support by extracting patient information and providing patients with information and simulation regarding the procedure. Intraoperatively, generative artificial intelligence can document data that is normally not captured as intraoperative adverse events or provide information to help decision-making. Postoperatively, GAIs can help with patient discharge and follow-up. The ability to provide real-time feedback and store it for later review is an important capability of GAIs. GAI applications are emerging as highly specialized, task-specific tools for tasks such as data extraction, synthesis, presentation, and communication within the realm of surgery. GAIs have the potential to play a pivotal role in facilitating interaction between surgeons and artificial intelligence.
Collapse
Affiliation(s)
- Severin Rodler
- USC Institute of Urology and Catherine and Joseph Aresty Department of Urology, Keck School of Medicine, University of Southern California, Los Angeles, CA; Artificial Intelligence Center at USC Urology, USC Institute of Urology, University of Southern California, Los Angeles, CA; Department of Urology, University Hospital of LMU Munich, Germany; Young Academic Working Group in Urologic Technology of the European Association of Urology, Arnhem, The Netherlands
| | - Conner Ganjavi
- USC Institute of Urology and Catherine and Joseph Aresty Department of Urology, Keck School of Medicine, University of Southern California, Los Angeles, CA; Artificial Intelligence Center at USC Urology, USC Institute of Urology, University of Southern California, Los Angeles, CA
| | - Pieter De Backer
- Young Academic Working Group in Urologic Technology of the European Association of Urology, Arnhem, The Netherlands; Department of Urology, Onze-Lieve-Vrouwziekenhuis Hospital, Aalst, Belgium; ORSI Academy, Ghent, Belgium
| | - Vasileios Magoulianitis
- Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA
| | - Lorenzo Storino Ramacciotti
- USC Institute of Urology and Catherine and Joseph Aresty Department of Urology, Keck School of Medicine, University of Southern California, Los Angeles, CA; Artificial Intelligence Center at USC Urology, USC Institute of Urology, University of Southern California, Los Angeles, CA
| | - Andre Luis De Castro Abreu
- USC Institute of Urology and Catherine and Joseph Aresty Department of Urology, Keck School of Medicine, University of Southern California, Los Angeles, CA; Artificial Intelligence Center at USC Urology, USC Institute of Urology, University of Southern California, Los Angeles, CA
| | - Inderbir S Gill
- USC Institute of Urology and Catherine and Joseph Aresty Department of Urology, Keck School of Medicine, University of Southern California, Los Angeles, CA; Artificial Intelligence Center at USC Urology, USC Institute of Urology, University of Southern California, Los Angeles, CA
| | - Giovanni E Cacciamani
- USC Institute of Urology and Catherine and Joseph Aresty Department of Urology, Keck School of Medicine, University of Southern California, Los Angeles, CA; Artificial Intelligence Center at USC Urology, USC Institute of Urology, University of Southern California, Los Angeles, CA; Young Academic Working Group in Urologic Technology of the European Association of Urology, Arnhem, The Netherlands.
| |
Collapse
|
11
|
Siegel MG, Rossi MJ, Lubowitz JH. Artificial Intelligence and Machine Learning May Resolve Health Care Information Overload. Arthroscopy 2024; 40:1721-1723. [PMID: 38218231 DOI: 10.1016/j.arthro.2024.01.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/06/2024] [Accepted: 01/06/2024] [Indexed: 01/15/2024]
Abstract
Biomedical information doubles almost every 2 months, and this very rate is expected to double by 2025. The result is information overload for clinicians and researchers. Today, artificial intelligence (AI) and machine learning (ML) research contribute to the deluge of information. In addition, AI large language models, although capable of automating scientific writing, are flawed. They hallucinate (make things up), are trained primarily on non-peer-reviewed content, raise ethical and legal issues, and lack human empathy. Still, when it comes to AI including ML, we are optimistic. The technology is improving rapidly. In the future, AI will help us manage unwieldy information by processing data, determining diagnoses, recommending treatments, and predicting outcomes. In research, AI and ML similarly promise efficient data analysis and literature review and will create new content in response to our instructions. Human touch will be required, and we will disclose use of AI proactively, including rationale for its use, our data input, our level of confidence in the output, and the patients or populations to whom the output may be applied. In addition, we will ensure data quality is high and bias is minimized. Most of all, we will provide essential reasoning, clinical and research guidance, and diligent oversight. Humans will remain accountable.
Collapse
|
12
|
Eng E, Mowers C, Sachdev D, Yerke-Hansen P, Jackson GR, Knapik DM, Sabesan VJ. ChatGPT-3.5 Responses Require Advanced Readability for the General Population and May Not Effectively Supplement Patient-Related Information Provided by the Treating Surgeon Regarding Common Questions About Rotator Cuff Repair. Arthroscopy 2024:S0749-8063(24)00369-4. [PMID: 38777000 DOI: 10.1016/j.arthro.2024.05.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/03/2023] [Revised: 04/23/2024] [Accepted: 05/02/2024] [Indexed: 05/25/2024]
Abstract
PURPOSE To investigate the accuracy of ChatGPT's responses to frequently asked questions prior to rotator cuff repair surgery. METHODS The 10 most common frequently asked questions related to rotator cuff repair were compiled from 4 institution websites. Questions were then input into ChatGPT-3.5 in 1 session. The provided ChatGPT-3.5 responses were analyzed by 2 orthopaedic surgeons for reliability, quality, and readability using the Journal of the American Medical Association Benchmark criteria, the DISCERN score, and the Flesch-Kincaid Grade Level. RESULTS The Journal of the American Medical Association Benchmark criteria score was 0, indicating the absence of reliable source material citations. The mean Flesch-Kincaid Grade Level was 13.4 (range, 11.2-15.0). The mean DISCERN score was 43.4 (range, 36-51), indicating that the quality of the responses overall was considered fair. All responses cited making final decision-making to be made with the treating physician. CONCLUSIONS ChatGPT-3.5 provided substandard patient-related information in alignment with recommendations from the treating surgeon regarding common questions around rotator cuff repair surgery. Additionally, the responses lacked reliable source material citations, and the readability of the responses was relatively advanced with a complex language style. CLINICAL RELEVANCE The findings of this study suggest that ChatGPT-3.5 may not effectively supplement patient-related information in the context of recommendations provided by the treating surgeon prior to rotator cuff repair surgery.
Collapse
Affiliation(s)
- Emma Eng
- Charles E. Schmidt College of Medicine at Florida Atlantic University, Boca Raton, Florida, U.S.A
| | - Colton Mowers
- Rush University Medical College, Chicago, Illinois, U.S.A
| | | | - Payton Yerke-Hansen
- Department of Orthopaedic Surgery, Louisiana State University Health-Shreveport, Shreveport, Louisiana, U.S.A
| | - Garrett R Jackson
- Department of Orthopaedic Surgery, University of Missouri, Columbia, Missouri, U.S.A..
| | - Derrick M Knapik
- Department of Orthopaedic Surgery, Washington University and Barnes-Jewish Orthopedic Center, Chesterfield, Missouri, U.S.A
| | - Vani J Sabesan
- HCA JFK/University of Miami Miller School of Medicine Orthopaedic Residency Program Palm Beach, Lake Worth, Florida, U.S.A
| |
Collapse
|
13
|
Fahy S, Niemann M, Böhm P, Winkler T, Oehme S. Assessment of the Quality and Readability of Information Provided by ChatGPT in Relation to the Use of Platelet-Rich Plasma Therapy for Osteoarthritis. J Pers Med 2024; 14:495. [PMID: 38793077 PMCID: PMC11122161 DOI: 10.3390/jpm14050495] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2024] [Revised: 04/28/2024] [Accepted: 05/04/2024] [Indexed: 05/26/2024] Open
Abstract
Objective: This study aimed to evaluate the quality and readability of information generated by ChatGPT versions 3.5 and 4 concerning platelet-rich plasma (PRP) therapy in the management of knee osteoarthritis (OA), exploring whether large language models (LLMs) could play a significant role in patient education. Design: A total of 23 common patient queries regarding the role of PRP therapy in knee OA management were presented to ChatGPT versions 3.5 and 4. The quality of the responses was assessed using the DISCERN criteria, and readability was evaluated using six established assessment tools. Results: Both ChatGPT versions 3.5 and 4 produced moderate quality information. The quality of information provided by ChatGPT version 4 was significantly better than version 3.5, with mean DISCERN scores of 48.74 and 44.59, respectively. Both models scored highly with respect to response relevance and had a consistent emphasis on the importance of shared decision making. However, both versions produced content significantly above the recommended 8th grade reading level for patient education materials (PEMs), with mean reading grade levels (RGLs) of 17.18 for ChatGPT version 3.5 and 16.36 for ChatGPT version 4, indicating a potential barrier to their utility in patient education. Conclusions: While ChatGPT versions 3.5 and 4 both demonstrated the capability to generate information of moderate quality regarding the role of PRP therapy for knee OA, the readability of the content remains a significant barrier to widespread usage, exceeding the recommended reading levels for PEMs. Although ChatGPT version 4 showed improvements in quality and source citation, future iterations must focus on producing more accessible content to serve as a viable resource in patient education. Collaboration between healthcare providers, patient organizations, and AI developers is crucial to ensure the generation of high quality, peer reviewed, and easily understandable information that supports informed healthcare decisions.
Collapse
Affiliation(s)
- Stephen Fahy
- Center for Musculoskeletal Surgery, Charité—Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, 10117 Berlin, Germany; (S.F.); (M.N.); (T.W.)
| | - Marcel Niemann
- Center for Musculoskeletal Surgery, Charité—Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, 10117 Berlin, Germany; (S.F.); (M.N.); (T.W.)
| | - Peter Böhm
- Deutsche Rheuma-Liga e.V., 53111 Bonn, Germany;
| | - Tobias Winkler
- Center for Musculoskeletal Surgery, Charité—Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, 10117 Berlin, Germany; (S.F.); (M.N.); (T.W.)
- Berlin Institute of Health Center for Regenerative Therapies, Berlin Institute of Health at Charité—Universitätsmedizin Berlin, 13353 Berlin, Germany
- Berlin Institute of Health at Charité—Universitätsmedizin Berlin, Julius Wolff Institute, 13353 Berlin, Germany
| | - Stephan Oehme
- Center for Musculoskeletal Surgery, Charité—Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, 10117 Berlin, Germany; (S.F.); (M.N.); (T.W.)
| |
Collapse
|
14
|
Gül Ş, Erdemir İ, Hanci V, Aydoğmuş E, Erkoç YS. How artificial intelligence can provide information about subdural hematoma: Assessment of readability, reliability, and quality of ChatGPT, BARD, and perplexity responses. Medicine (Baltimore) 2024; 103:e38009. [PMID: 38701313 PMCID: PMC11062651 DOI: 10.1097/md.0000000000038009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/21/2024] [Accepted: 04/04/2024] [Indexed: 05/05/2024] Open
Abstract
Subdural hematoma is defined as blood collection in the subdural space between the dura mater and arachnoid. Subdural hematoma is a condition that neurosurgeons frequently encounter and has acute, subacute and chronic forms. The incidence in adults is reported to be 1.72-20.60/100.000 people annually. Our study aimed to evaluate the quality, reliability and readability of the answers to questions asked to ChatGPT, Bard, and perplexity about "Subdural Hematoma." In this observational and cross-sectional study, we asked ChatGPT, Bard, and perplexity to provide the 100 most frequently asked questions about "Subdural Hematoma" separately. Responses from both chatbots were analyzed separately for readability, quality, reliability and adequacy. When the median readability scores of ChatGPT, Bard, and perplexity answers were compared with the sixth-grade reading level, a statistically significant difference was observed in all formulas (P < .001). All 3 chatbot responses were found to be difficult to read. Bard responses were more readable than ChatGPT's (P < .001) and perplexity's (P < .001) responses for all scores evaluated. Although there were differences between the results of the evaluated calculators, perplexity's answers were determined to be more readable than ChatGPT's answers (P < .05). Bard answers were determined to have the best GQS scores (P < .001). Perplexity responses had the best Journal of American Medical Association and modified DISCERN scores (P < .001). ChatGPT, Bard, and perplexity's current capabilities are inadequate in terms of quality and readability of "Subdural Hematoma" related text content. The readability standard for patient education materials as determined by the American Medical Association, National Institutes of Health, and the United States Department of Health and Human Services is at or below grade 6. The readability levels of the responses of artificial intelligence applications such as ChatGPT, Bard, and perplexity are significantly higher than the recommended 6th grade level.
Collapse
Affiliation(s)
- Şanser Gül
- Department of Neurosurgery, Ankara Ataturk Sanatory Education and Research Hospital, Ankara, Turkey
| | - İsmail Erdemir
- Department of Anesthesiology and Critical Care, Faculty of Medicine, Dokuz Eylül University, Izmir, Turkey
| | - Volkan Hanci
- Department of Anesthesiology and Reanimation, Ankara Sincan Education and Research Hospital, Ankara, Turkey
| | - Evren Aydoğmuş
- Department of Neurosurgery, Istanbul Kartal Dr Lütfi Kırdar City Hospital, Istanbul, Turkey
| | - Yavuz Selim Erkoç
- Department of Neurosurgery, Ankara Ataturk Sanatory Education and Research Hospital, Ankara, Turkey
| |
Collapse
|
15
|
Butler JJ, Puleo J, Harrington MC, Dahmen J, Rosenbaum AJ, Kerkhoffs GMMJ, Kennedy JG. From technical to understandable: Artificial Intelligence Large Language Models improve the readability of knee radiology reports. Knee Surg Sports Traumatol Arthrosc 2024; 32:1077-1086. [PMID: 38488217 DOI: 10.1002/ksa.12133] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Revised: 02/19/2024] [Accepted: 02/22/2024] [Indexed: 04/23/2024]
Abstract
PURPOSE The purpose of this study was to evaluate the effectiveness of an Artificial Intelligence-Large Language Model (AI-LLM) at improving the readability of knee radiology reports. METHODS Reports of 100 knee X-rays, 100 knee computed tomography (CT) scans and 100 knee magnetic resonance imaging (MRI) scans were retrieved. The following prompt command was inserted into the AI-LLM: 'Explain this radiology report to a patient in layman's terms in the second person:[Report Text]'. The Flesch-Kincaid reading level (FKRL) score, Flesch reading ease (FRE) score and report length were calculated for the original radiology report and the AI-LLM generated report. Any 'hallucination' or inaccurate text produced by the AI-LLM-generated report was documented. RESULTS Statistically significant improvements in mean FKRL scores in the AI-LLM generated X-ray report (12.7 ± 1.0-7.2 ± 0.6), CT report (13.4 ± 1.0-7.5 ± 0.5) and MRI report (13.5 ± 0.9-7.5 ± 0.6) were observed. Statistically significant improvements in mean FRE scores in the AI-LLM generated X-ray report (39.5 ± 7.5-76.8 ± 5.1), CT report (27.3 ± 5.9-73.1 ± 5.6) and MRI report (26.8 ± 6.4-73.4 ± 5.0) were observed. Superior FKRL scores and FRE scores were observed in the AI-LLM-generated X-ray report compared to the AI-LLM-generated CT report and MRI report, p < 0.001. The hallucination rates in the AI-LLM generated X-ray report, CT report and MRI report were 2%, 5% and 5%, respectively. CONCLUSIONS This study highlights the promising use of AI-LLMs as an innovative, patient-centred strategy to improve the readability of knee radiology reports. The clinical relevance of this study is that an AI-LLM-generated knee radiology report may enhance patients' understanding of their imaging reports, potentially reducing the responder burden placed on the ordering physicians. However, due to the 'hallucinations' produced by the AI-LLM-generated report, the ordering physician must always engage in a collaborative discussion with the patient regarding both reports and the corresponding images. LEVEL OF EVIDENCE Level IV.
Collapse
Affiliation(s)
- James J Butler
- Department of Orthopaedic Surgery, Foot and Ankle Division, NYU Langone Health, New York City, New York, USA
| | - James Puleo
- Albany Medical Center, Albany, New York, USA
| | | | - Jari Dahmen
- Department of Orthopaedic Surgery and Sports Medicine, Amsterdam Movement Sciences, Amsterdam UMC, University of Amsterdam, Location AMC, Amsterdam, The Netherlands
- Academic Center for Evidence-Based Sports Medicine, Amsterdam UMC, Amsterdam, The Netherlands
- Amsterdam Collaboration for Health and Safety in Sports, International Olympic Committee Research Center, Amsterdam UMC, Amsterdam, The Netherlands
| | | | - Gino M M J Kerkhoffs
- Department of Orthopaedic Surgery and Sports Medicine, Amsterdam Movement Sciences, Amsterdam UMC, University of Amsterdam, Location AMC, Amsterdam, The Netherlands
- Academic Center for Evidence-Based Sports Medicine, Amsterdam UMC, Amsterdam, The Netherlands
- Amsterdam Collaboration for Health and Safety in Sports, International Olympic Committee Research Center, Amsterdam UMC, Amsterdam, The Netherlands
| | - John G Kennedy
- Department of Orthopaedic Surgery, Foot and Ankle Division, NYU Langone Health, New York City, New York, USA
| |
Collapse
|
16
|
Cote MP, Lubowitz JH. Recommended Requirements and Essential Elements for Proper Reporting of the Use of Artificial Intelligence Machine Learning Tools in Biomedical Research and Scientific Publications. Arthroscopy 2024; 40:1033-1038. [PMID: 38300189 DOI: 10.1016/j.arthro.2023.12.027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/30/2023] [Accepted: 12/30/2023] [Indexed: 02/02/2024]
Abstract
Essential elements required for proper use of artificial intelligence machine learning tools in biomedical research and scientific publications include (1) explanation justifying why a machine learning approach contributes to the purpose of the study; (2) description of the adequacy of the data (input) to produce the desired results (output); (3) details of the algorithmic (i.e., computational) approach including methods for organizing the data (preprocessing); the machine learning computational algorithm(s) assessed; on what data the models were trained; the presence of bias and efforts to mitigate these effects; and the methods for quantifying the variables (features) most influential in determining the results (e.g., Shapley values); (4) description of methods, and reporting of results, quantitating performance in terms of both model accuracy and model calibration (level of confidence in the model's predictions); (5) availability of the programming code (including a link to the code when available-ideally, the code should be available); (6) discussion of model internal validation (results applicable and sensitive to the population investigated and data on which the model was trained) and external validation (were the results investigated as to whether they are generalizable to different populations? If not, consideration of this limitation and discussion of plans for external validation, i.e., next steps). As biomedical research submissions using artificial intelligence technology increase, these requirements could facilitate purposeful use and comprehensive methodological reporting.
Collapse
|
17
|
Sánchez-Rosenberg G, Magnéli M, Barle N, Kontakis MG, Müller AM, Wittauer M, Gordon M, Brodén C. ChatGPT-4 generates orthopedic discharge documents faster than humans maintaining comparable quality: a pilot study of 6 cases. Acta Orthop 2024; 95:152-156. [PMID: 38597205 PMCID: PMC10959013 DOI: 10.2340/17453674.2024.40182] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/01/2023] [Accepted: 01/28/2024] [Indexed: 04/11/2024] Open
Abstract
BACKGROUND AND PURPOSE Large language models like ChatGPT-4 have emerged. They hold the potential to reduce the administrative burden by generating everyday clinical documents, thus allowing the physician to spend more time with the patient. We aimed to assess both the quality and efficiency of discharge documents generated by ChatGPT-4 in comparison with those produced by physicians. PATIENTS AND METHODS To emulate real-world situations, the health records of 6 fictional orthopedic cases were created. Discharge documents for each case were generated by a junior attending orthopedic surgeon and an advanced orthopedic resident. ChatGPT-4 was then prompted to generate the discharge documents using the same health record information. The quality assessment was performed by an expert panel (n = 15) blinded to the source of the documents. As secondary outcome, the time required to generate the documents was compared, logging the duration of the creation of the discharge documents by the physician and by ChatGPT-4. RESULTS Overall, both ChatGPT-4 and physician-generated notes were comparable in quality. Notably, ChatGPT-4 generated discharge documents 10 times faster than the traditional method. 4 events of hallucinations were found in the ChatGPT-4-generated content, compared with 6 events in the human/physician produced notes. CONCLUSION ChatGPT-4 creates orthopedic discharge notes faster than physicians, with comparable quality. This shows it has great potential for making these documents more efficient in orthopedic care. ChatGPT-4 has the potential to significantly reduce the administrative burden on healthcare professionals.
Collapse
Affiliation(s)
| | - Martin Magnéli
- Karolinska Institute, Department of Clinical Sciences at Danderyd Hospital, Stockholm; Sweden
| | - Niklas Barle
- Karolinska Institute, Department of Clinical Sciences at Danderyd Hospital, Stockholm; Sweden
| | - Michael G Kontakis
- Department of Surgical Sciences, Orthopedics, Uppsala University Hospital, Uppsala, Sweden
| | - Andreas Marc Müller
- Department of Orthopedic and Trauma Surgery, University Hospital Basel, Switzerland
| | - Matthias Wittauer
- Department of Orthopedic and Trauma Surgery, University Hospital Basel, Switzerland
| | - Max Gordon
- Karolinska Institute, Department of Clinical Sciences at Danderyd Hospital, Stockholm; Sweden
| | - Cyrus Brodén
- Department of Surgical Sciences, Orthopedics, Uppsala University Hospital, Uppsala, Sweden.
| |
Collapse
|
18
|
Mastrokostas PG, Mastrokostas LE, Emara AK, Wellington IJ, Ginalis E, Houten JK, Khalsa AS, Saleh A, Razi AE, Ng MK. GPT-4 as a Source of Patient Information for Anterior Cervical Discectomy and Fusion: A Comparative Analysis Against Google Web Search. Global Spine J 2024:21925682241241241. [PMID: 38513636 DOI: 10.1177/21925682241241241] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 03/23/2024] Open
Abstract
STUDY DESIGN Comparative study. OBJECTIVES This study aims to compare Google and GPT-4 in terms of (1) question types, (2) response readability, (3) source quality, and (4) numerical response accuracy for the top 10 most frequently asked questions (FAQs) about anterior cervical discectomy and fusion (ACDF). METHODS "Anterior cervical discectomy and fusion" was searched on Google and GPT-4 on December 18, 2023. Top 10 FAQs were classified according to the Rothwell system. Source quality was evaluated using JAMA benchmark criteria and readability was assessed using Flesch Reading Ease and Flesch-Kincaid grade level. Differences in JAMA scores, Flesch-Kincaid grade level, Flesch Reading Ease, and word count between platforms were analyzed using Student's t-tests. Statistical significance was set at the .05 level. RESULTS Frequently asked questions from Google were varied, while GPT-4 focused on technical details and indications/management. GPT-4 showed a higher Flesch-Kincaid grade level (12.96 vs 9.28, P = .003), lower Flesch Reading Ease score (37.07 vs 54.85, P = .005), and higher JAMA scores for source quality (3.333 vs 1.800, P = .016). Numerically, 6 out of 10 responses varied between platforms, with GPT-4 providing broader recovery timelines for ACDF. CONCLUSIONS This study demonstrates GPT-4's ability to elevate patient education by providing high-quality, diverse information tailored to those with advanced literacy levels. As AI technology evolves, refining these tools for accuracy and user-friendliness remains crucial, catering to patients' varying literacy levels and information needs in spine surgery.
Collapse
Affiliation(s)
- Paul G Mastrokostas
- College of Medicine, State University of New York (SUNY) Downstate, Brooklyn, NY, USA
| | | | - Ahmed K Emara
- Department of Orthopaedic Surgery, Cleveland Clinic, Cleveland, OH, USA
| | - Ian J Wellington
- Department of Orthopaedic Surgery, University of Connecticut, Hartford, CT, USA
| | | | - John K Houten
- Department of Neurosurgery, Mount Sinai School of Medicine, New York, NY, USA
| | - Amrit S Khalsa
- Department of Orthopaedic Surgery, University of Pennsylvania, Philadelphia, PA, USA
| | - Ahmed Saleh
- Department of Orthopaedic Surgery, Maimonides Medical Center, Brooklyn, NY, USA
| | - Afshin E Razi
- Department of Orthopaedic Surgery, Maimonides Medical Center, Brooklyn, NY, USA
| | - Mitchell K Ng
- Department of Orthopaedic Surgery, Maimonides Medical Center, Brooklyn, NY, USA
| |
Collapse
|
19
|
Fahy S, Oehme S, Milinkovic D, Jung T, Bartek B. Assessment of Quality and Readability of Information Provided by ChatGPT in Relation to Anterior Cruciate Ligament Injury. J Pers Med 2024; 14:104. [PMID: 38248805 PMCID: PMC10817257 DOI: 10.3390/jpm14010104] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Revised: 12/17/2023] [Accepted: 12/20/2023] [Indexed: 01/23/2024] Open
Abstract
The aim of our study was to evaluate the potential role of Artificial Intelligence tools like ChatGPT in patient education. To do this, we assessed both the quality and readability of information provided by ChatGPT 3.5 and 4 in relation to Anterior Cruciate Ligament (ACL) injury and treatment. ChatGPT 3.5 and 4 were used to answer common patient queries relating to ACL injuries and treatment. The quality of the information was assessed using the DISCERN criteria. Readability was assessed with the use of seven readability formulae: the Flesch-Kincaid Reading Grade Level, the Flesch Reading Ease Score, the Raygor Estimate, the SMOG, the Fry, the FORCAST, and the Gunning Fog. The mean reading grade level (RGL) was compared with the recommended 8th-grade reading level, the mean RGL among adults in America. The perceived quality and mean RGL of answers given by both ChatGPT 3.5 and 4 was also compared. Both ChatGPT 3.5 and 4 yielded DISCERN scores suggesting "good" quality of information, with ChatGPT 4 slightly outperforming 3.5. However, readability levels for both versions significantly exceeded the average 8th-grade reading level for American patients. ChatGPT 3.5 had a mean RGL of 18.08, while the mean RGL of ChatGPT 4 was 17.9, exceeding the average American reading grade level by 10.08 grade levels and 9.09 grade levels, respectively. While ChatGPT can provide both reliable and good quality information on ACL injuries and treatment options, the readability of the content may limit its utility. Additionally, the consistent lack of source citation represents a significant area of concern for patients and clinicians alike. If AI is to play a role in patient education, it must reliably produce information which is accurate, easily comprehensible, and clearly sourced.
Collapse
Affiliation(s)
- Stephen Fahy
- Centrum für Muskuloskeletale Chirurgie, Charité Universitätsmedizin, 10117 Berlin, Germany; (S.O.); (T.J.); (B.B.)
| | | | | | | | | |
Collapse
|
20
|
Hernandez CA, Vazquez Gonzalez AE, Polianovskaia A, Amoro Sanchez R, Muyolema Arce V, Mustafa A, Vypritskaya E, Perez Gutierrez O, Bashir M, Eighaei Sedeh A. The Future of Patient Education: AI-Driven Guide for Type 2 Diabetes. Cureus 2023; 15:e48919. [PMID: 38024047 PMCID: PMC10654048 DOI: 10.7759/cureus.48919] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/16/2023] [Indexed: 12/01/2023] Open
Abstract
Introduction and aim The surging incidence of type 2 diabetes has become a growing concern for the healthcare sector. This chronic ailment, characterized by its complex blend of genetic and lifestyle determinants, has witnessed a notable increase in recent times, exerting substantial pressure on healthcare resources. As more individuals turn to online platforms for health guidance and embrace the utilization of Chat Generative Pre-trained Transformer (ChatGPT; San Francisco, CA: OpenAI), a text-generating AI (TGAI), to get insights into their well-being, evaluating its effectiveness and reliability becomes crucial. This research primarily aimed to evaluate the correctness of TGAI responses to type 2 diabetes (T2DM) inquiries via ChatGPT. Furthermore, this study aimed to examine the consistency of TGAI in addressing common queries on T2DM complications for patient education. Material and methods Questions on T2DM were formulated by experienced physicians and screened by research personnel before querying ChatGPT. Each question was posed thrice, and the collected answers were summarized. Responses were then sorted into three distinct categories as follows: (a) appropriate, (b) inappropriate, and (c) unreliable by two seasoned physicians. In instances of differing opinions, a third physician was consulted to achieve consensus. Results From the initial set of 110 T2DM questions, 40 were dismissed by experts for relevance, resulting in a final count of 70. An overwhelming 98.5% of the AI's answers were judged as appropriate, thus underscoring its reliability over traditional online search engines. Nonetheless, a 1.5% rate of inappropriate responses underlines the importance of ongoing AI improvements and strict adherence to medical protocols. Conclusion TGAI provides medical information of high quality and reliability. This study underscores TGAI's impressive effectiveness in delivering reliable information about T2DM, with 98.5% of responses aligning with the standard of care. These results hold promise for integrating AI platforms as supplementary tools to enhance patient education and outcomes.
Collapse
|