1
|
Hancı V, Ergün B, Gül Ş, Uzun Ö, Erdemir İ, Hancı FB. Assessment of readability, reliability, and quality of ChatGPT®, BARD®, Gemini®, Copilot®, Perplexity® responses on palliative care. Medicine (Baltimore) 2024; 103:e39305. [PMID: 39151545 PMCID: PMC11332738 DOI: 10.1097/md.0000000000039305] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/14/2024] [Revised: 07/20/2024] [Accepted: 07/25/2024] [Indexed: 08/19/2024] Open
Abstract
There is no study that comprehensively evaluates data on the readability and quality of "palliative care" information provided by artificial intelligence (AI) chatbots ChatGPT®, Bard®, Gemini®, Copilot®, Perplexity®. Our study is an observational and cross-sectional original research study. In our study, AI chatbots ChatGPT®, Bard®, Gemini®, Copilot®, and Perplexity® were asked to present the answers of the 100 questions most frequently asked by patients about palliative care. Responses from each 5 AI chatbots were analyzed separately. This study did not involve any human participants. Study results revealed significant differences between the readability assessments of responses from all 5 AI chatbots (P < .05). According to the results of our study, when different readability indexes were evaluated holistically, the readability of AI chatbot responses was evaluated as Bard®, Copilot®, Perplexity®, ChatGPT®, Gemini®, from easy to difficult (P < .05). In our study, the median readability indexes of each of the 5 AI chatbots Bard®, Copilot®, Perplexity®, ChatGPT®, Gemini® responses were compared to the "recommended" 6th grade reading level. According to the results of our study answers of all 5 AI chatbots were compared with the 6th grade reading level, statistically significant differences were observed in the all formulas (P < .001). The answers of all 5 artificial intelligence robots were determined to be at an educational level well above the 6th grade level. The modified DISCERN and Journal of American Medical Association scores was found to be the highest in Perplexity® (P < .001). Gemini® responses were found to have the highest Global Quality Scale score (P < .001). It is emphasized that patient education materials should have a readability level of 6th grade level. Of the 5 AI chatbots whose answers about palliative care were evaluated, Bard®, Copilot®, Perplexity®, ChatGPT®, Gemini®, their current answers were found to be well above the recommended levels in terms of readability of text content. Text content quality assessment scores are also low. Both the quality and readability of texts should be brought to appropriate recommended limits.
Collapse
Affiliation(s)
- Volkan Hancı
- Clinic of Anesthesiology and Critical Care, Sincan Education and Research Hospital, Ankara, Turkey
| | - Bişar Ergün
- Clinic of Internal Medicine and Critical Care, Dr. Ismail Fehmi Cumalioğlu City Hospital, Tekirdağ, Turkey
| | - Şanser Gül
- Clinic of Neurosurgery, Ankara Ataturk Sanatory Education and Research Hospital, Ankara, Turkey
| | - Özcan Uzun
- Clinic of Internal Medicine and Nephrology, Yalova City Hospital, Yalova, Turkey
| | - İsmail Erdemir
- Department of Anesthesiology and Critical Care, Faculty of Medicine, Dokuz Eylül University, Izmir, Turkey
| | - Ferid Baran Hancı
- Artificial Intelligence Engineering Department, Faculty of Engineering, Ostim Technical University, Ankara, Turkey
| |
Collapse
|
2
|
Denecke K, May R, Rivera Romero O. Potential of Large Language Models in Health Care: Delphi Study. J Med Internet Res 2024; 26:e52399. [PMID: 38739445 PMCID: PMC11130776 DOI: 10.2196/52399] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2023] [Revised: 10/10/2023] [Accepted: 04/19/2024] [Indexed: 05/14/2024] Open
Abstract
BACKGROUND A large language model (LLM) is a machine learning model inferred from text data that captures subtle patterns of language use in context. Modern LLMs are based on neural network architectures that incorporate transformer methods. They allow the model to relate words together through attention to multiple words in a text sequence. LLMs have been shown to be highly effective for a range of tasks in natural language processing (NLP), including classification and information extraction tasks and generative applications. OBJECTIVE The aim of this adapted Delphi study was to collect researchers' opinions on how LLMs might influence health care and on the strengths, weaknesses, opportunities, and threats of LLM use in health care. METHODS We invited researchers in the fields of health informatics, nursing informatics, and medical NLP to share their opinions on LLM use in health care. We started the first round with open questions based on our strengths, weaknesses, opportunities, and threats framework. In the second and third round, the participants scored these items. RESULTS The first, second, and third rounds had 28, 23, and 21 participants, respectively. Almost all participants (26/28, 93% in round 1 and 20/21, 95% in round 3) were affiliated with academic institutions. Agreement was reached on 103 items related to use cases, benefits, risks, reliability, adoption aspects, and the future of LLMs in health care. Participants offered several use cases, including supporting clinical tasks, documentation tasks, and medical research and education, and agreed that LLM-based systems will act as health assistants for patient education. The agreed-upon benefits included increased efficiency in data handling and extraction, improved automation of processes, improved quality of health care services and overall health outcomes, provision of personalized care, accelerated diagnosis and treatment processes, and improved interaction between patients and health care professionals. In total, 5 risks to health care in general were identified: cybersecurity breaches, the potential for patient misinformation, ethical concerns, the likelihood of biased decision-making, and the risk associated with inaccurate communication. Overconfidence in LLM-based systems was recognized as a risk to the medical profession. The 6 agreed-upon privacy risks included the use of unregulated cloud services that compromise data security, exposure of sensitive patient data, breaches of confidentiality, fraudulent use of information, vulnerabilities in data storage and communication, and inappropriate access or use of patient data. CONCLUSIONS Future research related to LLMs should not only focus on testing their possibilities for NLP-related tasks but also consider the workflows the models could contribute to and the requirements regarding quality, integration, and regulations needed for successful implementation in practice.
Collapse
Affiliation(s)
| | - Richard May
- Harz University of Applied Sciences, Wernigerode, Germany
| | - Octavio Rivera Romero
- Instituto de Ingeniería Informática (I3US), Universidad de Sevilla, Sevilla, Spain
- Department of Electronic Technology, Universidad de Sevilla, Sevilla, Spain
| |
Collapse
|
3
|
Pak S, Park SG, Park J, Cho ST, Lee YG, Ahn H. Applications of artificial intelligence in urologic oncology. Investig Clin Urol 2024; 65:202-216. [PMID: 38714511 PMCID: PMC11076794 DOI: 10.4111/icu.20230435] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/30/2023] [Revised: 02/24/2024] [Accepted: 03/11/2024] [Indexed: 05/10/2024] Open
Abstract
PURPOSE With the recent rising interest in artificial intelligence (AI) in medicine, many studies have explored the potential and usefulness of AI in urological diseases. This study aimed to comprehensively review recent applications of AI in urologic oncology. MATERIALS AND METHODS We searched the PubMed-MEDLINE databases for articles in English on machine learning (ML) and deep learning (DL) models related to general surgery and prostate, bladder, and kidney cancer. The search terms were a combination of keywords, including both "urology" and "artificial intelligence" with one of the following: "machine learning," "deep learning," "neural network," "renal cell carcinoma," "kidney cancer," "urothelial carcinoma," "bladder cancer," "prostate cancer," and "robotic surgery." RESULTS A total of 58 articles were included. The studies on prostate cancer were related to grade prediction, improved diagnosis, and predicting outcomes and recurrence. The studies on bladder cancer mainly used radiomics to identify aggressive tumors and predict treatment outcomes, recurrence, and survival rates. Most studies on the application of ML and DL in kidney cancer were focused on the differentiation of benign and malignant tumors as well as prediction of their grade and subtype. Most studies suggested that methods using AI may be better than or similar to existing traditional methods. CONCLUSIONS AI technology is actively being investigated in the field of urological cancers as a tool for diagnosis, prediction of prognosis, and decision-making and is expected to be applied in additional clinical areas soon. Despite technological, legal, and ethical concerns, AI will change the landscape of urological cancer management.
Collapse
Affiliation(s)
- Sahyun Pak
- Department of Urology, Kangnam Sacred Heart Hospital, Hallym University College of Medicine, Seoul, Korea
| | - Sung Gon Park
- Department of Urology, Kangnam Sacred Heart Hospital, Hallym University College of Medicine, Seoul, Korea
| | | | - Sung Tae Cho
- Department of Urology, Kangnam Sacred Heart Hospital, Hallym University College of Medicine, Seoul, Korea
| | - Young Goo Lee
- Department of Urology, Kangnam Sacred Heart Hospital, Hallym University College of Medicine, Seoul, Korea
| | - Hanjong Ahn
- Department of Urology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea.
| |
Collapse
|
4
|
Arango SD, Flynn JC, Zeitlin J, Lorenzana DJ, Miller AJ, Wilson MS, Strohl AB, Weiss LE, Weir TB. The Performance of ChatGPT on the American Society for Surgery of the Hand Self-Assessment Examination. Cureus 2024; 16:e58950. [PMID: 38800302 PMCID: PMC11126365 DOI: 10.7759/cureus.58950] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/24/2024] [Indexed: 05/29/2024] Open
Abstract
BACKGROUND This study aims to compare the performance of ChatGPT-3.5 (GPT-3.5) and ChatGPT-4 (GPT-4) on the American Society for Surgery of the Hand (ASSH) Self-Assessment Examination (SAE) to determine their potential as educational tools. METHODS This study assessed the proportion of correct answers to text-based questions on the 2021 and 2022 ASSH SAE between untrained ChatGPT versions. Secondary analyses assessed the performance of ChatGPT based on question difficulty and question category. The outcomes of ChatGPT were compared with the performance of actual examinees on the ASSH SAE. RESULTS A total of 238 questions were included in the analysis. Compared with GPT-3.5, GPT-4 provided significantly more correct answers overall (58.0% versus 68.9%, respectively; P = 0.013), on the 2022 SAE (55.9% versus 72.9%; P = 0.007), and more difficult questions (48.8% versus 63.6%; P = 0.02). In a multivariable logistic regression analysis, correct answers were predicted by GPT-4 (odds ratio [OR], 1.66; P = 0.011), increased question difficulty (OR, 0.59; P = 0.009), Bone and Joint questions (OR, 0.18; P < 0.001), and Soft Tissue questions (OR, 0.30; P = 0.013). Actual examinees scored a mean of 21.6% above GPT-3.5 and 10.7% above GPT-4. The mean percentage of correct answers by actual examinees was significantly higher for correct (versus incorrect) ChatGPT answers. CONCLUSIONS GPT-4 demonstrated improved performance over GPT-3.5 on the ASSH SAE, especially on more difficult questions. Actual examinees scored higher than both versions of ChatGPT, but the margin was cut in half by GPT-4.
Collapse
Affiliation(s)
- Sebastian D Arango
- Department of Orthopaedic Surgery, Philadelphia Hand to Shoulder Center, Philadelphia, USA
| | - Jason C Flynn
- Department of Orthopaedic Surgery, Sidney Kimmel Medical College, Philadelphia, USA
| | - Jacob Zeitlin
- Department of Orthopaedic Surgery, Philadelphia Hand to Shoulder Center, Philadelphia, USA
| | - Daniel J Lorenzana
- Department of Orthopaedic Surgery, Philadelphia Hand to Shoulder Center, Philadelphia, USA
| | - Andrew J Miller
- Department of Orthopaedic Surgery, Philadelphia Hand to Shoulder Center, Philadelphia, USA
| | - Matthew S Wilson
- Department of Orthopaedic Surgery, Philadelphia Hand to Shoulder Center, Philadelphia, USA
| | - Adam B Strohl
- Department of Orthopaedic Surgery, Philadelphia Hand to Shoulder Center, Philadelphia, USA
| | - Lawrence E Weiss
- Division of Orthopaedic Hand Surgery, OAA Orthopaedic Specialists, Allentown, USA
| | - Tristan B Weir
- Department of Orthopaedic Surgery, Philadelphia Hand to Shoulder Center, Philadelphia, USA
| |
Collapse
|
5
|
Lum ZC, Collins DP, Dennison S, Guntupalli L, Choudhary S, Saiz AM, Randall RL. Generative Artificial Intelligence Performs at a Second-Year Orthopedic Resident Level. Cureus 2024; 16:e56104. [PMID: 38618358 PMCID: PMC11014641 DOI: 10.7759/cureus.56104] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Accepted: 03/12/2024] [Indexed: 04/16/2024] Open
Abstract
Introduction Artificial intelligence (AI) models using large language models (LLMs) and non-specific domains have gained attention for their innovative information processing. As AI advances, it's essential to regularly evaluate these tools' competency to maintain high standards, prevent errors or biases, and avoid flawed reasoning or misinformation that could harm patients or spread inaccuracies. Our study aimed to determine the performance of Chat Generative Pre-trained Transformer (ChatGPT) by OpenAI and Google BARD (BARD) in orthopedic surgery, assess performance based on question types, contrast performance between different AIs and compare AI performance to orthopedic residents. Methods We administered ChatGPT and BARD 757 Orthopedic In-Training Examination (OITE) questions. After excluding image-related questions, the AIs answered 390 multiple choice questions, all categorized within 10 sub-specialties (basic science, trauma, sports medicine, spine, hip and knee, pediatrics, oncology, shoulder and elbow, hand, and food and ankle) and three taxonomy classes (recall, interpretation, and application of knowledge). Statistical analysis was performed to analyze the number of questions answered correctly by each AI model, the performance returned by each AI model within the categorized question sub-specialty designation, and the performance of each AI model in comparison to the results returned by orthopedic residents classified by their respective post-graduate year (PGY) level. Results BARD answered more overall questions correctly (58% vs 54%, p<0.001). ChatGPT performed better in sports medicine and basic science and worse in hand surgery, while BARD performed better in basic science (p<0.05). The AIs performed better in recall questions compared to the application of knowledge (p<0.05). Based on previous data, it ranked in the 42nd-96th percentile for post-graduate year ones (PGY1s), 27th-58th for PGY2s, 3rd-29th for PGY3s, 1st-21st for PGY4s, and 1st-17th for PGY5s. Discussion ChatGPT excelled in sports medicine but fell short in hand surgery, while both AIs performed well in the basic science sub-specialty but performed poorly in the application of knowledge-based taxonomy questions. BARD performed better than ChatGPT overall. Although the AI reached the second-year PGY orthopedic resident level, it fell short of passing the American Board of Orthopedic Surgery (ABOS). Its strengths in recall-based inquiries highlight its potential as an orthopedic learning and educational tool.
Collapse
Affiliation(s)
- Zachary C Lum
- Orthopedic Surgery, University of California (UC) Davis School of Medicine, Sacramento, USA
- Orthopedic Surgery, Nova Southeastern University, Pembroke Pines, USA
| | - Dylon P Collins
- College of Medicine, Nova Southeastern University Dr. Kiran C. Patel College of Osteopathic Medicine, Fort Lauderdale, USA
| | - Stanley Dennison
- College of Medicine, Nova Southeastern University Dr. Kiran C. Patel College of Osteopathic Medicine, Fort Lauderdale, USA
| | - Lohitha Guntupalli
- Osteopathic Medicine, Nova Southeastern University Dr. Kiran C. Patel College of Osteopathic Medicine, Clearwater, USA
| | - Soham Choudhary
- Orthopedic Surgery, University of California, Davis, Davis, USA
| | - Augustine M Saiz
- Orthopedic Surgery, University of California (UC) Davis Health, Sacramento, USA
| | - Robert L Randall
- Orthopedic Surgery, University of California (UC) Davis Health, Sacramento, USA
| |
Collapse
|
6
|
Chen Z, Yang D, Li A, Sun L, Zhao J, Liu J, Liu L, Zhou X, Chen Y, Cai Y, Wu Z, Cheng K, Cai H, Tang M, Peng B, Wang X. Decoding surgical skill: an objective and efficient algorithm for surgical skill classification based on surgical gesture features -experimental studies. Int J Surg 2024; 110:1441-1449. [PMID: 38079605 PMCID: PMC10942222 DOI: 10.1097/js9.0000000000000975] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2023] [Accepted: 11/21/2023] [Indexed: 03/16/2024]
Abstract
BACKGROUND Various surgical skills lead to differences in patient outcomes and identifying poorly skilled surgeons with constructive feedback contributes to surgical quality improvement. The aim of the study was to develop an algorithm for evaluating surgical skills in laparoscopic cholecystectomy based on the features of elementary functional surgical gestures (Surgestures). MATERIALS AND METHODS Seventy-five laparoscopic cholecystectomy videos were collected from 33 surgeons in five hospitals. The phase of mobilization hepatocystic triangle and gallbladder dissection from the liver bed of each video were annotated with 14 Surgestures. The videos were grouped into competent and incompetent based on the quantiles of modified global operative assessment of laparoscopic skills (mGOALS). Surgeon-related information, clinical data, and intraoperative events were analyzed. Sixty-three Surgesture features were extracted to develop the surgical skill classification algorithm. The area under the receiver operating characteristic curve of the classification and the top features were evaluated. RESULTS Correlation analysis revealed that most perioperative factors had no significant correlation with mGOALS scores. The incompetent group has a higher probability of cholecystic vascular injury compared to the competent group (30.8 vs 6.1%, P =0.004). The competent group demonstrated fewer inefficient Surgestures, lower shift frequency, and a larger dissection-exposure ratio of Surgestures during the procedure. The area under the receiver operating characteristic curve of the classification algorithm achieved 0.866. Different Surgesture features contributed variably to overall performance and specific skill items. CONCLUSION The computer algorithm accurately classified surgeons with different skill levels using objective Surgesture features, adding insight into designing automatic laparoscopic surgical skill assessment tools with technical feedback.
Collapse
Affiliation(s)
- Zixin Chen
- Department of General Surgery, Division of Pancreatic Surgery
- West China School of Medicine, West China Hospital of Sichuan University
| | - Dewei Yang
- Chongqing University of Posts and Telecommunications, School of Advanced Manufacturing Engineering, Chongqing
| | - Ang Li
- Department of General Surgery, Division of Pancreatic Surgery
- Guang’an People’s Hospital, Guang’an
| | - Louzong Sun
- Department of Hepatobiliary Surgery, Zigong First People’s Hospital, Zigong
| | - Jifan Zhao
- Chengdu Withai Innovations Technology Company, Chengdu
| | - Jie Liu
- Chengdu Withai Innovations Technology Company, Chengdu
| | - Linxun Liu
- Department of General Surgery, Qinghai Provincial People’s Hospital, Xining, People’s Republic of China
| | - Xiaobo Zhou
- School of Biomedical Informatics, McGovern Medical School, University of Texas Health Science Center, Houston, USA
| | - Yonghua Chen
- Department of General Surgery, Division of Pancreatic Surgery
| | - Yunqiang Cai
- Department of General Surgery, Division of Pancreatic Surgery
| | - Zhong Wu
- Department of General Surgery, Division of Pancreatic Surgery
| | - Ke Cheng
- Department of General Surgery, Division of Pancreatic Surgery
| | - He Cai
- Department of General Surgery, Division of Pancreatic Surgery
| | - Ming Tang
- Department of General Surgery, Division of Pancreatic Surgery
- West China School of Medicine, West China Hospital of Sichuan University
| | - Bing Peng
- Department of General Surgery, Division of Pancreatic Surgery
| | - Xin Wang
- Department of General Surgery, Division of Pancreatic Surgery
| |
Collapse
|
7
|
Goldenberg MG. Surgical Artificial Intelligence in Urology: Educational Applications. Urol Clin North Am 2024; 51:105-115. [PMID: 37945096 DOI: 10.1016/j.ucl.2023.06.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2023]
Abstract
Surgical education has seen immense change recently. Increased demand for iterative evaluation of trainees from medical school to independent practice has led to the generation of an overwhelming amount of data related to an individual's competency. Artificial intelligence has been proposed as a solution to automate and standardize the ability of stakeholders to assess the technical and nontechnical abilities of a surgical trainee. In both the simulation and clinical environments, evidence supports the use of machine learning algorithms to both evaluate trainee skill and provide real-time and automated feedback, enabling a shortened learning curve for many key procedural skills and ensuring patient safety.
Collapse
Affiliation(s)
- Mitchell G Goldenberg
- Catherine & Joseph Aresty Department of Urology, USC Institute of Urology, University of Southern California, 1441 Eastlake Avenue, Suite 7416, Los Angeles, CA 90033, USA.
| |
Collapse
|
8
|
Kaoukabani G, Gokcal F, Fanta A, Liu X, Shields M, Stricklin C, Friedman A, Kudsi OY. A multifactorial evaluation of objective performance indicators and video analysis in the context of case complexity and clinical outcomes in robotic-assisted cholecystectomy. Surg Endosc 2023; 37:8540-8551. [PMID: 37789179 DOI: 10.1007/s00464-023-10432-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2023] [Accepted: 08/31/2023] [Indexed: 10/05/2023]
Abstract
BACKGROUND The increased digitization in robotic surgical procedures today enables surgeons to quantify their movements through data captured directly from the robotic system. These calculations, called objective performance indicators (OPIs), offer unprecedented detail into surgical performance. In this study, we link case- and surgical step-specific OPIs to case complexity, surgical experience and console utilization, and post-operative clinical complications across 87 robotic cholecystectomy (RC) cases. METHODS Videos of RCs performed by a principal surgeon with and without fellows were segmented into eight surgical steps and linked to patients' clinical data. Data for OPI calculations were extracted from an Intuitive Data Recorder and the da Vinci ® robotic system. RC cases were each assigned a Nassar and Parkland Grading score and categorized as standard or complex. OPIs were compared across complexity groups, console attributions, and post-surgical complication severities to determine objective relationships across variables. RESULTS Across cases, differences in camera control and head positioning metrics of the principal surgeon were observed when comparing standard and complex cases. Further, OPI differences across the principal surgeon and the fellow(s) were observed in standard cases and include differences in arm swapping, camera control, and clutching behaviors. Monopolar coagulation energy usage differences were also observed. Select surgical step duration differences were observed across complexities and console attributions, and additional surgical task analyses determine the adhesion removal and liver bed hemostasis steps to be the most impactful steps for case complexity and post-surgical complications, respectively. CONCLUSION This is the first study to establish the association between OPIs, case complexities, and clinical complications in RC. We identified OPI differences in intra-operative behaviors and post-surgical complications dependent on surgeon expertise and case complexity, opening the door for more standardized assessments of teaching cases, surgical behaviors and case complexities.
Collapse
Affiliation(s)
| | - Fahri Gokcal
- Good Samaritan Medical Center, Brockton, MA, USA
| | - Abeselom Fanta
- Applied Research, Intuitive Surgical Inc., Peachtree City, GA, USA
| | - Xi Liu
- Applied Research, Intuitive Surgical Inc., Peachtree City, GA, USA
| | - Mallory Shields
- Applied Research, Intuitive Surgical Inc., Peachtree City, GA, USA
| | | | | | | |
Collapse
|
9
|
de Marinis R, Marigi EM, Atwan Y, Yang L, Oeding JF, Gupta P, Pareek A, Sanchez-Sotelo J, Sperling JW. Current clinical applications of artificial intelligence in shoulder surgery: what the busy shoulder surgeon needs to know and what's coming next. JSES REVIEWS, REPORTS, AND TECHNIQUES 2023; 3:447-453. [PMID: 37928999 PMCID: PMC10625013 DOI: 10.1016/j.xrrt.2023.07.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 11/07/2023]
Abstract
Background Artificial intelligence (AI) is a continuously expanding field with the potential to transform a variety of industries-including health care-by providing automation, efficiency, precision, accuracy, and decision-making support for simple and complex tasks. Basic knowledge of the key features as well as limitations of AI is paramount to understand current developments in this field and to successfully apply them to shoulder surgery. The purpose of the present review is to provide an overview of AI within orthopedics and shoulder surgery exploring current and forthcoming AI applications. Methods PubMed and Scopus databases were searched to provide a narrative review of the most relevant literature on AI applications in shoulder surgery. Results Despite the enormous clinical and research potential of AI, orthopedic surgery has been a relatively late adopter of AI technologies. Image evaluation, surgical planning, aiding decision-making, and facilitating patient evaluations over time are some of the current areas of development with enormous opportunities to improve surgical practice, research, and education. Furthermore, the advancement of AI-driven strategies has the potential to create a more efficient medical system that may reduce the overall cost of delivering and implementing quality health care for patients with shoulder pathology. Conclusion AI is an expanding field with the potential for broad clinical and research applications in orthopedic surgery. Many challenges still need to be addressed to fully leverage the potential of AI to clinical practice and research such as privacy issues, data ownership, and external validation of the proposed models.
Collapse
Affiliation(s)
- Rodrigo de Marinis
- Department of Orthopedic Surgery, Mayo Clinic, Rochester, MN, USA
- Department of Orthopedic Surgery, Pontificia Universidad Católica de Chile, Santiago, Chile
- Shoulder and Elbow Unit, Hospital Dr. Sótero del Rio, Santiago, Chile
| | - Erick M. Marigi
- Department of Orthopedic Surgery, Mayo Clinic, Rochester, MN, USA
| | - Yousif Atwan
- Department of Orthopedic Surgery, Mayo Clinic, Rochester, MN, USA
| | - Linjun Yang
- Orthopedic Surgery Artificial Intelligence Lab (OSAIL), Mayo Clinic, Rochester, MN, USA
| | - Jacob F. Oeding
- Department of Orthopedic Surgery, Mayo Clinic, Rochester, MN, USA
| | - Puneet Gupta
- Department of Orthopaedic Surgery, George Washington University School of Medicine and Health Sciences, Washington, DC, USA
| | - Ayoosh Pareek
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, NY, USA
| | | | - John W. Sperling
- Department of Orthopedic Surgery, Mayo Clinic, Rochester, MN, USA
| |
Collapse
|
10
|
Moodi Ghalibaf A, Moghadasin M, Emadzadeh A, Mastour H. Psychometric properties of the persian version of the Medical Artificial Intelligence Readiness Scale for Medical Students (MAIRS-MS). BMC MEDICAL EDUCATION 2023; 23:577. [PMID: 37582816 PMCID: PMC10428571 DOI: 10.1186/s12909-023-04553-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/28/2023] [Accepted: 07/30/2023] [Indexed: 08/17/2023]
Abstract
INTRODUCTION There are numerous cases where artificial intelligence (AI) can be applied to improve the outcomes of medical education. The extent to which medical practitioners and students are ready to work and leverage this paradigm is unclear in Iran. This study investigated the psychometric properties of a Persian version of the Medical Artificial Intelligence Readiness Scale for Medical Students (MAIRS-MS) developed by Karaca, et al. in 2021. In future studies, the medical AI readiness for Iranian medical students could be investigated using this scale, and effective interventions might be planned and implemented according to the results. METHODS In this study, 502 medical students (mean age 22.66(± 2.767); 55% female) responded to the Persian questionnaire in an online survey. The original questionnaire was translated into Persian using a back translation procedure, and all participants completed the demographic component and the entire MAIRS-MS. Internal and external consistencies, factor analysis, construct validity, and confirmatory factor analysis were examined to analyze the collected data. A P ≤ 0.05 was considered as the level of statistical significance. RESULTS Four subscales emerged from the exploratory factor analysis (Cognition, Ability, Vision, and Ethics), and confirmatory factor analysis confirmed the four subscales. The Cronbach alpha value for internal consistency was 0.944 for the total scale and 0.886, 0.905, 0.865, and 0.856 for cognition, ability, vision, and ethics, respectively. CONCLUSIONS The Persian version of MAIRS-MS was fairly equivalent to the original one regarding the conceptual and linguistic aspects. This study also confirmed the validity and reliability of the Persian version of MAIRS-MS. Therefore, the Persian version can be a suitable and brief instrument to assess Iranian Medical Students' readiness for medical artificial intelligence.
Collapse
Affiliation(s)
- AmirAli Moodi Ghalibaf
- Student Research Committee, Faculty of Medicine, Birjand University of Medical Sciences, Birjand, Iran
| | - Maryam Moghadasin
- Department of Clinical Psychology, Faculty of Psychology and Education, Kharazmi University, Tehran, Iran
| | - Ali Emadzadeh
- Department of Medical Education, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran
| | - Haniye Mastour
- Department of Medical Education, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran.
| |
Collapse
|
11
|
Lum ZC. Can Artificial Intelligence Pass the American Board of Orthopaedic Surgery Examination? Orthopaedic Residents Versus ChatGPT. Clin Orthop Relat Res 2023; 481:1623-1630. [PMID: 37220190 PMCID: PMC10344569 DOI: 10.1097/corr.0000000000002704] [Citation(s) in RCA: 33] [Impact Index Per Article: 33.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Accepted: 04/28/2023] [Indexed: 05/25/2023]
Abstract
BACKGROUND Advances in neural networks, deep learning, and artificial intelligence (AI) have progressed recently. Previous deep learning AI has been structured around domain-specific areas that are trained on dataset-specific areas of interest that yield high accuracy and precision. A new AI model using large language models (LLM) and nonspecific domain areas, ChatGPT (OpenAI), has gained attention. Although AI has demonstrated proficiency in managing vast amounts of data, implementation of that knowledge remains a challenge. QUESTIONS/PURPOSES (1) What percentage of Orthopaedic In-Training Examination questions can a generative, pretrained transformer chatbot (ChatGPT) answer correctly? (2) How does that percentage compare with results achieved by orthopaedic residents of different levels, and if scoring lower than the 10th percentile relative to 5th-year residents is likely to correspond to a failing American Board of Orthopaedic Surgery score, is this LLM likely to pass the orthopaedic surgery written boards? (3) Does increasing question taxonomy affect the LLM's ability to select the correct answer choices? METHODS This study randomly selected 400 of 3840 publicly available questions based on the Orthopaedic In-Training Examination and compared the mean score with that of residents who took the test over a 5-year period. Questions with figures, diagrams, or charts were excluded, including five questions the LLM could not provide an answer for, resulting in 207 questions administered with raw score recorded. The LLM's answer results were compared with the Orthopaedic In-Training Examination ranking of orthopaedic surgery residents. Based on the findings of an earlier study, a pass-fail cutoff was set at the 10th percentile. Questions answered were then categorized based on the Buckwalter taxonomy of recall, which deals with increasingly complex levels of interpretation and application of knowledge; comparison was made of the LLM's performance across taxonomic levels and was analyzed using a chi-square test. RESULTS ChatGPT selected the correct answer 47% (97 of 207) of the time, and 53% (110 of 207) of the time it answered incorrectly. Based on prior Orthopaedic In-Training Examination testing, the LLM scored in the 40th percentile for postgraduate year (PGY) 1s, the eighth percentile for PGY2s, and the first percentile for PGY3s, PGY4s, and PGY5s; based on the latter finding (and using a predefined cutoff of the 10th percentile of PGY5s as the threshold for a passing score), it seems unlikely that the LLM would pass the written board examination. The LLM's performance decreased as question taxonomy level increased (it answered 54% [54 of 101] of Tax 1 questions correctly, 51% [18 of 35] of Tax 2 questions correctly, and 34% [24 of 71] of Tax 3 questions correctly; p = 0.034). CONCLUSION Although this general-domain LLM has a low likelihood of passing the orthopaedic surgery board examination, testing performance and knowledge are comparable to that of a first-year orthopaedic surgery resident. The LLM's ability to provide accurate answers declines with increasing question taxonomy and complexity, indicating a deficiency in implementing knowledge. CLINICAL RELEVANCE Current AI appears to perform better at knowledge and interpretation-based inquires, and based on this study and other areas of opportunity, it may become an additional tool for orthopaedic learning and education.
Collapse
|
12
|
Satapathy P, Hermis AH, Rustagi S, Pradhan KB, Padhi BK, Sah R. Artificial intelligence in surgical education and training: opportunities, challenges, and ethical considerations - correspondence. Int J Surg 2023; 109:1543-1544. [PMID: 37037597 PMCID: PMC10389387 DOI: 10.1097/js9.0000000000000387] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2023] [Accepted: 03/27/2023] [Indexed: 04/12/2023]
Affiliation(s)
| | - Alaa H. Hermis
- Nursing Department, Al-Mustaqbal University College, Hillah, Babylon, Iraq
| | - Sarvesh Rustagi
- Sarvesh Rustagi, School of Applied and Life Sciences, Dehradun, Uttarakhand
| | - Keerti B. Pradhan
- Department of Healthcare Management, Chitkara University Punjab, Patiala
| | - Bijaya K. Padhi
- Department of Community Medicine and School of Public Health, Postgraduate Institute of Medical Education and Research, Chandigarh
| | - Ranjit Sah
- Department of Microbiology, Dr. D.Y. Patil Medical College, Hospital and Research Centre
- Department of Public Health Dentistry, Dr. D.Y. Patil Dental College and Hospital, Dr. D.Y. Patil Vidyapeeth, Pune, Maharashtra, India
- Tribhuvan University Teaching Hospital, Kathmandu, Nepal
| |
Collapse
|
13
|
Colborn K, Brat G, Callcut R. Predictive Analytics and Artificial Intelligence in Surgery-Opportunities and Risks. JAMA Surg 2023; 158:337-338. [PMID: 36576823 PMCID: PMC10204072 DOI: 10.1001/jamasurg.2022.5444] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/29/2022]
Abstract
This Viewpoint discusses the opportunities and risks of using 3 main areas of artificial intelligence in surgery: computer vision, digital transformation at the point of care, and electronic health records data.
Collapse
Affiliation(s)
- Kathryn Colborn
- University of Colorado Denver, Anschutz Medical Campus, Aurora
| | - Gabriel Brat
- Beth Israel Deaconess Medical Center, Boston, Massachusetts
| | | |
Collapse
|
14
|
Chadebecq F, Lovat LB, Stoyanov D. Artificial intelligence and automation in endoscopy and surgery. Nat Rev Gastroenterol Hepatol 2023; 20:171-182. [PMID: 36352158 DOI: 10.1038/s41575-022-00701-y] [Citation(s) in RCA: 21] [Impact Index Per Article: 21.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 10/03/2022] [Indexed: 11/10/2022]
Abstract
Modern endoscopy relies on digital technology, from high-resolution imaging sensors and displays to electronics connecting configurable illumination and actuation systems for robotic articulation. In addition to enabling more effective diagnostic and therapeutic interventions, the digitization of the procedural toolset enables video data capture of the internal human anatomy at unprecedented levels. Interventional video data encapsulate functional and structural information about a patient's anatomy as well as events, activity and action logs about the surgical process. This detailed but difficult-to-interpret record from endoscopic procedures can be linked to preoperative and postoperative records or patient imaging information. Rapid advances in artificial intelligence, especially in supervised deep learning, can utilize data from endoscopic procedures to develop systems for assisting procedures leading to computer-assisted interventions that can enable better navigation during procedures, automation of image interpretation and robotically assisted tool manipulation. In this Perspective, we summarize state-of-the-art artificial intelligence for computer-assisted interventions in gastroenterology and surgery.
Collapse
Affiliation(s)
- François Chadebecq
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK
| | - Laurence B Lovat
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK
| | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK.
| |
Collapse
|
15
|
Belmar F, Gaete MI, Escalona G, Carnier M, Durán V, Villagrán I, Asbun D, Cortés M, Neyem A, Crovari F, Alseidi A, Varas J. Artificial intelligence in laparoscopic simulation: a promising future for large-scale automated evaluations. Surg Endosc 2022:10.1007/s00464-022-09576-1. [PMID: 36192656 PMCID: PMC9529161 DOI: 10.1007/s00464-022-09576-1] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2022] [Accepted: 08/22/2022] [Indexed: 12/01/2022]
Abstract
Introduction A limitation to expanding laparoscopic simulation training programs is the scarcity of expert evaluators. In 2019, a new digital platform for remote and asynchronous laparoscopic simulation training was validated. Through this platform, 369 trainees have been trained in 14 institutions across Latin America, collecting 6729 videos of laparoscopic training exercises. The use of artificial intelligence (AI) has recently emerged in surgical simulation, showing usefulness in training assessment, virtual reality scenarios, and laparoscopic virtual reality simulation. An AI algorithm to assess basic laparoscopic simulation training exercises was developed. This study aimed to analyze the agreement between this AI algorithm and expert evaluators in assessing basic laparoscopic-simulated training exercises. Methods The AI algorithm was trained using 400-bean drop (BD) and 480-peg transfer (PT) videos and tested using 64-BD and 43-PT randomly selected videos, not previously used to train the algorithm. The agreement between AI and expert evaluators from the digital platform (EE) was then analyzed. The exercises being assessed involve using laparoscopic graspers to move objects across an acrylic board without dropping any objects in a determined time (BD < 24 s, PT < 55 s). The AI algorithm can detect object movement, identify if objects have fallen, track grasper clamps location, and measure exercise time. Cohen’s Kappa test was used to evaluate the agreement between AI assessments and those performed by EE, using a pass/fail nomenclature based on the time to complete the exercise. Results After the algorithm was trained, 79.69% and 93.02% agreement were observed in BD and PT, respectively. The Kappa coefficients test observed for BD and PT were 0.59 (moderate agreement) and 0.86 (almost perfect agreement), respectively. Conclusion This first approach of AI use in basic laparoscopic skills simulated training assessment shows promising results, providing a preliminary framework to expand the use of AI to other basic laparoscopic skills exercises. Supplementary Information The online version contains supplementary material available at 10.1007/s00464-022-09576-1.
Collapse
Affiliation(s)
- Francisca Belmar
- Experimental Surgery and Simulation Center, Department of Digestive Surgery, Catholic University of Chile, Santiago, Chile
| | - María Inés Gaete
- Experimental Surgery and Simulation Center, Department of Digestive Surgery, Catholic University of Chile, Santiago, Chile
| | - Gabriel Escalona
- Experimental Surgery and Simulation Center, Department of Digestive Surgery, Catholic University of Chile, Santiago, Chile
| | - Martín Carnier
- Experimental Surgery and Simulation Center, Department of Digestive Surgery, Catholic University of Chile, Santiago, Chile
| | - Valentina Durán
- Experimental Surgery and Simulation Center, Department of Digestive Surgery, Catholic University of Chile, Santiago, Chile
| | - Ignacio Villagrán
- Experimental Surgery and Simulation Center, Department of Digestive Surgery, Catholic University of Chile, Santiago, Chile.,Faculty of Medicine, Catholic University of Chile, Santiago, Chile
| | - Domenech Asbun
- Hepatobiliary & Pancreatic Surgery, Miami Cancer Institute, Miami, USA
| | - Matías Cortés
- Experimental Surgery and Simulation Center, Department of Digestive Surgery, Catholic University of Chile, Santiago, Chile
| | - Andrés Neyem
- Department of Computer Science, Engineering Faculty, Catholic University of Chile, Santiago, Chile
| | - Fernando Crovari
- Experimental Surgery and Simulation Center, Department of Digestive Surgery, Catholic University of Chile, Santiago, Chile.,Faculty of Medicine, Catholic University of Chile, Santiago, Chile
| | - Adnan Alseidi
- Department of Surgery, University of California San Francisco, San Francisco, USA
| | - Julián Varas
- Experimental Surgery and Simulation Center, Department of Digestive Surgery, Catholic University of Chile, Santiago, Chile. .,Faculty of Medicine, Catholic University of Chile, Santiago, Chile.
| |
Collapse
|