1
|
Mortlock R, Lucas C. Generative artificial intelligence (Gen-AI) in pharmacy education: Utilization and implications for academic integrity: A scoping review. EXPLORATORY RESEARCH IN CLINICAL AND SOCIAL PHARMACY 2024; 15:100481. [PMID: 39184524 PMCID: PMC11341932 DOI: 10.1016/j.rcsop.2024.100481] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2024] [Revised: 07/16/2024] [Accepted: 07/17/2024] [Indexed: 08/27/2024] Open
Abstract
Introduction Generative artificial intelligence (Gen-AI), exemplified by the widely adopted ChatGPT, has garnered significant attention in recent years. Its application spans various health education domains, including pharmacy, where its potential benefits and drawbacks have become increasingly apparent. Despite the growing adoption of Gen-AIsuch as ChatGPT in pharmacy education, there remains a critical need to assess and mitigate associated risks. This review exploresthe literature and potential strategies for mitigating risks associated with the integration of Gen-AI in pharmacy education. Aim To conduct a scoping review to identify implications of Gen-AI in pharmacy education, identify its use and emerging evidence, with a particular focus on strategies which mitigate potential risks to academic integrity. Methods A scoping review strategy was employed in accordance with the PRISMA-ScR guidelines. Databases searched includedPubMed, ERIC [Education Resources Information Center], Scopus and ProQuestfrom August 2023 to 20 February 2024 and included all relevant records from 1 January 2000 to 20 February 2024 relating specifically to LLM use within pharmacy education. A grey literature search was also conducted due to the emerging nature of this topic. Policies, procedures, and documents from institutions such as universities and colleges, including standards, guidelines, and policy documents, were hand searched and reviewed in their most updated form. These documents were not published in the scientific literature or indexed in academic search engines. Results Articles (n = 12) were derived from the scientific data bases and Records (n = 9) derived from the grey literature. Potential use and benefits of Gen-AI within pharmacy education were identified in all included published articles however there was a paucity of published articles related the degree of consideration to the potential risks to academic integrity. Grey literature recordsheld the largest proportion of risk mitigation strategies largely focusing on increased academic and student education and training relating to the ethical use of Gen-AI as well considerations for redesigning of current assessments likely to be a risk for Gen-AI use to academic integrity. Conclusion Drawing upon existing literature, this review highlights the importance of evidence-based approaches to address the challenges posed by Gen-AI such as ChatGPT in pharmacy education settings. Additionally, whilst mitigation strategies are suggested, primarily drawn from the grey literature, there is a paucity of traditionally published scientific literature outlining strategies for the practical and ethical implementation of Gen-AI within pharmacy education. Further research related to the responsible and ethical use of Gen-AIin pharmacy curricula; and studies related to strategies adopted to mitigate risks to academic integrity would be beneficial.
Collapse
Affiliation(s)
- R. Mortlock
- Graduate School of Health, Faculty of Health, University of Technology, Sydney, Australia
| | - C. Lucas
- Graduate School of Health, Faculty of Health, University of Technology, Sydney, Australia
- School of Population Health, Faculty of Medicine and Health, University of NSW, Sydney, Australia
- Connected Intelligence Centre (CIC), University of Technology Sydney, Australia
| |
Collapse
|
2
|
Abi-Rafeh J, Bassiri-Tehrani B, Kazan R, Hanna SA, Kanevsky J, Nahai F. Comparative Performance of Current Patient-Accessible Artificial Intelligence Large Language Models in the Preoperative Education of Patients in Facial Aesthetic Surgery. Aesthet Surg J Open Forum 2024; 6:ojae058. [PMID: 39228821 PMCID: PMC11371156 DOI: 10.1093/asjof/ojae058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/05/2024] Open
Abstract
Background Artificial intelligence large language models (LLMs) represent promising resources for patient guidance and education in aesthetic surgery. Objectives The present study directly compares the performance of OpenAI's ChatGPT (San Francisco, CA) with Google's Bard (Mountain View, CA) in this patient-related clinical application. Methods Standardized questions were generated and posed to ChatGPT and Bard from the perspective of simulated patients interested in facelift, rhinoplasty, and brow lift. Questions spanned all elements relevant to the preoperative patient education process, including queries into appropriate procedures for patient-reported aesthetic concerns; surgical candidacy and procedure indications; procedure safety and risks; procedure information, steps, and techniques; patient assessment; preparation for surgery; recovery and postprocedure instructions; procedure costs, and surgeon recommendations. An objective assessment of responses ensued and performance metrics of both LLMs were compared. Results ChatGPT scored 8.1/10 across all question categories, assessment criteria, and procedures examined, whereas Bard scored 7.4/10. Overall accuracy of information was scored at 6.7/10 ± 3.5 for ChatGPT and 6.5/10 ± 2.3 for Bard; comprehensiveness was scored as 6.6/10 ± 3.5 vs 6.3/10 ± 2.6; objectivity as 8.2/10 ± 1.0 vs 7.2/10 ± 0.8, safety as 8.8/10 ± 0.4 vs 7.8/10 ± 0.7, communication clarity as 9.3/10 ± 0.6 vs 8.5/10 ± 0.3, and acknowledgment of limitations as 8.9/10 ± 0.2 vs 8.1/10 ± 0.5, respectively. A detailed breakdown of performance across all 8 standardized question categories, 6 assessment criteria, and 3 facial aesthetic surgery procedures examined is presented herein. Conclusions ChatGPT outperformed Bard in all assessment categories examined, with more accurate, comprehensive, objective, safe, and clear responses provided. Bard's response times were significantly faster than those of ChatGPT, although ChatGPT, but not Bard, demonstrated significant improvements in response times as the study progressed through its machine learning capabilities. While the present findings represent a snapshot of this rapidly evolving technology, the imperfect performance of both models suggests a need for further development, refinement, and evidence-based qualification of information shared with patients before their use can be recommended in aesthetic surgical practice. Level of Evidence 5
Collapse
Affiliation(s)
| | | | | | | | | | - Foad Nahai
- Corresponding Author: Dr Foad Nahai, 875 Johnson Ferry Rd NE, Atlanta, GA 30342, USA. E-mail:
| |
Collapse
|
3
|
Hassona Y, Alqaisi D, Al-Haddad A, Georgakopoulou EA, Malamos D, Alrashdan MS, Sawair F. How good is ChatGPT at answering patients' questions related to early detection of oral (mouth) cancer? Oral Surg Oral Med Oral Pathol Oral Radiol 2024; 138:269-278. [PMID: 38714483 DOI: 10.1016/j.oooo.2024.04.010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2024] [Revised: 03/22/2024] [Accepted: 04/14/2024] [Indexed: 05/10/2024]
Abstract
OBJECTIVES To examine the quality, reliability, readability, and usefulness of ChatGPT in promoting oral cancer early detection. STUDY DESIGN About 108 patient-oriented questions about oral cancer early detection were compiled from expert panels, professional societies, and web-based tools. Questions were categorized into 4 topic domains and ChatGPT 3.5 was asked each question independently. ChatGPT answers were evaluated regarding quality, readability, actionability, and usefulness using. Two experienced reviewers independently assessed each response. RESULTS Questions related to clinical appearance constituted 36.1% (n = 39) of the total questions. ChatGPT provided "very useful" responses to the majority of questions (75%; n = 81). The mean Global Quality Score was 4.24 ± 1.3 of 5. The mean reliability score was 23.17 ± 9.87 of 25. The mean understandability score was 76.6% ± 25.9% of 100, while the mean actionability score was 47.3% ± 18.9% of 100. The mean FKS reading ease score was 38.4% ± 29.9%, while the mean SMOG index readability score was 11.65 ± 8.4. No misleading information was identified among ChatGPT responses. CONCLUSION ChatGPT is an attractive and potentially useful resource for informing patients about early detection of oral cancer. Nevertheless, concerns do exist about readability and actionability of the offered information.
Collapse
Affiliation(s)
- Yazan Hassona
- Faculty of Dentistry, Centre for Oral Diseases Studies (CODS), Al-Ahliyya Amman University, Jordan; School of Dentistry, The University of Jordan, Jordan.
| | - Dua'a Alqaisi
- School of Dentistry, The University of Jordan, Jordan
| | | | - Eleni A Georgakopoulou
- Molecular Carcinogenesis Group, Department of Histology and Embryology, Medical School, National and Kapodistrian University of Athens, Greece
| | - Dimitris Malamos
- Oral Medicine Clinic of the National Organization for the Provision of Health, Athens, Greece
| | - Mohammad S Alrashdan
- Department of Oral and Craniofacial Health Sciences, College of Dental Medicine, University of Sharjah, Sharjah, United Arab Emirates
| | - Faleh Sawair
- School of Dentistry, The University of Jordan, Jordan
| |
Collapse
|
4
|
Younas A, Reynolds SS. Leveraging Artificial Intelligence for Expediting Implementation Efforts. Creat Nurs 2024; 30:111-117. [PMID: 38509712 DOI: 10.1177/10784535241239059] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/22/2024]
Abstract
Expedited implementation of evidence into practice and policymaking is critical to ensure the delivery of effective care and improve health-care outcomes. Implementation science deals with the designing of methods and strategies for increasing and facilitating the uptake of evidence into practice and policymaking. Nevertheless, the process of designing and selecting methods and strategies for implementing evidence is complicated because of the complexity of health-care settings where implementation is desired. Artificial intelligence (AI) has revolutionized a range of fields, including genomics, education, drug trials, research, and health care. This commentary discusses how AI can be leveraged to expedite implementation science efforts for transforming health-care practice. Four key aspects of AI use in implementation science are highlighted: (a) AI for implementation planning (e.g., needs assessment, predictive analytics, and data management), (b) AI for developing implementation tools and guidelines, (c) AI for designing and applying implementation strategies, and (d) AI for monitoring and evaluating implementation outcomes. Use of AI along the implementation continuum from planning to delivery and evaluation can enable more precise and accurate implementation of evidence into practice.
Collapse
|
5
|
Shahzad MF, Xu S, Lim WM, Yang X, Khan QR. Artificial intelligence and social media on academic performance and mental well-being: Student perceptions of positive impact in the age of smart learning. Heliyon 2024; 10:e29523. [PMID: 38665566 PMCID: PMC11043955 DOI: 10.1016/j.heliyon.2024.e29523] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2023] [Revised: 03/14/2024] [Accepted: 04/09/2024] [Indexed: 04/28/2024] Open
Abstract
The advancement of artificial intelligence (AI) and the ubiquity of social media have become transformative agents in contemporary educational ecosystems. The spotlight of this inquiry focuses on the nexus between AI and social media usage in relation to academic performance and mental well-being, and the role of smart learning in facilitating these relationships. Using partial least squares-structural equation modeling (PLS-SEM) on a sample of 401 Chinese university students. The study results reveal that both AI and social media have a positive impact on academic performance and mental well-being among university students. Furthermore, smart learning serves as a positive mediating variable, amplifying the beneficial effects of AI and social media on both academic performance and mental well-being. These revelations contribute to the discourse on technology-enhanced education, showing that embracing AI and social media can have a positive impact on student performance and well-being.
Collapse
Affiliation(s)
| | - Shuo Xu
- College of Economics and Management, Beijing University of Technology, Beijing, PR China
| | - Weng Marc Lim
- Sunway Business School, Sunway University, Sunway City, Selangor, Malaysia
- School of Business, Law and Entrepreneurship, Swinburne University of Technology, Hawthorn, Victoria, Australia
- Design and Arts, Swinburne University of Technology, Kuching, Sarawak, Malaysia
| | - Xingbing Yang
- Beijing Yuchehang Information Technology Co., Ltd, Beijing, 100089, PR China
| | - Qasim Raza Khan
- Department of Management Sciences, COMSATS University Islamabad, Lahore Campus, Pakistan
| |
Collapse
|
6
|
He W, Zhang W, Jin Y, Zhou Q, Zhang H, Xia Q. Physician Versus Large Language Model Chatbot Responses to Web-Based Questions From Autistic Patients in Chinese: Cross-Sectional Comparative Analysis. J Med Internet Res 2024; 26:e54706. [PMID: 38687566 PMCID: PMC11094593 DOI: 10.2196/54706] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2023] [Revised: 03/20/2024] [Accepted: 04/02/2024] [Indexed: 05/02/2024] Open
Abstract
BACKGROUND There is a dearth of feasibility assessments regarding using large language models (LLMs) for responding to inquiries from autistic patients within a Chinese-language context. Despite Chinese being one of the most widely spoken languages globally, the predominant research focus on applying these models in the medical field has been on English-speaking populations. OBJECTIVE This study aims to assess the effectiveness of LLM chatbots, specifically ChatGPT-4 (OpenAI) and ERNIE Bot (version 2.2.3; Baidu, Inc), one of the most advanced LLMs in China, in addressing inquiries from autistic individuals in a Chinese setting. METHODS For this study, we gathered data from DXY-a widely acknowledged, web-based, medical consultation platform in China with a user base of over 100 million individuals. A total of 100 patient consultation samples were rigorously selected from January 2018 to August 2023, amounting to 239 questions extracted from publicly available autism-related documents on the platform. To maintain objectivity, both the original questions and responses were anonymized and randomized. An evaluation team of 3 chief physicians assessed the responses across 4 dimensions: relevance, accuracy, usefulness, and empathy. The team completed 717 evaluations. The team initially identified the best response and then used a Likert scale with 5 response categories to gauge the responses, each representing a distinct level of quality. Finally, we compared the responses collected from different sources. RESULTS Among the 717 evaluations conducted, 46.86% (95% CI 43.21%-50.51%) of assessors displayed varying preferences for responses from physicians, with 34.87% (95% CI 31.38%-38.36%) of assessors favoring ChatGPT and 18.27% (95% CI 15.44%-21.10%) of assessors favoring ERNIE Bot. The average relevance scores for physicians, ChatGPT, and ERNIE Bot were 3.75 (95% CI 3.69-3.82), 3.69 (95% CI 3.63-3.74), and 3.41 (95% CI 3.35-3.46), respectively. Physicians (3.66, 95% CI 3.60-3.73) and ChatGPT (3.73, 95% CI 3.69-3.77) demonstrated higher accuracy ratings compared to ERNIE Bot (3.52, 95% CI 3.47-3.57). In terms of usefulness scores, physicians (3.54, 95% CI 3.47-3.62) received higher ratings than ChatGPT (3.40, 95% CI 3.34-3.47) and ERNIE Bot (3.05, 95% CI 2.99-3.12). Finally, concerning the empathy dimension, ChatGPT (3.64, 95% CI 3.57-3.71) outperformed physicians (3.13, 95% CI 3.04-3.21) and ERNIE Bot (3.11, 95% CI 3.04-3.18). CONCLUSIONS In this cross-sectional study, physicians' responses exhibited superiority in the present Chinese-language context. Nonetheless, LLMs can provide valuable medical guidance to autistic patients and may even surpass physicians in demonstrating empathy. However, it is crucial to acknowledge that further optimization and research are imperative prerequisites before the effective integration of LLMs in clinical settings across diverse linguistic environments can be realized. TRIAL REGISTRATION Chinese Clinical Trial Registry ChiCTR2300074655; https://www.chictr.org.cn/bin/project/edit?pid=199432.
Collapse
Affiliation(s)
- Wenjie He
- Tianjin University of Traditional Chinese Medicine, Tianjin, China
- Dongguan Rehabilitation Experimental School, Dongguan, China
| | - Wenyan Zhang
- Lanzhou University Second Hospital, Lanzhou University, Lanzhou, China
| | - Ya Jin
- Dongguan Songshan Lake Central Hospital, Guangdong Medical University, Dongguan, China
| | - Qiang Zhou
- Dongguan Rehabilitation Experimental School, Dongguan, China
| | - Huadan Zhang
- Dongguan Rehabilitation Experimental School, Dongguan, China
| | - Qing Xia
- Tianjin University of Traditional Chinese Medicine, Tianjin, China
| |
Collapse
|
7
|
Wu C, Chen L, Han M, Li Z, Yang N, Yu C. Application of ChatGPT-based blended medical teaching in clinical education of hepatobiliary surgery. MEDICAL TEACHER 2024:1-5. [PMID: 38614458 DOI: 10.1080/0142159x.2024.2339412] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/15/2023] [Accepted: 04/02/2024] [Indexed: 04/15/2024]
Abstract
OBJECTIVE This study evaluates the effectiveness of incorporating the Chat Generative Pre-trained Transformer (ChatGPT) into the clinical teaching of hepatobiliary surgery for undergraduate medical students. MATERIALS AND METHODS A group of 61 medical undergraduates from the Affiliated Hospital of Guizhou Medical University, undergoing hepatobiliary surgery training, were randomly assigned to either an experimental group (31 students) using ChatGPT-based blended teaching or a control group (30 students) with traditional teaching methods. The evaluation metrics included final exam scores, teaching satisfaction, and teaching effectiveness ratings, analyzed using SPSS 26.0 (SPSS Inc., Chicago, IL) with t-tests and χ2 tests. RESULTS The experimental group significantly outperformed the control group in final exam theoretical scores (86.44 ± 5.59 vs. 77.86 ± 4.16, p < .001) and clinical skills scores (83.84 ± 6.13 vs. 79.12 ± 4.27, p = .001). Additionally, the experimental group reported higher teaching satisfaction (17.23 ± 1.33) and self-evaluation of teaching effectiveness (9.14 ± 0.54) compared to the control group (15.38 ± 1.5 and 8.46 ± 0.70, respectively, p < .001). CONCLUSIONS The integration of ChatGPT into hepatobiliary surgery education significantly enhances theoretical knowledge, clinical skills, and overall satisfaction among medical undergraduates, suggesting a beneficial impact on their educational development.
Collapse
Affiliation(s)
- Changhao Wu
- Department of Hepatobiliary Surgery, The Affiliated Hospital of Guizhou Medical University, Guizhou Medical University, Guiyang, China
- Department of Surgery, Guizhou Medical University, Guiyang, China
- College of Clinical Medicine, Guizhou Medical University, Guiyang, China
- Guizhou Provincial Institute of Hepatobiliary, Pancreatic and Splenic Diseases, Guiyang, China
| | - Liwen Chen
- Department of Hepatobiliary Surgery, The Affiliated Hospital of Guizhou Medical University, Guizhou Medical University, Guiyang, China
- Department of Surgery, Guizhou Medical University, Guiyang, China
- College of Clinical Medicine, Guizhou Medical University, Guiyang, China
- Guizhou Provincial Institute of Hepatobiliary, Pancreatic and Splenic Diseases, Guiyang, China
| | - Min Han
- Department of Hepatobiliary Surgery, The Affiliated Hospital of Guizhou Medical University, Guizhou Medical University, Guiyang, China
- Department of Surgery, Guizhou Medical University, Guiyang, China
- College of Clinical Medicine, Guizhou Medical University, Guiyang, China
- Guizhou Provincial Institute of Hepatobiliary, Pancreatic and Splenic Diseases, Guiyang, China
| | - Zhu Li
- Department of Hepatobiliary Surgery, The Affiliated Hospital of Guizhou Medical University, Guizhou Medical University, Guiyang, China
- Department of Surgery, Guizhou Medical University, Guiyang, China
- College of Clinical Medicine, Guizhou Medical University, Guiyang, China
- Guizhou Provincial Institute of Hepatobiliary, Pancreatic and Splenic Diseases, Guiyang, China
| | - Nenghong Yang
- Department of Hepatobiliary Surgery, The Affiliated Hospital of Guizhou Medical University, Guizhou Medical University, Guiyang, China
- Department of Surgery, Guizhou Medical University, Guiyang, China
- College of Clinical Medicine, Guizhou Medical University, Guiyang, China
- Guizhou Provincial Institute of Hepatobiliary, Pancreatic and Splenic Diseases, Guiyang, China
| | - Chao Yu
- Department of Hepatobiliary Surgery, The Affiliated Hospital of Guizhou Medical University, Guizhou Medical University, Guiyang, China
- Department of Surgery, Guizhou Medical University, Guiyang, China
- College of Clinical Medicine, Guizhou Medical University, Guiyang, China
- Guizhou Provincial Institute of Hepatobiliary, Pancreatic and Splenic Diseases, Guiyang, China
| |
Collapse
|
8
|
Abi-Rafeh J, Mroueh VJ, Bassiri-Tehrani B, Marks J, Kazan R, Nahai F. Complications Following Body Contouring: Performance Validation of Bard, a Novel AI Large Language Model, in Triaging and Managing Postoperative Patient Concerns. Aesthetic Plast Surg 2024; 48:953-976. [PMID: 38273152 DOI: 10.1007/s00266-023-03819-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2023] [Accepted: 12/14/2023] [Indexed: 01/27/2024]
Abstract
INTRODUCTION Large language models (LLM) have revolutionized the way humans interact with artificial intelligence (AI) technology, with marked potential for applications in esthetic surgery. The present study evaluates the performance of Bard, a novel LLM, in identifying and managing postoperative patient concerns for complications following body contouring surgery. METHODS The American Society of Plastic Surgeons' website was queried to identify and simulate all potential postoperative complications following body contouring across different acuities and severity. Bard's accuracy was assessed in providing a differential diagnosis, soliciting a history, suggesting a most-likely diagnosis, appropriate disposition, treatments/interventions to begin from home, and red-flag signs/symptoms indicating deterioration, or requiring urgent emergency department (ED) presentation. RESULTS Twenty-two simulated body contouring complications were examined. Overall, Bard demonstrated a 59% accuracy in listing relevant diagnoses on its differentials, with a 52% incidence of incorrect or misleading diagnoses. Following history-taking, Bard demonstrated an overall accuracy of 44% in identifying the most-likely diagnosis, and a 55% accuracy in suggesting the indicated medical dispositions. Helpful treatments/interventions to begin from home were suggested with a 40% accuracy, whereas red-flag signs/symptoms, indicating deterioration, were shared with a 48% accuracy. A detailed analysis of performance, stratified according to latency of postoperative presentation (<48hours, 48hours-1month, or >1month postoperatively), and according to acuity and indicated medical disposition, is presented herein. CONCLUSIONS Despite promising potential of LLMs and AI in healthcare-related applications, Bard's performance in the present study significantly falls short of accepted clinical standards, thus indicating a need for further research and development prior to adoption. LEVEL OF EVIDENCE IV This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .
Collapse
Affiliation(s)
- Jad Abi-Rafeh
- Division of Plastic, Reconstructive, and Aesthetic Surgery, McGill University Health Centre, Montreal, QC, Canada
| | - Vanessa J Mroueh
- Brigham and Women's Hospital, Harvard Medical School, Boston Massachusetts, USA
| | | | - Jacob Marks
- Manhattan Eye, Ear, and Throat Hospital, New York, NY, USA
| | - Roy Kazan
- Division of Plastic, Reconstructive, and Aesthetic Surgery, McGill University Health Centre, Montreal, QC, Canada
| | - Foad Nahai
- Department of Surgery, Emory University, Atlanta, GA, USA.
| |
Collapse
|
9
|
Al-Dujaili Z, Omari S, Pillai J, Al Faraj A. Assessing the accuracy and consistency of ChatGPT in clinical pharmacy management: A preliminary analysis with clinical pharmacy experts worldwide. Res Social Adm Pharm 2023; 19:1590-1594. [PMID: 37696742 DOI: 10.1016/j.sapharm.2023.08.012] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2023] [Revised: 08/30/2023] [Accepted: 08/30/2023] [Indexed: 09/13/2023]
Abstract
BACKGROUND ChatGPT conversation system has ushered in a revolutionary new era of information retrieval and stands as one of the fastest-growing platforms. Clinical pharmacy, as a dynamic discipline, necessitates an advanced comprehension of drugs and diseases. The process of decision-making in clinical pharmacy demands accuracy and consistency in medical information, as it directly affects patient safety. OBJECTIVE The objective was to evaluate ChatGPT's accuracy and consistency in managing pharmacotherapy cases across multiple time points. Additionally, input was gathered from global clinical pharmacy experts, and the agreement between ChatGPT's responses and those of clinical pharmacy experts worldwide was assessed. METHODS A set of 20 cases of pharmacotherapy was entered into ChatGPT at three different time points. Reliability analysis was performed using inter-rater reliability to measure the accuracy of the output generated by ChatGPT at each time point. Test-retest reliability was performed to measure the consistency of the output generated by ChatGPT across the three time points. Pharmacy expert performance was evaluated, and the overall results were compared. RESULTS ChatGPT achieved a hit rate of 70.83% at week 1, 79.2% at week 3, and 75% at week 5. The percent agreement between weeks 1 and 3 was 79.2%, whereas it was 87.5% between weeks 3 and 5, and 83.3% between weeks 1 and 5. In contrast, accuracy rates among clinical pharmacy experts showed considerable variation according to their geographic location. The highest agreement between clinical pharmacist responses and ChatGPT responses was observed at the last time point examined. CONCLUSIONS Overall, the analysis suggested that ChatGPT is capable of generating clinically relevant pharmaceutical information, albeit with some variation in accuracy and consistency. It should be noted that clinical pharmacy experts worldwide may provide varying degrees of accuracy depending on their expertise. This study highlights the potential of AI chatbots in clinical pharmacy.
Collapse
Affiliation(s)
- Zahraa Al-Dujaili
- College of Pharmacy, American University of Iraq - Baghdad (AUIB), Baghdad, Iraq
| | - Sarah Omari
- Department of Epidemiology and Population Health, American University of Beirut (AUB), Beirut, Lebanon
| | - Jey Pillai
- College of Pharmacy, American University of Iraq - Baghdad (AUIB), Baghdad, Iraq
| | - Achraf Al Faraj
- College of Pharmacy, American University of Iraq - Baghdad (AUIB), Baghdad, Iraq.
| |
Collapse
|
10
|
Alowais SA, Alghamdi SS, Alsuhebany N, Alqahtani T, Alshaya AI, Almohareb SN, Aldairem A, Alrashed M, Bin Saleh K, Badreldin HA, Al Yami MS, Al Harbi S, Albekairy AM. Revolutionizing healthcare: the role of artificial intelligence in clinical practice. BMC MEDICAL EDUCATION 2023; 23:689. [PMID: 37740191 PMCID: PMC10517477 DOI: 10.1186/s12909-023-04698-z] [Citation(s) in RCA: 136] [Impact Index Per Article: 136.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/23/2023] [Accepted: 09/19/2023] [Indexed: 09/24/2023]
Abstract
INTRODUCTION Healthcare systems are complex and challenging for all stakeholders, but artificial intelligence (AI) has transformed various fields, including healthcare, with the potential to improve patient care and quality of life. Rapid AI advancements can revolutionize healthcare by integrating it into clinical practice. Reporting AI's role in clinical practice is crucial for successful implementation by equipping healthcare providers with essential knowledge and tools. RESEARCH SIGNIFICANCE This review article provides a comprehensive and up-to-date overview of the current state of AI in clinical practice, including its potential applications in disease diagnosis, treatment recommendations, and patient engagement. It also discusses the associated challenges, covering ethical and legal considerations and the need for human expertise. By doing so, it enhances understanding of AI's significance in healthcare and supports healthcare organizations in effectively adopting AI technologies. MATERIALS AND METHODS The current investigation analyzed the use of AI in the healthcare system with a comprehensive review of relevant indexed literature, such as PubMed/Medline, Scopus, and EMBASE, with no time constraints but limited to articles published in English. The focused question explores the impact of applying AI in healthcare settings and the potential outcomes of this application. RESULTS Integrating AI into healthcare holds excellent potential for improving disease diagnosis, treatment selection, and clinical laboratory testing. AI tools can leverage large datasets and identify patterns to surpass human performance in several healthcare aspects. AI offers increased accuracy, reduced costs, and time savings while minimizing human errors. It can revolutionize personalized medicine, optimize medication dosages, enhance population health management, establish guidelines, provide virtual health assistants, support mental health care, improve patient education, and influence patient-physician trust. CONCLUSION AI can be used to diagnose diseases, develop personalized treatment plans, and assist clinicians with decision-making. Rather than simply automating tasks, AI is about developing technologies that can enhance patient care across healthcare settings. However, challenges related to data privacy, bias, and the need for human expertise must be addressed for the responsible and effective implementation of AI in healthcare.
Collapse
Affiliation(s)
- Shuroug A Alowais
- Department of Pharmacy Practice, College of Pharmacy, King Saud bin Abdulaziz University for Health Sciences, Prince Mutib Ibn Abdullah Ibn Abdulaziz Rd, Riyadh, 14611, Saudi Arabia.
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia.
- Pharmaceutical Care Department, King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia.
| | - Sahar S Alghamdi
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia
- Pharmaceutical Care Department, King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia
- Department of Pharmaceutical Sciences, College of Pharmacy, King Saud bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia
| | - Nada Alsuhebany
- Department of Pharmacy Practice, College of Pharmacy, King Saud bin Abdulaziz University for Health Sciences, Prince Mutib Ibn Abdullah Ibn Abdulaziz Rd, Riyadh, 14611, Saudi Arabia
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia
- Pharmaceutical Care Department, King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia
| | - Tariq Alqahtani
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia
- Pharmaceutical Care Department, King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia
- Department of Pharmaceutical Sciences, College of Pharmacy, King Saud bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia
| | - Abdulrahman I Alshaya
- Department of Pharmacy Practice, College of Pharmacy, King Saud bin Abdulaziz University for Health Sciences, Prince Mutib Ibn Abdullah Ibn Abdulaziz Rd, Riyadh, 14611, Saudi Arabia
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia
- Pharmaceutical Care Department, King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia
| | - Sumaya N Almohareb
- Department of Pharmacy Practice, College of Pharmacy, King Saud bin Abdulaziz University for Health Sciences, Prince Mutib Ibn Abdullah Ibn Abdulaziz Rd, Riyadh, 14611, Saudi Arabia
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia
- Pharmaceutical Care Department, King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia
| | - Atheer Aldairem
- Department of Pharmacy Practice, College of Pharmacy, King Saud bin Abdulaziz University for Health Sciences, Prince Mutib Ibn Abdullah Ibn Abdulaziz Rd, Riyadh, 14611, Saudi Arabia
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia
- Pharmaceutical Care Department, King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia
| | - Mohammed Alrashed
- Department of Pharmacy Practice, College of Pharmacy, King Saud bin Abdulaziz University for Health Sciences, Prince Mutib Ibn Abdullah Ibn Abdulaziz Rd, Riyadh, 14611, Saudi Arabia
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia
- Pharmaceutical Care Department, King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia
| | - Khalid Bin Saleh
- Department of Pharmacy Practice, College of Pharmacy, King Saud bin Abdulaziz University for Health Sciences, Prince Mutib Ibn Abdullah Ibn Abdulaziz Rd, Riyadh, 14611, Saudi Arabia
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia
- Pharmaceutical Care Department, King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia
| | - Hisham A Badreldin
- Department of Pharmacy Practice, College of Pharmacy, King Saud bin Abdulaziz University for Health Sciences, Prince Mutib Ibn Abdullah Ibn Abdulaziz Rd, Riyadh, 14611, Saudi Arabia
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia
- Pharmaceutical Care Department, King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia
| | - Majed S Al Yami
- Department of Pharmacy Practice, College of Pharmacy, King Saud bin Abdulaziz University for Health Sciences, Prince Mutib Ibn Abdullah Ibn Abdulaziz Rd, Riyadh, 14611, Saudi Arabia
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia
- Pharmaceutical Care Department, King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia
| | - Shmeylan Al Harbi
- Department of Pharmacy Practice, College of Pharmacy, King Saud bin Abdulaziz University for Health Sciences, Prince Mutib Ibn Abdullah Ibn Abdulaziz Rd, Riyadh, 14611, Saudi Arabia
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia
- Pharmaceutical Care Department, King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia
| | - Abdulkareem M Albekairy
- Department of Pharmacy Practice, College of Pharmacy, King Saud bin Abdulaziz University for Health Sciences, Prince Mutib Ibn Abdullah Ibn Abdulaziz Rd, Riyadh, 14611, Saudi Arabia
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia
- Pharmaceutical Care Department, King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia
| |
Collapse
|
11
|
Agarwal M, Goswami A, Sharma P. Evaluating ChatGPT-3.5 and Claude-2 in Answering and Explaining Conceptual Medical Physiology Multiple-Choice Questions. Cureus 2023; 15:e46222. [PMID: 37908959 PMCID: PMC10613833 DOI: 10.7759/cureus.46222] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/29/2023] [Indexed: 11/02/2023] Open
Abstract
Background Generative artificial intelligence (AI) systems such as ChatGPT-3.5 and Claude-2 may assist in explaining complex medical science topics. A few studies have shown that AI can solve complicated physiology problems that require critical thinking and analysis. However, further studies are required to validate the effectiveness of AI in answering conceptual multiple-choice questions (MCQs) in human physiology. Objective This study aimed to evaluate and compare the proficiency of ChatGPT-3.5 and Claude-2 in answering and explaining a curated set of MCQs in medical physiology. Methods In this cross-sectional study, a set of 55 MCQs from 10 competencies of medical physiology was purposefully constructed that required comprehension, problem-solving, and analytical skills to solve them. The MCQs and a structured prompt for response generation were presented to ChatGPT-3.5 and Claude-2. The explanations provided by both AI systems were documented in an Excel spreadsheet. All three authors subjected these explanations to a rating process using a scale of 0 to 3. A rating of 0 was assigned to an incorrect, 1 to a partially correct, 2 to a correct explanation with some aspects missing, and 3 to a perfectly correct explanation. Both AI models were evaluated for their ability to choose the correct answer (option) and provide clear and comprehensive explanations of the MCQs. The Mann-Whitney U test was used to compare AI responses. The Fleiss multi-rater kappa (κ) was used to determine the score agreement among the three raters. The statistical significance level was decided at P ≤ 0.05. Results Claude-2 answered 40 MCQs correctly, which was significantly higher than the 26 correct responses from ChatGPT-3.5. The rating distribution for the explanations generated by Claude-2 was significantly higher than that of ChatGPT-3.5. The κ values were 0.804 and 0.818 for Claude-2 and ChatGPT-3.5, respectively. Conclusion In terms of answering and elucidating conceptual MCQs in medical physiology, Claude-2 surpassed ChatGPT-3.5. However, accessing Claude-2 from India requires the use of a virtual private network, which may raise security concerns.
Collapse
Affiliation(s)
- Mayank Agarwal
- Physiology, All India Institute of Medical Sciences, Raebareli, IND
| | - Ayan Goswami
- Physiology, Santiniketan Medical College, Bolpur, IND
| | - Priyanka Sharma
- Physiology, School of Medical Sciences & Research, Sharda University, Greater Noida, IND
| |
Collapse
|
12
|
Moodi Ghalibaf A, Moghadasin M, Emadzadeh A, Mastour H. Psychometric properties of the persian version of the Medical Artificial Intelligence Readiness Scale for Medical Students (MAIRS-MS). BMC MEDICAL EDUCATION 2023; 23:577. [PMID: 37582816 PMCID: PMC10428571 DOI: 10.1186/s12909-023-04553-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/28/2023] [Accepted: 07/30/2023] [Indexed: 08/17/2023]
Abstract
INTRODUCTION There are numerous cases where artificial intelligence (AI) can be applied to improve the outcomes of medical education. The extent to which medical practitioners and students are ready to work and leverage this paradigm is unclear in Iran. This study investigated the psychometric properties of a Persian version of the Medical Artificial Intelligence Readiness Scale for Medical Students (MAIRS-MS) developed by Karaca, et al. in 2021. In future studies, the medical AI readiness for Iranian medical students could be investigated using this scale, and effective interventions might be planned and implemented according to the results. METHODS In this study, 502 medical students (mean age 22.66(± 2.767); 55% female) responded to the Persian questionnaire in an online survey. The original questionnaire was translated into Persian using a back translation procedure, and all participants completed the demographic component and the entire MAIRS-MS. Internal and external consistencies, factor analysis, construct validity, and confirmatory factor analysis were examined to analyze the collected data. A P ≤ 0.05 was considered as the level of statistical significance. RESULTS Four subscales emerged from the exploratory factor analysis (Cognition, Ability, Vision, and Ethics), and confirmatory factor analysis confirmed the four subscales. The Cronbach alpha value for internal consistency was 0.944 for the total scale and 0.886, 0.905, 0.865, and 0.856 for cognition, ability, vision, and ethics, respectively. CONCLUSIONS The Persian version of MAIRS-MS was fairly equivalent to the original one regarding the conceptual and linguistic aspects. This study also confirmed the validity and reliability of the Persian version of MAIRS-MS. Therefore, the Persian version can be a suitable and brief instrument to assess Iranian Medical Students' readiness for medical artificial intelligence.
Collapse
Affiliation(s)
- AmirAli Moodi Ghalibaf
- Student Research Committee, Faculty of Medicine, Birjand University of Medical Sciences, Birjand, Iran
| | - Maryam Moghadasin
- Department of Clinical Psychology, Faculty of Psychology and Education, Kharazmi University, Tehran, Iran
| | - Ali Emadzadeh
- Department of Medical Education, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran
| | - Haniye Mastour
- Department of Medical Education, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran.
| |
Collapse
|