1
|
Sallam M, Al-Mahzoum K, Almutawaa RA, Alhashash JA, Dashti RA, AlSafy DR, Almutairi RA, Barakat M. The performance of OpenAI ChatGPT-4 and Google Gemini in virology multiple-choice questions: a comparative analysis of English and Arabic responses. BMC Res Notes 2024; 17:247. [PMID: 39228001 PMCID: PMC11373487 DOI: 10.1186/s13104-024-06920-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2024] [Accepted: 08/28/2024] [Indexed: 09/05/2024] Open
Abstract
OBJECTIVE The integration of artificial intelligence (AI) in healthcare education is inevitable. Understanding the proficiency of generative AI in different languages to answer complex questions is crucial for educational purposes. The study objective was to compare the performance ChatGPT-4 and Gemini in answering Virology multiple-choice questions (MCQs) in English and Arabic, while assessing the quality of the generated content. Both AI models' responses to 40 Virology MCQs were assessed for correctness and quality based on the CLEAR tool designed for evaluation of AI-generated content. The MCQs were classified into lower and higher cognitive categories based on the revised Bloom's taxonomy. The study design considered the METRICS checklist for the design and reporting of generative AI-based studies in healthcare. RESULTS ChatGPT-4 and Gemini performed better in English compared to Arabic, with ChatGPT-4 consistently surpassing Gemini in correctness and CLEAR scores. ChatGPT-4 led Gemini with 80% vs. 62.5% correctness in English compared to 65% vs. 55% in Arabic. For both AI models, superior performance in lower cognitive domains was reported. Both ChatGPT-4 and Gemini exhibited potential in educational applications; nevertheless, their performance varied across languages highlighting the importance of continued development to ensure the effective AI integration in healthcare education globally.
Collapse
Affiliation(s)
- Malik Sallam
- Department of Pathology, Microbiology and Forensic Medicine, School of Medicine, The University of Jordan, Amman, 11942, Jordan.
- Department of Clinical Laboratories and Forensic Medicine, Jordan University Hospital, Queen Rania Al-Abdullah Street-Aljubeiha, P.O. Box: 13046, Amman, 11942, Jordan.
| | | | | | | | | | | | | | - Muna Barakat
- Department of Clinical Pharmacy and Therapeutics, Faculty of Pharmacy, Applied Science Private University, Amman, 11931, Jordan
| |
Collapse
|
2
|
Quah B, Zheng L, Sng TJH, Yong CW, Islam I. Reliability of ChatGPT in automated essay scoring for dental undergraduate examinations. BMC MEDICAL EDUCATION 2024; 24:962. [PMID: 39227811 PMCID: PMC11373238 DOI: 10.1186/s12909-024-05881-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/04/2024] [Accepted: 08/09/2024] [Indexed: 09/05/2024]
Abstract
BACKGROUND This study aimed to answer the research question: How reliable is ChatGPT in automated essay scoring (AES) for oral and maxillofacial surgery (OMS) examinations for dental undergraduate students compared to human assessors? METHODS Sixty-nine undergraduate dental students participated in a closed-book examination comprising two essays at the National University of Singapore. Using pre-created assessment rubrics, three assessors independently performed manual essay scoring, while one separate assessor performed AES using ChatGPT (GPT-4). Data analyses were performed using the intraclass correlation coefficient and Cronbach's α to evaluate the reliability and inter-rater agreement of the test scores among all assessors. The mean scores of manual versus automated scoring were evaluated for similarity and correlations. RESULTS A strong correlation was observed for Question 1 (r = 0.752-0.848, p < 0.001) and a moderate correlation was observed between AES and all manual scorers for Question 2 (r = 0.527-0.571, p < 0.001). Intraclass correlation coefficients of 0.794-0.858 indicated excellent inter-rater agreement, and Cronbach's α of 0.881-0.932 indicated high reliability. For Question 1, the mean AES scores were similar to those for manual scoring (p > 0.05), and there was a strong correlation between AES and manual scores (r = 0.829, p < 0.001). For Question 2, AES scores were significantly lower than manual scores (p < 0.001), and there was a moderate correlation between AES and manual scores (r = 0.599, p < 0.001). CONCLUSION This study shows the potential of ChatGPT for essay marking. However, an appropriate rubric design is essential for optimal reliability. With further validation, the ChatGPT has the potential to aid students in self-assessment or large-scale marking automated processes.
Collapse
Affiliation(s)
- Bernadette Quah
- Faculty of Dentistry, National University of Singapore, Singapore, Singapore
- Discipline of Oral and Maxillofacial Surgery, National University Centre for Oral Health, 9 Lower Kent Ridge Road, Singapore, Singapore
| | - Lei Zheng
- Faculty of Dentistry, National University of Singapore, Singapore, Singapore
- Discipline of Oral and Maxillofacial Surgery, National University Centre for Oral Health, 9 Lower Kent Ridge Road, Singapore, Singapore
| | - Timothy Jie Han Sng
- Faculty of Dentistry, National University of Singapore, Singapore, Singapore
- Discipline of Oral and Maxillofacial Surgery, National University Centre for Oral Health, 9 Lower Kent Ridge Road, Singapore, Singapore
| | - Chee Weng Yong
- Faculty of Dentistry, National University of Singapore, Singapore, Singapore
- Discipline of Oral and Maxillofacial Surgery, National University Centre for Oral Health, 9 Lower Kent Ridge Road, Singapore, Singapore
| | - Intekhab Islam
- Faculty of Dentistry, National University of Singapore, Singapore, Singapore.
- Discipline of Oral and Maxillofacial Surgery, National University Centre for Oral Health, 9 Lower Kent Ridge Road, Singapore, Singapore.
| |
Collapse
|
3
|
Cálem J, Moreira C, Jorge J. Intelligent systems in healthcare: A systematic survey of explainable user interfaces. Comput Biol Med 2024; 180:108908. [PMID: 39067152 DOI: 10.1016/j.compbiomed.2024.108908] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2024] [Revised: 07/05/2024] [Accepted: 07/15/2024] [Indexed: 07/30/2024]
Abstract
With radiology shortages affecting over half of the global population, the potential of artificial intelligence to revolutionize medical diagnosis and treatment is ever more important. However, lacking trust from medical professionals hinders the widespread adoption of AI models in health sciences. Explainable AI (XAI) aims to increase trust and understanding of black box models by identifying biases and providing transparent explanations. This is the first survey that explores explainable user interfaces (XUI) from a medical domain perspective, analysing the visualization and interaction methods employed in current medical XAI systems. We analysed 42 explainable interfaces following the PRISMA methodology, emphasizing the critical role of effectively conveying information to users as part of the explanation process. We contribute a taxonomy of interface design properties and identify five distinct clusters of research papers. Future research directions include contestability in medical decision support, counterfactual explanations for images, and leveraging Large Language Models to enhance XAI interfaces in healthcare.
Collapse
Affiliation(s)
- João Cálem
- Instituto Superior Técnico, Universidade de Lisboa, Portugal; INESC-ID, Portugal.
| | - Catarina Moreira
- Data Science Institute, University of Technology Sydney, Australia; INESC-ID, Portugal
| | - Joaquim Jorge
- Instituto Superior Técnico, Universidade de Lisboa, Portugal; INESC-ID, Portugal
| |
Collapse
|
4
|
Mortlock R, Lucas C. Generative artificial intelligence (Gen-AI) in pharmacy education: Utilization and implications for academic integrity: A scoping review. EXPLORATORY RESEARCH IN CLINICAL AND SOCIAL PHARMACY 2024; 15:100481. [PMID: 39184524 PMCID: PMC11341932 DOI: 10.1016/j.rcsop.2024.100481] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2024] [Revised: 07/16/2024] [Accepted: 07/17/2024] [Indexed: 08/27/2024] Open
Abstract
Introduction Generative artificial intelligence (Gen-AI), exemplified by the widely adopted ChatGPT, has garnered significant attention in recent years. Its application spans various health education domains, including pharmacy, where its potential benefits and drawbacks have become increasingly apparent. Despite the growing adoption of Gen-AIsuch as ChatGPT in pharmacy education, there remains a critical need to assess and mitigate associated risks. This review exploresthe literature and potential strategies for mitigating risks associated with the integration of Gen-AI in pharmacy education. Aim To conduct a scoping review to identify implications of Gen-AI in pharmacy education, identify its use and emerging evidence, with a particular focus on strategies which mitigate potential risks to academic integrity. Methods A scoping review strategy was employed in accordance with the PRISMA-ScR guidelines. Databases searched includedPubMed, ERIC [Education Resources Information Center], Scopus and ProQuestfrom August 2023 to 20 February 2024 and included all relevant records from 1 January 2000 to 20 February 2024 relating specifically to LLM use within pharmacy education. A grey literature search was also conducted due to the emerging nature of this topic. Policies, procedures, and documents from institutions such as universities and colleges, including standards, guidelines, and policy documents, were hand searched and reviewed in their most updated form. These documents were not published in the scientific literature or indexed in academic search engines. Results Articles (n = 12) were derived from the scientific data bases and Records (n = 9) derived from the grey literature. Potential use and benefits of Gen-AI within pharmacy education were identified in all included published articles however there was a paucity of published articles related the degree of consideration to the potential risks to academic integrity. Grey literature recordsheld the largest proportion of risk mitigation strategies largely focusing on increased academic and student education and training relating to the ethical use of Gen-AI as well considerations for redesigning of current assessments likely to be a risk for Gen-AI use to academic integrity. Conclusion Drawing upon existing literature, this review highlights the importance of evidence-based approaches to address the challenges posed by Gen-AI such as ChatGPT in pharmacy education settings. Additionally, whilst mitigation strategies are suggested, primarily drawn from the grey literature, there is a paucity of traditionally published scientific literature outlining strategies for the practical and ethical implementation of Gen-AI within pharmacy education. Further research related to the responsible and ethical use of Gen-AIin pharmacy curricula; and studies related to strategies adopted to mitigate risks to academic integrity would be beneficial.
Collapse
Affiliation(s)
- R. Mortlock
- Graduate School of Health, Faculty of Health, University of Technology, Sydney, Australia
| | - C. Lucas
- Graduate School of Health, Faculty of Health, University of Technology, Sydney, Australia
- School of Population Health, Faculty of Medicine and Health, University of NSW, Sydney, Australia
- Connected Intelligence Centre (CIC), University of Technology Sydney, Australia
| |
Collapse
|
5
|
Bains SS, Dubin JA, Hameed D, Sax OC, Douglas S, Mont MA, Nace J, Delanois RE. Use and Application of Large Language Models for Patient Questions Following Total Knee Arthroplasty. J Arthroplasty 2024; 39:2289-2294. [PMID: 38490569 DOI: 10.1016/j.arth.2024.03.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/15/2023] [Revised: 03/06/2024] [Accepted: 03/07/2024] [Indexed: 03/17/2024] Open
Abstract
BACKGROUND A consumer-focused health care model not only allows unprecedented access to information, but equally warrants consideration of the appropriateness of providing accurate patient health information. Nurses play a large role in influencing patient satisfaction following total knee arthroplasty (TKA), but they come at a cost. A specific natural language artificial intelligence (AI) model, ChatGPT (Chat Generative Pre-trained Transformer), has accumulated over 100 million users within months of launching. As such, we aimed to compare: (1) orthopaedic surgeons' evaluation of the appropriateness of the answers to the most frequently asked patient questions after TKA; and (2) patients' comfort level in answering their postoperative questions by using answers provided by arthroplasty-trained nurses and ChatGPT. METHODS We prospectively created 60 questions based on the most commonly asked patient questions following TKA. There were 3 fellowship-trained surgeons who assessed the answers provided by arthroplasty-trained nurses and ChatGPT-4 to each of the questions. The surgeons graded each set of responses based on clinical judgment as: (1) "appropriate," (2) "inappropriate" if the response contained inappropriate information, or (3) "unreliable," if the responses provided inconsistent content. Patients' comfort level and trust in AI were assessed using Research Electronic Data Capture (REDCap) hosted at our local hospital. RESULTS The surgeons graded 44 out of 60 (73.3%) responses for the arthroplasty-trained nurses and 44 out of 60 (73.3%) for ChatGPT to be "appropriate." There were 4 responses graded "inappropriate" and one response graded "unreliable" provided by the nurses. For the ChatGPT response, there were 5 responses graded "inappropriate" and no responses graded "unreliable." There were 136 patients (53.8%) who were more comfortable with the answers provided by ChatGPT compared to 86 patients (34.0%) who preferred the answers from arthroplasty-trained nurses. Of the 253 patients, 233 (92.1%) were uncertain if they would trust AI to answer their postoperative questions. There were 127 patients (50.2%) who answered that if they knew the previous answer was provided by ChatGPT, their comfort level in trusting the answer would change. CONCLUSIONS One potential use of ChatGPT can be found in providing appropriate answers to patient questions after TKA. At our institution, cost expenditures can potentially be minimized while maintaining patient satisfaction. Inevitably, successful implementation is dependent on the ability to provide information that is credible and in accordance with the objectives of both physicians and patients. LEVEL OF EVIDENCE III.
Collapse
Affiliation(s)
- Sandeep S Bains
- Rubin Institute for Advanced Orthopedics, LifeBridge Health, Sinai Hospital of Baltimore, Baltimore, Maryland
| | - Jeremy A Dubin
- Rubin Institute for Advanced Orthopedics, LifeBridge Health, Sinai Hospital of Baltimore, Baltimore, Maryland
| | - Daniel Hameed
- Rubin Institute for Advanced Orthopedics, LifeBridge Health, Sinai Hospital of Baltimore, Baltimore, Maryland
| | - Oliver C Sax
- Rubin Institute for Advanced Orthopedics, LifeBridge Health, Sinai Hospital of Baltimore, Baltimore, Maryland
| | - Scott Douglas
- Rubin Institute for Advanced Orthopedics, LifeBridge Health, Sinai Hospital of Baltimore, Baltimore, Maryland
| | - Michael A Mont
- Rubin Institute for Advanced Orthopedics, LifeBridge Health, Sinai Hospital of Baltimore, Baltimore, Maryland
| | - James Nace
- Rubin Institute for Advanced Orthopedics, LifeBridge Health, Sinai Hospital of Baltimore, Baltimore, Maryland
| | - Ronald E Delanois
- Rubin Institute for Advanced Orthopedics, LifeBridge Health, Sinai Hospital of Baltimore, Baltimore, Maryland
| |
Collapse
|
6
|
Alnaim N, AlSanad DS, Albelali S, Almulhem M, Almuhanna AF, Attar RW, Alsahli M, Albagmi S, Bakhshwain AM, Almazrou S, Almutairi SA, AboAlsamh HM, Arif WM, Alsadhan AA, Alsedrah IT, Alanezi F, Alibrahim D, Alqahtani NS. Effectiveness of ChatGPT in remote learning environments: An empirical study with medical students in Saudi Arabia. Nutr Health 2024:2601060241273596. [PMID: 39150345 DOI: 10.1177/02601060241273596] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/17/2024]
Abstract
Purpose: This study aims to assess the effectiveness of ChatGPT in remote learning among medical students. Methods: This cross-sectional survey study recruited 386 medical students from three public universities in Saudi Arabia. Participants completed an online questionnaire designed to assess perceptions of ChatGPT's effectiveness in remote learning. The questionnaire included Likert scale questions to evaluate various aspects of ChatGPT's support in remote learning, such as personalized learning, language and communication skills, and interactive quizzing. Data were analyzed using SPSS, employing descriptive statistics, independent samples t-tests, one-way ANOVA, and Cronbach's alpha to evaluate reliability. Results: Participants mostly used ChatGPT on a weekly (43.2%) or daily (48.7%) basis, primarily on personal computers (62.5%). Mean scores for ChatGPT's support in remote learning were high for personalized learning (4.35), language and communication skills (4.23), and interactive quizzing and assessments (4.01). Statistically significant differences were found based on gender for interactive quizzing (p = .0177) and continuity of education (p = .0122). Conclusion: Despite certain challenges and variations in perceptions based on gender and education level, the overwhelmingly positive attitudes toward ChatGPT highlight its potential as a valuable tool in medical education.
Collapse
Affiliation(s)
- Norah Alnaim
- Department of Computer, College of Science and Humanities, Imam Abdulrahman Bin Faisal University, Jubail, Saudi Arabia
| | | | - Salma Albelali
- Department of Computer, College of Science and Humanities, Imam Abdulrahman Bin Faisal University, Jubail, Saudi Arabia
| | - Munerah Almulhem
- Department of Mathematics, College of Science and Humanities, Imam Abdulrahman Bin Faisal University, Jubail, Saudi Arabia
| | - Afnan Fahd Almuhanna
- Radiology Department, College of Medicine, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
| | - Razaz Waheeb Attar
- Management Department, College of Business Administration, Princess Nourah Bint Abdulrahman University, Riyadh, Saudi Arabia
| | - Mohammed Alsahli
- Health Informatics Department, College of Health Science, Saudi Electronic University, Riyadh, Saudi Arabia
| | - Salem Albagmi
- Prince Sultan Military College of Health Sciences, Dhahran, Saudi Arabia
| | | | - Saja Almazrou
- Clinical Pharmacy Department, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia
| | | | | | - Wejdan M Arif
- Department of Radiological Sciences, College of Applied Medical Sciences, King Saud University, Riyadh, Saudi Arabia
| | | | - Ibrahim Tawfeeq Alsedrah
- College of Business Administration, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
| | - Fahad Alanezi
- Department Management Information Systems, College of Business Administration, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
| | - Daneah Alibrahim
- College of Dentistry, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
| | - Nasser S Alqahtani
- Community Health Department, Northern Border University, Arar, Saudi Arabia
| |
Collapse
|
7
|
Fatima A, Shafique MA, Alam K, Fadlalla Ahmed TK, Mustafa MS. ChatGPT in medicine: A cross-disciplinary systematic review of ChatGPT's (artificial intelligence) role in research, clinical practice, education, and patient interaction. Medicine (Baltimore) 2024; 103:e39250. [PMID: 39121303 PMCID: PMC11315549 DOI: 10.1097/md.0000000000039250] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Accepted: 07/19/2024] [Indexed: 08/11/2024] Open
Abstract
BACKGROUND ChatGPT, a powerful AI language model, has gained increasing prominence in medicine, offering potential applications in healthcare, clinical decision support, patient communication, and medical research. This systematic review aims to comprehensively assess the applications of ChatGPT in healthcare education, research, writing, patient communication, and practice while also delineating potential limitations and areas for improvement. METHOD Our comprehensive database search retrieved relevant papers from PubMed, Medline and Scopus. After the screening process, 83 studies met the inclusion criteria. This review includes original studies comprising case reports, analytical studies, and editorials with original findings. RESULT ChatGPT is useful for scientific research and academic writing, and assists with grammar, clarity, and coherence. This helps non-English speakers and improves accessibility by breaking down linguistic barriers. However, its limitations include probable inaccuracy and ethical issues, such as bias and plagiarism. ChatGPT streamlines workflows and offers diagnostic and educational potential in healthcare but exhibits biases and lacks emotional sensitivity. It is useful in inpatient communication, but requires up-to-date data and faces concerns about the accuracy of information and hallucinatory responses. CONCLUSION Given the potential for ChatGPT to transform healthcare education, research, and practice, it is essential to approach its adoption in these areas with caution due to its inherent limitations.
Collapse
Affiliation(s)
- Afia Fatima
- Department of Medicine, Jinnah Sindh Medical University, Karachi, Pakistan
| | | | - Khadija Alam
- Department of Medicine, Liaquat National Medical College, Karachi, Pakistan
| | | | | |
Collapse
|
8
|
Mohammed M, Kumar N, Zawiah M, Al-Ashwal FY, Bala AA, Lawal BK, Wada AS, Halboup A, Muhammad S, Ahmad R, Sha'aban A. Psychometric Properties and Assessment of Knowledge, Attitude, and Practice Towards ChatGPT in Pharmacy Practice and Education: a Study Protocol. J Racial Ethn Health Disparities 2024; 11:2284-2293. [PMID: 37428357 DOI: 10.1007/s40615-023-01696-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2023] [Revised: 06/17/2023] [Accepted: 06/21/2023] [Indexed: 07/11/2023]
Abstract
ChatGPT represents an advanced conversational artificial intelligence (AI), providing a powerful tool for generating human-like responses that could change pharmacy prospects. This protocol aims to describe the development, validation, and utilization of a tool to assess the knowledge, attitude, and practice towards ChatGPT (KAP-C) in pharmacy practice and education. The development and validation process of the KAP-C tool will include a comprehensive literature search to identify relevant constructs, content validation by a panel of experts for items relevancy using content validity index (CVI) and face validation by sample participants for items clarity using face validity index (FVI), readability and difficulty index using the Flesch-Kincaid Readability Test, Gunning Fog Index, or Simple Measure of Gobbledygook (SMOG), assessment of reliability using internal consistency (Cronbach's alpha), and exploratory factor analysis (EFA) to determine the underlying factor structures (eigenvalues, scree plot analysis, factor loadings, and varimax). The second phase will utilize the validated KAP-C tool to conduct KAP surveys among pharmacists and pharmacy students in selected low- and middle-income countries (LMICs) (Nigeria, Pakistan, and Yemen). The final data will be analyzed descriptively using frequencies, percentages, mean (standard deviation) or median (interquartile range), and inferential statistics like Chi-square or regression analyses using IBM SPSS version 28. A p<0.05 will be considered statistically significant. ChatGPT holds the potential to revolutionize pharmacy practice and education. This study will highlight the psychometric properties of the KAP-C tool that assesses the knowledge, attitude, and practice towards ChatGPT in pharmacy practice and education. The findings will contribute to the potential ethical integration of ChatGPT into pharmacy practice and education in LMICs, serve as a reference to other economies, and provide valuable evidence for leveraging AI advancements in pharmacy.
Collapse
Affiliation(s)
- Mustapha Mohammed
- Department of Clinical Pharmacy and Pharmacy Practice, Faculty of Pharmaceutical Sciences, Ahmadu Bello University, Zaria, Kaduna, Nigeria.
- School of Pharmaceutical Sciences, Universiti Sains Malaysia, Gelugor, Pulau Pinang, Malaysia.
- Vice President for Medical and Health Science Office, QU Health, Qatar University, Doha, Qatar.
| | - Narendar Kumar
- Department of Pharmacy Practice, Faculty of Pharmacy, University of Sindh Jamshoro, Sindh, Pakistan
| | - Mohammed Zawiah
- School of Pharmaceutical Sciences, Universiti Sains Malaysia, Gelugor, Pulau Pinang, Malaysia
- Department of Pharmacy Practice, Faculty of Clinical Pharmacy, Hodeidah University, Al Hodeidah, Yemen
| | - Fahmi Y Al-Ashwal
- Department of Pharmacy, Al-Maarif University College, Anbar, 31001, Iraq
| | - Auwal Adam Bala
- Department of Pharmacology, College of Medicine and Health Sciences, Federal University Dutse, Dutse, Jigawa, Nigeria
| | - Basira Kankia Lawal
- Department of Clinical Pharmacy and Pharmacy Management, Kaduna State University, Kaduna, Nigeria
| | - Abubakar Sadiq Wada
- Department of Pharmacology and Therapeutics, Bayero University, Kano, Nigeria
| | - Abdulsalam Halboup
- School of Pharmaceutical Sciences, Universiti Sains Malaysia, Gelugor, Pulau Pinang, Malaysia
- Department of Clinical Pharmacy and Pharmacy Practice, University of Science and Technology, Sana'a, Yemen
| | | | - Rabbiya Ahmad
- School of Pharmaceutical Sciences, Universiti Sains Malaysia, Gelugor, Pulau Pinang, Malaysia
| | - Abubakar Sha'aban
- Division of Population Medicine, School of Medicine, Cardiff University, Heath Park, Cardiff, CF14 4YS, Wales, UK
| |
Collapse
|
9
|
Hadi A, Tran E, Nagarajan B, Kirpalani A. Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians. PLoS One 2024; 19:e0307383. [PMID: 39083523 PMCID: PMC11290643 DOI: 10.1371/journal.pone.0307383] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2023] [Accepted: 07/03/2024] [Indexed: 08/02/2024] Open
Abstract
BACKGROUND ChatGPT is a large language model (LLM) trained on over 400 billion words from books, articles, and websites. Its extensive training draws from a large database of information, making it valuable as a diagnostic aid. Moreover, its capacity to comprehend and generate human language allows medical trainees to interact with it, enhancing its appeal as an educational resource. This study aims to investigate ChatGPT's diagnostic accuracy and utility in medical education. METHODS 150 Medscape case challenges (September 2021 to January 2023) were inputted into ChatGPT. The primary outcome was the number (%) of cases for which the answer given was correct. Secondary outcomes included diagnostic accuracy, cognitive load, and quality of medical information. A qualitative content analysis was also conducted to assess its responses. RESULTS ChatGPT answered 49% (74/150) cases correctly. It had an overall accuracy of 74%, a precision of 48.67%, sensitivity of 48.67%, specificity of 82.89%, and an AUC of 0.66. Most answers were considered low cognitive load 51% (77/150) and most answers were complete and relevant 52% (78/150). DISCUSSION ChatGPT in its current form is not accurate as a diagnostic tool. ChatGPT does not necessarily give factual correctness, despite the vast amount of information it was trained on. Based on our qualitative analysis, ChatGPT struggles with the interpretation of laboratory values, imaging results, and may overlook key information relevant to the diagnosis. However, it still offers utility as an educational tool. ChatGPT was generally correct in ruling out a specific differential diagnosis and providing reasonable next diagnostic steps. Additionally, answers were easy to understand, showcasing a potential benefit in simplifying complex concepts for medical learners. Our results should guide future research into harnessing ChatGPT's potential educational benefits, such as simplifying medical concepts and offering guidance on differential diagnoses and next steps.
Collapse
Affiliation(s)
- Ali Hadi
- Department of Paediatrics, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
| | - Edward Tran
- Department of Paediatrics, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
| | - Branavan Nagarajan
- Department of Paediatrics, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
| | - Amrit Kirpalani
- Department of Paediatrics, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
- Division of Nephrology, Children’s Hospital, London Health Sciences Centre, London, Ontario, Canada
| |
Collapse
|
10
|
Zhou J, He X, Sun L, Xu J, Chen X, Chu Y, Zhou L, Liao X, Zhang B, Afvari S, Gao X. Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4. Nat Commun 2024; 15:5649. [PMID: 38969632 PMCID: PMC11226626 DOI: 10.1038/s41467-024-50043-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2023] [Accepted: 06/26/2024] [Indexed: 07/07/2024] Open
Abstract
Large language models (LLMs) are seen to have tremendous potential in advancing medical diagnosis recently, particularly in dermatological diagnosis, which is a very important task as skin and subcutaneous diseases rank high among the leading contributors to the global burden of nonfatal diseases. Here we present SkinGPT-4, which is an interactive dermatology diagnostic system based on multimodal large language models. We have aligned a pre-trained vision transformer with an LLM named Llama-2-13b-chat by collecting an extensive collection of skin disease images (comprising 52,929 publicly available and proprietary images) along with clinical concepts and doctors' notes, and designing a two-step training strategy. We have quantitatively evaluated SkinGPT-4 on 150 real-life cases with board-certified dermatologists. With SkinGPT-4, users could upload their own skin photos for diagnosis, and the system could autonomously evaluate the images, identify the characteristics and categories of the skin conditions, perform in-depth analysis, and provide interactive treatment recommendations.
Collapse
Affiliation(s)
- Juexiao Zhou
- Computer Science Program, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia
- Computational Bioscience Research Center, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia
- DermAssure, LLC, New York, NY, USA
| | - Xiaonan He
- Emergency Critical Care Center, Beijing AnZhen Hospital, Affiliated to Capital Medical University, Beijing, China.
| | - Liyuan Sun
- Department of Dermatology, Beijing AnZhen Hospital, Affiliated to Capital Medical University, Beijing, China
| | - Jiannan Xu
- Department of Dermatology, Beijing AnZhen Hospital, Affiliated to Capital Medical University, Beijing, China
| | - Xiuying Chen
- Computer Science Program, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia
- Computational Bioscience Research Center, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia
| | - Yuetan Chu
- Computer Science Program, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia
- Computational Bioscience Research Center, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia
| | - Longxi Zhou
- Computer Science Program, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia
- Computational Bioscience Research Center, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia
| | - Xingyu Liao
- Computer Science Program, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia
- Computational Bioscience Research Center, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia
| | - Bin Zhang
- Computer Science Program, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia
- Computational Bioscience Research Center, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia
| | - Shawn Afvari
- DermAssure, LLC, New York, NY, USA
- Department of Dermatology, Brigham and Women's Hospital, Harvard University, Boston, MA, USA
- School of Medicine, New York Medical College, Valhalla, NY, USA
| | - Xin Gao
- Computer Science Program, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia.
- Computational Bioscience Research Center, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, Kingdom of Saudi Arabia.
| |
Collapse
|
11
|
Al‐Anezi FM. Exploring the use of ChatGPT as a virtual health coach for chronic disease management. Learn Health Syst 2024; 8:e10406. [PMID: 39036525 PMCID: PMC11257053 DOI: 10.1002/lrh2.10406] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2023] [Revised: 12/09/2023] [Accepted: 12/19/2023] [Indexed: 07/23/2024] Open
Abstract
Introduction ChatGPT has been widely researched for its potential in gealthcare applications. However, its efficcy as a virtual health coach is one of the important areas, which can significantly contribute to the sustainablility in healthcare operations, especially in managing critical illnesses. Therefore, this study aims to analyze the use of ChatGPT as a virtual health coach for chronic disease managemet. Methods This study used a quasi-experimental design because ChatGPT is a relatively new technology and few people have experience with it. Patients who were receiving care outside of the hospital were included. Semi-structured interviews were conducted after a 2-week period in which participants used ChatGPT to search for health information about chronic disease management. Thirty-nine outpatients were interviewed and thematic analysis was used to analyze the interview data. Results The findings suggested both opportunities and challenges of using ChatGPT as a virtual health coach for chronic disease management. The major opportunities identified included life-long learning, improved health literacy, cost-effectiveness, behavioral change support, scalability, and accessibility. The major challenges identified included limited physical examination, lack of human connection, legal and ethical complications, and lack of accuracy and reliability. Conclusion ChatGPT-based technologies may serve as a supplementary or intermediate support system. However, such applications for managing chronic diseases must protect privacy and promote both short- and long-term positive outcomes.
Collapse
Affiliation(s)
- Fahad M. Al‐Anezi
- Department of Management Information SystemsImam Abdulrahman Bin Faisal UniversityDammamSaudi Arabia
| |
Collapse
|
12
|
Xu R, Wang Z. Generative artificial intelligence in healthcare from the perspective of digital media: Applications, opportunities and challenges. Heliyon 2024; 10:e32364. [PMID: 38975200 PMCID: PMC11225727 DOI: 10.1016/j.heliyon.2024.e32364] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2024] [Accepted: 06/03/2024] [Indexed: 07/09/2024] Open
Abstract
Introduction The emergence and application of generative artificial intelligence/large language models (hereafter GenAI LLMs) have the potential for significant impact on the healthcare industry. However, there is currently a lack of systematic research on GenAI LLMs in healthcare based on reliable data. This article aims to conduct an exploratory study of the application of GenAI LLMs (i.e., ChatGPT) in healthcare from the perspective of digital media (i.e., online news), including the application scenarios, potential opportunities, and challenges. Methods This research used thematic qualitative text analysis in five steps: firstly, developing main topical categories based on relevant articles; secondly, encoding the search keywords using these categories; thirdly, conducting searches for news articles via Google ; fourthly, encoding the sub-categories using the elaborate category system; and finally, conducting category-based analysis and presenting the results. Natural language processing techniques, including the TermRaider and AntConc tool, were applied in the aforementioned steps to assist in text qualitative analysis. Additionally, this study built a framework, using for analyzing the above three topics, from the perspective of five different stakeholders, including healthcare demanders and providers. Results This study summarizes 26 applications (e.g., provide medical advice, provide diagnosis and triage recommendations, provide mental health support, etc.), 21 opportunities (e.g., make healthcare more accessible, reduce healthcare costs, improve patients care, etc.), and 17 challenges (e.g., generate inaccurate/misleading/wrong answers, raise privacy concerns, lack of transparency, etc.), and analyzes the reasons for the formation of these key items and the links between the three research topics. Conclusions The application of GenAI LLMs in healthcare is primarily focused on transforming the way healthcare demanders access medical services (i.e., making it more intelligent, refined, and humane) and optimizing the processes through which healthcare providers offer medical services (i.e., simplifying, ensuring timeliness, and reducing errors). As the application becomes more widespread and deepens, GenAI LLMs is expected to have a revolutionary impact on traditional healthcare service models, but it also inevitably raises ethical and security concerns. Furthermore, GenAI LLMs applied in healthcare is still in the initial stage, which can be accelerated from a specific healthcare field (e.g., mental health) or a specific mechanism (e.g., GenAI LLMs' economic benefits allocation mechanism applied to healthcare) with empirical or clinical research.
Collapse
Affiliation(s)
- Rui Xu
- School of Economics, Guangdong University of Technology, Guangzhou, China
| | - Zhong Wang
- School of Economics, Guangdong University of Technology, Guangzhou, China
- Key Laboratory of Digital Economy and Data Governance, Guangdong University of Technology, Guangzhou, China
| |
Collapse
|
13
|
Zhou B, Mui LG. Utilising chatbots in clinical nursing education: Application and obstacles. J Clin Nurs 2024; 33:2362-2363. [PMID: 38407407 DOI: 10.1111/jocn.17089] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2024] [Accepted: 02/20/2024] [Indexed: 02/27/2024]
Affiliation(s)
- Bo Zhou
- Faculty of Medicine, Bioscience and Nursing, MAHSA University, Jenjarom, Malaysia
| | - Lim Gek Mui
- Faculty of Medicine, Bioscience and Nursing, MAHSA University, Jenjarom, Malaysia
| |
Collapse
|
14
|
Bharatha A, Ojeh N, Fazle Rabbi AM, Campbell MH, Krishnamurthy K, Layne-Yarde RNA, Kumar A, Springer DCR, Connell KL, Majumder MAA. Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of Bloom's Taxonomy. ADVANCES IN MEDICAL EDUCATION AND PRACTICE 2024; 15:393-400. [PMID: 38751805 PMCID: PMC11094742 DOI: 10.2147/amep.s457408] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 12/31/2023] [Accepted: 05/01/2024] [Indexed: 05/18/2024]
Abstract
Introduction This research investigated the capabilities of ChatGPT-4 compared to medical students in answering MCQs using the revised Bloom's Taxonomy as a benchmark. Methods A cross-sectional study was conducted at The University of the West Indies, Barbados. ChatGPT-4 and medical students were assessed on MCQs from various medical courses using computer-based testing. Results The study included 304 MCQs. Students demonstrated good knowledge, with 78% correctly answering at least 90% of the questions. However, ChatGPT-4 achieved a higher overall score (73.7%) compared to students (66.7%). Course type significantly affected ChatGPT-4's performance, but revised Bloom's Taxonomy levels did not. A detailed association check between program levels and Bloom's taxonomy levels for correct answers by ChatGPT-4 showed a highly significant correlation (p<0.001), reflecting a concentration of "remember-level" questions in preclinical and "evaluate-level" questions in clinical courses. Discussion The study highlights ChatGPT-4's proficiency in standardized tests but indicates limitations in clinical reasoning and practical skills. This performance discrepancy suggests that the effectiveness of artificial intelligence (AI) varies based on course content. Conclusion While ChatGPT-4 shows promise as an educational tool, its role should be supplementary, with strategic integration into medical education to leverage its strengths and address limitations. Further research is needed to explore AI's impact on medical education and student performance across educational levels and courses.
Collapse
Affiliation(s)
- Ambadasu Bharatha
- Faculty of Medical Sciences, The University of the West Indies, Bridgetown, Barbados
| | - Nkemcho Ojeh
- Faculty of Medical Sciences, The University of the West Indies, Bridgetown, Barbados
| | | | - Michael H Campbell
- Faculty of Medical Sciences, The University of the West Indies, Bridgetown, Barbados
| | | | | | - Alok Kumar
- Faculty of Medical Sciences, The University of the West Indies, Bridgetown, Barbados
| | - Dale C R Springer
- Faculty of Medical Sciences, The University of the West Indies, Bridgetown, Barbados
| | - Kenneth L Connell
- Faculty of Medical Sciences, The University of the West Indies, Bridgetown, Barbados
| | | |
Collapse
|
15
|
Yüce A, Erkurt N, Yerli M, Misir A. The Potential of ChatGPT for High-Quality Information in Patient Education for Sports Surgery. Cureus 2024; 16:e58874. [PMID: 38800159 PMCID: PMC11116739 DOI: 10.7759/cureus.58874] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/23/2024] [Indexed: 05/29/2024] Open
Abstract
BACKGROUND AND OBJECTIVE Artificial intelligence (AI) advancements continue to have a profound impact on modern society, driving significant innovation and development across various fields. We sought to appraise the reliability of the information offered by Chat Generative Pre-Trained Transformer (ChatGPT) regarding diseases commonly associated with sports surgery. We hypothesized that ChatGPT could offer high-quality information on sports-related diseases and be used in patient education. METHODS On September 11, 2023, specific sports surgery-related diseases were identified to ask ChatGPT-4 (personal communication, March 4, 2023). The informative texts provided by ChatGPT were recorded by a non-observer senior orthopedic surgeon for this study. Ten texts provided by ChatGPT related to sports surgery diseases were evaluated blindly by two observers. Observers assessed and scored these texts based on the sports surgery-specific scoring (SSSS) and DISCERN criteria. The precision of the disease-related information offered by ChatGPT was evaluated. RESULTS The calculated average DISCERN score of the texts in the study was 44.75 points and the average SSSS score was 13.3 points. In the interclass correlation coefficient analysis of the measurements made by the observers, the agreement was found to be excellent (0.989; p < 0.001). CONCLUSION ChatGPT has the potential to be used in patient education for sports surgery-related diseases. The potential to provide quality information in this regard seems to be an advantage.
Collapse
Affiliation(s)
- Ali Yüce
- Department of Orthopedics and Traumatology, Prof. Dr. Cemil Taşçıoğlu City Hospital, Istanbul, TUR
| | - Nazım Erkurt
- Department of Orthopedics and Traumatology, Prof. Dr. Cemil Taşçıoğlu City Hospital, Istanbul, TUR
| | - Mustafa Yerli
- Department of Orthopedics and Traumatology, Prof. Dr. Cemil Taşçıoğlu City Hospital, Istanbul, TUR
| | - Abdulhamit Misir
- Department of Orthopedics and Traumatology, Bahcesehir University Göztepe Medicalpark Hospital, Istanbul, TUR
| |
Collapse
|
16
|
Jačisko J, Veselý V, Chang KV, Özçakar L. (How) ChatGPT-Artificial Intelligence Thinks It Can Help/Harm Physiatry. Am J Phys Med Rehabil 2024; 103:346-349. [PMID: 38112589 DOI: 10.1097/phm.0000000000002370] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/21/2023]
Abstract
ABSTRACT ChatGPT is a chatbot that is based on the generative pretrained transformer architecture as an artificial inteligence-based large language model. Its widespread use in healthcare practice, research, and education seems to be (increasingly) inevitable. Also considering the relevant limitations regarding privacy, ethics, bias, legal, and validity, in this article, its use as a supplement (for sure not as a substitute for physicians) is discussed in light of the recent literature. Particularly, the "opinion" of ChatGPT about how it can help/harm physiatry is exemplified.
Collapse
Affiliation(s)
- Jakub Jačisko
- From the Department of Rehabilitation and Sports Medicine, Second Faculty of Medicine, Charles University and University Hospital Motol, Prague, Czech Republic (JJ, VV); Department of Physical Medicine and Rehabilitation, National Taiwan University Hospital, Bei-Hu Branch, Taipei, Taiwan (K-VC); and Department of Physical and Rehabilitation Medicine, Hacettepe University Medical School, Ankara, Turkey (LO)
| | | | | | | |
Collapse
|
17
|
Lin KC, Chen TA, Lin MH, Chen YC, Chen TJ. Integration and Assessment of ChatGPT in Medical Case Reporting: A Multifaceted Approach. Eur J Investig Health Psychol Educ 2024; 14:888-901. [PMID: 38667812 PMCID: PMC11049282 DOI: 10.3390/ejihpe14040057] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2024] [Revised: 03/18/2024] [Accepted: 03/19/2024] [Indexed: 04/28/2024] Open
Abstract
ChatGPT, a large language model, has gained significance in medical writing, particularly in case reports that document the course of an illness. This article explores the integration of ChatGPT and how ChatGPT shapes the process, product, and politics of medical writing in the real world. We conducted a bibliometric analysis on case reports utilizing ChatGPT and indexed in PubMed, encompassing publication information. Furthermore, an in-depth analysis was conducted to categorize the applications and limitations of ChatGPT and the publication trend of application categories. A total of 66 case reports utilizing ChatGPT were identified, with a predominant preference for the online version and English input by the authors. The prevalent application categories were information retrieval and content generation. Notably, this trend remained consistent across different months. Within the subset of 32 articles addressing ChatGPT limitations in case report writing, concerns related to inaccuracies and a lack of clinical context were prominently emphasized. This pointed out the important role of clinical thinking and professional expertise, representing the foundational tenets of medical education, while also accentuating the distinction between physicians and generative artificial intelligence.
Collapse
Affiliation(s)
- Kuan-Chen Lin
- Department of Family Medicine, Taipei Veterans General Hospital, No. 201, Sec. 2, Shih-Pai Road, Taipei 11217, Taiwan; (K.-C.L.); (T.-A.C.); (M.-H.L.)
| | - Tsung-An Chen
- Department of Family Medicine, Taipei Veterans General Hospital, No. 201, Sec. 2, Shih-Pai Road, Taipei 11217, Taiwan; (K.-C.L.); (T.-A.C.); (M.-H.L.)
| | - Ming-Hwai Lin
- Department of Family Medicine, Taipei Veterans General Hospital, No. 201, Sec. 2, Shih-Pai Road, Taipei 11217, Taiwan; (K.-C.L.); (T.-A.C.); (M.-H.L.)
- School of Medicine, National Yang Ming Chiao Tung University, Taipei 30010, Taiwan
| | - Yu-Chun Chen
- Department of Family Medicine, Taipei Veterans General Hospital, No. 201, Sec. 2, Shih-Pai Road, Taipei 11217, Taiwan; (K.-C.L.); (T.-A.C.); (M.-H.L.)
- School of Medicine, National Yang Ming Chiao Tung University, Taipei 30010, Taiwan
- Institute of Hospital and Health Care Administration, School of Medicine, National Yang Ming Chiao Tung University, Taipei 30010, Taiwan
- Big Data Center, Taipei Veterans General Hospital, Taipei 11217, Taiwan
| | - Tzeng-Ji Chen
- Department of Family Medicine, Taipei Veterans General Hospital Hsinchu Branch, No. 81, Sec. 1, Zhongfeng Road, Zhudong Township, Hsinchu 310403, Taiwan
- Department of Post-Baccalaureate Medicine, National Chung Hsing University, No. 145, Xingda Road, South District, Taichung 402202, Taiwan
| |
Collapse
|
18
|
Abbas A, Rehman MS, Rehman SS. Comparing the Performance of Popular Large Language Models on the National Board of Medical Examiners Sample Questions. Cureus 2024; 16:e55991. [PMID: 38606229 PMCID: PMC11007479 DOI: 10.7759/cureus.55991] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/11/2024] [Indexed: 04/13/2024] Open
Abstract
INTRODUCTION Large language models (LLMs) have transformed various domains in medicine, aiding in complex tasks and clinical decision-making, with OpenAI's GPT-4, GPT-3.5, Google's Bard, and Anthropic's Claude among the most widely used. While GPT-4 has demonstrated superior performance in some studies, comprehensive comparisons among these models remain limited. Recognizing the significance of the National Board of Medical Examiners (NBME) exams in assessing the clinical knowledge of medical students, this study aims to compare the accuracy of popular LLMs on NBME clinical subject exam sample questions. METHODS The questions used in this study were multiple-choice questions obtained from the official NBME website and are publicly available. Questions from the NBME subject exams in medicine, pediatrics, obstetrics and gynecology, clinical neurology, ambulatory care, family medicine, psychiatry, and surgery were used to query each LLM. The responses from GPT-4, GPT-3.5, Claude, and Bard were collected in October 2023. The response by each LLM was compared to the answer provided by the NBME and checked for accuracy. Statistical analysis was performed using one-way analysis of variance (ANOVA). RESULTS A total of 163 questions were queried by each LLM. GPT-4 scored 163/163 (100%), GPT-3.5 scored 134/163 (82.2%), Bard scored 123/163 (75.5%), and Claude scored 138/163 (84.7%). The total performance of GPT-4 was statistically superior to that of GPT-3.5, Claude, and Bard by 17.8%, 15.3%, and 24.5%, respectively. The total performance of GPT-3.5, Claude, and Bard was not significantly different. GPT-4 significantly outperformed Bard in specific subjects, including medicine, pediatrics, family medicine, and ambulatory care, and GPT-3.5 in ambulatory care and family medicine. Across all LLMs, the surgery exam had the highest average score (18.25/20), while the family medicine exam had the lowest average score (3.75/5). Conclusion: GPT-4's superior performance on NBME clinical subject exam sample questions underscores its potential in medical education and practice. While LLMs exhibit promise, discernment in their application is crucial, considering occasional inaccuracies. As technological advancements continue, regular reassessments and refinements are imperative to maintain their reliability and relevance in medicine.
Collapse
Affiliation(s)
- Ali Abbas
- Medical School, University of Texas Southwestern Medical School, Dallas, USA
| | - Mahad S Rehman
- Medical School, University of Texas Southwestern Medical School, Dallas, USA
| | - Syed S Rehman
- Nephrology, Baptist Hospitals of Southeast Texas, Beaumont, USA
| |
Collapse
|
19
|
Roosan D, Padua P, Khan R, Khan H, Verzosa C, Wu Y. Effectiveness of ChatGPT in clinical pharmacy and the role of artificial intelligence in medication therapy management. J Am Pharm Assoc (2003) 2024; 64:422-428.e8. [PMID: 38049066 DOI: 10.1016/j.japh.2023.11.023] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2023] [Revised: 10/27/2023] [Accepted: 11/27/2023] [Indexed: 12/06/2023]
Abstract
BACKGROUND The use of artificial intelligence (AI) to optimize medication therapy management (MTM) in identifying drug interactions may potentially improve MTM efficiency. ChatGPT, an AI language model, may be applied to identify medication interventions by integrating patient and drug databases. ChatGPT has been shown to be effective in other areas of clinical medicine, from diagnosis to patient management. However, ChatGPT's ability to manage MTM related activities is little known. OBJECTIVES To evaluate the effectiveness of ChatGPT in MTM services in simple, complex, and very complex cases to understand AI contributions in MTM. METHODS Two clinical pharmacists rated and validated the difficulty of patient cases from simple, complex, and very complex. ChatGPT's response to the cases was assessed based on 3 criteria: the ability to identify drug interactions, precision in recommending alternatives, and appropriateness in devising management plans. Two clinical pharmacists validated the accuracy of ChatGPT's responses and compared them to actual answers for each complexity level. RESULTS ChatGPT 4.0 accurately solved 39 out of 39 (100 %) patient cases. ChatGPT successfully identified drug interactions, provided therapy recommendations and formulated general management plans, but it did not recommend specific dosages. Results suggest it can assist pharmacists in formulating MTM plans to improve overall efficiency. CONCLUSION The application of ChatGPT in MTM has the potential to enhance patient safety and involvement, lower healthcare costs, and assist healthcare providers in medication management and identifying drug interactions. Future pharmacists can utilize AI models such as ChatGPT to improve patient care. The future of the pharmacy profession will depend on how the field responds to the changing need for patient care optimized by AI and automation.
Collapse
|
20
|
Çiftci N, Sarman A, Yıldız M, Çiftci K. Use of ChatGPT in health: benefits, hazards, and recommendations. Public Health 2024; 228:e1-e2. [PMID: 38346914 DOI: 10.1016/j.puhe.2023.12.032] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Revised: 12/11/2023] [Accepted: 12/28/2023] [Indexed: 03/16/2024]
Affiliation(s)
- N Çiftci
- Department of Nursing, Faculty of Health Sciences, Muş Alparslan University, Muş, Turkey
| | - A Sarman
- Department of Pediatric Nursing, Faculty of Health Science, Bingöl University, Bingöl, Turkey.
| | - M Yıldız
- Department of Midwifery, Faculty of Health Science, Sakarya University, Sakarya, Turkey
| | - K Çiftci
- Department of Medical Services and Techniques, Vocational School of Health Services, Muş Alparslan University, Muş, Turkey
| |
Collapse
|
21
|
Sallam M, Barakat M, Sallam M. A Preliminary Checklist (METRICS) to Standardize the Design and Reporting of Studies on Generative Artificial Intelligence-Based Models in Health Care Education and Practice: Development Study Involving a Literature Review. Interact J Med Res 2024; 13:e54704. [PMID: 38276872 PMCID: PMC10905357 DOI: 10.2196/54704] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2023] [Revised: 12/18/2023] [Accepted: 01/26/2024] [Indexed: 01/27/2024] Open
Abstract
BACKGROUND Adherence to evidence-based practice is indispensable in health care. Recently, the utility of generative artificial intelligence (AI) models in health care has been evaluated extensively. However, the lack of consensus guidelines on the design and reporting of findings of these studies poses a challenge for the interpretation and synthesis of evidence. OBJECTIVE This study aimed to develop a preliminary checklist to standardize the reporting of generative AI-based studies in health care education and practice. METHODS A literature review was conducted in Scopus, PubMed, and Google Scholar. Published records with "ChatGPT," "Bing," or "Bard" in the title were retrieved. Careful examination of the methodologies employed in the included records was conducted to identify the common pertinent themes and the possible gaps in reporting. A panel discussion was held to establish a unified and thorough checklist for the reporting of AI studies in health care. The finalized checklist was used to evaluate the included records by 2 independent raters. Cohen κ was used as the method to evaluate the interrater reliability. RESULTS The final data set that formed the basis for pertinent theme identification and analysis comprised a total of 34 records. The finalized checklist included 9 pertinent themes collectively referred to as METRICS (Model, Evaluation, Timing, Range/Randomization, Individual factors, Count, and Specificity of prompts and language). Their details are as follows: (1) Model used and its exact settings; (2) Evaluation approach for the generated content; (3) Timing of testing the model; (4) Transparency of the data source; (5) Range of tested topics; (6) Randomization of selecting the queries; (7) Individual factors in selecting the queries and interrater reliability; (8) Count of queries executed to test the model; and (9) Specificity of the prompts and language used. The overall mean METRICS score was 3.0 (SD 0.58). The tested METRICS score was acceptable, with the range of Cohen κ of 0.558 to 0.962 (P<.001 for the 9 tested items). With classification per item, the highest average METRICS score was recorded for the "Model" item, followed by the "Specificity" item, while the lowest scores were recorded for the "Randomization" item (classified as suboptimal) and "Individual factors" item (classified as satisfactory). CONCLUSIONS The METRICS checklist can facilitate the design of studies guiding researchers toward best practices in reporting results. The findings highlight the need for standardized reporting algorithms for generative AI-based studies in health care, considering the variability observed in methodologies and reporting. The proposed METRICS checklist could be a preliminary helpful base to establish a universally accepted approach to standardize the design and reporting of generative AI-based studies in health care, which is a swiftly evolving research topic.
Collapse
Affiliation(s)
- Malik Sallam
- Department of Pathology, Microbiology and Forensic Medicine, School of Medicine, The University of Jordan, Amman, Jordan
- Department of Clinical Laboratories and Forensic Medicine, Jordan University Hospital, Amman, Jordan
- Department of Translational Medicine, Faculty of Medicine, Lund University, Malmo, Sweden
| | - Muna Barakat
- Department of Clinical Pharmacy and Therapeutics, Faculty of Pharmacy, Applied Science Private University, Amman, Jordan
| | - Mohammed Sallam
- Department of Pharmacy, Mediclinic Parkview Hospital, Mediclinic Middle East, Dubai, United Arab Emirates
| |
Collapse
|
22
|
Leal Neto O, Von Wyl V. Digital Transformation of Public Health for Noncommunicable Diseases: Narrative Viewpoint of Challenges and Opportunities. JMIR Public Health Surveill 2024; 10:e49575. [PMID: 38271097 PMCID: PMC10853859 DOI: 10.2196/49575] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2023] [Revised: 09/13/2023] [Accepted: 12/12/2023] [Indexed: 01/27/2024] Open
Abstract
The recent SARS-CoV-2 pandemic underscored the effectiveness and rapid deployment of digital public health interventions, notably the digital proximity tracing apps, leveraging Bluetooth capabilities to trace and notify users about potential infection exposures. Backed by renowned organizations such as the World Health Organization and the European Union, digital proximity tracings showcased the promise of digital public health. As the world pivots from pandemic responses, it becomes imperative to address noncommunicable diseases (NCDs) that account for a vast majority of health care expenses and premature disability-adjusted life years lost. The narrative of digital transformation in the realm of NCD public health is distinct from infectious diseases. Public health, with its multifaceted approach from disciplines such as medicine, epidemiology, and psychology, focuses on promoting healthy living and choices through functions categorized as "Assessment," "Policy Development," "Resource Allocation," "Assurance," and "Access." The power of artificial intelligence (AI) in this digital transformation is noteworthy. AI can automate repetitive tasks, facilitating health care providers to prioritize personal interactions, especially those that cannot be digitalized like emotional support. Moreover, AI presents tools for individuals to be proactive in their health management. However, the human touch remains irreplaceable; AI serves as a companion guiding through the health care landscape. Digital evolution, while revolutionary, poses its own set of challenges. Issues of equity and access are at the forefront. Vulnerable populations, whether due to economic constraints, geographical barriers, or digital illiteracy, face the threat of being marginalized further. This transformation mandates an inclusive strategy, focusing on not amplifying existing health disparities but eliminating them. Population-level digital interventions in NCD prevention demand societal agreement. Policies, like smoking bans or sugar taxes, though effective, might affect those not directly benefiting. Hence, all involved parties, from policy makers to the public, should have a balanced perspective on the advantages, risks, and expenses of these digital shifts. For a successful digital shift in public health, especially concerning NCDs, AI's potential to enhance efficiency, effectiveness, user experience, and equity-the "quadruple aim"-is undeniable. However, it is vital that AI-driven initiatives in public health domains remain purposeful, offering improvements without compromising other objectives. The broader success of digital public health hinges on transparent benchmarks and criteria, ensuring maximum benefits without sidelining minorities or vulnerable groups. Especially in population-centric decisions, like resource allocation, AI's ability to avoid bias is paramount. Therefore, the continuous involvement of stakeholders, including patients and minority groups, remains pivotal in the progression of AI-integrated digital public health.
Collapse
Affiliation(s)
- Onicio Leal Neto
- Department of Computer Science, ETH Zurich, Zurich, Switzerland
- Global Health Institute, Mel & Enid Zuckerman College of Public Health, University of Arizona, Tucson, AZ, United States
- Department of Epidemiology and Biostatistics, Mel & Enid Zuckerman College of Public Health, University of Arizona, Tucson, AZ, United States
| | - Viktor Von Wyl
- Institute for Implementation Science in Health Care, University of Zurich, Zurich, Switzerland
- Epidemiology, Biostatistics & Prevention Institute, University of Zurich, Zurich, Switzerland
| |
Collapse
|
23
|
Abdaljaleel M, Barakat M, Alsanafi M, Salim NA, Abazid H, Malaeb D, Mohammed AH, Hassan BAR, Wayyes AM, Farhan SS, Khatib SE, Rahal M, Sahban A, Abdelaziz DH, Mansour NO, AlZayer R, Khalil R, Fekih-Romdhane F, Hallit R, Hallit S, Sallam M. A multinational study on the factors influencing university students' attitudes and usage of ChatGPT. Sci Rep 2024; 14:1983. [PMID: 38263214 PMCID: PMC10806219 DOI: 10.1038/s41598-024-52549-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2023] [Accepted: 01/19/2024] [Indexed: 01/25/2024] Open
Abstract
Artificial intelligence models, like ChatGPT, have the potential to revolutionize higher education when implemented properly. This study aimed to investigate the factors influencing university students' attitudes and usage of ChatGPT in Arab countries. The survey instrument "TAME-ChatGPT" was administered to 2240 participants from Iraq, Kuwait, Egypt, Lebanon, and Jordan. Of those, 46.8% heard of ChatGPT, and 52.6% used it before the study. The results indicated that a positive attitude and usage of ChatGPT were determined by factors like ease of use, positive attitude towards technology, social influence, perceived usefulness, behavioral/cognitive influences, low perceived risks, and low anxiety. Confirmatory factor analysis indicated the adequacy of the "TAME-ChatGPT" constructs. Multivariate analysis demonstrated that the attitude towards ChatGPT usage was significantly influenced by country of residence, age, university type, and recent academic performance. This study validated "TAME-ChatGPT" as a useful tool for assessing ChatGPT adoption among university students. The successful integration of ChatGPT in higher education relies on the perceived ease of use, perceived usefulness, positive attitude towards technology, social influence, behavioral/cognitive elements, low anxiety, and minimal perceived risks. Policies for ChatGPT adoption in higher education should be tailored to individual contexts, considering the variations in student attitudes observed in this study.
Collapse
Affiliation(s)
- Maram Abdaljaleel
- Department of Pathology, Microbiology and Forensic Medicine, School of Medicine, The University of Jordan, Amman, 11942, Jordan
- Department of Clinical Laboratories and Forensic Medicine, Jordan University Hospital, Amman, 11942, Jordan
| | - Muna Barakat
- Department of Clinical Pharmacy and Therapeutics, Faculty of Pharmacy, Applied Science Private University, Amman, 11931, Jordan
| | - Mariam Alsanafi
- Department of Pharmacy Practice, Faculty of Pharmacy, Kuwait University, Kuwait City, Kuwait
- Department of Pharmaceutical Sciences, Public Authority for Applied Education and Training, College of Health Sciences, Safat, Kuwait
| | - Nesreen A Salim
- Prosthodontic Department, School of Dentistry, The University of Jordan, Amman, 11942, Jordan
- Prosthodontic Department, Jordan University Hospital, Amman, 11942, Jordan
| | - Husam Abazid
- Department of Clinical Pharmacy and Therapeutics, Faculty of Pharmacy, Applied Science Private University, Amman, 11931, Jordan
| | - Diana Malaeb
- College of Pharmacy, Gulf Medical University, P.O. Box 4184, Ajman, United Arab Emirates
| | - Ali Haider Mohammed
- School of Pharmacy, Monash University Malaysia, Jalan Lagoon Selatan, 47500, Bandar Sunway, Selangor Darul Ehsan, Malaysia
| | | | | | - Sinan Subhi Farhan
- Department of Anesthesia, Al Rafidain University College, Baghdad, 10001, Iraq
| | - Sami El Khatib
- Department of Biomedical Sciences, School of Arts and Sciences, Lebanese International University, Bekaa, Lebanon
- Center for Applied Mathematics and Bioinformatics (CAMB), Gulf University for Science and Technology (GUST), 32093, Hawally, Kuwait
| | - Mohamad Rahal
- School of Pharmacy, Lebanese International University, Beirut, 961, Lebanon
| | - Ali Sahban
- School of Dentistry, The University of Jordan, Amman, 11942, Jordan
| | - Doaa H Abdelaziz
- Pharmacy Practice and Clinical Pharmacy Department, Faculty of Pharmacy, Future University in Egypt, Cairo, 11835, Egypt
- Department of Clinical Pharmacy, Faculty of Pharmacy, Al-Baha University, Al-Baha, Saudi Arabia
| | - Noha O Mansour
- Clinical Pharmacy and Pharmacy Practice Department, Faculty of Pharmacy, Mansoura University, Mansoura, 35516, Egypt
- Clinical Pharmacy and Pharmacy Practice Department, Faculty of Pharmacy, Mansoura National University, Dakahlia Governorate, 7723730, Egypt
| | - Reem AlZayer
- Clinical Pharmacy Practice, Department of Pharmacy, Mohammed Al-Mana College for Medical Sciences, 34222, Dammam, Saudi Arabia
| | - Roaa Khalil
- Department of Pathology, Microbiology and Forensic Medicine, School of Medicine, The University of Jordan, Amman, 11942, Jordan
| | - Feten Fekih-Romdhane
- The Tunisian Center of Early Intervention in Psychosis, Department of Psychiatry "Ibn Omrane", Razi Hospital, 2010, Manouba, Tunisia
- Faculty of Medicine of Tunis, Tunis El Manar University, Tunis, Tunisia
| | - Rabih Hallit
- School of Medicine and Medical Sciences, Holy Spirit University of Kaslik, Jounieh, Lebanon
- Department of Infectious Disease, Bellevue Medical Center, Mansourieh, Lebanon
- Department of Infectious Disease, Notre Dame des Secours, University Hospital Center, Byblos, Lebanon
| | - Souheil Hallit
- School of Medicine and Medical Sciences, Holy Spirit University of Kaslik, Jounieh, Lebanon
- Research Department, Psychiatric Hospital of the Cross, Jal Eddib, Lebanon
| | - Malik Sallam
- Department of Pathology, Microbiology and Forensic Medicine, School of Medicine, The University of Jordan, Amman, 11942, Jordan.
- Department of Clinical Laboratories and Forensic Medicine, Jordan University Hospital, Amman, 11942, Jordan.
| |
Collapse
|
24
|
Younis HA, Eisa TAE, Nasser M, Sahib TM, Noor AA, Alyasiri OM, Salisu S, Hayder IM, Younis HA. A Systematic Review and Meta-Analysis of Artificial Intelligence Tools in Medicine and Healthcare: Applications, Considerations, Limitations, Motivation and Challenges. Diagnostics (Basel) 2024; 14:109. [PMID: 38201418 PMCID: PMC10802884 DOI: 10.3390/diagnostics14010109] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2023] [Revised: 12/02/2023] [Accepted: 12/04/2023] [Indexed: 01/12/2024] Open
Abstract
Artificial intelligence (AI) has emerged as a transformative force in various sectors, including medicine and healthcare. Large language models like ChatGPT showcase AI's potential by generating human-like text through prompts. ChatGPT's adaptability holds promise for reshaping medical practices, improving patient care, and enhancing interactions among healthcare professionals, patients, and data. In pandemic management, ChatGPT rapidly disseminates vital information. It serves as a virtual assistant in surgical consultations, aids dental practices, simplifies medical education, and aids in disease diagnosis. A total of 82 papers were categorised into eight major areas, which are G1: treatment and medicine, G2: buildings and equipment, G3: parts of the human body and areas of the disease, G4: patients, G5: citizens, G6: cellular imaging, radiology, pulse and medical images, G7: doctors and nurses, and G8: tools, devices and administration. Balancing AI's role with human judgment remains a challenge. A systematic literature review using the PRISMA approach explored AI's transformative potential in healthcare, highlighting ChatGPT's versatile applications, limitations, motivation, and challenges. In conclusion, ChatGPT's diverse medical applications demonstrate its potential for innovation, serving as a valuable resource for students, academics, and researchers in healthcare. Additionally, this study serves as a guide, assisting students, academics, and researchers in the field of medicine and healthcare alike.
Collapse
Affiliation(s)
- Hussain A. Younis
- College of Education for Women, University of Basrah, Basrah 61004, Iraq
| | | | - Maged Nasser
- Computer & Information Sciences Department, Universiti Teknologi PETRONAS, Seri Iskandar 32610, Malaysia;
| | - Thaeer Mueen Sahib
- Kufa Technical Institute, Al-Furat Al-Awsat Technical University, Kufa 54001, Iraq;
| | - Ameen A. Noor
- Computer Science Department, College of Education, University of Almustansirya, Baghdad 10045, Iraq;
| | | | - Sani Salisu
- Department of Information Technology, Federal University Dutse, Dutse 720101, Nigeria;
| | - Israa M. Hayder
- Qurna Technique Institute, Southern Technical University, Basrah 61016, Iraq;
| | - Hameed AbdulKareem Younis
- Department of Cybersecurity, College of Computer Science and Information Technology, University of Basrah, Basrah 61016, Iraq;
| |
Collapse
|
25
|
Abdel Aziz MH, Rowe C, Southwood R, Nogid A, Berman S, Gustafson K. A scoping review of artificial intelligence within pharmacy education. AMERICAN JOURNAL OF PHARMACEUTICAL EDUCATION 2024; 88:100615. [PMID: 37914030 DOI: 10.1016/j.ajpe.2023.100615] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/25/2023] [Revised: 10/19/2023] [Accepted: 10/25/2023] [Indexed: 11/03/2023]
Abstract
OBJECTIVES This scoping review aimed to summarize the available literature on the use of artificial intelligence (AI) in pharmacy education and identify gaps where additional research is needed. FINDINGS Seven studies specifically addressing the use of AI in pharmacy education were identified. Of these 7 studies, 5 focused on AI use in the context of teaching and learning, 1 on the prediction of academic performance for admissions, and the final study focused on using AI text generation to elucidate the benefits and limitations of ChatGPT use in pharmacy education. SUMMARY There are currently a limited number of available publications that describe AI use in pharmacy education. Several challenges exist regarding the use of AI in pharmacy education, including the need for faculty expertise and time, limited generalizability of tools, limited outcomes data, and several legal and ethical concerns. As AI use increases and implementation becomes more standardized, opportunities will be created for the inclusion of AI in pharmacy education.
Collapse
Affiliation(s)
- May H Abdel Aziz
- University of Texas at Tyler, Ben and Maytee Fisch College of Pharmacy, Department of Pharmaceutical Sciences and Health Outcomes, Tyler, TX, USA.
| | - Casey Rowe
- University of Florida College of Pharmacy, Department of Pharmacotherapy and Translational Research, Orlando, FL, USA
| | - Robin Southwood
- University of Georgia, College of Pharmacy, Department of Clinical and Administrative Pharmacy, Athens, GA, USA
| | - Anna Nogid
- Fairleigh Dickinson University, School of Pharmacy and Health Sciences, Department of Pharmacy Practice, Florham Park, NJ, USA
| | - Sarah Berman
- University of the Incarnate Word, Feik School of Pharmacy, Department of Pharmacy Practice, San Antonio, TX, USA
| | - Kyle Gustafson
- Northeast Ohio Medical University, Department of Pharmacy Practice, Rootstown, OH, USA
| |
Collapse
|
26
|
Sallam M, Al-Salahat K. Below average ChatGPT performance in medical microbiology exam compared to university students. FRONTIERS IN EDUCATION 2023; 8. [DOI: 10.3389/feduc.2023.1333415] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/01/2024]
Abstract
BackgroundThe transformative potential of artificial intelligence (AI) in higher education is evident, with conversational models like ChatGPT poised to reshape teaching and assessment methods. The rapid evolution of AI models requires a continuous evaluation. AI-based models can offer personalized learning experiences but raises accuracy concerns. MCQs are widely used for competency assessment. The aim of this study was to evaluate ChatGPT performance in medical microbiology MCQs compared to the students’ performance.MethodsThe study employed an 80-MCQ dataset from a 2021 medical microbiology exam at the University of Jordan Doctor of Dental Surgery (DDS) Medical Microbiology 2 course. The exam contained 40 midterm and 40 final MCQs, authored by a single instructor without copyright issues. The MCQs were categorized based on the revised Bloom’s Taxonomy into four categories: Remember, Understand, Analyze, or Evaluate. Metrics, including facility index and discriminative efficiency, were derived from 153 midterm and 154 final exam DDS student performances. ChatGPT 3.5 was used to answer questions, and responses were assessed for correctness and clarity by two independent raters.ResultsChatGPT 3.5 correctly answered 64 out of 80 medical microbiology MCQs (80%) but scored below the student average (80.5/100 vs. 86.21/100). Incorrect ChatGPT responses were more common in MCQs with longer choices (p = 0.025). ChatGPT 3.5 performance varied across cognitive domains: Remember (88.5% correct), Understand (82.4% correct), Analyze (75% correct), Evaluate (72% correct), with no statistically significant differences (p = 0.492). Correct ChatGPT responses received statistically significant higher average clarity and correctness scores compared to incorrect responses.ConclusionThe study findings emphasized the need for ongoing refinement and evaluation of ChatGPT performance. ChatGPT 3.5 showed the potential to correctly and clearly answer medical microbiology MCQs; nevertheless, its performance was below-bar compared to the students. Variability in ChatGPT performance in different cognitive domains should be considered in future studies. The study insights could contribute to the ongoing evaluation of the AI-based models’ role in educational assessment and to augment the traditional methods in higher education.
Collapse
|
27
|
Bagde H, Dhopte A, Alam MK, Basri R. A systematic review and meta-analysis on ChatGPT and its utilization in medical and dental research. Heliyon 2023; 9:e23050. [PMID: 38144348 PMCID: PMC10746423 DOI: 10.1016/j.heliyon.2023.e23050] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2023] [Revised: 10/24/2023] [Accepted: 11/24/2023] [Indexed: 12/26/2023] Open
Abstract
Since its release, ChatGPT has taken the world by storm with its utilization in various fields of life. This review's main goal was to offer a thorough and fact-based evaluation of ChatGPT's potential as a tool for medical and dental research, which could direct subsequent research and influence clinical practices. METHODS Different online databases were scoured for relevant articles that were in accordance with the study objectives. A team of reviewers was assembled to devise a proper methodological framework for inclusion of articles and meta-analysis. RESULTS 11 descriptive studies were considered for this review that evaluated the accuracy of ChatGPT in answering medical queries related to different domains such as systematic reviews, cancer, liver diseases, diagnostic imaging, education, and COVID-19 vaccination. The studies reported different accuracy ranges, from 18.3 % to 100 %, across various datasets and specialties. The meta-analysis showed an odds ratio (OR) of 2.25 and a relative risk (RR) of 1.47 with a 95 % confidence interval (CI), indicating that the accuracy of ChatGPT in providing correct responses was significantly higher compared to the total responses for queries. However, significant heterogeneity was present among the studies, suggesting considerable variability in the effect sizes across the included studies. CONCLUSION The observations indicate that ChatGPT has the ability to provide appropriate solutions to questions in the medical and dentistry areas, but researchers and doctors should cautiously assess its responses because they might not always be dependable. Overall, the importance of this study rests in shedding light on ChatGPT's accuracy in the medical and dentistry fields and emphasizing the need for additional investigation to enhance its performance. © 2017 Elsevier Inc. All rights reserved.
Collapse
Affiliation(s)
- Hiroj Bagde
- Department of Periodontology, Chhattisgarh Dental College and Research Institute, Rajnandgaon, Chhattisgarh, India
| | - Ashwini Dhopte
- Department of Oral Medicine and Radiology, Chhattisgarh Dental College and Research Institute, Rajnandgaon, Chhattisgarh, India
| | - Mohammad Khursheed Alam
- Preventive Dentistry Department, College of Dentistry, Jouf University, Sakaka, 72345, Saudi Arabia
- Department of Dental Research Cell, Saveetha Dental College and Hospitals, Saveetha Institute of Medical and Technical Sciences, Chennai, India
- Department of Public Health, Faculty of Allied Health Sciences, Daffodil International University, Dhaka, Bangladesh
| | - Rehana Basri
- Department of Internal Medicine, College of Medicine, Jouf University, Sakaka, 72345, Saudi Arabia
| |
Collapse
|
28
|
Abu-Farha R, Fino L, Al-Ashwal FY, Zawiah M, Gharaibeh L, Harahsheh MM, Darwish Elhajji F. Evaluation of community pharmacists' perceptions and willingness to integrate ChatGPT into their pharmacy practice: A study from Jordan. J Am Pharm Assoc (2003) 2023; 63:1761-1767.e2. [PMID: 37648157 DOI: 10.1016/j.japh.2023.08.020] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2023] [Revised: 08/10/2023] [Accepted: 08/22/2023] [Indexed: 09/01/2023]
Abstract
OBJECTIVES This study aimed to examine the extent of community pharmacists' awareness of Chat Generative Pretraining Transformer (ChatGPT), their willingness to embark on this new development of artificial intelligence (AI) development, and barriers that face the incorporation of this nonconventional source of information into pharmacy practice. METHODS A cross-sectional study was conducted among community pharmacists in Jordanian cities between April 26, 2023, and May 10, 2023. Convenience and snowball sampling techniques were used to select study participants owing to resource and time constraints. The questionnaire was distributed by research assistants through popular social media platforms. Logistic regression analysis was used to assess predictors affecting their willingness to use this service in the future. RESULTS A total of 221 community pharmacists participated in the current study (response rate was not calculated because opt-in recruitment strategies were used). Remarkably, nearly half of the pharmacists (n = 107, 48.4%) indicated a willingness to incorporate the ChatGPT into their pharmacy practice. Nearly half of the pharmacists (n = 105, 47.5%) demonstrated a high perceived benefit score for ChatGPT, whereas approximately 37% of pharmacists (n = 81) expressed a high concern score about ChatGPT. More than 70% of pharmacists believed that ChatGPT lacked the ability to use human judgment and make complicated ethical judgments in its responses (n = 168). Finally, logistics regression analysis showed that pharmacists who had previous experience in using ChatGPT were more willing to integrate ChatGPT in their pharmacy practice than those with no previous experience in using ChatGPT (odds ratio 2.312, P = 0.035). CONCLUSION Although pharmacists show a willingness to incorporate ChatGPT into their practice, especially those with previous experience, there are major concerns. These mainly revolve around the tool's ability to make human-like judgments and ethical decisions. These findings are crucial for the future development and integration of AI tools in pharmacy practice.
Collapse
|
29
|
Abujaber AA, Abd-Alrazaq A, Al-Qudimat AR, Nashwan AJ. A Strengths, Weaknesses, Opportunities, and Threats (SWOT) Analysis of ChatGPT Integration in Nursing Education: A Narrative Review. Cureus 2023; 15:e48643. [PMID: 38090452 PMCID: PMC10711690 DOI: 10.7759/cureus.48643] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/11/2023] [Indexed: 03/25/2024] Open
Abstract
Amidst evolving healthcare demands, nursing education plays a pivotal role in preparing future nurses for complex challenges. Traditional approaches, however, must be revised to meet modern healthcare needs. The ChatGPT, an AI-based chatbot, has garnered significant attention due to its ability to personalize learning experiences, enhance virtual clinical simulations, and foster collaborative learning in nursing education. This review aims to thoroughly assess the potential impact of integrating ChatGPT into nursing education. The hypothesis is that valuable insights can be provided for stakeholders through a comprehensive SWOT analysis examining the strengths, weaknesses, opportunities, and threats associated with ChatGPT. This will enable informed decisions about its integration, prioritizing improved learning outcomes. A thorough narrative literature review was undertaken to provide a solid foundation for the SWOT analysis. The materials included scholarly articles and reports, which ensure the study's credibility and allow for a holistic and unbiased assessment. The analysis identified accessibility, consistency, adaptability, cost-effectiveness, and staying up-to-date as crucial factors influencing the strengths, weaknesses, opportunities, and threats associated with ChatGPT integration in nursing education. These themes provided a framework to understand the potential risks and benefits of integrating ChatGPT into nursing education. This review highlights the importance of responsible and effective use of ChatGPT in nursing education and the need for collaboration among educators, policymakers, and AI developers. Addressing the identified challenges and leveraging the strengths of ChatGPT can lead to improved learning outcomes and enriched educational experiences for students. The findings emphasize the importance of responsibly integrating ChatGPT in nursing education, balancing technological advancement with careful consideration of associated risks, to achieve optimal outcomes.
Collapse
Affiliation(s)
| | - Alaa Abd-Alrazaq
- AI Center for Precision Health, Weill Cornell Medicine-Qatar, Doha, QAT
| | - Ahmad R Al-Qudimat
- Department of Public Health, Qatar University, Doha, QAT
- Surgical Research Section, Department of Surgery, Hamad Medical Corporation, Doha, QAT
| | | |
Collapse
|
30
|
Sallam M, Barakat M, Sallam M. Pilot Testing of a Tool to Standardize the Assessment of the Quality of Health Information Generated by Artificial Intelligence-Based Models. Cureus 2023; 15:e49373. [PMID: 38024074 PMCID: PMC10674084 DOI: 10.7759/cureus.49373] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/24/2023] [Indexed: 12/01/2023] Open
Abstract
Background Artificial intelligence (AI)-based conversational models, such as Chat Generative Pre-trained Transformer (ChatGPT), Microsoft Bing, and Google Bard, have emerged as valuable sources of health information for lay individuals. However, the accuracy of the information provided by these AI models remains a significant concern. This pilot study aimed to test a new tool with key themes for inclusion as follows: Completeness of content, Lack of false information in the content, Evidence supporting the content, Appropriateness of the content, and Relevance, referred to as "CLEAR", designed to assess the quality of health information delivered by AI-based models. Methods Tool development involved a literature review on health information quality, followed by the initial establishment of the CLEAR tool, which comprised five items that aimed to assess the following: completeness, lack of false information, evidence support, appropriateness, and relevance. Each item was scored on a five-point Likert scale from excellent to poor. Content validity was checked by expert review. Pilot testing involved 32 healthcare professionals using the CLEAR tool to assess content on eight different health topics deliberately designed with varying qualities. The internal consistency was checked with Cronbach's alpha (α). Feedback from the pilot test resulted in language modifications to improve the clarity of the items. The final CLEAR tool was used to assess the quality of health information generated by four distinct AI models on five health topics. The AI models were ChatGPT 3.5, ChatGPT 4, Microsoft Bing, and Google Bard, and the content generated was scored by two independent raters with Cohen's kappa (κ) for inter-rater agreement. Results The final five CLEAR items were: (1) Is the content sufficient?; (2) Is the content accurate?; (3) Is the content evidence-based?; (4) Is the content clear, concise, and easy to understand?; and (5) Is the content free from irrelevant information? Pilot testing on the eight health topics revealed acceptable internal consistency with a Cronbach's α range of 0.669-0.981. The use of the final CLEAR tool yielded the following average scores: Microsoft Bing (mean=24.4±0.42), ChatGPT-4 (mean=23.6±0.96), Google Bard (mean=21.2±1.79), and ChatGPT-3.5 (mean=20.6±5.20). The inter-rater agreement revealed the following Cohen κ values: for ChatGPT-3.5 (κ=0.875, P<.001), ChatGPT-4 (κ=0.780, P<.001), Microsoft Bing (κ=0.348, P=.037), and Google Bard (κ=.749, P<.001). Conclusions The CLEAR tool is a brief yet helpful tool that can aid in standardizing testing of the quality of health information generated by AI-based models. Future studies are recommended to validate the utility of the CLEAR tool in the quality assessment of AI-generated health-related content using a larger sample across various complex health topics.
Collapse
Affiliation(s)
- Malik Sallam
- Department of Pathology, Microbiology, and Forensic Medicine, School of Medicine, University of Jordan, Amman, JOR
- Department of Clinical Laboratories and Forensic Medicine, Jordan University Hospital, Amman, JOR
| | - Muna Barakat
- Department of Clinical Pharmacy and Therapeutics, School of Pharmacy, Applied Science Private University, Amman, JOR
- Department of Research, Middle East University, Amman, JOR
| | - Mohammed Sallam
- Department of Pharmacy, Mediclinic Parkview Hospital, Mediclinic Middle East, Dubai, ARE
| |
Collapse
|
31
|
Huang H. Performance of ChatGPT on Registered Nurse License Exam in Taiwan: A Descriptive Study. Healthcare (Basel) 2023; 11:2855. [PMID: 37958000 PMCID: PMC10649156 DOI: 10.3390/healthcare11212855] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2023] [Revised: 10/17/2023] [Accepted: 10/27/2023] [Indexed: 11/15/2023] Open
Abstract
(1) Background: AI (artificial intelligence) chatbots have been widely applied. ChatGPT could enhance individual learning capabilities and clinical reasoning skills and facilitate students' understanding of complex concepts in healthcare education. There is currently less emphasis on its application in nursing education. The application of ChatGPT in nursing education needs to be verified. (2) Methods: A descriptive study was used to analyze the scores of ChatGPT on the registered nurse license exam (RNLE) in 2022~2023, and to explore the response and explanations of ChatGPT. The process of data measurement encompassed input sourcing, encoding methods, and statistical analysis. (3) Results: ChatGPT promptly responded within seconds. The average score of four exams was around 51.6 to 63.75 by ChatGPT, and it passed the RNLE in 2022 1st and 2023 2nd. However, ChatGPT may generate misleading or inaccurate explanations, or it could lead to hallucination; confusion or misunderstanding about complicated scenarios; and languages bias. (4) Conclusions: ChatGPT may have the potential to assist with nursing education because of its advantages. It is recommended to integrate ChatGPT into different nursing courses, to assess its limitations and effectiveness through a variety of tools and methods.
Collapse
Affiliation(s)
- Huiman Huang
- School of Nursing, College of Nursing, Tzu Chi University of Science and Technology, Hualien 970302, Taiwan
| |
Collapse
|
32
|
Abu Hammour K, Alhamad H, Al-Ashwal FY, Halboup A, Abu Farha R, Abu Hammour A. ChatGPT in pharmacy practice: a cross-sectional exploration of Jordanian pharmacists' perception, practice, and concerns. J Pharm Policy Pract 2023; 16:115. [PMID: 37789443 PMCID: PMC10548710 DOI: 10.1186/s40545-023-00624-2] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2023] [Accepted: 09/22/2023] [Indexed: 10/05/2023] Open
Abstract
OBJECTIVES The purpose of this study is to find out how much pharmacists know and have used ChatGPT in their practice. We investigated the advantages and disadvantages of utilizing ChatGPT in a pharmacy context, the amount of training necessary to use it proficiently, and the influence on patient care using a survey. METHODS This cross-sectional study was carried out between May and June 2023 to assess the potential and problems that pharmacists observed while integrating chatbots powered by AI (ChatGPT) in pharmacy practice. The correlation between perceived benefits and concerns was evaluated using Spearman's rho correlation due to the data's non-normal distribution.Any pharmacists licensed by the Jordanian Pharmacists Association were included in the study. A convenient sampling technique was used to choose the participants, and the study questionnaire was distributed utilizing an online medium (Facebook and WhatsApp). Anyone who expressed interest in taking part was given a link to the study's instructions so they may read them before giving their electronic consent and accessing the survey. RESULTS The potential advantages of ChatGPT in the pharmacy practice were widely acknowledged by the participants. The majority of participants (69.9%) concurred that educational material about pharmacy items or therapeutic areas can be provided using ChatGPT, with 66.9% of respondents believing that ChatGPT is a machine learning algorithm. Concerns about the accuracy of AI-generated responses were also prevalent. More than half of the participants (55.7%) raised the possibility that AI systems such as ChatGPT could pick up on and replicate prejudices and discriminatory patterns from the data they were trained on. Analysis shows a statistically significant positive link, albeit a minor one, between the perceived advantages of ChatGPT and its drawbacks (r = 0.255, p < 0.001). However, concerns were strongly correlated with knowledge of ChatGPT. In contrast to those who were either unsure or had not heard of ChatGPT (64.2%), individuals who had heard of it were more likely to have strong concerns (79.8%) (p = 0.002). Finally, the results show a statistically significant association between the frequency of ChatGPT use and positive perceptions of the tool (p < 0.001). CONCLUSIONS Although ChatGPT has shown promise in health and pharmaceutical practice, its application should be rigorously regulated by evidence-based law. According to the study's findings, pharmacists support the use of ChatGPT in pharmacy practice but have concerns about its use due to ethical reasons, legal problems, privacy concerns, worries about the accuracy of the data generated, data learning, and bias risk.
Collapse
Affiliation(s)
- Khawla Abu Hammour
- Department of Clinical Pharmacy and Biopharmaceutics, Faculty of Pharmacy, University of Jordan, Amman, Jordan
| | - Hamza Alhamad
- Department of Clinical Pharmacy, Faculty of Pharmacy, Zarqa University, Zarqa, Jordan
| | - Fahmi Y Al-Ashwal
- Department of Clinical Pharmacy, College of Pharmacy, Al-Ayen University, Thi-Qar, Iraq.
- Department of Clinical Pharmacy and Pharmacy Practice, Faculty of Pharmacy, University of Science and Technology, Sana'a, Yemen.
| | - Abdulsalam Halboup
- Department of Clinical Pharmacy and Pharmacy Practice, Faculty of Pharmacy, University of Science and Technology, Sana'a, Yemen
- Discipline of Clinical Pharmacy, School of Pharmaceutical Sciences, University Sains Malaysia, Gelugor, Pulau Pinang, Malaysia
| | - Rana Abu Farha
- Clinical Pharmacy and Therapeutics Department, Faculty of Pharmacy, Applied Science Private University, P.O. Box 11937, Amman, Jordan
| | - Adnan Abu Hammour
- Medrise Medical Center, Dubai Healthcare City, Dubai, United Arab Emirates
| |
Collapse
|
33
|
Rahimzadeh V, Kostick-Quenet K, Blumenthal Barby J, McGuire AL. Ethics Education for Healthcare Professionals in the Era of ChatGPT and Other Large Language Models: Do We Still Need It? THE AMERICAN JOURNAL OF BIOETHICS : AJOB 2023; 23:17-27. [PMID: 37487184 DOI: 10.1080/15265161.2023.2233358] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/26/2023]
Abstract
In this paper, we contend with whether we still need traditional ethics education as part of healthcare professional training given the abilities of chatGPT (generative pre-trained transformer) and other large language models (LLM). We reflect on common programmatic goals to assess the current strengths and limitations of LLMs in helping to build ethics competencies among future clinicians. Through an actual case analysis, we highlight areas in which chatGPT and other LLMs are conducive to common bioethics education goals. We also comment on where such technologies remain an imperfect substitute for human-led ethics teaching and learning. Finally, we conclude that the relative strengths of chatGPT warrant its consideration as a teaching and learning tool in ethics education in ways that account for current limitations and build in flexibility as the technology evolves.
Collapse
|
34
|
Analyzing the PESTEL Factors on ChatGPT and Its Impact on the Population. ARTIFICIAL INTELLIGENCE APPLICATIONS USING CHATGPT IN EDUCATION 2023:126-137. [DOI: 10.4018/978-1-6684-9300-7.ch012] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/02/2024]
Abstract
An AI application called ChatGPT has spurred discussions about how it can affect education. The researcher used the PESTEL analysis framework to understand the political, economic, social, technological, environmental, and legal aspects of ChatGPT in education and research. The political aspect observed leaned towards the left orientation. The economic aspect is loosely inclined towards increased productivity and efficiency. The social aspect is directed towards improved social services sectors. The environmental aspect raised concerns about carbon emissions and is yet not explored in detail by researchers, leading to a clear research gap. Last, the authors discuss the legal impact of ChatGPT on the population, and there is an unclear picture towards the AI laws and regulations. There is a need for research to differentiate traditional AI and generative AI. Moreover, generative AI must subclassify before developing the policy for all stakeholders. In conclusion, the responsible use of ChatGPT will benefit the productivity of the general population.
Collapse
|
35
|
Sallam M, Salim NA, Barakat M, Al-Mahzoum K, Al-Tammemi AB, Malaeb D, Hallit R, Hallit S. Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study. JMIR MEDICAL EDUCATION 2023; 9:e48254. [PMID: 37578934 PMCID: PMC10509747 DOI: 10.2196/48254] [Citation(s) in RCA: 13] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/17/2023] [Revised: 07/25/2023] [Accepted: 08/14/2023] [Indexed: 08/16/2023]
Abstract
BACKGROUND ChatGPT is a conversational large language model that has the potential to revolutionize knowledge acquisition. However, the impact of this technology on the quality of education is still unknown considering the risks and concerns surrounding ChatGPT use. Therefore, it is necessary to assess the usability and acceptability of this promising tool. As an innovative technology, the intention to use ChatGPT can be studied in the context of the technology acceptance model (TAM). OBJECTIVE This study aimed to develop and validate a TAM-based survey instrument called TAME-ChatGPT (Technology Acceptance Model Edited to Assess ChatGPT Adoption) that could be employed to examine the successful integration and use of ChatGPT in health care education. METHODS The survey tool was created based on the TAM framework. It comprised 13 items for participants who heard of ChatGPT but did not use it and 23 items for participants who used ChatGPT. Using a convenient sampling approach, the survey link was circulated electronically among university students between February and March 2023. Exploratory factor analysis (EFA) was used to assess the construct validity of the survey instrument. RESULTS The final sample comprised 458 respondents, the majority among them undergraduate students (n=442, 96.5%). Only 109 (23.8%) respondents had heard of ChatGPT prior to participation and only 55 (11.3%) self-reported ChatGPT use before the study. EFA analysis on the attitude and usage scales showed significant Bartlett tests of sphericity scores (P<.001) and adequate Kaiser-Meyer-Olkin measures (0.823 for the attitude scale and 0.702 for the usage scale), confirming the factorability of the correlation matrices. The EFA showed that 3 constructs explained a cumulative total of 69.3% variance in the attitude scale, and these subscales represented perceived risks, attitude to technology/social influence, and anxiety. For the ChatGPT usage scale, EFA showed that 4 constructs explained a cumulative total of 72% variance in the data and comprised the perceived usefulness, perceived risks, perceived ease of use, and behavior/cognitive factors. All the ChatGPT attitude and usage subscales showed good reliability with Cronbach α values >.78 for all the deduced subscales. CONCLUSIONS The TAME-ChatGPT demonstrated good reliability, validity, and usefulness in assessing health care students' attitudes toward ChatGPT. The findings highlighted the importance of considering risk perceptions, usefulness, ease of use, attitudes toward technology, and behavioral factors when adopting ChatGPT as a tool in health care education. This information can aid the stakeholders in creating strategies to support the optimal and ethical use of ChatGPT and to identify the potential challenges hindering its successful implementation. Future research is recommended to guide the effective adoption of ChatGPT in health care education.
Collapse
Affiliation(s)
- Malik Sallam
- Department of Pathology, Microbiology and Forensic Medicine, School of Medicine, The University of Jordan, Amman, Jordan
- Department of Clinical Laboratories and Forensic Medicine, Jordan University Hospital, Amman, Jordan
| | - Nesreen A Salim
- Prosthodontic Department, School of Dentistry, The University of Jordan, Amman, Jordan
- Prosthodontic Department, Jordan University Hospital, Amman, Jordan
| | - Muna Barakat
- Department of Clinical Pharmacy and Therapeutics, Faculty of Pharmacy, Applied Science Private University, Amman, Jordan
- Middle East University Research Unit, Middle East University, Amman, Jordan
| | - Kholoud Al-Mahzoum
- Department of Pathology, Microbiology and Forensic Medicine, School of Medicine, The University of Jordan, Amman, Jordan
| | - Ala'a B Al-Tammemi
- Migration Health Division, International Organization for Migration, The United Nations Migration Agency, Amman, Jordan
| | - Diana Malaeb
- College of Pharmacy, Gulf Medical University, Ajman, United Arab Emirates
| | - Rabih Hallit
- School of Medicine and Medical Sciences, Holy Spirit University of Kaslik, Jounieh, Lebanon
- Department of Infectious Disease, Bellevue Medical Center, Mansourieh, Lebanon
- Department of Infectious Disease, Notre Dame des Secours, University Hospital Center, Byblos, Lebanon
| | - Souheil Hallit
- School of Medicine and Medical Sciences, Holy Spirit University of Kaslik, Jounieh, Lebanon
- Research Department, Psychiatric Hospital of the Cross, Jal Eddib, Lebanon
| |
Collapse
|
36
|
Abstract
BACKGROUND ChatGPT is an artificial intelligence model trained for conversations. ChatGPT has been widely applied in general medical education and cardiology, but its application in pharmacy has been lacking. This study examined the accuracy of ChatGPT on the Taiwanese Pharmacist Licensing Examination and investigated its potential role in pharmacy education. METHODS ChatGPT was used on the first Taiwanese Pharmacist Licensing Examination in 2023 in Mandarin and English. The questions were entered manually one by one. Graphical questions, chemical formulae, and tables were excluded. Textual questions were scored according to the number of correct answers. Chart question scores were determined by multiplying the number and the correct rate of text questions. This study was conducted from March 5 to March 10, 2023, by using ChatGPT 3.5. RESULTS The correct rate of ChatGPT in Chinese and English questions was 54.4% and 56.9% in the first stage, and 53.8% and 67.6% in the second stage. On the Chinese test, only pharmacology and pharmacochemistry sections received passing scores. The English test scores were higher than the Chinese test scores across all subjects and were significantly higher in dispensing pharmacy and clinical pharmacy as well as therapeutics. CONCLUSION ChatGPT 3.5 failed the Taiwanese Pharmacist Licensing Examination. Although it is not able to pass the examination, it can be improved quickly through deep learning. It reminds us that we should not only use multiple-choice questions to assess a pharmacist's ability, but also use more variety of evaluations in the future. Pharmacy education should be changed in line with the examination, and students must be able to use AI technology for self-learning. More importantly, we need to help students develop humanistic qualities and strengthen their ability to interact with patients, so that they can become warm-hearted healthcare professionals.
Collapse
Affiliation(s)
- Ying-Mei Wang
- Department of Medical Education and Research, Taipei Veterans General Hospital Hsinchu Branch, Hsinchu, Taiwan, ROC
- Department of Pharmacy, Taipei Veterans General Hospital Hsinchu Branch, Hsinchu, Taiwan, ROC
- School of Medicine, National Tsing Hua University, Hsinchu, Taiwan, ROC
| | - Hung-Wei Shen
- Department of Pharmacy, Taipei Veterans General Hospital Hsinchu Branch, Hsinchu, Taiwan, ROC
| | - Tzeng-Ji Chen
- Department of Family Medicine, Taipei Veterans General Hospital Hsinchu Branch, Hsinchu, Taiwan, ROC
- Department of Family Medicine, Taipei Veterans General Hospital, Taipei, Taiwan, ROC
- Department of Post-Baccalaureate Medicine, National Chung Hsing University, Taichung, Taiwan, ROC
| |
Collapse
|
37
|
Sharma S, Pajai S, Prasad R, Wanjari MB, Munjewar PK, Sharma R, Pathade A. A Critical Review of ChatGPT as a Potential Substitute for Diabetes Educators. Cureus 2023; 15:e38380. [PMID: 37265899 PMCID: PMC10231273 DOI: 10.7759/cureus.38380] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Accepted: 05/01/2023] [Indexed: 06/03/2023] Open
Abstract
This review article explores the potential of ChatGPT as a substitute for diabetes educators. Diabetes is a prevalent chronic disease that requires ongoing education and support for patients to effectively manage their condition. However, there is a shortage of diabetes educators, and traditional education methods have limitations in addressing patients' individual needs. ChatGPT is an artificial intelligence technology that offers a personalized and interactive approach to education and support. In this review, we provide an overview of ChatGPT technology, discuss the challenges facing diabetes educators, review evidence supporting the use of ChatGPT in diabetes education, and examine ethical considerations related to its use. We also provide recommendations for further research and development of ChatGPT in diabetes education and integration into clinical practice. ChatGPT has the potential to improve access to education and support for patients with diabetes, but further research is needed to better understand its effectiveness and limitations. It is important to ensure that ChatGPT is developed and integrated in an ethical and equitable manner to maximize its potential benefits and minimize potential risks.
Collapse
Affiliation(s)
- Samriddhi Sharma
- Medicine and Surgery, Jawaharlal Nehru Medical College, Datta Meghe Institute of Higher Education and Research, Wardha, IND
| | - Sandhya Pajai
- Obstetrics and Gynaecology, Jawaharlal Nehru Medical College, Datta Meghe Institute of Higher Education and Research, Wardha, IND
| | - Roshan Prasad
- Medicine and Surgery, Jawaharlal Nehru Medical College, Datta Meghe Institute of Higher Education and Research, Wardha, IND
| | - Mayur B Wanjari
- Research and Development, Jawaharlal Nehru Medical College, Datta Meghe Institute of Higher Education and Research, Wardha, IND
| | - Pratiksha K Munjewar
- Medical Surgical Nursing, Srimati Radhikabai Meghe Memorial College of Nursing, Datta Meghe Institute of Higher Education and Research, Wardha, IND
| | - Ranjana Sharma
- Medical Surgical Nursing, Srimati Radhikabai Meghe Memorial College of Nursing, Datta Meghe Institute of Higher Education and Research, Wardha, IND
| | - Aniket Pathade
- Research and Development, Jawaharlal Nehru Medical College, Datta Meghe Institute of Higher Education and Research, Wardha, IND
| |
Collapse
|