1
|
Chelli M, Descamps J, Lavoué V, Trojani C, Azar M, Deckert M, Raynier JL, Clowez G, Boileau P, Ruetsch-Chelli C. Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis. J Med Internet Res 2024; 26:e53164. [PMID: 38776130 PMCID: PMC11153973 DOI: 10.2196/53164] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2023] [Revised: 01/22/2024] [Accepted: 02/21/2024] [Indexed: 05/24/2024] Open
Abstract
BACKGROUND Large language models (LLMs) have raised both interest and concern in the academic community. They offer the potential for automating literature search and synthesis for systematic reviews but raise concerns regarding their reliability, as the tendency to generate unsupported (hallucinated) content persist. OBJECTIVE The aim of the study is to assess the performance of LLMs such as ChatGPT and Bard (subsequently rebranded Gemini) to produce references in the context of scientific writing. METHODS The performance of ChatGPT and Bard in replicating the results of human-conducted systematic reviews was assessed. Using systematic reviews pertaining to shoulder rotator cuff pathology, these LLMs were tested by providing the same inclusion criteria and comparing the results with original systematic review references, serving as gold standards. The study used 3 key performance metrics: recall, precision, and F1-score, alongside the hallucination rate. Papers were considered "hallucinated" if any 2 of the following information were wrong: title, first author, or year of publication. RESULTS In total, 11 systematic reviews across 4 fields yielded 33 prompts to LLMs (3 LLMs×11 reviews), with 471 references analyzed. Precision rates for GPT-3.5, GPT-4, and Bard were 9.4% (13/139), 13.4% (16/119), and 0% (0/104) respectively (P<.001). Recall rates were 11.9% (13/109) for GPT-3.5 and 13.7% (15/109) for GPT-4, with Bard failing to retrieve any relevant papers (P<.001). Hallucination rates stood at 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001). Further analysis of nonhallucinated papers retrieved by GPT models revealed significant differences in identifying various criteria, such as randomized studies, participant criteria, and intervention criteria. The study also noted the geographical and open-access biases in the papers retrieved by the LLMs. CONCLUSIONS Given their current performance, it is not recommended for LLMs to be deployed as the primary or exclusive tool for conducting systematic reviews. Any references generated by such models warrant thorough validation by researchers. The high occurrence of hallucinations in LLMs highlights the necessity for refining their training and functionality before confidently using them for rigorous academic purposes.
Collapse
Affiliation(s)
- Mikaël Chelli
- Institute for Sports and Reconstructive Bone and Joint Surgery, Groupe Kantys, Nice, France
| | - Jules Descamps
- Orthopedic and Traumatology Unit, Hospital Lariboisière, Assistance Publique-Hôpitaux de Paris, Paris, France
| | - Vincent Lavoué
- Institute for Sports and Reconstructive Bone and Joint Surgery, Groupe Kantys, Nice, France
| | - Christophe Trojani
- Institute for Sports and Reconstructive Bone and Joint Surgery, Groupe Kantys, Nice, France
| | - Michel Azar
- Institute for Sports and Reconstructive Bone and Joint Surgery, Groupe Kantys, Nice, France
| | - Marcel Deckert
- Université Côte d'Azur, INSERM, C3M, Team Microenvironment, Signalling and Cancer, Nice, France
| | - Jean-Luc Raynier
- Institute for Sports and Reconstructive Bone and Joint Surgery, Groupe Kantys, Nice, France
| | - Gilles Clowez
- Institute for Sports and Reconstructive Bone and Joint Surgery, Groupe Kantys, Nice, France
| | - Pascal Boileau
- Institute for Sports and Reconstructive Bone and Joint Surgery, Groupe Kantys, Nice, France
| | - Caroline Ruetsch-Chelli
- Université Côte d'Azur, INSERM, C3M, Team Microenvironment, Signalling and Cancer, Nice, France
| |
Collapse
|
2
|
Eubank BHF, Sheps DM, Dennett L, Connick A, Bouliane M, Panu A, Harding G, Beaupre LA. A scoping review and best evidence synthesis for treatment of partial-thickness rotator cuff tears. J Shoulder Elbow Surg 2024; 33:e126-e152. [PMID: 38103720 DOI: 10.1016/j.jse.2023.10.027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/01/2023] [Revised: 10/24/2023] [Accepted: 10/30/2023] [Indexed: 12/19/2023]
Abstract
BACKGROUND Rotator cuff disorders include a broad spectrum of pathological conditions including partial-thickness and full-thickness tears. Studies have shown partial-thickness rotator cuff tear (PTRCT) prevalence to be twice that of full-thickness tears. In the working population, PTRCTs are one of the most common causes of shoulder pain and often result in occupational disability due to pain, stiffness, and loss of shoulder function. Treatment of PTRCTs remains controversial. The purpose of this study was to consolidate the existing high-quality evidence on best management approaches in treating PTRCTs using both nonoperative and operative approaches. METHODS A scoping review with best evidence synthesis was performed as per the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews. MEDLINE (OVID), EMBASE (OVID), Cochrane Library (Wiley), SCOPUS, Web of Science Core Collection, CINAHL Plus with Full Text (EBSCOhost), PubMed Central, and Science Direct were searched from 2000 to March 3, 2023. Level 1 studies, and systematic reviews and meta-analyses that included level 1 and 2 studies, were included. RESULTS The search yielded 8276 articles. A total of 3930 articles were screened after removing 4346 duplicates. Application of inclusion criteria resulted in 662 articles that were selected for full-text review. Twenty-eight level 1 studies, 1 systematic review, 4 meta-analyses, and 1 network meta-analyses were included in the best evidence synthesis. Nonoperative strategies included injections (ie, platelet-rich plasma, corticosteroid, prolotherapy, sodium hyaluronate, anesthetic, and atelocollagen), exercise therapy, and physical agents. Operative interventions consisted of débridement, shaving of the tendon and footprint, transtendon repair, and traditional suture anchor repair techniques with and without tear completion. Both nonoperative and operative strategies demonstrated effectiveness at managing pain and functional outcome for PTRCTs. The evidence supports the effectiveness of surgical intervention in treating PTRCTs regardless of arthroscopic technique. CONCLUSION The results of this scoping review do not support superiority of operative over nonoperative management and suggest that both strategies can be effective at managing pain and functional outcome for PTRCTs. Surgery, however, is the most invasive and costly approach, with the highest risk of complications such as infection. Other variables such as patient expectation, treating practitioner bias, or preference may change which modalities are offered and in what sequence.
Collapse
Affiliation(s)
- Breda H F Eubank
- Faculty of Health, Community, and Education, Department of Health and Physical Education, Mount Royal University, Calgary, AB, Canada.
| | - David M Sheps
- Division of Orthopaedics, Department of Surgery, University of Alberta, Edmonton, AB, Canada
| | - Liz Dennett
- Health Sciences Librarian, University of Alberta, Edmonton, AB, Canada
| | - Abbie Connick
- Department of Physical Therapy, University of Alberta, Edmonton, AB, Canada
| | - Martin Bouliane
- Division of Orthopaedics, Department of Surgery, University of Alberta, Edmonton, AB, Canada
| | - Anukul Panu
- Division of Radiology & Diagnostic Imaging, University of Alberta, Edmonton, AB, Canada
| | - Graeme Harding
- Division of Orthopaedics, Department of Surgery, University of Alberta, Edmonton, AB, Canada
| | - Lauren A Beaupre
- Department of Physical Therapy, University of Alberta, Edmonton, AB, Canada
| |
Collapse
|
3
|
Fong HPY, Zhu MT, Rabago DP, Reeves KD, Chung VCH, Sit RWS. Effectiveness of Hypertonic Dextrose Injection (Prolotherapy) in Plantar Fasciopathy: A Systematic Review and Meta-analysis of Randomized Controlled Trials. Arch Phys Med Rehabil 2023; 104:1941-1953.e9. [PMID: 37098357 DOI: 10.1016/j.apmr.2023.03.027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2022] [Revised: 02/21/2023] [Accepted: 03/14/2023] [Indexed: 04/27/2023]
Abstract
OBJECTIVE To systematically review the effectiveness of hypertonic dextrose prolotherapy (DPT) in plantar fasciopathy (PF) compared with other non-surgical treatments. DATA SOURCES PubMed/MEDLINE, EMBASE, Cochrane Central Register of Controlled Trials, Web of Science, Allied and Complementary Medicine Database, Global Health, Ovid Nursing Database, Dimensions, and WHO ICTRP were searched from inception to April 30th, 2022. STUDY SELECTION Two independent reviewers selected randomized controlled trials (RCTs) that evaluated the effectiveness of DPT in PF compared with non-surgical treatments. Outcomes included pain intensity, foot and ankle function, and plantar fascia thickness. DATA EXTRACTION Two independent reviewers conducted data extraction. Risk of bias (RoB) assessment was conducted using the Cochrane Risk of Bias 2 (RoB 2) tool, and certainty of evidence was assessed with Grading of Recommendation Assessment, Development, and Evaluation (GRADE). DATA SYNTHESIS Eight RCTs (n=469) met the inclusion criteria. Pooled results favored the use of DPT versus normal saline (NS) injections in reducing pain (weighted mean difference [WMD] -41.72; 95% confidence interval [CI] -62.36 to -21.08; P<.01; low certainty evidence) and improving function [WMD -39.04; 95% CI -55.24 to -22.85; P<.01; low certainty evidence] in the medium term. Pooled results also showed corticosteroid (CS) injections was superior to DPT in reducing pain in the short term [standardized mean difference 0.77; 95% CI 0.40 to 1.14; P<.01; moderate certainty evidence]. Overall RoB varied from "some concerns" to "high". The overall certainty of evidence presented ranges from very low to moderate based on the assessment with the GRADE approach. CONCLUSION Low certainty evidence demonstrated that DPT was superior to NS injections in reducing pain and improving function in the medium term, but moderate certainty evidence showed that it was inferior to CS in reducing pain in the short term. Further high-quality RCTs with standard protocol, longer-term follow-up, and adequate sample size are needed to confirm its role in clinical practice.
Collapse
Affiliation(s)
- Hugo P Y Fong
- The Jockey Club School of Public Health and Primary Care, the Chinese University of Hong Kong, Hong Kong, China
| | - Meng-Ting Zhu
- The Jockey Club School of Public Health and Primary Care, the Chinese University of Hong Kong, Hong Kong, China
| | - David P Rabago
- Department of Family and Community Medicine, Penn State College of Medicine, Hershey, PA
| | | | - Vincent C H Chung
- The Jockey Club School of Public Health and Primary Care, the Chinese University of Hong Kong, Hong Kong, China
| | - Regina W S Sit
- The Jockey Club School of Public Health and Primary Care, the Chinese University of Hong Kong, Hong Kong, China.
| |
Collapse
|
4
|
Lin LC, Lee YH, Chen YW, Hsu TH, Vitoonpong T, Liou TH, Huang SW. Comparison Clinical Effects of Hypertonic Dextrose and Steroid Injections on Chronic Subacromial Bursitis: A Double-Blind Randomized Controlled Trial. Am J Phys Med Rehabil 2023; 102:867-872. [PMID: 36897810 DOI: 10.1097/phm.0000000000002232] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/11/2023]
Abstract
OBJECTIVE The aim of the study is to determine and compare the treatment efficacy of subacromial steroid injections and dextrose prolotherapy for chronic subacromial bursitis patients. DESIGN Fifty-four patients with chronic subacromial bursitis were enrolled in this double-blind randomized controlled trial. Shoulder Pain and Disability Index and visual analog scale were the primary outcomes. RESULTS The steroid group ( n = 26) exhibited significant visual analog scale score improvements comparing with baseline at weeks 2, 6, and 12; the dextrose prolotherapy group ( n = 28) exhibited visual analog scale score improvements at weeks 6 and 12. The steroid group displayed significant Shoulder Pain and Disability Index score improvements compared with baseline at weeks 2, 6, and 12; the dextrose prolotherapy group exhibited significant score decreases at weeks 2 and 6. Compared with the dextrose prolotherapy group, the steroid group demonstrated significantly greater decreases in visual analog scale scores at weeks 2 and 6; the steroid group showed significantly greater decreases in Shoulder Pain and Disability Index scores at weeks 2, 6, and 12. CONCLUSIONS Both hypertonic dextrose prolotherapy and steroid injections can provide short-term improvements of pain and disability among chronic subacromial bursitis patients. Moreover, steroid injections showed better effectiveness than hypertonic dextrose prolotherapy in ameliorating pain and improving function.
Collapse
Affiliation(s)
- Lien-Chieh Lin
- From the Department of Physical Medicine and Rehabilitation, Shuang Ho Hospital, Taipei Medical University, Taipei, Taiwan (L-CL, Y-HL, Y-WC, T-HH, T-HL, S-WH); Department of Physical Medicine and Rehabilitation, School of Medicine, College of Medicine, Taipei Medical University, Taipei, Taiwan (L-CL, Y-HL, Y-WC, T-HH, T-HL, S-WH); and Rehabilitation Department, King Chulalongkorn Memorial Hospital, Bangkok, Thailand (TV)
| | | | | | | | | | | | | |
Collapse
|
5
|
Emam M, Cohen C, Willeford S, Mahesh K, Le MQ, Wilckens J. Role of Conservative Treatment vs Surgical Treatment for Rotator Cuff Tears: A Narrative Review. CURRENT PHYSICAL MEDICINE AND REHABILITATION REPORTS 2023. [DOI: 10.1007/s40141-023-00389-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/22/2023]
|
6
|
Prolotherapy. Phys Med Rehabil Clin N Am 2023; 34:165-180. [DOI: 10.1016/j.pmr.2022.08.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
7
|
El-Sherif SM, Abdel-Hamid MM, Noureldin JMAM, Fahmy HM, Abdel-Naby HMA. Effectiveness of lyophilized growth factors injection for subacromial impingement syndrome: a prospective randomized double-blind placebo-controlled study. J Orthop Surg Res 2023; 18:78. [PMID: 36721157 PMCID: PMC9887845 DOI: 10.1186/s13018-023-03548-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/12/2022] [Accepted: 01/17/2023] [Indexed: 02/02/2023] Open
Abstract
BACKGROUND Platelet-derived lyophilized growth factors (L-GFs) use a standardized number of allogenic pathogen-free platelets instead of autologous platelets used in PRP as a source of growth factors. This study aimed to evaluate the efficacy of L-GF injection versus placebo in subacromial impingement (SIS) treatment. METHODS The current randomized double-blind placebo-controlled study included sixty patients (40 females and 20 males, aged between 24 and 75 years) diagnosed with SIS (both clinically and sonographically). Patients were randomly assigned to two equal groups. Under ultrasound guidance, group 1 received subacromial saline injection, and group 2 received L-GF injection. Clinical examination, pain visual analogue scale (VAS), shoulder pain and disability index (SPADI) and shoulder ultrasound were performed before and at the 8th week after injection. RESULTS Follow-up assessment showed statistically significant improvement in the L-GF group regarding active flexion, active and passive internal rotation and extension, SPADI-disability scale, VAS and thickness of the supraspinatus tendon by US. Regression analysis showed that group 1 was approximately 30 times more likely than the L-GF group to experience painful arc at follow-up. Both groups showed statistically significant improvement in SPADI-pain scale and SPADI-total, flexion and abduction (still the mean value of abduction was significantly higher in the L-GF group). CONCLUSIONS L-GF injection resulted in clinically significant reductions in pain and functional disability outcomes in patients with SIS. An objective significant reduction in the thickness of the supraspinatus tendon, measured by ultrasound, in the L-GF group hopefully encourages proper healing and functioning in SIS. TRIAL REGISTRATION The identification number is NCT04330027, date of first registration (01/04/2020). Unique on 21/11/2019, Protocol ID: 0106178.
Collapse
Affiliation(s)
- Sherine Mahmoud El-Sherif
- Present Address: Department Physical Medicine, Rheumatology and Rehabilitation, Faculty of Medicine, Alexandria University, Medaan El-Khartoom Square, Al-Azaritah, Alexandria, Egypt
| | - Mowaffak Moustafa Abdel-Hamid
- Present Address: Department Physical Medicine, Rheumatology and Rehabilitation, Faculty of Medicine, Alexandria University, Medaan El-Khartoom Square, Al-Azaritah, Alexandria, Egypt
| | - Jailan Mohamed Ashraf Mohamed Noureldin
- Present Address: Department Physical Medicine, Rheumatology and Rehabilitation, Faculty of Medicine, Alexandria University, Medaan El-Khartoom Square, Al-Azaritah, Alexandria, Egypt
| | | | - Hoda Mohamed Aly Abdel-Naby
- Present Address: Department Physical Medicine, Rheumatology and Rehabilitation, Faculty of Medicine, Alexandria University, Medaan El-Khartoom Square, Al-Azaritah, Alexandria, Egypt
| |
Collapse
|
8
|
Effectiveness of Ultrasound-Guided Corticosteroid Injections, Prolotherapy, and Exercise Therapy on Partial-Thickness Supraspinatus Tears. J Sport Rehabil 2022; 31:869-875. [PMID: 35453118 DOI: 10.1123/jsr.2021-0254] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2021] [Revised: 03/03/2022] [Accepted: 03/04/2022] [Indexed: 11/18/2022]
Abstract
CONTEXT To investigate the effects of steroid injection (STE), prolotherapy (PRO), and exercise therapy in the treatment of partial tears of the supraspinatus. DESIGN A retrospective cohort study. METHODS A total of 64 patients with clinically and radiologically diagnosed partial-thickness supraspinatus tear who received either a cortisone injection (STE), dextrose PRO, or physical therapy combined with home-based exercise therapy were included. Main outcome measures were patients' visual analog scale scores, Western Ontario Rotator Cuff (WORC) Index scores, and the Shoulder Pain and Disability Index scores at the baseline, 3 weeks, and 3 months. RESULTS The effect of group, time, and group-time interaction on visual analog scale, WORC, and Shoulder Pain and Disability Index scores was statistically significant (P < .001). Visual analog scale and Shoulder Pain and Disability Index scores were the lowest in the STE group at week 3, and the lowest in the PRO group at month 3 (P < .001). WORC scores of the STE group were the highest at week 3 (P < .001). At month 3, WORC scores of STE and PRO groups were similar (P = .089), but significantly higher than exercise therapy. CONCLUSIONS Corticosteroids provide a fast pain-relieving effect and improvement in function in partial-thickness rotator cuff tears, but these effects diminish over time, whereas PRO provides a long-lasting effect.
Collapse
|