1
|
Cao C, Sang J, Arora R, Chen D, Kloosterman R, Cecere M, Gorla J, Saleh R, Drennan I, Teja B, Fehlings M, Ronksley P, Leung AA, Weisz DE, Ware H, Whelan M, Emerson DB, Arora RK, Bobrovitz N. Development of Prompt Templates for Large Language Model-Driven Screening in Systematic Reviews. Ann Intern Med 2025; 178:389-401. [PMID: 39993313 DOI: 10.7326/annals-24-02189] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/26/2025] Open
Abstract
BACKGROUND Systematic reviews (SRs) are hindered by the initial rigorous article screen, which delays access to reliable information synthesis. OBJECTIVE To develop generic prompt templates for large language model (LLM)-driven abstract and full-text screening that can be adapted to different reviews. DESIGN Diagnostic test accuracy. SETTING 48 425 citations were tested for abstract screening across 10 SRs. Full-text screening evaluated all 12 690 freely available articles from the original search. Prompt development used the GPT4-0125-preview model (OpenAI). PARTICIPANTS None. MEASUREMENTS Large language models were prompted to include or exclude articles based on SR eligibility criteria. Model outputs were compared with original SR author decisions after full-text screening to evaluate performance (accuracy, sensitivity, and specificity). RESULTS Optimized prompts using GPT4-0125-preview achieved a weighted sensitivity of 97.7% (range, 86.7% to 100%) and specificity of 85.2% (range, 68.3% to 95.9%) in abstract screening and weighted sensitivity of 96.5% (range, 89.7% to 100.0%) and specificity of 91.2% (range, 80.7% to 100%) in full-text screening across 10 SRs. In contrast, zero-shot prompts had poor sensitivity (49.0% abstract, 49.1% full-text). Across LLMs, Claude-3.5 (Anthropic) and GPT4 variants had similar performance, whereas Gemini Pro (Google) and GPT3.5 (OpenAI) models underperformed. Direct screening costs for 10 000 citations differed substantially: Where single human abstract screening was estimated to require more than 83 hours and $1666.67 USD, our LLM-based approach completed screening in under 1 day for $157.02 USD. LIMITATIONS Further prompt optimizations may exist. Retrospective study. Convenience sample of SRs. Full-text screening evaluations were limited to free PubMed Central full-text articles. CONCLUSION A generic prompt for abstract and full-text screening achieving high sensitivity and specificity that can be adapted to other SRs and LLMs was developed. Our prompting innovations may have value to SR investigators and researchers conducting similar criteria-based tasks across the medical sciences. PRIMARY FUNDING SOURCE None.
Collapse
Affiliation(s)
- Christian Cao
- Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, and Centre for Health Informatics, Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada (C.C.)
| | - Jason Sang
- Stripe, San Francisco, California (J.S.)
| | - Rohit Arora
- Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts (R.A.)
| | - David Chen
- Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada (D.C., R.K., M.C., J.G., R.S.)
| | - Robert Kloosterman
- Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada (D.C., R.K., M.C., J.G., R.S.)
| | - Matthew Cecere
- Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada (D.C., R.K., M.C., J.G., R.S.)
| | - Jaswanth Gorla
- Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada (D.C., R.K., M.C., J.G., R.S.)
| | - Richard Saleh
- Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada (D.C., R.K., M.C., J.G., R.S.)
| | - Ian Drennan
- Temerty Faculty of Medicine, University of Toronto, Department of Emergency Services and Sunnybrook Research Institute, Sunnybrook Health Sciences Centre, and Ornge Air Ambulance and Critical Care Transport, Toronto, Ontario, Canada (I.D.)
| | - Bijan Teja
- Department of Anesthesiology and Pain Medicine, University of Toronto, and Department of Anesthesia and Critical Care Medicine, St. Michael's Hospital, Toronto, Ontario, Canada (B.T.)
| | - Michael Fehlings
- Department of Surgery, Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada (M.F.)
| | - Paul Ronksley
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada (P.R.)
| | - Alexander A Leung
- Department of Medicine and Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada (A.A.L.)
| | - Dany E Weisz
- Department of Newborn and Developmental Paediatrics, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada (D.E.W.)
| | - Harriet Ware
- Centre for Health Informatics, Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada (H.W., M.W., R.K.A.)
| | - Mairead Whelan
- Centre for Health Informatics, Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada (H.W., M.W., R.K.A.)
| | | | - Rahul K Arora
- Centre for Health Informatics, Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada (H.W., M.W., R.K.A.)
| | - Niklas Bobrovitz
- Centre for Health Informatics, Department of Community Health Sciences, and Department of Emergency Medicine, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada (N.B.)
| |
Collapse
|
2
|
Mondal S, Emmanuel M, Reindorf R, Mali M, Toursavadkohi S, Ghoreishi M, Williams B, Deshpande SP, Fernando RJ, Augoustides JG, Blacker SN, Smeltz AM. Anesthetic and Perioperative Considerations in Endovascular Aortic Valve, Aortic Root, and Ascending Aorta Repair (Endo-Bentall Procedure) for Acute Type A Aortic Dissection: A Single Academic Center Experience. J Cardiothorac Vasc Anesth 2025:S1053-0770(25)00125-9. [PMID: 40011137 DOI: 10.1053/j.jvca.2025.02.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/07/2025] [Accepted: 02/08/2025] [Indexed: 02/28/2025]
Affiliation(s)
- Samhati Mondal
- Department of Anesthesiology, University of Maryland School of Medicine, Baltimore, MD
| | - Maxwell Emmanuel
- Office of Student Research, University of Maryland School of Medicine, Baltimore, MD
| | - Rachel Reindorf
- Department of Anesthesiology, University of Maryland School of Medicine, Baltimore, MD
| | | | | | - Mehrdad Ghoreishi
- Department of Surgery, Baptist Health Heart and Vascular Care, Miami Cardiac and Vascular Institute, Miami, FL
| | - Brittney Williams
- Department of Anesthesiology, University of Maryland School of Medicine, Baltimore, MD
| | - Seema P Deshpande
- Department of Anesthesiology, University of Maryland School of Medicine, Baltimore, MD
| | - Rohesh J Fernando
- Department of Anesthesiology, Cardiothoracic Section, Wake Forest University School of Medicine, Medical Center Boulevard, Winston Salem, NC.
| | - John G Augoustides
- Cardiovascular and Thoracic Division, Department of Anesthesiology and Critical Care, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA
| | - Samuel N Blacker
- University of North Carolina School of Medicine, Chapel Hill, NC
| | - Alan M Smeltz
- University of North Carolina School of Medicine, Chapel Hill, NC
| |
Collapse
|
3
|
Nie C, Chen K, Gu S, Lyu F, Jiang J, Xia X, Zheng C. Hybrid decompression-based surgical strategy for treating multilevel thoracic ossification of the ligamentum flavum: a retrospective study. Asian Spine J 2025; 19:74-84. [PMID: 40037322 PMCID: PMC11895116 DOI: 10.31616/asj.2024.0366] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/04/2024] [Revised: 10/04/2024] [Accepted: 11/01/2024] [Indexed: 03/06/2025] Open
Abstract
STUDY DESIGN A retrospective study design was adopted. PURPOSE This study investigated the surgical modification of laminectomy, including piecemeal and en bloc resections, and compared this hybrid approach with conventional en bloc laminectomy for treating multilevel thoracic ossification of the ligamentum flavum (TOLF). OVERVIEW OF LITERATURE En bloc laminectomy is the most commonly used method for managing symptomatic TOLF. However, this approach can easily cause intraoperative spinal cord irritation, dural tear, and cerebrospinal fluid leakage (CFL). METHODS Motor-evoked potentials (MEPs) and somatosensory-evoked potentials (SEPs) were recorded in 48 patients with TOLF (hybrid 20 vs. en bloc 28) during surgery. Patients were categorized based on MEP/SEP improvement, deterioration, or no change, and MEP/SEP improvement rates were measured in the improvement group. Furthermore, all patients were assessed using the Ashworth and modified Japanese Orthopedic Association scores. RESULTS The incidences of both MEP/SEP improvement (21.4% vs. 25.0%, p=0.772) and deterioration (21.4% vs. 20.0%, p=0.904) were similar between the en bloc and hybrid laminectomy groups, and no difference in preoperative and postoperative clinical assessments was observed between the two groups (p>0.05). In four patients (4/28, 14.3%) undergoing en bloc laminectomy, MEP amplitudes initially increased after OLF removal but gradually decreased. This delayed MEP reduction did not occur in the hybrid laminectomy group. Furthermore, more patients undergoing en bloc laminectomy had CFL than those undergoing hybrid laminectomy (46.4% vs. 15.0%, p=0.023). In the improvement group, the hybrid laminectomy group exhibited higher MEP improvement rates in the bilateral abductor hallucis than the en bloc laminectomy group (left side: 213.4%±35.9% vs. 152.5%±41.0%, p=0.028; right side: 201.2%±32.0% vs. 145.2%±46.3%, p=0.043). CONCLUSIONS Compared with en bloc laminectomy, hybrid laminectomy may be a safe and effective method for treating multilevel TOLF, potentially reducing intraoperative spinal cord irritation and CFL and causing relatively better functional recovery.
Collapse
Affiliation(s)
- Cong Nie
- Department of Orthopaedics, Huashan Hospital, Fudan University, Shanghai,
China
| | - Kaiwen Chen
- Department of Orthopaedics, Huashan Hospital, Fudan University, Shanghai,
China
| | - Shenyan Gu
- Department of Orthopaedics, Huashan Hospital, Fudan University, Shanghai,
China
| | - Feizhou Lyu
- Department of Orthopaedics, Huashan Hospital, Fudan University, Shanghai,
China
- Department of Orthopaedics, The Fifth People’s Hospital, Fudan University, Shanghai,
China
| | - Jianyuan Jiang
- Department of Orthopaedics, Huashan Hospital, Fudan University, Shanghai,
China
| | - Xinlei Xia
- Department of Orthopaedics, Huashan Hospital, Fudan University, Shanghai,
China
| | - Chaojun Zheng
- Department of Orthopaedics, Huashan Hospital, Fudan University, Shanghai,
China
| |
Collapse
|
4
|
Keil LG, Bomar JD, Bower CR, Venne MH, Curran PF, Upasani VV. Intraoperative Neuromonitoring During Periacetabular Osteotomy Provides Actionable Alerts. JB JS Open Access 2025; 10:e24.00126. [PMID: 40094075 PMCID: PMC11896103 DOI: 10.2106/jbjs.oa.24.00126] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 03/19/2025] Open
Abstract
Background Bernese periacetabular osteotomy (PAO) for symptomatic acetabular dysplasia and femoroacetabular impingement has become increasingly common, with a corresponding increase in the incidence of adverse outcomes. The rate of major neurological injury (excluding lateral femoral cutaneous nerve injury) during PAO has been reported to be around 2%. Previous publications have recommended the use of intraoperative neuromonitoring (IONM) to mitigate risk of major neurological injury during PAO, but its use has not become universal among PAO surgeons as it has among spine surgeons. The purpose of this study was to report the incidence and clinical significance of IONM alerts in a single-surgeon, consecutive cohort of patients treated with Bernese PAO. Methods After a permanent peripheral nerve injury during a PAO without IONM, IONM has been used at our institution in every PAO. Motor evoked potentials and somatosensory monitoring are performed throughout the procedure. We conducted a retrospective review of all PAOs performed after this practice change between 2017 and 2023. Medical records were reviewed for all IONM alerts, surgical team responses to alerts, and postoperative neurological status. Results All 94 PAOs performed with IONM in 82 patients during the study period were included. The mean age was 19 years (range 11-38). Significant IONM alerts occurred in 10 of 94 PAOs (11%) in 10 patients. Of these 10 alerts, 6 resulted in action taken by the surgical team including adjustment of acetabular fragment correction, leg repositioning, or stockinette or boot loosening. The remaining 4 alerts were due to anesthetic or systemic causes or technical issues with the neuromonitoring electrodes. No patients had a detectable neurological deficit postoperatively. Conclusions IONM may produce alerts in approximately 1 in 9 periacetabular osteotomies. These alerts are actionable and may improve patient safety and minimize the non-negligible risk of major nerve injury. This study provides additional evidence to support the utility of IONM in PAO. Level of Evidence Level III-retrospective cohort study. See Instructions for Authors for a complete description of levels of evidence.
Collapse
Affiliation(s)
- Lukas G. Keil
- Department of Orthopedics, Rady Children's Hospital, San Diego, California
| | - James D. Bomar
- Department of Orthopedics, Rady Children's Hospital, San Diego, California
| | | | | | - Patrick F. Curran
- Department of Orthopedics, Rady Children's Hospital, San Diego, California
| | | |
Collapse
|
5
|
Rocos B, Wong IH, Jentzsch T, Strantzas S, Lewis SJ. The Effect of Anaemia on Intra-operative Neuromonitoring Following Correction of Large Scoliosis Curves: Two Case Reports. Cureus 2024; 16:e59353. [PMID: 38817484 PMCID: PMC11138233 DOI: 10.7759/cureus.59353] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/25/2024] [Indexed: 06/01/2024] Open
Abstract
The correction of anemia is important in reversing significant intraoperative bilateral motor-evoked potential (MEP) loss following rod placement for correction of large scoliosis curves. This article presents a retrospective review of intraoperative neuromonitoring (IONM) data, anesthesia records, and medical charts of two patients with significant bilateral MEP changes associated with posterior spinal surgery for deformity correction. A 70 kg 12-year-old and a 44 kg 16-year-old female with main thoracic curves underwent a posterior scoliosis correction with multilevel posterior column osteotomies. Following rod insertion, significant reduction in the bilateral lower extremity MEP occurred in both cases despite mean arterial pressure exceeding 70 mmHg, which was presumed to be due to the scale of the correction attempted in the setting of haemorrhage which rendered the patient acutely anaemic, thus compromising cord vasculature and oxygen delivery. The rods were removed and packed red blood cell transfusions were administered in response to acute anaemia as a result of haemorrhage in both cases. Neither was noted to be anaemic preoperatively. Once the MEP signals improved, the rods were reinserted and correction was attempted, limited by neuromonitoring signals and resistance of the bony anchors to pullout. At closure, the MEPs were near baseline in the first case and >50% of baseline in the second. There were no changes in the somatosensory evoked potential signals in either case. Post-operative neurological function was normal in both patients. Correcting the circulating haemoglobin concentration through blood product resuscitation allowed for safe correction of spinal deformity in two cases with significant bilateral MEP loss following the initial placement of rods.
Collapse
Affiliation(s)
- Brett Rocos
- Orthopaedic Surgery, Duke University, Durham, USA
| | - Ian H Wong
- Division of Orthopaedic Surgery, Toronto Western Hospital, University Health Network, Toronto, CAN
| | | | | | - Stephen J Lewis
- Division of Orthopaedic Surgery, Toronto Western Hospital, University Health Network, Toronto, CAN
| |
Collapse
|