1
|
Woo KMC, Simon GW, Akindutire O, Aphinyanaphongs Y, Austrian JS, Kim JG, Genes N, Goldenring JA, Major VJ, Pariente CS, Pineda EG, Kang SK. Evaluation of GPT-4 ability to identify and generate patient instructions for actionable incidental radiology findings. J Am Med Inform Assoc 2024; 31:1983-1993. [PMID: 38778578 PMCID: PMC11339516 DOI: 10.1093/jamia/ocae117] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2023] [Revised: 03/30/2024] [Accepted: 05/06/2024] [Indexed: 05/25/2024] Open
Abstract
OBJECTIVES To evaluate the proficiency of a HIPAA-compliant version of GPT-4 in identifying actionable, incidental findings from unstructured radiology reports of Emergency Department patients. To assess appropriateness of artificial intelligence (AI)-generated, patient-facing summaries of these findings. MATERIALS AND METHODS Radiology reports extracted from the electronic health record of a large academic medical center were manually reviewed to identify non-emergent, incidental findings with high likelihood of requiring follow-up, further sub-stratified as "definitely actionable" (DA) or "possibly actionable-clinical correlation" (PA-CC). Instruction prompts to GPT-4 were developed and iteratively optimized using a validation set of 50 reports. The optimized prompt was then applied to a test set of 430 unseen reports. GPT-4 performance was primarily graded on accuracy identifying either DA or PA-CC findings, then secondarily for DA findings alone. Outputs were reviewed for hallucinations. AI-generated patient-facing summaries were assessed for appropriateness via Likert scale. RESULTS For the primary outcome (DA or PA-CC), GPT-4 achieved 99.3% recall, 73.6% precision, and 84.5% F-1. For the secondary outcome (DA only), GPT-4 demonstrated 95.2% recall, 77.3% precision, and 85.3% F-1. No findings were "hallucinated" outright. However, 2.8% of cases included generated text about recommendations that were inferred without specific reference. The majority of True Positive AI-generated summaries required no or minor revision. CONCLUSION GPT-4 demonstrates proficiency in detecting actionable, incidental findings after refined instruction prompting. AI-generated patient instructions were most often appropriate, but rarely included inferred recommendations. While this technology shows promise to augment diagnostics, active clinician oversight via "human-in-the-loop" workflows remains critical for clinical implementation.
Collapse
Affiliation(s)
- Kar-mun C Woo
- Ronald O. Perelman Department of Emergency Medicine, NYU Grossman School of Medicine, New York, NY 10016, United States
| | - Gregory W Simon
- Ronald O. Perelman Department of Emergency Medicine, NYU Grossman School of Medicine, New York, NY 10016, United States
| | - Olumide Akindutire
- Ronald O. Perelman Department of Emergency Medicine, NYU Grossman School of Medicine, New York, NY 10016, United States
| | - Yindalon Aphinyanaphongs
- Department of Population Health, NYU Grossman School of Medicine, New York, NY 10016, United States
- Department of Health Informatics, Medical Center IT, NYU Langone Health, New York, NY 10016, United States
| | - Jonathan S Austrian
- Department of Health Informatics, Medical Center IT, NYU Langone Health, New York, NY 10016, United States
- Department of Medicine, NYU Grossman School of Medicine, New York, NY 10016, United States
| | - Jung G Kim
- Ronald O. Perelman Department of Emergency Medicine, NYU Grossman School of Medicine, New York, NY 10016, United States
- Institute for Innovations in Medical Education, NYU Langone Health, New York, NY 10016, United States
| | - Nicholas Genes
- Ronald O. Perelman Department of Emergency Medicine, NYU Grossman School of Medicine, New York, NY 10016, United States
- Department of Health Informatics, Medical Center IT, NYU Langone Health, New York, NY 10016, United States
| | - Jacob A Goldenring
- Ronald O. Perelman Department of Emergency Medicine, NYU Grossman School of Medicine, New York, NY 10016, United States
| | - Vincent J Major
- Department of Population Health, NYU Grossman School of Medicine, New York, NY 10016, United States
- Department of Health Informatics, Medical Center IT, NYU Langone Health, New York, NY 10016, United States
| | - Chloé S Pariente
- Department of Health Informatics, Medical Center IT, NYU Langone Health, New York, NY 10016, United States
| | - Edwin G Pineda
- MCIT Clinical Systems—ASAP application, NYU Langone Health, New York, NY 10016, United States
| | - Stella K Kang
- Department of Population Health, NYU Grossman School of Medicine, New York, NY 10016, United States
- Department of Radiology, NYU Grossman School of Medicine, New York, NY 10016, United States
| |
Collapse
|
2
|
Clifford SK, Gadoura AM, Ishola OO, Bashir M. Mass Effect of Large Pelvic Lipoma Resulting in Femoral Hernia. Cureus 2024; 16:e61148. [PMID: 38933648 PMCID: PMC11200145 DOI: 10.7759/cureus.61148] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/25/2024] [Indexed: 06/28/2024] Open
Abstract
A 73-year-old lady presented with a three-day history of constipation, vomiting, and abdominal pain. On examination, a right femoral hernia was identified, and this was confirmed on computed tomography imaging with evidence of mechanical small bowel obstruction. There was an incidental finding of a large pelvic lipoma causing a mass effect. This lady underwent open repair of the femoral hernia. Intra-abdominal lipomatosis is a rare finding and can present itself in a variety of manifestations, or it can be identified as an incidental finding on cross-sectional imaging. Bowel obstructions, abdominal pain, lipoma, and abdominopelvic hernias are some examples of symptomatic presentations of intra-abdominal lipomas.
Collapse
Affiliation(s)
| | - Ahmed M Gadoura
- General Surgery, Our Lady of Lourdes Hospital, Drogheda, Drogheda, IRL
| | | | - Masoud Bashir
- General Surgery, Our Lady of Lourdes Hospital, Drogheda, Drogheda, IRL
| |
Collapse
|
3
|
Zimmer L, McDade C, Beyhaghi H, Purser M, Textoris J, Krause A, Blanc E, Pavlov V, Earnshaw S. Cost-Effectiveness of Blood-Based Brain Biomarkers for Screening Adults with Mild Traumatic Brain Injury in the French Health Care Setting. J Neurotrauma 2023; 40:706-719. [PMID: 36267001 PMCID: PMC10061334 DOI: 10.1089/neu.2022.0270] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
Two blood-based brain biomarker tests such as the combination of glial fibrillary acidic protein and ubiquitin C-terminal hydrolase-L1 (GFAP+UCH-L1) or S100B have potential to reduce the need for head computed tomography (CT) scanning in patients with mild traumatic brain injury (mTBI). We assessed the clinical and economic impact of using GFAP+UCH-L1 versus CT scan and GFAP+UCH-L1 versus S100B to screen adults with suspected mTBI presenting to an emergency department (ED). A decision model was developed to estimate costs and health outcomes of GFAP+UCH-L1, CT scan, and S100B associated with these screening protocols. Model parameters were extracted from peer-reviewed articles, clinical guidelines, and expert opinion. Analysis was performed from a French health care system perspective (costs in 2020 euros). In the model, patients with a positive biomarker receive a CT scan to confirm the presence of intracranial lesions (ICLs). Depending on clinical state and biomarker and CT results, patients were discharged immediately, kept for observation in the ED, admitted for in-hospital stay and observation, or admitted for surgical management. Incorrect test results may lead to delayed treatment and poor outcomes or overtreatment. GFAP+UCH-L1 use was associated with an overall decrease in CT scans when compared with CT screening or S100B use (325.42 and 46.43 CTs per 1000 patients, respectively). The use of GFAP+UCH-L1 resulted in modest cost savings when compared with CT scanning and with S100B. In all cases, use of GFAP+UCH-L1 marginally improved quality-adjusted life-years (QALYs) and outcomes. Thus, screening with GFAP+UCH-L1 reduced the need for CT scans when compared with systematic CT scan screening or use of S100B while maintaining similar costs and health outcomes.
Collapse
Affiliation(s)
| | - Cheryl McDade
- RTI Health Solutions, Research Triangle Park, North Carolina, USA
| | | | - Molly Purser
- RTI Health Solutions, Research Triangle Park, North Carolina, USA
| | | | | | | | | | | |
Collapse
|
4
|
Pacella CB, Yealy DM. Follow-up of Incidental Radiology Findings: Is the Cart Ahead of the Horse? Ann Emerg Med 2022; 80:257-259. [PMID: 35995513 DOI: 10.1016/j.annemergmed.2022.07.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Revised: 07/08/2022] [Accepted: 07/11/2022] [Indexed: 11/27/2022]
Affiliation(s)
- Charissa B Pacella
- Department of Emergency Medicine, University of Pittsburgh School of Medicine, Pittsburgh, PA.
| | - Donald M Yealy
- Department of Emergency Medicine, University of Pittsburgh School of Medicine, Pittsburgh, PA
| |
Collapse
|
5
|
Barrett TW, Garland NM, Freeman CL, Klar K, Dahlke J, Lancaster P, Prisco L, Chang SS, Goff LW, Russ S, Jones ID. Catching Those Who Fall Through the Cracks: Integrating a Follow-Up Process for Emergency Department Patients with Incidental Radiologic Findings. Ann Emerg Med 2022; 80:235-242. [PMID: 35752517 DOI: 10.1016/j.annemergmed.2022.04.026] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2022] [Revised: 04/15/2022] [Accepted: 04/20/2022] [Indexed: 11/01/2022]
Abstract
STUDY OBJECTIVE Abnormal findings unrelated to the indication for testing are identified on emergency department (ED) imaging studies. We report the design and implementation of an electronic health record-based interdisciplinary referral system and our experience from the first 13 months of ensuring that patients with incidental radiology findings were connected with the appropriate outpatient surveillance. METHODS Our informatics team standardized the contemporaneous reporting of critical radiology alerts using our ED trackboard and created a companion follow-up request form for the treating ED clinicians to complete. The forms were routed to nurse case managers, who arranged follow-ups based on the findings and clinical significance. The primary outcome was the proportion of ED patient visits with identified incidental findings that had documented communication of the incidental findings and surveillance plans. RESULTS Over the first 13 months after implementation, 932 ED patient visits had critical radiology alert referrals, for a total of 982 incidental findings. The primary outcome (confirmed post-ED communication and documented follow-up plan) was attained in 888 (95.3%, 95% confidence interval [CI] 93.9% to 96.6%) ED patient visits with confirmed post-ED communication and documented follow-up plans. The team was unable to contact or confirm follow-up with 44 (4.7%, 95% CI 3.4 to 6.1) patients by telephone or through the health care system's electronic communication tools. CONCLUSION We report the implementation of a standardized notification and referral system for ED patients with incidental radiology findings. The development of a reliable notification and follow-up system is an important patient safety intervention given the opportunity to potentially identify undiagnosed malignancies.
Collapse
Affiliation(s)
- Tyler W Barrett
- Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, TN; Vanderbilt Ingram Cancer Center, Vanderbilt University Medical Center, Nashville, TN.
| | - Nicholas M Garland
- Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, TN; Vanderbilt Ingram Cancer Center, Vanderbilt University Medical Center, Nashville, TN
| | - Clifford L Freeman
- Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, TN; Vanderbilt Ingram Cancer Center, Vanderbilt University Medical Center, Nashville, TN
| | - Katharine Klar
- Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, TN; Vanderbilt Ingram Cancer Center, Vanderbilt University Medical Center, Nashville, TN
| | - Jan Dahlke
- Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, TN; Vanderbilt Ingram Cancer Center, Vanderbilt University Medical Center, Nashville, TN
| | - Penny Lancaster
- Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, TN; Vanderbilt Ingram Cancer Center, Vanderbilt University Medical Center, Nashville, TN
| | - Larry Prisco
- Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, TN; Vanderbilt Ingram Cancer Center, Vanderbilt University Medical Center, Nashville, TN
| | - Sam S Chang
- Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, TN; Vanderbilt Ingram Cancer Center, Vanderbilt University Medical Center, Nashville, TN
| | - Laura W Goff
- Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, TN; Vanderbilt Ingram Cancer Center, Vanderbilt University Medical Center, Nashville, TN
| | - Stephan Russ
- Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, TN; Vanderbilt Ingram Cancer Center, Vanderbilt University Medical Center, Nashville, TN
| | - Ian D Jones
- Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, TN; Vanderbilt Ingram Cancer Center, Vanderbilt University Medical Center, Nashville, TN
| |
Collapse
|
6
|
Evans CS, Arthur R, Kane M, Omofoye F, Chung AE, Moreton E, Moore C. Incidental Radiology Findings on Computed Tomography Studies in Emergency Department Patients: A Systematic Review and Meta-Analysis. Ann Emerg Med 2022; 80:243-256. [PMID: 35717273 DOI: 10.1016/j.annemergmed.2022.03.027] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2022] [Revised: 03/16/2022] [Accepted: 03/30/2022] [Indexed: 11/01/2022]
Abstract
STUDY OBJECTIVE An incidental finding is defined as a newly discovered mass or lesion detected on imaging performed for an unrelated reason. The identification of an incidental finding may be an opportunity for the early detection of a serious medical condition, including a malignancy. However, little is known about the prevalence of incidental findings in the emergency department (ED) setting and the strategies that can be used to mitigate the risk associated with them in the ED. This study aimed to estimate the overall prevalence of incidental findings and to summarize the currently described measures to mitigate the risks associated with incidental findings. METHODS On November 22, 2020, a systematic literature search of PubMed, EMBASE, and Scopus was performed for studies that were published in peer-reviewed journals and reported the prevalence of incidental findings in computed tomography (CT) scans in patients in the ED. Patients who received CT scans that included the head, neck, chest, or abdomen/pelvis were included. The study characteristics, overall prevalence of incidental findings, prevalence of incidental findings by body region, and prespecified subgroups were extracted. The criteria used for risk stratification within individual studies were also extracted. Pooled estimates were calculated using a random-effects meta-analysis. RESULTS A total of 1,385 studies were identified, and 69 studies met the inclusion criteria. The included studies represented 147,763 ED encounters or radiology reports across 16 countries, and 83% of studies were observational, cross-sectional studies. A total of 35 studies (50.7%) were in trauma patients. A large degree of heterogeneity was observed across the included studies. The overall pooled prevalence estimate for any incidental finding was 31.3% (95% confidence interval 24.4% to 39.1%). We found great variation in the methods described to mitigate the risk associated with incidental findings, including a lack of standardized risk stratification, inconsistent documentation practices, and only a small subset of studies describing prospective interventions aimed at improving the recognition and management of incidental findings from the ED. CONCLUSION In patients in the ED receiving CT scans, incidental findings are commonly encountered across a broad range of ED chief complaints. This review highlights the existence of great heterogeneity in the definitions used to classify incidental findings. Future studies are needed to determine a clinically feasible categorization standard or terminology for commonly encountered incidental findings in the ED setting to standardize classification and documentation.
Collapse
Affiliation(s)
- Christopher S Evans
- Clinical Informatics Fellowship Program, UNC Hospitals, Chapel Hill, NC; Department of Emergency Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC.
| | - Rodney Arthur
- School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Michael Kane
- Clinical Informatics Fellowship Program, UNC Hospitals, Chapel Hill, NC; Department of Psychiatry, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Fola Omofoye
- School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Arlene E Chung
- Clinical Informatics Fellowship Program, UNC Hospitals, Chapel Hill, NC; Department of Biostatistics & Bioinformatics, Duke School of Medicine, Durham, NC
| | - Elizabeth Moreton
- Health Sciences Library, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Carlton Moore
- Clinical Informatics Fellowship Program, UNC Hospitals, Chapel Hill, NC; Department of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC
| |
Collapse
|
7
|
Niedermeier S, Wania R, Lampart A, Stahl R, Trumm C, Kammerlander C, Böcker W, Nickel CH, Bingisser R, Armbruster M, Pedersen V. Incidental CT Findings in the Elderly with Low-Energy Falls: Prevalence and Implications. Diagnostics (Basel) 2022; 12:diagnostics12020354. [PMID: 35204445 PMCID: PMC8871195 DOI: 10.3390/diagnostics12020354] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2021] [Revised: 01/10/2022] [Accepted: 01/24/2022] [Indexed: 02/04/2023] Open
Abstract
Background: Computed tomography (CT) is commonly used in trauma care, with increasing implementation during the emergency work-up of elderly patients with low-energy falls (LEF). The prevalence of incidental findings (IFs) resulting from CT imaging and requiring down-stream actions in this patient cohort is unknown. We have investigated the prevalence and urgency of IFs from emergency CT examinations in these patients. Methods: A total of 2871 patients with LEF and emergency CT examinations were consecutively included in this retrospective cohort study. The primary endpoint was the prevalence of IFs; the secondary endpoint was their urgency. Results: The median age was 82 years (64.2% were women). IFs were identified in 73.9% of patients, with an average of 1.6 IFs per patient. Of all IFs, 16.4% were classified as urgent or relevant, predominantly in the abdomen, chest and neck. Increasing age was associated with the prevalence of an IF (odds ratio: 1.053, 95% confidence interval: 1.042–1.064). Significantly more IFs were found in female patients (75.2% vs. 71.5%). Conclusion: IFs resulting from CT examinations of the elderly are frequent, but in more than 8 out of 10, they are harmless or currently asymptomatic. For the benefit of an accurate diagnosis of traumatic lesions, concerns about IFs with respect to disease burden, further work-up and resource utilisation might be disregarded.
Collapse
Affiliation(s)
- Sandra Niedermeier
- Department of General, Trauma and Reconstructive Surgery, University Hospital, LMU Munich, Marchioninistr. 15, 81377 Munich, Germany; (S.N.); (R.W.); (C.K.); (W.B.)
| | - Rebecca Wania
- Department of General, Trauma and Reconstructive Surgery, University Hospital, LMU Munich, Marchioninistr. 15, 81377 Munich, Germany; (S.N.); (R.W.); (C.K.); (W.B.)
| | - Alina Lampart
- Department of Medicine, Kantonsspital Lucerne, Spitalstrasse, 6000 Lucerne, Switzerland;
| | - Robert Stahl
- Institute of Diagnostic and Interventional Neuroradiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377 Munich, Germany; (R.S.); (C.T.)
| | - Christoph Trumm
- Institute of Diagnostic and Interventional Neuroradiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377 Munich, Germany; (R.S.); (C.T.)
| | - Christian Kammerlander
- Department of General, Trauma and Reconstructive Surgery, University Hospital, LMU Munich, Marchioninistr. 15, 81377 Munich, Germany; (S.N.); (R.W.); (C.K.); (W.B.)
- Trauma Hospital Styria, Goestinger Straße 24, 8020 Graz, Austria
| | - Wolfgang Böcker
- Department of General, Trauma and Reconstructive Surgery, University Hospital, LMU Munich, Marchioninistr. 15, 81377 Munich, Germany; (S.N.); (R.W.); (C.K.); (W.B.)
| | - Christian H. Nickel
- Department of Emergency Medicine, University Hospital Basel, Petersgraben 2, 4031 Basel, Switzerland; (C.H.N.); (R.B.)
| | - Roland Bingisser
- Department of Emergency Medicine, University Hospital Basel, Petersgraben 2, 4031 Basel, Switzerland; (C.H.N.); (R.B.)
| | - Marco Armbruster
- Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377 Munich, Germany;
| | - Vera Pedersen
- Department of General, Trauma and Reconstructive Surgery, University Hospital, LMU Munich, Marchioninistr. 15, 81377 Munich, Germany; (S.N.); (R.W.); (C.K.); (W.B.)
- Correspondence: ; Tel.: +49-89-440072005; Fax: +49-89-440072102
| |
Collapse
|