1
|
Croxford E, Gao Y, Patterson B, To D, Tesch S, Dligach D, Mayampurath A, Churpek MM, Afshar M. Development of a Human Evaluation Framework and Correlation with Automated Metrics for Natural Language Generation of Medical Diagnoses. medRxiv 2024:2024.03.20.24304620. [PMID: 38562730 PMCID: PMC10984060 DOI: 10.1101/2024.03.20.24304620] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
Abstract
In the evolving landscape of clinical Natural Language Generation (NLG), assessing abstractive text quality remains challenging, as existing methods often overlook generative task complexities. This work aimed to examine the current state of automated evaluation metrics in NLG in healthcare. To have a robust and well-validated baseline with which to examine the alignment of these metrics, we created a comprehensive human evaluation framework. Employing ChatGPT-3.5-turbo generative output, we correlated human judgments with each metric. None of the metrics demonstrated high alignment; however, the SapBERT score-a Unified Medical Language System (UMLS)- showed the best results. This underscores the importance of incorporating domain-specific knowledge into evaluation efforts. Our work reveals the deficiency in quality evaluations for generated text and introduces our comprehensive human evaluation framework as a baseline. Future efforts should prioritize integrating medical knowledge databases to enhance the alignment of automated metrics, particularly focusing on refining the SapBERT score for improved assessments.
Collapse
Affiliation(s)
- Emma Croxford
- Department of Medicine, School of Medicine and Public Health, University of Wisconsin Madison
| | - Yanjun Gao
- Department of Medicine, School of Medicine and Public Health, University of Wisconsin Madison
| | - Brian Patterson
- Department of Emergency Medicine, School of Medicine and Public Health, University of Wisconsin Madison
| | - Daniel To
- Department of Medicine, School of Medicine and Public Health, University of Wisconsin Madison
| | - Samuel Tesch
- Department of Medicine, School of Medicine and Public Health, University of Wisconsin Madison
| | | | - Anoop Mayampurath
- Biostatistics and Medical Informatics, School of Medicine and Public Health, University of Wisconsin Madison
| | - Matthew M Churpek
- Department of Medicine, School of Medicine and Public Health, University of Wisconsin Madison
| | - Majid Afshar
- Department of Medicine, School of Medicine and Public Health, University of Wisconsin Madison
| |
Collapse
|
2
|
Blitzer GC, Paz C, Glassey A, Ganz OR, Giri J, Pennati A, Meyers RO, Bates AM, Nickel KP, Weiss M, Morris ZS, Mattison RJ, McDowell KA, Croxford E, Chappell RJ, Glazer TA, Rogus-Pulia NM, Galipeau J, Kimple RJ. Functionality of bone marrow mesenchymal stromal cells derived from head and neck cancer patients - A FDA-IND enabling study regarding MSC-based treatments for radiation-induced xerostomia. Radiother Oncol 2024; 192:110093. [PMID: 38224919 PMCID: PMC10922976 DOI: 10.1016/j.radonc.2024.110093] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2023] [Revised: 01/09/2024] [Accepted: 01/10/2024] [Indexed: 01/17/2024]
Abstract
PURPOSE Salivary dysfunction is a significant side effect of radiation therapy for head and neck cancer (HNC). Preliminary data suggests that mesenchymal stromal cells (MSCs) can improve salivary function. Whether MSCs from HNC patients who have completed chemoradiation are functionally similar to those from healthy patients is unknown. We performed a pilot clinical study to determine whether bone marrow-derived MSCs [MSC(M)] from HNC patients could be used for the treatment of RT-induced salivary dysfunction. METHODS An IRB-approved pilot clinical study was undertaken on HNC patients with xerostomia who had completed treatment two or more years prior. Patients underwent iliac crest bone marrow aspirate and MSC(M) were isolated and cultured. Culture-expanded MSC(M) were stimulated with IFNγ and cryopreserved prior to reanimation and profiling for functional markers by flow cytometry and ELISA. MSC(M) were additionally injected into mice with radiation-induced xerostomia and the changes in salivary gland histology and salivary production were examined. RESULTS A total of six subjects were enrolled. MSC(M) from all subjects were culture expanded to > 20 million cells in a median of 15.5 days (range 8-20 days). Flow cytometry confirmed that cultured cells from HNC patients were MSC(M). Functional flow cytometry demonstrated that these IFNγ-stimulated MSC(M) acquired an immunosuppressive phenotype. IFNγ-stimulated MSC(M) from HNC patients were found to express GDNF, WNT1, and R-spondin 1 as well as pro-angiogenesis and immunomodulatory cytokines. In mice, IFNγ-stimulated MSC(M) injection after radiation decreased the loss of acinar cells, decreased the formation of fibrosis, and increased salivary production. CONCLUSIONS MSC (M) from previously treated HNC patients can be expanded for auto-transplantation and are functionally active. Furthermore IFNγ-stimulated MSC(M) express proteins implicated in salivary gland regeneration. This study provides preliminary data supporting the feasibility of using autologous MSC(M) from HNC patients to treat RT-induced salivary dysfunction.
Collapse
Affiliation(s)
- Grace C Blitzer
- Department of Human Oncology, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Cristina Paz
- Department of Human Oncology, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Annemarie Glassey
- Department of Human Oncology, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Olga R Ganz
- Department of Medicine, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Jayeeta Giri
- Department of Medicine, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Andrea Pennati
- Department of Medicine, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA; UW Carbone Cancer Center, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Ross O Meyers
- Department of Human Oncology, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA; Department of Medicine, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Amber M Bates
- Department of Human Oncology, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Kwangok P Nickel
- Department of Human Oncology, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Marissa Weiss
- Department of Human Oncology, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Zachary S Morris
- Department of Human Oncology, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Ryan J Mattison
- Department of Medicine, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA; UW Carbone Cancer Center, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Kimberly A McDowell
- Department of Medicine, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Emma Croxford
- Department of Biostatistics and Medical Informatics, 610 Walnut Street, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53726 USA
| | - Richard J Chappell
- Department of Biostatistics and Medical Informatics, 610 Walnut Street, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53726 USA; UW Carbone Cancer Center, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Tiffany A Glazer
- Department of Surgery, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Nicole M Rogus-Pulia
- Department of Medicine, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA; UW Carbone Cancer Center, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA; Geriatric Research Education and Clinical Center, 2500 Overlook Terrace, William S. Middleton Memorial Veterans Hospital, Madison, WI 53705 USA
| | - Jacques Galipeau
- Department of Medicine, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA; UW Carbone Cancer Center, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA
| | - Randall J Kimple
- Department of Human Oncology, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA; UW Carbone Cancer Center, 600 Highland Ave, University of Wisconsin, School of Medicine and Public Health, Madison, WI 53705 USA.
| |
Collapse
|