1
|
Lam BD, Chrysafi P, Chiasakul T, Khosla H, Karagkouni D, McNichol M, Adamski A, Reyes N, Abe K, Mantha S, Vlachos IS, Zwicker JI, Patell R. Machine learning natural language processing for identifying venous thromboembolism: systematic review and meta-analysis. Blood Adv 2024; 8:2991-3000. [PMID: 38522096 PMCID: PMC11215191 DOI: 10.1182/bloodadvances.2023012200] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2023] [Revised: 02/22/2024] [Accepted: 02/22/2024] [Indexed: 03/26/2024] Open
Abstract
ABSTRACT Venous thromboembolism (VTE) is a leading cause of preventable in-hospital mortality. Monitoring VTE cases is limited by the challenges of manual medical record review and diagnosis code interpretation. Natural language processing (NLP) can automate the process. Rule-based NLP methods are effective but time consuming. Machine learning (ML)-NLP methods present a promising solution. We conducted a systematic review and meta-analysis of studies published before May 2023 that use ML-NLP to identify VTE diagnoses in the electronic health records. Four reviewers screened all manuscripts, excluding studies that only used a rule-based method. A meta-analysis evaluated the pooled performance of each study's best performing model that evaluated for pulmonary embolism and/or deep vein thrombosis. Pooled sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) with confidence interval (CI) were calculated by DerSimonian and Laird method using a random-effects model. Study quality was assessed using an adapted TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) tool. Thirteen studies were included in the systematic review and 8 had data available for meta-analysis. Pooled sensitivity was 0.931 (95% CI, 0.881-0.962), specificity 0.984 (95% CI, 0.967-0.992), PPV 0.910 (95% CI, 0.865-0.941) and NPV 0.985 (95% CI, 0.977-0.990). All studies met at least 13 of the 21 NLP-modified TRIPOD items, demonstrating fair quality. The highest performing models used vectorization rather than bag-of-words and deep-learning techniques such as convolutional neural networks. There was significant heterogeneity in the studies, and only 4 validated their model on an external data set. Further standardization of ML studies can help progress this novel technology toward real-world implementation.
Collapse
Affiliation(s)
- Barbara D. Lam
- Division of Hematology, Department of Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA
- Division of Clinical Informatics, Department of Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA
| | - Pavlina Chrysafi
- Department of Medicine, Mount Auburn Hospital, Harvard Medical School, Boston, MA
| | - Thita Chiasakul
- Center of Excellence in Translational Hematology, Division of Hematology, Department of Medicine, Faculty of Medicine, Chulalongkorn University and King Chulalongkorn Memorial Hospital, Thai Red Cross Society, Bangkok, Thailand
| | - Harshit Khosla
- Department of Medicine, Saint Vincent Hospital, Worcester, MA
| | - Dimitra Karagkouni
- Department of Pathology, Cancer Research Institute, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA
| | - Megan McNichol
- Library Sciences, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA
| | - Alys Adamski
- Division of Blood Disorders, National Center on Birth Defects and Developmental Disabilities, Centers for Disease Control and Prevention, Atlanta, GA
| | - Nimia Reyes
- Division of Blood Disorders, National Center on Birth Defects and Developmental Disabilities, Centers for Disease Control and Prevention, Atlanta, GA
| | - Karon Abe
- Division of Blood Disorders, National Center on Birth Defects and Developmental Disabilities, Centers for Disease Control and Prevention, Atlanta, GA
| | - Simon Mantha
- Division of Hematology, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY
| | - Ioannis S. Vlachos
- Department of Pathology, Cancer Research Institute, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA
| | - Jeffrey I. Zwicker
- Division of Hematology, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY
| | - Rushad Patell
- Division of Hematology, Department of Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA
| |
Collapse
|
2
|
Tsai AY, Carter SR, Greene AC. Artificial intelligence in pediatric surgery. Semin Pediatr Surg 2024; 33:151390. [PMID: 38242061 DOI: 10.1016/j.sempedsurg.2024.151390] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/21/2024]
Abstract
Artificial intelligence (AI) is rapidly changing the landscape of medicine and is already being utilized in conjunction with medical diagnostics and imaging analysis. We hereby explore AI applications in surgery and examine its relevance to pediatric surgery, covering its evolution, current state, and promising future. The various fields of AI are explored including machine learning and applications to predictive analytics and decision support in surgery, computer vision and image analysis in preoperative planning, image segmentation, surgical navigation, and finally, natural language processing assist in expediting clinical documentation, identification of clinical indications, quality improvement, outcome research, and other types of automated data extraction. The purpose of this review is to familiarize the pediatric surgical community with the rise of AI and highlight the ongoing advancements and challenges in its adoption, including data privacy, regulatory considerations, and the imperative for interdisciplinary collaboration. We hope this review serves as a comprehensive guide to AI's transformative influence on surgery, demonstrating its potential to enhance pediatric surgical patient outcomes, improve precision, and usher in a new era of surgical excellence.
Collapse
Affiliation(s)
- Anthony Y Tsai
- Division of Pediatric Surgery, Penn State Health Children's Hospital, 500 University Drive, Hershey, PA 17033, United States.
| | - Stewart R Carter
- Division of Pediatric Surgery, University of Louisville School of Medicine, Louisville, KY, United States
| | - Alicia C Greene
- Division of Pediatric Surgery, Penn State Health Children's Hospital, 500 University Drive, Hershey, PA 17033, United States
| |
Collapse
|
3
|
Bikdeli B, Lo YC, Khairani CD, Bejjani A, Jimenez D, Barco S, Mahajan S, Caraballo C, Secemsky EA, Klok FA, Hunsaker AR, Aghayev A, Muriel A, Wang Y, Hussain MA, Appah-Sampong A, Lu Y, Lin Z, Aneja S, Khera R, Goldhaber SZ, Zhou L, Monreal M, Krumholz HM, Piazza G. Developing Validated Tools to Identify Pulmonary Embolism in Electronic Databases: Rationale and Design of the PE-EHR+ Study. Thromb Haemost 2023; 123:649-662. [PMID: 36809777 PMCID: PMC11200175 DOI: 10.1055/a-2039-3222] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/23/2023]
Abstract
BACKGROUND Contemporary pulmonary embolism (PE) research, in many cases, relies on data from electronic health records (EHRs) and administrative databases that use International Classification of Diseases (ICD) codes. Natural language processing (NLP) tools can be used for automated chart review and patient identification. However, there remains uncertainty with the validity of ICD-10 codes or NLP algorithms for patient identification. METHODS The PE-EHR+ study has been designed to validate ICD-10 codes as Principal Discharge Diagnosis, or Secondary Discharge Diagnoses, as well as NLP tools set out in prior studies to identify patients with PE within EHRs. Manual chart review by two independent abstractors by predefined criteria will be the reference standard. Sensitivity, specificity, and positive and negative predictive values will be determined. We will assess the discriminatory function of code subgroups for intermediate- and high-risk PE. In addition, accuracy of NLP algorithms to identify PE from radiology reports will be assessed. RESULTS A total of 1,734 patients from the Mass General Brigham health system have been identified. These include 578 with ICD-10 Principal Discharge Diagnosis codes for PE, 578 with codes in the secondary position, and 578 without PE codes during the index hospitalization. Patients within each group were selected randomly from the entire pool of patients at the Mass General Brigham health system. A smaller subset of patients will also be identified from the Yale-New Haven Health System. Data validation and analyses will be forthcoming. CONCLUSIONS The PE-EHR+ study will help validate efficient tools for identification of patients with PE in EHRs, improving the reliability of efficient observational studies or randomized trials of patients with PE using electronic databases.
Collapse
Affiliation(s)
- Behnood Bikdeli
- Division of Cardiovascular Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
- Thrombosis Research Group, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
- Cardiovascular Research Foundation (CRF), New York, New York, United States
| | - Ying-Chih Lo
- Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Candrika D Khairani
- Thrombosis Research Group, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Antoine Bejjani
- Thrombosis Research Group, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - David Jimenez
- Respiratory Department, Hospital Ramón y Cajal and Medicine Department, Universidad de Alcalá (Instituto de Ramón y Cajal de Investigación Sanitaria), Centro de Investigación Biomédica en Red de Enfermedades Respiratorias, Madrid, Spain
| | - Stefano Barco
- Department of Angiology, University Hospital Zurich, Zurich, Switzerland
- Center for Thrombosis and Hemostasis, Johannes Gutenberg University Mainz, Mainz, Germany
| | - Shiwani Mahajan
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
- Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, United States
| | - César Caraballo
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
| | - Eric A Secemsky
- Richard A. and Susan F. Smith Center for Outcomes Research in Cardiology, Department of Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States
- Harvard Medical School, Boston, Massachusetts, United States
- Division of Cardiology, Department of Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States
| | - Frederikus A Klok
- Department of Medicine - Thrombosis and Hemostasis, Leiden University Medical Centre, Leiden, The Netherlands
| | - Andetta R Hunsaker
- Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Ayaz Aghayev
- Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Alfonso Muriel
- Clinical Biostatistics Unit. Hospital Universitario Ramón y Cajal. IRYCIS, CIBERESP: Universidad de Alcalá. Madrid, Spain
| | - Yun Wang
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
- Richard A. and Susan F. Smith Center for Outcomes Research in Cardiology, Department of Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States
| | - Mohamad A Hussain
- Division of Vascular and Endovascular Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
- Centre for Surgery and Public Health, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Abena Appah-Sampong
- Department of Surgery, Brigham and Women's Hospital, Boston, Massachusetts, United States
| | - Yuan Lu
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
| | - Zhenqiu Lin
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
| | - Sanjay Aneja
- Department of Therapeutic Radiology, Yale School of Medicine, New Haven, Connecticut, United States
| | - Rohan Khera
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
- Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, United States
| | - Samuel Z Goldhaber
- Division of Cardiovascular Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
- Thrombosis Research Group, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Li Zhou
- Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Manuel Monreal
- Cátedra de Enfermedad Tromboembólica, Universidad Católica de Murcia, Murcia, Spain
| | - Harlan M Krumholz
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
- Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, United States
- Department of Health Policy and Management, Yale School of Public Health, New Haven, Connecticut, United States
| | - Gregory Piazza
- Division of Cardiovascular Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
- Thrombosis Research Group, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| |
Collapse
|
4
|
Fernandes MB, Valizadeh N, Alabsi HS, Quadri SA, Tesh RA, Bucklin AA, Sun H, Jain A, Brenner LN, Ye E, Ge W, Collens SI, Lin S, Das S, Robbins GK, Zafar SF, Mukerji SS, Westover MB. Classification of neurologic outcomes from medical notes using natural language processing. EXPERT SYSTEMS WITH APPLICATIONS 2023; 214:119171. [PMID: 36865787 PMCID: PMC9974159 DOI: 10.1016/j.eswa.2022.119171] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
Neurologic disability level at hospital discharge is an important outcome in many clinical research studies. Outside of clinical trials, neurologic outcomes must typically be extracted by labor intensive manual review of clinical notes in the electronic health record (EHR). To overcome this challenge, we set out to develop a natural language processing (NLP) approach that automatically reads clinical notes to determine neurologic outcomes, to make it possible to conduct larger scale neurologic outcomes studies. We obtained 7314 notes from 3632 patients hospitalized at two large Boston hospitals between January 2012 and June 2020, including discharge summaries (3485), occupational therapy (1472) and physical therapy (2357) notes. Fourteen clinical experts reviewed notes to assign scores on the Glasgow Outcome Scale (GOS) with 4 classes, namely 'good recovery', 'moderate disability', 'severe disability', and 'death' and on the Modified Rankin Scale (mRS), with 7 classes, namely 'no symptoms', 'no significant disability', 'slight disability', 'moderate disability', 'moderately severe disability', 'severe disability', and 'death'. For 428 patients' notes, 2 experts scored the cases generating interrater reliability estimates for GOS and mRS. After preprocessing and extracting features from the notes, we trained a multiclass logistic regression model using LASSO regularization and 5-fold cross validation for hyperparameter tuning. The model performed well on the test set, achieving a micro average area under the receiver operating characteristic and F-score of 0.94 (95% CI 0.93-0.95) and 0.77 (0.75-0.80) for GOS, and 0.90 (0.89-0.91) and 0.59 (0.57-0.62) for mRS, respectively. Our work demonstrates that an NLP algorithm can accurately assign neurologic outcomes based on free text clinical notes. This algorithm increases the scale of research on neurological outcomes that is possible with EHR data.
Collapse
Affiliation(s)
- Marta B. Fernandes
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
- Clinical Data Animation Center (CDAC), MGH, Boston, MA, United States
| | - Navid Valizadeh
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
| | - Haitham S. Alabsi
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
| | - Syed A. Quadri
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
- Clinical Data Animation Center (CDAC), MGH, Boston, MA, United States
| | - Ryan A. Tesh
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
- Clinical Data Animation Center (CDAC), MGH, Boston, MA, United States
| | - Abigail A. Bucklin
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
- Clinical Data Animation Center (CDAC), MGH, Boston, MA, United States
| | - Haoqi Sun
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
- Clinical Data Animation Center (CDAC), MGH, Boston, MA, United States
| | - Aayushee Jain
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Clinical Data Animation Center (CDAC), MGH, Boston, MA, United States
| | - Laura N. Brenner
- Harvard Medical School, Boston, MA, United States
- Division of Pulmonary and Critical Care Medicine, MGH, Boston, MA, United States
- Division of General Internal Medicine, MGH, Boston, MA, United States
| | - Elissa Ye
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Clinical Data Animation Center (CDAC), MGH, Boston, MA, United States
| | - Wendong Ge
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
- Clinical Data Animation Center (CDAC), MGH, Boston, MA, United States
| | - Sarah I. Collens
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
| | - Stacie Lin
- Harvard Medical School, Boston, MA, United States
| | - Sudeshna Das
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
| | - Gregory K. Robbins
- Harvard Medical School, Boston, MA, United States
- Division of Infectious Diseases, MGH, Boston, MA, United States
| | - Sahar F. Zafar
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
| | - Shibani S. Mukerji
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
- Vaccine and Immunotherapy Center, Division of Infectious Diseases, MGH, Boston, MA, United States
| | - M. Brandon Westover
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
- Clinical Data Animation Center (CDAC), MGH, Boston, MA, United States
- McCance Center for Brain Health, MGH, Boston, MA, United States
| |
Collapse
|
5
|
Kobritz M, Patel V, Rindskopf D, Demyan L, Jarrett M, Coppa G, Antonacci AC. Practice-Based Learning and Improvement: Improving Morbidity and Mortality Review Using Natural Language Processing. J Surg Res 2023; 283:351-356. [PMID: 36427445 DOI: 10.1016/j.jss.2022.10.075] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2022] [Revised: 09/21/2022] [Accepted: 10/18/2022] [Indexed: 11/23/2022]
Abstract
INTRODUCTION Practice-Based Learning and Improvement, a core competency identified by the Accreditation Council for Graduate Medical Education, carries importance throughout a physician's career. Practice-Based Learning and Improvement is cultivated by a critical review of complications, yet methods to accurately identify complications are inadequate. Machine-learning algorithms show promise in improving identification of complications. We compare a manual-supplemented natural language processing (ms-NLP) methodology against a validated electronic morbidity and mortality (MM) database, the Morbidity and Mortality Adverse Event Reporting System (MARS) to understand the utility of NLP in MM review. METHODS The number and severity of complications were compared between MARS and ms-NLP of surgical hospitalization discharge summaries among three academic medical centers. Clavien-Dindo (CD) scores were assigned to cases with identified complications and classified into minor (CD I-II) or major (CD III-IV) harm. RESULTS Of 7774 admissions, 987 cases were identified to have 1659 complications by MARS and 1296 by ms-NLP. MARS identified 611 (62%) cases, whereas ms-NLP identified 670 (68%) cases. Less than one-third of cases (299, 30.3%) were detected by both methods. MARS identified a greater number of complications with major harm (457, 46.30%) than did ms-NLP (P < 0.0001). CONCLUSIONS Both a prospectively maintained MM database and ms-NLP review of discharge summaries fail to identify a significant proportion of postoperative complications and overlap 1/3 of the time. ms-NLP more frequently identifies cases with minor complications, whereas prospective voluntary reporting more frequently identifies major complications. The educational benefit of reporting and analysis of complication data may be supplemented by ms-NLP but not replaced by it at this time.
Collapse
Affiliation(s)
- Molly Kobritz
- Northwell Health North Shore/Long Island Jewish General Surgery, Manhasset, New York; Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York.
| | - Vihas Patel
- Northwell Health North Shore/Long Island Jewish General Surgery, Manhasset, New York; Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York
| | - David Rindskopf
- City University of New York, Graduate School And University Center, New York, New York
| | - Lyudmyla Demyan
- Northwell Health North Shore/Long Island Jewish General Surgery, Manhasset, New York; Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York
| | - Mark Jarrett
- Northwell Health North Shore/Long Island Jewish General Surgery, Manhasset, New York; Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York
| | - Gene Coppa
- Northwell Health North Shore/Long Island Jewish General Surgery, Manhasset, New York; Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York
| | - Anthony C Antonacci
- Northwell Health North Shore/Long Island Jewish General Surgery, Manhasset, New York; Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York
| |
Collapse
|
6
|
Colborn KL, Zhuang Y, Dyas AR, Henderson WG, Madsen HJ, Bronsert MR, Matheny ME, Lambert-Kerzner A, Myers QWO, Meguid RA. Development and validation of models for detection of postoperative infections using structured electronic health records data and machine learning. Surgery 2023; 173:464-471. [PMID: 36470694 PMCID: PMC10204069 DOI: 10.1016/j.surg.2022.10.026] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2022] [Revised: 10/18/2022] [Accepted: 10/26/2022] [Indexed: 12/04/2022]
Abstract
BACKGROUND Postoperative infections constitute more than half of all postoperative complications. Surveillance of these complications is primarily done through manual chart review, which is time consuming, expensive, and typically only covers 10% to 15% of all operations. Automated surveillance would permit the timely evaluation of and reporting of all operations. METHODS The goal of this study was to develop and validate parsimonious, interpretable models for conducting surveillance of postoperative infections using structured electronic health records data. This was a retrospective study using 30,639 unique operations from 5 major hospitals between 2013 and 2019. Structured electronic health records data were linked to postoperative outcomes data from the American College of Surgeons National Surgical Quality Improvement Program. Predictors from the electronic health records included diagnoses, procedures, and medications. Infectious complications included surgical site infection, urinary tract infection, sepsis, and pneumonia within 30 days of surgery. The knockoff filter, a penalized regression technique that controls type I error, was applied for variable selection. Models were validated in a chronological held-out dataset. RESULTS Seven percent of patients experienced at least one type of postoperative infection. Models selected contained between 4 and 8 variables and achieved >0.91 area under the receiver operating characteristic curve, >81% specificity, >87% sensitivity, >99% negative predictive value, and 10% to 15% positive predictive value in a held-out test dataset. CONCLUSION Surveillance and reporting of postoperative infection rates can be implemented for all operations with high accuracy using electronic health records data and simple linear regression models.
Collapse
Affiliation(s)
- Kathryn L Colborn
- Department of Surgery, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO; Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO; Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO; Adult and Child Consortium for Health Outcomes Research and Delivery Science, University of Colorado Anschutz Medical Campus, Aurora, CO.
| | - Yaxu Zhuang
- Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO
| | - Adam R Dyas
- Department of Surgery, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO; Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - William G Henderson
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO; Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO
| | - Helen J Madsen
- Department of Surgery, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO; Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - Michael R Bronsert
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO; Adult and Child Consortium for Health Outcomes Research and Delivery Science, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - Michael E Matheny
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN; Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN; Division of General Internal Medicine, Vanderbilt University Medical Center, Nashville, TN
| | - Anne Lambert-Kerzner
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - Quintin W O Myers
- Department of Surgery, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO; Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - Robert A Meguid
- Department of Surgery, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO; Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, CO; Adult and Child Consortium for Health Outcomes Research and Delivery Science, University of Colorado Anschutz Medical Campus, Aurora, CO
| |
Collapse
|
7
|
Schumm M, Hu MY, Sant V, Kim J, Tseng CH, Sanz J, Raman S, Yu R, Livhits M. Automated extraction of incidental adrenal nodules from electronic health records. Surgery 2023; 173:52-58. [PMID: 36207197 DOI: 10.1016/j.surg.2022.07.028] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2022] [Revised: 05/25/2022] [Accepted: 07/19/2022] [Indexed: 12/13/2022]
Abstract
BACKGROUND Many adrenal incidentalomas do not undergo appropriate biochemical testing and complete imaging characterization to assess for hormone hypersecretion and malignancy. With the growing availability of clinical narratives in the electronic medical record, automated surveillance using advanced data analytic techniques may represent a promising method to improve management. METHODS A data provisioning process using a series of structured query language scripts was used to abstract all chest and abdominal computed tomography and magnetic resonance imaging reports from an academic health care system in 2018. The narratives and impressions were queried for key text relating to the identification of adrenal incidentalomas. Patients with a history of extra-adrenal malignancy undergoing staging or surveillance imaging were excluded. The prevalence and radiographic characteristics were analyzed. Patients with adrenal incidentalomas newly identified in 2018 were assessed for biochemical testing and nodule stability through August 2021. RESULTS Of 36,618 patients queried, 8,557 were excluded owing to a history of extra-adrenal malignancy. Data from 447 patients were flagged by the structured query language scripts and electronically abstracted. On internal validation, 307/447 (69%) patients were correctly identified as having adrenal nodules (1.1% overall prevalence). The median patient age was 67 years, and 56% were female. The median nodule size was 1.7 (IQR 1.3-2.5) cm, 9% were bilateral, and 63% were low density (unenhanced Hounsfield units <10). Adrenal carcinoma was identified in 10 patients. In 2018, 121 patients were diagnosed with a new adrenal incidentaloma. Of 32 (27%) patients who had follow-up imaging at a median of 1.9 years, 97% of nodules were stable in size. Biochemical testing was performed in 53 patients (44%), of which 31 (26%) had complete hormonal assessment; 14 (26%) were functional nodules: 7 aldosterone-secreting, 4 cortisol-secreting, and 3 pheochromocytoma. CONCLUSION Only one-fourth of patients received appropriate biochemical testing after incidental diagnosis of an adrenal nodule, and most nodules with indeterminate imaging characteristics did not undergo follow-up imaging. Advanced data analytic techniques on electronic imaging reports may aid in the clinical identification and improved management of patients with adrenal incidentalomas.
Collapse
Affiliation(s)
- Max Schumm
- Section of Endocrine Surgery, Department of Surgery, University of California-Los Angeles (UCLA) David Geffen School of Medicine, CA.
| | - Ming-Yeah Hu
- Section of Endocrine Surgery, Department of Surgery, University of California-Los Angeles (UCLA) David Geffen School of Medicine, CA. https://twitter.com/MingYeahHu
| | - Vivek Sant
- Section of Endocrine Surgery, Department of Surgery, University of California-Los Angeles (UCLA) David Geffen School of Medicine, CA. https://twitter.com/VivekSantMD
| | - Jiyoon Kim
- Department of Biostatistics, UCLA Fielding School of Public Health, Los Angeles, CA
| | - Chi-Hong Tseng
- Division of General Internal Medicine and Health Services Research, Department of Medicine, UCLA David Geffen School of Medicine, CA
| | - Javier Sanz
- Department of Medicine, Clinical and Translational Science Institute, UCLA David Geffen School of Medicine, Los Angeles, CA
| | - Steven Raman
- Department of Interventional and Diagnostic Radiology, UCLA David Geffen School of Medicine, Los Angeles, CA. https://twitter.com/StevenSRaman_MD
| | - Run Yu
- Division of Endocrinology, Diabetes, and Metabolism; Department of Medicine, UCLA David Geffen School of Medicine, Los Angeles, CA
| | - Masha Livhits
- Section of Endocrine Surgery, Department of Surgery, University of California-Los Angeles (UCLA) David Geffen School of Medicine, CA. https://twitter.com/mashalivhitsMD
| |
Collapse
|
8
|
Dyas AR, Zhuang Y, Meguid RA, Henderson WG, Madsen HJ, Bronsert MR, Colborn KL. Development and validation of a model for surveillance of postoperative bleeding complications using structured electronic health records data. Surgery 2022; 172:1728-1732. [PMID: 36150923 PMCID: PMC10204070 DOI: 10.1016/j.surg.2022.08.021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2022] [Revised: 07/01/2022] [Accepted: 08/22/2022] [Indexed: 01/28/2023]
Abstract
BACKGROUND Postoperative bleeding complications surveillance is done primarily through manual chart review. The purpose of this study was to develop and validate a detection model for postoperative bleeding complications using structured electronic health records data. METHODS Patients who underwent operations at 1 of 5 hospitals within our local health system between 2013 and 2019 and whose complications were reported by the American College of Surgeons National Surgical Quality Improvement Program were included. Electronic health records data were linked to American College of Surgeons National Surgical Quality Improvement Program data using personal health identifiers. Electronic health records predictors included diagnosis codes mapped to PheCodes, procedure names, and medications within 30 days after surgery. We defined bleeding events as the transfusion of red blood cell components within 30 days after surgery. The knockoff filter and the lasso were used to develop a model in a training set of operations from January 2013 to March 2017. Performance of each model was tested in a held-out data set of patients who underwent operations from March 2017 to October 2019. RESULTS A total of 30,639 patients were included; 1,112 patients (3.6%) had a bleeding event. Eight predictor variables were selected by the knockoff filter. When applied to the test set, specificity was 94%, sensitivity was 94%, area under the curve was 0.97, and accuracy was 93%. Calibration was consistent in lower predicted risk patients, whereas the model slightly overpredicted risk in high-risk patients. CONCLUSION We created a parsimonious, accurate model for identifying patients with bleeding complications. This model can be used to augment manual chart review for surveillance and reporting of perioperative bleeding complications, enabling inclusion of all surgeries in quality improvement efforts.
Collapse
Affiliation(s)
- Adam R Dyas
- Department of Surgery, University of Colorado School of Medicine, Aurora, CO; Surgical Outcomes and Applied Research Program, University of Colorado School of Medicine, Aurora, CO.
| | - Yaxu Zhuang
- Department of Surgery, University of Colorado School of Medicine, Aurora, CO; Surgical Outcomes and Applied Research Program, University of Colorado School of Medicine, Aurora, CO; Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO
| | - Robert A Meguid
- Department of Surgery, University of Colorado School of Medicine, Aurora, CO; Surgical Outcomes and Applied Research Program, University of Colorado School of Medicine, Aurora, CO; Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora, CO
| | - William G Henderson
- Surgical Outcomes and Applied Research Program, University of Colorado School of Medicine, Aurora, CO; Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora, CO; Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO
| | - Helen J Madsen
- Department of Surgery, University of Colorado School of Medicine, Aurora, CO; Surgical Outcomes and Applied Research Program, University of Colorado School of Medicine, Aurora, CO
| | - Michael R Bronsert
- Surgical Outcomes and Applied Research Program, University of Colorado School of Medicine, Aurora, CO; Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora, CO
| | - Kathryn L Colborn
- Department of Surgery, University of Colorado School of Medicine, Aurora, CO; Surgical Outcomes and Applied Research Program, University of Colorado School of Medicine, Aurora, CO; Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO
| |
Collapse
|
9
|
Jin ZG, Zhang H, Tai MH, Yang Y, Yao Y, Guo YT. Natural Language Processing for Identification of Venous Thromboembolism in a Clinical Decision Support System: Validation Study (Preprint). J Med Internet Res 2022; 25:e43153. [PMID: 37093636 PMCID: PMC10167583 DOI: 10.2196/43153] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2022] [Revised: 11/20/2022] [Accepted: 03/29/2023] [Indexed: 03/31/2023] Open
Abstract
BACKGROUND It remains unknown whether capturing data from electronic health records (EHRs) using natural language processing (NLP) can improve venous thromboembolism (VTE) detection in different clinical settings. OBJECTIVE The aim of this study was to validate the NLP algorithm in a clinical decision support system for VTE risk assessment and integrated care (DeVTEcare) to identify VTEs from EHRs. METHODS All inpatients aged ≥18 years in the Sixth Medical Center of the Chinese People's Liberation Army General Hospital from January 1 to December 31, 2021, were included as the validation cohort. The sensitivity, specificity, positive and negative likelihood ratios (LR+ and LR-, respectively), area under the receiver operating characteristic curve (AUC), and F1-scores along with their 95% CIs were used to analyze the performance of the NLP tool, with manual review of medical records as the reference standard for detecting deep vein thrombosis (DVT) and pulmonary embolism (PE). The primary end point was the performance of the NLP approach embedded into the EHR for VTE identification. The secondary end points were the performances to identify VTE among different hospital departments with different VTE risks. Subgroup analyses were performed among age, sex, and the study season. RESULTS Among 30,152 patients (median age 56 [IQR 41-67] years; 14,247/30,152, 47.3% females), the prevalence of VTE, PE, and DVT was 2.1% (626/30,152), 0.6% (177/30,152), and 1.8% (532/30,152), respectively. The sensitivity, specificity, LR+, LR-, AUC, and F1-score of NLP-facilitated VTE detection were 89.9% (95% CI 87.3%-92.2%), 99.8% (95% CI 99.8%-99.9%), 483 (95% CI 370-629), 0.10 (95% CI 0.08-0.13), 0.95 (95% CI 0.94-0.96), and 0.90 (95% CI 0.90-0.91), respectively. Among departments of surgery, internal medicine, and intensive care units, the highest specificity (100% vs 99.7% vs 98.8%, respectively), LR+ (3202 vs 321 vs 77, respectively), and F1-score (0.95 vs 0.89 vs 0.92, respectively) were in the surgery department (all P<.001). Among low, intermediate, and high VTE risks in hospital departments, the low-risk department had the highest AUC (1.00 vs 0.94 vs 0.96, respectively) and F1-score (0.97 vs 0.90 vs 0.90, respectively) as well as the lowest LR- (0.00 vs 0.13 vs 0.08, respectively) (DeLong test for AUC; all P<.001). Subgroup analysis of the age, sex, and season demonstrated consistently good performance of VTE detection with >87% sensitivity and specificity and >89% AUC and F1-score. The NLP algorithm performed better among patients aged ≤65 years than among those aged >65 years (F1-score 0.93 vs 0.89, respectively; P<.001). CONCLUSIONS The NLP algorithm in our DeVTEcare identified VTE well across different clinical settings, especially in patients in surgery units, departments with low-risk VTE, and patients aged ≤65 years. This algorithm can help to inform accurate in-hospital VTE rates and enhance risk-classified VTE integrated care in future research.
Collapse
Affiliation(s)
- Zhi-Geng Jin
- Department of Pulmonary Vascular and Thrombotic Disease, Sixth Medical Center of Chinese People's Liberation Army General Hospital, Beijing, China
| | - Hui Zhang
- Department of Pulmonary Vascular and Thrombotic Disease, Sixth Medical Center of Chinese People's Liberation Army General Hospital, Beijing, China
| | - Mei-Hui Tai
- Chinese People's Liberation Army Medical School, Beijing, China
| | - Ying Yang
- Quality Management Division, Sixth Medical Center of Chinese People's Liberation Army General Hospital, Beijing, China
| | - Yuan Yao
- Institute for Hospital Management Research, Chinese People's Liberation Army General Hospital, Beijing, China
| | - Yu-Tao Guo
- Department of Pulmonary Vascular and Thrombotic Disease, Sixth Medical Center of Chinese People's Liberation Army General Hospital, Beijing, China
| |
Collapse
|
10
|
Saber I, Adamski A, Kuchibhatla M, Abe K, Beckman M, Reyes N, Schulteis R, Pendurthi Singh B, Sitlinger A, Thames EH, Ortel TL. Racial differences in venous thromboembolism: A surveillance program in Durham County, North Carolina. Res Pract Thromb Haemost 2022; 6:e12769. [PMID: 35873215 PMCID: PMC9301530 DOI: 10.1002/rth2.12769] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2021] [Revised: 06/15/2022] [Accepted: 06/24/2022] [Indexed: 11/06/2022] Open
Abstract
Background Venous thromboembolism (VTE) affects approximately 1-2 individuals per 1000 annually and is associated with an increased risk for pulmonary hypertension, postthrombotic syndrome, and recurrent VTE. Objective To determine risk factors, incidence, treatments, and outcomes of VTE through a 2-year surveillance program initiated in Durham County, North Carolina (population approximately 280,000 at time of study). Patients/Methods We performed a retrospective analysis of data actively collected from three hospitals in Durham County during the surveillance period. Results A total of 987 patients were diagnosed with VTE, for an annual rate of 1.76 per 1000 individuals. Hospital-associated VTE occurred in 167 hospitalized patients (16.9%) and 271 outpatients who were hospitalized within 90 days of diagnosis (27.5%). Annual incidence was 1.98 per 1000 Black individuals compared to 1.25 per 1000 White individuals (p < 0.0001), and Black individuals with VTE were younger than White individuals (p < 0.0001). Common risk factors included active cancer, prolonged immobility, and obesity, and approximately half were still taking anticoagulant therapy 1 year later. A total of 224 patients died by 1 year (28.5% of patients for whom outcomes could be confirmed), and Black patients were more likely to have recurrent VTE than White patients during the first 6 months following initial presentation (9.4% vs. 4.1%, p = 0.01). Conclusions Ongoing surveillance provides an effective strategy to identify patients with VTE and monitor treatment and outcomes. We demonstrated that hospital-associated VTE continues to be a major contributor to the burden of VTE and confirmed the higher incidence of VTE in Black compared to White individuals.
Collapse
Affiliation(s)
- Ibrahim Saber
- Division of Hematology, Department of Medicine Duke University Medical Center Durham North Carolina USA
| | - Alys Adamski
- Centers for Disease Control and Prevention Atlanta Georgia USA
| | - Maragatha Kuchibhatla
- Department of Biostatistics and Bioinformatics Duke University Durham North Carolina USA
| | - Karon Abe
- Centers for Disease Control and Prevention Atlanta Georgia USA
| | - Michele Beckman
- Centers for Disease Control and Prevention Atlanta Georgia USA
| | - Nimia Reyes
- Centers for Disease Control and Prevention Atlanta Georgia USA
| | - Ryan Schulteis
- Durham Veterans' Administration Medical Center Durham North Carolina USA
| | | | - Andrea Sitlinger
- Division of Hematologic Malignancies and Cellular Therapy, Department of Medicine Duke University Medical Center Durham North Carolina USA
| | - Elizabeth H Thames
- Division of Hematology, Department of Medicine Duke University Medical Center Durham North Carolina USA
| | - Thomas L Ortel
- Division of Hematology, Department of Medicine Duke University Medical Center Durham North Carolina USA.,Department of Pathology Duke University Medical Center Durham North Carolina USA
| |
Collapse
|
11
|
Xiong Y, Zhang Y, Zhang F, Wu C, Qin F, Yuan J. Applications of artificial intelligence in the diagnosis and prediction of erectile dysfunction: a narrative review. Int J Impot Res 2022; 35:95-102. [PMID: 35027721 DOI: 10.1038/s41443-022-00528-w] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2021] [Revised: 12/24/2021] [Accepted: 01/06/2022] [Indexed: 02/05/2023]
Abstract
Despite the high prevalence of erectile dysfunction, patients are reluctant to seek medical advice, which leads to low diagnostic rates in clinical practice. Artificial intelligence has been widely applied in the diagnosis of many diseases and may alleviate the situation. However, the applications of artificial intelligence in erectile dysfunction have not been reviewed to date. Therefore, the assistance from artificial intelligence needs to be summarized. In this review, 418 publications before January 10, 2021, regarding artificial intelligence applications in diagnosing and predicting erectile dysfunction, were retrieved from five databases, including PubMed, EMBASE, the Cochrane Library, and two Chinese databases (WANFANG and CNKI). In addition, the reference lists of the included studies or relevant reviews were checked to avoid bias. Finally, 30 articles were reviewed to summarize the current status, merits, and limitations of applying artificial intelligence in diagnosing and predicting erectile dysfunction. The results showed that artificial intelligence contributed to developing novel diagnostic questionnaires, equipment, expert systems, classifiers by images and predictive models. However, most of the included studies were not subjected to external validations, resulting in doubt on the generalizability. In the future, more rigorously designed studies with high-quality datasets for erectile dysfunction are required.
Collapse
Affiliation(s)
- Yang Xiong
- Andrology Laboratory, West China Hospital, Sichuan University, Chengdu, China.,Department of Urology, West China Hospital, Sichuan University, Chengdu, China
| | - Yangchang Zhang
- Department of Epidemiology and Health Statistics, School of Public Health and Management, Chongqing Medical University, Chongqing, China
| | - Fuxun Zhang
- Andrology Laboratory, West China Hospital, Sichuan University, Chengdu, China.,Department of Urology, West China Hospital, Sichuan University, Chengdu, China
| | - Changjing Wu
- Andrology Laboratory, West China Hospital, Sichuan University, Chengdu, China
| | - Feng Qin
- Andrology Laboratory, West China Hospital, Sichuan University, Chengdu, China
| | - Jiuhong Yuan
- Andrology Laboratory, West China Hospital, Sichuan University, Chengdu, China. .,Department of Urology, West China Hospital, Sichuan University, Chengdu, China.
| |
Collapse
|
12
|
Wang Q, Yuan L, Ding X, Zhou Z. Prediction and Diagnosis of Venous Thromboembolism Using Artificial Intelligence Approaches: A Systematic Review and Meta-Analysis. Clin Appl Thromb Hemost 2021; 27:10760296211021162. [PMID: 34184560 PMCID: PMC8246532 DOI: 10.1177/10760296211021162] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022] Open
Abstract
Venous thromboembolism (VTE) is a fatal disease and has become a burden on the global health system. Recent studies have suggested that artificial intelligence (AI) could be used to make a diagnosis and predict venous thrombosis more accurately. Thus, we performed a meta-analysis to better evaluate the performance of AI in the prediction and diagnosis of venous thrombosis. PubMed, Web of Science, and EMBASE were used to identify relevant studies. Of the 741 studies, 12 met the inclusion criteria and were included in the meta-analysis. Among them, 5 studies included a training set and test set, and 7 studies included only a training set. In the training set, the pooled sensitivity was 0.87 (95% CI 0.79-0.92), the pooled specificity was 0.95 (95% CI 0.89-0.97), and the area under the summary receiver operating characteristic (SROC) curve was 0.97 (95% CI 0.95-0.98). In the test set, the pooled sensitivity was 0.87 (95% CI 0.74-0.93), the pooled specificity was 0.96 (95% CI 0.79-0.99), and the area under the SROC curve was 0.98 (95% CI 0.97-0.99). The combined results remained significant in the subgroup analyzes, which included venous thrombosis type, AI type, model type (diagnosis/prediction), and whether the period was perioperative. In conclusion, AI may aid in the diagnosis and prediction of venous thrombosis, demonstrating high sensitivity, specificity and area under the SROC curve values. Thus, AI has important clinical value.
Collapse
Affiliation(s)
- Qi Wang
- Department of Neurology, The First Affiliated Hospital of Wannan Medical College, Yijishan Hospital, Wuhu, China
| | - Lili Yuan
- Department of Neurology, The First Affiliated Hospital of Wannan Medical College, Yijishan Hospital, Wuhu, China
| | - Xianhui Ding
- Department of Neurology, The First Affiliated Hospital of Wannan Medical College, Yijishan Hospital, Wuhu, China
| | - Zhiming Zhou
- Department of Neurology, The First Affiliated Hospital of Wannan Medical College, Yijishan Hospital, Wuhu, China
| |
Collapse
|
13
|
Natural language processing for the surveillance of postoperative venous thromboembolism. Surgery 2021; 170:1175-1182. [PMID: 34090671 DOI: 10.1016/j.surg.2021.04.027] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2021] [Revised: 04/07/2021] [Accepted: 04/20/2021] [Indexed: 11/20/2022]
Abstract
BACKGROUND The objective of the study was to develop a portal natural language processing approach to aid in the identification of postoperative venous thromboembolism events from free-text clinical notes. METHODS We abstracted clinical notes from 25,494 operative events from 2 independent health care systems. A venous thromboembolism detected as part of the American College of Surgeons National Surgical Quality Improvement Program was used as the reference standard. A natural language processing engine, easy clinical information extractor-pulmonary embolism/deep vein thrombosis (EasyCIE-PEDVT), was trained to detect pulmonary embolism and deep vein thrombosis from clinical notes. International Classification of Diseases discharge diagnosis codes for venous thromboembolism were used as baseline comparators. The classification performance of EasyCIE-PEDVT was compared with International Classification of Diseases codes using sensitivity, specificity, area under the receiver operating characteristic curve, using an internal and external validation cohort. RESULTS To detect pulmonary embolism, EasyCIE-PEDVT had a sensitivity of 0.714 and 0.815 in internal and external validation, respectively. To detect deep vein thrombosis, EasyCIE-PEDVT had a sensitivity of 0.846 and 0.849 in internal and external validation, respectively. EasyCIE-PEDVT had significantly higher discrimination for deep vein thrombosis compared with International Classification of Diseases codes in internal validation (area under the receiver operating characteristic curve: 0.920 vs 0.761; P < .001) and external validation (area under the receiver operating characteristic curve: 0.921 vs 0.794; P < .001). There was no significant difference in the discrimination for pulmonary embolism between EasyCIE-PEDVT and International Classification of Diseases codes. CONCLUSION Accurate surveillance of postoperative venous thromboembolism may be achieved using natural language processing on clinical notes in 2 independent health care systems. These findings suggest natural language processing may augment manual chart abstraction for large registries such as National Surgical Quality Improvement Program.
Collapse
|
14
|
Abstract
OBJECTIVE The aim of this study was to systematically assess the application and potential benefits of natural language processing (NLP) in surgical outcomes research. SUMMARY BACKGROUND DATA Widespread implementation of electronic health records (EHRs) has generated a massive patient data source. Traditional methods of data capture, such as billing codes and/or manual review of free-text narratives in EHRs, are highly labor-intensive, costly, subjective, and potentially prone to bias. METHODS A literature search of PubMed, MEDLINE, Web of Science, and Embase identified all articles published starting in 2000 that used NLP models to assess perioperative surgical outcomes. Evaluation metrics of NLP systems were assessed by means of pooled analysis and meta-analysis. Qualitative synthesis was carried out to assess the results and risk of bias on outcomes. RESULTS The present study included 29 articles, with over half (n = 15) published after 2018. The most common outcome identified using NLP was postoperative complications (n = 14). Compared to traditional non-NLP models, NLP models identified postoperative complications with higher sensitivity [0.92 (0.87-0.95) vs 0.58 (0.33-0.79), P < 0.001]. The specificities were comparable at 0.99 (0.96-1.00) and 0.98 (0.95-0.99), respectively. Using summary of likelihood ratio matrices, traditional non-NLP models have clinical utility for confirming documentation of outcomes/diagnoses, whereas NLP models may be reliably utilized for both confirming and ruling out documentation of outcomes/diagnoses. CONCLUSIONS NLP usage to extract a range of surgical outcomes, particularly postoperative complications, is accelerating across disciplines and areas of clinical outcomes research. NLP and traditional non-NLP approaches demonstrate similar performance measures, but NLP is superior in ruling out documentation of surgical outcomes.
Collapse
|
15
|
Deng Z, Yin K, Bao Y, Armengol VD, Wang C, Tiwari A, Barzilay R, Parmigiani G, Braun D, Hughes KS. Validation of a Semiautomated Natural Language Processing-Based Procedure for Meta-Analysis of Cancer Susceptibility Gene Penetrance. JCO Clin Cancer Inform 2020; 3:1-9. [PMID: 31419182 DOI: 10.1200/cci.19.00043] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022] Open
Abstract
PURPOSE Quantifying the risk of cancer associated with pathogenic mutations in germline cancer susceptibility genes-that is, penetrance-enables the personalization of preventive management strategies. Conducting a meta-analysis is the best way to obtain robust risk estimates. We have previously developed a natural language processing (NLP) -based abstract classifier which classifies abstracts as relevant to penetrance, prevalence of mutations, both, or neither. In this work, we evaluate the performance of this NLP-based procedure. MATERIALS AND METHODS We compared the semiautomated NLP-based procedure, which involves automated abstract classification and text mining, followed by human review of identified studies, with the traditional procedure that requires human review of all studies. Ten high-quality gene-cancer penetrance meta-analyses spanning 16 gene-cancer associations were used as the gold standard by which to evaluate the performance of our procedure. For each meta-analysis, we evaluated the number of abstracts that required human review (workload) and the ability to identify the studies that were included by the authors in their quantitative analysis (coverage). RESULTS Compared with the traditional procedure, the semiautomated NLP-based procedure led to a lower workload across all 10 meta-analyses, with an overall 84% reduction (2,774 abstracts v 16,941 abstracts) in the amount of human review required. Overall coverage was 93%-we are able to identify 132 of 142 studies-before reviewing references of identified studies. Reasons for the 10 missed studies included blank and poorly written abstracts. After reviewing references, nine of the previously missed studies were identified and coverage improved to 99% (141 of 142 studies). CONCLUSION We demonstrated that an NLP-based procedure can significantly reduce the review workload without compromising the ability to identify relevant studies. NLP algorithms have promising potential for reducing human efforts in the literature review process.
Collapse
Affiliation(s)
| | - Kanhua Yin
- Massachusetts General Hospital, Boston, MA
| | - Yujia Bao
- Massachusetts Institute of Technology, Boston, MA
| | | | - Cathy Wang
- Harvard TH Chan School of Public Health, Boston, MA.,Dana-Farber Cancer Institute, Boston, MA
| | | | | | - Giovanni Parmigiani
- Harvard TH Chan School of Public Health, Boston, MA.,Dana-Farber Cancer Institute, Boston, MA
| | - Danielle Braun
- Harvard TH Chan School of Public Health, Boston, MA.,Dana-Farber Cancer Institute, Boston, MA
| | - Kevin S Hughes
- Massachusetts General Hospital, Boston, MA.,Harvard Medical School, Boston, MA
| |
Collapse
|