1
|
Bosschieter TM, Xu Z, Lan H, Lengerich BJ, Nori H, Painter I, Souter V, Caruana R. Interpretable Predictive Models to Understand Risk Factors for Maternal and Fetal Outcomes. JOURNAL OF HEALTHCARE INFORMATICS RESEARCH 2024; 8:65-87. [PMID: 38273984 PMCID: PMC10805688 DOI: 10.1007/s41666-023-00151-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2023] [Revised: 09/18/2023] [Accepted: 09/19/2023] [Indexed: 01/27/2024]
Abstract
Although most pregnancies result in a good outcome, complications are not uncommon and can be associated with serious implications for mothers and babies. Predictive modeling has the potential to improve outcomes through a better understanding of risk factors, heightened surveillance for high-risk patients, and more timely and appropriate interventions, thereby helping obstetricians deliver better care. We identify and study the most important risk factors for four types of pregnancy complications: (i) severe maternal morbidity, (ii) shoulder dystocia, (iii) preterm preeclampsia, and (iv) antepartum stillbirth. We use an Explainable Boosting Machine (EBM), a high-accuracy glass-box learning method, for the prediction and identification of important risk factors. We undertake external validation and perform an extensive robustness analysis of the EBM models. EBMs match the accuracy of other black-box ML methods, such as deep neural networks and random forests, and outperform logistic regression, while being more interpretable. EBMs prove to be robust. The interpretability of the EBM models reveal surprising insights into the features contributing to risk (e.g., maternal height is the second most important feature for shoulder dystocia) and may have potential for clinical application in the prediction and prevention of serious complications in pregnancy.
Collapse
Affiliation(s)
| | - Zifei Xu
- Stanford University, Stanford, CA USA
| | - Hui Lan
- Stanford University, Stanford, CA USA
| | | | | | - Ian Painter
- Foundation for Healthcare Quality, Seattle, WA USA
| | | | | |
Collapse
|
2
|
Cárdenas-Fuentes G, Bosch de Basea M, Cobo I, Subirana I, Ceresa M, Famada E, Gimeno-Santos E, Delgado-Ortiz L, Faner R, Molina-Molina M, Agustí À, Muñoz X, Sibila O, Gea J, Garcia-Aymerich J. Validity of prognostic models of critical COVID-19 is variable. A systematic review with external validation. J Clin Epidemiol 2023; 159:274-288. [PMID: 37142168 PMCID: PMC10152752 DOI: 10.1016/j.jclinepi.2023.04.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2022] [Revised: 01/26/2023] [Accepted: 04/25/2023] [Indexed: 05/06/2023]
Abstract
OBJECTIVES To identify prognostic models which estimate the risk of critical COVID-19 in hospitalized patients and to assess their validation properties. STUDY DESIGN AND SETTING We conducted a systematic review in Medline (up to January 2021) of studies developing or updating a model that estimated the risk of critical COVID-19, defined as death, admission to intensive care unit, and/or use of mechanical ventilation during admission. Models were validated in two datasets with different backgrounds (HM [private Spanish hospital network], n = 1,753, and ICS [public Catalan health system], n = 1,104), by assessing discrimination (area under the curve [AUC]) and calibration (plots). RESULTS We validated 18 prognostic models. Discrimination was good in nine of them (AUCs ≥ 80%) and higher in those predicting mortality (AUCs 65%-87%) than those predicting intensive care unit admission or a composite outcome (AUCs 53%-78%). Calibration was poor in all models providing outcome's probabilities and good in four models providing a point-based score. These four models used mortality as outcome and included age, oxygen saturation, and C-reactive protein among their predictors. CONCLUSION The validity of models predicting critical COVID-19 by using only routinely collected predictors is variable. Four models showed good discrimination and calibration when externally validated and are recommended for their use.
Collapse
Affiliation(s)
- Gabriela Cárdenas-Fuentes
- ISGlobal, Barcelona, Spain; Universitat Pompeu Fabra (UPF), Barcelona, Spain; CIBER Epidemiología y Salud Pública (CIBERESP), Barcelona, Spain; School of Health Sciences, Blanquerna-Universitat Ramon Llull, Barcelona, Spain.
| | - Magda Bosch de Basea
- ISGlobal, Barcelona, Spain; Universitat Pompeu Fabra (UPF), Barcelona, Spain; CIBER Epidemiología y Salud Pública (CIBERESP), Barcelona, Spain
| | - Inés Cobo
- ISGlobal, Barcelona, Spain; Universitat Pompeu Fabra (UPF), Barcelona, Spain; CIBER Epidemiología y Salud Pública (CIBERESP), Barcelona, Spain
| | - Isaac Subirana
- Instituto Hospital del Mar de Investigaciones Médicas (IMIM), Barcelona, Spain; CIBER Enfermedades Cardiovasculares (CIBERCV), ISCIII, Spain
| | - Mario Ceresa
- BCNMedTech, Universitat Pompeu Fabra (UPF), Barcelona, Spain
| | | | - Elena Gimeno-Santos
- ISGlobal, Barcelona, Spain; Universitat Pompeu Fabra (UPF), Barcelona, Spain; CIBER Epidemiología y Salud Pública (CIBERESP), Barcelona, Spain; Respiratory Institute, Hospital Clinic, Barcelona, Spain
| | - Laura Delgado-Ortiz
- ISGlobal, Barcelona, Spain; Universitat Pompeu Fabra (UPF), Barcelona, Spain; CIBER Epidemiología y Salud Pública (CIBERESP), Barcelona, Spain
| | - Rosa Faner
- Instituto de Investigaciones Biomédicas August Pi i Sunyer (IDIBAPS), Barcelona, Spain; Universitat de Barcelona, Barcelona, Spain; CIBER Enfermedades Respiratorias (CIBERES), ISCIII, Spain
| | - María Molina-Molina
- CIBER Enfermedades Respiratorias (CIBERES), ISCIII, Spain; Servicio de Neumología, Hospital Universitario de Bellvitge, L'Hospitalet de Llobregat, Spain; Instituto de Investigación Biomédica de Bellvitge (IDIBELL), L'Hospitalet de Llobregat, Spain
| | - Àlvar Agustí
- Respiratory Institute, Hospital Clinic, Barcelona, Spain; Instituto de Investigaciones Biomédicas August Pi i Sunyer (IDIBAPS), Barcelona, Spain; Universitat de Barcelona, Barcelona, Spain; CIBER Enfermedades Respiratorias (CIBERES), ISCIII, Spain
| | - Xavier Muñoz
- CIBER Enfermedades Respiratorias (CIBERES), ISCIII, Spain; Servicio de Neumología, Hospital Universitario Vall d'Hebron, Barcelona, Spain; Departamento de Biología celular, fisiología e inmunología, Universitat Autònoma de Barcelona, Barcelona, Spain
| | - Oriol Sibila
- Respiratory Institute, Hospital Clinic, Barcelona, Spain; Instituto de Investigaciones Biomédicas August Pi i Sunyer (IDIBAPS), Barcelona, Spain; CIBER Enfermedades Respiratorias (CIBERES), ISCIII, Spain
| | - Joaquim Gea
- Universitat Pompeu Fabra (UPF), Barcelona, Spain; CIBER Enfermedades Respiratorias (CIBERES), ISCIII, Spain; Servicio de Neumología, Hospital del Mar-IMIM, Barcelona, Spain; Fundació Barcelona Respiratory Network (BRN), Barcelona, Spain
| | - Judith Garcia-Aymerich
- ISGlobal, Barcelona, Spain; Universitat Pompeu Fabra (UPF), Barcelona, Spain; CIBER Epidemiología y Salud Pública (CIBERESP), Barcelona, Spain
| |
Collapse
|
3
|
Levy TJ, Coppa K, Cang J, Barnaby DP, Paradis MD, Cohen SL, Makhnevich A, van Klaveren D, Kent DM, Davidson KW, Hirsch JS, Zanos TP. Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients. Nat Commun 2022; 13:6812. [PMID: 36357420 PMCID: PMC9648888 DOI: 10.1038/s41467-022-34646-2] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2022] [Accepted: 11/02/2022] [Indexed: 11/12/2022] Open
Abstract
Clinical prognostic models can assist patient care decisions. However, their performance can drift over time and location, necessitating model monitoring and updating. Despite rapid and significant changes during the pandemic, prognostic models for COVID-19 patients do not currently account for these drifts. We develop a framework for continuously monitoring and updating prognostic models and apply it to predict 28-day survival in COVID-19 patients. We use demographic, laboratory, and clinical data from electronic health records of 34912 hospitalized COVID-19 patients from March 2020 until May 2022 and compare three modeling methods. Model calibration performance drift is immediately detected with minor fluctuations in discrimination. The overall calibration on the prospective validation cohort is significantly improved when comparing the dynamically updated models against their static counterparts. Our findings suggest that, using this framework, models remain accurate and well-calibrated across various waves, variants, race and sex and yield positive net-benefits.
Collapse
Affiliation(s)
- Todd J Levy
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
| | - Kevin Coppa
- Clinical Digital Solutions, Northwell Health, New Hyde Park, NY, 11042, USA
| | - Jinxuan Cang
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
| | - Douglas P Barnaby
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY, 11549, USA
| | - Marc D Paradis
- Northwell Holdings, Northwell Health, Manhasset, NY, 11030, USA
| | - Stuart L Cohen
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY, 11549, USA
| | - Alex Makhnevich
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY, 11549, USA
| | - David van Klaveren
- Department of Public Health, Erasmus MC University Medical Center, Rotterdam, Netherlands
- Predictive Analytics and Comparative Effectiveness Center, Tufts Medical Center, Boston, MA, USA
| | - David M Kent
- Predictive Analytics and Comparative Effectiveness Center, Tufts Medical Center, Boston, MA, USA
| | - Karina W Davidson
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY, 11549, USA
| | - Jamie S Hirsch
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Clinical Digital Solutions, Northwell Health, New Hyde Park, NY, 11042, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY, 11549, USA
| | - Theodoros P Zanos
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA.
- Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA.
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY, 11549, USA.
| |
Collapse
|