1
|
Al‐Farra H, Ravelli ACJ, Henriques JPS, Houterman S, de Mol BAJM, Abu‐Hanna A. Development and validation of a prediction model for early mortality after transcatheter aortic valve implantation (TAVI) based on the Netherlands Heart Registration (NHR): The TAVI-NHR risk model. Catheter Cardiovasc Interv 2022; 100:879-889. [PMID: 36069120 PMCID: PMC9826169 DOI: 10.1002/ccd.30398] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/25/2022] [Revised: 08/02/2022] [Accepted: 08/25/2022] [Indexed: 01/11/2023]
Abstract
BACKGROUND The currently available mortality prediction models (MPM) have suboptimal performance when predicting early mortality (30-days) following transcatheter aortic valve implantation (TAVI) on various external populations. We developed and validated a new TAVI-MPM based on a large number of predictors with recent data from a national heart registry. METHODS We included all TAVI-patients treated in the Netherlands between 2013 and 2018, from the Netherlands Heart Registration. We used logistic-regression analysis based on the Akaike Information Criterion for variable selection. We multiply imputed missing values, but excluded variables with >30% missing values. For internal validation, we used ten-fold cross-validation. For temporal (prospective) validation, we used the 2018-data set for testing. We assessed discrimination by the c-statistic, predicted probability accuracy by the Brier score, and calibration by calibration graphs, and calibration-intercept and calibration slope. We compared our new model to the updated ACC-TAVI and IRRMA MPMs on our population. RESULTS We included 9144 TAVI-patients. The observed early mortality was 4.0%. The final MPM had 10 variables, including: critical-preoperative state, procedure-acuteness, body surface area, serum creatinine, and diabetes-mellitus status. The median c-statistic was 0.69 (interquartile range [IQR] 0.646-0.75). The median Brier score was 0.038 (IQR 0.038-0.040). No signs of miscalibration were observed. The c-statistic's temporal-validation was 0.71 (95% confidence intervals 0.64-0.78). Our model outperformed the updated currently available MPMs ACC-TAVI and IRRMA (p value < 0.05). CONCLUSION The new TAVI-model used additional variables and showed fair discrimination and good calibration. It outperformed the updated currently available TAVI-models on our population. The model's good calibration benefits preprocedural risk-assessment and patient counseling.
Collapse
Affiliation(s)
- Hatem Al‐Farra
- Department of Medical Informatics, Amsterdam UMCLocation University of AmsterdamAmsterdamThe Netherlands
- Heart Centre, Amsterdam Cardiovascular Sciences, Amsterdam UMCLocation University of AmsterdamAmsterdamThe Netherlands
- Amsterdam Public HealthAmsterdamThe Netherlands
| | - Anita C. J. Ravelli
- Department of Medical Informatics, Amsterdam UMCLocation University of AmsterdamAmsterdamThe Netherlands
- Amsterdam Public HealthAmsterdamThe Netherlands
| | - José P. S. Henriques
- Heart Centre, Amsterdam Cardiovascular Sciences, Amsterdam UMCLocation University of AmsterdamAmsterdamThe Netherlands
| | | | - Bas A. J. M. de Mol
- Heart Centre, Amsterdam Cardiovascular Sciences, Amsterdam UMCLocation University of AmsterdamAmsterdamThe Netherlands
| | - Ameen Abu‐Hanna
- Department of Medical Informatics, Amsterdam UMCLocation University of AmsterdamAmsterdamThe Netherlands
| | | |
Collapse
|
2
|
Chi S, Tian Y, Wang F, Zhou T, Jin S, Li J. A novel lifelong machine learning-based method to eliminate calibration drift in clinical prediction models. Artif Intell Med 2022; 125:102256. [DOI: 10.1016/j.artmed.2022.102256] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2021] [Revised: 01/14/2022] [Accepted: 02/09/2022] [Indexed: 02/03/2023]
|
3
|
Fernandez-Felix BM, Barca LV, Garcia-Esquinas E, Correa-Pérez A, Fernández-Hidalgo N, Muriel A, Lopez-Alcalde J, Álvarez-Diaz N, Pijoan JI, Ribera A, Elorza EN, Muñoz P, Fariñas MDC, Goenaga MÁ, Zamora J. Prognostic models for mortality after cardiac surgery in patients with infective endocarditis: a systematic review and aggregation of prediction models. Clin Microbiol Infect 2021; 27:1422-1430. [PMID: 34620380 DOI: 10.1016/j.cmi.2021.05.051] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2021] [Revised: 05/19/2021] [Accepted: 05/31/2021] [Indexed: 12/13/2022]
Abstract
BACKGROUND There are several prognostic models to estimate the risk of mortality after surgery for active infective endocarditis (IE). However, these models incorporate different predictors and their performance is uncertain. OBJECTIVE We systematically reviewed and critically appraised all available prediction models of postoperative mortality in patients undergoing surgery for IE, and aggregated them into a meta-model. DATA SOURCES We searched Medline and EMBASE databases from inception to June 2020. STUDY ELIGIBILITY CRITERIA We included studies that developed or updated a prognostic model of postoperative mortality in patient with IE. METHODS We assessed the risk of bias of the models using PROBAST (Prediction model Risk Of Bias ASsessment Tool) and we aggregated them into an aggregate meta-model based on stacked regressions and optimized it for a nationwide registry of IE patients. The meta-model performance was assessed using bootstrap validation methods and adjusted for optimism. RESULTS We identified 11 prognostic models for postoperative mortality. Eight models had a high risk of bias. The meta-model included weighted predictors from the remaining three models (EndoSCORE, specific ES-I and specific ES-II), which were not rated as high risk of bias and provided full model equations. Additionally, two variables (age and infectious agent) that had been modelled differently across studies, were estimated based on the nationwide registry. The performance of the meta-model was better than the original three models, with the corresponding performance measures: C-statistics 0.79 (95% CI 0.76-0.82), calibration slope 0.98 (95% CI 0.86-1.13) and calibration-in-the-large -0.05 (95% CI -0.20 to 0.11). CONCLUSIONS The meta-model outperformed published models and showed a robust predictive capacity for predicting the individualized risk of postoperative mortality in patients with IE. PROTOCOL REGISTRATION PROSPERO (registration number CRD42020192602).
Collapse
Affiliation(s)
- Borja M Fernandez-Felix
- Clinical Biostatistics Unit, Hospital Universitario Ramon y Cajal (IRYCIS), Madrid, Spain; CIBER Epidemiology and Public Health (CIBERESP), Madrid, Spain.
| | - Laura Varela Barca
- Department of Cardiovascular Surgery, Fundacion Jimenez Diaz University Hospital, Madrid, Spain
| | - Esther Garcia-Esquinas
- CIBER Epidemiology and Public Health (CIBERESP), Madrid, Spain; Department of Preventive Medicine and Public Health, School of Medicine, Universidad Autónoma de Madrid, Madrid, Spain; IdiPaz (Hospital Universitario La Paz-Universidad Autónoma de Madrid), Madrid, Spain
| | - Andrea Correa-Pérez
- Clinical Biostatistics Unit, Hospital Universitario Ramon y Cajal (IRYCIS), Madrid, Spain; Faculty of Medicine, Universidad Francisco de Vitoria, Madrid, Spain
| | - Nuria Fernández-Hidalgo
- Servei de Malalties Infeccioses, Hospital Universitari Vall d'Hebron, Barcelona, Spain; Red Española de Investigación en Patología Infecciosa (REIPI), Instituto de Salud Carlos III, Madrid, Spain
| | - Alfonso Muriel
- Clinical Biostatistics Unit, Hospital Universitario Ramon y Cajal (IRYCIS), Madrid, Spain; CIBER Epidemiology and Public Health (CIBERESP), Madrid, Spain
| | - Jesus Lopez-Alcalde
- Clinical Biostatistics Unit, Hospital Universitario Ramon y Cajal (IRYCIS), Madrid, Spain; CIBER Epidemiology and Public Health (CIBERESP), Madrid, Spain; Faculty of Medicine, Universidad Francisco de Vitoria, Madrid, Spain; Institute for Complementary and Integrative Medicine, University Hospital Zurich and University of Zurich, Zurich, Switzerland
| | - Noelia Álvarez-Diaz
- Medical Library, Hospital Universitario Ramon y Cajal (IRYCIS), Madrid, Madrid, Spain
| | - Jose I Pijoan
- CIBER Epidemiology and Public Health (CIBERESP), Madrid, Spain; Hospital Universitario Cruces/OSI EEC, Barakaldo, Spain; Biocruces-Bizkaia Health Research Institute, Barakaldo, Spain
| | - Aida Ribera
- CIBER Epidemiology and Public Health (CIBERESP), Madrid, Spain; Cardiovascular Epidemiology and Research Unit, Hospital Universitari Vall d'Hebron, Barcelona, Spain
| | - Enrique Navas Elorza
- Department of Infectology, Hospital Universitario Ramon y Cajal (IRYCIS), Madrid, Spain
| | - Patricia Muñoz
- Clinical Microbiology and Infectious Diseases Service, Hospital General Universitario Gregorio Marañón, Instituto de Investigación Sanitaria Gregorio Marañón, CIBER Enfermedades Respiratorias-CIBERES, Facultad de Medicina, Universidad Complutense de Madrid, Madrid, Spain
| | - María Del Carmen Fariñas
- Infectious Diseases Service, Hospital Universitario Marqués de Valdecilla-IDIVAL, Universidad de Cantabria, Santander, Spain
| | - Miguel Ángel Goenaga
- Infectious Diseases Service, Hospital Universitario Donostia, IIS Biodonostia, OSI Donostialdea, San Sebastián, Spain
| | - Javier Zamora
- Clinical Biostatistics Unit, Hospital Universitario Ramon y Cajal (IRYCIS), Madrid, Spain; CIBER Epidemiology and Public Health (CIBERESP), Madrid, Spain; WHO Collaborating Centre for Global Women's Health, Institute of Metabolism and Systems Research, University of Birmingham, Birmingham, UK
| |
Collapse
|
4
|
Al-Farra H, de Mol BAJM, Ravelli ACJ, Ter Burg WJPP, Houterman S, Henriques JPS, Abu-Hanna A, Vis MM, Vos J, Timmers L, Tonino WAL, Schotborgh CE, Roolvink V, Porta F, Stoel MG, Kats S, Amoroso G, van der Werf HW, Stella PR, de Jaegere P. Update and, internal and temporal-validation of the FRANCE-2 and ACC-TAVI early-mortality prediction models for Transcatheter Aortic Valve Implantation (TAVI) using data from the Netherlands heart registration (NHR). IJC HEART & VASCULATURE 2021; 32:100716. [PMID: 33537406 PMCID: PMC7843396 DOI: 10.1016/j.ijcha.2021.100716] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2020] [Revised: 12/30/2020] [Accepted: 01/04/2021] [Indexed: 01/08/2023]
Abstract
Background The predictive performance of the models FRANCE-2 and ACC-TAVI for early-mortality after Transcatheter Aortic Valve Implantation (TAVI) can decline over time and can be enhanced by updating them on new populations. We aim to update and internally and temporally validate these models using a recent TAVI-cohort from the Netherlands Heart Registration (NHR). Methods We used data of TAVI-patients treated in 2013-2017. For each original-model, the best update-method (model-intercept, model-recalibration, or model-revision) was selected by a closed-testing procedure. We internally validated both updated models with 1000 bootstrap samples. We also updated the models on the 2013-2016 dataset and temporally validated them on the 2017-dataset. Performance measures were the Area-Under ROC-curve (AU-ROC), Brier-score, and calibration graphs. Results We included 6177 TAVI-patients, with 4.5% observed early-mortality. The selected update-method for FRANCE-2 was model-intercept-update. Internal validation showed an AU-ROC of 0.63 (95%CI 0.62-0.66) and Brier-score of 0.04 (0.04-0.05). Calibration graphs show that it overestimates early-mortality. In temporal-validation, the AU-ROC was 0.61 (0.53-0.67).The selected update-method for ACC-TAVI was model-revision. In internal-validation, the AU-ROC was 0.63 (0.63-0.66) and Brier-score was 0.04 (0.04-0.05). The updated ACC-TAVI calibrates well up to a probability of 20%, and subsequently underestimates early-mortality. In temporal-validation the AU-ROC was 0.65 (0.58-0.72). Conclusion Internal-validation of the updated models FRANCE-2 and ACC-TAVI with data from the NHR demonstrated improved performance, which was better than in external-validation studies and comparable to the original studies. In temporal-validation, ACC-TAVI outperformed FRANCE-2 because it suffered less from changes over time.
Collapse
Key Words
- ACC-TAVI (ACC TVT), American College of Cardiology Transcatheter Valve Therapy
- AU-PRC, Area Under the Precision-Recall Curve
- AU-ROC, Area Under the Receiver Operating-Characteristic Curve
- Amsterdam UMC, Amsterdam University Medical Center - location AMC (Academic Medical Center)
- BSS, Brier-skill score
- Closed-testing procedure
- EuroSCORE, European System for Cardiac Operative Risk Evaluation
- External Validation
- FRANCE-2, French Aortic National CoreValve and Edwards [15]
- LVEF, Left Ventricular Ejection Fraction
- MPM, Mortality Prediction Models
- Model recalibration
- Model updating
- NHR, Netherlands Heart Registration (“Nederlandse Hart Registratie in Dutch”)
- NYHA, New York Heart Association
- Prediction models
- SAVR, Surgical Aortic Valve Replacement
- TAVI (TAVR), Transcatheter Aortic Valve Implantation (Replacement)
- Transcatheter Aortic Valve Implantation (TAVI)
Collapse
Affiliation(s)
- Hatem Al-Farra
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands.,Heart Center, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Bas A J M de Mol
- Heart Center, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Anita C J Ravelli
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - W J P P Ter Burg
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | | | - José P S Henriques
- Heart Center, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Ameen Abu-Hanna
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - M M Vis
- Heart Center, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - J Vos
- Amphia Hospital, the Netherlands
| | - L Timmers
- St. Antonius Hospital, the Netherlands
| | | | | | | | - F Porta
- Leeuwarden Medical Center, the Netherlands
| | - M G Stoel
- Medisch Spectrum Twente, the Netherlands
| | - S Kats
- Maastricht University Medical Center, the Netherlands
| | - G Amoroso
- Onze Lieve Vrouwe Gasthuis, the Netherlands
| | | | - P R Stella
- University Medical Center Utrecht, the Netherlands
| | - P de Jaegere
- Erasmus University Medical Center, the Netherlands
| | | |
Collapse
|
5
|
Jenkins DA, Martin GP, Sperrin M, Riley RD, Debray TPA, Collins GS, Peek N. Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems? Diagn Progn Res 2021; 5:1. [PMID: 33431065 PMCID: PMC7797885 DOI: 10.1186/s41512-020-00090-3] [Citation(s) in RCA: 46] [Impact Index Per Article: 15.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/14/2020] [Accepted: 12/08/2020] [Indexed: 01/01/2023] Open
Abstract
Clinical prediction models (CPMs) have become fundamental for risk stratification across healthcare. The CPM pipeline (development, validation, deployment, and impact assessment) is commonly viewed as a one-time activity, with model updating rarely considered and done in a somewhat ad hoc manner. This fails to address the fact that the performance of a CPM worsens over time as natural changes in populations and care pathways occur. CPMs need constant surveillance to maintain adequate predictive performance. Rather than reactively updating a developed CPM once evidence of deteriorated performance accumulates, it is possible to proactively adapt CPMs whenever new data becomes available. Approaches for validation then need to be changed accordingly, making validation a continuous rather than a discrete effort. As such, "living" (dynamic) CPMs represent a paradigm shift, where the analytical methods dynamically generate updated versions of a model through time; one then needs to validate the system rather than each subsequent model revision.
Collapse
Affiliation(s)
- David A Jenkins
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.
- NIHR Greater Manchester Patient Safety Translational Research Centre, The University of Manchester, Manchester, UK.
| | - Glen P Martin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Matthew Sperrin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Richard D Riley
- Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Staffordshire, UK
| | - Thomas P A Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Niels Peek
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
- NIHR Greater Manchester Patient Safety Translational Research Centre, The University of Manchester, Manchester, UK
- NIHR Manchester Biomedical Research Centre, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| |
Collapse
|
6
|
Martin GP, Sperrin M, Snell KIE, Buchan I, Riley RD. Clinical prediction models to predict the risk of multiple binary outcomes: a comparison of approaches. Stat Med 2020; 40:498-517. [PMID: 33107066 DOI: 10.1002/sim.8787] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2020] [Revised: 08/25/2020] [Accepted: 10/07/2020] [Indexed: 12/13/2022]
Abstract
Clinical prediction models (CPMs) can predict clinically relevant outcomes or events. Typically, prognostic CPMs are derived to predict the risk of a single future outcome. However, there are many medical applications where two or more outcomes are of interest, meaning this should be more widely reflected in CPMs so they can accurately estimate the joint risk of multiple outcomes simultaneously. A potentially naïve approach to multi-outcome risk prediction is to derive a CPM for each outcome separately, then multiply the predicted risks. This approach is only valid if the outcomes are conditionally independent given the covariates, and it fails to exploit the potential relationships between the outcomes. This paper outlines several approaches that could be used to develop CPMs for multiple binary outcomes. We consider four methods, ranging in complexity and conditional independence assumptions: namely, probabilistic classifier chain, multinomial logistic regression, multivariate logistic regression, and a Bayesian probit model. These are compared with methods that rely on conditional independence: separate univariate CPMs and stacked regression. Employing a simulation study and real-world example, we illustrate that CPMs for joint risk prediction of multiple outcomes should only be derived using methods that model the residual correlation between outcomes. In such a situation, our results suggest that probabilistic classification chains, multinomial logistic regression or the Bayesian probit model are all appropriate choices. We call into question the development of CPMs for each outcome in isolation when multiple correlated or structurally related outcomes are of interest and recommend more multivariate approaches to risk prediction.
Collapse
Affiliation(s)
- Glen P Martin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
| | - Matthew Sperrin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
| | - Kym I E Snell
- Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Staffordshire, UK
| | - Iain Buchan
- Institute of Population Health Sciences, Faculty of Health and Life Sciences, University of Liverpool, Liverpool, UK
| | - Richard D Riley
- Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Staffordshire, UK
| |
Collapse
|
7
|
Establishment and evaluation of a multicenter collaborative prediction model construction framework supporting model generalization and continuous improvement: A pilot study. Int J Med Inform 2020; 141:104173. [PMID: 32531725 DOI: 10.1016/j.ijmedinf.2020.104173] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2020] [Revised: 04/10/2020] [Accepted: 05/09/2020] [Indexed: 12/13/2022]
Abstract
BACKGROUND AND OBJECTIVE In recent years, an increasing number of clinical prediction models have been developed to serve clinical care. Establishing a data-driven prediction model based on large-scale electronic health record (EHR) data can provide a more empirical basis for clinical decision making. However, research on model generalization and continuous improvement is insufficiently focused, which also hinders the application and evaluation of prediction models in real clinical environments. Therefore, this study proposes a multicenter collaborative prediction model construction framework to build a prediction model with greater generalizability and continuous improvement capabilities while preserving patient data security and privacy. MATERIALS AND METHODS Based on a multicenter collaborative research network, such as the Observational Health Data Sciences and Informatics (OHDSI), a multicenter collaborative prediction model construction framework is proposed. Based on the idea of multi-source transfer learning, in each source hospital, a base classifier was trained according to the model research setting. Then, in the target hospital with missing calibration data, a prediction model was established through weighted integration of base classifiers from source hospitals based on the smoothness assumption. Moreover, a passive-aggressive online learning algorithm was used for continuous improvement of the prediction model, which can help to maintain a high predictive performance to provide reliable clinical decision-making abilities. To evaluate the proposed prediction model construction framework, a prototype system for colorectal cancer prognosis prediction was developed. To evaluate the performance of models, 70,906 patients were screened, including 70,090 from 5 US hospital-specific datasets and 816 from a Chinese hospital-specific dataset. The area under the receiver operating characteristic curve (AUC) and the estimated calibration index (ECI) were used to evaluate the discrimination and calibration of models. RESULTS Regarding the colorectal cancer prognosis prediction in our prototype system, compared with the reference models, our model achieved a better performance in model calibration (ECI = 9.294 [9.146, 9.441]) and a similar ability in model discrimination (AUC = 0.783 [0.780, 0.786]). Furthermore, the online learning process provided in this study can continuously improve the performance of the prediction model when patient data with specified labels arrive (the AUC value increased from 0.709 to 0.715 and the ECI value decreased from 13.013 to 9.634 after 650 patient instances with specified labels from the Chinese hospital arrived), enabling the prediction model to maintain a good predictive performance during clinical application. CONCLUSIONS This study proposes and evaluates a multicenter collaborative prediction model construction framework that can support the construction of prediction models with better generalizability and continuous improvement capabilities without the need to aggregate multicenter patient-level data.
Collapse
|
8
|
Nguyen T, Collins GS, Pellegrini F, Moons KG, Debray TP. On the aggregation of published prognostic scores for causal inference in observational studies. Stat Med 2020; 39:1440-1457. [PMID: 32022311 PMCID: PMC7187258 DOI: 10.1002/sim.8489] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2019] [Revised: 12/12/2019] [Accepted: 01/14/2020] [Indexed: 12/23/2022]
Abstract
As real world evidence on drug efficacy involves nonrandomized studies, statistical methods adjusting for confounding are needed. In this context, prognostic score (PGS) analysis has recently been proposed as a method for causal inference. It aims to restore balance across the different treatment groups by identifying subjects with a similar prognosis for a given reference exposure ("control"). This requires the development of a multivariable prognostic model in the control arm of the study sample, which is then extrapolated to the different treatment arms. Unfortunately, large cohorts for developing prognostic models are not always available. Prognostic models are therefore subject to a dilemma between overfitting and parsimony; the latter being prone to a violation of the assumption of no unmeasured confounders when important covariates are ignored. Although it is possible to limit overfitting by using penalization strategies, an alternative approach is to adopt evidence synthesis. Aggregating previously published prognostic models may improve the generalizability of PGS, while taking account of a large set of covariates-even when limited individual participant data are available. In this article, we extend a method for prediction model aggregation to PGS analysis in nonrandomized studies. We conduct extensive simulations to assess the validity of model aggregation, compared with other methods of PGS analysis for estimating marginal treatment effects. We show that aggregating existing PGS into a "meta-score" is robust to misspecification, even when elementary scores wrongfully omit confounders or focus on different outcomes. We illustrate our methods in a setting of treatments for asthma.
Collapse
Affiliation(s)
- Tri‐Long Nguyen
- Section of Epidemiology, Department of Public HealthUniversity of CopenhagenCopenhagenDenmark
- Julius Center for Health Sciences and Primary CareUniversity Medical Center Utrecht, Utrecht UniversityUtrechtThe Netherlands
- Department of PharmacyNîmes University Hospital CentreNîmesFrance
| | - Gary S. Collins
- National Institute for Health Research Oxford Biomedical Research CentreJohn Radcliffe HospitalOxfordUK
| | | | - Karel G.M. Moons
- Julius Center for Health Sciences and Primary CareUniversity Medical Center Utrecht, Utrecht UniversityUtrechtThe Netherlands
- Cochrane NetherlandsUniversity Medical Center UtrechtUtrechtThe Netherlands
| | - Thomas P.A. Debray
- Julius Center for Health Sciences and Primary CareUniversity Medical Center Utrecht, Utrecht UniversityUtrechtThe Netherlands
- Cochrane NetherlandsUniversity Medical Center UtrechtUtrechtThe Netherlands
| |
Collapse
|
9
|
Debray TP, de Jong VM, Moons KG, Riley RD. Evidence synthesis in prognosis research. Diagn Progn Res 2019; 3:13. [PMID: 31338426 PMCID: PMC6621956 DOI: 10.1186/s41512-019-0059-4] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/23/2018] [Accepted: 04/16/2019] [Indexed: 12/11/2022] Open
Abstract
Over the past few years, evidence synthesis has become essential to investigate and improve the generalizability of medical research findings. This strategy often involves a meta-analysis to formally summarize quantities of interest, such as relative treatment effect estimates. The use of meta-analysis methods is, however, less straightforward in prognosis research because substantial variation exists in research objectives, analysis methods and the level of reported evidence. We present a gentle overview of statistical methods that can be used to summarize data of prognostic factor and prognostic model studies. We discuss how aggregate data, individual participant data, or a combination thereof can be combined through meta-analysis methods. Recent examples are provided throughout to illustrate the various methods.
Collapse
Affiliation(s)
- Thomas P.A. Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
| | - Valentijn M.T. de Jong
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
| | - Karel G.M. Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
| | - Richard D. Riley
- Research Institute for Primary Care & Health Sciences, Keele University, Staffordshire, ST5 5BG UK
| |
Collapse
|
10
|
Chi S, Li X, Tian Y, Li J, Kong X, Ding K, Weng C, Li J. Semi-supervised learning to improve generalizability of risk prediction models. J Biomed Inform 2019; 92:103117. [DOI: 10.1016/j.jbi.2019.103117] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/11/2022]
|
11
|
Martin GP, Sperrin M, Mamas MA. Pre-procedural risk models for patients undergoing transcatheter aortic valve implantation. J Thorac Dis 2018; 10:S3560-S3567. [PMID: 30505535 DOI: 10.21037/jtd.2018.05.67] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
Transcatheter aortic valve implantation (TAVI) has emerged as the standard treatment option for patients with symptomatic aortic stenosis who are considered intermediate to high surgical risk. Nonetheless, optimal clinical outcomes following the procedure require careful consideration of procedural risk by the Heart Team. While this decision-making could be supported through the development of TAVI-specific clinical prediction models (CPMs), current models remain suboptimal. In this review paper, we aimed to outline the performance of several recently derived TAVI CPMs that predict mortality and present some future research directions. We discuss how the existing risk models have achieved only moderate discrimination but highlight that some of the models are well calibrated across multiple populations, indicating the feasibility of using them to aid benchmarking analyses. Moreover, we suggest that future work should focus on the development of CPMs in cohorts of patients with aortic stenosis that include multiple treatment modalities. Supported by appropriate modelling of 'what if' scenarios, this would allow the Heart Teams to predict and compare outcomes across surgical aortic valve replacement, medical management and TAVI, thereby allowing one to personalise treatment decisions to the individual patient. Such a goal could be facilitated by considering novel risk factors, shifting the focus to endpoints other than mortality, and through collaborative efforts to combine the evidence base and existing models across wider populations.
Collapse
Affiliation(s)
- Glen P Martin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Matthew Sperrin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Mamas A Mamas
- Keele Cardiovascular Research Group, Centre for Prognosis Research, Institute for Primary Care and Health Sciences, Keele University, Stoke-on-Trent, UK
| |
Collapse
|
12
|
Martin GP, Mamas MA, Peek N, Buchan I, Sperrin M. A multiple-model generalisation of updating clinical prediction models. Stat Med 2017; 37:1343-1358. [PMID: 29250812 PMCID: PMC5873448 DOI: 10.1002/sim.7586] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2017] [Revised: 11/15/2017] [Accepted: 11/19/2017] [Indexed: 12/23/2022]
Abstract
There is growing interest in developing clinical prediction models (CPMs) to aid local healthcare decision‐making. Frequently, these CPMs are developed in isolation across different populations, with repetitive de novo derivation a common modelling strategy. However, this fails to utilise all available information and does not respond to changes in health processes through time and space. Alternatively, model updating techniques have previously been proposed that adjust an existing CPM to suit the new population, but these techniques are restricted to a single model. Therefore, we aimed to develop a generalised method for updating and aggregating multiple CPMs. The proposed “hybrid method” re‐calibrates multiple CPMs using stacked regression while concurrently revising specific covariates using individual participant data (IPD) under a penalised likelihood. The performance of the hybrid method was compared with existing methods in a clinical example of mortality risk prediction after transcatheter aortic valve implantation, and in 2 simulation studies. The simulation studies explored the effect of sample size and between‐population‐heterogeneity on the method, with each representing a situation of having multiple distinct CPMs and 1 set of IPD. When the sample size of the IPD was small, stacked regression and the hybrid method had comparable but highest performance across modelling methods. Conversely, in large IPD samples, development of a new model and the hybrid method gave the highest performance. Hence, the proposed strategy can inform the choice between utilising existing CPMs or developing a model de novo, thereby incorporating IPD, existing research, and prior (clinical) knowledge into the modelling strategy.
Collapse
Affiliation(s)
- Glen P Martin
- Farr Institute, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Mamas A Mamas
- Farr Institute, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.,Keele Cardiovascular Research Group, Keele University, Stoke-on-Trent, UK
| | - Niels Peek
- Farr Institute, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.,NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre, University of Manchester, Manchester, UK
| | - Iain Buchan
- Farr Institute, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.,Microsoft Research, Cambridge, UK
| | - Matthew Sperrin
- Farr Institute, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| |
Collapse
|