651
|
Vodencarevic A, Tascilar K, Hartmann F, Reiser M, Hueber AJ, Haschka J, Bayat S, Meinderink T, Knitza J, Mendez L, Hagen M, Krönke G, Rech J, Manger B, Kleyer A, Zimmermann-Rittereiser M, Schett G, Simon D. Advanced machine learning for predicting individual risk of flares in rheumatoid arthritis patients tapering biologic drugs. Arthritis Res Ther 2021; 23:67. [PMID: 33640008 PMCID: PMC7913400 DOI: 10.1186/s13075-021-02439-5] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2020] [Accepted: 02/10/2021] [Indexed: 12/27/2022] Open
Abstract
BACKGROUND Biological disease-modifying anti-rheumatic drugs (bDMARDs) can be tapered in some rheumatoid arthritis (RA) patients in sustained remission. The purpose of this study was to assess the feasibility of building a model to estimate the individual flare probability in RA patients tapering bDMARDs using machine learning methods. METHODS Longitudinal clinical data of RA patients on bDMARDs from a randomized controlled trial of treatment withdrawal (RETRO) were used to build a predictive model to estimate the probability of a flare. Four basic machine learning models were trained, and their predictions were additionally combined to train an ensemble learning method, a stacking meta-classifier model to predict the individual flare probability within 14 weeks after each visit. Prediction performance was estimated using nested cross-validation as the area under the receiver operating curve (AUROC). Predictor importance was estimated using the permutation importance approach. RESULTS Data of 135 visits from 41 patients were included. A model selection approach based on nested cross-validation was implemented to find the most suitable modeling formalism for the flare prediction task as well as the optimal model hyper-parameters. Moreover, an approach based on stacking different classifiers was successfully applied to create a powerful and flexible prediction model with the final measured AUROC of 0.81 (95%CI 0.73-0.89). The percent dose change of bDMARDs, clinical disease activity (DAS-28 ESR), disease duration, and inflammatory markers were the most important predictors of a flare. CONCLUSION Machine learning methods were deemed feasible to predict flares after tapering bDMARDs in RA patients in sustained remission.
Collapse
Affiliation(s)
| | - Koray Tascilar
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany
| | - Fabian Hartmann
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany
| | - Michaela Reiser
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany
| | - Axel J Hueber
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany.,Section Rheumatology, Sozialstiftung Bamberg, 96049, Bamberg, Germany
| | - Judith Haschka
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany.,Vinforce Study Group, St. Vincent Hospital, Medical University of Vienna, 1090, Vienna, Austria
| | - Sara Bayat
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany
| | - Timo Meinderink
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany
| | - Johannes Knitza
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany
| | - Larissa Mendez
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany
| | - Melanie Hagen
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany
| | - Gerhard Krönke
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany
| | - Jürgen Rech
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany
| | - Bernhard Manger
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany
| | - Arnd Kleyer
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany
| | | | - Georg Schett
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany.,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany
| | - David Simon
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054, Erlangen, Germany. .,Deutsches Zentrum fuer Immuntherapie (DZI), 91054, Erlangen, Germany.
| | | |
Collapse
|
652
|
Kuipers H, Hoogwater FJH, Holtman GA, Slangen JJG, de Haas RJ, de Boer MT. Diagnostic performance of preoperative CT in differentiating between benign and malignant origin of suspicious gallbladder lesions. Eur J Radiol 2021; 138:109619. [PMID: 33676357 DOI: 10.1016/j.ejrad.2021.109619] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2020] [Revised: 02/06/2021] [Accepted: 02/23/2021] [Indexed: 10/22/2022]
Abstract
PURPOSE To determine diagnostic performance of preoperative CT in differentiating between benign and malignant suspicious gallbladder lesions and to develop a preoperative risk score. METHOD All patients referred between January 2007 and September 2018 for suspicion of gallbladder cancer (GBC) or incidentally found GBC were retrospectively analyzed. Patients were excluded when preoperative CT or histopathologic examination was lacking. Two radiologists, blinded to histopathology results, independently reviewed CT images to differentiate benign disease from GBC. Multivariable analysis and internal validation were used to develop a risk score for GBC. Model discrimination, calibration, and diagnostic performance were assessed. RESULTS In total, 118 patients with 39 malignant (33 %) and 79 benign (67 %) lesions were included. Sensitivity of CT for diagnosing GBC was 90 % (95 % confidence interval [CI]: 76-97). Specificity rates were 61 % (95 % CI: 49-72) and 59 % (95 % CI: 48-70). Three predictors of GBC (irregular lesion aspect, absence of fat stranding, and locoregional lymphadenopathy) were included in the risk score ranging from -1 to 4. Adequate performance was found (AUC: 0.79, calibration slope: 0.89). In patients allocated >0 points, the model showed higher performance in excluding GBC than the radiologists (sensitivity 92 % [95 % CI: 79-98]). Moreover, when allocated >3 points, the risk score was superior in diagnosing GBC (specificity 99 % [95 % CI: 93-100]). CONCLUSIONS Sensitivity rates of CT for differentiation between benign and malignant gallbladder lesions are high, however specificity rates are relatively low. The proposed risk score may facilitate differentiation between benign and malignant suspicious gallbladder lesions.
Collapse
Affiliation(s)
- Hendrien Kuipers
- Department of Surgery, Section Hepato-Pancreato-Biliary Surgery and Liver Transplantation, University of Groningen, University Medical Center Groningen, Hanzeplein 1, 9713 GZ, Groningen, The Netherlands
| | - Frederik J H Hoogwater
- Department of Surgery, Section Hepato-Pancreato-Biliary Surgery and Liver Transplantation, University of Groningen, University Medical Center Groningen, Hanzeplein 1, 9713 GZ, Groningen, The Netherlands
| | - Gea A Holtman
- Department of General Practice and Elderly Care Medicine, University of Groningen, University Medical Center Groningen, Hanzeplein 1, 9713 GZ, Groningen, The Netherlands
| | - Jules J G Slangen
- Department of Radiology, University of Groningen, University Medical Center Groningen, Hanzeplein 1, 9713 GZ, Groningen, The Netherlands
| | - Robbert J de Haas
- Department of Radiology, University of Groningen, University Medical Center Groningen, Hanzeplein 1, 9713 GZ, Groningen, The Netherlands.
| | - Marieke T de Boer
- Department of Surgery, Section Hepato-Pancreato-Biliary Surgery and Liver Transplantation, University of Groningen, University Medical Center Groningen, Hanzeplein 1, 9713 GZ, Groningen, The Netherlands
| |
Collapse
|
653
|
Zanoletti E, Frigo AC, Franz L, Cazzador D, Franchella S, Favaretto N, Calvanese L, Nicolai P, Mazzoni A, Marioni G. Reply to Letter to the Editor regarding "Temporal bone carcinoma: Novel prognostic score based on clinical and histological features". Head Neck 2021; 43:1687-1689. [PMID: 33599342 DOI: 10.1002/hed.26650] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2021] [Accepted: 02/03/2021] [Indexed: 12/11/2022] Open
Affiliation(s)
- Elisabetta Zanoletti
- Section of Otolaryngology, Department of Neuroscience (DNS), Padova University, Padova, Italy
| | - Anna Chiara Frigo
- Department of Cardiac-Thoracic-Vascular Sciences and Public Health, Padova University, Padova, Italy
| | - Leonardo Franz
- Section of Otolaryngology, Department of Neuroscience (DNS), Padova University, Padova, Italy
| | - Diego Cazzador
- Section of Otolaryngology, Department of Neuroscience (DNS), Padova University, Padova, Italy.,Section of Human Anatomy, Department of Neuroscience (DNS), Padova University, Padova, Italy
| | - Sebastiano Franchella
- Section of Otolaryngology, Department of Neuroscience (DNS), Padova University, Padova, Italy.,Department of Women's and Children's Health, University of Padova, Padova, Italy
| | - Niccolò Favaretto
- Section of Otolaryngology, Department of Neuroscience (DNS), Padova University, Padova, Italy
| | - Leonardo Calvanese
- Section of Otolaryngology, Department of Neuroscience (DNS), Padova University, Padova, Italy
| | - Piero Nicolai
- Section of Otolaryngology, Department of Neuroscience (DNS), Padova University, Padova, Italy
| | - Antonio Mazzoni
- Section of Otolaryngology, Department of Neuroscience (DNS), Padova University, Padova, Italy
| | - Gino Marioni
- Section of Otolaryngology, Department of Neuroscience (DNS), Padova University, Padova, Italy
| |
Collapse
|
654
|
Nabuurs CH, Hannink G, Kunst HPM. Comment on "Temporal bone carcinoma: Novel prognostic score based on clinical and histological features". Head Neck 2021; 43:1683-1686. [PMID: 33599344 DOI: 10.1002/hed.26649] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2020] [Accepted: 02/03/2021] [Indexed: 12/23/2022] Open
Affiliation(s)
- Cindy H Nabuurs
- Department of Otorhinolaryngology - Head and Neck Surgery, Academic Alliance Skull Base Pathology Radboudumc - MUMC+, Radboud University Medical Center, Nijmegen, the Netherlands.,Rare Cancers, Radboud Institute for Health Sciences, Nijmegen, the Netherlands
| | - Gerjon Hannink
- Department of Operating Rooms, Radboud University Nijmegen Medical Center, Nijmegen, the Netherlands
| | - Henricus P M Kunst
- Department of Otorhinolaryngology - Head and Neck Surgery, Academic Alliance Skull Base Pathology Radboudumc - MUMC+, Radboud University Medical Center, Nijmegen, the Netherlands.,Rare Cancers, Radboud Institute for Health Sciences, Nijmegen, the Netherlands.,Department of Otorhinolaryngology - Head and Neck Surgery, Academic Alliance Skull Base Pathology Radboudumc - MUMC+, Maastricht University Medical Center, Maastricht, the Netherlands
| |
Collapse
|
655
|
Schwab P, Mehrjou A, Parbhoo S, Celi LA, Hetzel J, Hofer M, Schölkopf B, Bauer S. Real-time prediction of COVID-19 related mortality using electronic health records. Nat Commun 2021; 12:1058. [PMID: 33594046 PMCID: PMC7886884 DOI: 10.1038/s41467-020-20816-7] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2020] [Accepted: 12/15/2020] [Indexed: 12/15/2022] Open
Abstract
Coronavirus disease 2019 (COVID-19) is a respiratory disease with rapid human-to-human transmission caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Due to the exponential growth of infections, identifying patients with the highest mortality risk early is critical to enable effective intervention and prioritisation of care. Here, we present the COVID-19 early warning system (CovEWS), a risk scoring system for assessing COVID-19 related mortality risk that we developed using data amounting to a total of over 2863 years of observation time from a cohort of 66 430 patients seen at over 69 healthcare institutions. On an external cohort of 5005 patients, CovEWS predicts mortality from 78.8% (95% confidence interval [CI]: 76.0, 84.7%) to 69.4% (95% CI: 57.6, 75.2%) specificity at sensitivities greater than 95% between, respectively, 1 and 192 h prior to mortality events. CovEWS could enable earlier intervention, and may therefore help in preventing or mitigating COVID-19 related mortality.
Collapse
Affiliation(s)
| | - Arash Mehrjou
- Max Planck Institute for Intelligent Systems, Tübingen, Germany
- ETH Zurich, Zurich, Switzerland
| | - Sonali Parbhoo
- John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, USA
| | - Leo Anthony Celi
- Department of Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, USA
- MIT Critical Data, Laboratory for Computational Physiology, Institute for Medical Engineering and Science, Harvard-MIT Health Sciences and Technology, Cambridge, USA
| | - Jürgen Hetzel
- Department of Medical Oncology and Pneumology, University Hospital of Tübingen, Tübingen, Germany
- Department of Pneumology, Kantonsspital Winterthur, Winterthur, Switzerland
| | - Markus Hofer
- Department of Pneumology, Kantonsspital Winterthur, Winterthur, Switzerland
| | - Bernhard Schölkopf
- Max Planck Institute for Intelligent Systems, Tübingen, Germany
- ETH Zurich, Zurich, Switzerland
| | - Stefan Bauer
- Max Planck Institute for Intelligent Systems, Tübingen, Germany
- CIFAR Azrieli Global Scholar, Toronto, Canada
| |
Collapse
|
656
|
Clift AK, Le Lannou E, Tighe CP, Shah SS, Beatty M, Hyvärinen A, Lane SJ, Strauss T, Dunn DD, Lu J, Aral M, Vahdat D, Ponzo S, Plans D. Development and Validation of Risk Scores for All-Cause Mortality for a Smartphone-Based "General Health Score" App: Prospective Cohort Study Using the UK Biobank. JMIR Mhealth Uhealth 2021; 9:e25655. [PMID: 33591285 PMCID: PMC7925156 DOI: 10.2196/25655] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2020] [Revised: 12/16/2020] [Accepted: 01/20/2021] [Indexed: 12/23/2022] Open
Abstract
Background Given the established links between an individual’s behaviors and lifestyle factors and potentially adverse health outcomes, univariate or simple multivariate health metrics and scores have been developed to quantify general health at a given point in time and estimate risk of negative future outcomes. However, these health metrics may be challenging for widespread use and are unlikely to be successful at capturing the broader determinants of health in the general population. Hence, there is a need for a multidimensional yet widely employable and accessible way to obtain a comprehensive health metric. Objective The objective of the study was to develop and validate a novel, easily interpretable, points-based health score (“C-Score”) derived from metrics measurable using smartphone components and iterations thereof that utilize statistical modeling and machine learning (ML) approaches. Methods A literature review was conducted to identify relevant predictor variables for inclusion in the first iteration of a points-based model. This was followed by a prospective cohort study in a UK Biobank population for the purposes of validating the C-Score and developing and comparatively validating variations of the score using statistical and ML models to assess the balance between expediency and ease of interpretability and model complexity. Primary and secondary outcome measures were discrimination of a points-based score for all-cause mortality within 10 years (Harrell c-statistic) and discrimination and calibration of Cox proportional hazards models and ML models that incorporate C-Score values (or raw data inputs) and other predictors to predict the risk of all-cause mortality within 10 years. Results The study cohort comprised 420,560 individuals. During a cohort follow-up of 4,526,452 person-years, there were 16,188 deaths from any cause (3.85%). The points-based model had good discrimination (c-statistic=0.66). There was a 31% relative reduction in risk of all-cause mortality per decile of increasing C-Score (hazard ratio of 0.69, 95% CI 0.663-0.675). A Cox model integrating age and C-Score had improved discrimination (8 percentage points; c-statistic=0.74) and good calibration. ML approaches did not offer improved discrimination over statistical modeling. Conclusions The novel health metric (“C-Score”) has good predictive capabilities for all-cause mortality within 10 years. Embedding the C-Score within a smartphone app may represent a useful tool for democratized, individualized health risk prediction. A simple Cox model using C-Score and age balances parsimony and accuracy of risk predictions and could be used to produce absolute risk estimations for app users.
Collapse
Affiliation(s)
- Ashley K Clift
- Department of Surgery and Cancer, Imperial College London, London, United Kingdom
| | | | - Christian P Tighe
- Department of Surgery and Cancer, Imperial College London, London, United Kingdom.,Huma Therapeutics, London, United Kingdom
| | | | | | | | | | | | | | - Jiahe Lu
- Huma Therapeutics, London, United Kingdom
| | - Mert Aral
- Huma Therapeutics, London, United Kingdom
| | - Dan Vahdat
- Huma Therapeutics, London, United Kingdom
| | | | - David Plans
- Huma Therapeutics, London, United Kingdom.,Department of Science, Innovation, Technology, and Entrepreneurship, University of Exeter, Exeter, United Kingdom.,Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom
| |
Collapse
|
657
|
Mamprin M, Zelis JM, Tonino PAL, Zinger S, de With PHN. Decision Trees for Predicting Mortality in Transcatheter Aortic Valve Implantation. Bioengineering (Basel) 2021; 8:bioengineering8020022. [PMID: 33572063 PMCID: PMC7915084 DOI: 10.3390/bioengineering8020022] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2020] [Revised: 01/27/2021] [Accepted: 02/04/2021] [Indexed: 12/23/2022] Open
Abstract
Current prognostic risk scores in cardiac surgery do not benefit yet from machine learning (ML). This research aims to create a machine learning model to predict one-year mortality of a patient after transcatheter aortic valve implantation (TAVI). We adopt a modern gradient boosting on decision trees classifier (GBDTs), specifically designed for categorical features. In combination with a recent technique for model interpretations, we developed a feature analysis and selection stage, enabling the identification of the most important features for the prediction. We base our prediction model on the most relevant features, after interpreting and discussing the feature analysis results with clinical experts. We validated our model on 270 consecutive TAVI cases, reaching a C-statistic of 0.83 with CI [0.82, 0.84]. The model has achieved a positive predictive value ranging from 57% to 64%, suggesting that the patient selection made by the heart team of professionals can be further improved by taking into consideration the clinical data we identified as important and by exploiting ML approaches in the development of clinical risk scores. Our approach has shown promising predictive potential also with respect to widespread prognostic risk scores, such as logistic European system for cardiac operative risk evaluation (EuroSCORE II) and the society of thoracic surgeons (STS) risk score, which are broadly adopted by cardiologists worldwide.
Collapse
Affiliation(s)
- Marco Mamprin
- Department of Electrical Engineering, Eindhoven University of Technology, 5612 AJ Eindhoven, The Netherlands; (S.Z.); (P.H.N.d.W.)
- Correspondence:
| | - Jo M. Zelis
- Department of Cardiology, Catharina Hospital, 5623 EJ Eindhoven, The Netherlands; (J.M.Z.); (P.A.L.T.)
| | - Pim A. L. Tonino
- Department of Cardiology, Catharina Hospital, 5623 EJ Eindhoven, The Netherlands; (J.M.Z.); (P.A.L.T.)
| | - Sveta Zinger
- Department of Electrical Engineering, Eindhoven University of Technology, 5612 AJ Eindhoven, The Netherlands; (S.Z.); (P.H.N.d.W.)
| | - Peter H. N. de With
- Department of Electrical Engineering, Eindhoven University of Technology, 5612 AJ Eindhoven, The Netherlands; (S.Z.); (P.H.N.d.W.)
| |
Collapse
|
658
|
Ali I, Kalra PA. A validation study of the 4-variable and 8-variable kidney failure risk equation in transplant recipients in the United Kingdom. BMC Nephrol 2021; 22:57. [PMID: 33563222 PMCID: PMC7874608 DOI: 10.1186/s12882-021-02259-4] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2020] [Accepted: 12/31/2020] [Indexed: 12/23/2022] Open
Abstract
Background There is emerging evidence that the 4-variable Kidney Failure Risk Equation (KFRE) can be used for risk prediction of graft failure in transplant recipients. However, geographical validation of the 4-variable KFRE in transplant patients is lacking, as is whether the more extensive 8-variable KFRE improves predictive accuracy. This study aimed to validate the 4- and 8-variable KFRE predictions of the 5-year death-censored risk of graft failure in patients in the United Kingdom. Methods A retrospective cohort study involved 415 transplant recipients who had their first renal transplant between 2003 and 2015 and were under follow-up at Salford Royal NHS Foundation Trust. The KFRE risk scores were calculated on variables taken 1-year post-transplant. The area under the receiver operating characteristic curves (AUC) and calibration plots were evaluated to determine discrimination and calibration of the 4- and 8-variable KFREs in the whole cohort as well as in a subgroup analysis of living and deceased donor recipients and in patients with an eGFR< 45 ml/min/1.73m2. Results There were 16 graft failure events (4%) in the whole cohort. The 4- and 8-variable KFREs showed good discrimination with AUC of 0.743 (95% confidence interval [CI] 0.610–0.876) and 0.751 (95% CI 0.629–0.872) respectively. In patients with an eGFR< 45 ml/min/1.73m2, the 8-variable KFRE had good discrimination with an AUC of 0.785 (95% CI 0.558–0.982) but the 4-variable provided excellent discrimination in this group with an AUC of 0.817 (0.646–0.988). Calibration plots however showed poor calibration with risk scores tending to underestimate risk of graft failure in low-risk patients and overestimate risk in high-risk patients, which was seen in the primary and subgroup analyses. Conclusions Despite adequate discrimination, the 4- and 8-variable KFREs are imprecise in predicting graft failure in transplant recipients using data 1-year post-transplant. Larger, international studies involving diverse patient populations should be considered to corroborate these findings. Supplementary Information The online version contains supplementary material available at 10.1186/s12882-021-02259-4.
Collapse
Affiliation(s)
- Ibrahim Ali
- Department of Renal Medicine, Salford Royal NHS Foundation Trust, Stott Lane, Salford, M6 8HD, UK. .,Division of Cardiovascular Sciences, University of Manchester, Manchester, M13 9PL, UK.
| | - Philip A Kalra
- Department of Renal Medicine, Salford Royal NHS Foundation Trust, Stott Lane, Salford, M6 8HD, UK.,Division of Cardiovascular Sciences, University of Manchester, Manchester, M13 9PL, UK
| |
Collapse
|
659
|
Taylor RA, Haimovich AD. Machine Learning in Emergency Medicine: Keys to Future Success. Acad Emerg Med 2021; 28:263-267. [PMID: 33277733 DOI: 10.1111/acem.14189] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Affiliation(s)
- R. Andrew Taylor
- Yale Department of Emergency Medicine Yale School of Medicine New Haven CTUSA
| | - Adrian D. Haimovich
- Yale Department of Emergency Medicine Yale School of Medicine New Haven CTUSA
| |
Collapse
|
660
|
Lassau N, Ammari S, Chouzenoux E, Gortais H, Herent P, Devilder M, Soliman S, Meyrignac O, Talabard MP, Lamarque JP, Dubois R, Loiseau N, Trichelair P, Bendjebbar E, Garcia G, Balleyguier C, Merad M, Stoclin A, Jegou S, Griscelli F, Tetelboum N, Li Y, Verma S, Terris M, Dardouri T, Gupta K, Neacsu A, Chemouni F, Sefta M, Jehanno P, Bousaid I, Boursin Y, Planchet E, Azoulay M, Dachary J, Brulport F, Gonzalez A, Dehaene O, Schiratti JB, Schutte K, Pesquet JC, Talbot H, Pronier E, Wainrib G, Clozel T, Barlesi F, Bellin MF, Blum MGB. Integrating deep learning CT-scan model, biological and clinical variables to predict severity of COVID-19 patients. Nat Commun 2021; 12:634. [PMID: 33504775 PMCID: PMC7840774 DOI: 10.1038/s41467-020-20657-4] [Citation(s) in RCA: 97] [Impact Index Per Article: 32.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2020] [Accepted: 12/09/2020] [Indexed: 12/11/2022] Open
Abstract
The SARS-COV-2 pandemic has put pressure on intensive care units, so that identifying predictors of disease severity is a priority. We collect 58 clinical and biological variables, and chest CT scan data, from 1003 coronavirus-infected patients from two French hospitals. We train a deep learning model based on CT scans to predict severity. We then construct the multimodal AI-severity score that includes 5 clinical and biological variables (age, sex, oxygenation, urea, platelet) in addition to the deep learning model. We show that neural network analysis of CT-scans brings unique prognosis information, although it is correlated with other markers of severity (oxygenation, LDH, and CRP) explaining the measurable but limited 0.03 increase of AUC obtained when adding CT-scan information to clinical variables. Here, we show that when comparing AI-severity with 11 existing severity scores, we find significantly improved prognosis performance; AI-severity can therefore rapidly become a reference scoring approach.
Collapse
Affiliation(s)
- Nathalie Lassau
- Imaging Department, Gustave Roussy, Université Paris -Saclay, Villejuif, 94805, France
- Biomaps, UMR 1281 INSERM, CEA, CNRS, Université Paris-Saclay, Villejuif, 94805, France
| | - Samy Ammari
- Imaging Department, Gustave Roussy, Université Paris -Saclay, Villejuif, 94805, France
- Biomaps, UMR 1281 INSERM, CEA, CNRS, Université Paris-Saclay, Villejuif, 94805, France
| | - Emilie Chouzenoux
- Centre de Vision Numérique, Université Paris-Saclay, CentraleSupélec, Inria, 91190, Gif-sur-Yvette, France
| | - Hugo Gortais
- Radiology Department, Hôpital de Bicêtre - AP-HP, Université Paris-Saclay, Le Kremlin-Bicêtre, France
| | | | - Matthieu Devilder
- Radiology Department, Hôpital de Bicêtre - AP-HP, Université Paris-Saclay, Le Kremlin-Bicêtre, France
| | - Samer Soliman
- Radiology Department, Hôpital de Bicêtre - AP-HP, Université Paris-Saclay, Le Kremlin-Bicêtre, France
| | - Olivier Meyrignac
- Radiology Department, Hôpital de Bicêtre - AP-HP, Université Paris-Saclay, Le Kremlin-Bicêtre, France
| | - Marie-Pauline Talabard
- Radiology Department, Hôpital de Bicêtre - AP-HP, Université Paris-Saclay, Le Kremlin-Bicêtre, France
| | - Jean-Philippe Lamarque
- Imaging Department, Gustave Roussy, Université Paris -Saclay, Villejuif, 94805, France
- Biomaps, UMR 1281 INSERM, CEA, CNRS, Université Paris-Saclay, Villejuif, 94805, France
| | | | | | | | | | - Gabriel Garcia
- Imaging Department, Gustave Roussy, Université Paris -Saclay, Villejuif, 94805, France
| | - Corinne Balleyguier
- Imaging Department, Gustave Roussy, Université Paris -Saclay, Villejuif, 94805, France
- Biomaps, UMR 1281 INSERM, CEA, CNRS, Université Paris-Saclay, Villejuif, 94805, France
| | - Mansouria Merad
- Département Interdisciplinaire d'Organisation des Parcours Patients, Gustave Roussy, Université Paris-Saclay, Villejuif, 94805, France
| | - Annabelle Stoclin
- Département Interdisciplinaire d'Organisation des Parcours Patients, Gustave Roussy, Université Paris-Saclay, Villejuif, 94805, France
| | | | - Franck Griscelli
- Département de Biologie, Gustave Roussy, Université Paris-Saclay, Villejuif, 94805, France
| | - Nicolas Tetelboum
- Imaging Department, Gustave Roussy, Université Paris -Saclay, Villejuif, 94805, France
| | - Yingping Li
- Biomaps, UMR 1281 INSERM, CEA, CNRS, Université Paris-Saclay, Villejuif, 94805, France
- Centre de Vision Numérique, Université Paris-Saclay, CentraleSupélec, Inria, 91190, Gif-sur-Yvette, France
| | - Sagar Verma
- Centre de Vision Numérique, Université Paris-Saclay, CentraleSupélec, Inria, 91190, Gif-sur-Yvette, France
| | - Matthieu Terris
- Centre de Vision Numérique, Université Paris-Saclay, CentraleSupélec, Inria, 91190, Gif-sur-Yvette, France
| | - Tasnim Dardouri
- Centre de Vision Numérique, Université Paris-Saclay, CentraleSupélec, Inria, 91190, Gif-sur-Yvette, France
| | - Kavya Gupta
- Centre de Vision Numérique, Université Paris-Saclay, CentraleSupélec, Inria, 91190, Gif-sur-Yvette, France
| | - Ana Neacsu
- Centre de Vision Numérique, Université Paris-Saclay, CentraleSupélec, Inria, 91190, Gif-sur-Yvette, France
| | - Frank Chemouni
- Département Interdisciplinaire d'Organisation des Parcours Patients, Gustave Roussy, Université Paris-Saclay, Villejuif, 94805, France
| | | | | | - Imad Bousaid
- Direction de la Transformation Numérique et des Systèmes d'Information, Gustave Roussy, Université Paris-Saclay, 94805, Villejuif, France
| | - Yannick Boursin
- Direction de la Transformation Numérique et des Systèmes d'Information, Gustave Roussy, Université Paris-Saclay, 94805, Villejuif, France
| | - Emmanuel Planchet
- Direction de la Transformation Numérique et des Systèmes d'Information, Gustave Roussy, Université Paris-Saclay, 94805, Villejuif, France
| | - Mikael Azoulay
- Direction de la Transformation Numérique et des Systèmes d'Information, Gustave Roussy, Université Paris-Saclay, 94805, Villejuif, France
| | | | | | | | | | | | | | - Jean-Christophe Pesquet
- Centre de Vision Numérique, Université Paris-Saclay, CentraleSupélec, Inria, 91190, Gif-sur-Yvette, France
| | - Hugues Talbot
- Centre de Vision Numérique, Université Paris-Saclay, CentraleSupélec, Inria, 91190, Gif-sur-Yvette, France
| | | | | | | | - Fabrice Barlesi
- Département d'Oncologie Médicale, Gustave Roussy, Université Paris-Saclay, Villejuif, 94805, France
| | - Marie-France Bellin
- Radiology Department, Hôpital de Bicêtre - AP-HP, Université Paris-Saclay, Le Kremlin-Bicêtre, France
| | | |
Collapse
|
661
|
Benevenuta S, Capriotti E, Fariselli P. Calibrating variant-scoring methods for clinical decision making. Bioinformatics 2021; 36:5709-5711. [PMID: 33492342 PMCID: PMC8023678 DOI: 10.1093/bioinformatics/btaa943] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2020] [Revised: 09/27/2020] [Accepted: 10/28/2020] [Indexed: 12/22/2022] Open
Abstract
Summary Identifying pathogenic variants and annotating them is a major challenge in human genetics, especially for the non-coding ones. Several tools have been developed and used to predict the functional effect of genetic variants. However, the calibration assessment of the predictions has received little attention. Calibration refers to the idea that if a model predicts a group of variants to be pathogenic with a probability P, it is expected that the same fraction P of true positive is found in the observed set. For instance, a well-calibrated classifier should label the variants such that among the ones to which it gave a probability value close to 0.7, approximately 70% actually belong to the pathogenic class. Poorly calibrated algorithms can be misleading and potentially harmful for clinical decision making. Avaliability and implementation The dataset used for testing the methods is available through the DOI:10.5281/zenodo.4448197. Supplementary information Supplementary data are available at Bioinformatics online.
Collapse
Affiliation(s)
- Silvia Benevenuta
- Department of Medical Sciences, University of Torino, Via Santena, 19, 10126, Torino, Italy
| | - Emidio Capriotti
- BioFolD Unit, Department of Pharmacy and Biotechnology (FaBiT), University of Bologna, Via Selmi 3, 40126, Bologna, Italy
| | - Piero Fariselli
- Department of Medical Sciences, University of Torino, Via Santena, 19, 10126, Torino, Italy
| |
Collapse
|
662
|
Harrison JH, Gilbertson JR, Hanna MG, Olson NH, Seheult JN, Sorace JM, Stram MN. Introduction to Artificial Intelligence and Machine Learning for Pathology. Arch Pathol Lab Med 2021; 145:1228-1254. [PMID: 33493264 DOI: 10.5858/arpa.2020-0541-cp] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/10/2020] [Indexed: 11/06/2022]
Abstract
CONTEXT.— Recent developments in machine learning have stimulated intense interest in software that may augment or replace human experts. Machine learning may impact pathology practice by offering new capabilities in analysis, interpretation, and outcomes prediction using images and other data. The principles of operation and management of machine learning systems are unfamiliar to pathologists, who anticipate a need for additional education to be effective as expert users and managers of the new tools. OBJECTIVE.— To provide a background on machine learning for practicing pathologists, including an overview of algorithms, model development, and performance evaluation; to examine the current status of machine learning in pathology and consider possible roles and requirements for pathologists in local deployment and management of machine learning systems; and to highlight existing challenges and gaps in deployment methodology and regulation. DATA SOURCES.— Sources include the biomedical and engineering literature, white papers from professional organizations, government reports, electronic resources, and authors' experience in machine learning. References were chosen when possible for accessibility to practicing pathologists without specialized training in mathematics, statistics, or software development. CONCLUSIONS.— Machine learning offers an array of techniques that in recent published results show substantial promise. Data suggest that human experts working with machine learning tools outperform humans or machines separately, but the optimal form for this combination in pathology has not been established. Significant questions related to the generalizability of machine learning systems, local site verification, and performance monitoring remain to be resolved before a consensus on best practices and a regulatory environment can be established.
Collapse
Affiliation(s)
- James H Harrison
- From the Department of Pathology, University of Virginia School of Medicine, Charlottesville (Harrison)
| | - John R Gilbertson
- the Departments of Biomedical Informatics and Pathology, University of Pittsburgh, Pittsburgh, Pennsylvania (Gilbertson)
| | - Matthew G Hanna
- the Department of Pathology, Memorial Sloan Kettering Cancer Center, New York, New York (Hanna)
| | - Niels H Olson
- the Defense Innovation Unit, Mountain View, California (Olson).,the Department of Pathology, Uniformed Services University, Bethesda, Maryland (Olson)
| | - Jansen N Seheult
- the Department of Pathology, University of Pittsburgh, and Vitalant Specialty Labs, Pittsburgh, Pennsylvania (Seheult)
| | - James M Sorace
- the US Department of Health and Human Services, retired, Lutherville, Maryland (Sorace)
| | - Michelle N Stram
- the Department of Forensic Medicine, New York University, and Office of Chief Medical Examiner, New York, New York (Stram)
| |
Collapse
|
663
|
Carr E, Bendayan R, Bean D, Stammers M, Wang W, Zhang H, Searle T, Kraljevic Z, Shek A, Phan HTT, Muruet W, Gupta RK, Shinton AJ, Wyatt M, Shi T, Zhang X, Pickles A, Stahl D, Zakeri R, Noursadeghi M, O'Gallagher K, Rogers M, Folarin A, Karwath A, Wickstrøm KE, Köhn-Luque A, Slater L, Cardoso VR, Bourdeaux C, Holten AR, Ball S, McWilliams C, Roguski L, Borca F, Batchelor J, Amundsen EK, Wu X, Gkoutos GV, Sun J, Pinto A, Guthrie B, Breen C, Douiri A, Wu H, Curcin V, Teo JT, Shah AM, Dobson RJB. Evaluation and improvement of the National Early Warning Score (NEWS2) for COVID-19: a multi-hospital study. BMC Med 2021; 19:23. [PMID: 33472631 PMCID: PMC7817348 DOI: 10.1186/s12916-020-01893-3] [Citation(s) in RCA: 60] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/29/2020] [Accepted: 12/16/2020] [Indexed: 01/08/2023] Open
Abstract
BACKGROUND The National Early Warning Score (NEWS2) is currently recommended in the UK for the risk stratification of COVID-19 patients, but little is known about its ability to detect severe cases. We aimed to evaluate NEWS2 for the prediction of severe COVID-19 outcome and identify and validate a set of blood and physiological parameters routinely collected at hospital admission to improve upon the use of NEWS2 alone for medium-term risk stratification. METHODS Training cohorts comprised 1276 patients admitted to King's College Hospital National Health Service (NHS) Foundation Trust with COVID-19 disease from 1 March to 30 April 2020. External validation cohorts included 6237 patients from five UK NHS Trusts (Guy's and St Thomas' Hospitals, University Hospitals Southampton, University Hospitals Bristol and Weston NHS Foundation Trust, University College London Hospitals, University Hospitals Birmingham), one hospital in Norway (Oslo University Hospital), and two hospitals in Wuhan, China (Wuhan Sixth Hospital and Taikang Tongji Hospital). The outcome was severe COVID-19 disease (transfer to intensive care unit (ICU) or death) at 14 days after hospital admission. Age, physiological measures, blood biomarkers, sex, ethnicity, and comorbidities (hypertension, diabetes, cardiovascular, respiratory and kidney diseases) measured at hospital admission were considered in the models. RESULTS A baseline model of 'NEWS2 + age' had poor-to-moderate discrimination for severe COVID-19 infection at 14 days (area under receiver operating characteristic curve (AUC) in training cohort = 0.700, 95% confidence interval (CI) 0.680, 0.722; Brier score = 0.192, 95% CI 0.186, 0.197). A supplemented model adding eight routinely collected blood and physiological parameters (supplemental oxygen flow rate, urea, age, oxygen saturation, C-reactive protein, estimated glomerular filtration rate, neutrophil count, neutrophil/lymphocyte ratio) improved discrimination (AUC = 0.735; 95% CI 0.715, 0.757), and these improvements were replicated across seven UK and non-UK sites. However, there was evidence of miscalibration with the model tending to underestimate risks in most sites. CONCLUSIONS NEWS2 score had poor-to-moderate discrimination for medium-term COVID-19 outcome which raises questions about its use as a screening tool at hospital admission. Risk stratification was improved by including readily available blood and physiological parameters measured at hospital admission, but there was evidence of miscalibration in external sites. This highlights the need for a better understanding of the use of early warning scores for COVID.
Collapse
Affiliation(s)
- Ewan Carr
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience (IoPPN), King's College London, 16 De Crespigny Park, London, SE5 8AF, UK.
| | - Rebecca Bendayan
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience (IoPPN), King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
- NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King's College London, London, UK
| | - Daniel Bean
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience (IoPPN), King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
- Health Data Research UK London, University College London, London, UK
| | - Matt Stammers
- Clinical Informatics Research Unit, University of Southampton, Coxford Rd., Southampton, SO16 5AF, UK
- NIHR Biomedical Research Centre at University Hospital Southampton NHS Trust, Coxford Road, Southampton, UK
- UHS Digital, University Hospital Southampton, Tremona Road, Southampton, SO16 6YD, UK
| | - Wenjuan Wang
- School of Population Health and Environmental Sciences, King's College London, London, UK
| | - Huayu Zhang
- Usher Institute, University of Edinburgh, Edinburgh, UK
| | - Thomas Searle
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience (IoPPN), King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
- NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King's College London, London, UK
| | - Zeljko Kraljevic
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience (IoPPN), King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
| | - Anthony Shek
- Department of Clinical Neuroscience, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
| | - Hang T T Phan
- Clinical Informatics Research Unit, University of Southampton, Coxford Rd., Southampton, SO16 5AF, UK
- NIHR Biomedical Research Centre at University Hospital Southampton NHS Trust, Coxford Road, Southampton, UK
| | - Walter Muruet
- School of Population Health and Environmental Sciences, King's College London, London, UK
| | - Rishi K Gupta
- UCL Institute for Global Health, University College London Hospitals NHS Trust, London, UK
| | - Anthony J Shinton
- UHS Digital, University Hospital Southampton, Tremona Road, Southampton, SO16 6YD, UK
| | - Mike Wyatt
- University Hospitals Bristol and Weston NHS Foundation Trust, Bristol, UK
| | - Ting Shi
- Usher Institute, University of Edinburgh, Edinburgh, UK
| | - Xin Zhang
- Department of Pulmonary and Critical Care Medicine, People's Liberation Army Joint Logistic Support Force 920th Hospital, Kunming, Yunnan, China
| | - Andrew Pickles
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience (IoPPN), King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
- NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King's College London, London, UK
| | - Daniel Stahl
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience (IoPPN), King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
| | - Rosita Zakeri
- King's College Hospital NHS Foundation Trust, London, UK
- School of Cardiovascular Medicine & Sciences, King's College London British Heart Foundation Centre of Excellence, London, SE5 9NU, UK
| | - Mahdad Noursadeghi
- UCL Division of Infection and Immunity, University College London Hospitals NHS Trust, London, UK
| | - Kevin O'Gallagher
- King's College Hospital NHS Foundation Trust, London, UK
- School of Cardiovascular Medicine & Sciences, King's College London British Heart Foundation Centre of Excellence, London, SE5 9NU, UK
| | - Matt Rogers
- University Hospitals Bristol and Weston NHS Foundation Trust, Bristol, UK
| | - Amos Folarin
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience (IoPPN), King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
- Health Data Research UK London, University College London, London, UK
- Institute of Health Informatics, University College London, London, UK
- NIHR Biomedical Research Centre at University College London Hospitals NHS Foundation Trust, London, UK
| | - Andreas Karwath
- College of Medical and Dental Sciences, Institute of Cancer and Genomics, University of Birmingham, Birmingham, UK
- Institute of Translational Medicine, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Health Data Research UK Midlands, Birmingham, UK
| | - Kristin E Wickstrøm
- Department of Medical Biochemistry, Blood Cell Research Group, Oslo University Hospital, Oslo, Norway
| | - Alvaro Köhn-Luque
- Oslo Centre for Biostatistics and Epidemiology, Faculty of Medicine, University of Oslo, Oslo, Norway
| | - Luke Slater
- College of Medical and Dental Sciences, Institute of Cancer and Genomics, University of Birmingham, Birmingham, UK
- Institute of Translational Medicine, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Health Data Research UK Midlands, Birmingham, UK
| | - Victor Roth Cardoso
- College of Medical and Dental Sciences, Institute of Cancer and Genomics, University of Birmingham, Birmingham, UK
- Institute of Translational Medicine, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Health Data Research UK Midlands, Birmingham, UK
| | | | - Aleksander Rygh Holten
- Department of Acute Medicine, Oslo University Hospital and Institute of Clinical Medicine, University of Oslo, Oslo, Norway
| | - Simon Ball
- Health Data Research UK Midlands, Birmingham, UK
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
| | - Chris McWilliams
- Department of Engineering Mathematics, University of Bristol, Bristol, UK
| | - Lukasz Roguski
- Health Data Research UK London, University College London, London, UK
- Institute of Health Informatics, University College London, London, UK
- Institute of Translational Medicine, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
| | - Florina Borca
- Clinical Informatics Research Unit, University of Southampton, Coxford Rd., Southampton, SO16 5AF, UK
- NIHR Biomedical Research Centre at University Hospital Southampton NHS Trust, Coxford Road, Southampton, UK
- UHS Digital, University Hospital Southampton, Tremona Road, Southampton, SO16 6YD, UK
| | - James Batchelor
- Clinical Informatics Research Unit, University of Southampton, Coxford Rd., Southampton, SO16 5AF, UK
| | - Erik Koldberg Amundsen
- Department of Medical Biochemistry, Blood Cell Research Group, Oslo University Hospital, Oslo, Norway
| | - Xiaodong Wu
- Department of Pulmonary and Critical Care Medicine, Shanghai East Hospital, Tongji University, Shanghai, China
- Department of Pulmonary and Critical Care Medicine, Taikang Tongji Hospital, Wuhan, China
| | - Georgios V Gkoutos
- College of Medical and Dental Sciences, Institute of Cancer and Genomics, University of Birmingham, Birmingham, UK
- Institute of Translational Medicine, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Health Data Research UK Midlands, Birmingham, UK
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
| | - Jiaxing Sun
- Department of Pulmonary and Critical Care Medicine, Shanghai East Hospital, Tongji University, Shanghai, China
| | - Ashwin Pinto
- UHS Digital, University Hospital Southampton, Tremona Road, Southampton, SO16 6YD, UK
| | - Bruce Guthrie
- Usher Institute, University of Edinburgh, Edinburgh, UK
| | - Cormac Breen
- School of Population Health and Environmental Sciences, King's College London, London, UK
| | - Abdel Douiri
- School of Population Health and Environmental Sciences, King's College London, London, UK
| | - Honghan Wu
- Health Data Research UK London, University College London, London, UK
- Institute of Health Informatics, University College London, London, UK
| | - Vasa Curcin
- School of Population Health and Environmental Sciences, King's College London, London, UK
| | - James T Teo
- Department of Clinical Neuroscience, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
- King's College Hospital NHS Foundation Trust, London, UK
| | - Ajay M Shah
- King's College Hospital NHS Foundation Trust, London, UK
- School of Cardiovascular Medicine & Sciences, King's College London British Heart Foundation Centre of Excellence, London, SE5 9NU, UK
| | - Richard J B Dobson
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience (IoPPN), King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
- NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King's College London, London, UK
- Health Data Research UK London, University College London, London, UK
- Institute of Health Informatics, University College London, London, UK
- NIHR Biomedical Research Centre at University College London Hospitals NHS Foundation Trust, London, UK
| |
Collapse
|
664
|
Archer L, Snell KIE, Ensor J, Hudda MT, Collins GS, Riley RD. Minimum sample size for external validation of a clinical prediction model with a continuous outcome. Stat Med 2021; 40:133-146. [PMID: 33150684 DOI: 10.1002/sim.8766] [Citation(s) in RCA: 40] [Impact Index Per Article: 13.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2020] [Revised: 08/06/2020] [Accepted: 09/11/2020] [Indexed: 01/12/2023]
Abstract
Clinical prediction models provide individualized outcome predictions to inform patient counseling and clinical decision making. External validation is the process of examining a prediction model's performance in data independent to that used for model development. Current external validation studies often suffer from small sample sizes, and subsequently imprecise estimates of a model's predictive performance. To address this, we propose how to determine the minimum sample size needed for external validation of a clinical prediction model with a continuous outcome. Four criteria are proposed, that target precise estimates of (i) R2 (the proportion of variance explained), (ii) calibration-in-the-large (agreement between predicted and observed outcome values on average), (iii) calibration slope (agreement between predicted and observed values across the range of predicted values), and (iv) the variance of observed outcome values. Closed-form sample size solutions are derived for each criterion, which require the user to specify anticipated values of the model's performance (in particular R2 ) and the outcome variance in the external validation dataset. A sensible starting point is to base values on those for the model development study, as obtained from the publication or study authors. The largest sample size required to meet all four criteria is the recommended minimum sample size needed in the external validation dataset. The calculations can also be applied to estimate expected precision when an existing dataset with a fixed sample size is available, to help gauge if it is adequate. We illustrate the proposed methods on a case-study predicting fat-free mass in children.
Collapse
Affiliation(s)
- Lucinda Archer
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| | - Kym I E Snell
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| | - Joie Ensor
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| | - Mohammed T Hudda
- Population Health Research Institute, St George's, University of London, London, UK
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Richard D Riley
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| |
Collapse
|
665
|
Oskar S, Stingone JA. Machine Learning Within Studies of Early-Life Environmental Exposures and Child Health: Review of the Current Literature and Discussion of Next Steps. Curr Environ Health Rep 2021; 7:170-184. [PMID: 32578067 DOI: 10.1007/s40572-020-00282-5] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
Abstract
PURPOSE OF REVIEW The goal of this article is to review the use of machine learning (ML) within studies of environmental exposures and children's health, identify common themes across studies, and provide recommendations to advance their use in research and practice. RECENT FINDINGS We identified 42 articles reporting upon the use of ML within studies of environmental exposures and children's health between 2017 and 2019. The common themes among the articles were analysis of mixture data, exposure prediction, disease prediction and forecasting, analysis of complex data, and causal inference. With the increasing complexity of environmental health data, we anticipate greater use of ML to address the challenges that cannot be handled by traditional analytics. In order for these methods to beneficially impact public health, the ML techniques we use need to be appropriate for our study questions, rigorously evaluated and reported in a way that can be critically assessed by the scientific community.
Collapse
Affiliation(s)
- Sabine Oskar
- Department of Epidemiology, Columbia University Mailman School of Public Health, 722 West 168th St, Room 1608, New York, NY, 10032, USA
| | - Jeanette A Stingone
- Department of Epidemiology, Columbia University Mailman School of Public Health, 722 West 168th St, Room 1608, New York, NY, 10032, USA.
| |
Collapse
|
666
|
Eyler Dang L, Hubbard A, Dissak-Delon FN, Chichom Mefire A, Juillard C. Right population, right resources, right algorithm: Using machine learning efficiently and effectively in surgical systems where data are a limited resource. Surgery 2021; 170:325-328. [PMID: 33413920 DOI: 10.1016/j.surg.2020.11.043] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2020] [Revised: 11/24/2020] [Accepted: 11/27/2020] [Indexed: 11/16/2022]
Abstract
There is a growing interest in using machine learning algorithms to support surgical care, diagnostics, and public health surveillance in low- and middle-income countries. From our own experience and the literature, we share several lessons for developing such models in settings where the data necessary for algorithm training and implementation is a limited resource. First, the training cohort should be as similar as possible to the population of interest, and recalibration can be used to improve risk estimates when a model is transported to a new context. Second, algorithms should incorporate existing data sources or data that is easily obtainable by frontline health workers or assistants in order to optimize available resources and facilitate integration into clinical practice. Third, the Super Learner ensemble machine learning algorithm can be used to define the optimal model for a given prediction problem while minimizing bias in the algorithm selection process. By considering the right population, right resources, and right algorithm, researchers can train prediction models that are both context-appropriate and resource-conscious. There remain gaps in data availability, affordable computing capacity, and implementation studies that hinder clinical algorithm development and use in low-resource settings, although these barriers are decreasing over time. We advocate for researchers to create open-source code, apps, and training materials to allow new machine learning models to be adapted to different populations and contexts in order to support surgical providers and health care systems in low- and middle-income countries worldwide.
Collapse
Affiliation(s)
- Lauren Eyler Dang
- University of California, Berkeley, School of Public Health, Division of Biostatistics, Berkeley, CA; University of California, San Francisco, Department of Surgery, San Francisco, CA
| | - Alan Hubbard
- University of California, Berkeley, School of Public Health, Division of Biostatistics, Berkeley, CA
| | | | - Alain Chichom Mefire
- University of Buea, Faculty of Health Sciences, Department of Surgery, Buea, Cameroon
| | - Catherine Juillard
- University of California, Los Angeles, Department of Surgery, Los Angeles, CA.
| |
Collapse
|
667
|
Fu B, Wei X, Wang Q, Yang Z, Chen J, Yu D. Use of the Thrombolysis in Myocardial Infarction Risk Index for Elderly Patients With ST-Segment Elevation Myocardial Infarction. Front Cardiovasc Med 2021; 8:743678. [PMID: 34869648 PMCID: PMC8639686 DOI: 10.3389/fcvm.2021.743678] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2021] [Accepted: 10/28/2021] [Indexed: 02/05/2023] Open
Abstract
Background: Thrombolysis in Myocardial Infarction (TIMI) Risk Index (TRI) is a simple risk assessment tool for patients with ST-segment elevation myocardial infarction (STEMI). However, its applicability to elderly patients with STEMI undergoing percutaneous coronary intervention (PCI) is uncertain. Methods: This was a retrospective analysis of elderly (≥60 years) patients who underwent PCI for STEMI from January 2010 to April 2016. TRI was calculated on admission using the following formula: heart rate × (age/10)2/systolic blood pressure. Discrimination and calibration of TRI for in-hospital events and 1 year mortality were analyzed. Results: Totally 1,054 patients were divided into three groups according to the tertiles of the TRI: <27 (n = 348), 27-36 (n = 360) and >36 (n = 346). The incidence of acute kidney injury (AKI; 7.8 vs. 8.6 vs. 24.0%, p < 0.001), AHF (3.5 vs. 6.6 vs. 16.2%, p < 0.001), in-hospital death (0.6 vs. 3.3 vs. 11.6%, p < 0.001) and MACEs (5.2 vs. 5.8 vs. 15.9%, p < 0.001) was significantly higher in the third tertile. TRI showed good discrimination for in-hospital death [area under the curve (AUC) = 0.804, p < 0.001; Hosmer-Lemeshow p = 0.302], which was superior to its prediction for AKI (AUC = 0.678, p < 0.001; Hosmer-Lemeshow p = 0.121), and in-hospital MACEs (AUC = 0.669, p < 0.001; Hosmer-Lemeshow p = 0.077). Receiver-operation characteristics curve showed that TRI > 42.0 had a sensitivity of 64.8% and specificity of 82.2% for predicting in-hospital death. Kaplan-Meier analysis showed that patients with TRI > 42.0 had higher 1 year mortality (Log-rank = 79.2, p < 0.001). Conclusion: TRI is suitable for risk stratification in elderly patients with STEMI undergoing PCI, and is thus of continuing value for an aging population.
Collapse
Affiliation(s)
- Bingqi Fu
- Division of Cardiology, Guangdong Cardiovascular Institute, Guangdong Provincial Key Laboratory of Coronary Heart Disease Prevention, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China
- Shantou University Medical College, Shantou, China
| | - Xuebiao Wei
- Division of Cardiology, Guangdong Cardiovascular Institute, Guangdong Provincial Key Laboratory of Coronary Heart Disease Prevention, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China
- Division of Geriatric Intensive Medicine, Guangdong Provincial Geriatrics Institute, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China
| | - Qi Wang
- Division of Cardiology, Guangdong Cardiovascular Institute, Guangdong Provincial Key Laboratory of Coronary Heart Disease Prevention, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China
- Shantou University Medical College, Shantou, China
| | - Zhiwen Yang
- Division of Cardiology, Guangdong Cardiovascular Institute, Guangdong Provincial Key Laboratory of Coronary Heart Disease Prevention, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China
- Shantou University Medical College, Shantou, China
| | - Jiyan Chen
- Division of Cardiology, Guangdong Cardiovascular Institute, Guangdong Provincial Key Laboratory of Coronary Heart Disease Prevention, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China
| | - Danqing Yu
- Division of Cardiology, Guangdong Cardiovascular Institute, Guangdong Provincial Key Laboratory of Coronary Heart Disease Prevention, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China
- *Correspondence: Danqing Yu
| |
Collapse
|
668
|
Khurshid F, Coo H, Khalil A, Messiha J, Ting JY, Wong J, Shah PS. Comparison of Multivariable Logistic Regression and Machine Learning Models for Predicting Bronchopulmonary Dysplasia or Death in Very Preterm Infants. Front Pediatr 2021; 9:759776. [PMID: 34950616 PMCID: PMC8688959 DOI: 10.3389/fped.2021.759776] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/17/2021] [Accepted: 11/01/2021] [Indexed: 11/30/2022] Open
Abstract
Bronchopulmonary dysplasia (BPD) is the most prevalent and clinically significant complication of prematurity. Accurate identification of at-risk infants would enable ongoing intervention to improve outcomes. Although postnatal exposures are known to affect an infant's likelihood of developing BPD, most existing BPD prediction models do not allow risk to be evaluated at different time points, and/or are not suitable for use in ethno-diverse populations. A comprehensive approach to developing clinical prediction models avoids assumptions as to which method will yield the optimal results by testing multiple algorithms/models. We compared the performance of machine learning and logistic regression models in predicting BPD/death. Our main cohort included infants <33 weeks' gestational age (GA) admitted to a Canadian Neonatal Network site from 2016 to 2018 (n = 9,006) with all analyses repeated for the <29 weeks' GA subcohort (n = 4,246). Models were developed to predict, on days 1, 7, and 14 of admission to neonatal intensive care, the composite outcome of BPD/death prior to discharge. Ten-fold cross-validation and a 20% hold-out sample were used to measure area under the curve (AUC). Calibration intercepts and slopes were estimated by regressing the outcome on the log-odds of the predicted probabilities. The model AUCs ranged from 0.811 to 0.886. Model discrimination was lower in the <29 weeks' GA subcohort (AUCs 0.699-0.790). Several machine learning models had a suboptimal calibration intercept and/or slope (k-nearest neighbor, random forest, artificial neural network, stacking neural network ensemble). The top-performing algorithms will be used to develop multinomial models and an online risk estimator for predicting BPD severity and death that does not require information on ethnicity.
Collapse
Affiliation(s)
- Faiza Khurshid
- Department of Pediatrics, Queen's University, Kingston, ON, Canada
| | - Helen Coo
- Department of Pediatrics, Queen's University, Kingston, ON, Canada
| | - Amal Khalil
- Centre for Advanced Computing, Queen's University, Kingston, ON, Canada
| | - Jonathan Messiha
- Smith School of Business, Queen's University, Kingston, ON, Canada
| | - Joseph Y Ting
- Department of Pediatrics, University of British Columbia, Vancouver, BC, Canada
| | - Jonathan Wong
- Department of Pediatrics, University of British Columbia, Vancouver, BC, Canada
| | - Prakesh S Shah
- Department of Paediatrics, University of Toronto, Toronto, ON, Canada.,Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada.,Lunenfeld-Tanenbaum Research Institute, Sinai Health, Toronto, ON, Canada
| |
Collapse
|
669
|
Mijderwijk HJ, Beez T, Hänggi D, Nieboer D. Application of clinical prediction modeling in pediatric neurosurgery: a case study. Childs Nerv Syst 2021; 37:1495-1504. [PMID: 33783617 PMCID: PMC8084798 DOI: 10.1007/s00381-021-05112-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/30/2020] [Accepted: 03/02/2021] [Indexed: 12/23/2022]
Abstract
There has been an increasing interest in articles reporting on clinical prediction models in pediatric neurosurgery. Clinical prediction models are mathematical equations that combine patient-related risk factors for the estimation of an individual's risk of an outcome. If used sensibly, these evidence-based tools may help pediatric neurosurgeons in medical decision-making processes. Furthermore, they may help to communicate anticipated future events of diseases to children and their parents and facilitate shared decision-making accordingly. A basic understanding of this methodology is incumbent when developing or applying a prediction model. This paper addresses this methodology tailored to pediatric neurosurgery. For illustration, we use original pediatric data from our institution to illustrate this methodology with a case study. The developed model is however not externally validated, and clinical impact has not been assessed; therefore, the model cannot be recommended for clinical use in its current form.
Collapse
Affiliation(s)
- Hendrik-Jan Mijderwijk
- Medical Faculty, Department of Neurosurgery, Heinrich Heine University, Moorenstraße 5, 40225, Düsseldorf, Germany.
| | - Thomas Beez
- Medical Faculty, Department of Neurosurgery, Heinrich Heine University, Moorenstraße 5, 40225 Düsseldorf, Germany
| | - Daniel Hänggi
- Medical Faculty, Department of Neurosurgery, Heinrich Heine University, Moorenstraße 5, 40225 Düsseldorf, Germany
| | - Daan Nieboer
- Department of Public Health, Erasmus University Medical Center, Rotterdam, The Netherlands
| |
Collapse
|
670
|
Adil SM, Elahi C, Gramer R, Spears CA, Fuller AT, Haglund MM, Dunn TW. Predicting the Individual Treatment Effect of Neurosurgery for Patients with Traumatic Brain Injury in the Low-Resource Setting: A Machine Learning Approach in Uganda. J Neurotrauma 2020; 38:928-939. [PMID: 33054545 DOI: 10.1089/neu.2020.7262] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/17/2022] Open
Abstract
Traumatic brain injury (TBI) disproportionately affects low- and middle-income countries (LMICs). In these low-resource settings, effective triage of patients with TBI-including the decision of whether or not to perform neurosurgery-is critical in optimizing patient outcomes and healthcare resource utilization. Machine learning may allow for effective predictions of patient outcomes both with and without surgery. Data from patients with TBI was collected prospectively at Mulago National Referral Hospital in Kampala, Uganda, from 2016 to 2019. One linear and six non-linear machine learning models were designed to predict good versus poor outcome near hospital discharge and internally validated using nested five-fold cross-validation. The 13 predictors included clinical variables easily acquired on admission and whether or not the patient received surgery. Using an elastic-net regularized logistic regression model (GLMnet), with predictions calibrated using Platt scaling, the probability of poor outcome was calculated for each patient both with and without surgery (with the difference quantifying the "individual treatment effect," ITE). Relative ITE represents the percent reduction in chance of poor outcome, equaling this ITE divided by the probability of poor outcome with no surgery. Ultimately, 1766 patients were included. Areas under the receiver operating characteristic curve (AUROCs) ranged from 83.1% (single C5.0 ruleset) to 88.5% (random forest), with the GLMnet at 87.5%. The two variables promoting good outcomes in the GLMnet model were high Glasgow Coma Scale score and receiving surgery. For the subgroup not receiving surgery, the median relative ITE was 42.9% (interquartile range [IQR], 32.7% to 53.5%); similarly, in those receiving surgery, it was 43.2% (IQR, 32.9% to 54.3%). We provide the first machine learning-based model to predict TBI outcomes with and without surgery in LMICs, thus enabling more effective surgical decision making in the resource-limited setting. Predicted ITE similarity between surgical and non-surgical groups suggests that, currently, patients are not being chosen optimally for neurosurgical intervention. Our clinical decision aid has the potential to improve outcomes.
Collapse
Affiliation(s)
- Syed M Adil
- Division of Global Neurosurgery and Neurology, Duke University Medical Center, Durham, North Carolina, USA
| | - Cyrus Elahi
- Division of Global Neurosurgery and Neurology, Duke University Medical Center, Durham, North Carolina, USA
| | - Robert Gramer
- Division of Global Neurosurgery and Neurology, Duke University Medical Center, Durham, North Carolina, USA
| | - Charis A Spears
- Division of Global Neurosurgery and Neurology, Duke University Medical Center, Durham, North Carolina, USA
| | - Anthony T Fuller
- Division of Global Neurosurgery and Neurology, Duke University Medical Center, Durham, North Carolina, USA.,Department of Neurosurgery, Duke University Medical Center, Durham, North Carolina, USA.,Duke Global Health Institute, Duke University, Durham, North Carolina. USA
| | - Michael M Haglund
- Division of Global Neurosurgery and Neurology, Duke University Medical Center, Durham, North Carolina, USA.,Department of Neurosurgery, Duke University Medical Center, Durham, North Carolina, USA.,Duke Global Health Institute, Duke University, Durham, North Carolina. USA
| | - Timothy W Dunn
- Division of Global Neurosurgery and Neurology, Duke University Medical Center, Durham, North Carolina, USA.,Department of Neurosurgery, Duke University Medical Center, Durham, North Carolina, USA.,Department of Statistical Science, Duke University Medical Center, Durham, North Carolina, USA
| |
Collapse
|
671
|
Czajka S, Ziębińska K, Marczenko K, Posmyk B, Szczepańska AJ, Krzych ŁJ. Validation of APACHE II, APACHE III and SAPS II scores in in-hospital and one year mortality prediction in a mixed intensive care unit in Poland: a cohort study. BMC Anesthesiol 2020; 20:296. [PMID: 33267777 PMCID: PMC7709291 DOI: 10.1186/s12871-020-01203-7] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2020] [Accepted: 11/10/2020] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND There are several scores used for in-hospital mortality prediction in critical illness. Their application in a local scenario requires validation to ensure appropriate diagnostic accuracy. Moreover, their use in assessing post-discharge mortality in intensive care unit (ICU) survivors has not been extensively studied. We aimed to validate APACHE II, APACHE III and SAPS II scores in short- and long-term mortality prediction in a mixed adult ICU in Poland. APACHE II, APACHE III and SAPS II scores, with corresponding predicted mortality ratios, were calculated for 303 consecutive patients admitted to a 10-bed ICU in 2016. Short-term (in-hospital) and long-term (12-month post-discharge) mortality was assessed. RESULTS Median APACHE II, APACHE III and SAPS II scores were 19 (IQR 12-24), 67 (36.5-88) and 44 (27-56) points, with corresponding in-hospital mortality ratios of 25.8% (IQR 12.1-46.0), 18.5% (IQR 3.8-41.8) and 34.8% (IQR 7.9-59.8). Observed in-hospital mortality was 35.6%. Moreover, 12-month post-discharge mortality reached 17.4%. All the scores predicted in-hospital mortality (p < 0.05): APACHE II (AUC = 0.78; 95%CI 0.73-0.83), APACHE III (AUC = 0.79; 95%CI 0.74-0.84) and SAPS II (AUC = 0.79; 95%CI 0.74-0.84); as well as mortality after hospital discharge (p < 0.05): APACHE II (AUC = 0.71; 95%CI 0.64-0.78), APACHE III (AUC = 0.72; 95%CI 0.65-0.78) and SAPS II (AUC = 0.69; 95%CI 0.62-0.76), with no statistically significant difference between the scores (p > 0.05). The calibration of the scores was good. CONCLUSIONS All the scores are acceptable predictors of in-hospital mortality. In the case of post-discharge mortality, their diagnostic accuracy is lower and of borderline clinical relevance. Further studies are needed to create scores estimating the long-term prognosis of subjects successfully discharged from the ICU.
Collapse
Affiliation(s)
- Szymon Czajka
- Department of Anesthesiology and Intensive Care, School of Medicine in Katowice, Medical University of Silesia, Katowice, Poland.
| | - Katarzyna Ziębińska
- Students' Scientific Society, Department of Anesthesiology and Intensive Care, School of Medicine in Katowice, Medical University of Silesia, Katowice, Poland
| | - Konstanty Marczenko
- Students' Scientific Society, Department of Anesthesiology and Intensive Care, School of Medicine in Katowice, Medical University of Silesia, Katowice, Poland
| | - Barbara Posmyk
- Students' Scientific Society, Department of Anesthesiology and Intensive Care, School of Medicine in Katowice, Medical University of Silesia, Katowice, Poland
| | - Anna J Szczepańska
- Department of Anesthesiology and Intensive Care, School of Medicine in Katowice, Medical University of Silesia, Katowice, Poland
| | - Łukasz J Krzych
- Department of Anesthesiology and Intensive Care, School of Medicine in Katowice, Medical University of Silesia, Katowice, Poland
| |
Collapse
|
672
|
|
673
|
Davis SE, Greevy RA, Lasko TA, Walsh CG, Matheny ME. Detection of calibration drift in clinical prediction models to inform model updating. J Biomed Inform 2020; 112:103611. [PMID: 33157313 PMCID: PMC8627243 DOI: 10.1016/j.jbi.2020.103611] [Citation(s) in RCA: 45] [Impact Index Per Article: 11.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2020] [Revised: 10/21/2020] [Accepted: 10/29/2020] [Indexed: 10/23/2022]
Abstract
Model calibration, critical to the success and safety of clinical prediction models, deteriorates over time in response to the dynamic nature of clinical environments. To support informed, data-driven model updating strategies, we present and evaluate a calibration drift detection system. Methods are developed for maintaining dynamic calibration curves with optimized online stochastic gradient descent and for detecting increasing miscalibration with adaptive sliding windows. These methods are generalizable to support diverse prediction models developed using a variety of learning algorithms and customizable to address the unique needs of clinical use cases. In both simulation and case studies, our system accurately detected calibration drift. When drift is detected, our system further provides actionable alerts by including information on a window of recent data that may be appropriate for model updating. Simulations showed these windows were primarily composed of data accruing after drift onset, supporting the potential utility of the windows for model updating. By promoting model updating as calibration deteriorates rather than on pre-determined schedules, implementations of our drift detection system may minimize interim periods of insufficient model accuracy and focus analytic resources on those models most in need of attention.
Collapse
Affiliation(s)
- Sharon E Davis
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA.
| | - Robert A Greevy
- Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN, USA.
| | - Thomas A Lasko
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA.
| | - Colin G Walsh
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA; Department of Medicine, Vanderbilt University Medical Center, Nashville, TN, USA; Department of Psychiatry, Vanderbilt University Medical Center, Nashville, TN, USA.
| | - Michael E Matheny
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA; Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN, USA; Department of Medicine, Vanderbilt University Medical Center, Nashville, TN, USA; Geriatrics Research, Education, and Clinical Care, Tennessee Valley Healthcare System VA, Nashville, TN, USA.
| |
Collapse
|
674
|
Brathwaite R, Rocha TBM, Kieling C, Kohrt BA, Mondelli V, Adewuya AO, Fisher HL. Predicting the risk of future depression among school-attending adolescents in Nigeria using a model developed in Brazil. Psychiatry Res 2020; 294:113511. [PMID: 33113451 PMCID: PMC7732701 DOI: 10.1016/j.psychres.2020.113511] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/04/2020] [Accepted: 10/12/2020] [Indexed: 01/29/2023]
Abstract
Depression commonly emerges in adolescence and is a major public health issue in low- and middle-income countries where 90% of the world's adolescents live. Thus efforts to prevent depression onset are crucial in countries like Nigeria, where two-thirds of the population are aged under 24. Therefore, we tested the ability of a prediction model developed in Brazil to predict future depression in a Nigerian adolescent sample. Data were obtained from school students aged 14-16 years in Lagos, who were assessed in 2016 and 2019 for depression using a self-completed version of the Mini International Neuropsychiatric Interview for Children and Adolescents. Only the 1,928 students free of depression at baseline were included. Penalized logistic regression was used to predict individualized risk of developing depression at follow-up for each adolescent based on the 7 matching baseline sociodemographic predictors from the Brazilian model. Discrimination between adolescents who did and did not develop depression was better than chance (area under the curve = 0.62 (bootstrap-corrected 95% CI: 0.58-0.66). However, the model was not well-calibrated even after adjustment of the intercept, indicating poorer overall performance compared to the original Brazilian cohort. Updating the model with context-specific factors may improve prediction of depression in this setting.
Collapse
Affiliation(s)
- Rachel Brathwaite
- King's College London, Social, Genetic & Developmental Psychiatry Centre, Institute of Psychiatry, Psychology & Neuroscience, London, United Kingdom
| | - Thiago Botter-Maio Rocha
- Department of Psychiatry, Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil; Child & Adolescent Psychiatry Division, Hospital de Clínicas de Porto Alegre, Brazil
| | - Christian Kieling
- Department of Psychiatry, Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil; Child & Adolescent Psychiatry Division, Hospital de Clínicas de Porto Alegre, Brazil
| | - Brandon A Kohrt
- Division of Global Mental Health, George Washington University, Washington DC, United States
| | - Valeria Mondelli
- King's College London, Department of Psychological Medicine, Institute of Psychiatry, Psychology & Neuroscience, London, United Kingdom; National Institute for Health Research Mental Health Biomedical Research Centre, South London and Maudsley NHS Foundation Trust and King's College London, London, United Kingdom
| | - Abiodun O Adewuya
- Department of Behavioural Medicine, Lagos State University College of Medicine, Lagos, Nigeria; Centre for Mental Health Research and Initiative (CEMHRI), Lagos, Nigeria
| | - Helen L Fisher
- King's College London, Social, Genetic & Developmental Psychiatry Centre, Institute of Psychiatry, Psychology & Neuroscience, London, United Kingdom; ESRC Centre for Society and Mental Health, King's College London, London, United Kingdom.
| |
Collapse
|
675
|
Chi QW, Zhao C, Li ST. Development and validation of a HOXB8 gene-based prognostic model and nomogram for colorectal cancer patients. Shijie Huaren Xiaohua Zazhi 2020; 28:1128-1136. [DOI: 10.11569/wcjd.v28.i22.1128] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/06/2023] Open
Abstract
BACKGROUND At present, colorectal cancer (CRC) is still associated with a high rate of recurrence and distant metastasis with a poor prognosis. HOXB8 gene is related to the tumorigenesis and development in CRC.
AIM To explore the prognostic value of HOXB8 gene in CRC patients, and provide a novel insight into the monitoring of disease progression and cancer recurrence in patients with high-risk CRC.
METHODS The mRNA sequencing data of HOXB8 in CRC patients was downloaded from The Cancer Genome Atlas database. Then, we analyzed the relationship between HOXB8 expression and clinicopathologic features in CRC, and performed survival analysis based on HOXB8 expression. Univariate and multivariate Cox regression analyese were performed for identifying prognostic factors for CRC, and then a nomogram was established and evaluated by concordance index, calibration curve, and decision curve analysis (DCA).
RESULTS HOXB8 mRNA expression was significantly correlated with CRC tumor tissue (P < 0.001), right-side CRC (P < 0.001), T stage (P = 0.024), and M stage (P = 0.0074). Survival analysis showed that overexpression of HOXB8 was associated with a poor progression-free survival (PFS) in CRC patients (P = 0.0019). Univariate and multivariate COX analyses suggested that the expression level of HOXB8 [HR: 1.539 (1.066-2.221), P = 0.021] and TNM stage were independent prognostic factors for PFS of CRC patients. A nomogram was established to predict 3- and 5-year PFS of CRC patients based on four factors including HOXB8 expression and TNM stage. The concordance index was 0.735, suggesting good discrimination; the calibration curve and DCA showed that the nomogram had good predictive power and clinical practicability.
CONCLUSION The expression of HOXB8 is significantly related to the prognosis of CRC patients, and it has appreciated predictive ability for disease progression and cancer recurrence in CRC patients. HOXB8 could act as a potential biomarker to identify high-risk CRC patients and become a novel therapeutic target and prognostic indicator for CRC.
Collapse
Affiliation(s)
- Qiang-Wei Chi
- Department of Colorectal Surgery, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou 325015, Zhejiang Province, China
| | - Chang Zhao
- Department of Colorectal Surgery, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou 325015, Zhejiang Province, China
| | - Shao-Tang Li
- Department of Colorectal Surgery, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou 325015, Zhejiang Province, China
| |
Collapse
|
676
|
Karas M, Marinsek N, Goldhahn J, Foschini L, Ramirez E, Clay I. Predicting Subjective Recovery from Lower Limb Surgery Using Consumer Wearables. Digit Biomark 2020; 4:73-86. [PMID: 33442582 DOI: 10.1159/000511531] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2020] [Accepted: 09/11/2020] [Indexed: 12/20/2022] Open
Abstract
Introduction A major challenge in the monitoring of rehabilitation is the lack of long-term individual baseline data which would enable accurate and objective assessment of functional recovery. Consumer-grade wearable devices enable the tracking of individual everyday functioning prior to illness or other medical events which necessitate the monitoring of recovery trajectories. Methods For 1,324 individuals who underwent surgery on a lower limb, we collected their Fitbit device data of steps, heart rate, and sleep from 26 weeks before to 26 weeks after the self-reported surgery date. We identified subgroups of individuals who self-reported surgeries for bone fracture repair (n = 355), tendon or ligament repair/reconstruction (n = 773), and knee or hip joint replacement (n = 196). We used linear mixed models to estimate the average effect of time relative to surgery on daily activity measurements while adjusting for gender, age, and the participant-specific activity baseline. We used a sub-cohort of 127 individuals with dense wearable data who underwent tendon/ligament surgery and employed XGBoost to predict the self-reported recovery time. Results The 1,324 study individuals were all US residents, predominantly female (84%), white or Caucasian (85%), and young to middle-aged (mean age 36.2 years). We showed that 12 weeks pre- and 26 weeks post-surgery trajectories of daily behavioral measurements (steps sum, heart rate, sleep efficiency score) can capture activity changes relative to an individual's baseline. We demonstrated that the trajectories differ across surgery types, recapitulate the documented effect of age on functional recovery, and highlight differences in relative activity change across self-reported recovery time groups. Finally, using a sub-cohort of 127 individuals, we showed that long-term recovery can be accurately predicted, on an individual level, only 1 month after surgery (AUROC 0.734, AUPRC 0.8). Furthermore, we showed that predictions are most accurate when long-term, individual baseline data are available. Discussion Leveraging long-term, passively collected wearable data promises to enable relative assessment of individual recovery and is a first step towards data-driven intervention for individuals.
Collapse
Affiliation(s)
- Marta Karas
- Evidation Health Inc., San Mateo, California, USA.,Department of Biostatistics, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, Maryland, USA
| | | | - Jörg Goldhahn
- Institute of Translational Medicine, Department of Health Sciences and Technology, Eidgenössische Technische Hochschule (ETH), Zurich, Switzerland
| | | | | | - Ieuan Clay
- Evidation Health Inc., San Mateo, California, USA
| |
Collapse
|
677
|
Rafiei A, Rezaee A, Hajati F, Gheisari S, Golzan M. SSP: Early prediction of sepsis using fully connected LSTM-CNN model. Comput Biol Med 2020; 128:104110. [PMID: 33227577 DOI: 10.1016/j.compbiomed.2020.104110] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2020] [Revised: 11/05/2020] [Accepted: 11/05/2020] [Indexed: 01/31/2023]
Abstract
BACKGROUND Sepsis is a life-threatening condition that occurs due to the body's reaction to infections, and it is a leading cause of morbidity and mortality in hospitals. Early prediction of sepsis onset facilitates early interventions that promote the survival of suspected patients. However, reliable and intelligent systems for predicting sepsis are scarce. METHODS This paper presents a novel technique called Smart Sepsis Predictor (SSP) to predict sepsis onset in patients admitted to an intensive care unit (ICU). SSP is a deep neural network architecture that encompasses long short-term memory (LSTM), convolutional, and fully connected layers to achieve early prediction of sepsis. SSP can work in two modes; Mode 1 uses demographic data and vital signs, and Mode 2 uses laboratory test results in addition to demographic data and vital signs. To evaluate SSP, we have used the 2019 PhysioNet/CinC Challenge dataset, which includes the records of 40,366 patients admitted to the ICU. RESULTS To compare SSP with existing state-of-the-art methods, we have measured the accuracy of the SSP in 4-, 8-, and 12-h prediction windows using publicly available data. Our results show that the SSP performance in Mode 1 and Mode 2 is much higher than existing methods, achieving an area under the receiver operating characteristic curve (AUROC) of 0.89 and 0.92, 0.88 and 0.87, and 0.86 and 0.84 for 4 h, 8 h, and 12 h before sepsis onset, respectively. CONCLUSIONS Using ICU data, sepsis onset can be predicted up to 12 h in advance. Our findings offer an early solution for mitigating the risk of sepsis onset.
Collapse
Affiliation(s)
- Alireza Rafiei
- Intelligent Mobile Robot Lab (IMRL), Department of Mechatronics Engineering, Faculty of New Sciences and Technologies, University of Tehran, Tehran, Iran.
| | - Alireza Rezaee
- Intelligent Mobile Robot Lab (IMRL), Department of Mechatronics Engineering, Faculty of New Sciences and Technologies, University of Tehran, Tehran, Iran.
| | - Farshid Hajati
- College of Engineering and Science, Victoria University Sydney, Australia.
| | - Soheila Gheisari
- Vision Science Group, Graduate School of Health, University of Technology Sydney, Australia.
| | - Mojtaba Golzan
- Vision Science Group, Graduate School of Health, University of Technology Sydney, Australia.
| |
Collapse
|
678
|
Semenova E, Williams DP, Afzal AM, Lazic SE. A Bayesian neural network for toxicity prediction. ACTA ACUST UNITED AC 2020. [DOI: 10.1016/j.comtox.2020.100133] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
|
679
|
Beyene KM, El Ghouch A. Smoothed time-dependent receiver operating characteristic curve for right censored survival data. Stat Med 2020; 39:3373-3396. [PMID: 32687225 DOI: 10.1002/sim.8671] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2019] [Revised: 03/30/2020] [Accepted: 06/05/2020] [Indexed: 11/08/2022]
Abstract
The prediction reliability is of primary concern in many clinical studies when the objective is to develop new predictive models or improve existing risk scores. In fact, before using a model in any clinical decision making, it is very important to check its ability to discriminate between subjects who are at risk of, for example, developing certain disease in a near future from those who will not. To that end, the time-dependent receiver operating characteristic (ROC) curve is the most commonly used method in practice. Several approaches have been proposed in the literature to estimate the ROC nonparametrically in the context of survival data. But, except one recent approach, all the existing methods provide a nonsmooth ROC estimator whereas, by definition, the ROC curve is smooth. In this article we propose and study a new nonparametric smooth ROC estimator based on a weighted kernel smoother. More precisely, our approach relies on a well-known kernel method used to estimate cumulative distribution functions of random variables with bounded supports. We derived some asymptotic properties for the proposed estimator. As bandwidth is the main parameter to be set, we present and study different methods to appropriately select one. A simulation study is conducted, under different scenarios, to prove the consistency of the proposed method and to compare its finite sample performance with a competitor. The results show that the proposed method performs better and appear to be quite robust to bandwidth choice. As for inference purposes, our results also reveal the good performances of a proposed nonparametric bootstrap procedure. Furthermore, we illustrate the method using a real data example.
Collapse
Affiliation(s)
- Kassu Mehari Beyene
- Institute of Statistics, Biostatistics and Actuarial Sciences, Catholic University of Louvain, Louvain la Neuve, Belgium
| | - Anouar El Ghouch
- Institute of Statistics, Biostatistics and Actuarial Sciences, Catholic University of Louvain, Louvain la Neuve, Belgium
| |
Collapse
|
680
|
Hill NR, Arden C, Beresford-Hulme L, Camm AJ, Clifton D, Davies DW, Farooqui U, Gordon J, Groves L, Hurst M, Lawton S, Lister S, Mallen C, Martin AC, McEwan P, Pollock KG, Rogers J, Sandler B, Sugrue DM, Cohen AT. Identification of undiagnosed atrial fibrillation patients using a machine learning risk prediction algorithm and diagnostic testing (PULsE-AI): Study protocol for a randomised controlled trial. Contemp Clin Trials 2020; 99:106191. [PMID: 33091585 PMCID: PMC7571442 DOI: 10.1016/j.cct.2020.106191] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2020] [Revised: 10/14/2020] [Accepted: 10/16/2020] [Indexed: 12/29/2022]
Abstract
Atrial fibrillation (AF) is associated with an increased risk of stroke, enhanced stroke severity, and other comorbidities. However, AF is often asymptomatic, and frequently remains undiagnosed until complications occur. Current screening approaches for AF lack either cost-effectiveness or diagnostic sensitivity; thus, there is interest in tools that could be used for population screening. An AF risk prediction algorithm, developed using machine learning from a UK dataset of 2,994,837 patients, was found to be more effective than existing models at identifying patients at risk of AF. Therefore, the aim of the trial is to assess the effectiveness of this risk prediction algorithm combined with diagnostic testing for the identification of AF in a real-world primary care setting. Eligible participants (aged ≥30 years and without an existing AF diagnosis) registered at participating UK general practices will be randomised into intervention and control arms. Intervention arm participants identified at highest risk of developing AF (algorithm risk score ≥ 7.4%) will be invited for a 12‑lead electrocardiogram (ECG) followed by two-weeks of home-based ECG monitoring with a KardiaMobile device. Control arm participants will be used for comparison and will be managed routinely. The primary outcome is the number of AF diagnoses in the intervention arm compared with the control arm during the research window. If the trial is successful, there is potential for the risk prediction algorithm to be implemented throughout primary care for narrowing the population considered at highest risk for AF who could benefit from more intensive screening for AF. Trial Registration: NCT04045639.
Collapse
Affiliation(s)
- Nathan R Hill
- Bristol Myers Squibb Pharmaceutical Ltd, Uxbridge, UK.
| | - Chris Arden
- Park Surgery, Chandlers Ford, Hampshire, UK.
| | | | - A John Camm
- Cardiology Clinical Academic Group, Molecular & Clinical Sciences Research Institute, St. George's University of London, London, UK.
| | - David Clifton
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, UK.
| | | | | | - Jason Gordon
- Health Economics and Outcomes Research Ltd, Cardiff, UK.
| | - Lara Groves
- Health Economics and Outcomes Research Ltd, Cardiff, UK.
| | - Michael Hurst
- Health Economics and Outcomes Research Ltd, Cardiff, UK.
| | - Sarah Lawton
- School of Medicine, Keele University, Staffordshire, UK.
| | - Steven Lister
- Bristol Myers Squibb Pharmaceutical Ltd, Uxbridge, UK.
| | | | - Anne-Celine Martin
- Université de Paris, Innovative Therapies in Haemostasis, INSERM, Hôpital Européen Georges Pompidou, Service de Cardiologie, 20 rue Leblanc, Paris, France
| | - Phil McEwan
- Health Economics and Outcomes Research Ltd, Cardiff, UK.
| | | | | | | | | | - Alexander T Cohen
- Department of Haematological Medicine, Guys and St Thomas' NHS Foundation Trust, King's College London, London, UK.
| |
Collapse
|
681
|
Best practices for authors of healthcare-related artificial intelligence manuscripts. NPJ Digit Med 2020; 3:134. [PMID: 33083569 PMCID: PMC7567805 DOI: 10.1038/s41746-020-00336-w] [Citation(s) in RCA: 26] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2020] [Accepted: 09/17/2020] [Indexed: 12/23/2022] Open
Abstract
Since its inception in 2017, npj Digital Medicine has attracted a disproportionate number of manuscripts reporting on uses of artificial intelligence. This field has matured rapidly in the past several years. There was initial fascination with the algorithms themselves (machine learning, deep learning, convoluted neural networks) and the use of these algorithms to make predictions that often surpassed prevailing benchmarks. As the discipline has matured, individuals have called attention to aberrancies in the output of these algorithms. In particular, criticisms have been widely circulated that algorithmically developed models may have limited generalizability due to overfitting to the training data and may systematically perpetuate various forms of biases inherent in the training data, including race, gender, age, and health state or fitness level (Challen et al. BMJ Qual. Saf. 28:231–237, 2019; O’neil. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, Broadway Book, 2016). Given our interest in publishing the highest quality papers and the growing volume of submissions using AI algorithms, we offer a list of criteria that authors should consider before submitting papers to npj Digital Medicine.
Collapse
|
682
|
A prognostic model predicted deterioration in health-related quality of life in older patients with multimorbidity and polypharmacy. J Clin Epidemiol 2020; 130:1-12. [PMID: 33065164 DOI: 10.1016/j.jclinepi.2020.10.006] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2020] [Revised: 08/12/2020] [Accepted: 10/07/2020] [Indexed: 01/07/2023]
Abstract
OBJECTIVES To develop and validate a prognostic model to predict deterioration in health-related quality of life (dHRQoL) in older general practice patients with at least one chronic condition and one chronic prescription. STUDY DESIGN AND SETTING We used individual participant data from five cluster-randomized trials conducted in the Netherlands and Germany to predict dHRQoL, defined as a decrease in EQ-5D-3 L index score of ≥5% after 6-month follow-up in logistic regression models with stratified intercepts to account for between-study heterogeneity. The model was validated internally and by using internal-external cross-validation (IECV). RESULTS In 3,582 patients with complete data, of whom 1,046 (29.2%) showed deterioration in HRQoL, and 12/87 variables were selected that were related to single (chronic) conditions, inappropriate medication, medication underuse, functional status, well-being, and HRQoL. Bootstrap internal validation showed a C-statistic of 0.71 (0.69 to 0.72) and a calibration slope of 0.88 (0.78 to 0.98). In the IECV loop, the model provided a pooled C-statistic of 0.68 (0.65 to 0.70) and calibration-in-the-large of 0 (-0.13 to 0.13). HRQoL/functionality had the strongest prognostic value. CONCLUSION The model performed well in terms of discrimination, calibration, and generalizability and might help clinicians identify older patients at high risk of dHRQoL. REGISTRATION PROSPERO ID: CRD42018088129.
Collapse
|
683
|
Devroe J, Peeraer K, Verbeke G, Spiessens C, Vriens J, Dancet E. Predicting the chance on live birth per cycle at each step of the IVF journey: external validation and update of the van Loendersloot multivariable prognostic model. BMJ Open 2020; 10:e037289. [PMID: 33033089 PMCID: PMC7545639 DOI: 10.1136/bmjopen-2020-037289] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/07/2023] Open
Abstract
OBJECTIVE To study the performance of the 'van Loendersloot' prognostic model for our clinic's in vitro fertilisation (IVF) in its original version, the refitted version and in an adapted version replacing previous by current cycle IVF laboratory variables. METHODS This retrospective cohort study in our academic tertiary fertility clinic analysed 1281 IVF cycles of 591 couples, who completed at least one 2nd-6th IVF cycle with own fresh gametes after a previous IVF cycle with the same partner in our clinic between 2010 and 2018. The outcome of interest was the chance on a live birth after one complete IVF cycle (including all fresh and frozen embryo transfers from the same episode of ovarian stimulation). Model performance was expressed in terms of discrimination (c-statistics) and calibration (calibration model, comparison of prognosis to observed ratios of five disjoint groups formed by the quintiles of the IVF prognoses and a calibration plot). RESULTS A total of 344 live births were obtained (26.9%). External validation of the original van Loendersloot model showed a poor c-statistic of 0.64 (95% CI: 0.61 to 0.68) and an underestimation of IVF success. The refitted and the adapted models showed c-statistics of respectively 0.68 (95% CI: 0.65 to 0.71) and 0.74 (95% CI: 0.70 to 0.77). Similar c-statistics were found with cross-validation. Both models showed a good calibration model; refitted model: intercept=0.00 (95% CI: -0.23 to 0.23) and slope=1.00 (95% CI: 0.79 to 1.21); adapted model: intercept=0.00 (95% CI: -0.18 to 0.18) and slope=1.00 (95% CI: 0.83 to 1.17). Prognoses and observed success rates of the disjoint groups matched well for the refitted model and even better for the adapted model. CONCLUSION External validation of the original van Loendersloot model indicated that model updating was recommended. The good performance of the refitted and adapted models allows informing couples about their IVF prognosis prior to an IVF cycle and at the time of embryo transfer. Whether this has an impact on couple's expected success rates, distress and IVF discontinuation can now be studied.
Collapse
Affiliation(s)
- Johanna Devroe
- Leuven University Fertility Centre, University Hospital Leuven, Leuven, Belgium
- Development and Regeneration, Laboratory of Endometrium, Endometriosis & Reproductive Medicine, Leuven, Belgium
| | - Karen Peeraer
- Leuven University Fertility Centre, University Hospital Leuven, Leuven, Belgium
- Development and Regeneration, Laboratory of Endometrium, Endometriosis & Reproductive Medicine, Leuven, Belgium
| | - Geert Verbeke
- Public Health and Primary Care, Leuven Biostatistics and statistical Bioinformatics Centre, Leuven, Belgium
- Interuniversity Institute for Biostatistics and Statistical Bioinformatics, Leuven, Belgium
| | - Carl Spiessens
- Leuven University Fertility Centre, University Hospital Leuven, Leuven, Belgium
| | - Joris Vriens
- Development and Regeneration, Laboratory of Endometrium, Endometriosis & Reproductive Medicine, Leuven, Belgium
| | - Eline Dancet
- Leuven University Fertility Centre, University Hospital Leuven, Leuven, Belgium
- Development and Regeneration, Laboratory of Endometrium, Endometriosis & Reproductive Medicine, Leuven, Belgium
- Postdoctoral fellow, Research Foundation, Flanders, Belgium
| |
Collapse
|
684
|
Bicknell R, Lim WK, Maier AB, LoGiuidice D. A study protocol for the development of a multivariable model predicting 6- and 12-month mortality for people with dementia living in residential aged care facilities (RACFs) in Australia. Diagn Progn Res 2020; 4:17. [PMID: 33033746 PMCID: PMC7538167 DOI: 10.1186/s41512-020-00085-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/12/2019] [Accepted: 09/23/2020] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND For residential aged care facility (RACF) residents with dementia, lack of prognostic guidance presents a significant challenge for end of life care planning. In an attempt to address this issue, models have been developed to assess mortality risk for people with advanced dementia, predominantly using long-term care minimum data set (MDS) information from the USA. A limitation of these models is that the information contained within the MDS used for model development was not collected for the purpose of identifying prognostic factors. The models developed using MDS data have had relatively modest ability to discriminate mortality risk and are difficult to apply outside the MDS setting. This study will aim to develop a model to estimate 6- and 12-month mortality risk for people with dementia from prognostic indicators recorded during usual clinical care provided in RACFs in Australia. METHODS A secondary analysis will be conducted for a cohort of people with dementia from RACFs participating in a cluster-randomized trial of a palliative care education intervention (IMPETUS-D). Ten prognostic indicator variables were identified based on a literature review of clinical features associated with increased mortality for people with dementia living in RACFs. Variables will be extracted from RACF files at baseline and mortality measured at 6 and 12 months after baseline data collection. A multivariable logistic regression model will be developed for 6- and 12-month mortality outcome measures using backwards elimination with a fractional polynomial approach for continuous variables. Internal validation will be undertaken using bootstrapping methods. Discrimination of the model for 6- and 12-month mortality will be presented as receiver operating curves with c statistics. Calibration curves will be presented comparing observed and predicted event rates for each decile of risk as well as flexible calibration curves derived using loess-based functions. DISCUSSION The model developed in this study aims to improve clinical assessment of mortality risk for people with dementia living in RACFs in Australia. Further external validation in different populations will be required before the model could be developed into a tool to assist with clinical decision-making in the future.
Collapse
Affiliation(s)
- Ross Bicknell
- Department of Medicine and Aged Care, @AgeMelbourne, Melbourne Health–Royal Melbourne Hospital, University of Melbourne, 6 North Main Building, Royal Melbourne Hospital, 300 Grattan Street, Parkville, Victoria 3050 Australia
| | - Wen Kwang Lim
- Department of Medicine and Aged Care, @AgeMelbourne, Melbourne Health–Royal Melbourne Hospital, University of Melbourne, 6 North Main Building, Royal Melbourne Hospital, 300 Grattan Street, Parkville, Victoria 3050 Australia
| | - Andrea B. Maier
- Department of Medicine and Aged Care, @AgeMelbourne, Melbourne Health–Royal Melbourne Hospital, University of Melbourne, 6 North Main Building, Royal Melbourne Hospital, 300 Grattan Street, Parkville, Victoria 3050 Australia
- Department of Human Movement Sciences, @AgeAmsterdam, Amsterdam Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Dina LoGiuidice
- Department of Medicine and Aged Care, @AgeMelbourne, Melbourne Health–Royal Melbourne Hospital, University of Melbourne, 6 North Main Building, Royal Melbourne Hospital, 300 Grattan Street, Parkville, Victoria 3050 Australia
| |
Collapse
|
685
|
Affiliation(s)
- Artuur M Leeuwenberg
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht 3508 GA, Netherlands
| | - Ewoud Schuit
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht 3508 GA, Netherlands
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht 3508 GA, Netherlands
| |
Collapse
|
686
|
Yadaw AS, Li YC, Bose S, Iyengar R, Bunyavanich S, Pandey G. Clinical features of COVID-19 mortality: development and validation of a clinical prediction model. Lancet Digit Health 2020; 2:e516-e525. [PMID: 32984797 PMCID: PMC7508513 DOI: 10.1016/s2589-7500(20)30217-x] [Citation(s) in RCA: 150] [Impact Index Per Article: 37.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/08/2023]
Abstract
Background The COVID-19 pandemic has affected millions of individuals and caused hundreds of thousands of deaths worldwide. Predicting mortality among patients with COVID-19 who present with a spectrum of complications is very difficult, hindering the prognostication and management of the disease. We aimed to develop an accurate prediction model of COVID-19 mortality using unbiased computational methods, and identify the clinical features most predictive of this outcome. Methods In this prediction model development and validation study, we applied machine learning techniques to clinical data from a large cohort of patients with COVID-19 treated at the Mount Sinai Health System in New York City, NY, USA, to predict mortality. We analysed patient-level data captured in the Mount Sinai Data Warehouse database for individuals with a confirmed diagnosis of COVID-19 who had a health system encounter between March 9 and April 6, 2020. For initial analyses, we used patient data from March 9 to April 5, and randomly assigned (80:20) the patients to the development dataset or test dataset 1 (retrospective). Patient data for those with encounters on April 6, 2020, were used in test dataset 2 (prospective). We designed prediction models based on clinical features and patient characteristics during health system encounters to predict mortality using the development dataset. We assessed the resultant models in terms of the area under the receiver operating characteristic curve (AUC) score in the test datasets. Findings Using the development dataset (n=3841) and a systematic machine learning framework, we developed a COVID-19 mortality prediction model that showed high accuracy (AUC=0·91) when applied to test datasets of retrospective (n=961) and prospective (n=249) patients. This model was based on three clinical features: patient's age, minimum oxygen saturation over the course of their medical encounter, and type of patient encounter (inpatient vs outpatient and telehealth visits). Interpretation An accurate and parsimonious COVID-19 mortality prediction model based on three features might have utility in clinical settings to guide the management and prognostication of patients affected by this disease. External validation of this prediction model in other populations is needed. Funding National Institutes of Health.
Collapse
Affiliation(s)
- Arjun S Yadaw
- Department of Pharmacological Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Yan-Chak Li
- Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Sonali Bose
- Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Ravi Iyengar
- Department of Pharmacological Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Supinda Bunyavanich
- Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Gaurav Pandey
- Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| |
Collapse
|
687
|
Bongers MER, Karhade AV, Villavieja J, Groot OQ, Bilsky MH, Laufer I, Schwab JH. Does the SORG algorithm generalize to a contemporary cohort of patients with spinal metastases on external validation? Spine J 2020; 20:1646-1652. [PMID: 32428674 DOI: 10.1016/j.spinee.2020.05.003] [Citation(s) in RCA: 32] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/06/2020] [Revised: 05/05/2020] [Accepted: 05/07/2020] [Indexed: 02/03/2023]
Abstract
BACKGROUND CONTEXT The SORG machine-learning algorithms were previously developed for preoperative prediction of overall survival in spinal metastatic disease. On sub-group analysis of a previous external validation, these algorithms were found to have diminished performance on patients treated after 2010. PURPOSE The purpose of this study was to assess the performance of these algorithms on a large contemporary cohort of consecutive spinal metastatic disease patients. STUDY DESIGN/SETTING Retrospective study performed at a tertiary care referral center. PATIENT SAMPLE Patients of 18 years and older treated with surgery for metastatic spinal disease between 2014 and 2016. OUTCOME MEASURES Ninety-day and one-year mortality. METHODS Baseline patient and tumor characteristics of the validation cohort were compared to the development cohort using bivariate logistic regression. Performance of the SORG algorithms on external validation in the contemporary cohort was assessed with discrimination (c-statistic and receiver operating curve), calibration (calibration plot, intercept, and slope), overall performance (Brier score compared to the null-model Brier score), and decision curve analysis. RESULTS Overall, 200 patients were included with 90-day and 1-year mortality rates of 55 (27.6%) and 124 (62.9%), respectively. The contemporary external validation cohort and the developmental cohort differed significantly on primary tumor histology, presence of visceral metastases, American Spinal Injury Association impairment scale, and preoperative laboratory values. The SORG algorithms for 90-day and 1-year mortality retained good discriminative ability (c-statistic of 0.81 [95% confidence interval [CI], 0.74-0.87] and 0.84 [95% CI, 0.77-0.89]), overall performance, and decision curve analysis. The algorithm for 90-day mortality showed almost perfect calibration reflected in an overall calibration intercept of -0.07 (95% CI: -0.50, 0.35). The 1-year mortality algorithm underestimated mortality mainly for the lowest predicted probabilities with an overall intercept of 0.57 (95% CI: 0.18, 0.96). CONCLUSIONS The SORG algorithms for survival in spinal metastatic disease generalized well to a contemporary cohort of consecutively treated patients from an external institutional. Further validation in international cohorts and large, prospective multi-institutional trials is required to confirm or refute the findings presented here. The open-access algorithms are available here: https://sorg-apps.shinyapps.io/spinemetssurvival/.
Collapse
Affiliation(s)
- Michiel E R Bongers
- Department of Orthopedic Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, MA 02114, USA.
| | - Aditya V Karhade
- Department of Orthopedic Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, MA 02114, USA
| | - Jemma Villavieja
- Department of Neurosurgery, Memorial Sloan-Kettering Cancer Center, New York, NY, USA
| | - Olivier Q Groot
- Department of Orthopedic Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, MA 02114, USA
| | - Mark H Bilsky
- Department of Neurosurgery, Memorial Sloan-Kettering Cancer Center, New York, NY, USA
| | - Ilya Laufer
- Department of Neurosurgery, Memorial Sloan-Kettering Cancer Center, New York, NY, USA
| | - Joseph H Schwab
- Department of Orthopedic Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, MA 02114, USA
| |
Collapse
|
688
|
Van Calster B, Wynants L, Collins GS, Verbakel JY, Steyerberg EW. ROC curves for clinical prediction models part 3. The ROC plot: a picture that needs a 1000 words. J Clin Epidemiol 2020; 126:220-223. [PMID: 32562835 DOI: 10.1016/j.jclinepi.2020.05.037] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2020] [Accepted: 05/24/2020] [Indexed: 12/23/2022]
Affiliation(s)
- Ben Van Calster
- KU Leuven, Department of Development and Regeneration, Leuven, Belgium; Department of Biomedical Data Sciences, Leiden University Medical Centre (LUMC), Leiden, the Netherlands; EPI-Centre, KU Leuven, Leuven, Belgium.
| | - Laure Wynants
- KU Leuven, Department of Development and Regeneration, Leuven, Belgium; Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, the Netherlands
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Musculoskeletal Sciences, University of Oxford, Oxford, UK; NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, UK
| | - Jan Y Verbakel
- EPI-Centre, KU Leuven, Leuven, Belgium; Academic Centre for Primary Care, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium; Nuffield Department of Primary Care Health Sciences, University of Oxford, UK
| | - Ewout W Steyerberg
- Department of Biomedical Data Sciences, Leiden University Medical Centre (LUMC), Leiden, the Netherlands
| |
Collapse
|
689
|
Comparison of Conventional Statistical Methods with Machine Learning in Medicine: Diagnosis, Drug Development, and Treatment. ACTA ACUST UNITED AC 2020; 56:medicina56090455. [PMID: 32911665 PMCID: PMC7560135 DOI: 10.3390/medicina56090455] [Citation(s) in RCA: 138] [Impact Index Per Article: 34.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2020] [Revised: 09/02/2020] [Accepted: 09/07/2020] [Indexed: 01/22/2023]
Abstract
Futurists have anticipated that novel autonomous technologies, embedded with machine learning (ML), will substantially influence healthcare. ML is focused on making predictions as accurate as possible, while traditional statistical models are aimed at inferring relationships between variables. The benefits of ML comprise flexibility and scalability compared with conventional statistical approaches, which makes it deployable for several tasks, such as diagnosis and classification, and survival predictions. However, much of ML-based analysis remains scattered, lacking a cohesive structure. There is a need to evaluate and compare the performance of well-developed conventional statistical methods and ML on patient outcomes, such as survival, response to treatment, and patient-reported outcomes (PROs). In this article, we compare the usefulness and limitations of traditional statistical methods and ML, when applied to the medical field. Traditional statistical methods seem to be more useful when the number of cases largely exceeds the number of variables under study and a priori knowledge on the topic under study is substantial such as in public health. ML could be more suited in highly innovative fields with a huge bulk of data, such as omics, radiodiagnostics, drug development, and personalized treatment. Integration of the two approaches should be preferred over a unidirectional choice of either approach.
Collapse
|
690
|
Steyerberg EW, Bonneville EF. Praise to Robust Prediction Modeling in Large Datasets. JACC CardioOncol 2020; 2:411-413. [PMID: 32955520 PMCID: PMC7491996 DOI: 10.1016/j.jaccao.2020.08.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/05/2022] Open
Affiliation(s)
- Ewout W. Steyerberg
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
| | - Edouard F. Bonneville
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
| |
Collapse
|
691
|
Mervin LH, Afzal AM, Engkvist O, Bender A. Comparison of Scaling Methods to Obtain Calibrated Probabilities of Activity for Protein–Ligand Predictions. J Chem Inf Model 2020; 60:4546-4559. [DOI: 10.1021/acs.jcim.0c00476] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/16/2022]
Affiliation(s)
- Lewis H. Mervin
- Hit Discovery, Discovery Sciences, R&D, AstraZeneca, Cambridge CB2 0AA, U.K
| | - Avid M. Afzal
- Data Sciences & Quantitative Biology, Discovery Sciences, R&D, AstraZeneca, Cambridge CB2 0AA, U.K
| | - Ola Engkvist
- Hit Discovery, Discovery Sciences, R&D, AstraZeneca, Mölndal SE-431 83, Sweden
| | - Andreas Bender
- Centre for Molecular Informatics, Department of Chemistry, University of Cambridge, Cambridge CB2 1TN, U.K
| |
Collapse
|
692
|
Kreuzberger N, Damen JA, Trivella M, Estcourt LJ, Aldin A, Umlauff L, Vazquez-Montes MD, Wolff R, Moons KG, Monsef I, Foroutan F, Kreuzer KA, Skoetz N. Prognostic models for newly-diagnosed chronic lymphocytic leukaemia in adults: a systematic review and meta-analysis. Cochrane Database Syst Rev 2020; 7:CD012022. [PMID: 32735048 PMCID: PMC8078230 DOI: 10.1002/14651858.cd012022.pub2] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/06/2023]
Abstract
BACKGROUND Chronic lymphocytic leukaemia (CLL) is the most common cancer of the lymphatic system in Western countries. Several clinical and biological factors for CLL have been identified. However, it remains unclear which of the available prognostic models combining those factors can be used in clinical practice to predict long-term outcome in people newly-diagnosed with CLL. OBJECTIVES To identify, describe and appraise all prognostic models developed to predict overall survival (OS), progression-free survival (PFS) or treatment-free survival (TFS) in newly-diagnosed (previously untreated) adults with CLL, and meta-analyse their predictive performances. SEARCH METHODS We searched MEDLINE (from January 1950 to June 2019 via Ovid), Embase (from 1974 to June 2019) and registries of ongoing trials (to 5 March 2020) for development and validation studies of prognostic models for untreated adults with CLL. In addition, we screened the reference lists and citation indices of included studies. SELECTION CRITERIA We included all prognostic models developed for CLL which predict OS, PFS, or TFS, provided they combined prognostic factors known before treatment initiation, and any studies that tested the performance of these models in individuals other than the ones included in model development (i.e. 'external model validation studies'). We included studies of adults with confirmed B-cell CLL who had not received treatment prior to the start of the study. We did not restrict the search based on study design. DATA COLLECTION AND ANALYSIS We developed a data extraction form to collect information based on the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS). Independent pairs of review authors screened references, extracted data and assessed risk of bias according to the Prediction model Risk Of Bias ASsessment Tool (PROBAST). For models that were externally validated at least three times, we aimed to perform a quantitative meta-analysis of their predictive performance, notably their calibration (proportion of people predicted to experience the outcome who do so) and discrimination (ability to differentiate between people with and without the event) using a random-effects model. When a model categorised individuals into risk categories, we pooled outcome frequencies per risk group (low, intermediate, high and very high). We did not apply GRADE as guidance is not yet available for reviews of prognostic models. MAIN RESULTS From 52 eligible studies, we identified 12 externally validated models: six were developed for OS, one for PFS and five for TFS. In general, reporting of the studies was poor, especially predictive performance measures for calibration and discrimination; but also basic information, such as eligibility criteria and the recruitment period of participants was often missing. We rated almost all studies at high or unclear risk of bias according to PROBAST. Overall, the applicability of the models and their validation studies was low or unclear; the most common reasons were inappropriate handling of missing data and serious reporting deficiencies concerning eligibility criteria, recruitment period, observation time and prediction performance measures. We report the results for three models predicting OS, which had available data from more than three external validation studies: CLL International Prognostic Index (CLL-IPI) This score includes five prognostic factors: age, clinical stage, IgHV mutational status, B2-microglobulin and TP53 status. Calibration: for the low-, intermediate- and high-risk groups, the pooled five-year survival per risk group from validation studies corresponded to the frequencies observed in the model development study. In the very high-risk group, predicted survival from CLL-IPI was lower than observed from external validation studies. Discrimination: the pooled c-statistic of seven external validation studies (3307 participants, 917 events) was 0.72 (95% confidence interval (CI) 0.67 to 0.77). The 95% prediction interval (PI) of this model for the c-statistic, which describes the expected interval for the model's discriminative ability in a new external validation study, ranged from 0.59 to 0.83. Barcelona-Brno score Aimed at simplifying the CLL-IPI, this score includes three prognostic factors: IgHV mutational status, del(17p) and del(11q). Calibration: for the low- and intermediate-risk group, the pooled survival per risk group corresponded to the frequencies observed in the model development study, although the score seems to overestimate survival for the high-risk group. Discrimination: the pooled c-statistic of four external validation studies (1755 participants, 416 events) was 0.64 (95% CI 0.60 to 0.67); 95% PI 0.59 to 0.68. MDACC 2007 index score The authors presented two versions of this model including six prognostic factors to predict OS: age, B2-microglobulin, absolute lymphocyte count, gender, clinical stage and number of nodal groups. Only one validation study was available for the more comprehensive version of the model, a formula with a nomogram, while seven studies (5127 participants, 994 events) validated the simplified version of the model, the index score. Calibration: for the low- and intermediate-risk groups, the pooled survival per risk group corresponded to the frequencies observed in the model development study, although the score seems to overestimate survival for the high-risk group. Discrimination: the pooled c-statistic of the seven external validation studies for the index score was 0.65 (95% CI 0.60 to 0.70); 95% PI 0.51 to 0.77. AUTHORS' CONCLUSIONS Despite the large number of published studies of prognostic models for OS, PFS or TFS for newly-diagnosed, untreated adults with CLL, only a minority of these (N = 12) have been externally validated for their respective primary outcome. Three models have undergone sufficient external validation to enable meta-analysis of the model's ability to predict survival outcomes. Lack of reporting prevented us from summarising calibration as recommended. Of the three models, the CLL-IPI shows the best discrimination, despite overestimation. However, performance of the models may change for individuals with CLL who receive improved treatment options, as the models included in this review were tested mostly on retrospective cohorts receiving a traditional treatment regimen. In conclusion, this review shows a clear need to improve the conducting and reporting of both prognostic model development and external validation studies. For prognostic models to be used as tools in clinical practice, the development of the models (and their subsequent validation studies) should adapt to include the latest therapy options to accurately predict performance. Adaptations should be timely.
Collapse
Key Words
- adult
- female
- humans
- male
- age factors
- bias
- biomarkers, tumor
- calibration
- confidence intervals
- discriminant analysis
- disease-free survival
- genes, p53
- genes, p53/genetics
- immunoglobulin heavy chains
- immunoglobulin heavy chains/genetics
- immunoglobulin variable region
- immunoglobulin variable region/genetics
- leukemia, lymphocytic, chronic, b-cell
- leukemia, lymphocytic, chronic, b-cell/mortality
- leukemia, lymphocytic, chronic, b-cell/pathology
- models, theoretical
- neoplasm staging
- prognosis
- progression-free survival
- receptors, antigen, b-cell
- receptors, antigen, b-cell/genetics
- reproducibility of results
- tumor suppressor protein p53
- tumor suppressor protein p53/genetics
Collapse
MESH Headings
- Adult
- Age Factors
- Bias
- Biomarkers, Tumor
- Calibration
- Confidence Intervals
- Discriminant Analysis
- Disease-Free Survival
- Female
- Genes, p53/genetics
- Humans
- Immunoglobulin Heavy Chains/genetics
- Immunoglobulin Variable Region/genetics
- Leukemia, Lymphocytic, Chronic, B-Cell/mortality
- Leukemia, Lymphocytic, Chronic, B-Cell/pathology
- Male
- Models, Theoretical
- Neoplasm Staging
- Prognosis
- Progression-Free Survival
- Receptors, Antigen, B-Cell/genetics
- Reproducibility of Results
- Tumor Suppressor Protein p53/genetics
Collapse
Affiliation(s)
- Nina Kreuzberger
- Cochrane Haematology, Department I of Internal Medicine, Center for Integrated Oncology Aachen Bonn Cologne Duesseldorf, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany
| | - Johanna Aag Damen
- Cochrane Netherlands, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
| | | | - Lise J Estcourt
- Haematology/Transfusion Medicine, NHS Blood and Transplant, Oxford, UK
| | - Angela Aldin
- Cochrane Haematology, Department I of Internal Medicine, Center for Integrated Oncology Aachen Bonn Cologne Duesseldorf, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany
| | - Lisa Umlauff
- Cochrane Haematology, Department I of Internal Medicine, Center for Integrated Oncology Aachen Bonn Cologne Duesseldorf, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany
| | | | | | - Karel Gm Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
| | - Ina Monsef
- Cochrane Haematology, Department I of Internal Medicine, Center for Integrated Oncology Aachen Bonn Cologne Duesseldorf, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany
| | - Farid Foroutan
- Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Canada
| | - Karl-Anton Kreuzer
- Center of Integrated Oncology Cologne-Bonn, Department I of Internal Medicine, University of Cologne, Faculty of Medicine and University Hospital Cologne, Cologne, Germany
| | - Nicole Skoetz
- Cochrane Cancer, Department I of Internal Medicine, Center for Integrated Oncology Aachen Bonn Cologne Duesseldorf, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany
| |
Collapse
|
693
|
Deelen JWT, Rottier WC, Giron Ortega JA, Rodriguez-Baño J, Harbarth S, Tacconelli E, Jacobsson G, Zahar JR, van Werkhoven CH, Bonten MJM. An international prospective cohort study to validate two prediction rules for infections caused by 3rd-generation cephalosporin-resistant Enterobacterales. Clin Infect Dis 2020; 73:e4475-e4483. [PMID: 32640024 PMCID: PMC8849131 DOI: 10.1093/cid/ciaa950] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2020] [Accepted: 07/03/2020] [Indexed: 11/13/2022] Open
Abstract
Background The possibility of bloodstream infections caused by third-generation cephalosporin-resistant Enterobacterales (3GC-R-BSI) leads to a trade-off between empiric inappropriate treatment (IAT) and unnecessary carbapenem use (UCU). Accurately predicting 3GC-R-BSI could reduce IAT and UCU. We externally validate 2 previously derived prediction rules for community-onset (CO) and hospital-onset (HO) suspected bloodstream infections. Methods In 33 hospitals in 13 countries we prospectively enrolled 200 patients per hospital in whom blood cultures were obtained and intravenous antibiotics with coverage for Enterobacterales were empirically started. Cases were defined as 3GC-R-BSI or 3GC-R gram-negative infection (3GC-R-GNI) (analysis 2); all other outcomes served as a comparator. Model discrimination and calibration were assessed. Impact on carbapenem use was assessed at several cutoff points. Results 4650 CO infection episodes were included and the prevalence of 3GC-R-BSI was 2.1% (n = 97). IAT occurred in 69 of 97 (71.1%) 3GC-R-BSI and UCU in 398 of 4553 non–3GC-R-BSI patients (8.7%). Model calibration was good, and the AUC was .79 (95% CI, .75–.83) for 3GC-R-BSI. The prediction rule potentially reduced IAT to 62% (60/97) while keeping UCU comparable at 8.4% or could reduce UCU to 6.3% (287/4553) while keeping IAT equal. IAT and UCU in all 3GC-R-GNIs (analysis 2) improved at similar percentages. 1683 HO infection episodes were included and the prevalence of 3GC-R-BSI was 4.9% (n = 83). Here model calibration was insufficient. Conclusions A prediction rule for CO 3GC-R infection was validated in an international cohort and could improve empirical antibiotic use. Validation of the HO rule yielded suboptimal performance.
Collapse
Affiliation(s)
- J W Timotëus Deelen
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Wouter C Rottier
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands
| | - José A Giron Ortega
- Unidad Clínica de Enfermedades Infecciosas, Microbiología y Medicina Preventiva, Hospital Universitario Virgen Macarena/Departamento de Medicina, Universidad de Sevilla/Instituto de Biomedicina de Sevilla (IBiS), Seville, Spain
| | - Jesús Rodriguez-Baño
- Unidad Clínica de Enfermedades Infecciosas, Microbiología y Medicina Preventiva, Hospital Universitario Virgen Macarena/Departamento de Medicina, Universidad de Sevilla/Instituto de Biomedicina de Sevilla (IBiS), Seville, Spain
| | - Stephan Harbarth
- Infection Control Program, Geneva University Hospitals and Faculty of Medicine, Geneva, Switzerland
| | - Evelina Tacconelli
- Division of Infectious Diseases, Department of Diagnostic and Public Health, University of Verona, Verona, Italy
| | - Gunnar Jacobsson
- Region Västra Götaland, Skaraborg Hospital, Department of Infectious Diseases, Skövde, Sweden
| | - Jean-Ralph Zahar
- IAME, UMR 1137, Université Paris 13, Sorbonne Paris Cité, France; Service de Microbiologie Clinique et Unité de Contrôle et de Prévention Du Risque Infectieux, Groupe Hospitalier Paris Seine Saint-Denis, AP-HP, Rue de Stalingrad, Bobigny, France
| | - Cornelis H van Werkhoven
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Marc J M Bonten
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands.,Department of Medical Microbiology, University Medical Center Utrecht, Utrecht, the Netherlands
| | | |
Collapse
|
694
|
Affiliation(s)
- Joseph H Schwab
- Department of Orthopedic Surgery, Harvard Medical School, Massachusetts General Hospital, Boston, MA 02114, USA.
| |
Collapse
|
695
|
Fontana M, Carrasco-Labra A, Spallek H, Eckert G, Katz B. Improving Caries Risk Prediction Modeling: A Call for Action. J Dent Res 2020; 99:1215-1220. [PMID: 32600174 DOI: 10.1177/0022034520934808] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/11/2022] Open
Abstract
Dentistry has entered an era of personalized/precision care in which targeting care to groups, individuals, or even tooth surfaces based on their caries risk has become a reality to address the skewed distribution of the disease. The best approach to determine a patient's prognosis relies on the development of caries risk prediction models (CRPMs). A desirable model should be derived and validated to appropriately discriminate between patients who will develop disease from those who will not, and it should provide an accurate estimation of the patient's absolute risk (i.e., calibration). However, evidence suggests there is a need to improve the methodological standards and increase consistency in the way CRPMs are developed and evaluated. In fact, although numerous caries risk assessment tools are available, most are not routinely used in practice or used to influence treatment decisions, and choice is not commonly based on high-quality evidence. Research will propose models that will become more complex, incorporating new factors with high prognostic value (e.g., human genetic markers, microbial biomarkers). Big data and predictive analytic methods will be part of the new approaches for the identification of promising predictors with the ability to monitor patients' risk in real time. Eventually, the implementation of validated, accurate CRPMs will have to follow a user-centered design respecting the patient-clinician dynamic, with no disruption to the clinical workflow, and needs to operate at low cost. The resulting predictive risk estimate needs to be presented to the patient in an understandable way so that it triggers behavior change and effectively informs health care decision making, to ultimately improve caries outcomes. However, research on these later aspects is largely missing and increasingly needed in dentistry.
Collapse
Affiliation(s)
- M Fontana
- Department of Cariology, Restorative Sciences and Endodontics, School of Dentistry, University of Michigan, Ann Arbor, MI, USA
| | - A Carrasco-Labra
- Department of Evidence Synthesis and Translation Research, Science and Research Institute, American Dental Association, Chicago, IL, USA.,Department of Oral and Craniofacial Health Science, School of Dentistry, University of North Carolina at Chapel Hill, NC, USA
| | - H Spallek
- The University of Sydney School of Dentistry, Westmead, New South Wales, Australia
| | - G Eckert
- Department of Biostatistics, School of Medicine and Richard M. Fairbanks School of Public Health, Indiana University, Indianapolis, IN, USA
| | - B Katz
- Department of Biostatistics, School of Medicine and Richard M. Fairbanks School of Public Health, Indiana University, Indianapolis, IN, USA
| |
Collapse
|
696
|
van Smeden M, Groenwold RHH, Moons KG. A cautionary note on the use of the missing indicator method for handling missing data in prediction research. J Clin Epidemiol 2020; 125:188-190. [PMID: 32565213 DOI: 10.1016/j.jclinepi.2020.06.007] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2020] [Accepted: 06/09/2020] [Indexed: 01/07/2023]
Affiliation(s)
- Maarten van Smeden
- Julius Center for Health Science and Primary Care, University Medical Center Utrecht, University of Utrecht, Utrecht, the Netherlands.
| | - Rolf H H Groenwold
- Department of Clinical Epidemiology, Leiden University Medical Centre, Leiden, the Netherlands; Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, the Netherlands
| | - Karel Gm Moons
- Julius Center for Health Science and Primary Care, University Medical Center Utrecht, University of Utrecht, Utrecht, the Netherlands
| |
Collapse
|
697
|
Fang AHS, Lim WT, Balakrishnan T. Early warning score validation methodologies and performance metrics: a systematic review. BMC Med Inform Decis Mak 2020; 20:111. [PMID: 32552702 PMCID: PMC7301346 DOI: 10.1186/s12911-020-01144-8] [Citation(s) in RCA: 22] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2019] [Accepted: 06/03/2020] [Indexed: 01/06/2023] Open
Abstract
BACKGROUND Early warning scores (EWS) have been developed as clinical prognostication tools to identify acutely deteriorating patients. In the past few years, there has been a proliferation of studies that describe the development and validation of novel machine learning-based EWS. Systematic reviews of published studies which focus on evaluating performance of both well-established and novel EWS have shown conflicting conclusions. A possible reason is the heterogeneity in validation methods applied. In this review, we aim to examine the methodologies and metrics used in studies which perform EWS validation. METHODS A systematic review of all eligible studies from the MEDLINE database and other sources, was performed. Studies were eligible if they performed validation on at least one EWS and reported associations between EWS scores and inpatient mortality, intensive care unit (ICU) transfers, or cardiac arrest (CA) of adults. Two reviewers independently did a full-text review and performed data abstraction by using standardized data-worksheet based on the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) checklist. Meta-analysis was not performed due to heterogeneity. RESULTS The key differences in validation methodologies identified were (1) validation dataset used, (2) outcomes of interest, (3) case definition, time of EWS use and aggregation methods, and (4) handling of missing values. In terms of case definition, among the 48 eligible studies, 34 used the patient episode case definition while 12 used the observation set case definition, and 2 did the validation using both case definitions. Of those that used the patient episode case definition, 18 studies validated the EWS at a single point of time, mostly using the first recorded observation. The review also found more than 10 different performance metrics reported among the studies. CONCLUSIONS Methodologies and performance metrics used in studies performing validation on EWS were heterogeneous hence making it difficult to interpret and compare EWS performance. Standardizing EWS validation methodology and reporting can potentially address this issue.
Collapse
Affiliation(s)
| | - Wan Tin Lim
- Department of Internal Medicine, Singapore General Hospital, Singapore, Singapore
| | | |
Collapse
|
698
|
Turner DP, Deng H. A Conceptual Introduction to Regression. Headache 2020; 60:1047-1055. [PMID: 32474925 DOI: 10.1111/head.13834] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2020] [Accepted: 04/20/2020] [Indexed: 01/05/2023]
Affiliation(s)
- Dana P Turner
- Department of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Hao Deng
- Department of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.,Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, USA
| |
Collapse
|
699
|
Adaptation of the prostate biopsy collaborative group risk calculator in patients with PSA less than 10 ng/ml improves its performance. Int Urol Nephrol 2020; 52:1811-1819. [PMID: 32468165 DOI: 10.1007/s11255-020-02517-8] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2020] [Accepted: 05/22/2020] [Indexed: 10/24/2022]
Abstract
PURPOSES The prostate biopsy collaborative group risk calculator (PBCGRC) is a newly developed risk estimator for predicting prostate biopsy outcomes. However, its clinical usefulness is still unknown within the so-called gray area of PSA values. This study aimed to determine whether updating the PBCGRC improves its predictive performance for predicting any-grade and high-grade (HG), defined as biopsy Gleason score ≥ 7, prostate cancer (PCa) in patients with prostate-specific antigen (PSA) less than 10 ng/ml. METHODS The risk of any-grade and HGPCa was calculated using the PBCG risk calculation formulas updated by recalibration in the large, logistic recalibration and model revision. Predictive performances of the PBCGRC and the updated models were compared using discrimination, calibration, and clinical utility. RESULTS Within the study sample of 526 patients, PCa was detected in 193 (36.7%), and 78 (14.8%) of them had HGPCa. According to the calibration curves, the PBCGRC overestimated the risk of PCa. Predictive accuracy of the revised model was higher [the area under the receiver-operating characteristic curve (AUCs), 65.4% and 70.2%] than that of the PBCGRC (AUCs, 60.4% and 64.3%) for any-grade and HGPCa. The net benefit was greater for model revision in comparison with the original model. CONCLUSION The performance accuracy of PBCGRC for the prediction of any and HGPC in men undergoing prostate biopsy with PSA levels below 10 ng/ml is suboptimal. The model revision resulted with significant improvement in model performance. However, external validation of the revised model is necessary before its routine use in clinical practice.
Collapse
|
700
|
Kent DM, Paulus JK, Sharp RR, Hajizadeh N. When predictions are used to allocate scarce health care resources: three considerations for models in the era of Covid-19. Diagn Progn Res 2020; 4:11. [PMID: 32455168 PMCID: PMC7238723 DOI: 10.1186/s41512-020-00079-y] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/13/2020] [Accepted: 04/29/2020] [Indexed: 02/06/2023] Open
Abstract
BACKGROUND The need for life-saving interventions such as mechanical ventilation may threaten to outstrip resources during the Covid-19 pandemic. Allocation of these resources to those most likely to benefit can be supported by clinical prediction models. The ethical and practical considerations relevant to predictions supporting decisions about microallocation are distinct from those that inform shared decision-making in ways important for model design. MAIN BODY We review three issues of importance for microallocation: (1) Prediction of benefit (or of medical futility) may be technically very challenging; (2) When resources are scarce, calibration is less important for microallocation than is ranking to prioritize patients, since capacity determines thresholds for resource utilization; (3) The concept of group fairness, which is not germane in shared decision-making, is of central importance in microallocation. Therefore, model transparency is important. CONCLUSION Prediction supporting allocation of life-saving interventions should be explicit, data-driven, frequently updated and open to public scrutiny. This implies a preference for simple, easily understood and easily applied prognostic models.
Collapse
Affiliation(s)
- David M. Kent
- Predictive Analytics and Comparative Effectiveness (PACE) Center, Tufts Medical Center, Boston, MA USA
| | - Jessica K. Paulus
- Predictive Analytics and Comparative Effectiveness (PACE) Center, Tufts Medical Center, Boston, MA USA
| | | | - Negin Hajizadeh
- Feinstein Institutes for Medical Research, Northwell Health, New York City, NY USA
| |
Collapse
|