101
|
Modern Learning from Big Data in Critical Care: Primum Non Nocere. Neurocrit Care 2022; 37:174-184. [PMID: 35513752 PMCID: PMC9071245 DOI: 10.1007/s12028-022-01510-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2021] [Accepted: 04/06/2022] [Indexed: 12/13/2022]
Abstract
Large and complex data sets are increasingly available for research in critical care. To analyze these data, researchers use techniques commonly referred to as statistical learning or machine learning (ML). The latter is known for large successes in the field of diagnostics, for example, by identification of radiological anomalies. In other research areas, such as clustering and prediction studies, there is more discussion regarding the benefit and efficiency of ML techniques compared with statistical learning. In this viewpoint, we aim to explain commonly used statistical learning and ML techniques and provide guidance for responsible use in the case of clustering and prediction questions in critical care. Clustering studies have been increasingly popular in critical care research, aiming to inform how patients can be characterized, classified, or treated differently. An important challenge for clustering studies is to ensure and assess generalizability. This limits the application of findings in these studies toward individual patients. In the case of predictive questions, there is much discussion as to what algorithm should be used to most accurately predict outcome. Aspects that determine usefulness of ML, compared with statistical techniques, include the volume of the data, the dimensionality of the preferred model, and the extent of missing data. There are areas in which modern ML methods may be preferred. However, efforts should be made to implement statistical frameworks (e.g., for dealing with missing data or measurement error, both omnipresent in clinical data) in ML methods. To conclude, there are important opportunities but also pitfalls to consider when performing clustering or predictive studies with ML techniques. We advocate careful valuation of new data-driven findings. More interaction is needed between the engineer mindset of experts in ML methods, the insight in bias of epidemiologists, and the probabilistic thinking of statisticians to extract as much information and knowledge from data as possible, while avoiding harm.
Collapse
|
102
|
Halvachizadeh S, Störmann PJ, Özkurtul O, Berk T, Teuben M, Sprengel K, Pape HC, Lefering R, Jensen KO. Discrimination and calibration of a prediction model for mortality is decreased in secondary transferred patients: a validation in the TraumaRegister DGU. BMJ Open 2022; 12:e056381. [PMID: 35418430 PMCID: PMC9014053 DOI: 10.1136/bmjopen-2021-056381] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/07/2022] Open
Abstract
INTRODUCTION The Revised Injury Severity Classification II (RISC II) score represents a data-derived score that aims to predict mortality in severely injured patients. The aim of this study was to assess the discrimination and calibration of RISC II in secondary transferred polytrauma patients. METHODS This study was performed on the multicentre database of the TraumaRegister DGU. Inclusion criteria included Injury Severity Score (ISS)≥9 points and complete demographic data. Exclusion criteria included patients with 'do not resuscitate' orders or late transfers (>24 hours after initial trauma). Patients were stratified based on way of admission into patients transferred to a European trauma centre after initial treatment in another hospital (group Tr) and primary admitted patients who were not transferred out (group P). The RISC II score was calculated within each group at admission after secondary transfer (group Tr) and at primary admission (group P) and compared with the observed mortality rate. The calibration and discrimination of prediction were analysed. RESULTS Group P included 116 112 (91%) patients and group Tr included 11 604 (9%) patients. The study population was predominantly male (n=86 280, 70.1%), had a mean age of 53.2 years and a mean ISS of 20.7 points. Patients in group Tr were marginally older (54 years vs 52 years) and a had slightly higher ISS (21.5 points vs 20.1 points). Median time from accident site to hospital admission was 60 min in group P and 241 min (4 hours) in group Tr. Observed and predicted mortality based on RISC II were nearly identical in group P (10.9% and 11.0%, respectively) but predicted mortality was worse (13.4%) than observed mortality (11.1%) in group Tr. CONCLUSION The way of admission alters the calibration of prediction models for mortality in polytrauma patients. Mortality prediction in secondary transferred polytrauma patients should be calculated separately from primary admitted polytrauma patients.
Collapse
Affiliation(s)
- Sascha Halvachizadeh
- Department of Trauma, University Hospital Zurich, Zurich, Switzerland
- Harald Tscherne Research Laboratory for Orthopaedic Trauma, University of Zurich, Faculty of Medicine, Zurich, Switzerland
| | - P J Störmann
- Department of Trauma, Hand, and Reconstructive Surgery, University Hospital Frankfurt/Main, Frankfurt am Main, Germany
| | - Orkun Özkurtul
- Department of Orthopaedic, University Hospital Leipzig, Leipzig, Germany
| | - Till Berk
- Department of Trauma, University Hospital Zurich, Zurich, Switzerland
| | - Michel Teuben
- Department of Trauma, University Hospital Zurich, Zurich, Switzerland
| | - Kai Sprengel
- Department of Trauma, University Hospital Zurich, Zurich, Switzerland
| | | | - Rolf Lefering
- Institute for Research in Operative Medicine (IFOM), Witten/Herdecke University, Witten, Germany
| | - Kai Oliver Jensen
- Department of Trauma, University Hospital Zurich, Zurich, Switzerland
| |
Collapse
|
103
|
McNamara ME, Zisser M, Beevers CG, Shumake J. Not just “big” data: Importance of sample size, measurement error, and uninformative predictors for developing prognostic models for digital interventions. Behav Res Ther 2022; 153:104086. [DOI: 10.1016/j.brat.2022.104086] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/18/2021] [Revised: 03/11/2022] [Accepted: 04/05/2022] [Indexed: 11/24/2022]
|
104
|
Riley RD, Collins GS, Ensor J, Archer L, Booth S, Mozumder SI, Rutherford MJ, van Smeden M, Lambert PC, Snell KIE. Minimum sample size calculations for external validation of a clinical prediction model with a time-to-event outcome. Stat Med 2022; 41:1280-1295. [PMID: 34915593 DOI: 10.1002/sim.9275] [Citation(s) in RCA: 26] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2021] [Revised: 11/15/2021] [Accepted: 11/16/2021] [Indexed: 12/23/2022]
Abstract
Previous articles in Statistics in Medicine describe how to calculate the sample size required for external validation of prediction models with continuous and binary outcomes. The minimum sample size criteria aim to ensure precise estimation of key measures of a model's predictive performance, including measures of calibration, discrimination, and net benefit. Here, we extend the sample size guidance to prediction models with a time-to-event (survival) outcome, to cover external validation in datasets containing censoring. A simulation-based framework is proposed, which calculates the sample size required to target a particular confidence interval width for the calibration slope measuring the agreement between predicted risks (from the model) and observed risks (derived using pseudo-observations to account for censoring) on the log cumulative hazard scale. Precise estimation of calibration curves, discrimination, and net-benefit can also be checked in this framework. The process requires assumptions about the validation population in terms of the (i) distribution of the model's linear predictor and (ii) event and censoring distributions. Existing information can inform this; in particular, the linear predictor distribution can be approximated using the C-index or Royston's D statistic from the model development article, together with the overall event risk. We demonstrate how the approach can be used to calculate the sample size required to validate a prediction model for recurrent venous thromboembolism. Ideally the sample size should ensure precise calibration across the entire range of predicted risks, but must at least ensure adequate precision in regions important for clinical decision-making. Stata and R code are provided.
Collapse
Affiliation(s)
- Richard D Riley
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
- NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | - Joie Ensor
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| | - Lucinda Archer
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| | - Sarah Booth
- Biostatistics Research Group, Department of Health Sciences, George Davies Centre, University of Leicester, Leicester, UK
| | - Sarwar I Mozumder
- Biostatistics Research Group, Department of Health Sciences, George Davies Centre, University of Leicester, Leicester, UK
| | - Mark J Rutherford
- Biostatistics Research Group, Department of Health Sciences, George Davies Centre, University of Leicester, Leicester, UK
| | - Maarten van Smeden
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, University of Utrecht, Utrecht, The Netherlands
| | - Paul C Lambert
- Biostatistics Research Group, Department of Health Sciences, George Davies Centre, University of Leicester, Leicester, UK
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
| | - Kym I E Snell
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| |
Collapse
|
105
|
Haller MC, Aschauer C, Wallisch C, Leffondré K, van Smeden M, Oberbauer R, Heinze G. Prediction models for living organ transplantation are poorly developed, reported and validated: a systematic review. J Clin Epidemiol 2022; 145:126-135. [DOI: 10.1016/j.jclinepi.2022.01.025] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2021] [Revised: 01/27/2022] [Accepted: 01/31/2022] [Indexed: 12/12/2022]
|
106
|
Kontakiotis N, Rushton AB, Billis E, Papathanasiou G, Gioftsos G. Development of a clinical prediction model to inform clinical decision making for classification of patients with sciatica, based on their clinical characteristics, in the Greek health system: protocol for a prospective predictive exploratory study. BMJ Open 2022; 12:e052119. [PMID: 35105622 PMCID: PMC8804639 DOI: 10.1136/bmjopen-2021-052119] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/27/2022] Open
Abstract
INTRODUCTION Sciatica is one of the most common reasons for seeking healthcare for musculoskeletal pain. Sciatica is primarily considered as neuropathic in nature when neural tissue in the low back is compromised, but sometimes other non-neural structures may be involved. Appropriate assessment and management are important for patients with sciatica. Therapists use several outcome measures to assess patients to inform selection of the most suitable treatment. There is limited evidence for the best treatment of sciatica, and this is likely contributed to by having no reliable algorithm to categorise patients based on their clinical characteristics to inform physiotherapy treatment. The purpose of this study is to develop a clinical prediction model to categorise patients with sciatica, in terms of early clinical outcome, based on their initial clinical characteristics. METHODS AND ANALYSIS A prospective observational multicentre design will recruit consecutive patients (n=467) with sciatica referred for physiotherapy. Each patient will be evaluated to determine whether or not they will be accepted into the study by answering some questions that will confirm the study's eligibility criteria. Patients' basic characteristics, patient-reported outcome measures and performance-based measures will be collected at baseline from multiple sites in the Greek territory using this same protocol, prior to commencement of treatment. The main researcher of this study will be responsible for data collection in all sites. On completion of the standard referred physiotherapy treatment after 3 weeks' time, participants will be asked by telephone to evaluate their outcome using the Global Perceived Effect Scale. For the descriptive statistical analysis, the continuous variables will be expressed in the form of 'mean' and 'SD'. In order to assess the prognostic value of each predictor, in terms of the level of improvement or worsening of the symptoms, multiple variable regression analysis will be used. ETHICS AND DISSEMINATION Τhis study is approved from the Ethics and Deontology Committee of the University of West Attica, Athens, Greece, protocol number: 38313-09/06/2020, 10226-10/02/2021. The study's findings will be published in a peer-reviewed journal and disseminated at national and international conferences and through social media. PROSPERO REGISTRATION NUMBER CRD42020168467.
Collapse
Affiliation(s)
- Nikolaos Kontakiotis
- Physiotherapy Department, Laboratory of Advanced Physiotherapy, Faculty of Health and Caring Sciences, University of West Attica, Athens, Greece
| | - Alison B Rushton
- School of Physical Therapy, Western University Faculty of Health Sciences, London, Ontario, Canada
| | - Evdokia Billis
- Physiotherapy Department, School of Health Rehabilitation Sciences, University of Patras, Patra, Greece
| | - George Papathanasiou
- Physiotherapy Department, Laboratory of Neuromuscular and Cardiovascular Study of Motion, Faculty of Health and Caring Sciences, University of West Attica, Athens, Greece
| | - George Gioftsos
- Physiotherapy Department, Laboratory of Advanced Physiotherapy, Faculty of Health and Caring Sciences, University of West Attica, Athens, Greece
| |
Collapse
|
107
|
de Hond AAH, Leeuwenberg AM, Hooft L, Kant IMJ, Nijman SWJ, van Os HJA, Aardoom JJ, Debray TPA, Schuit E, van Smeden M, Reitsma JB, Steyerberg EW, Chavannes NH, Moons KGM. Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review. NPJ Digit Med 2022; 5:2. [PMID: 35013569 PMCID: PMC8748878 DOI: 10.1038/s41746-021-00549-7] [Citation(s) in RCA: 105] [Impact Index Per Article: 52.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/26/2021] [Accepted: 12/13/2021] [Indexed: 12/23/2022] Open
Abstract
While the opportunities of ML and AI in healthcare are promising, the growth of complex data-driven prediction models requires careful quality and applicability assessment before they are applied and disseminated in daily practice. This scoping review aimed to identify actionable guidance for those closely involved in AI-based prediction model (AIPM) development, evaluation and implementation including software engineers, data scientists, and healthcare professionals and to identify potential gaps in this guidance. We performed a scoping review of the relevant literature providing guidance or quality criteria regarding the development, evaluation, and implementation of AIPMs using a comprehensive multi-stage screening strategy. PubMed, Web of Science, and the ACM Digital Library were searched, and AI experts were consulted. Topics were extracted from the identified literature and summarized across the six phases at the core of this review: (1) data preparation, (2) AIPM development, (3) AIPM validation, (4) software development, (5) AIPM impact assessment, and (6) AIPM implementation into daily healthcare practice. From 2683 unique hits, 72 relevant guidance documents were identified. Substantial guidance was found for data preparation, AIPM development and AIPM validation (phases 1-3), while later phases clearly have received less attention (software development, impact assessment and implementation) in the scientific literature. The six phases of the AIPM development, evaluation and implementation cycle provide a framework for responsible introduction of AI-based prediction models in healthcare. Additional domain and technology specific research may be necessary and more practical experience with implementing AIPMs is needed to support further guidance.
Collapse
Affiliation(s)
- Anne A H de Hond
- Department of Information Technology and Digital Innovation, Leiden University Medical Center, Leiden, The Netherlands.
- Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, The Netherlands.
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands.
| | - Artuur M Leeuwenberg
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.
| | - Lotty Hooft
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Ilse M J Kant
- Department of Information Technology and Digital Innovation, Leiden University Medical Center, Leiden, The Netherlands
- Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, The Netherlands
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands
| | - Steven W J Nijman
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Hendrikus J A van Os
- Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, The Netherlands
- National eHealth Living Lab, Leiden, The Netherlands
| | - Jiska J Aardoom
- National eHealth Living Lab, Leiden, The Netherlands
- Department of Public Health and Primary Care, Leiden University Medical Center, Leiden, The Netherlands
| | - Thomas P A Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Ewoud Schuit
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Maarten van Smeden
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Johannes B Reitsma
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Ewout W Steyerberg
- Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, The Netherlands
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands
| | - Niels H Chavannes
- National eHealth Living Lab, Leiden, The Netherlands
- Department of Public Health and Primary Care, Leiden University Medical Center, Leiden, The Netherlands
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
108
|
Homeyer A, Geißler C, Schwen LO, Zakrzewski F, Evans T, Strohmenger K, Westphal M, Bülow RD, Kargl M, Karjauv A, Munné-Bertran I, Retzlaff CO, Romero-López A, Sołtysiński T, Plass M, Carvalho R, Steinbach P, Lan YC, Bouteldja N, Haber D, Rojas-Carulla M, Vafaei Sadr A, Kraft M, Krüger D, Fick R, Lang T, Boor P, Müller H, Hufnagl P, Zerbe N. Recommendations on compiling test datasets for evaluating artificial intelligence solutions in pathology. Mod Pathol 2022; 35:1759-1769. [PMID: 36088478 PMCID: PMC9708586 DOI: 10.1038/s41379-022-01147-y] [Citation(s) in RCA: 24] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2022] [Revised: 07/24/2022] [Accepted: 07/25/2022] [Indexed: 12/24/2022]
Abstract
Artificial intelligence (AI) solutions that automatically extract information from digital histology images have shown great promise for improving pathological diagnosis. Prior to routine use, it is important to evaluate their predictive performance and obtain regulatory approval. This assessment requires appropriate test datasets. However, compiling such datasets is challenging and specific recommendations are missing. A committee of various stakeholders, including commercial AI developers, pathologists, and researchers, discussed key aspects and conducted extensive literature reviews on test datasets in pathology. Here, we summarize the results and derive general recommendations on compiling test datasets. We address several questions: Which and how many images are needed? How to deal with low-prevalence subsets? How can potential bias be detected? How should datasets be reported? What are the regulatory requirements in different countries? The recommendations are intended to help AI developers demonstrate the utility of their products and to help pathologists and regulatory agencies verify reported performance measures. Further research is needed to formulate criteria for sufficiently representative test datasets so that AI solutions can operate with less user intervention and better support diagnostic workflows in the future.
Collapse
Affiliation(s)
- André Homeyer
- Fraunhofer Institute for Digital Medicine MEVIS, Max-von-Laue-Straße 2, 28359, Bremen, Germany.
| | - Christian Geißler
- grid.6734.60000 0001 2292 8254Technische Universität Berlin, DAI-Labor, Ernst-Reuter-Platz 7, 10587 Berlin, Germany
| | - Lars Ole Schwen
- grid.428590.20000 0004 0496 8246Fraunhofer Institute for Digital Medicine MEVIS, Max-von-Laue-Straße 2, 28359 Bremen, Germany
| | - Falk Zakrzewski
- grid.412282.f0000 0001 1091 2917Institute of Pathology, Carl Gustav Carus University Hospital Dresden (UKD), TU Dresden (TUD), Fetscherstrasse 74, 01307 Dresden, Germany
| | - Theodore Evans
- grid.6734.60000 0001 2292 8254Technische Universität Berlin, DAI-Labor, Ernst-Reuter-Platz 7, 10587 Berlin, Germany
| | - Klaus Strohmenger
- grid.6363.00000 0001 2218 4662Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117 Berlin, Germany
| | - Max Westphal
- grid.428590.20000 0004 0496 8246Fraunhofer Institute for Digital Medicine MEVIS, Max-von-Laue-Straße 2, 28359 Bremen, Germany
| | - Roman David Bülow
- grid.412301.50000 0000 8653 1507Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Michaela Kargl
- grid.11598.340000 0000 8988 2476Medical University of Graz, Diagnostic and Research Center for Molecular BioMedicine, Diagnostic & Research Institute of Pathology, Neue Stiftingtalstrasse 6, 8010 Graz, Austria
| | - Aray Karjauv
- grid.6734.60000 0001 2292 8254Technische Universität Berlin, DAI-Labor, Ernst-Reuter-Platz 7, 10587 Berlin, Germany
| | - Isidre Munné-Bertran
- MoticEurope, S.L.U., C. Les Corts, 12 Poligono Industrial, 08349 Barcelona, Spain
| | - Carl Orge Retzlaff
- grid.6734.60000 0001 2292 8254Technische Universität Berlin, DAI-Labor, Ernst-Reuter-Platz 7, 10587 Berlin, Germany
| | | | | | - Markus Plass
- grid.11598.340000 0000 8988 2476Medical University of Graz, Diagnostic and Research Center for Molecular BioMedicine, Diagnostic & Research Institute of Pathology, Neue Stiftingtalstrasse 6, 8010 Graz, Austria
| | - Rita Carvalho
- grid.6363.00000 0001 2218 4662Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117 Berlin, Germany
| | - Peter Steinbach
- grid.40602.300000 0001 2158 0612Helmholtz-Zentrum Dresden Rossendorf, Bautzner Landstraße 400, 01328 Dresden, Germany
| | - Yu-Chia Lan
- grid.412301.50000 0000 8653 1507Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Nassim Bouteldja
- grid.412301.50000 0000 8653 1507Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074 Aachen, Germany
| | - David Haber
- Lakera AI AG, Zelgstrasse 7, 8003 Zürich, Switzerland
| | | | - Alireza Vafaei Sadr
- grid.412301.50000 0000 8653 1507Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074 Aachen, Germany
| | | | - Daniel Krüger
- grid.474385.90000 0004 4676 7928Olympus Soft Imaging Solutions GmbH, Johann-Krane-Weg 39, 48149 Münster, Germany
| | - Rutger Fick
- Tribun Health, 2 Rue du Capitaine Scott, 75015 Paris, France
| | - Tobias Lang
- Mindpeak GmbH, Zirkusweg 2, 20359 Hamburg, Germany
| | - Peter Boor
- grid.412301.50000 0000 8653 1507Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Heimo Müller
- grid.11598.340000 0000 8988 2476Medical University of Graz, Diagnostic and Research Center for Molecular BioMedicine, Diagnostic & Research Institute of Pathology, Neue Stiftingtalstrasse 6, 8010 Graz, Austria
| | - Peter Hufnagl
- grid.6363.00000 0001 2218 4662Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117 Berlin, Germany
| | - Norman Zerbe
- grid.6363.00000 0001 2218 4662Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117 Berlin, Germany
| |
Collapse
|
109
|
Screening for neurocognitive impairment following out-of-hospital cardiac arrest: anyone for a MoCA? Resuscitation 2022; 172:137-138. [DOI: 10.1016/j.resuscitation.2022.01.010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2022] [Accepted: 01/12/2022] [Indexed: 11/23/2022]
|
110
|
Campagner A, Carobene A, Cabitza F. External validation of Machine Learning models for COVID-19 detection based on Complete Blood Count. Health Inf Sci Syst 2021; 9:37. [PMID: 34721844 PMCID: PMC8540880 DOI: 10.1007/s13755-021-00167-3] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/08/2021] [Accepted: 09/29/2021] [Indexed: 01/13/2023] Open
Abstract
PURPOSE The rRT-PCR for COVID-19 diagnosis is affected by long turnaround time, potential shortage of reagents, high false-negative rates and high costs. Routine hematochemical tests are a faster and less expensive alternative for diagnosis. Thus, Machine Learning (ML) has been applied to hematological parameters to develop diagnostic tools and help clinicians in promptly managing positive patients. However, few ML models have been externally validated, making their real-world applicability unclear. METHODS We externally validate 6 state-of-the-art diagnostic ML models, based on Complete Blood Count (CBC) and trained on a dataset encompassing 816 COVID-19 positive cases. The external validation was performed based on two datasets, collected at two different hospitals in northern Italy and encompassing 163 and 104 COVID-19 positive cases, in terms of both error rate and calibration. RESULTS AND CONCLUSION We report an average AUC of 95% and average Brier score of 0.11, out-performing existing ML methods, and showing good cross-site transportability. The best performing model (SVM) reported an average AUC of 97.5% (Sensitivity: 87.5%, Specificity: 94%), comparable with the performance of RT-PCR, and was also the best calibrated. The validated models can be useful in the early identification of potential COVID-19 patients, due to the rapid availability of CBC exams, and in multiple test settings.
Collapse
Affiliation(s)
| | - Anna Carobene
- Laboratory Medicine, IRCCS San Raffaele Scientific Institute, Milan, Italy
| | | |
Collapse
|
111
|
Miles J, Jacques R, Turner J, Mason S. The Safety INdEx of Prehospital On Scene Triage (SINEPOST) study: the development and validation of a risk prediction model to support ambulance clinical transport decisions on-scene-a protocol. Diagn Progn Res 2021; 5:18. [PMID: 34749832 PMCID: PMC8573562 DOI: 10.1186/s41512-021-00108-4] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/06/2021] [Accepted: 10/25/2021] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Demand for both the ambulance service and the emergency department (ED) is rising every year and when this demand is excessive in both systems, ambulance crews queue at the ED waiting to hand patients over. Some transported ambulance patients are 'low-acuity' and do not require the treatment of the ED. However, paramedics can find it challenging to identify these patients accurately. Decision support tools have been developed using expert opinion to help identify these low acuity patients but have failed to show a benefit beyond regular decision-making. Predictive algorithms may be able to build accurate models, which can be used in the field to support the decision not to take a low-acuity patient to an ED. METHODS AND ANALYSIS All patients in Yorkshire who were transported to the ED by ambulance between July 2019 and February 2020 will be included. Ambulance electronic patient care record (ePCR) clinical data will be used as candidate predictors for the model. These will then be linked to the corresponding ED record, which holds the outcome of a 'non-urgent attendance'. The estimated sample size is 52,958, with 4767 events and an EPP of 7.48. An XGBoost algorithm will be used for model development. Initially, a model will be derived using all the data and the apparent performance will be assessed. Then internal-external validation will use non-random nested cross-validation (CV) with test sets held out for each ED (spatial validation). After all models are created, a random-effects meta-analysis will be undertaken. This will pool performance measures such as goodness of fit, discrimination and calibration. It will also generate a prediction interval and measure heterogeneity between clusters. The performance of the full model will be updated with the pooled results. DISCUSSION Creating a risk prediction model in this area will lead to further development of a clinical decision support tool that ensures every ambulance patient can get to the right place of care, first time. If this study is successful, it could help paramedics evaluate the benefit of transporting a patient to the ED before they leave the scene. It could also reduce congestion in the urgent and emergency care system. TRIAL REGISTRATION This study was retrospectively registered with the ISRCTN: 12121281.
Collapse
Affiliation(s)
- Jamie Miles
- CURE Group, School of Health and Related Research, University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK.
- Yorkshire Ambulance Service, Brindley Way, Wakefield, WF2 0XQ, UK.
| | - Richard Jacques
- School of Health and Related Research, University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK
| | - Janette Turner
- CURE Group, School of Health and Related Research, University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK
| | - Suzanne Mason
- CURE Group, School of Health and Related Research, University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK
| |
Collapse
|
112
|
Sun H, Depraetere K, Meesseman L, Cabanillas Silva P, Szymanowsky R, Fliegenschmidt J, Hulde N, von Dossow V, Vanbiervliet M, De Baerdemaeker J, Roccaro-Waldmeyer DM, Stieg J, Domínguez Hidalgo M, Dahlweid FM. Evaluating live performance of machine learning based prediction models for different clinical risks: a study of live systems in different hospitals (Preprint). J Med Internet Res 2021; 24:e34295. [PMID: 35502887 PMCID: PMC9214618 DOI: 10.2196/34295] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2021] [Revised: 02/25/2022] [Accepted: 04/12/2022] [Indexed: 11/30/2022] Open
Abstract
Background Machine learning algorithms are currently used in a wide array of clinical domains to produce models that can predict clinical risk events. Most models are developed and evaluated with retrospective data, very few are evaluated in a clinical workflow, and even fewer report performances in different hospitals. In this study, we provide detailed evaluations of clinical risk prediction models in live clinical workflows for three different use cases in three different hospitals. Objective The main objective of this study was to evaluate clinical risk prediction models in live clinical workflows and compare their performance in these setting with their performance when using retrospective data. We also aimed at generalizing the results by applying our investigation to three different use cases in three different hospitals. Methods We trained clinical risk prediction models for three use cases (ie, delirium, sepsis, and acute kidney injury) in three different hospitals with retrospective data. We used machine learning and, specifically, deep learning to train models that were based on the Transformer model. The models were trained using a calibration tool that is common for all hospitals and use cases. The models had a common design but were calibrated using each hospital’s specific data. The models were deployed in these three hospitals and used in daily clinical practice. The predictions made by these models were logged and correlated with the diagnosis at discharge. We compared their performance with evaluations on retrospective data and conducted cross-hospital evaluations. Results The performance of the prediction models with data from live clinical workflows was similar to the performance with retrospective data. The average value of the area under the receiver operating characteristic curve (AUROC) decreased slightly by 0.6 percentage points (from 94.8% to 94.2% at discharge). The cross-hospital evaluations exhibited severely reduced performance: the average AUROC decreased by 8 percentage points (from 94.2% to 86.3% at discharge), which indicates the importance of model calibration with data from the deployment hospital. Conclusions Calibrating the prediction model with data from different deployment hospitals led to good performance in live settings. The performance degradation in the cross-hospital evaluation identified limitations in developing a generic model for different hospitals. Designing a generic process for model development to generate specialized prediction models for each hospital guarantees model performance in different hospitals.
Collapse
Affiliation(s)
- Hong Sun
- Dedalus Healthcare, Antwerp, Belgium
| | | | | | | | | | - Janis Fliegenschmidt
- Institute of Anesthesiology and Pain Therapy, Heart and Diabetes Centre North Rhine-Westphalia, University Hospital of Ruhr-University Bochum, Bad Oeynhausen, Germany
| | - Nikolai Hulde
- Institute of Anesthesiology and Pain Therapy, Heart and Diabetes Centre North Rhine-Westphalia, University Hospital of Ruhr-University Bochum, Bad Oeynhausen, Germany
| | - Vera von Dossow
- Institute of Anesthesiology and Pain Therapy, Heart and Diabetes Centre North Rhine-Westphalia, University Hospital of Ruhr-University Bochum, Bad Oeynhausen, Germany
| | | | | | | | | | | | | |
Collapse
|
113
|
López-Jiménez N, García-Sánchez F, Hernández-Pailos R, Rodrigo-Álvaro V, Pascual-Pedreño A, Moreno-Cid M, Delgado-Rodríguez M, Hernández-Martínez A. Risk of caesarean delivery in labour induction: a systematic review and external validation of predictive models. BJOG 2021; 129:685-695. [PMID: 34559942 DOI: 10.1111/1471-0528.16947] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/21/2021] [Indexed: 11/30/2022]
Abstract
BACKGROUND Despite the existence of numerous published models predicting the risk of caesarean delivery in women undergoing induction of labour (IOL), validated models are scarce. OBJECTIVES To systematically review and externally assess the predictive capacity of caesarean delivery risk models in women undergoing IOL. SEARCH STRATEGY Studies published up to 15 January 2021 were identified through PubMed, CINAHL, Scopus and ClinicalTrials.gov, without temporal or language restrictions. SELECTION CRITERIA Studies describing the derivation of new models for predicting the risk of caesarean delivery in labour induction. DATA COLLECTION AND ANALYSIS Three authors independently screened the articles and assessed the risk of bias (ROB) according to the prediction model risk of bias assessment tool (PROBAST). External validation was performed in a prospective cohort of 468 pregnancies undergoing IOL from February 2019 to August 2020. The predictive capacity of the models was assessed by creating areas under the receiver operating characteristic curve (AUCs), calibration plots and decision curve analysis (DCA). MAIN RESULTS Fifteen studies met the eligibility criteria; 12 predictive models were validated. The quality of most of the included studies was not adequate. The AUC of the models varied from 0.520 to 0.773. The three models with the best discriminative capacity were those of Levine et al. (AUC 0.773, 95% CI 0.720-0.827), Hernández et al. (AUC 0.762, 95% CI 0.715-0.809) and Rossi et al. (AUC 0.752, 95% CI 0.707-0.797). CONCLUSIONS Predictive capacity and methodological quality were limited; therefore, we cannot currently recommend the use of any of the models for decision making in clinical practice. TWEETABLE ABSTRACT Predictive models that predict the risk of cesarean section in labor inductions are currently not applicable.
Collapse
Affiliation(s)
- N López-Jiménez
- Department of Obstetrics and Gynaecology, La Mancha Centro Hospital, Alcázar de San Juan, Ciudad Real, Spain
| | - F García-Sánchez
- Department of Obstetrics and Gynaecology, La Mancha Centro Hospital, Alcázar de San Juan, Ciudad Real, Spain
| | - R Hernández-Pailos
- Department of Obstetrics and Gynaecology, La Mancha Centro Hospital, Alcázar de San Juan, Ciudad Real, Spain
| | - V Rodrigo-Álvaro
- Department of Obstetrics and Gynaecology, La Mancha Centro Hospital, Alcázar de San Juan, Ciudad Real, Spain
| | - A Pascual-Pedreño
- Department of Obstetrics and Gynaecology, La Mancha Centro Hospital, Alcázar de San Juan, Ciudad Real, Spain
| | - M Moreno-Cid
- Department of Obstetrics and Gynaecology, La Mancha Centro Hospital, Alcázar de San Juan, Ciudad Real, Spain
| | - M Delgado-Rodríguez
- Consortium for Biomedical Research in Epidemiology and Public Health (CIBERESP), Madrid, Spain.,Department of Health Sciences, University of Jaen, Jaen, Spain
| | - A Hernández-Martínez
- Department of Obstetrics and Gynaecology, La Mancha Centro Hospital, Alcázar de San Juan, Ciudad Real, Spain.,Department of Nursing, Faculty of Nursing of Ciudad Real, University of Castilla-La Mancha, Ciudad Real, Spain
| |
Collapse
|
114
|
Cabitza F, Campagner A, Soares F, García de Guadiana-Romualdo L, Challa F, Sulejmani A, Seghezzi M, Carobene A. The importance of being external. methodological insights for the external validation of machine learning models in medicine. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 208:106288. [PMID: 34352688 DOI: 10.1016/j.cmpb.2021.106288] [Citation(s) in RCA: 59] [Impact Index Per Article: 19.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/03/2021] [Accepted: 07/09/2021] [Indexed: 06/13/2023]
Abstract
UNLABELLED Background and Objective Medical machine learning (ML) models tend to perform better on data from the same cohort than on new data, often due to overfitting, or co-variate shifts. For these reasons, external validation (EV) is a necessary practice in the evaluation of medical ML. However, there is still a gap in the literature on how to interpret EV results and hence assess the robustness of ML models. METHODS We fill this gap by proposing a meta-validation method, to assess the soundness of EV procedures. In doing so, we complement the usual way to assess EV by considering both dataset cardinality, and the similarity of the EV dataset with respect to the training set. We then investigate how the notions of cardinality and similarity can be used to inform on the reliability of a validation procedure, by integrating them into two summative data visualizations. RESULTS We illustrate our methodology by applying it to the validation of a state-of-the-art COVID-19 diagnostic model on 8 EV sets, collected across 3 different continents. The model performance was moderately impacted by data similarity (Pearson ρ = 0.38, p< 0.001). In the EV, the validated model reported good AUC (average: 0.84), acceptable calibration (average: 0.17) and utility (average: 0.50). The validation datasets were adequate in terms of dataset cardinality and similarity, thus suggesting the soundness of the results. We also provide a qualitative guideline to evaluate the reliability of validation procedures, and we discuss the importance of proper external validation in light of the obtained results. CONCLUSIONS In this paper, we propose a novel, lean methodology to: 1) study how the similarity between training and validation sets impacts the generalizability of a ML model; 2) assess the soundness of EV evaluations along three complementary performance dimensions: discrimination, utility and calibration; 3) draw conclusions on the robustness of the model under validation. We applied this methodology to a state-of-the-art model for the diagnosis of COVID-19 from routine blood tests, and showed how to interpret the results in light of the presented framework.
Collapse
Affiliation(s)
- Federico Cabitza
- University of Milano-Bicocca, Viale Sarca 336, Milano, 20126, Italy.
| | - Andrea Campagner
- University of Milano-Bicocca, Viale Sarca 336, Milano, 20126, Italy
| | - Felipe Soares
- Department of Industrial Engineering - Universidade Federal do Rio Grande do Sul. Porto Alegre, Brazil
| | | | - Feyissa Challa
- National Reference Laboratory for Clinical Chemistry, Ethiopian Public Health Institute, Addis Ababa, Ethiopia
| | - Adela Sulejmani
- Laboratorio di chimica clinica, Ospedale di Desio e Monza, ASST-Monza, Dipartimento di medicina e chirurgia, Universit di Milano-Bicocca, Monza, Italy
| | - Michela Seghezzi
- Laboratorio di chimica clinica, Ospedale Papa Giovanni XXIII, Bergamo, Italy
| | - Anna Carobene
- Laboratory Medicine, IRCCS San Raffaele Scientific Institute, Milan, Italy
| |
Collapse
|
115
|
Heinze G, van Smeden M, Wynants L, Steyerberg E, van Calster B. Prediction models: stepwise development and simultaneous validation is a step back. J Clin Epidemiol 2021; 142:330-331. [PMID: 34348179 DOI: 10.1016/j.jclinepi.2021.07.019] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2021] [Accepted: 07/28/2021] [Indexed: 12/23/2022]
Affiliation(s)
- Georg Heinze
- Section for Clinical Biometrics, Center for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna, Spitalgasse 23, 1090 Vienna, Austria.
| | - Maarten van Smeden
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Laure Wynants
- Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Netherlands; Department of Development and Regenaration, KU Leuven, Leuven, Belgium; EPI-center, KU Leuven, Belgium
| | - Ewout Steyerberg
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
| | - Ben van Calster
- Department of Development and Regenaration, KU Leuven, Leuven, Belgium; EPI-center, KU Leuven, Belgium; Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
| |
Collapse
|
116
|
Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, Bonten MMJ, Dahly DL, Damen JAA, Debray TPA, de Jong VMT, De Vos M, Dhiman P, Haller MC, Harhay MO, Henckaerts L, Heus P, Kammer M, Kreuzberger N, Lohmann A, Luijken K, Ma J, Martin GP, McLernon DJ, Andaur Navarro CL, Reitsma JB, Sergeant JC, Shi C, Skoetz N, Smits LJM, Snell KIE, Sperrin M, Spijker R, Steyerberg EW, Takada T, Tzoulaki I, van Kuijk SMJ, van Bussel B, van der Horst ICC, van Royen FS, Verbakel JY, Wallisch C, Wilkinson J, Wolff R, Hooft L, Moons KGM, van Smeden M. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 2020; 369:m1328. [PMID: 32265220 PMCID: PMC7222643 DOI: 10.1136/bmj.m1328] [Citation(s) in RCA: 1647] [Impact Index Per Article: 411.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 03/31/2020] [Indexed: 12/12/2022]
Abstract
OBJECTIVE To review and appraise the validity and usefulness of published and preprint reports of prediction models for diagnosing coronavirus disease 2019 (covid-19) in patients with suspected infection, for prognosis of patients with covid-19, and for detecting people in the general population at increased risk of covid-19 infection or being admitted to hospital with the disease. DESIGN Living systematic review and critical appraisal by the COVID-PRECISE (Precise Risk Estimation to optimise covid-19 Care for Infected or Suspected patients in diverse sEttings) group. DATA SOURCES PubMed and Embase through Ovid, up to 1 July 2020, supplemented with arXiv, medRxiv, and bioRxiv up to 5 May 2020. STUDY SELECTION Studies that developed or validated a multivariable covid-19 related prediction model. DATA EXTRACTION At least two authors independently extracted data using the CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist; risk of bias was assessed using PROBAST (prediction model risk of bias assessment tool). RESULTS 37 421 titles were screened, and 169 studies describing 232 prediction models were included. The review identified seven models for identifying people at risk in the general population; 118 diagnostic models for detecting covid-19 (75 were based on medical imaging, 10 to diagnose disease severity); and 107 prognostic models for predicting mortality risk, progression to severe disease, intensive care unit admission, ventilation, intubation, or length of hospital stay. The most frequent types of predictors included in the covid-19 prediction models are vital signs, age, comorbidities, and image features. Flu-like symptoms are frequently predictive in diagnostic models, while sex, C reactive protein, and lymphocyte counts are frequent prognostic factors. Reported C index estimates from the strongest form of validation available per model ranged from 0.71 to 0.99 in prediction models for the general population, from 0.65 to more than 0.99 in diagnostic models, and from 0.54 to 0.99 in prognostic models. All models were rated at high or unclear risk of bias, mostly because of non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, high risk of model overfitting, and unclear reporting. Many models did not include a description of the target population (n=27, 12%) or care setting (n=75, 32%), and only 11 (5%) were externally validated by a calibration plot. The Jehi diagnostic model and the 4C mortality score were identified as promising models. CONCLUSION Prediction models for covid-19 are quickly entering the academic literature to support medical decision making at a time when they are urgently needed. This review indicates that almost all pubished prediction models are poorly reported, and at high risk of bias such that their reported predictive performance is probably optimistic. However, we have identified two (one diagnostic and one prognostic) promising models that should soon be validated in multiple cohorts, preferably through collaborative efforts and data sharing to also allow an investigation of the stability and heterogeneity in their performance across populations and settings. Details on all reviewed models are publicly available at https://www.covprecise.org/. Methodological guidance as provided in this paper should be followed because unreliable predictions could cause more harm than benefit in guiding clinical decisions. Finally, prediction model authors should adhere to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) reporting guideline. SYSTEMATIC REVIEW REGISTRATION Protocol https://osf.io/ehc47/, registration https://osf.io/wy245. READERS' NOTE This article is a living systematic review that will be updated to reflect emerging evidence. Updates may occur for up to two years from the date of original publication. This version is update 3 of the original article published on 7 April 2020 (BMJ 2020;369:m1328). Previous updates can be found as data supplements (https://www.bmj.com/content/369/bmj.m1328/related#datasupp). When citing this paper please consider adding the update number and date of access for clarity.
Collapse
Affiliation(s)
- Laure Wynants
- Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Peter Debyeplein 1, 6229 HA Maastricht, Netherlands
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
| | - Ben Van Calster
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Musculoskeletal Sciences, University of Oxford, Oxford, UK
- NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, UK
| | - Richard D Riley
- Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Keele, UK
| | - Georg Heinze
- Section for Clinical Biometrics, Centre for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna, Vienna, Austria
| | - Ewoud Schuit
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Marc M J Bonten
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- Department of Medical Microbiology, University Medical Centre Utrecht, Utrecht, Netherlands
| | - Darren L Dahly
- HRB Clinical Research Facility, Cork, Ireland
- School of Public Health, University College Cork, Cork, Ireland
| | - Johanna A A Damen
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Thomas P A Debray
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Valentijn M T de Jong
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Maarten De Vos
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Electrical Engineering, ESAT Stadius, KU Leuven, Leuven, Belgium
| | - Paul Dhiman
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Musculoskeletal Sciences, University of Oxford, Oxford, UK
- NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, UK
| | - Maria C Haller
- Section for Clinical Biometrics, Centre for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna, Vienna, Austria
- Ordensklinikum Linz, Hospital Elisabethinen, Department of Nephrology, Linz, Austria
| | - Michael O Harhay
- Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Palliative and Advanced Illness Research Center and Division of Pulmonary and Critical Care Medicine, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Liesbet Henckaerts
- Department of Microbiology, Immunology and Transplantation, KU Leuven-University of Leuven, Leuven, Belgium
- Department of General Internal Medicine, KU Leuven-University Hospitals Leuven, Leuven, Belgium
| | - Pauline Heus
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Michael Kammer
- Section for Clinical Biometrics, Centre for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna, Vienna, Austria
- Department of Nephrology, Medical University of Vienna, Vienna, Austria
| | - Nina Kreuzberger
- Evidence-Based Oncology, Department I of Internal Medicine and Centre for Integrated Oncology Aachen Bonn Cologne Dusseldorf, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany
| | - Anna Lohmann
- Department of Clinical Epidemiology, Leiden University Medical Centre, Leiden, Netherlands
| | - Kim Luijken
- Department of Clinical Epidemiology, Leiden University Medical Centre, Leiden, Netherlands
| | - Jie Ma
- NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, UK
| | - Glen P Martin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
| | - David J McLernon
- Institute of Applied Health Sciences, University of Aberdeen, Aberdeen, UK
| | - Constanza L Andaur Navarro
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Johannes B Reitsma
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Jamie C Sergeant
- Centre for Biostatistics, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
- Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Chunhu Shi
- Division of Nursing, Midwifery and Social Work, School of Health Sciences, University of Manchester, Manchester, UK
| | - Nicole Skoetz
- Department of Nephrology, Medical University of Vienna, Vienna, Austria
| | - Luc J M Smits
- Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Peter Debyeplein 1, 6229 HA Maastricht, Netherlands
| | - Kym I E Snell
- Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Keele, UK
| | - Matthew Sperrin
- Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK
| | - René Spijker
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- Amsterdam UMC, University of Amsterdam, Amsterdam Public Health, Medical Library, Netherlands
| | - Ewout W Steyerberg
- Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands
| | - Toshihiko Takada
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Ioanna Tzoulaki
- Department of Epidemiology and Biostatistics, Imperial College London School of Public Health, London, UK
- Department of Hygiene and Epidemiology, University of Ioannina Medical School, Ioannina, Greece
| | - Sander M J van Kuijk
- Department of Clinical Epidemiology and Medical Technology Assessment, Maastricht University Medical Centre+, Maastricht, Netherlands
| | - Bas van Bussel
- Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Peter Debyeplein 1, 6229 HA Maastricht, Netherlands
- Department of Intensive Care, Maastricht University Medical Centre+, Maastricht University, Maastricht, Netherlands
| | - Iwan C C van der Horst
- Department of Intensive Care, Maastricht University Medical Centre+, Maastricht University, Maastricht, Netherlands
| | - Florien S van Royen
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Jan Y Verbakel
- EPI-Centre, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
| | - Christine Wallisch
- Section for Clinical Biometrics, Centre for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna, Vienna, Austria
- Charité Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, Berlin, Germany
- Berlin Institute of Health, Berlin, Germany
| | - Jack Wilkinson
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
| | | | - Lotty Hooft
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Maarten van Smeden
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| |
Collapse
|