1
|
Palmowski L, Nowak H, Witowski A, Koos B, Wolf A, Weber M, Kleefisch D, Unterberg M, Haberl H, von Busch A, Ertmer C, Zarbock A, Bode C, Putensen C, Limper U, Wappler F, Köhler T, Henzler D, Oswald D, Ellger B, Ehrentraut SF, Bergmann L, Rump K, Ziehe D, Babel N, Sitek B, Marcus K, Frey UH, Thoral PJ, Adamzik M, Eisenacher M, Rahmel T. Assessing SOFA score trajectories in sepsis using machine learning: A pragmatic approach to improve the accuracy of mortality prediction. PLoS One 2024; 19:e0300739. [PMID: 38547245 PMCID: PMC10977876 DOI: 10.1371/journal.pone.0300739] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2023] [Accepted: 03/04/2024] [Indexed: 04/01/2024] Open
Abstract
INTRODUCTION An increasing amount of longitudinal health data is available on critically ill septic patients in the age of digital medicine, including daily sequential organ failure assessment (SOFA) score measurements. Thus, the assessment in sepsis focuses increasingly on the evaluation of the individual disease's trajectory. Machine learning (ML) algorithms may provide a promising approach here to improve the evaluation of daily SOFA score dynamics. We tested whether ML algorithms can outperform the conventional ΔSOFA score regarding the accuracy of 30-day mortality prediction. METHODS We used the multicentric SepsisDataNet.NRW study cohort that prospectively enrolled 252 sepsis patients between 03/2018 and 09/2019 for training ML algorithms, i.e. support vector machine (SVM) with polynomial kernel and artificial neural network (aNN). We used the Amsterdam UMC database covering 1,790 sepsis patients for external and independent validation. RESULTS Both SVM (AUC 0.84; 95% CI: 0.71-0.96) and aNN (AUC 0.82; 95% CI: 0.69-0.95) assessing the SOFA scores of the first seven days led to a more accurate prognosis of 30-day mortality compared to the ΔSOFA score between day 1 and 7 (AUC 0.73; 95% CI: 0.65-0.80; p = 0.02 and p = 0.05, respectively). These differences were even more prominent the shorter the time interval considered. Using the SOFA scores of day 1 to 3 SVM (AUC 0.82; 95% CI: 0.68 0.95) and aNN (AUC 0.80; 95% CI: 0.660.93) led to a more accurate prognosis of 30-day mortality compared to the ΔSOFA score (AUC 0.66; 95% CI: 0.58-0.74; p < 0.01 and p < 0.01, respectively). Strikingly, all these findings could be confirmed in the independent external validation cohort. CONCLUSIONS The ML-based algorithms using daily SOFA scores markedly improved the accuracy of mortality compared to the conventional ΔSOFA score. Therefore, this approach could provide a promising and automated approach to assess the individual disease trajectory in sepsis. These findings reflect the potential of incorporating ML algorithms as robust and generalizable support tools on intensive care units.
Collapse
Affiliation(s)
- Lars Palmowski
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
| | - Hartmuth Nowak
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
- Zentrum für Künstliche Intelligenz, Medizininformatik und Datenwissenschaften, Universitätsklinikum Knappschaftskrankenhaus Bochum, Bochum, Germany
| | - Andrea Witowski
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
| | - Björn Koos
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
| | - Alexander Wolf
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
| | - Maike Weber
- Medizinische Fakultät, Medizinisches Proteom-Center, Ruhr Universität Bochum, Bochum, Germany
- Zentrum für Proteindiagnostik (PRODI), Ruhr Universität Bochum, Bochum, Germany
| | - Daniel Kleefisch
- Medizinische Fakultät, Medizinisches Proteom-Center, Ruhr Universität Bochum, Bochum, Germany
- Zentrum für Proteindiagnostik (PRODI), Ruhr Universität Bochum, Bochum, Germany
| | - Matthias Unterberg
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
| | - Helge Haberl
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
| | - Alexander von Busch
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
| | - Christian Ertmer
- Klinik für Anästhesiologie, Operative Intensivmedizin und Schmerztherapie, Universitätsklinikum Münster, Münster, Germany
| | - Alexander Zarbock
- Klinik für Anästhesiologie, Operative Intensivmedizin und Schmerztherapie, Universitätsklinikum Münster, Münster, Germany
| | - Christian Bode
- Klinik für Anästhesiologie und Operative Intensivmedizin, Universitätsklinikum Bonn, Bonn, Germany
| | - Christian Putensen
- Klinik für Anästhesiologie und Operative Intensivmedizin, Universitätsklinikum Bonn, Bonn, Germany
| | - Ulrich Limper
- Klinik für Anästhesiologie und Operative Intensivmedizin, Universität Witten/Herdecke, Krankenhaus Köln-Merheim, Köln, Germany
| | - Frank Wappler
- Klinik für Anästhesiologie und Operative Intensivmedizin, Universität Witten/Herdecke, Krankenhaus Köln-Merheim, Köln, Germany
| | - Thomas Köhler
- Klinik für Anästhesiologie und Operative Intensiv-, Rettungsmedizin und Schmerztherapie, Klinikum Herford, Herford, Germany
- Klinik für Anästhesiologie und Intensivmedizin, AMEOS-Klinikum Halberstadt, Halberstadt, Germany
| | - Dietrich Henzler
- Klinik für Anästhesiologie und Operative Intensiv-, Rettungsmedizin und Schmerztherapie, Klinikum Herford, Herford, Germany
| | - Daniel Oswald
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Klinikum Westfalen, Dortmund, Germany
| | - Björn Ellger
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Klinikum Westfalen, Dortmund, Germany
| | - Stefan F. Ehrentraut
- Klinik für Anästhesiologie und Operative Intensivmedizin, Universitätsklinikum Bonn, Bonn, Germany
| | - Lars Bergmann
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
| | - Katharina Rump
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
| | - Dominik Ziehe
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
| | - Nina Babel
- Centrum für Translationale Medizin, Medizinische Klinik I, Marien Hospital Herne, Universitätsklinikum der Ruhr-Universität Bochum, Herne, Germany
| | - Barbara Sitek
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
- Medizinische Fakultät, Medizinisches Proteom-Center, Ruhr Universität Bochum, Bochum, Germany
| | - Katrin Marcus
- Medizinische Fakultät, Medizinisches Proteom-Center, Ruhr Universität Bochum, Bochum, Germany
| | - Ulrich H. Frey
- Klinik für Anästhesiologie, Operative Intensivmedizin, Schmerz- und Palliativmedizin, Marien Hospital Herne, Universitätsklinikum der Ruhr-Universität Bochum, Bochum, Germany
| | - Patrick J. Thoral
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Cardiovascular Science (ACS), Amsterdam Infection and Immunity Institute (AI&II), Amsterdam UMC, Location VUmc, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Michael Adamzik
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
| | - Martin Eisenacher
- Medizinische Fakultät, Medizinisches Proteom-Center, Ruhr Universität Bochum, Bochum, Germany
- Zentrum für Proteindiagnostik (PRODI), Ruhr Universität Bochum, Bochum, Germany
| | - Tim Rahmel
- Klinik für Anästhesiologie, Intensivmedizin und Schmerztherapie, Universitätsklinikum Knappschaftskrankenhaus Bochum, Ruhr Universität Bochum, Bochum, Germany
| | | |
Collapse
|
2
|
Otten M, Jagesar AR, Dam TA, Biesheuvel LA, den Hengst F, Ziesemer KA, Thoral PJ, de Grooth HJ, Girbes ARJ, François-Lavet V, Hoogendoorn M, Elbers PWG. Does Reinforcement Learning Improve Outcomes for Critically Ill Patients? A Systematic Review and Level-of-Readiness Assessment. Crit Care Med 2024; 52:e79-e88. [PMID: 37938042 DOI: 10.1097/ccm.0000000000006100] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2023]
Abstract
OBJECTIVE Reinforcement learning (RL) is a machine learning technique uniquely effective at sequential decision-making, which makes it potentially relevant to ICU treatment challenges. We set out to systematically review, assess level-of-readiness and meta-analyze the effect of RL on outcomes for critically ill patients. DATA SOURCES A systematic search was performed in PubMed, Embase.com, Clarivate Analytics/Web of Science Core Collection, Elsevier/SCOPUS and the Institute of Electrical and Electronics Engineers Xplore Digital Library from inception to March 25, 2022, with subsequent citation tracking. DATA EXTRACTION Journal articles that used an RL technique in an ICU population and reported on patient health-related outcomes were included for full analysis. Conference papers were included for level-of-readiness assessment only. Descriptive statistics, characteristics of the models, outcome compared with clinician's policy and level-of-readiness were collected. RL-health risk of bias and applicability assessment was performed. DATA SYNTHESIS A total of 1,033 articles were screened, of which 18 journal articles and 18 conference papers, were included. Thirty of those were prototyping or modeling articles and six were validation articles. All articles reported RL algorithms to outperform clinical decision-making by ICU professionals, but only in retrospective data. The modeling techniques for the state-space, action-space, reward function, RL model training, and evaluation varied widely. The risk of bias was high in all articles, mainly due to the evaluation procedure. CONCLUSION In this first systematic review on the application of RL in intensive care medicine we found no studies that demonstrated improved patient outcomes from RL-based technologies. All studies reported that RL-agent policies outperformed clinician policies, but such assessments were all based on retrospective off-policy evaluation.
Collapse
Affiliation(s)
- Martijn Otten
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | - Ameet R Jagesar
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | - Tariq A Dam
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | - Laurens A Biesheuvel
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | - Floris den Hengst
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | | | - Patrick J Thoral
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Harm-Jan de Grooth
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Armand R J Girbes
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Vincent François-Lavet
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | - Mark Hoogendoorn
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | - Paul W G Elbers
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| |
Collapse
|
3
|
Dam TA, Fleuren LM, Roggeveen LF, Otten M, Biesheuvel L, Jagesar AR, Lalisang RCA, Kullberg RFJ, Hendriks T, Girbes ARJ, Hoogendoorn M, Thoral PJ, Elbers PWG. Augmented intelligence facilitates concept mapping across different electronic health records. Int J Med Inform 2023; 179:105233. [PMID: 37748329 DOI: 10.1016/j.ijmedinf.2023.105233] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2023] [Revised: 09/15/2023] [Accepted: 09/21/2023] [Indexed: 09/27/2023]
Abstract
INTRODUCTION With the advent of artificial intelligence, the secondary use of routinely collected medical data from electronic healthcare records (EHR) has become increasingly popular. However, different EHR systems typically use different names for the same medical concepts. This obviously hampers scalable model development and subsequent clinical implementation for decision support. Therefore, converting original parameter names to a so-called ontology, a standardized set of predefined concepts, is necessary but time-consuming and labor-intensive. We therefore propose an augmented intelligence approach to facilitate ontology alignment by predicting correct concepts based on parameter names from raw electronic health record data exports. METHODS We used the manually mapped parameter names from the multicenter "Dutch ICU data warehouse against COVID-19" sourced from three types of EHR systems to train machine learning models for concept mapping. Data from 29 intensive care units on 38,824 parameters mapped to 1,679 relevant and unique concepts and 38,069 parameters labeled as irrelevant were used for model development and validation. We used the Natural Language Toolkit (NLTK) to preprocess the parameter names based on WordNet cognitive synonyms transformed by term-frequency inverse document frequency (TF-IDF), yielding numeric features. We then trained linear classifiers using stochastic gradient descent for multi-class prediction. Finally, we fine-tuned these predictions using information on distributions of the data associated with each parameter name through similarity score and skewness comparisons. RESULTS The initial model, trained using data from one hospital organization for each of three EHR systems, scored an overall top 1 precision of 0.744, recall of 0.771, and F1-score of 0.737 on a total of 58,804 parameters. Leave-one-hospital-out analysis returned an average top 1 recall of 0.680 for relevant parameters, which increased to 0.905 for the top 5 predictions. When reducing the training dataset to only include relevant parameters, top 1 recall was 0.811 and top 5 recall was 0.914 for relevant parameters. Performance improvement based on similarity score or skewness comparisons affected at most 5.23% of numeric parameters. CONCLUSION Augmented intelligence is a promising method to improve concept mapping of parameter names from raw electronic health record data exports. We propose a robust method for mapping data across various domains, facilitating the integration of diverse data sources. However, recall is not perfect, and therefore manual validation of mapping remains essential.
Collapse
Affiliation(s)
- Tariq A Dam
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence (C4I), Amsterdam Medical Data Science (AMDS), Amsterdam Public Health (APH), Amsterdam Cardiovascular Science (ACS), Amsterdam Institute for Infection and Immunity (AII), Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands; Pacmed, Amsterdam, the Netherlands.
| | - Lucas M Fleuren
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence (C4I), Amsterdam Medical Data Science (AMDS), Amsterdam Public Health (APH), Amsterdam Cardiovascular Science (ACS), Amsterdam Institute for Infection and Immunity (AII), Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands.
| | - Luca F Roggeveen
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence (C4I), Amsterdam Medical Data Science (AMDS), Amsterdam Public Health (APH), Amsterdam Cardiovascular Science (ACS), Amsterdam Institute for Infection and Immunity (AII), Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands.
| | - Martijn Otten
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence (C4I), Amsterdam Medical Data Science (AMDS), Amsterdam Public Health (APH), Amsterdam Cardiovascular Science (ACS), Amsterdam Institute for Infection and Immunity (AII), Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands.
| | - Laurens Biesheuvel
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence (C4I), Amsterdam Medical Data Science (AMDS), Amsterdam Public Health (APH), Amsterdam Cardiovascular Science (ACS), Amsterdam Institute for Infection and Immunity (AII), Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands.
| | - Ameet R Jagesar
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence (C4I), Amsterdam Medical Data Science (AMDS), Amsterdam Public Health (APH), Amsterdam Cardiovascular Science (ACS), Amsterdam Institute for Infection and Immunity (AII), Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands.
| | | | | | | | - Armand R J Girbes
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence (C4I), Amsterdam Medical Data Science (AMDS), Amsterdam Public Health (APH), Amsterdam Cardiovascular Science (ACS), Amsterdam Institute for Infection and Immunity (AII), Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands.
| | - Mark Hoogendoorn
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, the Netherlands.
| | - Patrick J Thoral
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence (C4I), Amsterdam Medical Data Science (AMDS), Amsterdam Public Health (APH), Amsterdam Cardiovascular Science (ACS), Amsterdam Institute for Infection and Immunity (AII), Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands.
| | - Paul W G Elbers
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence (C4I), Amsterdam Medical Data Science (AMDS), Amsterdam Public Health (APH), Amsterdam Cardiovascular Science (ACS), Amsterdam Institute for Infection and Immunity (AII), Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands.
| |
Collapse
|
4
|
Liu X, Hu P, Yeung W, Zhang Z, Ho V, Liu C, Dumontier C, Thoral PJ, Mao Z, Cao D, Mark RG, Zhang Z, Feng M, Li D, Celi LA. Illness severity assessment of older adults in critical illness using machine learning (ELDER-ICU): an international multicentre study with subgroup bias evaluation. Lancet Digit Health 2023; 5:e657-e667. [PMID: 37599147 DOI: 10.1016/s2589-7500(23)00128-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2022] [Revised: 05/31/2023] [Accepted: 06/22/2023] [Indexed: 08/22/2023]
Abstract
BACKGROUND Comorbidity, frailty, and decreased cognitive function lead to a higher risk of death in elderly patients (more than 65 years of age) during acute medical events. Early and accurate illness severity assessment can support appropriate decision making for clinicians caring for these patients. We aimed to develop ELDER-ICU, a machine learning model to assess the illness severity of older adults admitted to the intensive care unit (ICU) with cohort-specific calibration and evaluation for potential model bias. METHODS In this retrospective, international multicentre study, the ELDER-ICU model was developed using data from 14 US hospitals, and validated in 171 hospitals from the USA and Netherlands. Data were extracted from the Medical Information Mart for Intensive Care database, electronic ICU Collaborative Research Database, and Amsterdam University Medical Centers Database. We used six categories of data as predictors, including demographics and comorbidities, physical frailty, laboratory tests, vital signs, treatments, and urine output. Patient data from the first day of ICU stay were used to predict in-hospital mortality. We used the eXtreme Gradient Boosting algorithm (XGBoost) to develop models and the SHapley Additive exPlanations method to explain model prediction. The trained model was calibrated before internal, external, and temporal validation. The final XGBoost model was compared against three other machine learning algorithms and five clinical scores. We performed subgroup analysis based on age, sex, and race. We assessed the discrimination and calibration of models using the area under receiver operating characteristic (AUROC) and standardised mortality ratio (SMR) with 95% CIs. FINDINGS Using the development dataset (n=50 366) and predictive model building process, the XGBoost algorithm performed the best in all types of validations compared with other machine learning algorithms and clinical scores (internal validation with 5037 patients from 14 US hospitals, AUROC=0·866 [95% CI 0·851-0·880]; external validation in the US population with 20 541 patients from 169 hospitals, AUROC=0·838 [0·829-0·847]; external validation in European population with 2411 patients from one hospital, AUROC=0·833 [0·812-0·853]; temporal validation with 4311 patients from one hospital, AUROC=0·884 [0·869-0·897]). In the external validation set (US population), the median AUROCs of bias evaluations covering eight subgroups were above 0·81, and the overall SMR was 0·99 (0·96-1·03). The top ten risk predictors were the minimum Glasgow Coma Scale score, total urine output, average respiratory rate, mechanical ventilation use, best state of activity, Charlson Comorbidity Index score, geriatric nutritional risk index, code status, age, and maximum blood urea nitrogen. A simplified model containing only the top 20 features (ELDER-ICU-20) had similar predictive performance to the full model. INTERPRETATION The ELDER-ICU model reliably predicts the risk of in-hospital mortality using routinely collected clinical features. The predictions could inform clinicians about patients who are at elevated risk of deterioration. Prospective validation of this model in clinical practice and a process for continuous performance monitoring and model recalibration are needed. FUNDING National Institutes of Health, National Natural Science Foundation of China, National Special Health Science Program, Health Science and Technology Plan of Zhejiang Province, Fundamental Research Funds for the Central Universities, Drug Clinical Evaluate Research of Chinese Pharmaceutical Association, and National Key R&D Program of China.
Collapse
Affiliation(s)
- Xiaoli Liu
- School of Biological Science and Medical Engineering, Beihang University, Beijing, China; Center for Artificial Intelligence in Medicine, Chinese PLA General Hospital, Beijing, China; Laboratory for Computational Physiology, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Pan Hu
- Department of Anesthesiology, The 920 Hospital of Joint Logistic Support Force of Chinese PLA, Kunming Yunnan, China; Department of Critical Care Medicine, The First Medical Center of PLA General Hospital, Beijing, China
| | - Wesley Yeung
- Laboratory for Computational Physiology, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA; Department of Cardiology, National University Heart Centre, Singapore
| | - Zhongheng Zhang
- Department of Emergency Medicine, Key Laboratory of Precision Medicine in Diagnosis and Monitoring Research of Zhejiang Province, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Vanda Ho
- Division of Geriatric Medicine, Department of Medicine, National University Hospital, Singapore
| | - Chao Liu
- Department of Critical Care Medicine, The First Medical Center of PLA General Hospital, Beijing, China
| | - Clark Dumontier
- New England Geriatric Research Education and Clinical Center, VA Boston Healthcare System, Boston, MA, USA; Division of Aging, Brigham and Women's Hospital, Boston, MA, USA
| | - Patrick J Thoral
- Center for Critical Care Computational Intelligence, Department of Intensive Care Medicine, Amsterdam UMC, Vrije Universiteit, Amsterdam, Netherlands
| | - Zhi Mao
- Department of Critical Care Medicine, The First Medical Center of PLA General Hospital, Beijing, China
| | - Desen Cao
- Department of Biomedical Engineering, Chinese PLA General Hospital, Beijing, China
| | - Roger G Mark
- Laboratory for Computational Physiology, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Zhengbo Zhang
- Center for Artificial Intelligence in Medicine, Chinese PLA General Hospital, Beijing, China
| | - Mengling Feng
- Saw Swee Hock School of Public Health and the Institute of Data Science, National University of Singapore, Singapore
| | - Deyu Li
- School of Biological Science and Medical Engineering, Beihang University, Beijing, China; National Key Lab for Virtual Reality Technology and Systems, Beihang University, Beijing, China.
| | - Leo Anthony Celi
- Laboratory for Computational Physiology, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA; Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA; Department of Biostatistics, Harvard T H Chan School of Public Health, Boston, MA, USA
| |
Collapse
|
5
|
de Groot R, Püttmann DP, Fleuren LM, Thoral PJ, Elbers PWG, de Keizer NF, Cornet R. Determining and assessing characteristics of data element names impacting the performance of annotation using Usagi. Int J Med Inform 2023; 178:105200. [PMID: 37703800 DOI: 10.1016/j.ijmedinf.2023.105200] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2023] [Revised: 08/11/2023] [Accepted: 08/23/2023] [Indexed: 09/15/2023]
Abstract
INTRODUCTION Hospitals generate large amounts of data and this data is generally modeled and labeled in a proprietary way, hampering its exchange and integration. Manually annotating data element names to internationally standardized data element identifiers is a time-consuming effort. Tools can support performing this task automatically. This study aimed to determine what factors influence the quality of automatic annotations. METHODS Data element names were used from the Dutch COVID-19 ICU Data Warehouse containing data on intensive care patients with COVID-19 from 25 hospitals in the Netherlands. In this data warehouse, the data had been merged using a proprietary terminology system while also storing the original hospital labels (synonymous names). Usagi, an OHDSI annotation tool, was used to perform the annotation for the data. A gold standard was used to determine if Usagi made correct annotations. Logistic regression was used to determine if the number of characters, number of words, match score (Usagi's certainty) and hospital label origin influenced Usagi's performance to annotate correctly. RESULTS Usagi automatically annotated 30.5% of the data element names correctly and 5.5% of the synonymous names. The match score is the best predictor for Usagi finding the correct annotation. It was determined that the AUC of data element names was 0.651 and 0.752 for the synonymous names respectively. The AUC for the individual hospital label origins varied between 0.460 to 0.905. DISCUSSION The results show that Usagi performed better to annotate the data element names than the synonymous names. The hospital origin in the synonymous names dataset was associated with the amount of correctly annotated concepts. Hospitals that performed better had shorter synonymous names and fewer words. Using shorter data element names or synonymous names should be considered to optimize the automatic annotating process. Overall, the performance of Usagi is too poor to completely rely on for automatic annotation.
Collapse
Affiliation(s)
- Rowdy de Groot
- Amsterdam UMC Location University of Amsterdam, Department of Medical Informatics, Amsterdam, the Netherlands.
| | - Daniel P Püttmann
- Amsterdam UMC Location University of Amsterdam, Department of Medical Informatics, Amsterdam, the Netherlands
| | - Lucas M Fleuren
- Department of Intensive Care Medicine, Center for Critical Care Computation Intelligence (C4i), Amsterdam Medical Data Science (AMDS), Amsterdam Public Health (APH), Amsterdam Cardiovascular Science (ACS), Amsterdam Institute for Infection and Immunity (AII), Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands
| | - Patrick J Thoral
- Department of Intensive Care Medicine, Center for Critical Care Computation Intelligence (C4i), Amsterdam Medical Data Science (AMDS), Amsterdam Public Health (APH), Amsterdam Cardiovascular Science (ACS), Amsterdam Institute for Infection and Immunity (AII), Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands
| | - Paul W G Elbers
- Department of Intensive Care Medicine, Center for Critical Care Computation Intelligence (C4i), Amsterdam Medical Data Science (AMDS), Amsterdam Public Health (APH), Amsterdam Cardiovascular Science (ACS), Amsterdam Institute for Infection and Immunity (AII), Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands
| | - Nicolette F de Keizer
- Amsterdam UMC Location University of Amsterdam, Department of Medical Informatics, Amsterdam, the Netherlands
| | - Ronald Cornet
- Amsterdam UMC Location University of Amsterdam, Department of Medical Informatics, Amsterdam, the Netherlands
| |
Collapse
|
6
|
Siepel S, Dam TA, Fleuren LM, Girbes AR, Hoogendoorn M, Thoral PJ, Elbers PW, Bennis FC. Evolution of Clinical Phenotypes of COVID-19 Patients During Intensive Care Treatment: An Unsupervised Machine Learning Analysis. J Intensive Care Med 2023:8850666231153393. [PMID: 36744415 PMCID: PMC9902809 DOI: 10.1177/08850666231153393] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/07/2023]
Abstract
BACKGROUND Identification of clinical phenotypes in critically ill COVID-19 patients could improve understanding of the disease heterogeneity and enable prognostic and predictive enrichment. However, previous attempts did not take into account temporal dynamics with high granularity. By including the dimension of time, we aim to gain further insights into the heterogeneity of COVID-19. METHODS We used granular data from 3202 adult COVID patients in the Dutch Data Warehouse that were admitted to one of 25 Dutch ICUs between February 2020 and March 2021. Parameters including demographics, clinical observations, medications, laboratory values, vital signs, and data from life support devices were selected. Twenty-one datasets were created that each covered 24 h of ICU data for each day of ICU treatment. Clinical phenotypes in each dataset were identified by performing cluster analyses. Both evolution of the clinical phenotypes over time and patient allocation to these clusters over time were tracked. RESULTS The final patient cohort consisted of 2438 COVID-19 patients with a ICU mortality outcome. Forty-one parameters were chosen for cluster analysis. On admission, both a mild and a severe clinical phenotype were found. After day 4, the severe phenotype split into an intermediate and a severe phenotype for 11 consecutive days. Heterogeneity between phenotypes appears to be driven by inflammation and dead space ventilation. During the 21-day period, only 8.2% and 4.6% of patients in the initial mild and severe clusters remained assigned to the same phenotype respectively. The clinical phenotype half-life was between 5 and 6 days for the mild and severe phenotypes, and about 3 days for the medium severe phenotype. CONCLUSIONS Patients typically do not remain in the same cluster throughout intensive care treatment. This may have important implications for prognostic or predictive enrichment. Prominent dissimilarities between clinical phenotypes are predominantly driven by inflammation and dead space ventilation.
Collapse
Affiliation(s)
- Sander Siepel
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, the Netherlands
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands
| | - Tariq A. Dam
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, the Netherlands
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands
| | - Lucas M. Fleuren
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, the Netherlands
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands
| | - Armand R.J. Girbes
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands
| | - Mark Hoogendoorn
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, the Netherlands
| | - Patrick J. Thoral
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands
| | - Paul W.G. Elbers
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands
| | - Frank C. Bennis
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, the Netherlands
- Nivel, Netherlands Institute for Health Services Research, Utrecht, the Netherlands
- Frank Bennis, Quantitative Data Analytics Group, Department of Computer Science, VU Amsterdam, De Boelelaan 1111, 1081 HV, Amsterdam, the Netherlands.
| | | |
Collapse
|
7
|
de Hond AAH, Kant IMJ, Fornasa M, Cinà G, Elbers PWG, Thoral PJ, Sesmu Arbous M, Steyerberg EW. Predicting Readmission or Death After Discharge From the ICU: External Validation and Retraining of a Machine Learning Model. Crit Care Med 2023; 51:291-300. [PMID: 36524820 PMCID: PMC9848213 DOI: 10.1097/ccm.0000000000005758] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
Abstract
OBJECTIVES Many machine learning (ML) models have been developed for application in the ICU, but few models have been subjected to external validation. The performance of these models in new settings therefore remains unknown. The objective of this study was to assess the performance of an existing decision support tool based on a ML model predicting readmission or death within 7 days after ICU discharge before, during, and after retraining and recalibration. DESIGN A gradient boosted ML model was developed and validated on electronic health record data from 2004 to 2021. We performed an independent validation of this model on electronic health record data from 2011 to 2019 from a different tertiary care center. SETTING Two ICUs in tertiary care centers in The Netherlands. PATIENTS Adult patients who were admitted to the ICU and stayed for longer than 12 hours. INTERVENTIONS None. MEASUREMENTS AND MAIN RESULTS We assessed discrimination by area under the receiver operating characteristic curve (AUC) and calibration (slope and intercept). We retrained and recalibrated the original model and assessed performance via a temporal validation design. The final retrained model was cross-validated on all data from the new site. Readmission or death within 7 days after ICU discharge occurred in 577 of 10,052 ICU admissions (5.7%) at the new site. External validation revealed moderate discrimination with an AUC of 0.72 (95% CI 0.67-0.76). Retrained models showed improved discrimination with AUC 0.79 (95% CI 0.75-0.82) for the final validation model. Calibration was poor initially and good after recalibration via isotonic regression. CONCLUSIONS In this era of expanding availability of ML models, external validation and retraining are key steps to consider before applying ML models to new settings. Clinicians and decision-makers should take this into account when considering applying new ML models to their local settings.
Collapse
Affiliation(s)
- Anne A H de Hond
- Department of Information Technology and Digital Innovation, Leiden University Medical Centre, Leiden, The Netherlands
- Department of Biomedical Informatics, Stanford Medicine, Stanford, CA
- Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, The Netherlands
| | - Ilse M J Kant
- Department of Information Technology and Digital Innovation, Leiden University Medical Centre, Leiden, The Netherlands
- Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, The Netherlands
| | | | - Giovanni Cinà
- Pacmed, Stadhouderskade 55, Amsterdam, The Netherlands
- Institute of Logic, Language and Computation, University of Amsterdam, Amsterdam, The Netherlands
| | - Paul W G Elbers
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam UMC, Amsterdam, The Netherlands
| | - Patrick J Thoral
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam UMC, Amsterdam, The Netherlands
| | - M Sesmu Arbous
- Department of Intensive Care Medicine, Leiden University Medical Centre, Leiden, The Netherlands
| | - Ewout W Steyerberg
- Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, The Netherlands
| |
Collapse
|
8
|
Kushniruk A, de Hond AAH, Thoral PJ, Steyerberg EW, Kant IMJ, Cinà G, Arbous MS. Intensive Care Unit Physicians' Perspectives on Artificial Intelligence-Based Clinical Decision Support Tools: Preimplementation Survey Study. JMIR Hum Factors 2023; 10:e39114. [PMID: 36602843 PMCID: PMC9853335 DOI: 10.2196/39114] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2022] [Revised: 11/21/2022] [Accepted: 11/27/2022] [Indexed: 11/29/2022] Open
Abstract
BACKGROUND Artificial intelligence-based clinical decision support (AI-CDS) tools have great potential to benefit intensive care unit (ICU) patients and physicians. There is a gap between the development and implementation of these tools. OBJECTIVE We aimed to investigate physicians' perspectives and their current decision-making behavior before implementing a discharge AI-CDS tool for predicting readmission and mortality risk after ICU discharge. METHODS We conducted a survey of physicians involved in decision-making on discharge of patients at two Dutch academic ICUs between July and November 2021. Questions were divided into four domains: (1) physicians' current decision-making behavior with respect to discharging ICU patients, (2) perspectives on the use of AI-CDS tools in general, (3) willingness to incorporate a discharge AI-CDS tool into daily clinical practice, and (4) preferences for using a discharge AI-CDS tool in daily workflows. RESULTS Most of the 64 respondents (of 93 contacted, 69%) were familiar with AI (62/64, 97%) and had positive expectations of AI, with 55 of 64 (86%) believing that AI could support them in their work as a physician. The respondents disagreed on whether the decision to discharge a patient was complex (23/64, 36% agreed and 22/64, 34% disagreed); nonetheless, most (59/64, 92%) agreed that a discharge AI-CDS tool could be of value. Significant differences were observed between physicians from the 2 academic sites, which may be related to different levels of involvement in the development of the discharge AI-CDS tool. CONCLUSIONS ICU physicians showed a favorable attitude toward the integration of AI-CDS tools into the ICU setting in general, and in particular toward a tool to predict a patient's risk of readmission and mortality within 7 days after discharge. The findings of this questionnaire will be used to improve the implementation process and training of end users.
Collapse
Affiliation(s)
| | - Anne A H de Hond
- Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, Netherlands.,Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, Netherlands
| | - Patrick J Thoral
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam University Medical Centers, Amsterdam, Netherlands
| | - Ewout W Steyerberg
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, Netherlands
| | - Ilse M J Kant
- Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, Netherlands.,Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, Netherlands
| | - Giovanni Cinà
- Pacmed, Amsterdam, Netherlands.,Institute for Logic, Language and Computation, University of Amsterdam, Amsterdam, Netherlands.,Department of Medical Informatics, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, Netherlands
| | - M Sesmu Arbous
- Department of Intensive Care Medicine, Leiden University Medical Center, Leiden, Netherlands
| |
Collapse
|
9
|
Dam TA, Roggeveen LF, van Diggelen F, Fleuren LM, Jagesar AR, Otten M, de Vries HJ, Gommers D, Cremer OL, Bosman RJ, Rigter S, Wils EJ, Frenzel T, Dongelmans DA, de Jong R, Peters MAA, Kamps MJA, Ramnarain D, Nowitzky R, Nooteboom FGCA, de Ruijter W, Urlings-Strop LC, Smit EGM, Mehagnoul-Schipper DJ, Dormans T, de Jager CPC, Hendriks SHA, Achterberg S, Oostdijk E, Reidinga AC, Festen-Spanjer B, Brunnekreef GB, Cornet AD, van den Tempel W, Boelens AD, Koetsier P, Lens J, Faber HJ, Karakus A, Entjes R, de Jong P, Rettig TCD, Arbous S, Vonk SJJ, Machado T, Herter WE, de Grooth HJ, Thoral PJ, Girbes ARJ, Hoogendoorn M, Elbers PWG. Predicting responders to prone positioning in mechanically ventilated patients with COVID-19 using machine learning. Ann Intensive Care 2022; 12:99. [PMID: 36264358 PMCID: PMC9583049 DOI: 10.1186/s13613-022-01070-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2022] [Accepted: 10/06/2022] [Indexed: 11/24/2022] Open
Abstract
Background For mechanically ventilated critically ill COVID-19 patients, prone positioning has quickly become an important treatment strategy, however, prone positioning is labor intensive and comes with potential adverse effects. Therefore, identifying which critically ill intubated COVID-19 patients will benefit may help allocate labor resources. Methods From the multi-center Dutch Data Warehouse of COVID-19 ICU patients from 25 hospitals, we selected all 3619 episodes of prone positioning in 1142 invasively mechanically ventilated patients. We excluded episodes longer than 24 h. Berlin ARDS criteria were not formally documented. We used supervised machine learning algorithms Logistic Regression, Random Forest, Naive Bayes, K-Nearest Neighbors, Support Vector Machine and Extreme Gradient Boosting on readily available and clinically relevant features to predict success of prone positioning after 4 h (window of 1 to 7 h) based on various possible outcomes. These outcomes were defined as improvements of at least 10% in PaO2/FiO2 ratio, ventilatory ratio, respiratory system compliance, or mechanical power. Separate models were created for each of these outcomes. Re-supination within 4 h after pronation was labeled as failure. We also developed models using a 20 mmHg improvement cut-off for PaO2/FiO2 ratio and using a combined outcome parameter. For all models, we evaluated feature importance expressed as contribution to predictive performance based on their relative ranking. Results The median duration of prone episodes was 17 h (11–20, median and IQR, N = 2632). Despite extensive modeling using a plethora of machine learning techniques and a large number of potentially clinically relevant features, discrimination between responders and non-responders remained poor with an area under the receiver operator characteristic curve of 0.62 for PaO2/FiO2 ratio using Logistic Regression, Random Forest and XGBoost. Feature importance was inconsistent between models for different outcomes. Notably, not even being a previous responder to prone positioning, or PEEP-levels before prone positioning, provided any meaningful contribution to predicting a successful next proning episode. Conclusions In mechanically ventilated COVID-19 patients, predicting the success of prone positioning using clinically relevant and readily available parameters from electronic health records is currently not feasible. Given the current evidence base, a liberal approach to proning in all patients with severe COVID-19 ARDS is therefore justified and in particular regardless of previous results of proning. Supplementary Information The online version contains supplementary material available at 10.1186/s13613-022-01070-0.
Collapse
Affiliation(s)
- Tariq A Dam
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands.
| | - Luca F Roggeveen
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Fuda van Diggelen
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, VU University, Amsterdam, The Netherlands
| | - Lucas M Fleuren
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Ameet R Jagesar
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Martijn Otten
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Heder J de Vries
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Diederik Gommers
- Department of Intensive Care, Erasmus Medical Center, Rotterdam, The Netherlands
| | - Olaf L Cremer
- Intensive Care, UMC Utrecht, Utrecht, The Netherlands
| | | | - Sander Rigter
- Department of Anesthesiology and Intensive Care, St. Antonius Hospital, Nieuwegein, The Netherlands
| | - Evert-Jan Wils
- Department of Intensive Care, Franciscus Gasthuis & Vlietland, Rotterdam, The Netherlands
| | - Tim Frenzel
- Department of Intensive Care Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Dave A Dongelmans
- Department of Intensive Care Medicine, Amsterdam UMC, Amsterdam, The Netherlands
| | - Remko de Jong
- Intensive Care, Bovenij Ziekenhuis, Amsterdam, The Netherlands
| | - Marco A A Peters
- Intensive Care, Canisius Wilhelmina Ziekenhuis, Nijmegen, The Netherlands
| | - Marlijn J A Kamps
- Intensive Care, Catharina Ziekenhuis Eindhoven, Eindhoven, The Netherlands
| | | | - Ralph Nowitzky
- Intensive Care, HagaZiekenhuis, Den Haag, The Netherlands
| | | | - Wouter de Ruijter
- Department of Intensive Care Medicine, Northwest Clinics, Alkmaar, The Netherlands
| | | | - Ellen G M Smit
- Intensive Care, Spaarne Gasthuis, Haarlem en Hoofddorp, The Netherlands
| | | | - Tom Dormans
- Intensive Care, Zuyderland MC, Heerlen, The Netherlands
| | | | | | | | | | - Auke C Reidinga
- ICU, SEH, BWC, Martiniziekenhuis, Groningen, The Netherlands
| | | | - Gert B Brunnekreef
- Department of Intensive Care, Ziekenhuisgroep Twente, Almelo, The Netherlands
| | - Alexander D Cornet
- Department of Intensive Care, Medisch Spectrum Twente, Enschede, The Netherlands
| | - Walter van den Tempel
- Department of Intensive Care, Ikazia Ziekenhuis Rotterdam, Rotterdam, The Netherlands
| | | | - Peter Koetsier
- Intensive Care, Medisch Centrum Leeuwarden, Leeuwarden, The Netherlands
| | - Judith Lens
- ICU, IJsselland Ziekenhuis, Capelle aan den IJssel, The Netherlands
| | | | - A Karakus
- Department of Intensive Care, Diakonessenhuis Hospital, Utrecht, The Netherlands
| | - Robert Entjes
- Department of Intensive Care, Adrz, Goes, The Netherlands
| | - Paul de Jong
- Department of Anesthesia and Intensive Care, Slingeland Ziekenhuis, Doetinchem, The Netherlands
| | - Thijs C D Rettig
- Department of Anesthesiology, Intensive Care and Pain Medicine, Amphia Ziekenhuis, Breda, The Netherlands
| | | | | | | | | | - Harm-Jan de Grooth
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Patrick J Thoral
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Armand R J Girbes
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Mark Hoogendoorn
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, VU University, Amsterdam, The Netherlands
| | - Paul W G Elbers
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | | |
Collapse
|
10
|
Vagliano I, Schut MC, Abu-Hanna A, Dongelmans DA, de Lange DW, Gommers D, Cremer OL, Bosman RJ, Rigter S, Wils EJ, Frenzel T, de Jong R, Peters MAA, Kamps MJA, Ramnarain D, Nowitzky R, Nooteboom FGCA, de Ruijter W, Urlings-Strop LC, Smit EGM, Mehagnoul-Schipper DJ, Dormans T, de Jager CPC, Hendriks SHA, Achterberg S, Oostdijk E, Reidinga AC, Festen-Spanjer B, Brunnekreef GB, Cornet AD, van den Tempel W, Boelens AD, Koetsier P, Lens J, Faber HJ, Karakus A, Entjes R, de Jong P, Rettig TCD, Reuland MC, Arbous S, Fleuren LM, Dam TA, Thoral PJ, Lalisang RCA, Tonutti M, de Bruin DP, Elbers PWG, de Keizer NF. Assess and validate predictive performance of models for in-hospital mortality in COVID-19 patients: A retrospective cohort study in the Netherlands comparing the value of registry data with high-granular electronic health records. Int J Med Inform 2022; 167:104863. [PMID: 36162166 PMCID: PMC9492397 DOI: 10.1016/j.ijmedinf.2022.104863] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2022] [Revised: 08/19/2022] [Accepted: 09/03/2022] [Indexed: 11/17/2022]
Abstract
PURPOSE To assess, validate and compare the predictive performance of models for in-hospital mortality of COVID-19 patients admitted to the intensive care unit (ICU) over two different waves of infections. Our models were built with high-granular Electronic Health Records (EHR) data versus less-granular registry data. METHODS Observational study of all COVID-19 patients admitted to 19 Dutch ICUs participating in both the national quality registry National Intensive Care Evaluation (NICE) and the EHR-based Dutch Data Warehouse (hereafter EHR). Multiple models were developed on data from the first 24 h of ICU admissions from February to June 2020 (first COVID-19 wave) and validated on prospective patients admitted to the same ICUs between July and December 2020 (second COVID-19 wave). We assessed model discrimination, calibration, and the degree of relatedness between development and validation population. Coefficients were used to identify relevant risk factors. RESULTS A total of 1533 patients from the EHR and 1563 from the registry were included. With high granular EHR data, the average AUROC was 0.69 (standard deviation of 0.05) for the internal validation, and the AUROC was 0.75 for the temporal validation. The registry model achieved an average AUROC of 0.76 (standard deviation of 0.05) in the internal validation and 0.77 in the temporal validation. In the EHR data, age, and respiratory-system related variables were the most important risk factors identified. In the NICE registry data, age and chronic respiratory insufficiency were the most important risk factors. CONCLUSION In our study, prognostic models built on less-granular but readily-available registry data had similar performance to models built on high-granular EHR data and showed similar transportability to a prospective COVID-19 population. Future research is needed to verify whether this finding can be confirmed for upcoming waves.
Collapse
Affiliation(s)
- Iacopo Vagliano
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands.
| | - Martijn C Schut
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands
| | - Ameen Abu-Hanna
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands
| | - Dave A Dongelmans
- National Intensive Care Evaluation (NICE) foundation, Amsterdam, The Netherlands; Department of Intensive Care Medicine, Amsterdam UMC, University of Amsterdam, Amsterdam, The Netherlands
| | - Dylan W de Lange
- National Intensive Care Evaluation (NICE) foundation, Amsterdam, The Netherlands; Department of Intensive Care Medicine, University Medical Center Utrecht, University Utrecht, Utrecht, The Netherlands
| | - Diederik Gommers
- Department of Intensive Care, Erasmus Medical Center, Rotterdam, The Netherlands
| | - Olaf L Cremer
- Intensive Care, UMC Utrecht, Utrecht, The Netherlands
| | | | - Sander Rigter
- Department of Anesthesiology and Intensive Care, St. Antonius Hospital, Nieuwegein, The Netherlands
| | - Evert-Jan Wils
- Department of Intensive Care, Franciscus Gasthuis & Vlietland, Rotterdam, The Netherlands
| | - Tim Frenzel
- Department of Intensive Care Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Remko de Jong
- Intensive Care, Bovenij Ziekenhuis, Amsterdam, The Netherlands
| | - Marco A A Peters
- Intensive Care, Canisius Wilhelmina Ziekenhuis, Nijmegen, The Netherlands
| | - Marlijn J A Kamps
- Intensive Care, Catharina Ziekenhuis Eindhoven, Eindhoven, The Netherlands
| | | | - Ralph Nowitzky
- Intensive Care, Haga Ziekenhuis, Den Haag, The Netherlands
| | | | - Wouter de Ruijter
- Department of Intensive Care Medicine, Northwest Clinics, Alkmaar, The Netherlands
| | | | - Ellen G M Smit
- Intensive Care, Spaarne Gasthuis, Haarlem en Hoofddorp, The Netherlands
| | | | - Tom Dormans
- Intensive care, Zuyderland MC, Heerlen, The Netherlands
| | | | | | | | | | - Auke C Reidinga
- ICU, SEH, BWC, Martiniziekenhuis, Groningen, The Netherlands
| | | | - Gert B Brunnekreef
- Department of Intensive Care, Ziekenhuisgroep Twente, Almelo, The Netherlands
| | - Alexander D Cornet
- Department of Intensive Care, Medisch Spectrum Twente, Enschede, The Netherlands
| | - Walter van den Tempel
- Department of Intensive Care, Ikazia Ziekenhuis Rotterdam, Rotterdam, The Netherlands
| | - Age D Boelens
- Anesthesiology, Antonius Ziekenhuis Sneek, Sneek, The Netherlands
| | - Peter Koetsier
- Intensive Care, Medisch Centrum Leeuwarden, Leeuwarden, The Netherlands
| | - Judith Lens
- ICU, IJsselland Ziekenhuis, Capelle aan den IJssel, The Netherlands
| | | | - A Karakus
- Department of Intensive Care, Diakonessenhuis Hospital, Utrecht, The Netherlands
| | - Robert Entjes
- Department of Intensive Care, Adrz, Goes, The Netherlands
| | - Paul de Jong
- Department of Anesthesia and Intensive Care, Slingeland Ziekenhuis, Doetinchem, The Netherlands
| | - Thijs C D Rettig
- Department of Anesthesiology, Intensive Care and Pain Medicine, Amphia Ziekenhuis, Breda, The Netherlands
| | - M C Reuland
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands
| | | | - Lucas M Fleuren
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Tariq A Dam
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Patrick J Thoral
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | | | | | | | - Paul W G Elbers
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Nicolette F de Keizer
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands; National Intensive Care Evaluation (NICE) foundation, Amsterdam, The Netherlands
| | | |
Collapse
|
11
|
de Vos J, Visser LA, de Beer AA, Fornasa M, Thoral PJ, Elbers PWG, Cinà G. The Potential Cost-Effectiveness of a Machine Learning Tool That Can Prevent Untimely Intensive Care Unit Discharge. Value Health 2022; 25:359-367. [PMID: 35227446 DOI: 10.1016/j.jval.2021.06.018] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/11/2021] [Revised: 05/06/2021] [Accepted: 06/29/2021] [Indexed: 06/14/2023]
Abstract
OBJECTIVES The machine learning prediction model Pacmed Critical (PC), currently under development, may guide intensivists in their decision-making process on the most appropriate time to discharge a patient from the intensive care unit (ICU). Given the financial pressure on healthcare budgets, this study assessed whether PC has the potential to be cost-effective compared with standard care, without the use of PC, for Dutch patients in the ICU from a societal perspective. METHODS A 1-year, 7-state Markov model reflecting the ICU care pathway and incorporating the PC decision tool was developed. A hypothetical cohort of 1000 adult Dutch patients admitted in the ICU was entered in the model. We used the literature, expert opinion, and data from Amsterdam University Medical Center for model parameters. The uncertainty surrounding the incremental cost-effectiveness ratio was assessed using deterministic and probabilistic sensitivity analyses and scenario analyses. RESULTS PC was a cost-effective strategy with an incremental cost-effectiveness ratio of €18 507 per quality-adjusted life-year. PC remained cost-effective over standard care in multiple scenarios and sensitivity analyses. The likelihood that PC will be cost-effective was 71% at a willingness-to-pay threshold of €30 000 per quality-adjusted life-year. The key driver of the results was the parameter "reduction in ICU length of stay." CONCLUSIONS We showed that PC has the potential to be cost-effective for Dutch ICUs in a time horizon of 1 year. This study is one of the first cost-effectiveness analyses of a machine learning device. Further research is needed to validate the effectiveness of PC, thereby focusing on the key parameter "reduction in ICU length of stay" and potential spill-over effects.
Collapse
Affiliation(s)
- Juliette de Vos
- Pacmed B.V., Amsterdam, The Netherlands; Erasmus School of Health Policy & Management, Erasmus University Rotterdam, Rotterdam, The Netherlands.
| | - Laurenske A Visser
- Erasmus School of Health Policy & Management, Erasmus University Rotterdam, Rotterdam, The Netherlands
| | | | | | - Patrick J Thoral
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Paul W G Elbers
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | | |
Collapse
|
12
|
Plečko D, Bennett N, Mårtensson J, Dam TA, Entjes R, Rettig TCD, Dongelmans DA, Boelens AD, Rigter S, Hendriks SHA, Jong R, Kamps MJA, Peters M, Karakus A, Gommers D, Ramnarain D, Wils E, Achterberg S, Nowitzky R, Tempel W, Jager CPC, Nooteboom FGCA, Oostdijk E, Koetsier P, Cornet AD, Reidinga AC, Ruijter W, Bosman RJ, Frenzel T, Urlings‐Strop LC, Jong P, Smit EG, Cremer OL, Mehagnoul‐Schipper DJ, Faber HJ, Lens J, Brunnekreef GB, Festen‐Spanjer B, Dormans T, Bruin DP, Lalisang RCA, Vonk SJJ, Haan ME, Fleuren LM, Thoral PJ, Elbers PWG, Bellomo R. Rapid Evaluation of Coronavirus Illness Severity (RECOILS) in intensive care: Development and validation of a prognostic tool for in-hospital mortality. Acta Anaesthesiol Scand 2022; 66:65-75. [PMID: 34622441 PMCID: PMC8652966 DOI: 10.1111/aas.13991] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2021] [Revised: 08/16/2021] [Accepted: 09/27/2021] [Indexed: 01/08/2023]
Abstract
Background The prediction of in‐hospital mortality for ICU patients with COVID‐19 is fundamental to treatment and resource allocation. The main purpose was to develop an easily implemented score for such prediction. Methods This was an observational, multicenter, development, and validation study on a national critical care dataset of COVID‐19 patients. A systematic literature review was performed to determine variables possibly important for COVID‐19 mortality prediction. Using a logistic multivariable model with a LASSO penalty, we developed the Rapid Evaluation of Coronavirus Illness Severity (RECOILS) score and compared its performance against published scores. Results Our development (validation) cohort consisted of 1480 (937) adult patients from 14 (11) Dutch ICUs admitted between March 2020 and April 2021. Median age was 65 (65) years, 31% (26%) died in hospital, 74% (72%) were males, average length of ICU stay was 7.83 (10.25) days and average length of hospital stay was 15.90 (19.92) days. Age, platelets, PaO2/FiO2 ratio, pH, blood urea nitrogen, temperature, PaCO2, Glasgow Coma Scale (GCS) score measured within +/−24 h of ICU admission were used to develop the score. The AUROC of RECOILS score was 0.75 (CI 0.71–0.78) which was higher than that of any previously reported predictive scores (0.68 [CI 0.64–0.71], 0.61 [CI 0.58–0.66], 0.67 [CI 0.63–0.70], 0.70 [CI 0.67–0.74] for ISARIC 4C Mortality Score, SOFA, SAPS‐III, and age, respectively). Conclusions Using a large dataset from multiple Dutch ICUs, we developed a predictive score for mortality of COVID‐19 patients admitted to ICU, which outperformed other predictive scores reported so far.
Collapse
Affiliation(s)
- Drago Plečko
- Department of Intensive Care Medicine Laboratory for Critical Care Computational Intelligence Amsterdam Medical Data Science Amsterdam UMC Amsterdam The Netherlands
- Department of Mathematics Seminar for Statistics ETH Zürich Zurich Switzerland
| | - Nicolas Bennett
- Department of Mathematics Seminar for Statistics ETH Zürich Zurich Switzerland
| | - Johan Mårtensson
- Department of Physiology and Pharmacology Section of Anaesthesia and Intensive Care Karolinska Institutet Stockholm Sweden
- Department of Perioperative Medicine and Intensive Care Karolinska University Hospital Stockholm Sweden
| | - Tariq A. Dam
- Department of Intensive Care Medicine Laboratory for Critical Care Computational Intelligence Amsterdam Medical Data Science Amsterdam UMC Amsterdam The Netherlands
| | - Robert Entjes
- Department of Intensive Care Admiraal De Ruyter Ziekenhuis Goes The Netherlands
| | | | - Dave A. Dongelmans
- Department of Intensive Care Medicine Laboratory for Critical Care Computational Intelligence Amsterdam Medical Data Science Amsterdam UMC Amsterdam The Netherlands
| | | | - Sander Rigter
- Department of Anesthesiology and Intensive Care St. Antonius Hospital Nieuwegein The Netherlands
| | | | - Remko Jong
- Intensive Care Bovenij Ziekenhuis Amsterdam The Netherlands
| | | | - Marco Peters
- Intensive Care Canisius Wilhelmina Ziekenhuis Nijmegen The Netherlands
| | - Attila Karakus
- Department of Intensive Care Diakonessenhuis Hospital Utrecht The Netherlands
| | - Diederik Gommers
- Department of Intensive Care Erasmus Medical Center Rotterdam The Netherlands
| | | | - Evert‐Jan Wils
- Department of Intensive Care Franciscus Gasthuis & Vlietland Rotterdam The Netherlands
| | | | | | - Walter Tempel
- Department of Intensive Care Ikazia Ziekenhuis Rotterdam Rotterdam The Netherlands
| | | | | | | | - Peter Koetsier
- Intensive Care Medisch Centrum Leeuwarden Leeuwarden The Netherlands
| | - Alexander D. Cornet
- Department of Intensive Care Medisch Spectrum Twente Enschede The Netherlands
| | | | - Wouter Ruijter
- Department of Intensive Care Medicine Northwest Clinics Alkmaar The Netherlands
| | | | - Tim Frenzel
- Department of Intensive Care Medicine Radboud University Medical Center Nijmegen The Netherlands
| | | | - Paul Jong
- Department of Anesthesia and Intensive Care Slingeland Ziekenhuis Doetinchem The Netherlands
| | - Ellen G.M. Smit
- Intensive Care Spaarne GasthuisHaarlem en Hoofddorp The Netherlands
| | | | | | | | - Judith Lens
- ICU ICU, IJsselland ZiekenhuisCapelle aan den IJssel The Netherlands
| | | | | | - Tom Dormans
- Intensive care Zuyderland MC Heerlen The Netherlands
| | | | | | | | - Martin E. Haan
- Department of Intensive Care Medicine Laboratory for Critical Care Computational Intelligence Amsterdam Medical Data Science Amsterdam UMC Amsterdam The Netherlands
| | - Lucas M. Fleuren
- Department of Intensive Care Medicine Laboratory for Critical Care Computational Intelligence Amsterdam Medical Data Science Amsterdam UMC Amsterdam The Netherlands
| | - Patrick J. Thoral
- Department of Intensive Care Medicine Laboratory for Critical Care Computational Intelligence Amsterdam Medical Data Science Amsterdam UMC Amsterdam The Netherlands
| | - Paul W. G. Elbers
- Department of Intensive Care Medicine Laboratory for Critical Care Computational Intelligence Amsterdam Medical Data Science Amsterdam UMC Amsterdam The Netherlands
| | - Rinaldo Bellomo
- Australian and New Zealand Intensive Care Research CentreSchool of Public Health and Preventative MedicineMonash University Melbourne Australia
- Department of Critical Care The University of Melbourne Melbourne Australia
- Data Analytics Research and Evaluation Centre Department of Medicine and Radiology The University of Melbourne
- Austin Hospital Melbourne Australia
| |
Collapse
|
13
|
Fleuren LM, Dam TA, Tonutti M, de Bruin DP, Lalisang RCA, Gommers D, Cremer OL, Bosman RJ, Rigter S, Wils EJ, Frenzel T, Dongelmans DA, de Jong R, Peters M, Kamps MJA, Ramnarain D, Nowitzky R, Nooteboom FGCA, de Ruijter W, Urlings-Strop LC, Smit EGM, Mehagnoul-Schipper DJ, Dormans T, de Jager CPC, Hendriks SHA, Achterberg S, Oostdijk E, Reidinga AC, Festen-Spanjer B, Brunnekreef GB, Cornet AD, van den Tempel W, Boelens AD, Koetsier P, Lens J, Faber HJ, Karakus A, Entjes R, de Jong P, Rettig TCD, Arbous S, Vonk SJJ, Fornasa M, Machado T, Houwert T, Hovenkamp H, Noorduijn Londono R, Quintarelli D, Scholtemeijer MG, de Beer AA, Cinà G, Kantorik A, de Ruijter T, Herter WE, Beudel M, Girbes ARJ, Hoogendoorn M, Thoral PJ, Elbers PWG. Predictors for extubation failure in COVID-19 patients using a machine learning approach. Crit Care 2021; 25:448. [PMID: 34961537 PMCID: PMC8711075 DOI: 10.1186/s13054-021-03864-3] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2021] [Accepted: 12/13/2021] [Indexed: 12/17/2022] Open
Abstract
INTRODUCTION Determining the optimal timing for extubation can be challenging in the intensive care. In this study, we aim to identify predictors for extubation failure in critically ill patients with COVID-19. METHODS We used highly granular data from 3464 adult critically ill COVID patients in the multicenter Dutch Data Warehouse, including demographics, clinical observations, medications, fluid balance, laboratory values, vital signs, and data from life support devices. All intubated patients with at least one extubation attempt were eligible for analysis. Transferred patients, patients admitted for less than 24 h, and patients still admitted at the time of data extraction were excluded. Potential predictors were selected by a team of intensive care physicians. The primary and secondary outcomes were extubation without reintubation or death within the next 7 days and within 48 h, respectively. We trained and validated multiple machine learning algorithms using fivefold nested cross-validation. Predictor importance was estimated using Shapley additive explanations, while cutoff values for the relative probability of failed extubation were estimated through partial dependence plots. RESULTS A total of 883 patients were included in the model derivation. The reintubation rate was 13.4% within 48 h and 18.9% at day 7, with a mortality rate of 0.6% and 1.0% respectively. The grandient-boost model performed best (area under the curve of 0.70) and was used to calculate predictor importance. Ventilatory characteristics and settings were the most important predictors. More specifically, a controlled mode duration longer than 4 days, a last fraction of inspired oxygen higher than 35%, a mean tidal volume per kg ideal body weight above 8 ml/kg in the day before extubation, and a shorter duration in assisted mode (< 2 days) compared to their median values. Additionally, a higher C-reactive protein and leukocyte count, a lower thrombocyte count, a lower Glasgow coma scale and a lower body mass index compared to their medians were associated with extubation failure. CONCLUSION The most important predictors for extubation failure in critically ill COVID-19 patients include ventilatory settings, inflammatory parameters, neurological status, and body mass index. These predictors should therefore be routinely captured in electronic health records.
Collapse
Affiliation(s)
- Lucas M. Fleuren
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Tariq A. Dam
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | | | | | | | - Diederik Gommers
- Department of Intensive Care, Erasmus Medical Center, Rotterdam, The Netherlands
| | - Olaf L. Cremer
- Department of Intensive Care, UMC Utrecht, Utrecht, The Netherlands
| | | | - Sander Rigter
- Department of Anesthesiology and Intensive Care, St. Antonius Hospital, Nieuwegein, The Netherlands
| | - Evert-Jan Wils
- Department of Intensive Care, Franciscus Gasthuis and Vlietland, Rotterdam, The Netherlands
| | - Tim Frenzel
- Department of Intensive Care Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Dave A. Dongelmans
- Department of Intensive Care Medicine, Amsterdam UMC, Amsterdam, The Netherlands
| | - Remko de Jong
- Intensive Care, Bovenij Ziekenhuis, Amsterdam, The Netherlands
| | - Marco Peters
- Intensive Care, Canisius Wilhelmina Ziekenhuis, Nijmegen, The Netherlands
| | | | | | - Ralph Nowitzky
- Intensive Care, HagaZiekenhuis, Den Haag, The Netherlands
| | | | - Wouter de Ruijter
- Department of Intensive Care Medicine, Northwest Clinics, Alkmaar, The Netherlands
| | | | - Ellen G. M. Smit
- Intensive Care, Spaarne Gasthuis, Haarlem en Hoofddorp, The Netherlands
| | | | - Tom Dormans
- Intensive Care, Zuyderland MC, Heerlen, The Netherlands
| | | | | | | | | | | | | | - Gert B. Brunnekreef
- Department of Intensive Care, Ziekenhuisgroep Twente, Almelo, The Netherlands
| | - Alexander D. Cornet
- Department of Intensive Care, Medisch Spectrum Twente, Enschede, The Netherlands
| | - Walter van den Tempel
- Department of Intensive Care, Ikazia Ziekenhuis Rotterdam, Rotterdam, The Netherlands
| | | | - Peter Koetsier
- Intensive Care, Medisch Centrum Leeuwarden, Leeuwarden, The Netherlands
| | - Judith Lens
- ICU, IJsselland Ziekenhuis, Capelle Aan Den IJssel, The Netherlands
| | | | - A. Karakus
- Department of Intensive Care, Diakonessenhuis Hospital, Utrecht, The Netherlands
| | - Robert Entjes
- Department of Intensive Care, Adrz, Goes, The Netherlands
| | - Paul de Jong
- Department of Anesthesia and Intensive Care, Slingeland Ziekenhuis, Doetinchem, The Netherlands
| | - Thijs C. D. Rettig
- Department of Anesthesiology, Intensive Care and Pain Medicine, Amphia Ziekenhuis, Breda, The Netherlands
| | - Sesmu Arbous
- Department of Intensive Care, LUMC, Leiden, The Netherlands
| | | | | | | | | | | | | | | | | | | | | | | | | | | | - Martijn Beudel
- Department of Neurology, Amsterdam UMC, Universiteit Van Amsterdam, Amsterdam, The Netherlands
| | - Armand R. J. Girbes
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Mark Hoogendoorn
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | - Patrick J. Thoral
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Paul W. G. Elbers
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| |
Collapse
|
14
|
Fleuren LM, Dam TA, Tonutti M, de Bruin DP, Lalisang RCA, Gommers D, Cremer OL, Bosman RJ, Rigter S, Wils EJ, Frenzel T, Dongelmans DA, de Jong R, Peters M, Kamps MJA, Ramnarain D, Nowitzky R, Nooteboom FGCA, de Ruijter W, Urlings-Strop LC, Smit EGM, Mehagnoul-Schipper DJ, Dormans T, de Jager CPC, Hendriks SHA, Achterberg S, Oostdijk E, Reidinga AC, Festen-Spanjer B, Brunnekreef GB, Cornet AD, van den Tempel W, Boelens AD, Koetsier P, Lens J, Faber HJ, Karakus A, Entjes R, de Jong P, Rettig TCD, Arbous S, Vonk SJJ, Fornasa M, Machado T, Houwert T, Hovenkamp H, Noorduijn-Londono R, Quintarelli D, Scholtemeijer MG, de Beer AA, Cina G, Beudel M, Herter WE, Girbes ARJ, Hoogendoorn M, Thoral PJ, Elbers PWG. The Dutch Data Warehouse, a multicenter and full-admission electronic health records database for critically ill COVID-19 patients. Crit Care 2021; 25:304. [PMID: 34425864 PMCID: PMC8381710 DOI: 10.1186/s13054-021-03733-z] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2021] [Accepted: 08/16/2021] [Indexed: 12/04/2022] Open
Abstract
BACKGROUND The Coronavirus disease 2019 (COVID-19) pandemic has underlined the urgent need for reliable, multicenter, and full-admission intensive care data to advance our understanding of the course of the disease and investigate potential treatment strategies. In this study, we present the Dutch Data Warehouse (DDW), the first multicenter electronic health record (EHR) database with full-admission data from critically ill COVID-19 patients. METHODS A nation-wide data sharing collaboration was launched at the beginning of the pandemic in March 2020. All hospitals in the Netherlands were asked to participate and share pseudonymized EHR data from adult critically ill COVID-19 patients. Data included patient demographics, clinical observations, administered medication, laboratory determinations, and data from vital sign monitors and life support devices. Data sharing agreements were signed with participating hospitals before any data transfers took place. Data were extracted from the local EHRs with prespecified queries and combined into a staging dataset through an extract-transform-load (ETL) pipeline. In the consecutive processing pipeline, data were mapped to a common concept vocabulary and enriched with derived concepts. Data validation was a continuous process throughout the project. All participating hospitals have access to the DDW. Within legal and ethical boundaries, data are available to clinicians and researchers. RESULTS Out of the 81 intensive care units in the Netherlands, 66 participated in the collaboration, 47 have signed the data sharing agreement, and 35 have shared their data. Data from 25 hospitals have passed through the ETL and processing pipeline. Currently, 3464 patients are included in the DDW, both from wave 1 and wave 2 in the Netherlands. More than 200 million clinical data points are available. Overall ICU mortality was 24.4%. Respiratory and hemodynamic parameters were most frequently measured throughout a patient's stay. For each patient, all administered medication and their daily fluid balance were available. Missing data are reported for each descriptive. CONCLUSIONS In this study, we show that EHR data from critically ill COVID-19 patients may be lawfully collected and can be combined into a data warehouse. These initiatives are indispensable to advance medical data science in the field of intensive care medicine.
Collapse
Affiliation(s)
- Lucas M. Fleuren
- Laboratory for Critical Care Computational Intelligence, Department of Intensive Care Medicine, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Tariq A. Dam
- Laboratory for Critical Care Computational Intelligence, Department of Intensive Care Medicine, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | | | | | | | - Diederik Gommers
- Department of Intensive Care, Erasmus Medical Center, Rotterdam, The Netherlands
| | | | | | - Sander Rigter
- Department of Anesthesiology and Intensive Care, St. Antonius Hospital, Nieuwegein, The Netherlands
| | - Evert-Jan Wils
- Department of Intensive Care, Franciscus Gasthuis & Vlietland, Rotterdam, The Netherlands
| | - Tim Frenzel
- Department of Intensive Care Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Dave A. Dongelmans
- Department of Intensive Care Medicine, Amsterdam UMC, Amsterdam, The Netherlands
| | - Remko de Jong
- Intensive Care, Bovenij Ziekenhuis, Amsterdam, The Netherlands
| | - Marco Peters
- Intensive Care, Canisius Wilhelmina Ziekenhuis, Nijmegen, The Netherlands
| | | | | | - Ralph Nowitzky
- Intensive Care, HagaZiekenhuis, Den Haag, The Netherlands
| | | | - Wouter de Ruijter
- Department of Intensive Care Medicine, Northwest Clinics, Alkmaar, The Netherlands
| | | | - Ellen G. M. Smit
- Intensive Care, Spaarne Gasthuis, Haarlem en Hoofddorp, The Netherlands
| | | | - Tom Dormans
- Intensive Care, Zuyderland MC, Heerlen, The Netherlands
| | | | | | | | | | | | | | - Gert B. Brunnekreef
- Department of Intensive Care, Ziekenhuisgroep Twente, Almelo, The Netherlands
| | - Alexander D. Cornet
- Department of Intensive Care, Medisch Spectrum Twente, Enschede, The Netherlands
| | - Walter van den Tempel
- Department of Intensive Care, Ikazia Ziekenhuis Rotterdam, Rotterdam, The Netherlands
| | - Age D. Boelens
- Anesthesiology, Antonius Ziekenhuis Sneek, Sneek, The Netherlands
| | - Peter Koetsier
- Intensive Care, Medisch Centrum Leeuwarden, Leeuwarden, The Netherlands
| | - Judith Lens
- ICU, ICU, IJsselland Ziekenhuis, Capelle aan den IJssel, The Netherlands
| | | | - A. Karakus
- Department of Intensive Care, Diakonessenhuis Hospital, Utrecht, The Netherlands
| | - Robert Entjes
- Department of Intensive Care, Admiraal De Ruyter Ziekenhuis, Goes, The Netherlands
| | - Paul de Jong
- Department of Anesthesia and Intensive Care, Slingeland Ziekenhuis, Doetinchem, The Netherlands
| | | | - Sesmu Arbous
- Department of Intensive Care, LUMC, Leiden, The Netherlands
| | | | | | | | | | | | | | | | | | | | | | - Martijn Beudel
- Department of Neurology, Amsterdam UMC, Universiteit van Amsterdam, Amsterdam, The Netherlands
| | | | - Armand R. J. Girbes
- Laboratory for Critical Care Computational Intelligence, Department of Intensive Care Medicine, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Mark Hoogendoorn
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrjie Universiteit, Amsterdam, The Netherlands
| | - Patrick J. Thoral
- Laboratory for Critical Care Computational Intelligence, Department of Intensive Care Medicine, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Paul W. G. Elbers
- Laboratory for Critical Care Computational Intelligence, Department of Intensive Care Medicine, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| |
Collapse
|
15
|
Thoral PJ, Peppink JM, Driessen RH, Sijbrands EJG, Kompanje EJO, Kaplan L, Bailey H, Kesecioglu J, Cecconi M, Churpek M, Clermont G, van der Schaar M, Ercole A, Girbes ARJ, Elbers PWG. Sharing ICU Patient Data Responsibly Under the Society of Critical Care Medicine/European Society of Intensive Care Medicine Joint Data Science Collaboration: The Amsterdam University Medical Centers Database (AmsterdamUMCdb) Example. Crit Care Med 2021; 49:e563-e577. [PMID: 33625129 PMCID: PMC8132908 DOI: 10.1097/ccm.0000000000004916] [Citation(s) in RCA: 74] [Impact Index Per Article: 24.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
OBJECTIVES Critical care medicine is a natural environment for machine learning approaches to improve outcomes for critically ill patients as admissions to ICUs generate vast amounts of data. However, technical, legal, ethical, and privacy concerns have so far limited the critical care medicine community from making these data readily available. The Society of Critical Care Medicine and the European Society of Intensive Care Medicine have identified ICU patient data sharing as one of the priorities under their Joint Data Science Collaboration. To encourage ICUs worldwide to share their patient data responsibly, we now describe the development and release of Amsterdam University Medical Centers Database (AmsterdamUMCdb), the first freely available critical care database in full compliance with privacy laws from both the United States and Europe, as an example of the feasibility of sharing complex critical care data. SETTING University hospital ICU. SUBJECTS Data from ICU patients admitted between 2003 and 2016. INTERVENTIONS We used a risk-based deidentification strategy to maintain data utility while preserving privacy. In addition, we implemented contractual and governance processes, and a communication strategy. Patient organizations, supporting hospitals, and experts on ethics and privacy audited these processes and the database. MEASUREMENTS AND MAIN RESULTS AmsterdamUMCdb contains approximately 1 billion clinical data points from 23,106 admissions of 20,109 patients. The privacy audit concluded that reidentification is not reasonably likely, and AmsterdamUMCdb can therefore be considered as anonymous information, both in the context of the U.S. Health Insurance Portability and Accountability Act and the European General Data Protection Regulation. The ethics audit concluded that responsible data sharing imposes minimal burden, whereas the potential benefit is tremendous. CONCLUSIONS Technical, legal, ethical, and privacy challenges related to responsible data sharing can be addressed using a multidisciplinary approach. A risk-based deidentification strategy, that complies with both U.S. and European privacy regulations, should be the preferred approach to releasing ICU patient data. This supports the shared Society of Critical Care Medicine and European Society of Intensive Care Medicine vision to improve critical care outcomes through scientific inquiry of vast and combined ICU datasets.
Collapse
Affiliation(s)
- Patrick J Thoral
- Department of Intensive Care Medicine, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Sciences (ACS), Amsterdam Infection and Immunity Institute (AI&II), Amsterdam UMC, Vrije Universiteit, Universiteit van Amsterdam, Amsterdam, The Netherlands
| | - Jan M Peppink
- Department of Intensive Care Medicine, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Sciences (ACS), Amsterdam Infection and Immunity Institute (AI&II), Amsterdam UMC, Vrije Universiteit, Universiteit van Amsterdam, Amsterdam, The Netherlands
| | - Ronald H Driessen
- Department of Intensive Care Medicine, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Sciences (ACS), Amsterdam Infection and Immunity Institute (AI&II), Amsterdam UMC, Vrije Universiteit, Universiteit van Amsterdam, Amsterdam, The Netherlands
| | | | - Erwin J O Kompanje
- Department of Intensive Care Medicine, Erasmus MC, Rotterdam, The Netherlands
| | - Lewis Kaplan
- Division of Trauma, Surgical Critical Care and Emergency Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA
- Executive Committee, Society of Critical Care Medicine, Mount Prospect, IL
| | - Heatherlee Bailey
- Department of Emergency Medicine, Durham VA Medical Center, Durham, NC
- Executive Committee, Society of Critical Care Medicine, Mount Prospect, IL
| | - Jozef Kesecioglu
- Department of Intensive Care Medicine, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Executive Committee, European Society of Intensive Care Medicine, Brussels, Belgium
| | - Maurizio Cecconi
- Executive Committee, European Society of Intensive Care Medicine, Brussels, Belgium
- Department of Anaesthesia and Intensive Care, Humanitas Research Hospital, Humanitas University, Milan, Italy
| | - Matthew Churpek
- Department of Medicine, University of Wisconsin, Madison, WI
| | - Gilles Clermont
- Department of Critical Care Medicine, CRISMA Laboratory, University of Pittsburgh, Pittsburgh, PA
| | - Mihaela van der Schaar
- University of Cambridge, Cambridge, United Kingdom
- Alan Turing Institute, London, United Kingdom
| | - Ari Ercole
- Division of Anaesthesia, University of Cambridge, Cambridge, United Kingdom
- Data Science Section, European Society of Intensive Care Medicine, Brussels, Belgium
| | - Armand R J Girbes
- Department of Intensive Care Medicine, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Sciences (ACS), Amsterdam Infection and Immunity Institute (AI&II), Amsterdam UMC, Vrije Universiteit, Universiteit van Amsterdam, Amsterdam, The Netherlands
- Executive Committee, European Society of Intensive Care Medicine, Brussels, Belgium
| | - Paul W G Elbers
- Department of Intensive Care Medicine, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Sciences (ACS), Amsterdam Infection and Immunity Institute (AI&II), Amsterdam UMC, Vrije Universiteit, Universiteit van Amsterdam, Amsterdam, The Netherlands
- Data Science Section, European Society of Intensive Care Medicine, Brussels, Belgium
| |
Collapse
|
16
|
Stapel SN, de Boer RJ, Thoral PJ, Vervloet MG, Girbes ARJ, Oudemans-van Straaten HM. Amino Acid Loss during Continuous Venovenous Hemofiltration in Critically Ill Patients. Blood Purif 2019; 48:321-329. [PMID: 31291614 DOI: 10.1159/000500998] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/27/2019] [Accepted: 05/14/2019] [Indexed: 11/19/2022]
Abstract
BACKGROUND/OBJECTIVES During continuous venovenous hemofiltration (CVVH), there is unwanted loss of amino acids (AA) in the ultrafiltrate (UF). Solutes may also be removed by adsorption to the filter membrane. The aim was to quantify the total loss of AA via the CVVH circuit using a high-flux polysulfone membrane and to differentiate between the loss by ultrafiltration and adsorption. METHODS Prospective observational study in ten critically ill patients, receiving predilution CVVH with a new filter, blood flow 180 mL/min, and predilution flow 2,400 mL/h. Arterial blood, postfilter blood, and UF samples were taken at baseline, and 1, 8, and 24-h after CVVH initiation, to determine AA concentrations and hematocrit. Mass transfer calculations were used to determine AA loss in the filter and by UF, and the difference between these 2. RESULTS The median AA loss in the filter was 10.4 g/day, the median AA loss by UF was 13.4 g/day, and the median difference was -2.9 g/day (IQR -5.9 to -1.4 g/day). For the individual AA, the difference ranged from -1 g/day to +0.4 g/day, suggesting that some AA were consumed or adsorbed and others were generated. AA losses did not significantly change over the 24-h study period. CONCLUSION During CVVH with a modern polysulfone membrane, the estimated AA loss was 13.4 g/day, which corresponds to a loss of about 11.2 g of protein per day. Adsorption did not play a major role. However, individual AA behaved differently, suggesting complex interactions and processes at the filter membrane or peripheral AA production.
Collapse
Affiliation(s)
- Sandra N Stapel
- Department of Adult Intensive Care Medicine, Amsterdam University Medical Centers, Vrije Universiteit, Amsterdam, The Netherlands,
| | - Ruben J de Boer
- Department of Adult Intensive Care Medicine, Amsterdam University Medical Centers, Vrije Universiteit, Amsterdam, The Netherlands
| | - Patrick J Thoral
- Department of Adult Intensive Care Medicine, Amsterdam University Medical Centers, Vrije Universiteit, Amsterdam, The Netherlands
| | - Marc G Vervloet
- Department of Nephrology, Amsterdam University Medical Centers, Vrije Universiteit, Amsterdam, The Netherlands
| | - Armand R J Girbes
- Department of Adult Intensive Care Medicine, Amsterdam University Medical Centers, Vrije Universiteit, Amsterdam, The Netherlands
| | - Heleen M Oudemans-van Straaten
- Department of Adult Intensive Care Medicine, Amsterdam University Medical Centers, Vrije Universiteit, Amsterdam, The Netherlands
| |
Collapse
|