Impact of the Covid-19 pandemic on the performance of machine learning algorithms for predicting perioperative mortality

BACKGROUND

Machine-learning models are susceptible to external influences which can result in performance deterioration. The aim of our study was to elucidate the impact of a sudden shift in covariates, like the one caused by the Covid-19 pandemic, on model performance.

METHODS

After ethical approval and registration in Clinical Trials (NCT04092933, initial release 17/09/2019), we developed different models for the prediction of perioperative mortality based on preoperative data: one for the pre-pandemic data period until March 2020, one including data before the pandemic and from the first wave until May 2020, and one that covers the complete period before and during the pandemic until October 2021. We applied XGBoost as well as a Deep Learning neural network (DL). Performance metrics of each model during the different pandemic phases were determined, and XGBoost models were analysed for changes in feature importance.

RESULTS

XGBoost and DL provided similar performance on the pre-pandemic data with respect to area under receiver operating characteristic (AUROC, 0.951 vs. 0.942) and area under precision-recall curve (AUPR, 0.144 vs. 0.187). Validation in patient cohorts of the different pandemic waves showed high fluctuations in performance from both AUROC and AUPR for DL, whereas the XGBoost models seemed more stable. Change in variable frequencies with onset of the pandemic were visible in age, ASA score, and the higher proportion of emergency operations, among others. Age consistently showed the highest information gain. Models based on pre-pandemic data performed worse during the first pandemic wave (AUROC 0.914 for XGBoost and DL) whereas models augmented with data from the first wave lacked performance after the first wave (AUROC 0.907 for XGBoost and 0.747 for DL). The deterioration was also visible in AUPR, which worsened by over 50% in both XGBoost and DL in the first phase after re-training.

CONCLUSIONS

A sudden shift in data impacts model performance. Re-training the model with updated data may cause degradation in predictive accuracy if the changes are only transient. Too early re-training should therefore be avoided, and close model surveillance is necessary.

Sex-specific issues of central and peripheral arginine-vasopressin concentrations in neurocritical care patients

BACKGROUND

Arginine-Vasopressin (AVP) is a nonapeptide that exerts multiple functions within the central nervous system and in the blood circulation that might contribute to outcome in critically ill patients. Sex differences have been found for mental and physical effects of AVP. For example, stress response and response due to hemorrhage differ between males and females, at least in animal studies. Data on humans -especially on AVP within the central nervous system (CNS)-are scarce, as cerebrospinal fluid (CSF) which is said to represent central AVP activity, has to be collected by means of invasive procedures. Here we present data on 30 neurocritical care patients where we simultaneously collected blood, CSF and saliva to analyze concentrations in the central and peripheral compartments.

PATIENTS AND METHODS

30 neurocritical care patients were included (13 male, 13 postmenopausal female, 4 premenopausal female) with a median age of 60 years. CSF, plasma and saliva were obtained simultaneously once in each patient and analyzed for AVP concentrations. Correlations between the central compartment represented by CSF, and the peripheral compartment represented by plasma and saliva, were identified. Relations between AVP concentrations and serum sodium and hematocrit were also determined.

RESULTS

In the whole patient collective, only very weak to weak correlations could be detected between AVP plasma/CSF, plasma/saliva and CSF/saliva as well as between AVP concentrations in each of the compartments and serum sodium/hematocrit. Regarding the subgroup of postmenopausal females, a significant moderate correlation could be detected for AVP in plasma and CSF and AVP CSF and serum sodium.

CONCLUSION

Absolute concentrations of AVP in central and peripheral compartments did not show sex differences. However, correlations between AVP plasma and CSF and AVP CSF and serum sodium in postmenopausal females indicate differences in AVP secretion and AVP response to triggers that deserve further examination.

[Potential effect of the stimulus threshold level of the nociceptive flexion reflex (NFRT) on mortality and delirium incidence in the critically ill patient: a retrospective cohort analysis]

BACKGROUND

Mortality and delirium in critically ill patients are affected by the provided analgesics and sedatives. The deeper the sedation and the higher the dose of analgesics applied, the more difficult it is to assess pain and the depth of sedation. Therefore, instrumental measurement methods, such as the measurement of the stimulus threshold of the nociceptive flexion reflex (NFRT), are becoming increasingly more important.

OBJECTIVE

The aim of the present study is to investigate a potential association between the level of the nociceptive flexion reflex, mortality, and the occurrence of delirium.

MATERIAL AND METHODS

By retrospectively analyzing a pilot data set of 57 ICU patients from the interdisciplinary surgical ICU of Ulm University Hospital surveyed between 11/2018 and 03/2020, a possible association between the NFRT, mortality, and the occurrence of delirium was calculated in an adjusted logistic regression model. Depending on the cut-off value, the stimulus threshold corridors result in the following comparison pairs: < 20 mA vs. 20-40 mA/20-50 mA/20-60 mA, > 40 mA vs. 20-40 mA, > 50 mA vs. 20-50 mA and > 60 mA vs. 20-60 mA. Results are presented as odds ratios (OR) adjusted for age, sex, height, TISS-28, SAPS II, RASS, BPS, and applied analgesics. Pain assessment was performed, in addition to the Behavioral Pain scale, ≥ 3 times daily by measuring NFRT.

RESULTS

A statistically nonsignificant tendency for an increase in mortality incidence occurred with an NFRT > 50 mA, versus a stimulus threshold corridor of 20-50 mA (OR 3.3, CI: 0.89-12.43, p = 0.07). A trend toward a reduction in delirium incidence occurred at an NFRT < 20 mA, versus a stimulus threshold corridor of 20-40 mA (OR 0.40, CI: 0.18-0.92, p = 0.03).

CONCLUSION

Based on the level of the NFRT, no recommendation can be made at this point to adjust the analgesic regimen of critically ill patients, who are unable to communicate. The observation of a tendency towards an increase in mortality at high stimulus thresholds or a reduction in the occurrence of delirium at low stimulus thresholds of the NFRT must be verified in standardized studies.

[Managing the pandemic-relocation concept for COVID-19 intensive care patients and non-COVID-19 intensive care patients in Baden-Württemberg]

BACKGROUND

A sharp rise in COVID-19 infections threatened to lead to a local overload of intensive care units in autumn 2020. To prevent this scenario a nationwide relocation concept was developed.

METHODS

For the development of the concept publicly available infection rates of the leading infection authority in Germany were used. Within this concept six medical care regions (clusters) were designed around a center of maximum intensive care (ECMO option) based on the number of intensive care beds per 100,000 inhabitants. The concept describes the management structure including a structural chart, the individual tasks, the organization and the cluster assignment of the clinics. The transfers of intensive care patients within and between the clusters were recorded from 11 December 2020 to 31 January 2021.

RESULT

In Germany and Baden-Württemberg, 1.5% of patients newly infected with SARS-CoV‑2 required intensive care treatment in mid-December 2020. With a 7-day incidence of 192 new infections in Germany, the hospitalization rate was 10% and 28-35% of the intensive care beds were occupied by COVID-19 patients. Only 16.8% of the intensive care beds were still available, in contrast to 35% in June 2020. The developed relocation concept has been in use in Baden-Württemberg starting from 10 December 2020. From then until 7 February 2021, a median of 24 ± 5/54 intensive care patients were transferred within the individual clusters, in total 154 intensive care patients. Between the clusters, a minimum of 1 and a maximum of 15 (median 12.5) patients were transferred, 21 intensive care patients were transferred to other federal states and 21 intensive care patients were admitted from these states. The total number of intensive care patients transferred was 261.

CONCLUSION

If the number of infections with SARS-CoV‑2 increases, a nationwide relocation concept for COVID-19 intensive care patients and non-COVID-19 intensive care patients should be installed at an early stage in order not to overwhelm the capacities of hospitals. Supply regions around a leading clinic with maximum intensive care options are to be defined with a central management that organizes the necessary relocations in cooperation with regional and superregional rescue service control centers. With this concept and the intensive care transports carried out, it was possible to effectively prevent the overload of individual clinics with COVID-19 patients in Baden-Württemberg. Due to that an almost unchanged number of patients requiring regular intensive care could be treated.

[Machine learning in anesthesiology]

The application of artificial intelligence (AI) is currently changing very different areas of life. Artificial intelligence involves the emulation of human behavior with the aid of methods from mathematics and informatics. Machine learning (ML) represents a subdivision of AI. Algorithms for ML have the potential to optimize patient care, in that they can be utilized in a supportive way in personalized medicine, decision making and risk prediction. Although the majority of the applications in medicine are still limited to data analysis and research, it is certain that ML will become increasingly more important in scientific and clinical aspects in this supportive function. Therefore, it is necessary for clinicians to have at least a basic understanding of the functional principles, strengths and weaknesses of ML.

Effects of cardiopulmonary bypass on neurocognitive performance and cytokine release in old and diabetic rats

BACKGROUND

Age and diabetes mellitus have been identified as independent risk factors for cognitive decline after cardiac surgery with cardiopulmonary bypass (CPB). We tested the effects of CPB on cognitive function in aged and diabetic rats utilizing the Morris water maze (MWM).

METHODS

Aged rats (26 months) were randomized into a sham group (cannulation but no CPB, n = 11) and a 90 min CPB group (n = 11). In addition, young rats (n = 14) were made diabetic with streptozotocin 9 weeks before experimentation and randomized to a sham or 90 min CPB group. Cytokine release [interleukin (IL-6)] and short-term MWM performance (days 8-14 after operation) were assessed in all animals. Long-term MWM performance (8 weeks after operation) was assessed in aged rats only.

RESULTS

There were no differences between the aged groups in short-term (P = 0.58) or long-term MWM performances (P = 0.69). The diabetic animals also showed no differences between the sham and CPB groups in MWM performance (P = 0.64). IL-6 assays showed an increased inflammatory response after CPB in the diabetic animals, but not in the elderly groups.

CONCLUSIONS

Ninety minutes of normothermic CPB had no deleterious effect on neurocognitive outcome in elderly or chronically diabetic animals, suggesting that CPB in itself is not a sufficient stressor of the rat central nervous system.