Diagnostic and prognostic prediction models in ventilator-associated pneumonia: Systematic review and meta-analysis of prediction modelling studies

Prediction models for the risk of ventilator-associated pneumonia in patients on mechanical ventilation: A systematic review and meta-analysis

BACKGROUND

Identifying patients at risk of ventilator-associated pneumonia through prediction models can facilitate medical decision-making. Our objective was to evaluate the current models for ventilator-associated pneumonia in patients with mechanical ventilation.

METHODS

Nine databases systematically retrieved from establishment to March 6, 2024. Two independent reviewers performed study selection, data extraction, and quality assessment, respectively. The Prediction Model Risk of Bias Assessment Tool was used to evaluate the risk of model bias and applicability. Stata 17.0 was used to conduct a meta-analysis of discrimination of model validation.

RESULTS

The total of 34 studies were included, with reported 52 prediction models. The most frequent predictors in the models were mechanical ventilation duration, length of intensive care unit stay, and age. Each study was essentially considered having a high risk of bias. A meta-analysis of 17 studies containing 33 models with validation was performed with a pooled area under the receiver-operating curve of 0.80 (95% confidence interval: 0.78-0.83).

CONCLUSIONS

Despite the relatively excellent performance of the models, there is a high risk of bias of the model development process. Enhancing the methodological quality, especially the external validation, practical application, and optimization of the models need urgent attention.

Evaluating automated machine learning platforms for use in healthcare

Objective

To describe development and application of a checklist of criteria for selecting an automated machine learning (Auto ML) platform for use in creating clinical ML models.

Materials and Methods

Evaluation criteria for selecting an Auto ML platform suited to ML needs of a local health district were developed in 3 steps: (1) identification of key requirements, (2) a market scan, and (3) an assessment process with desired outcomes.

Results

The final checklist comprising 21 functional and 6 non-functional criteria was applied to vendor submissions in selecting a platform for creating a ML heparin dosing model as a use case.

Discussion

A team of clinicians, data scientists, and key stakeholders developed a checklist which can be adapted to ML needs of healthcare organizations, the use case providing a relevant example.

Conclusion

An evaluative checklist was developed for selecting Auto ML platforms which requires validation in larger multi-site studies.

Quantitative ultrasound imaging parameters in patients with cancerous thyroid nodules: development of a diagnostic model

OBJECTIVE

This study aimed to develop a diagnostic model utilizing quantitative ultrasound parameters to accurately differentiate benign from malignant thyroid nodules.

METHODS

A retrospective analysis of 194 patients with thyroid nodules, encompassing 65 malignant and 129 benign cases, was performed. Clinical data, ultrasound characteristics, and hemodynamic indicators were compared. Receiver operating characteristic (ROC) curves and logistic regression analysis identified independent diagnostic markers.

RESULTS

No significant differences in clinical data were observed between the groups (P>0.05). Malignant nodules, however, were more likely to exhibit solid composition, hypoechoicity, irregular shapes, calcifications, central blood flow, and unclear margins (P<0.05). Hemodynamic parameters showed that malignant nodules had lower end-diastolic volume (EDV) but higher peak systolic velocity (PSV), resistive index (RI), and vascularization flow index (VFI) (P<0.001). Independent diagnostic factors identified included calcification, margin definition, RI, and VFI. A risk prediction model was formulated, demonstrating significantly lower scores for benign nodules (P<0.0001), achieving an ROC area of 0.964.

CONCLUSION

Color Doppler ultrasound effectively distinguishes malignant from benign thyroid nodules. The diagnostic model emphasizes the importance of calcification, margin clarity, RI, and VFI as critical elements, enhancing the accuracy of thyroid nodule characterization and facilitating informed clinical decisions.

Ventilator-Associated Pneumonia Prediction Models Based on AI: Scoping Review

Background

Ventilator-associated pneumonia (VAP) is a serious complication of mechanical ventilation therapy that affects patients' treatments and prognoses. Owing to its excellent data mining capabilities, artificial intelligence (AI) has been increasingly used to predict VAP.

Objective

This paper reviews VAP prediction models that are based on AI, providing a reference for the early identification of high-risk groups in future clinical practice.

Methods

A scoping review was conducted in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. The Wanfang database, the Chinese Biomedical Literature Database, Cochrane Library, Web of Science, PubMed, MEDLINE, and Embase were searched to identify relevant articles. Study selection and data extraction were independently conducted by 2 reviewers. The data extracted from the included studies were synthesized narratively.

Results

Of the 137 publications retrieved, 11 were included in this scoping review. The included studies reported the use of AI for predicting VAP. All 11 studies predicted VAP occurrence, and studies on VAP prognosis were excluded. Further, these studies used text data, and none of them involved imaging data. Public databases were the primary sources of data for model building (studies: 6/11, 55%), and 5 studies had sample sizes of <1000. Machine learning was the primary algorithm for studying the VAP prediction models. However, deep learning and large language models were not used to construct VAP prediction models. The random forest model was the most commonly used model (studies: 5/11, 45%). All studies only performed internal validations, and none of them addressed how to implement and apply the final model in real-life clinical settings.

Conclusions

This review presents an overview of studies that used AI to predict and diagnose VAP. AI models have better predictive performance than traditional methods and are expected to provide indispensable tools for VAP risk prediction in the future. However, the current research is in the model construction and validation stage, and the implementation of and guidance for clinical VAP prediction require further research.

Early prediction of ventilator-associated pneumonia with machine learning models: A systematic review and meta-analysis of prediction model performance✰

BACKGROUND

Machine learning-based prediction models can catalog, classify, and correlate large amounts of multimodal data to aid clinicians at diagnostic, prognostic, and therapeutic levels. Early prediction of ventilator-associated pneumonia (VAP) may accelerate the diagnosis and guide preventive interventions. The performance of a variety of machine learning-based prediction models were analyzed among adults undergoing invasive mechanical ventilation.

METHODS

This systematic review and meta-analysis was conducted in accordance with the Cochrane Collaboration. Machine learning-based prediction models were identified from a search of nine multi-disciplinary databases. Two authors independently selected and extracted data using predefined criteria and data extraction forms. The predictive performance, the interpretability, the technological readiness level, and the risk of bias of the included studies were evaluated.

RESULTS

Final analysis included 10 static prediction models using supervised learning. The pooled area under the receiver operating characteristics curve, sensitivity, and specificity for VAP were 0.88 (95 % CI 0.82-0.94, I2 98.4 %), 0.72 (95 % CI 0.45-0.98, I2 97.4 %) and 0.90 (95 % CI 0.85-0.94, I2 97.9 %), respectively. All included studies had either a high or unclear risk of bias without significant improvements in applicability. The care-related risk factors for the best performing models were the duration of mechanical ventilation, the length of ICU stay, blood transfusion, nutrition strategy, and the presence of antibiotics.

CONCLUSION

A variety of the prediction models, prediction intervals, and prediction windows were identified to facilitate timely diagnosis. In addition, care-related risk factors susceptible for preventive interventions were identified. In future, there is a need for dynamic machine learning models using time-depended predictors in conjunction with feature importance of the models to predict real-time risk of VAP and related outcomes to optimize bundled care.

Systematic reviews of machine learning in healthcare: a literature review

INTRODUCTION

The increasing availability of data and computing power has made machine learning (ML) a viable approach to faster, more efficient healthcare delivery.

METHODS

A systematic literature review (SLR) of published SLRs evaluating ML applications in healthcare settings published between1 January 2010 and 27 March 2023 was conducted.

RESULTS

In total 220 SLRs covering 10,462 ML algorithms were reviewed. The main application of AI in medicine related to the clinical prediction and disease prognosis in oncology and neurology with the use of imaging data. Accuracy, specificity, and sensitivity were provided in 56%, 28%, and 25% SLRs respectively. Internal and external validation was reported in 53% and less than 1% of the cases respectively. The most common modeling approach was neural networks (2,454 ML algorithms), followed by support vector machine and random forest/decision trees (1,578 and 1,522 ML algorithms, respectively).

EXPERT OPINION

The review indicated considerable reporting gaps in terms of the ML's performance, both internal and external validation. Greater accessibility to healthcare data for developers can ensure the faster adoption of ML algorithms into clinical practice.

VAP risk index: Early prediction and hospital phenotyping of ventilator-associated pneumonia using machine learning

BACKGROUND

Ventilator-associated pneumonia (VAP) is a leading cause of morbidity and mortality in intensive care units (ICUs). Early identification of patients at risk of VAP enables early intervention, which in turn improves patient outcomes. We developed a predictive model for individualized risk assessment utilizing machine learning to identify patients at risk of developing VAP.

METHODS

The Philips eRI dataset, a multi-institution electronic medical record (EMR), was used for model development. For adult (≥18y) patients, we propose a set of criteria using indications of the start of a new antibiotic treatment temporally contiguous to a microbiological test to mark suspected infection events, of which those with a positive culture are labeled as presumed VAP if 1) the event occurs at least 48 h after intubation, and 2) there are no indications of community-acquired pneumonia (CAP) or other hospital-acquired infections (HAI) in the patient charts. The resulting VAP and no-VAP (control) cases were then used to build an ensemble of decision trees to predict the risk of VAP in the next 24 h using data on patients' demographics, vitals, labs, and ventilator settings.

RESULTS

The resulting model predicts the development of VAP 24 h in advance with an AUC of 76 % and AUPRC of 75 %. Additionally, we group hospitals that are similar in healthcare processes into distinct clusters and characterize VAP prediction for the identified hospital clusters. We show inter-hospital (teaching status and healthcare processes) and cohort-specific (age groups, gender, early vs late VAP, ICU mortality status) differences in VAP prediction and associated symptomologies.

CONCLUSIONS

Our proposed VAP criteria use clinical actions to mark incidences of presumed VAP infection, which enables the development of models for early detection of these events. We curated a patient cohort using these criteria and used it to build a model for predicting impending VAP events prior to clinical suspicions. We present a clustering approach for tailoring the VAP prediction model for different hospital types based on their EMR data characteristics. The model provides an instantaneous risk score that allows early interventions and confirmatory diagnostic actions.

Prevention of ventilator-associated pneumonia through care bundles: A systematic review and meta-analysis

Background

Ventilator-associated pneumonia (VAP) represents a common hospital-acquired infection among mechanically ventilated patients. We summarized evidence concerning ventilator care bundles to prevent VAP.

Methods

A systematic review and meta-analysis were performed. Randomized controlled trials and controlled observational studies of adults undergoing mechanical ventilation (MV) for at least 48 h were considered for inclusion. Outcomes of interest were the number of VAP episodes, duration of MV, hospital and intensive care unit (ICU) length of stay, and mortality. A systematic search was conducted in the MEDLINE, the Cochrane Library, and the Web of Science between 1985 and 2022. Results are reported as odds ratio (OR) or mean difference (MD) with 95% confidence intervals (CI). The PROSPERO registration number is CRD42022341780.

Results

Thirty-six studies including 116,873 MV participants met the inclusion criteria. A total of 84,031 participants underwent care bundles for VAP prevention. The most reported component of the ventilator bundle was head-of-bed elevation (n=83,146), followed by oral care (n=80,787). A reduction in the number of VAP episodes was observed among those receiving ventilator care bundles, compared with the non-care bundle group (OR=0.42, 95% CI: 0.33, 0.54). Additionally, the implementation of care bundles decreased the duration of MV (MD=-0.59, 95% CI: -1.03, -0.15) and hospital length of stay (MD=-1.24, 95% CI: -2.30, -0.18) in studies where educational activities were part of the bundle. Data regarding mortality were inconclusive.

Conclusions

The implementation of ventilator care bundles reduced the number of VAP episodes and the duration of MV in adult ICUs. Their application in combination with educational activities seemed to improve clinical outcomes.

Novel pneumonia score based on a machine learning model for predicting mortality in pneumonia patients on admission to the intensive care unit

BACKGROUND

Scores for predicting the long-term mortality of severe pneumonia are lacking. The purpose of this study is to use machine learning methods to develop new pneumonia scores to predict the 1-year mortality and hospital mortality of pneumonia patients on admission to the intensive care unit (ICU).

METHODS

The study population was screened from the MIMIC-IV and eICU databases. The main outcomes evaluated were 1-year mortality and hospital mortality in the MIMIC-IV database and hospital mortality in the eICU database. From the full data set, we separated patients diagnosed with community-acquired pneumonia (CAP) and ventilator-associated pneumonia (VAP) for subgroup analysis. We used common shallow machine learning algorithms, including logistic regression, decision tree, random forest, multilayer perceptron and XGBoost.

RESULTS

The full data set of the MIMIC-IV database contained 4697 patients, while that of the eICU database contained 13760 patients. We defined a new pneumonia score, the "Integrated CCI-APS", using a multivariate logistic regression model including six variables: metastatic solid tumor, Charlson Comorbidity Index, readmission, congestive heart failure, age, and Acute Physiology Score III. The area under the curve (AUC) and accuracy of the integrated CCI-APS were assessed in three data sets (full, CAP, and VAP) using both the test set derived from the MIMIC-IV database and the external validation set derived from the eICU database. The AUC value ranges in predicting 1-year and hospital mortality were 0.784-0.797 and 0.691-0.780, respectively, and the corresponding accuracy ranges were 0.723-0.725 and 0.641-0.718, respectively.

CONCLUSIONS

The main contribution of this study was a benchmark for using machine learning models to build pneumonia scores. Based on the idea of integrated learning, we propose a new integrated CCI-APS score for severe pneumonia. In the prediction of 1-year mortality and hospital mortality, our new pneumonia score outperformed the existing score.

Ventilator-associated events: From surveillance to optimizing management

Mechanical ventilation (MV) is a life-support therapy that may predispose to morbid and lethal complications, with ventilator-associated pneumonia (VAP) being the most prevalent. In 2013, the Center for Disease Control (CDC) defined criteria for ventilator-associated events (VAE). Ten years later, a growing number of studies assessing or validating its clinical applicability and the potential benefits of its inclusion have been published. Surveillance with VAE criteria is retrospective and the focus is often on a subset of patients with higher than lower severity. To date, it is estimated that around 30% of ventilated patients in the intensive care unit (ICU) develop VAE. While surveillance enhances the detection of infectious and non-infectious MV-related complications that are severe enough to impact the patient's outcomes, there are still many gaps in its classification and management. In this review, we provide an update by discussing VAE etiologies, epidemiology, and classification. Preventive strategies on optimizing ventilation, sedative and neuromuscular blockade therapy, and restrictive fluid management are warranted. An ideal VAE bundle is likely to minimize the period of intubation. We believe that it is time to progress from just surveillance to clinical care. Therefore, with this review, we have aimed to provide a roadmap for future research on the subject.

Construction and Validation of a Predictive Model for the Risk of Ventilator-Associated Pneumonia in Elderly ICU Patients

Background

Ventilator-associated pneumonia (VAP) is among the most important hospital-acquired infections in an intensive-care unit setting. However, clinical practice lacks effective theoretical tools for preventing VAP in the elderly.

Aim

To describe the independent factors associated with VAP in elderly intensive-care unit (ICU) patients on mechanical ventilation (MV) and to construct a risk prediction model.

Methods

A total of 1851 elderly patients with MV in ICUs from January 2015 to September 2019 were selected from 12 tertiary hospitals. Study subjects were divided into a model group (n = 1219) and a validation group (n = 632). Two groups of patients were divided into a VAP group and a non-VAP group and compared. Univariate and logistic regression analyses were used to explore influencing factors for VAP in elderly ICU patients with MV, establish a risk prediction model, and draw a nomogram. We used the area under the receiver operating characteristic curve (AUROC) and the Hosmer-Lemeshow goodness-of-fit test to evaluate the predictive effect of the model. Findings regarding the length of ICU stay, surgery, C-reactive protein (CRP), and the number of reintubations were independent risk factors for VAP in elderly ICU patients with MV. Predictive-model verification results showed that the area under the curve (AUC) of VAP risk after MV in the modeling and verification groups was 0.859 and 0.813 (P < 0.001), respectively, while P values for the Hosmer-Lemeshow test in these two groups were 0.365 and 0.485, respectively.

Conclusion

The model could effectively predict the occurrence of VAP in elderly patients with MV in ICUs. This study is a retrospective study, so it has not been registered as a clinical study.

Predicting pneumonia during hospitalization in flail chest patients using machine learning approaches

Objective

Pneumonia is a common pulmonary complication of flail chest, causing high morbidity and mortality rates in affected patients. The existing methods for identifying pneumonia have low accuracy, and their use may delay antimicrobial therapy. However, machine learning can be combined with electronic medical record systems to identify information and assist in quick clinical decision-making. Our study aimed to develop a novel machine-learning model to predict pneumonia risk in flail chest patients.

Methods

From January 2011 to December 2021, the electronic medical records of 169 adult patients with flail chest at a tertiary teaching hospital in an urban level I Trauma Centre in Chongqing were retrospectively analysed. Then, the patients were randomly divided into training and test sets at a ratio of 7:3. Using the Fisher score, the best subset of variables was chosen. The performance of the seven models was evaluated by computing the area under the receiver operating characteristic curve (AUC). The output of the XGBoost model was shown using the Shapley Additive exPlanation (SHAP) method.

Results

Of 802 multiple rib fracture patients, 169 flail chest patients were eventually included, and 86 (50.80%) were diagnosed with pneumonia. The XGBoost model performed the best among all seven machine-learning models. The AUC of the XGBoost model was 0.895 (sensitivity: 84.3%; specificity: 80.0%).Pneumonia in flail chest patients was associated with several features: systolic blood pressure, pH value, blood transfusion, and ISS.

Conclusion

Our study demonstrated that the XGBoost model with 32 variables had high reliability in assessing risk indicators of pneumonia in flail chest patients. The SHAP method can identify vital pneumonia risk factors, making the XGBoost model's output clinically meaningful.