External validation of the parsimonious EuroLung risk models: analysis of the Brazilian Lung Cancer Registry

OBJECTIVE

The purpose of this study was to assess performance in the Brazilian Lung Cancer Registry Database by using the parsimonious EuroLung risk models for morbidity and mortality.

METHODS

The EuroLung1 and EuroLung2 models were tested and evaluated through calibration (calibration plot, Brier score, and the Hosmer-Lemeshow test) and discrimination (ROC AUCs), in a national multicenter registry of 1,031 patients undergoing anatomic lung resection.

RESULTS

The evaluation of performance in Brazilian health care facilities utilizing risk-adjustment models, specifically EuroLung1 and EuroLung2, revealed substantial miscalibration, as evidenced by calibration plots and Hosmer-Lemeshow tests in both models. In terms of calibration, EuroLung1 exhibited a calibration plot with overlapping points, characterized by a slope of 1.11 and a Brier score of 0.15; the Hosmer-Lemeshow test yielded a statistically significant p-value of 0.015; and the corresponding ROC AUC was 0.678 (95% CI: 0.636-0.721). The EuroLung2 model displayed better calibration, featuring fewer overlapping points in the calibration plot, with a slope of 1.22, with acceptable discrimination, as indicated by a ROC AUC of 0.756 (95% CI: 0.670-0.842). Both models failed to accurately predict morbidity and mortality outcomes in this specific health care context.

CONCLUSIONS

Discrepancies between the EuroLung model predictions and outcomes in Brazil underscore the need for model refinement and for a probe into inefficiencies in the Brazilian health care system.

A comparative study of four thoracic mortality scores

BACKGROUND

The percentage of patients in resectable stages at initial diagnosis of non-small cell lung cancer (NSCLC) raises due to better screening programs. Therefore, risk prediction models are becoming more critical. Here, we validated and compared four established scoring models, the Thoracoscore, Epithor, Eurloung 2, and the simplified Eurolung 2 (2b), in their ability to predict 30-day mortality.

METHODS

All consecutive patients undergoing anatomical pulmonary resection were included. The performance of the four scoring systems was assessed with Hosmer-Lemeshow goodness-of-fit test (calibration) and receiver operating characteristic (ROC) curves (discrimination). We compared the area under the curve (AUC) of the ROC curves by DeLong's method.

RESULTS

A total of 624 patients underwent surgery for NSCLC at our institution between 2012 and 2018 30-day mortality was 2.2% (14 patients). The AUC for Eurolung 2 and the simplified Eurolung 2 (0.82) were greater than those of the other scoring systems, Epithor (0.71) and Thoracoscore (0.65). In addition, the DeLong analysis showed a significant superiority of Eurolung 2 and Eurolung 2b over the Thoracoscore (p = 0.04); there were no significant differences compared to Epithor.

CONCLUSION

Eurolung 2 and the simplified Eurolung 2 were the favorable scoring systems for predicting 30-day mortality compared to Thoracoscore and Epithor. Therefore, we recommend using Eurolung 2 or the simplified Eurolung 2 for preoperative risk stratification.

External validation of risk prediction scores in patients undergoing anatomic video-assisted thoracoscopic resection

Background

EuroLung Risk scores were established to predict postoperative morbidity and mortality in patients undergoing anatomic lung resections. We aimed to perform an external validation of the EuroLung scores, which were calculated from data of the European Society of Thoracic Surgeons database, in our video-assisted thoracoscopic surgery cohort.

Methods

All available EuroLung scores were calculated for 718 patients scheduled for anatomic video-assisted thoracoscopic surgery resections between 2009 and 2019. Morbidity and mortality according to the definitions of the EuroLung scores were analyzed in a prospectively maintained database.

Results

Overall observed complication rate was 10.45%. Scores showed weak individual correlation (η = 0.155–0.174). The EuroLung1 app score showed the biggest area under the receiver operative characteristic (ROC) curve with 0.660. Binary logistic regression analysis showed that predicted postoperative forced expiratory volume in 1 s was associated with increased complications in both EuroLung1 and parsimonious EuroLung1 scores. Thirty-day mortality was 0.7% (predicted 1.10–1.40%) and was associated with predicted postoperative forced expiratory volume in 1 s for both EuroLung2 and parsimonious EuroLung2 scores. The EuroLung2 (2016) showed the biggest area under the ROC curve with 0.673. Only a very weak eta correlation between predicted and observed mortality was found for both aggregate EuroLung2, EuroLung2 (2016), EuroLung2 (2019), and parsimonious EuroLung2 (2016) (η = 0.025/0.015/0.011/0.009).

Conclusion

EuroLung scores help to estimate postoperative morbidity. However, even with the highest aggregate EuroLung scores possible only 50% suffer from postoperative morbidity. Although calibration of the scores was acceptable, discrimination between predicted and observed events was poor. Therefore, individual correlation between predicted and observed events is weak. Therefore, EuroLung scores may be best used to compare institutional quality of care to the European Society of Thoracic Surgeons database but should not be used to preclude patients from surgical treatment.

Prediction of postoperative cardiopulmonary complications after lung resection in a Chinese population: A machine learning-based study

Background

Approximately 20% of patients with lung cancer would experience postoperative cardiopulmonary complications after anatomic lung resection. Current prediction models for postoperative complications were not suitable for Chinese patients. This study aimed to develop and validate novel prediction models based on machine learning algorithms in a Chinese population.

Methods

Patients with lung cancer receiving anatomic lung resection and no neoadjuvant therapies from September 1, 2018 to August 31, 2019 were enrolled. The dataset was split into two cohorts at a 7:3 ratio. The logistic regression, random forest, and extreme gradient boosting were applied to construct models in the derivation cohort with 5-fold cross validation. The validation cohort accessed the model performance. The area under the curves measured the model discrimination, while the Spiegelhalter z test evaluated the model calibration.

Results

A total of 1085 patients were included, and 760 were assigned to the derivation cohort. 8.4% and 8.0% of patients experienced postoperative cardiopulmonary complications in the two cohorts. All baseline characteristics were balanced. The values of the area under the curve were 0.728, 0.721, and 0.767 for the logistic, random forest and extreme gradient boosting models, respectively. No significant differences existed among them. They all showed good calibration (p > 0.05). The logistic model consisted of male, arrhythmia, cerebrovascular disease, the percentage of predicted postoperative forced expiratory volume in one second, and the ratio of forced expiratory volume in one second to forced vital capacity. The last two variables, the percentage of forced vital capacity and age ranked in the top five important variables for novel machine learning models. A nomogram was plotted for the logistic model.

Conclusion

Three models were developed and validated for predicting postoperative cardiopulmonary complications among Chinese patients with lung cancer. They all exerted good discrimination and calibration. The percentage of predicted postoperative forced expiratory volume in one second and the ratio of forced expiratory volume in one second to forced vital capacity might be the most important variables. Further validation in different scenarios is still warranted.

External validation of five predictive models for postoperative cardiopulmonary morbidity in a Chinese population receiving lung resection

BACKGROUND

No postoperative cardiopulmonary morbidity models have been developed or validated in Chinese patients with lung resection. This study aims to externally validate five predictive models, including Eurolung models, the Brunelli model and the Age-adjusted Charlson Comorbidity Index, in a Chinese population.

METHODS

Patients with lung cancer who underwent anatomic lung resection between 2018/09/01 and 2019/08/31 in our center were involved. Model discrimination was assessed by the area under the receiver operating characteristic curve. Model calibration was evaluated by the Hosmer-Lemeshow test. Calibration curves were plotted. Specificity, sensitivity, negative predictive value, positive predictive value and accuracy were calculated. Model updating was achieved by re-estimating the intercept and/or the slope of the linear predictor and re-estimating all coefficients.

RESULTS

Among 1085 patients, 91 patients had postoperative cardiopulmonary complications defined by the European Society of Thoracic Surgeons. For original models, only parsimonious Eurolung1 had acceptable discrimination (area under the receiver operating characteristic curve = 0.688, 95% confidence interval 0.630-0.745) and calibration (p = 0.23 > 0.05) abilities simultaneously. Its sensitivity, specificity, positive predictive value, negative predictive value and accuracy were 0.700, 0.649, 0.153, 0.960 and 0.653, respectively. In the secondary analysis, increased pleural effusion (n = 94), which was nonchylous and nonpurulent, was labeled as a kind of postoperative complication. The area under the receiver operating characteristic curve of the models increased slightly, but all models were miscalibrated. The original Eurolung1 model had the highest discrimination ability but poor calibration, and thus it was updated by three methods. After model updating, new models showed good calibration and small improvements in discrimination. The discrimination ability was still merely acceptable.

CONCLUSIONS

Overall, none of the models performed well on postoperative cardiopulmonary morbidity prediction in this Chinese population. The original parsimonious Eurolung1 and the updated Eurolung1 were the best-performing models on morbidity prediction, but their discrimination ability only achieved an acceptable level. A multicenter study with more relevant variables and sophisticated statistical methods is warranted to develop new models among Chinese patients in the future.

A systematic review of risk prediction models for perioperative mortality after thoracic surgery

OBJECTIVES

Guidelines advocate that patients being considered for thoracic surgery should undergo a comprehensive preoperative risk assessment. Multiple risk prediction models to estimate the risk of mortality after thoracic surgery have been developed, but their quality and performance has not been reviewed in a systematic way. The objective was to systematically review these models and critically appraise their performance.

METHODS

The Cochrane Library and the MEDLINE database were searched for articles published between 1990 and 2019. Studies that developed or validated a model predicting perioperative mortality after thoracic surgery were included. Data were extracted based on the checklist for critical appraisal and data extraction for systematic reviews of prediction modelling studies.

RESULTS

A total of 31 studies describing 22 different risk prediction models were identified. There were 20 models developed specifically for thoracic surgery with two developed in other surgical specialties. A total of 57 different predictors were included across the identified models. Age, sex and pneumonectomy were the most frequently included predictors in 19, 13 and 11 models, respectively. Model performance based on either discrimination or calibration was inadequate for all externally validated models. The most recent data included in validation studies were from 2018. Risk of bias (assessed using Prediction model Risk Of Bias ASsessment Tool) was high for all except two models.

CONCLUSIONS

Despite multiple risk prediction models being developed to predict perioperative mortality after thoracic surgery, none could be described as appropriate for contemporary thoracic surgery. Contemporary validation of available models or new model development is required to ensure that appropriate estimates of operative risk are available for contemporary thoracic surgical practice.

Validation of the Japanese National Clinical Database Risk calculator for lung cancer surgery focused on postoperative morbidity

OBJECTIVES

To validate the efficacy of the Japanese National Clinical Database risk calculator, which predicts major morbidity in lung cancer surgery based on preoperative clinical characteristics.

METHODS

In total, 660 patients who underwent complete surgical resection of primary lung cancer were enrolled. The predicted rate of major morbidity determined using the risk calculator was compared between the patients with and without major morbidity. We performed receiver operating characteristic curve analysis to determine their cut-off values to predict major morbidity and assessed the associated factors with major morbidity. Major morbidity was defined as the Clavien-Dindo classification grade IIIa or greater.

RESULTS

The predicted rate of major morbidity was significantly higher in patients with major morbidity than in those without (P < 0.001). The cut-off value of the predicted rate of major morbidity to predict major morbidity was 3.0% (area under curve 0.741; sensitivity and specificity, 85.3% and 54.3%, respectively). The predicted rate of major morbidity ≥ 3.0% was significantly associated with occurrence of major morbidities (odds ratio 6.9; 95% confidence interval 2.63-18.04; P < 0.001) and the predicted rate of major morbidity had the highest odds ratio over other risk factors. This condition, namely the predicted rate of major morbidity ≥ 3.0%, was met in 315 (47%) of the total cases. However, only 29 (9%) of these cases had major morbidity in practice.

CONCLUSIONS

The risk calculator was fairly useful for estimating high-risk patients; however, it was not possible to identify a specific cut-off value to predict major morbidity in this cohort.

Ninety-Day Mortality: Redefining the Perioperative Period After Lung Resection

Operative mortality is an important outcome for patients, surgeons, healthcare institutions, and policy makers. Although measures of perioperative mortality have conventionally been limited to in-hospital and 30-day mortality (or a composite endpoint combining both), there is a large body of evidence emerging to support the extension of the perioperative period after lung resection to a minimum of 90 days after surgery. Several large-volume studies from centers across the world have reported that 90-day mortality after lung resection is double 30-day mortality. Hence, true perioperative mortality after lung resection is likely to be significantly higher than what is currently reported. In the contemporary era, where new treatment modalities such as stereotactic ablative body radiotherapy are emerging as viable nonsurgical alternatives for the treatment of lung cancer, accurate estimation of perioperative risk and reliable reporting of perioperative mortality are of particular importance. It is likely that shifting the discussion from 30-day to 90-day mortality will lead to altered decision making, particularly for specific patient subgroups at an increased risk of 90-day mortality. We believe that 90-day mortality should be adopted as the standard measure of perioperative mortality after lung resection and that strategies to reduce the risk of mortality within 90 days of surgery should be investigated.

External validation of six existing multivariable clinical prediction models for short-term mortality in patients undergoing lung resection

 

OBJECTIVES

National guidelines advocate the use of clinical prediction models to estimate perioperative mortality for patients undergoing lung resection. Several models have been developed that may potentially be useful but contemporary external validation studies are lacking. The aim of this study was to validate existing models in a multicentre patient cohort.

METHODS

The Thoracoscore, Modified Thoracoscore, Eurolung, Modified Eurolung, European Society Objective Score and Brunelli models were validated using a database of 6600 patients who underwent lung resection between 2012 and 2018. Models were validated for in-hospital or 30-day mortality (depending on intended outcome of each model) and also for 90-day mortality. Model calibration (calibration intercept, calibration slope, observed to expected ratio and calibration plots) and discrimination (area under receiver operating characteristic curve) were assessed as measures of model performance.

RESULTS

Mean age was 66.8 years (±10.9 years) and 49.7% (n = 3281) of patients were male. In-hospital, 30-day, perioperative (in-hospital or 30-day) and 90-day mortality were 1.5% (n = 99), 1.4% (n = 93), 1.8% (n = 121) and 3.1% (n = 204), respectively. Model area under the receiver operating characteristic curves ranged from 0.67 to 0.73. Calibration was inadequate in five models and mortality was significantly overestimated in five models. No model was able to adequately predict 90-day mortality.

CONCLUSIONS

Five of the validated models were poorly calibrated and had inadequate discriminatory ability. The modified Eurolung model demonstrated adequate statistical performance but lacked clinical validity. Development of accurate models that can be used to estimate the contemporary risk of lung resection is required.