A Novel “Maximizing Kappa” Approach for Assessing the Ability of a Diagnostic Marker and Its Optimal Cutoff Value

A Comparison of Additional Benefit Assessment Methods for Time-to-Event Endpoints Using Hazard Ratio Point Estimates or Confidence Interval Limits by Means of a Simulation Study

BACKGROUND

For time-to-event endpoints, three additional benefit assessment methods have been developed aiming at an unbiased knowledge about the magnitude of clinical benefit of newly approved treatments. The American Society of Clinical Oncology (ASCO) defines a continuous score using the hazard ratio point estimate (HR-PE). The European Society for Medical Oncology (ESMO) and the German Institute for Quality and Efficiency in Health Care (IQWiG) developed methods with an ordinal outcome using lower and upper limits of the 95% HR confidence interval (HR-CI), respectively. We describe all three frameworks for additional benefit assessment aiming at a fair comparison across different stakeholders. Furthermore, we determine which ASCO score is consistent with which ESMO/IQWiG category.

METHODS

In a comprehensive simulation study with different failure time distributions and treatment effects, we compare all methods using Spearman's correlation and descriptive measures. For determination of ASCO values consistent with categories of ESMO/IQWiG, maximizing weighted Cohen's Kappa approach was used.

RESULTS

Our research depicts a high positive relationship between ASCO/IQWiG and a low positive relationship between ASCO/ESMO. An ASCO score smaller than 17, 17 to 20, 20 to 24, and greater than 24 corresponds to ESMO categories. Using ASCO values of 21 and 38 as cutoffs represents IQWiG categories.

LIMITATIONS

We investigated the statistical aspects of the methods and hence implemented slightly reduced versions of all methods.

CONCLUSIONS

IQWiG and ASCO are more conservative than ESMO, which often awards the maximal category independent of the true effect and is at risk of overcompensating with various failure time distributions. ASCO has similar characteristics as IQWiG. Delayed treatment effects and underpowered/overpowered studies influence all methods in some degree. Nevertheless, ESMO is the most liberal one.

HIGHLIGHTS

For the additional benefit assessment, the American Society of Clinical Oncology (ASCO) uses the hazard ratio point estimate (HR-PE) for their continuous score. In contrast, the European Society for Medical Oncology (ESMO) and the German Institute for Quality and Efficiency in Health Care (IQWiG) use the lower and upper 95% HR confidence interval (HR-CI) to specific thresholds, respectively. ESMO generously assigns maximal scores, while IQWiG is more conservative.This research provides the first comparison between IQWiG and ASCO and describes all three frameworks for additional benefit assessment aiming for a fair comparison across different stakeholders. Furthermore, thresholds for ASCO consistent with ESMO and IQWiG categories are determined, enabling a comparison of the methods in practice in a fair manner.IQWiG and ASCO are the more conservative methods, while ESMO awards high percentages of maximal categories, especially with various failure time distributions. ASCO has similar characteristics as IQWiG. Delayed treatment effects and under/-overpowered studies influence all methods. Nevertheless, ESMO is the most liberal one. An ASCO score smaller than 17, 17 to 20, 20 to 24, and greater than 24 correspond to the categories of ESMO. Using ASCO values of 21 and 38 as cutoffs represents categories of IQWiG.

Machine Learning Algorithms Predict Successful Weaning From Mechanical Ventilation Before Intubation: Retrospective Analysis From the Medical Information Mart for Intensive Care IV Database

BACKGROUND

The prediction of successful weaning from mechanical ventilation (MV) in advance of intubation can facilitate discussions regarding end-of-life care before unnecessary intubation.

OBJECTIVE

We aimed to develop a machine learning-based model that predicts successful weaning from ventilator support based on routine clinical and laboratory data taken before or immediately after intubation.

METHODS

We used the Medical Information Mart for Intensive Care IV database, which is an open-access database covering 524,740 admissions of 382,278 patients in Beth Israel Deaconess Medical Center, United States, from 2008 to 2019. We selected adult patients who underwent MV in the intensive care unit (ICU). Clinical and laboratory variables that are considered relevant to the prognosis of the patient in the ICU were selected. Data collected before or within 24 hours of intubation were used to develop machine learning models that predict the probability of successful weaning within 14 days of ventilator support. Developed models were integrated into an ensemble model. Performance metrics were calculated by 5-fold cross-validation for each model, and a permutation feature importance and Shapley additive explanations analysis was conducted to better understand the impacts of individual variables on outcome prediction.

RESULTS

Of the 23,242 patients, 19,025 (81.9%) patients were successfully weaned from MV within 14 days. Using the preselected 46 clinical and laboratory variables, the area under the receiver operating characteristic curve of CatBoost classifier, random forest classifier, and regularized logistic regression classifier models were 0.860 (95% CI 0.852-0.868), 0.855 (95% CI 0.848-0.863), and 0.823 (95% CI 0.813-0.832), respectively. Using the ensemble voting classifier using the 3 models above, the final model revealed the area under the receiver operating characteristic curve of 0.861 (95% CI 0.853-0.869), which was significantly better than that of Simplified Acute Physiology Score II (0.749, 95% CI 0.742-0.756) and Sequential Organ Failure Assessment (0.588, 95% CI 0.566-0.609). The top features included lactate and anion gap. The model's performance achieved a plateau with approximately the top 21 variables.

CONCLUSIONS

We developed machine learning algorithms that can predict successful weaning from MV in advance to intubation in the ICU. Our models can aid the appropriate management for patients who hesitate to decide on ventilator support or meaningless end-of-life care.

Comparative Assessment of Clinical Benefit Using the ESMO-Magnitude of Clinical Benefit Scale Version 1.1 and the ASCO Value Framework Net Health Benefit Score

PURPOSE

To better understand the European Society for Medical Oncology-Magnitude of Clinical Benefit Scale version 1.1 (ESMO-MCBS v1.1) and the ASCO Value Framework Net Health Benefit score version 2 (ASCO-NHB v2), ESMO and ASCO collaborated to evaluate the concordance between the frameworks when used to assess clinical benefit attributable to new therapies.

METHODS

The 102 randomized controlled trials in the noncurative setting already evaluated in the field testing of ESMO-MCBS v1.1 were scored using ASCO-NHB v2 by its developers. Measures of agreement between the frameworks were calculated and receiver operating characteristic curves used to define thresholds for the ASCO-NHB v2 corresponding to ESMO-MCBS v1.1 categories. Studies with discordant scoring were identified and evaluated to understand the reasons for discordance.

RESULTS

The correlation of the 102 pairs of scores for studies in the noncurative setting is estimated to be 0.68 (Spearman's rank correlation coefficient; overall survival, 0.71; progression-free survival, 0.67). Receiver operating characteristic curves identified thresholds for ASCO-NHB v2 for facilitating comparisons with ESMO-MCBS v1.1 categories. After applying pragmatic threshold scores of 40 or less (ASCO-NHB v2) and 2 or less (ESMO-MCBS v1.1) for low benefit and 45 or greater (ASCO-NHB v2) and 4 to 5 (ESMO-MCBS v1.1) for substantial benefit, 37 discordant studies were identified. Major factors that contributed to discordance were different approaches to evaluation of relative and absolute gain for overall survival and progression-free survival, crediting tail of the curve gains, and assessing toxicity.

CONCLUSION

The agreement between the frameworks was higher than observed in other studies that sought to compare them. The factors that contributed to discordant scores suggest potential approaches to improve convergence between the scales.