Interval Coded Scoring: a toolbox for interpretable scoring systems

Over the last decades, clinical decision support systems have been gaining importance. They help clinicians to make effective use of the overload of available information to obtain correct diagnoses and appropriate treatments. However, their power often comes at the cost of a black box model which cannot be interpreted easily. This interpretability is of paramount importance in a medical setting with regard to trust and (legal) responsibility. In contrast, existing medical scoring systems are easy to understand and use, but they are often a simplified rule-of-thumb summary of previous medical experience rather than a well-founded system based on available data. Interval Coded Scoring (ICS) connects these two approaches, exploiting the power of sparse optimization to derive scoring systems from training data. The presented toolbox interface makes this theory easily applicable to both small and large datasets. It contains two possible problem formulations based on linear programming or elastic net. Both allow to construct a model for a binary classification problem and establish risk profiles that can be used for future diagnosis. All of this requires only a few lines of code. ICS differs from standard machine learning through its model consisting of interpretable main effects and interactions. Furthermore, insertion of expert knowledge is possible because the training can be semi-automatic. This allows end users to make a trade-off between complexity and performance based on cross-validation results and expert knowledge. Additionally, the toolbox offers an accessible way to assess classification performance via accuracy and the ROC curve, whereas the calibration of the risk profile can be evaluated via a calibration curve. Finally, the colour-coded model visualization has particular appeal if one wants to apply ICS manually on new observations, as well as for validation by experts in the specific application domains. The validity and applicability of the toolbox is demonstrated by comparing it to standard Machine Learning approaches such as Naive Bayes and Support Vector Machines for several real-life datasets. These case studies on medical problems show its applicability as a decision support system. ICS performs similarly in terms of classification and calibration. Its slightly lower performance is countered by its model simplicity which makes it the method of choice if interpretability is a key issue.

Visualizing Risk Prediction Models

OBJECTIVE

Risk prediction models can assist clinicians in making decisions. To boost the uptake of these models in clinical practice, it is important that end-users understand how the model works and can efficiently communicate its results. We introduce novel methods for interpretable model visualization.

METHODS

The proposed visualization techniques are applied to two prediction models from the Framingham Heart Study for the prediction of intermittent claudication and stroke after atrial fibrillation. We represent models using color bars, and visualize the risk estimation process for a specific patient using patient-specific contribution charts.

RESULTS

The color-based model representations provide users with an attractive tool to instantly gauge the relative importance of the predictors. The patient-specific representations allow users to understand the relative contribution of each predictor to the patient's estimated risk, potentially providing insightful information on which to base further patient management. Extensions towards non-linear models and interactions are illustrated on an artificial dataset.

CONCLUSION

The proposed methods summarize risk prediction models and risk predictions for specific patients in an alternative way. These representations may facilitate communication between clinicians and patients.

Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study

OBJECTIVES

To develop a risk prediction model to preoperatively discriminate between benign, borderline, stage I invasive, stage II-IV invasive, and secondary metastatic ovarian tumours.

DESIGN

Observational diagnostic study using prospectively collected clinical and ultrasound data.

SETTING

24 ultrasound centres in 10 countries.

PARTICIPANTS

Women with an ovarian (including para-ovarian and tubal) mass and who underwent a standardised ultrasound examination before surgery. The model was developed on 3506 patients recruited between 1999 and 2007, temporally validated on 2403 patients recruited between 2009 and 2012, and then updated on all 5909 patients.

MAIN OUTCOME MEASURES

Histological classification and surgical staging of the mass.

RESULTS

The Assessment of Different NEoplasias in the adneXa (ADNEX) model contains three clinical and six ultrasound predictors: age, serum CA-125 level, type of centre (oncology centres v other hospitals), maximum diameter of lesion, proportion of solid tissue, more than 10 cyst locules, number of papillary projections, acoustic shadows, and ascites. The area under the receiver operating characteristic curve (AUC) for the classic discrimination between benign and malignant tumours was 0.94 (0.93 to 0.95) on temporal validation. The AUC was 0.85 for benign versus borderline, 0.92 for benign versus stage I cancer, 0.99 for benign versus stage II-IV cancer, and 0.95 for benign versus secondary metastatic. AUCs between malignant subtypes varied between 0.71 and 0.95, with an AUC of 0.75 for borderline versus stage I cancer and 0.82 for stage II-IV versus secondary metastatic. Calibration curves showed that the estimated risks were accurate.

CONCLUSIONS

The ADNEX model discriminates well between benign and malignant tumours and offers fair to excellent discrimination between four types of ovarian malignancy. The use of ADNEX has the potential to improve triage and management decisions and so reduce morbidity and mortality associated with adnexal pathology.

Extending the c-statistic to nominal polytomous outcomes: the Polytomous Discrimination Index

Diagnostic problems in medicine are sometimes polytomous, meaning that the outcome has more than two distinct categories. For example, ovarian tumors can be benign, borderline, primary invasive, or metastatic. Extending the main measure of binary discrimination, the c-statistic or area under the ROC curve, to nominal polytomous settings is not straightforward. This paper reviews existing measures and presents the polytomous discrimination index (PDI) as an alternative. The PDI assesses all sets of k cases consisting of one case from each outcome category. For each category i (i = 1, … ,k), it is assessed whether the risk of category i is highest for the case from category i. A score of 1∕k is given per category for which this holds, yielding a set score between 0 and 1 to indicate the level of discrimination. The PDI is the average set score and is interpreted as the probability to correctly identify a case from a randomly selected category within a set of k cases. This probability can be split up by outcome category, yielding k category-specific values that result in the PDI when averaged. We demonstrate the measures on two diagnostic problems (residual mass histology after chemotherapy for testicular cancer; diagnosis of ovarian tumors). We compare the behavior of the measures on theoretical data, showing that PDI is more strongly influenced by simultaneous discrimination between all categories than by partial discrimination between pairs of categories. In conclusion, the PDI is attractive because it better matches the requirements of a measure to summarize polytomous discrimination.

Discrimination ability of prediction models for ordinal outcomes: relationships between existing measures and a new measure

In this paper, we focus on measures to evaluate discrimination of prediction models for ordinal outcomes. We review existing extensions of the dichotomous c-index-which is equivalent to the area under the receiver operating characteristic (ROC) curve--suggest a new measure, and study their relationships. The volume under the ROC surface (VUS) scores sets of cases including one case from each outcome category. VUS considers sets as either correctly or incorrectly ordered by the model. All other existing measures assess pairs of cases. We propose an ordinal c-index (ORC) that is set-based but, contrary to VUS, scores sets more gradually by indicating the closeness of the model-based ordering to the perfect ordering. As a result, the ORC does not decrease rapidly as the number of outcome categories increases. It turns out that the ORC can be rewritten as the average of pairwise c-indexes. Hence, the ORC has both a set- and pair-based interpretation. There are several relationships between the existing measures, leading to only two types of existing measures: a prevalence-weighted average of pairwise c-indexes and the VUS. Our suggested measure ORC positions itself in between as it is set-based but turns out to equal an unweighted average of pairwise c-indexes. The measures are demonstrated through a case study on the prediction of six-month outcome after traumatic brain injury. In conclusion, the set-based nature and graded scoring system make the ORC an attractive measure with a simple interpretation, together with its prevalence-independence that is a natural property of a discrimination measure.

Support vector methods for survival analysis: a comparison between ranking and regression approaches

OBJECTIVE

To compare and evaluate ranking, regression and combined machine learning approaches for the analysis of survival data.

METHODS

The literature describes two approaches based on support vector machines to deal with censored observations. In the first approach the key idea is to rephrase the task as a ranking problem via the concordance index, a problem which can be solved efficiently in a context of structural risk minimization and convex optimization techniques. In a second approach, one uses a regression approach, dealing with censoring by means of inequality constraints. The goal of this paper is then twofold: (i) introducing a new model combining the ranking and regression strategy, which retains the link with existing survival models such as the proportional hazards model via transformation models; and (ii) comparison of the three techniques on 6 clinical and 3 high-dimensional datasets and discussing the relevance of these techniques over classical approaches fur survival data.

RESULTS

We compare svm-based survival models based on ranking constraints, based on regression constraints and models based on both ranking and regression constraints. The performance of the models is compared by means of three different measures: (i) the concordance index, measuring the model's discriminating ability; (ii) the logrank test statistic, indicating whether patients with a prognostic index lower than the median prognostic index have a significant different survival than patients with a prognostic index higher than the median; and (iii) the hazard ratio after normalization to restrict the prognostic index between 0 and 1. Our results indicate a significantly better performance for models including regression constraints above models only based on ranking constraints.

CONCLUSIONS

This work gives empirical evidence that svm-based models using regression constraints perform significantly better than svm-based models based on ranking constraints. Our experiments show a comparable performance for methods including only regression or both regression and ranking constraints on clinical data. On high dimensional data, the former model performs better. However, this approach does not have a theoretical link with standard statistical models for survival data. This link can be made by means of transformation models when ranking constraints are included.

Qualitative Assessment of the Progesterone Receptor and HER2 Improves the Nottingham Prognostic Index Up to 5 Years After Breast Cancer Diagnosis

Purpose

To investigate whether the estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2) can improve the Nottingham Prognostic Index (NPI) in the classification of patients with primary operable breast cancer for disease-free survival (DFS).

Patients and Methods

The analysis is based on 1,927 patients with breast cancer treated between 2000 and 2005 at the University Hospitals, Leuven. We compared performances of NPI with and without ER, PR and/or HER2. Validation was done on two external data sets containing 862 and 2,805 patients from Oslo (Norway) and Auckland (New Zealand), respectively.

Results

In the Leuven cohort, median follow-up was 66 months, and 13.7% of patients experienced a breast cancer–related event. Positive staining for ER, PR, and HER2 was detected, respectively, in 86.9%, 75.5%, and 11.9% of patients. Based on multivariate Cox regression modeling, the improved NPI (iNPI) was derived as NPI − PR positivity + HER2 positivity. Validation results showed a risk group reclassification of 20% to 30% of patients when using iNPI with its optimal risk boundaries versus NPI, in a majority of patients to more appropriate risk groups. An additional 10% of patients were classified into the extreme risk groups, where clinical actions are less ambiguous. Survival curves of reclassified patients resembled more closely those for patients in the same iNPI group than those for patients in the same NPI group.

Conclusion

The addition of PR and HER2 to NPI increases its 5-year prognostic accuracy. The iNPI can be considered as a clinically useful tool for stratification of patients with breast cancer receiving standard of care.

Plasma MMP1 and MMP8 expression in breast cancer: protective role of MMP8 against lymph node metastasis

BACKGROUND

Elevated levels of matrix metalloproteinases have been found to associate with poor prognosis in various carcinomas. This study aimed at evaluating plasma levels of MMP1, MMP8 and MMP13 as diagnostic and prognostic markers of breast cancer.

METHODS

A total of 208 breast cancer patients, of which 21 with inflammatory breast cancer, and 42 healthy controls were included. Plasma MMP1, MMP8 and MMP13 levels were measured using ELISA and correlated with clinicopathological characteristics.

RESULTS

Median plasma MMP1 levels were higher in controls than in breast cancer patients (3.45 vs. 2.01 ng/ml), while no difference was found for MMP8 (10.74 vs. 10.49 ng/ml). ROC analysis for MMP1 revealed an AUC of 0.67, sensitivity of 80% and specificity of 24% at a cut-off value of 4.24 ng/ml. Plasma MMP13 expression could not be detected. No correlation was found between MMP1 and MMP8 levels. We found a trend of lower MMP1 levels with increasing tumour size (p = 0.07); and higher MMP8 levels with premenopausal status (p = 0.06) and NPI (p = 0.04). The median plasma MMP1 (p = 0.02) and MMP8 (p = 0.007) levels in the non-inflammatory breast cancer patients were almost twice as high as those found in the inflammatory breast cancer patients. Intriguingly, plasma MMP8 levels were positively associated with lymph node involvement but showed a negative correlation with the risk of distant metastasis. Both controls and lymph node negative patients (pN0) had lower MMP8 levels than patients with moderate lymph node involvement (pN1, pN2) (p = 0.001); and showed a trend for higher MMP8 levels compared to patients with extensive lymph node involvement (pN3) and a strong predisposition to distant metastasis (p = 0.11). Based on the hypothesis that blood and tissue protein levels are in reverse association, these results suggest that MMP8 in the tumour may have a protective effect against lymph node metastasis.

CONCLUSION

In summary, we observed differences in MMP1 and MMP8 plasma levels between healthy controls and breast cancer patients as well as between breast cancer patients. Interestingly, our results suggest that MMP8 may affect the metastatic behaviour of breast cancer cells through protection against lymph node metastasis, underlining the importance of anti-target identification in drug development.