Discrimination ability of prediction models for ordinal outcomes: relationships between existing measures and a new measure

Prediction model for recommending coronary artery calcium score screening (CAC-prob) in cardiology outpatient units: A development study

Despite the well-established significance of the CAC score as a cardiovascular risk marker, the timing of using CAC score in routine clinical practice remains unclear. We aim to develop a prediction model for patients visiting outpatient cardiology units, which can recommend whether CAC score screening is necessary. A prediction model using retrospective cross-sectional design was conducted. Patients who underwent CAC score screening were included. Eight candidate predictors were preselected, including age, gender, DM or primary hypertension, angina chest pain, LDL-C (≥130 mg/dl), presence of low HDL-C, triglyceride (≥150 mg/dl), and eGFR. The outcome of interest was the level of CAC score (CAC score 0, CAC score 1-99, CAC score ≥100). The model was developed using ordinal logistic regression, and model performance was evaluated in terms of discriminative ability and calibration. A total of 360 patients were recruited for analysis, comprising 136 with CAC score 0, 133 with CAC score 1-99, and 111 with CAC score ≥100. The final predictors identified were age, male gender, presence of hypertension or DM, and low HDL-C. The model demonstrated excellent discriminative ability (Ordinal C-statistics of 0.81) with visually good agreement on calibration plots. The implementation of this model (CAC-prob) has the potential to enhance precision in recommending CAC screening. However, external validation is necessary to assess its robustness in new patient cohorts.

Using 30-day modified rankin scale score to predict 90-day score in patients with intracranial hemorrhage: Derivation and validation of prediction model

Whether 30-day modified Rankin Scale (mRS) scores can predict 90-day scores is unclear. This study derived and validated a model to predict ordinal 90-day mRS score in an intracerebral hemorrhage (ICH) population using 30-day mRS values and routinely available baseline variables. Adults enrolled in the Antihypertensive Treatment of Acute Cerebral Hemorrhage-2 (ATACH-2) trial between May 2011 and September 2015 with acute ICH, who were alive at 30 days and had mRS scores reported at both 30 and 90 days were included in this post-hoc analysis. A proportional odds regression model for predicting ordinal 90-day mRS scores was developed and internally validated using bootstrapping. Variables in the model included: mRS score at 30 days, age (years), hematoma volume (cm3), hematoma location (deep [basal ganglia, thalamus], lobar, or infratentorial), presence of intraventricular hemorrhage (IVH), baseline Glasgow Coma Scale (GCS) score, and National Institutes of Health Stroke Scale (NIHSS) score at randomization. We assessed model fit, calibration, discrimination, and agreement (ordinal, dichotomized functional independence), and EuroQol-5D ([EQ-5D] utility weighted) between predicted and observed 90-day mRS. A total of 898/1000 participants were included. Following bootstrap internal validation, our model (calibration slope = 0.967) had an optimism-corrected c-index of 0.884 (95% CI = 0.873-0.896) and R2 = 0.712 for 90-day mRS score. The weighted ĸ for agreement between observed and predicted ordinal 90-day mRS score was 0.811 (95% CI = 0.787-0.834). Agreement between observed and predicted functional independence (mRS score of 0-2) at 90 days was 74.3% (95% CI = 69.9-78.7%). The mean ± SD absolute difference between predicted and observed EQ-5D-weighted mRS score was negligible (0.005 ± 0.145). This tool allows practitioners and researchers to utilize clinically available information along with the mRS score 30 days after ICH to reliably predict the mRS score at 90 days.

Prediction of long-term functional outcome following different rehabilitation pathways after stroke unit discharge

OBJECTIVE

To investigate whether referral for different types of rehabilitation on discharge from Swedish stroke units can predict functional outcomes at 1 and 5 years after a stroke.

DESIGN

A longitudinal and registry-based study.

SUBJECTS/PATIENTS

A total of 5,118 participants with index stroke in 2011 were followed-up at 1 and 5 years after the stroke.

METHODS

Ordinal logistic regression models were developed to predict the category of functional outcome: independent, dependent, or dead. The primary predictors were planned rehabilitation in a home setting, inpatient rehabilitation, and outpatient rehabilitation, with no planned rehabilitation as the reference category.

RESULTS

Planned outpatient rehabilitation predicted independence (compared with death) at 1 year. Planned rehabilitation in the home setting predicted independence (compared with death) at 1 and 5 years. Compared with other planned pathways, participants planned for inpatient rehabilitation had more severe conditions, and planned inpatient rehabilitation did not predict independence.

CONCLUSION

Planning for outpatient or home-based rehabilitation appeared to lead more effectively to participants achieving independence over the course of 1-5 years. This may have been due to the less severe nature of these participants' conditions, compared with those requiring inpatient rehabilitation.

Data-driven distillation and precision prognosis in traumatic brain injury with interpretable machine learning

Traumatic brain injury (TBI) affects how the brain functions in the short and long term. Resulting patient outcomes across physical, cognitive, and psychological domains are complex and often difficult to predict. Major challenges to developing personalized treatment for TBI include distilling large quantities of complex data and increasing the precision with which patient outcome prediction (prognoses) can be rendered. We developed and applied interpretable machine learning methods to TBI patient data. We show that complex data describing TBI patients' intake characteristics and outcome phenotypes can be distilled to smaller sets of clinically interpretable latent factors. We demonstrate that 19 clusters of TBI outcomes can be predicted from intake data, a ~ 6× improvement in precision over clinical standards. Finally, we show that 36% of the outcome variance across patients can be predicted. These results demonstrate the importance of interpretable machine learning applied to deeply characterized patients for data-driven distillation and precision prognosis.

A comparison of the NIRUDAK models and WHO algorithm for dehydration assessment in older children and adults with acute diarrhoea: a prospective, observational study

BACKGROUND

Despite the importance of accurate and rapid assessment of hydration status in patients with acute diarrhoea, no validated tools exist to help clinicians assess dehydration severity in older children and adults. The aim of this study is to validate a clinical decision support tool (CDST) and a simplified score for dehydration severity in older children and adults with acute diarrhoea (both developed during the NIRUDAK study) and compare their accuracy and reliability with current WHO guidelines.

METHODS

A random sample of patients aged 5 years or older presenting with diarrhoea to the icddr,b Dhaka Hospital in Bangladesh between Jan 30 and Dec 13, 2022 were included in this prospective cohort study. Patients with fewer than three loose stools per day, more than 7 days of symptoms, previous enrolment in the study, or a diagnosis other than acute gastroenteritis were excluded. Patients were weighed on arrival and assessed separately by two nurses using both our novel clinical tools and WHO guidelines. Patients were weighed every 4 h to determine their percent weight change with rehydration, our criterion standard for dehydration. Accuracy for the diagnosis of dehydration category (none, some, or severe) was assessed using the ordinal c-index (ORC). Reliability was assessed by comparing the prediction of severe dehydration from each nurse's independent assessment using the intraclass correlation coefficient (ICC).

FINDINGS

1580 patients were included in our primary analysis, of whom 921 (58·3%) were female and 659 (41·7%) male. The ORC was 0·74 (95% CI 0·71-0·77) for the CDST, 0·75 (0·71-0·78) for the simplified score, and 0·64 (0·61-0·67) for the WHO guidelines. The ICC was 0·98 (95% CI 0·97-0·98) for the CDST, 0·94 (0·93-0·95) for the simplified score, and 0·56 (0·52-0·60) for the WHO guidelines.

INTERPRETATION

Use of our CDST or simplified score by clinicians could reduce undertreatment and overtreatment of older children and adults with acute diarrhoea, potentially reducing morbidity and mortality for this common disease.

FUNDING

US National Institutes of Health.

TRANSLATION

For the Bangla translation of the abstract see Supplementary Materials section.

Mining the contribution of intensive care clinical course to outcome after traumatic brain injury

Existing methods to characterise the evolving condition of traumatic brain injury (TBI) patients in the intensive care unit (ICU) do not capture the context necessary for individualising treatment. Here, we integrate all heterogenous data stored in medical records (1166 pre-ICU and ICU variables) to model the individualised contribution of clinical course to 6-month functional outcome on the Glasgow Outcome Scale -Extended (GOSE). On a prospective cohort (n = 1550, 65 centres) of TBI patients, we train recurrent neural network models to map a token-embedded time series representation of all variables (including missing values) to an ordinal GOSE prognosis every 2 h. The full range of variables explains up to 52% (95% CI: 50-54%) of the ordinal variance in functional outcome. Up to 91% (95% CI: 90-91%) of this explanation is derived from pre-ICU and admission information (i.e., static variables). Information collected in the ICU (i.e., dynamic variables) increases explanation (by up to 5% [95% CI: 4-6%]), though not enough to counter poorer overall performance in longer-stay (>5.75 days) patients. Highest-contributing variables include physician-based prognoses, CT features, and markers of neurological function. Whilst static information currently accounts for the majority of functional outcome explanation after TBI, data-driven analysis highlights investigative avenues to improve the dynamic characterisation of longer-stay patients. Moreover, our modelling strategy proves useful for converting large patient records into interpretable time series with missing data integration and minimal processing.

Development and External Validation of Partial Proportional Odds Risk Prediction Models for Cancer Stage at Diagnosis among Males and Females in Canada

Risk prediction models for cancer stage at diagnosis may identify individuals at higher risk of late-stage cancer diagnoses. Partial proportional odds risk prediction models for cancer stage at diagnosis for males and females were developed using data from Alberta's Tomorrow Project (ATP). Prediction models were validated on the British Columbia Generations Project (BCGP) cohort using discrimination and calibration measures. Among ATP males, older age at diagnosis was associated with an earlier stage at diagnosis, while full- or part-time employment, prostate-specific antigen testing, and former/current smoking were associated with a later stage at diagnosis. Among ATP females, mammogram and sigmoidoscopy or colonoscopy were associated with an earlier stage at diagnosis, while older age at diagnosis, number of pregnancies, and hysterectomy were associated with a later stage at diagnosis. On external validation, discrimination results were poor for both males and females while calibration results indicated that the models did not over- or under-fit to derivation data or over- or under-predict risk. Multiple factors associated with cancer stage at diagnosis were identified among ATP participants. While the prediction model calibration was acceptable, discrimination was poor when applied to BCGP data. Updating our models with additional predictors may help improve predictive performance.

AutoScore-Ordinal: an interpretable machine learning framework for generating scoring models for ordinal outcomes

Background

Risk prediction models are useful tools in clinical decision-making which help with risk stratification and resource allocations and may lead to a better health care for patients. AutoScore is a machine learning–based automatic clinical score generator for binary outcomes. This study aims to expand the AutoScore framework to provide a tool for interpretable risk prediction for ordinal outcomes.

Methods

The AutoScore-Ordinal framework is generated using the same 6 modules of the original AutoScore algorithm including variable ranking, variable transformation, score derivation (from proportional odds models), model selection, score fine-tuning, and model evaluation. To illustrate the AutoScore-Ordinal performance, the method was conducted on electronic health records data from the emergency department at Singapore General Hospital over 2008 to 2017. The model was trained on 70% of the data, validated on 10% and tested on the remaining 20%.

Results

This study included 445,989 inpatient cases, where the distribution of the ordinal outcome was 80.7% alive without 30-day readmission, 12.5% alive with 30-day readmission, and 6.8% died inpatient or by day 30 post discharge. Two point-based risk prediction models were developed using two sets of 8 predictor variables identified by the flexible variable selection procedure. The two models indicated reasonably good performance measured by mean area under the receiver operating characteristic curve (0.758 and 0.793) and generalized c-index (0.737 and 0.760), which were comparable to alternative models.

Conclusion

AutoScore-Ordinal provides an automated and easy-to-use framework for development and validation of risk prediction models for ordinal outcomes, which can systematically identify potential predictors from high-dimensional data.

The leap to ordinal: Detailed functional prognosis after traumatic brain injury with a flexible modelling approach

When a patient is admitted to the intensive care unit (ICU) after a traumatic brain injury (TBI), an early prognosis is essential for baseline risk adjustment and shared decision making. TBI outcomes are commonly categorised by the Glasgow Outcome Scale–Extended (GOSE) into eight, ordered levels of functional recovery at 6 months after injury. Existing ICU prognostic models predict binary outcomes at a certain threshold of GOSE (e.g., prediction of survival [GOSE > 1]). We aimed to develop ordinal prediction models that concurrently predict probabilities of each GOSE score. From a prospective cohort (n = 1,550, 65 centres) in the ICU stratum of the Collaborative European NeuroTrauma Effectiveness Research in TBI (CENTER-TBI) patient dataset, we extracted all clinical information within 24 hours of ICU admission (1,151 predictors) and 6-month GOSE scores. We analysed the effect of two design elements on ordinal model performance: (1) the baseline predictor set, ranging from a concise set of ten validated predictors to a token-embedded representation of all possible predictors, and (2) the modelling strategy, from ordinal logistic regression to multinomial deep learning. With repeated k-fold cross-validation, we found that expanding the baseline predictor set significantly improved ordinal prediction performance while increasing analytical complexity did not. Half of these gains could be achieved with the addition of eight high-impact predictors to the concise set. At best, ordinal models achieved 0.76 (95% CI: 0.74–0.77) ordinal discrimination ability (ordinal c-index) and 57% (95% CI: 54%– 60%) explanation of ordinal variation in 6-month GOSE (Somers’ D_xy). Model performance and the effect of expanding the predictor set decreased at higher GOSE thresholds, indicating the difficulty of predicting better functional outcomes shortly after ICU admission. Our results motivate the search for informative predictors that improve confidence in prognosis of higher GOSE and the development of ordinal dynamic prediction models.

Classical Regression and Predictive Modeling

BACKGROUND

With the advent of personalized and stratified medicine, there has been much discussion about predictive modeling and the role of classical regression in modern medical research. We describe and distinguish the goals in these 2 frameworks for analysis.

METHODS

The assumptions underlying and utility of classical regression are reviewed for continuous and binary outcomes. The tenets of predictive modeling are then discussed and contrasted. Principles are illustrated by simulation and through application of methods to a neurosurgical study.

RESULTS

Classical regression can be used for insights into causal mechanisms if careful thought is given to the role of variables of interest and potential confounders. In predictive modeling, interest lies more in accuracy of predictions and so alternative metrics are used to judge adequacy of models and methods; methods which average predictions over several contending models can improve predictive performance but these do not admit a single risk score.

CONCLUSIONS

Both classical regression and predictive modeling have important roles in modern medical research. Understanding the distinction between the 2 frameworks for analysis is important to place them in their appropriate context and interpreting findings from published studies appropriately.

Development of a prediction model in female pure or predominant urge urinary incontinence: a retrospective cohort study

Background:

Urinary incontinence is a prevalent form of pelvic floor dysfunction, with a non-negligible impact on a patient’s quality of life. There are several treatment options, varying from conservative to invasive. The aim of this study is to predict treatment outcomes of pure or predominant urge urinary incontinence (UUI) in women to support shared decision-making and manage patient expectations.

Methods:

Data on patient characteristics, disease history, and investigations of 512 consecutive women treated for UUI in three hospitals in the Netherlands were retrospectively collected. The predicted outcome was the short-term subjective continence outcome, defined as patient-reported continence 3 months after treatment categorized as cure (no urinary leakage), improvement (any degree of improvement of urinary leakage), and failure (no improvement or worsening of urinary leakage). Multivariable ordinal regression with backward stepwise selection was performed to analyze association between outcome and patient’s characteristics. Interactions between patient characteristics and treatment were added to estimate individual treatment benefit. Discriminative ability was assessed with the ordinal c-statistic.

Results:

Conservative treatment was applied in 12% of the patients, pharmacological in 62%, and invasive in 26%. Subjective continence outcome was cure, improvement, and failure in 20%, 49%, and 31%, respectively. Number of incontinence episodes per day, voiding frequency during the day, subjective quantity of UI, coexistence of stress urinary incontinence (SUI), night incontinence, and bladder capacity and the interactions between these variables were included in the model. After internal validation, the ordinal c-statistic was 0.699.

Conclusions:

Six variables were of value to predict pure or predominant UUI treatment outcome in women. Further development into a comprehensive set of models for the use in various pelvic floor disorders and treatments is recommended to optimize individualized care. This model requires external validation before implementation in clinical practice.

A nonparametric approach to confidence intervals for concordance index and difference between correlated indices

Concordance refers to the probability that subjects with high values on one variable also have high values on another variable. This index has wide application in practice, as a measure of effect size in group-comparison studies, an index of accuracy in diagnostic studies, and a discrimination index for prediction models. Herein, we provide a unified framework for statistical inference involving concordance indices for standard variables of binary, ordinal, and continuous types. In particular, we develop confidence interval procedures for a single concordance index and differences between two correlated indices. Simulation results show that procedures based on logit-transformation for a single index and Fisher's z-transformation for a difference between indices perform very well in terms of coverage and tail errors even when the sample size is as small as 30, unless the concordance is high and the standard is a binary variable for which at least 50 subjects are needed. We illustrate the procedures for a variety of standard variables with previously published data. Illustrative SAS code is provided.

Risk prediction models for discrete ordinal outcomes: Calibration and the impact of the proportional odds assumption

Calibration is a vital aspect of the performance of risk prediction models, but research in the context of ordinal outcomes is scarce. This study compared calibration measures for risk models predicting a discrete ordinal outcome, and investigated the impact of the proportional odds assumption on calibration and overfitting. We studied the multinomial, cumulative, adjacent category, continuation ratio, and stereotype logit/logistic models. To assess calibration, we investigated calibration intercepts and slopes, calibration plots, and the estimated calibration index. Using large sample simulations, we studied the performance of models for risk estimation under various conditions, assuming that the true model has either a multinomial logistic form or a cumulative logit proportional odds form. Small sample simulations were used to compare the tendency for overfitting between models. As a case study, we developed models to diagnose the degree of coronary artery disease (five categories) in symptomatic patients. When the true model was multinomial logistic, proportional odds models often yielded poor risk estimates, with calibration slopes deviating considerably from unity even on large model development datasets. The stereotype logistic model improved the calibration slope, but still provided biased risk estimates for individual patients. When the true model had a cumulative logit proportional odds form, multinomial logistic regression provided biased risk estimates, although these biases were modest. Nonproportional odds models require more parameters to be estimated from the data, and hence suffered more from overfitting. Despite larger sample size requirements, we generally recommend multinomial logistic regression for risk prediction modeling of discrete ordinal outcomes.

Ordinal logistic regression model describing factors associated with extent of nodal involvement in oral cancer patients and its prospective validation

Background

Oral cancer is the most common cancer among Indian men, and has strong tendency of metastatic spread to neck lymph node which strongly influences prognosis especially 5 year survival-rate and also guides the related managements more effectively. Therefore, a reliable and accurate means of preoperative evaluation of extent of nodal involvement becomes crucial. However, earlier researchers have preferred to address mainly its dichotomous form (involved/not-involved) instead of ordinal form while dealing with epidemiology of nodal involvement. As a matter of fact, consideration of ordinal form appropriately may increase not only the efficiency of the developed model but also accuracy in the results and related implications. Hence, to develop a model describing factors associated with ordinal form of nodal involvement was major focus of this study.

Methods

The data for model building were taken from the Department of Surgical Oncology, Dr.BRA-IRCH, AIIMS, New Delhi, India. All the OSCC patients (duly operated including neck dissection) and confirmed histopathologically from 1995 to 2013 were included. Further, another data of 204 patients collected prospectively from 2014 to 2015 was considered for the validation of the developed model. To assess the factors associated with extent of nodal involvement, as a first attempt in the field of OSCC, stepwise multivariable regression procedure was used and results are presented as odds-ratio and corresponding 95% confidence interval (CI). For appropriate accounting of ordinal form, the ordinal models were assessed and compared. Also, performance of the developed model was validated on a prospectively collected another data.

Results

Under multivariable proportional odds model, pain at the time of presentation, sub mucous fibrosis, palpable neck node, oral site and degree of differentiation were found to be significantly associated factors with extent of nodal involvement. In addition, tumor size also emerged to be significant under partial-proportional odds model.

Conclusions

The analytical results under the present study reveal that in case of ordinal form of the outcome, appropriate ordinal regression may be a preferred choice. Present data suggest that, pain, sub mucous fibrosis, palpable neck node, oral site, degree of differentiation and tumor size are the most probable associated factors with extent of nodal involvement.

Differentiation of regenerative nodule, dysplastic nodule, and small hepatocellular carcinoma in cirrhotic patients: a contrast-enhanced ultrasound-based multivariable model analysis

OBJECTIVE

To develop a contrast-enhanced ultrasound (CEUS)-based model for differentiating cirrhotic liver lesions and for active surveillance of hepatocellular carcinoma (HCC).

METHODS

Patients with focal liver lesions (FLLs) with biopsy/resection-proven pathology and pre-procedure CEUS were enrolled from our institution between January 2011 and November 2014. Univariable and multivariable regression models were constructed using qualitative CEUS features and/or contrast arrival time ratio (CAT_R). The optimism-adjusted Harrell's generalized concordance index (C_H) was used to quantify the discriminatory ability of each CEUS feature and model.

RESULTS

A total of 149 patients (113 men and 36 women) with 162 FLLs were enrolled with mean age 53.4 ± 12.7 years. A 0.1-unit reduction in CAT_R was associated with a 68% increase in the odds of having a higher nodule ranking (RN < DN < small HCC) (OR, 0.32; 95% CI, 0.20-0.50, p < .001). Arterial phase hypoenhancement and isoenhancement were inversely associated with a higher nodule ranking compared to hyperenhancement. Late-phase isoenhancement was associated with lower odds of a higher nodule ranking. The CEUS + CAT_R model (C_H 0.92, 0.89-0.95) provided greater discriminatory ability when compared to the CAT_R model (ΔC_H 0.09, 0.04-0.13, p < .001) and the CEUS model (ΔC_H 0.03, 0.01-0.05, p = .02).

CONCLUSIONS

Our results provide preliminary evidence that multivariable regression model constructed using both qualitative CEUS features and CAT_R provides the greatest discriminatory ability to differentiate RN, DN, and small HCC in patients with cirrhosis, and might allow for active surveillance of the progression of cirrhotic liver lesions.

KEY POINTS

• Proportional odds logistic regression models based on qualitative CEUS features and/or CAT_R can be used for differentiating cirrhotic liver lesions and for active surveillance of HCC. • The reduction of CAT_R (RN < DN < small HCC) was strongly associated with high-risk cirrhotic liver nodules. • Inclusion of CAT_R in the CEUS prediction model significantly improved its performance for cirrhotic liver lesions risk-stratification.

Fundamentals of Clinical Prediction Modeling for the Neurosurgeon

Clinical prediction models in neurosurgery are increasingly reported. These models aim to provide an evidence-based approach to the estimation of the probability of a neurosurgical outcome by combining 2 or more prognostic variables. Model development and model reporting are often suboptimal. A basic understanding of the methodology of clinical prediction modeling is needed when interpreting these models. We address basic statistical background, 7 modeling steps, and requirements of these models such that they may fulfill their potential for major impact for our daily clinical practice and for future scientific work.

Can clinical prediction models assess antibiotic need in childhood pneumonia? A validation study in paediatric emergency care

Objectives

Pneumonia is the most common bacterial infection in children at the emergency department (ED). Clinical prediction models for childhood pneumonia have been developed (using chest x-ray as their reference standard), but without implementation in clinical practice. Given current insights in the diagnostic limitations of chest x-ray, this study aims to validate these prediction models for a clinical diagnosis of pneumonia, and to explore their potential to guide decisions on antibiotic treatment at the ED.

Methods

We systematically identified clinical prediction models for childhood pneumonia and assessed their quality. We evaluated the validity of these models in two populations, using a clinical reference standard (1. definite/probable bacterial, 2. bacterial syndrome, 3. unknown bacterial/viral, 4. viral syndrome, 5. definite/probable viral), measuring performance by the ordinal c-statistic (ORC). Validation populations included prospectively collected data of children aged 1 month to 5 years attending the ED of Rotterdam (2012–2013) or Coventry (2005–2006) with fever and cough or dyspnoea.

Results

We identified eight prediction models and could evaluate the validity of seven, with original good performance. In the Dutch population 22/248 (9%) had a bacterial infection, in Coventry 53/301 (17%), antibiotic prescription was 21% and 35% respectively. Three models predicted a higher risk in children with bacterial infections than in those with viral disease (ORC ≥0.55) and could identify children at low risk of bacterial infection.

Conclusions

Three clinical prediction models for childhood pneumonia could discriminate fairly well between a clinical reference standard of bacterial versus viral infection. However, they all require the measurement of biomarkers, raising questions on the exact target population when implementing these models in clinical practice. Moreover, choosing optimal thresholds to guide antibiotic prescription is challenging and requires careful consideration of potential harms and benefits.

Integrating biomarkers across omic platforms: an approach to improve stratification of patients with indolent and aggressive prostate cancer

Classifying indolent prostate cancer represents a significant clinical challenge. We investigated whether integrating data from different omic platforms could identify a biomarker panel with improved performance compared to individual platforms alone. DNA methylation, transcripts, protein and glycosylation biomarkers were assessed in a single cohort of patients treated by radical prostatectomy. Novel multiblock statistical data integration approaches were used to deal with missing data and modelled via stepwise multinomial logistic regression, or LASSO. After applying leave‐one‐out cross‐validation to each model, the probabilistic predictions of disease type for each individual panel were aggregated to improve prediction accuracy using all available information for a given patient. Through assessment of three performance parameters of area under the curve (AUC) values, calibration and decision curve analysis, the study identified an integrated biomarker panel which predicts disease type with a high level of accuracy, with Multi AUC value of 0.91 (0.89, 0.94) and Ordinal C‐Index (ORC) value of 0.94 (0.91, 0.96), which was significantly improved compared to the values for the clinical panel alone of 0.67 (0.62, 0.72) Multi AUC and 0.72 (0.67, 0.78) ORC. Biomarker integration across different omic platforms significantly improves prediction accuracy. We provide a novel multiplatform approach for the analysis, determination and performance assessment of novel panels which can be applied to other diseases. With further refinement and validation, this panel could form a tool to help inform appropriate treatment strategies impacting on patient outcome in early stage prostate cancer.

Ultrasound characteristics of endometrial cancer as defined by International Endometrial Tumor Analysis (IETA) consensus nomenclature: prospective multicenter study

OBJECTIVE

To describe the sonographic features of endometrial cancer in relation to tumor stage, grade and histological type, using the International Endometrial Tumor Analysis (IETA) terminology.

METHODS

This was a prospective multicenter study of 1714 women with biopsy-confirmed endometrial cancer undergoing standardized transvaginal grayscale and Doppler ultrasound examination according to the IETA study protocol, by experienced ultrasound examiners using high-end ultrasound equipment. Clinical and sonographic data were entered into a web-based database. We assessed how strongly sonographic characteristics, according to IETA, were associated with outcome at hysterectomy, i.e. tumor stage, grade and histological type, using univariable logistic regression and the c-statistic.

RESULTS

In total, 1538 women were included in the final analysis. Median age was 65 (range, 27-98) years, median body mass index was 28.4 (range 16-67) kg/m² , 1377 (89.5%) women were postmenopausal and 1296 (84.3%) reported abnormal vaginal bleeding. Grayscale and color Doppler features varied according to grade and stage of tumor. High-risk tumors, compared with low-risk tumors, were less likely to have regular endometrial-myometrial junction (difference of -23%; 95% CI, -27 to -18%), were larger (mean endometrial thickness; difference of +9%; 95% CI, +8 to +11%), and were more likely to have non-uniform echogenicity (difference of +7%; 95% CI, +1 to +13%), a multiple, multifocal vessel pattern (difference of +21%; 95% CI, +16 to +26%) and a moderate or high color score (difference of +22%; 95% CI, +18 to +27%).

CONCLUSION

Grayscale and color Doppler sonographic features are associated with grade and stage of tumor, and differ between high- and low-risk endometrial cancer. Copyright © 2017 ISUOG. Published by John Wiley & Sons Ltd.

Development and validation of the Dutch Stroke Score for predicting disability and functional outcome after ischemic stroke: A tool to support efficient discharge planning

Introduction

We aimed to develop and validate a prognostic score for disability at discharge and functional outcome at three months in patients with acute ischemic stroke based on clinical information available on admission.

Patients and methods

The Dutch Stroke Score (DSS) was developed in 1227 patients with ischemic stroke included in the Paracetamol (Acetaminophen) In Stroke study. Predictors for Barthel Index (BI) at discharge (‘DSS-discharge’) and modified Rankin Scale (mRS) at three months (‘DSS-3 months’) were identified in multivariable ordinal regression. The models were internally validated with bootstrapping techniques. The DSS-3 months was externally validated in the PRomoting ACute Thrombolysis in Ischemic StrokE study (1589 patients) and the Preventive Antibiotics in Stroke Study (2107 patients). Model performance was assessed in terms of discrimination, expressed by the area under the receiver operating characteristic curve (AUC), and calibration.

Results

At model development, the strongest predictors of Barthel Index at discharge were age per decade over 60 (odds ratio = 1.55, 95% confidence interval (CI) 1.41–1.68), National Institutes of Health Stroke Scale (odds ratio = 1.24 per point, 95% CI 1.22–1.26) and diabetes (odds ratio = 1.62, 95% CI 1.32–1.91). The internally validated AUC was 0.76 (95% CI 0.75–0.79). The DSS-3 months, additionally consisting of previous stroke and atrial fibrillation, performed similarly at internal (AUC 0.75, 95% CI 0.74–0.77) and external validation (AUC 0.74 in PRomoting ACute Thrombolysis in Ischemic StrokE (95% CI 0.72–0.76) and 0.69 in Preventive Antibiotics in Stroke Study (95% CI 0.69–0.72)). Observed outcome was slightly better than predicted.

Discussion: The DSS had satisfactory performance in predicting BI at discharge and mRS at three months in ischemic stroke patients.

Conclusion

If further validated, the DSS may contribute to efficient stroke unit discharge planning alongside patients' contextual factors and therapeutic needs.

Using ordinal outcomes to construct and select biomarker combinations for single-level prediction

BACKGROUND

Biomarker studies may involve an ordinal outcome, such as no, mild, or severe disease. There is often interest in predicting one particular level of the outcome due to its clinical significance.

METHODS

A simple approach to constructing biomarker combinations in this context involves dichotomizing the outcome and using a binary logistic regression model. We assessed whether more sophisticated methods offer advantages over this simple approach. It is often necessary to select among several candidate biomarker combinations. One strategy involves selecting a combination based on its ability to predict the outcome level of interest. We propose an algorithm that leverages the ordinal outcome to inform combination selection. We apply this algorithm to data from a study of acute kidney injury after cardiac surgery, where kidney injury may be absent, mild, or severe.

RESULTS

Using more sophisticated modeling approaches to construct combinations provided gains over the simple binary logistic regression approach in specific settings. In the examples considered, the proposed algorithm for combination selection tended to reduce the impact of bias due to selection and to provide combinations with improved performance.

CONCLUSIONS

Methods that utilize the ordinal nature of the outcome in the construction and/or selection of biomarker combinations have the potential to yield better combinations.

Multiple performance measures are needed to evaluate triage systems in the emergency department

OBJECTIVES

Emergency department triage systems can be considered prediction rules with an ordinal outcome, where different directions of misclassification have different clinical consequences. We evaluated strategies to compare the performance of triage systems and aimed to propose a set of performance measures that should be used in future studies.

STUDY DESIGN AND SETTING

We identified performance measures based on literature review and expert knowledge. Their properties are illustrated in a case study evaluating two triage modifications in a cohort of 14,485 pediatric emergency department visits. Strengths and weaknesses of the performance measures were systematically appraised.

RESULTS

Commonly reported performance measures are measures of statistical association (34/60 studies) and diagnostic accuracy (17/60 studies). The case study illustrates that none of the performance measures fulfills all criteria for triage evaluation. Decision curves are the performance measures with the most attractive features but require dichotomization. In addition, paired diagnostic accuracy measures can be recommended for dichotomized analysis, and the triage-weighted kappa and Nagelkerke's R² for ordinal analyses. Other performance measures provide limited additional information.

CONCLUSION

When comparing modifications of triage systems, decision curves and diagnostic accuracy measures should be used in a dichotomized analysis, and the triage-weighted kappa and Nagelkerke's R² in an ordinal approach.

Estimates of Disease Course in Inflammatory Bowel Disease Using Administrative Data: A Population-level Study

BACKGROUND AND AIMS

We sought develop a predictive model of disease course in inflammatory bowel disease [IBD] using health care utilization measures from administrative health data, and to apply this model to estimate disease course at a population level over time.

METHODS

Study participants were IBD patients who were prospectively followed for a 1-year period between 2009 and 2010 in a Canadian clinic setting to assess their IBD disease course [i.e. remission, mild, moderate, severe]. Clinic data were linked with population-based administrative health data. A multivariable partial proportional odds model tested health care utilization measures that discriminated disease course groups. The model was applied to project the distribution of disease course for the Manitoba IBD population for 1995-2013.

RESULTS

There were 407 participants (54.3% females, 64.4% Crohn's disease [CD]) with mean age at diagnosis of 29.8 years [SD 14.9]. Forty-one per cent of participants were clinically in remission, while 14.0% had severe IBD. Mild, moderate or severe disease was associated with three or more gastroenterologist visits (odds ratio [OR] = 3.33, 95% confidence interval [CI]: 2.03-5.54) or three or more general practitioner visits [OR = 2.97, 95% CI: 1.44-6.37] with an IBD diagnosis and ≥1 radiology test [OR = 2.22, 95% CI: 1.31-3.80]. The percentages of the Manitoba IBD population in remission rose steadily from 1995 to 2013 [43.6 to 59.9%], while the percentages of individuals with mild, moderate or severe disease declined.

CONCLUSION

This study demonstrated that health care utilization measures from administrative data can be used to predict disease course in the IBD population.

Nonparametric estimation and inference for polytomous discrimination index

Polytomous discrimination index is a novel and important diagnostic accuracy measure for multi-category classification. After reconstructing its probabilistic definition, we propose a nonparametric approach to the estimation of polytomous discrimination index based on an empirical sample of biomarker values. In this paper, we provide the finite-sample and asymptotic properties of the proposed estimators and such analytic results may facilitate the statistical inference. Simulation studies are performed to examine the performance of the nonparametric estimators. Two real data examples are analysed to illustrate our methodology.

External validation of the DHAKA score and comparison with the current IMCI algorithm for the assessment of dehydration in children with diarrhoea: a prospective cohort study

BACKGROUND

Dehydration due to diarrhoea is a leading cause of child death worldwide, yet no clinical tools for assessing dehydration have been validated in resource-limited settings. The Dehydration: Assessing Kids Accurately (DHAKA) score was derived for assessing dehydration in children with diarrhoea in a low-income country setting. In this study, we aimed to externally validate the DHAKA score in a new population of children and compare its accuracy and reliability to the current Integrated Management of Childhood Illness (IMCI) algorithm.

METHODS

DHAKA was a prospective cohort study done in children younger than 60 months presenting to the International Centre for Diarrhoeal Disease Research, Bangladesh, with acute diarrhoea (defined by WHO as three or more loose stools per day for less than 14 days). Local nurses assessed children and classified their dehydration status using both the DHAKA score and the IMCI algorithm. Serial weights were obtained and dehydration status was established by percentage weight change with rehydration. We did regression analyses to validate the DHAKA score and compared the accuracy and reliability of the DHAKA score and IMCI algorithm with receiver operator characteristic (ROC) curves and the weighted κ statistic. This study was registered with ClinicalTrials.gov, number NCT02007733.

FINDINGS

Between March 22, 2015, and May 15, 2015, 496 patients were included in our primary analyses. On the basis of our criterion standard, 242 (49%) of 496 children had no dehydration, 184 (37%) of 496 had some dehydration, and 70 (14%) of 496 had severe dehydration. In multivariable regression analyses, each 1-point increase in the DHAKA score predicted an increase of 0·6% in the percentage dehydration of the child and increased the odds of both some and severe dehydration by a factor of 1·4. Both the accuracy and reliability of the DHAKA score were significantly greater than those of the IMCI algorithm.

INTERPRETATION

The DHAKA score is the first clinical tool for assessing dehydration in children with acute diarrhoea to be externally validated in a low-income country. Further validation studies in a diverse range of settings and paediatric populations are warranted.

FUNDING

National Institutes of Health Fogarty International Center.

Clinical prediction model to identify vulnerable patients in ambulatory surgery: towards optimal medical decision-making

BACKGROUND

Ambulatory surgery patients are at risk of adverse psychological outcomes such as anxiety, aggression, fatigue, and depression. We developed and validated a clinical prediction model to identify patients who were vulnerable to these psychological outcome parameters.

METHODS

We prospectively assessed 383 mixed ambulatory surgery patients for psychological vulnerability, defined as the presence of anxiety (state/trait), aggression (state/trait), fatigue, and depression seven days after surgery. Three psychological vulnerability categories were considered-i.e., none, one, or multiple poor scores, defined as a score exceeding one standard deviation above the mean for each single outcome according to normative data. The following determinants were assessed preoperatively: sociodemographic (age, sex, level of education, employment status, marital status, having children, religion, nationality), medical (heart rate and body mass index), and psychological variables (self-esteem and self-efficacy), in addition to anxiety, aggression, fatigue, and depression. A prediction model was constructed using ordinal polytomous logistic regression analysis, and bootstrapping was applied for internal validation. The ordinal c-index (ORC) quantified the discriminative ability of the model, in addition to measures for overall model performance (Nagelkerke's R (2) ).

RESULTS

In this population, 137 (36%) patients were identified as being psychologically vulnerable after surgery for at least one of the psychological outcomes. The most parsimonious and optimal prediction model combined sociodemographic variables (level of education, having children, and nationality) with psychological variables (trait anxiety, state/trait aggression, fatigue, and depression). Model performance was promising: R (2)  = 30% and ORC = 0.76 after correction for optimism.

CONCLUSION

This study identified a substantial group of vulnerable patients in ambulatory surgery. The proposed clinical prediction model could allow healthcare professionals the opportunity to identify vulnerable patients in ambulatory surgery, although additional modification and validation are needed. (ClinicalTrials.gov number, NCT01441843).

Assessing the suitability of sets-based approaches: estimating the discriminative power of risk models for ordinal outcome treatments

BACKGROUND

In order to evaluate the discrimination performance of an ordinal model for improved disease screening, a new test was proposed where information was obtained across all samples simultaneously.

METHODS

The ordinal c-index builds upon the volume under the surface methodology without focusing on the accompanying receiver operating characteristic surfaces. However, it can be simplified to an average of pairwise c-indexes. In this paper, a set-based estimate (information was obtained across all samples simultaneously) was proposed by summing all correctly ordered groups. The asymptotic distribution of this proposed estimate was derived using U-statistics.

RESULTS

A predictive model was applied using the blood urea nitrogen/creatinine ratio to discriminate stroke in evolution in acute ischemic stroke patients, which could potentially be life-saving in emergency departments.

CONCLUSIONS

By conducting Monte Carlo simulations, it was concluded that the measure proposed herein is a better choice for clinical use because of the asymmetry of the predicted probabilities of groups.

The Prognostic Value of the Work Ability Index for Sickness Absence among Office Workers

BACKGROUND

The work ability index (WAI) is a frequently used tool in occupational health to identify workers at risk for a reduced work performance and for work-related disability. However, information about the prognostic value of the WAI to identify workers at risk for sickness absence is scarce.

OBJECTIVES

To investigate the prognostic value of the WAI for sickness absence, and whether the discriminative ability differs across demographic subgroups.

METHODS

At baseline, the WAI (score 7-49) was assessed among 1,331 office workers from a Dutch financial service company. Sickness absence was registered during 12-months follow-up and categorised as 0 days, 0<days<5, 5≤days<15, and ≥15 days in one year. Associations between WAI and sickness absence were estimated by multinomial regression analyses. Discriminative ability of the WAI was assessed by the Area Under the Curve (AUC) and Ordinal c-index (ORC). Test characteristics were determined for dichotomised outcomes. Additional analyses were performed for separate WAI dimensions, and subgroup analyses for demographic groups.

RESULTS

A lower WAI was associated with sickness absence (≥15 days vs. 0 days: per point lower WAI score OR=1.27; 95%CI 1.21-1.33). The WAI showed reasonable ability to discriminate between categories of sickness absence (ORC=0.65; 95%CI 0.63-0.68). Highest discrimination was found for comparing workers with ≥15 sick days with 0 sick days (AUC=0.77) or with 1-5 sick days (AUC=0.69). At the cut-off for poor work ability (WAI≤27) the sensitivity to identify workers at risk for ≥15 sick days was 7.5%, the specificity 99.6%, and the positive predictive value 82%. The performance was similar across demographic subgroups.

CONCLUSIONS

The WAI could be used to identify workers at high risk for prolonged sickness absence. However, due to low sensitivity many workers will be missed. Hence, additional factors are required to better identify workers at highest risk.

Grid multi-category response logistic models

Background

Multi-category response models are very important complements to binary logistic models in medical decision-making. Decomposing model construction by aggregating computation developed at different sites is necessary when data cannot be moved outside institutions due to privacy or other concerns. Such decomposition makes it possible to conduct grid computing to protect the privacy of individual observations.

Methods

This paper proposes two grid multi-category response models for ordinal and multinomial logistic regressions. Grid computation to test model assumptions is also developed for these two types of models. In addition, we present grid methods for goodness-of-fit assessment and for classification performance evaluation.

Results

Simulation results show that the grid models produce the same results as those obtained from corresponding centralized models, demonstrating that it is possible to build models using multi-center data without losing accuracy or transmitting observation-level data. Two real data sets are used to evaluate the performance of our proposed grid models.

Conclusions

The grid fitting method offers a practical solution for resolving privacy and other issues caused by pooling all data in a central site. The proposed method is applicable for various likelihood estimation problems, including other generalized linear models.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-015-0133-y) contains supplementary material, which is available to authorized users.

Comparison of statistical and clinical predictions of functional outcome after ischemic stroke

Background

To determine whether the predictions of functional outcome after ischemic stroke made at the bedside using a doctor’s clinical experience were more or less accurate than the predictions made by clinical prediction models (CPMs).

Methods and Findings

A prospective cohort study of nine hundred and thirty one ischemic stroke patients recruited consecutively at the outpatient, inpatient and emergency departments of the Western General Hospital, Edinburgh between 2002 and 2005. Doctors made informal predictions of six month functional outcome on the Oxford Handicap Scale (OHS). Patients were followed up at six months with a validated postal questionnaire. For each patient we calculated the absolute predicted risk of death or dependence (OHS≥3) using five previously described CPMs. The specificity of a doctor’s informal predictions of OHS≥3 at six months was good 0.96 (95% CI: 0.94 to 0.97) and similar to CPMs (range 0.94 to 0.96); however the sensitivity of both informal clinical predictions 0.44 (95% CI: 0.39 to 0.49) and clinical prediction models (range 0.38 to 0.45) was poor. The prediction of the level of disability after stroke was similar for informal clinical predictions (ordinal c-statistic 0.74 with 95% CI 0.72 to 0.76) and CPMs (range 0.69 to 0.75). No patient or clinician characteristic affected the accuracy of informal predictions, though predictions were more accurate in outpatients.

Conclusions

CPMs are at least as good as informal clinical predictions in discriminating between good and bad functional outcome after ischemic stroke. The place of these models in clinical practice has yet to be determined.

Key concepts and limitations of statistical methods for evaluating biomarkers of kidney disease

Interest in developing and using novel markers of kidney injury is increasing. To maintain scientific rigour in these endeavors, a comprehensive understanding of statistical methodology is required to rigorously assess the incremental value of novel biomarkers in existing clinical risk prediction models. Such knowledge is especially relevant, because no single statistical method is sufficient to evaluate a novel biomarker. In this review, we highlight the strengths and limitations of various traditional and novel statistical methods used in the literature for biomarker studies and use biomarkers of AKI as examples to show limitations of some popular statistical methods.

Assessing calibration of multinomial risk prediction models

Calibration, that is, whether observed outcomes agree with predicted risks, is important when evaluating risk prediction models. For dichotomous outcomes, several tools exist to assess different aspects of model calibration, such as calibration-in-the-large, logistic recalibration, and (non-)parametric calibration plots. We aim to extend these tools to prediction models for polytomous outcomes. We focus on models developed using multinomial logistic regression (MLR): outcome Y with k categories is predicted using k - 1 equations comparing each category i (i = 2, … ,k) with reference category 1 using a set of predictors, resulting in k - 1 linear predictors. We propose a multinomial logistic recalibration framework that involves an MLR fit where Y is predicted using the k - 1 linear predictors from the prediction model. A non-parametric alternative may use vector splines for the effects of the linear predictors. The parametric and non-parametric frameworks can be used to generate multinomial calibration plots. Further, the parametric framework can be used for the estimation and statistical testing of calibration intercepts and slopes. Two illustrative case studies are presented, one on the diagnosis of malignancy of ovarian tumors and one on residual mass diagnosis in testicular cancer patients treated with cisplatin-based chemotherapy. The risk prediction models were developed on data from 2037 and 544 patients and externally validated on 1107 and 550 patients, respectively. We conclude that calibration tools can be extended to polytomous outcomes. The polytomous calibration plots are particularly informative through the visual summary of the calibration performance.

Assessing the discriminative ability of risk models for more than two outcome categories

The discriminative ability of risk models for dichotomous outcomes is often evaluated with the concordance index (c-index). However, many medical prediction problems are polytomous, meaning that more than two outcome categories need to be predicted. Unfortunately such problems are often dichotomized in prediction research. We present a perspective on the evaluation of discriminative ability of polytomous risk models, which may instigate researchers to consider polytomous prediction models more often. First, we suggest a "discrimination plot" as a tool to visualize the model's discriminative ability. Second, we discuss the use of one overall polytomous c-index versus a set of dichotomous measures to summarize the performance of the model. Third, we address several aspects to consider when constructing a polytomous c-index. These involve the assessment of concordance in pairs versus sets of patients, weighting by outcome prevalence, the value related to models with random performance, the reduction to the dichotomous c-index for dichotomous problems, and interpretation. We illustrate these issues on case studies dealing with ovarian cancer (four outcome categories) and testicular cancer (three categories). We recommend the use of a discrimination plot together with an overall c-index such as the Polytomous Discrimination Index. If the overall c-index suggests that the model has relevant discriminative ability, pairwise c-indexes for each pair of outcome categories are informative. For pairwise c-indexes we recommend the 'conditional-risk' method which is consistent with the analytical approach of the multinomial logistic regression used to develop polytomous risk models.