Assessing the discriminative ability of risk models for more than two outcome categories

Perseverance time of informal caregivers for people with dementia: construct validity, responsiveness and predictive validity

Background

Informal care is essential for many people with dementia (PwD), but it often results in a considerable burden for the caregiver. The perseverance time instrument integrates the aspect of perceived burden with the caregiver’s capacity to cope with the burden, in contrast to most available instruments, which measure solely the burden of caregiving. The aim of this study was to extend insight into psychometric properties of the perseverance time instrument, specifically the construct validity, responsiveness, and predictive validity, within the population of informal caregivers for PwD.

Methods

Data from two studies among informal caregivers of community-dwelling PwD in the Netherlands were used. The first study included 198 caregivers from a single region in the Netherlands and lasted 1 year. The second was a cross-sectional nationwide study with 166 caregivers for PwD. Questionnaires of both studies included questions regarding demographics and informal care, perseverance time, and other informal caregiver outcomes (Caregiver Strain Index, Self-rated Burden scale, Care-related Quality of Life instrument, and visual analogue scale health scores). Construct validity and responsiveness were assessed using a hypothesis-testing approach. The predictive validity of demographic characteristics and perseverance time for living situation after 1 year (living at home, institutionalized, or deceased) was assessed with multivariable multinomial regression.

Results

All but one of the hypotheses regarding construct validity were met. Three of five hypotheses regarding responsiveness were met. Perseverance time scores at baseline were associated with living situation after 1 year (p < 0.01), unlike age, sex, and relationship with PwD. Perseverance time strongly increased predictive power for living situation after 1 year (c-index between 0.671 and 0.775) in addition to demographic characteristics.

Conclusions

This study supports previous findings regarding the construct validity of the perseverance time instrument and adds new evidence of good construct validity, responsiveness, and predictive validity. The predictive power of perseverance time scores for living situation exceeds the predictive power of other burden measures and indicates informal care as an important factor for maintaining the patient at home.

Nonparametric estimation and inference for polytomous discrimination index

Polytomous discrimination index is a novel and important diagnostic accuracy measure for multi-category classification. After reconstructing its probabilistic definition, we propose a nonparametric approach to the estimation of polytomous discrimination index based on an empirical sample of biomarker values. In this paper, we provide the finite-sample and asymptotic properties of the proposed estimators and such analytic results may facilitate the statistical inference. Simulation studies are performed to examine the performance of the nonparametric estimators. Two real data examples are analysed to illustrate our methodology.

Validation and updating of risk models based on multinomial logistic regression

BACKGROUND

Risk models often perform poorly at external validation in terms of discrimination or calibration. Updating methods are needed to improve performance of multinomial logistic regression models for risk prediction.

METHODS

We consider simple and more refined updating approaches to extend previously proposed methods for dichotomous outcomes. These include model recalibration (adjustment of intercept and/or slope), revision (re-estimation of individual model coefficients), and extension (revision with additional markers). We suggest a closed testing procedure to assist in deciding on the updating complexity. These methods are demonstrated on a case study of women with pregnancies of unknown location (PUL). A previously developed risk model predicts the probability that a PUL is a failed, intra-uterine, or ectopic pregnancy. We validated and updated this model on more recent patients from the development setting (temporal updating; n = 1422) and on patients from a different hospital (geographical updating; n = 873). Internal validation of updated models was performed through bootstrap resampling.

RESULTS

Contrary to dichotomous models, we noted that recalibration can also affect discrimination for multinomial risk models. If the number of outcome categories is higher than the number of variables, logistic recalibration is obsolete because straightforward model refitting does not require the estimation of more parameters. Although recalibration strongly improved performance in the case study, the closed testing procedure selected model revision. Further, revision of functional form of continuous predictors had a positive effect on discrimination, whereas penalized estimation of changes in model coefficients was beneficial for calibration.

CONCLUSIONS

Methods for updating of multinomial risk models are now available to improve predictions in new settings. A closed testing procedure is helpful to decide whether revision is preferred over recalibration. Because multicategory outcomes increase the number of parameters to be estimated, we recommend full model revision only when the sample size for each outcome category is large.

Managing pregnancy of unknown location based on initial serum progesterone and serial serum hCG levels: development and validation of a two-step triage protocol

OBJECTIVES

A uniform rationalized management protocol for pregnancies of unknown location (PUL) is lacking. We developed a two-step triage protocol to select PUL at high risk of ectopic pregnancy (EP), based on serum progesterone level at presentation (step 1) and the serum human chorionic gonadotropin (hCG) ratio, defined as the ratio of hCG at 48 h to hCG at presentation (step 2).

METHODS

This was a cohort study of 2753 PUL (301 EP), involving a secondary analysis of prospectively and consecutively collected PUL data from two London-based university teaching hospitals. Using a chronological split we used 1449 PUL for development and 1304 for validation. We aimed to assign PUL as low risk with high confidence (high negative predictive value (NPV)) while classifying most EP as high risk (high sensitivity). The first triage step assigned PUL as low risk using a threshold of serum progesterone at presentation. The remaining PUL were triaged using a novel logistic regression risk model based on hCG ratio and initial serum progesterone (second step), defining low risk as an estimated EP risk of < 5%.

RESULTS

On validation, initial serum progesterone ≤ 2 nmol/L (step 1) classified 16.1% PUL as low risk. Second-step classification with the risk model selected an additional 46.0% of all PUL as low risk. Overall, the two-step protocol classified 62.1% of PUL as low risk, with an NPV of 98.6% and a sensitivity of 92.0%. When the risk model was used in isolation (i.e. without the first step), 60.5% of PUL were classified as low risk with 99.1% NPV and 94.9% sensitivity.

CONCLUSION

PUL can be classified efficiently into being either high or low risk for complications using a two-step protocol involving initial progesterone and hCG levels and the hCG ratio. Copyright © 2016 ISUOG. Published by John Wiley & Sons Ltd.

Evaluating the risk of ovarian cancer before surgery using the ADNEX model: a multicentre external validation study

BACKGROUND

The International Ovarian Tumour Analysis (IOTA) group have developed the ADNEX (The Assessment of Different NEoplasias in the adneXa) model to predict the risk that an ovarian mass is benign, borderline, stage I, stages II-IV or metastatic. We aimed to externally validate the ADNEX model in the hands of examiners with varied training and experience.

METHODS

This was a multicentre cross-sectional cohort study for diagnostic accuracy. Patients were recruited from three cancer centres in Europe. Patients who underwent transvaginal ultrasonography and had a histological diagnosis of surgically removed tissue were included. The diagnostic performance of the ADNEX model with and without the use of CA125 as a predictor was calculated.

RESULTS

Data from 610 women were analysed. The overall prevalence of malignancy was 30%. The area under the receiver operator curve (AUC) for the ADNEX diagnostic performance to differentiate between benign and malignant masses was 0.937 (95% CI: 0.915-0.954) when CA125 was included, and 0.925 (95% CI: 0.902-0.943) when CA125 was excluded. The calibration plots suggest good correspondence between the total predicted risk of malignancy and the observed proportion of malignancies. The model showed good discrimination between the different subtypes.

CONCLUSIONS

The performance of the ADNEX model retains its performance on external validation in the hands of ultrasound examiners with varied training and experience.

Subjective ultrasound assessment, the ADNEX model and ultrasound-guided tru-cut biopsy to differentiate disseminated primary ovarian cancer from metastatic non-ovarian cancer

OBJECTIVES

To compare subjective ultrasound assessment and the ADNEX model with ultrasound-guided tru-cut biopsy to differentiate disseminated primary ovarian cancer from metastatic non-ovarian cancer.

METHODS

This was a prospective study including 143 consecutive women with disseminated malignancy of unknown primary origin, with a pelvic tumor/carcinosis. Women underwent either transvaginal or transrectal ultrasound as well as transabdominal ultrasound examination followed by tru-cut biopsy. The ultrasound examiner assessed tumor morphology, spread in the pelvis and abdomen, and predicted tumor origin as primary ovarian or metastatic using both subjective assessment and the ADNEX model. Histology from tru-cut biopsy served as the gold standard for assessment of diagnostic accuracy. Biopsy adequacy and the complication rate were assessed.

RESULTS

Tru-cut biopsy was performed transvaginally in 131/143 (92%) women. Two women needed inpatient care (one had abdominal wall hematoma, and one infection). Biopsy resulted in a conclusive diagnosis in 126/143 (88%) women, amongst whom cytoreductive surgery was performed in 30/126 confirming the diagnosis in all cases. Non-ovarian metastatic cancer was found in 37/126 (29%) women and primary ovarian cancer in 89/126 (71%) women. Subjective ultrasound evaluation had a sensitivity of 82% (73/89) and a specificity of 70% (26/37) in predicting primary ovarian cancer. The ADNEX model had an area under the receiver-operating characteristics curve of 0.891 (95% CI, 0.794-0.946) (in women with an ovarian lesion, n = 104). Tumor origin was associated with age, CA 125, previous neoplasia, presence of omental cake and tumor mobility.

CONCLUSIONS

Subjective ultrasound assessment and the ADNEX model can both be used to predict whether a pelvic tumor is metastatic and of non-ovarian origin, indicating the need for tru-cut biopsy, which is associated with very few complications and will provide a conclusive diagnosis in nine out of 10 women. Copyright © 2015 ISUOG.

A nomogram was developed to enhance the use of multinomial logistic regression modeling in diagnostic research

OBJECTIVES

We developed a nomogram to facilitate the interpretation and presentation of results from multinomial logistic regression models.

STUDY DESIGN AND SETTING

We analyzed data from 376 frail elderly with complaints of dyspnea. Potential underlying disease categories were heart failure (HF), chronic obstructive pulmonary disease (COPD), the combination of both (HF and COPD), and any other outcome (other). A nomogram for multinomial model was developed to depict the relative importance of each predictor and to calculate the probability for each disease category for a given patient. Additionally, model performance of the multinomial regression model was assessed.

RESULTS

Prevalence of HF and COPD was 14% (n = 54), HF 24% (n = 90), COPD 20% (n = 75), and Other 42% (n = 157). The relative importance of the individual predictors varied across these disease categories or was even reversed. The pairwise C statistics ranged from 0.75 (between HF and Other) to 0.96 (between HF and COPD and Other). The nomogram can be used to rank the disease categories from most to least likely within each patient or to calculate the predicted probabilities.

CONCLUSIONS

Our new nomogram is a useful tool to present and understand the results of a multinomial regression model and could enhance the applicability of such models in daily practice.

Impact of a clinical decision model for febrile children at risk for serious bacterial infections at the emergency department: a randomized controlled trial

OBJECTIVES

To assess the impact of a clinical decision model for febrile children at risk for serious bacterial infections (SBI) attending the emergency department (ED).

METHODS

Randomized controlled trial with 439 febrile children, aged 1 month-16 years, attending the pediatric ED of a Dutch university hospital during 2010-2012. Febrile children were randomly assigned to the intervention (clinical decision model; n = 219) or the control group (usual care; n = 220). The clinical decision model included clinical symptoms, vital signs, and C-reactive protein and provided high/low-risks for "pneumonia" and "other SBI". Nurses were guided by the intervention to initiate additional tests for high-risk children. The clinical decision model was evaluated by 1) area-under-the-receiver-operating-characteristic-curve (AUC) to indicate discriminative ability and 2) feasibility, to measure nurses' compliance to model recommendations. Primary patient outcome was defined as correct SBI diagnoses. Secondary process outcomes were defined as length of stay; diagnostic tests; antibiotic treatment; hospital admission; revisits and medical costs.

RESULTS

The decision model had good discriminative ability for both pneumonia (n = 33; AUC 0.83 (95% CI 0.75-0.90)) and other SBI (n = 22; AUC 0.81 (95% CI 0.72-0.90)). Compliance to model recommendations was high (86%). No differences in correct SBI determination were observed. Application of the clinical decision model resulted in less full-blood-counts (14% vs. 22%, p-value < 0.05) and more urine-dipstick testing (71% vs. 61%, p-value < 0.05).

CONCLUSIONS

In contrast to our expectations no substantial impact on patient outcome was perceived. The clinical decision model preserved, however, good discriminatory ability to detect SBI, achieved good compliance among nurses and resulted in a more standardized diagnostic approach towards febrile children, with less full blood-counts and more rightfully urine-dipstick testing.

TRIAL REGISTRATION

Nederlands Trial Register NTR2381.

The Prognostic Value of the Work Ability Index for Sickness Absence among Office Workers

BACKGROUND

The work ability index (WAI) is a frequently used tool in occupational health to identify workers at risk for a reduced work performance and for work-related disability. However, information about the prognostic value of the WAI to identify workers at risk for sickness absence is scarce.

OBJECTIVES

To investigate the prognostic value of the WAI for sickness absence, and whether the discriminative ability differs across demographic subgroups.

METHODS

At baseline, the WAI (score 7-49) was assessed among 1,331 office workers from a Dutch financial service company. Sickness absence was registered during 12-months follow-up and categorised as 0 days, 0<days<5, 5≤days<15, and ≥15 days in one year. Associations between WAI and sickness absence were estimated by multinomial regression analyses. Discriminative ability of the WAI was assessed by the Area Under the Curve (AUC) and Ordinal c-index (ORC). Test characteristics were determined for dichotomised outcomes. Additional analyses were performed for separate WAI dimensions, and subgroup analyses for demographic groups.

RESULTS

A lower WAI was associated with sickness absence (≥15 days vs. 0 days: per point lower WAI score OR=1.27; 95%CI 1.21-1.33). The WAI showed reasonable ability to discriminate between categories of sickness absence (ORC=0.65; 95%CI 0.63-0.68). Highest discrimination was found for comparing workers with ≥15 sick days with 0 sick days (AUC=0.77) or with 1-5 sick days (AUC=0.69). At the cut-off for poor work ability (WAI≤27) the sensitivity to identify workers at risk for ≥15 sick days was 7.5%, the specificity 99.6%, and the positive predictive value 82%. The performance was similar across demographic subgroups.

CONCLUSIONS

The WAI could be used to identify workers at high risk for prolonged sickness absence. However, due to low sensitivity many workers will be missed. Hence, additional factors are required to better identify workers at highest risk.

Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study

OBJECTIVES

To develop a risk prediction model to preoperatively discriminate between benign, borderline, stage I invasive, stage II-IV invasive, and secondary metastatic ovarian tumours.

DESIGN

Observational diagnostic study using prospectively collected clinical and ultrasound data.

SETTING

24 ultrasound centres in 10 countries.

PARTICIPANTS

Women with an ovarian (including para-ovarian and tubal) mass and who underwent a standardised ultrasound examination before surgery. The model was developed on 3506 patients recruited between 1999 and 2007, temporally validated on 2403 patients recruited between 2009 and 2012, and then updated on all 5909 patients.

MAIN OUTCOME MEASURES

Histological classification and surgical staging of the mass.

RESULTS

The Assessment of Different NEoplasias in the adneXa (ADNEX) model contains three clinical and six ultrasound predictors: age, serum CA-125 level, type of centre (oncology centres v other hospitals), maximum diameter of lesion, proportion of solid tissue, more than 10 cyst locules, number of papillary projections, acoustic shadows, and ascites. The area under the receiver operating characteristic curve (AUC) for the classic discrimination between benign and malignant tumours was 0.94 (0.93 to 0.95) on temporal validation. The AUC was 0.85 for benign versus borderline, 0.92 for benign versus stage I cancer, 0.99 for benign versus stage II-IV cancer, and 0.95 for benign versus secondary metastatic. AUCs between malignant subtypes varied between 0.71 and 0.95, with an AUC of 0.75 for borderline versus stage I cancer and 0.82 for stage II-IV versus secondary metastatic. Calibration curves showed that the estimated risks were accurate.

CONCLUSIONS

The ADNEX model discriminates well between benign and malignant tumours and offers fair to excellent discrimination between four types of ovarian malignancy. The use of ADNEX has the potential to improve triage and management decisions and so reduce morbidity and mortality associated with adnexal pathology.

Assessing calibration of multinomial risk prediction models

Calibration, that is, whether observed outcomes agree with predicted risks, is important when evaluating risk prediction models. For dichotomous outcomes, several tools exist to assess different aspects of model calibration, such as calibration-in-the-large, logistic recalibration, and (non-)parametric calibration plots. We aim to extend these tools to prediction models for polytomous outcomes. We focus on models developed using multinomial logistic regression (MLR): outcome Y with k categories is predicted using k - 1 equations comparing each category i (i = 2, … ,k) with reference category 1 using a set of predictors, resulting in k - 1 linear predictors. We propose a multinomial logistic recalibration framework that involves an MLR fit where Y is predicted using the k - 1 linear predictors from the prediction model. A non-parametric alternative may use vector splines for the effects of the linear predictors. The parametric and non-parametric frameworks can be used to generate multinomial calibration plots. Further, the parametric framework can be used for the estimation and statistical testing of calibration intercepts and slopes. Two illustrative case studies are presented, one on the diagnosis of malignancy of ovarian tumors and one on residual mass diagnosis in testicular cancer patients treated with cisplatin-based chemotherapy. The risk prediction models were developed on data from 2037 and 544 patients and externally validated on 1107 and 550 patients, respectively. We conclude that calibration tools can be extended to polytomous outcomes. The polytomous calibration plots are particularly informative through the visual summary of the calibration performance.

The Rotterdam Study: 2014 objectives and design update

The Rotterdam Study is a prospective cohort study ongoing since 1990 in the city of Rotterdam in The Netherlands. The study targets cardiovascular, endocrine, hepatic, neurological, ophthalmic, psychiatric, dermatological, oncological, and respiratory diseases. As of 2008, 14,926 subjects aged 45 years or over comprise the Rotterdam Study cohort. The findings of the Rotterdam Study have been presented in over a 1,000 research articles and reports (see www.erasmus-epidemiology.nl/rotterdamstudy ). This article gives the rationale of the study and its design. It also presents a summary of the major findings and an update of the objectives and methods.

Clinical prediction model to aid emergency doctors managing febrile children at risk of serious bacterial infections: diagnostic study

OBJECTIVE

To derive, cross validate, and externally validate a clinical prediction model that assesses the risks of different serious bacterial infections in children with fever at the emergency department.

DESIGN

Prospective observational diagnostic study.

SETTING

Three paediatric emergency care units: two in the Netherlands and one in the United Kingdom.

PARTICIPANTS

Children with fever, aged 1 month to 15 years, at three paediatric emergency care units: Rotterdam (n=1750) and the Hague (n=967), the Netherlands, and Coventry (n=487), United Kingdom. A prediction model was constructed using multivariable polytomous logistic regression analysis and included the predefined predictor variables age, duration of fever, tachycardia, temperature, tachypnoea, ill appearance, chest wall retractions, prolonged capillary refill time (>3 seconds), oxygen saturation <94%, and C reactive protein.

MAIN OUTCOME MEASURES

Pneumonia, other serious bacterial infections (SBIs, including septicaemia/meningitis, urinary tract infections, and others), and no SBIs.

RESULTS

Oxygen saturation <94% and presence of tachypnoea were important predictors of pneumonia. A raised C reactive protein level predicted the presence of both pneumonia and other SBIs, whereas chest wall retractions and oxygen saturation <94% were useful to rule out the presence of other SBIs. Discriminative ability (C statistic) to predict pneumonia was 0.81 (95% confidence interval 0.73 to 0.88); for other SBIs this was even better: 0.86 (0.79 to 0.92). Risk thresholds of 10% or more were useful to identify children with serious bacterial infections; risk thresholds less than 2.5% were useful to rule out the presence of serious bacterial infections. External validation showed good discrimination for the prediction of pneumonia (0.81, 0.69 to 0.93); discriminative ability for the prediction of other SBIs was lower (0.69, 0.53 to 0.86).

CONCLUSION

A validated prediction model, including clinical signs, symptoms, and C reactive protein level, was useful for estimating the likelihood of pneumonia and other SBIs in children with fever, such as septicaemia/meningitis and urinary tract infections.