When and how to use data from randomised trials to develop or validate prognostic models

An interrater reliability analysis of preoperative mortality risk calculators used for elective high-risk noncardiac surgical patients shows poor to moderate reliability

BACKGROUND

Multiple preoperative calculators are available online to predict preoperative mortality risk for noncardiac surgical patients. However, it is currently unknown how these risk calculators perform across different raters. The current study investigated the interrater reliability of three preoperative mortality risk calculators in an elective high-risk noncardiac surgical patient population to evaluate if these calculators can be safely used for identification of high-risk noncardiac surgical patients for a preoperative multidisciplinary team discussion.

METHODS

Five anesthesiologists assessed the preoperative mortality risk of 34 high-risk patients using the preoperative score to calculate postoperative mortality risks (POSPOM), the American College of Surgeons surgical risk calculator (SRC), and the surgical outcome risk tool (SORT). In total, 170 calculations per calculator were gathered.

RESULTS

Interrater reliability was poor for SORT (ICC (C.I. 95%) = 0.46 (0.30-0.63)) and moderate for SRC (ICC = 0.65 (0.51-0.78)) and POSPOM (ICC = 0.63 (0.49-0.77). The absolute range of calculated mortality risk was 0.2-72% for POSPOM, 0-36% for SRC, and 0.4-17% for SORT. The coefficient of variation increased in higher risk classes for POSPOM and SORT. The extended Bland-Altman limits of agreement suggested that all raters contributed to the variation in calculated risks.

CONCLUSION

The current results indicate that the preoperative risk calculators POSPOM, SRC, and SORT exhibit poor to moderate interrater reliability. These calculators are not sufficiently accurate for clinical identification and preoperative counseling of high-risk surgical patients. Clinicians should be trained in using mortality risk calculators. Also, clinicians should be cautious when using predicted mortality estimates from these calculators to identify high-risk noncardiac surgical patients for elective surgery.

Development and validation of a lifetime prediction model for incident type 2 diabetes in patients with established cardiovascular disease: the CVD2DM model

AIMS

Identifying patients with established cardiovascular disease (CVD) who are at high risk of type 2 diabetes (T2D) may allow for early interventions, reducing the development of T2D and associated morbidity. The aim of this study was to develop and externally validate the CVD2DM model to estimate the 10-year and lifetime risks of T2D in patients with established CVD.

METHODS AND RESULTS

Sex-specific, competing risk-adjusted Cox proportional hazard models were derived in 19 281 participants with established CVD and without diabetes at baseline from the UK Biobank. The core model's pre-specified predictors were age, current smoking, family history of diabetes mellitus, body mass index, systolic blood pressure, fasting plasma glucose, and HDL cholesterol. The extended model also included HbA1c. The model was externally validated in 3481 patients from the UCC-SMART study. During a median follow-up of 12.2 years (interquartile interval 11.3-13.1), 1628 participants with established CVD were diagnosed with T2D in the UK Biobank. External validation c-statistics were 0.79 [95% confidence interval (CI) 0.76-0.82] for the core model and 0.81 (95% CI 0.78-0.84) for the extended model. Calibration plots showed agreement between predicted and observed 10-year risk of T2D.

CONCLUSION

The 10-year and lifetime risks of T2D can be estimated with the CVD2DM model in patients with established CVD, using readily available clinical predictors. The model would benefit from further validation across diverse ethnic groups to enhance its applicability. Informing patients about their T2D risk could motivate them further to adhere to a healthy lifestyle.

A Dynamic Prediction Model Supporting Individual Life Expectancy Prediction Based on Longitudinal Time-Dependent Covariates

In the field of clinical chronic diseases, common prediction results (such as survival rate) and effect size hazard ratio (HR) are relative indicators, resulting in more abstract information. However, clinicians and patients are more interested in simple and intuitive concepts of (survival) time, such as how long a patient may live or how much longer a patient in a treatment group will live. In addition, due to the long follow-up time, resulting in generation of longitudinal time-dependent covariate information, patients are interested in how long they will survive at each follow-up visit. In this study, based on a time scale indicator-restricted mean survival time (RMST)-we proposed a dynamic RMST prediction model by considering longitudinal time-dependent covariates and utilizing joint model techniques. The model can describe the change trajectory of longitudinal time-dependent covariates and predict the average survival times of patients at different time points (such as follow-up visits). Simulation studies through Monte Carlo cross-validation showed that the dynamic RMST prediction model was superior to the static RMST model. In addition, the dynamic RMST prediction model was applied to a primary biliary cirrhosis (PBC) population to dynamically predict the average survival times of the patients, and the average C-index of the internal validation of the model reached 0.81, which was better than that of the static RMST regression. Therefore, the proposed dynamic RMST prediction model has better performance in prediction and can provide a scientific basis for clinicians and patients to make clinical decisions.

Propensity-based standardization to enhance the validation and interpretation of prediction model discrimination for a target population

External validation of the discriminative ability of prediction models is of key importance. However, the interpretation of such evaluations is challenging, as the ability to discriminate depends on both the sample characteristics (ie, case-mix) and the generalizability of predictor coefficients, but most discrimination indices do not provide any insight into their respective contributions. To disentangle differences in discriminative ability across external validation samples due to a lack of model generalizability from differences in sample characteristics, we propose propensity-weighted measures of discrimination. These weighted metrics, which are derived from propensity scores for sample membership, are standardized for case-mix differences between the model development and validation samples, allowing for a fair comparison of discriminative ability in terms of model characteristics in a target population of interest. We illustrate our methods with the validation of eight prediction models for deep vein thrombosis in 12 external validation data sets and assess our methods in a simulation study. In the illustrative example, propensity score standardization reduced between-study heterogeneity of discrimination, indicating that between-study variability was partially attributable to case-mix. The simulation study showed that only flexible propensity-score methods (allowing for non-linear effects) produced unbiased estimates of model discrimination in the target population, and only when the positivity assumption was met. Propensity score-based standardization may facilitate the interpretation of (heterogeneity in) discriminative ability of a prediction model as observed across multiple studies, and may guide model updating strategies for a particular target population. Careful propensity score modeling with attention for non-linear relations is recommended.

Predicting relapse or recurrence of depression: systematic review of prognostic models

BACKGROUND

Relapse and recurrence of depression are common, contributing to the overall burden of depression globally. Accurate prediction of relapse or recurrence while patients are well would allow the identification of high-risk individuals and may effectively guide the allocation of interventions to prevent relapse and recurrence.

AIMS

To review prognostic models developed to predict the risk of relapse, recurrence, sustained remission, or recovery in adults with remitted major depressive disorder.

METHOD

We searched the Cochrane Library (current issue); Ovid MEDLINE (1946 onwards); Ovid Embase (1980 onwards); Ovid PsycINFO (1806 onwards); and Web of Science (1900 onwards) up to May 2021. We included development and external validation studies of multivariable prognostic models. We assessed risk of bias of included studies using the Prediction model risk of bias assessment tool (PROBAST).

RESULTS

We identified 12 eligible prognostic model studies (11 unique prognostic models): 8 model development-only studies, 3 model development and external validation studies and 1 external validation-only study. Multiple estimates of performance measures were not available and meta-analysis was therefore not necessary. Eleven out of the 12 included studies were assessed as being at high overall risk of bias and none examined clinical utility.

CONCLUSIONS

Due to high risk of bias of the included studies, poor predictive performance and limited external validation of the models identified, presently available clinical prediction models for relapse and recurrence of depression are not yet sufficiently developed for deploying in clinical settings. There is a need for improved prognosis research in this clinical area and future studies should conform to best practice methodological and reporting guidelines.

External Validation of the ASSURE Model for Predicting Oncological Outcomes After Resection of High-risk Renal Cell Carcinoma (RESCUE Study: UroCCR 88)

A prognostic model based on the population of the ASSURE phase 3 trial has recently been described. The ASSURE model stratifies patients into risk groups to predict survival after surgical resection of intermediate- and high-risk localised kidney cancer. We evaluated this model in an independent cohort of 1372 patients using discrimination, calibration, and decision curve analysis. Regarding disease-free survival, the ASSURE model showed modest discrimination (65%), miscalibration, and poor net benefit compared with the UCLA Integrated Staging System (UISS) and Leibovich 2018 models. Similarly, the ability of the ASSURE model to predict overall survival was poor in terms of discrimination (63%), with overestimation on calibration plots and a modest net benefit for the probability threshold of between 10% and 40%. Overall, our results show that the performance of the ASSURE model was less optimistic than expected, and not associated with a clear improvement in patient selection and clinical usefulness in comparison to with available models. We propose an updated version using the recalibration method, which leads to a (slight) improvement in performance but should be validated in another external population.

Patient summary

The recent ASSURE model evaluates survival after surgery for nonmetastatic kidney cancer. We found no clear improvement in patient classification when we compared ASSURE with older models, so use of this model for patients with nonmetastatic kidney cancer still needs to be clarified.

Clinical Prediction Models in Sports Medicine: A Guide for Clinicians and Researchers

SYNOPSIS

Participating in sport carries inherent risk of injury. Clinicians execute high-level clinical reasoning and decision making to support athletes to achieve the best outcomes. Accurately diagnosing a problem, estimating prognosis, or selecting the most suitable intervention for each athlete is challenging. Clinical prediction models are tools to assist clinicians in estimating the risk or probability of a health outcome for an individual by using data from multiple predictors. Although common in general medical literature, clinical prediction models are rare in sports medicine. The purpose of this article was to (1) describe the steps required to develop and validate (ie, evaluate) a clinical prediction model for clinical researchers, and (2) help sports medicine clinicians understand and interpret clinical prediction model studies. Using a case study to illustrate how to implement clinical prediction models in practice, we address the following issues in developing and validating a clinical prediction model: study design and data, sample size, missing data, selecting predictors, handling continuous predictors, model fitting, internal and external validation, performance measures, reporting, and model presentation. Our work builds on initiatives to improve diagnostic and prognostic clinical research, including the PROGnosis RESearch Strategy (PROGRESS) series of papers and textbook and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. J Orthop Sports Phys Ther 2021;51(10):517-525. doi:10.2519/jospt.2021.10697.

Expected individual benefit of prophylactic platelet transfusions in hemato-oncology patients based on bleeding risks

BACKGROUND

Prophylactic platelet transfusions prevent bleeding in hemato-oncology patients, but it is unclear how any benefit varies between patients. Our aim was to assess if patients with different baseline risks for bleeding benefit differently from a prophylactic platelet transfusion strategy.

STUDY DESIGN AND METHODS

Using the data from the randomized controlled TOPPS trial (Trial of Platelet Prophylaxis), we developed a prediction model for World Health Organization grades 2, 3, and 4 bleeding risk (defined as at least one bleeding episode in a 30 days period) and grouped patients in four risk-quartiles based on this predicted baseline risk. Predictors in the model were baseline platelet count, age, diagnosis, disease modifying treatment, disease status, previous stem cell transplantation, and the randomization arm.

RESULTS

The model had a c-statistic of 0.58 (95% confidence interval [CI] 0.54-0.64). There was little variation in predicted risks (quartiles 46%, 47%, and 51%), but prophylactic platelet transfusions gave a risk reduction in all risk quartiles. The absolute risk difference (ARD) was 3.4% (CI -12.2 to 18.9) in the lowest risk quartile (quartile 1), 7.4% (95% CI -8.4 to 23.3) in quartile 2, 6.8% (95% CI -9.1 to 22.9) in quartile 3, and 12.8% (CI -3.1 to 28.7) in the highest risk quartile (quartile 4).

CONCLUSION

In our study, generally accepted bleeding risk predictors had limited predictive power (expressed by the low c-statistic), and, given the wide confidence intervals of predicted ARD, could not aid in identifying subgroups of patients who might benefit more (or less) from prophylactic platelet transfusion.

The development and validation of prognostic models for overall survival in the presence of missing data in the training dataset: a strategy with a detailed example

BACKGROUND

The United Kingdom Myeloma Research Alliance (UK-MRA) Myeloma Risk Profile is a prognostic model for overall survival. It was trained and tested on clinical trial data, aiming to improve the stratification of transplant ineligible (TNE) patients with newly diagnosed multiple myeloma. Missing data is a common problem which affects the development and validation of prognostic models, where decisions on how to address missingness have implications on the choice of methodology.

METHODS

Model building The training and test datasets were the TNE pathways from two large randomised multicentre, phase III clinical trials. Potential prognostic factors were identified by expert opinion. Missing data in the training dataset was imputed using multiple imputation by chained equations. Univariate analysis fitted Cox proportional hazards models in each imputed dataset with the estimates combined by Rubin's rules. Multivariable analysis applied penalised Cox regression models, with a fixed penalty term across the imputed datasets. The estimates from each imputed dataset and bootstrap standard errors were combined by Rubin's rules to define the prognostic model. Model assessment Calibration was assessed by visualising the observed and predicted probabilities across the imputed datasets. Discrimination was assessed by combining the prognostic separation D-statistic from each imputed dataset by Rubin's rules. Model validation The D-statistic was applied in a bootstrap internal validation process in the training dataset and an external validation process in the test dataset, where acceptable performance was pre-specified. Development of risk groups Risk groups were defined using the tertiles of the combined prognostic index, obtained by combining the prognostic index from each imputed dataset by Rubin's rules.

RESULTS

The training dataset included 1852 patients, 1268 (68.47%) with complete case data. Ten imputed datasets were generated. Five hundred twenty patients were included in the test dataset. The D-statistic for the prognostic model was 0.840 (95% CI 0.716-0.964) in the training dataset and 0.654 (95% CI 0.497-0.811) in the test dataset and the corrected D-Statistic was 0.801.

CONCLUSION

The decision to impute missing covariate data in the training dataset influenced the methods implemented to train and test the model. To extend current literature and aid future researchers, we have presented a detailed example of one approach. Whilst our example is not without limitations, a benefit is that all of the patient information available in the training dataset was utilised to develop the model.

TRIAL REGISTRATION

Both trials were registered; Myeloma IX- ISRCTN68454111 , registered 21 September 2000. Myeloma XI- ISRCTN49407852 , registered 24 June 2009.

Prognostic nomogram for progression-free survival in patients with BRCA mutations and platinum-sensitive recurrent ovarian cancer on maintenance olaparib therapy following response to chemotherapy

BACKGROUND

The impact of maintenance therapy with PARP inhibitors (PARPi) on progression-free survival (PFS) in patients with BRCA mutations and platinum-sensitive recurrent ovarian cancer (PSROC) varies widely. Individual prognostic factors do not reliably distinguish patients who progress early from those who have durable benefit. We developed and validated a prognostic nomogram to predict PFS in these patients.

METHODS

The nomogram was developed using data from a training patient cohort with BRCA mutations and high-grade serous PSROC on the placebo arm of two maintenance therapy trials, Study 19 and SOLO2/ENGOT-ov21. We performed multivariable Cox regression analysis based on pre-treatment characteristics to develop a nomogram that predicts PFS. We assessed the discrimination and validation of the nomogram in independent validation patient cohorts treated with maintenance olaparib.

RESULTS

The nomogram includes four PFS predictors: CA-125 at randomisation, platinum-free interval, presence of measurable disease and number of prior lines of platinum therapy. In the training (placebo) cohort (internal validation C-index 0.64), median PFS in the model-predicted good, intermediate and poor-risk groups was: 7.7 (95% CI 5.3-11.3), 5.4 (4.8-5.8) and 2.9 (2.8-4.4) months, respectively. In the validation (olaparib) cohort (C-index 0.71), median PFS in the model-predicted good, intermediate and poor-risk groups was: not reached, 16.6 (13.1-22.4) and 8.3 (7.1-10.8) months, respectively. The nomogram showed good calibration in the validation cohort (calibration plot).

CONCLUSIONS

This nomogram can be used to predict PFS and counsel patients with BRCA mutations and PSROC prior to maintenance olaparib and for stratification of patients in trials of maintenance therapies.

The development and validation of a prognostic model to PREDICT Relapse of depression in adult patients in primary care: protocol for the PREDICTR study

BACKGROUND

Most patients who present with depression are treated in primary care by general practitioners (GPs). Relapse of depression is common (at least 50% of patients treated for depression will relapse after a single episode) and leads to considerable morbidity and decreased quality of life for patients. The majority of patients will relapse within 6 months, and those with a history of relapse are more likely to relapse in the future than those with no such history. GPs see a largely undifferentiated case-mix of patients, and once patients with depression reach remission, there is limited guidance to help GPs stratify patients according to risk of relapse. We aim to develop a prognostic model to predict an individual's risk of relapse within 6-8 months of entering remission. The long-term objective is to inform the clinical management of depression after the acute phase.

METHODS

We will develop a prognostic model using secondary analysis of individual participant data drawn from seven RCTs and one longitudinal cohort study in primary or community care settings. We will use logistic regression to predict the outcome of relapse of depression within 6-8 months. We plan to include the following established relapse predictors in the model: residual depressive symptoms, number of previous depressive episodes, co-morbid anxiety and severity of index episode. We will use a "full model" development approach, including all available predictors. Performance statistics (optimism-adjusted C-statistic, calibration-in-the-large, calibration slope) and calibration plots (with smoothed calibration curves) will be calculated. Generalisability of predictive performance will be assessed through internal-external cross-validation. Clinical utility will be explored through net benefit analysis.

DISCUSSION

We will derive a statistical model to predict relapse of depression in remitted depressed patients in primary care. Assuming the model has sufficient predictive performance, we outline the next steps including independent external validation and further assessment of clinical utility and impact.

STUDY REGISTRATION

ClinicalTrials.gov ID: NCT04666662.

Prognostic models for predicting relapse or recurrence of major depressive disorder in adults

BACKGROUND

Relapse (the re-emergence of depressive symptoms after some level of improvement but preceding recovery) and recurrence (onset of a new depressive episode after recovery) are common in depression, lead to worse outcomes and quality of life for patients and exert a high economic cost on society. Outcomes can be predicted by using multivariable prognostic models, which use information about several predictors to produce an individualised risk estimate. The ability to accurately predict relapse or recurrence while patients are well (in remission) would allow the identification of high-risk individuals and may improve overall treatment outcomes for patients by enabling more efficient allocation of interventions to prevent relapse and recurrence.

OBJECTIVES

To summarise the predictive performance of prognostic models developed to predict the risk of relapse, recurrence, sustained remission or recovery in adults with major depressive disorder who meet criteria for remission or recovery.

SEARCH METHODS

We searched the Cochrane Library (current issue); Ovid MEDLINE (1946 onwards); Ovid Embase (1980 onwards); Ovid PsycINFO (1806 onwards); and Web of Science (1900 onwards) up to May 2020. We also searched sources of grey literature, screened the reference lists of included studies and performed a forward citation search. There were no restrictions applied to the searches by date, language or publication status .

SELECTION CRITERIA

We included development and external validation (testing model performance in data separate from the development data) studies of any multivariable prognostic models (including two or more predictors) to predict relapse, recurrence, sustained remission, or recovery in adults (aged 18 years and over) with remitted depression, in any clinical setting. We included all study designs and accepted all definitions of relapse, recurrence and other related outcomes. We did not specify a comparator prognostic model.

DATA COLLECTION AND ANALYSIS

Two review authors independently screened references; extracted data (using a template based on the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS)); and assessed risks of bias of included studies (using the Prediction model Risk Of Bias ASsessment Tool (PROBAST)). We referred any disagreements to a third independent review author. Where we found sufficient (10 or more) external validation studies of an individual model, we planned to perform a meta-analysis of its predictive performance, specifically with respect to its calibration (how well the predicted probabilities match the observed proportions of individuals that experience the outcome) and discrimination (the ability of the model to differentiate between those with and without the outcome). Recommendations could not be qualified using the GRADE system, as guidance is not yet available for prognostic model reviews.

MAIN RESULTS

We identified 11 eligible prognostic model studies (10 unique prognostic models). Seven were model development studies; three were model development and external validation studies; and one was an external validation-only study. Multiple estimates of performance measures were not available for any of the models and, meta-analysis was therefore not possible. Ten out of the 11 included studies were assessed as being at high overall risk of bias. Common weaknesses included insufficient sample size, inappropriate handling of missing data and lack of information about discrimination and calibration. One paper (Klein 2018) was at low overall risk of bias and presented a prognostic model including the following predictors: number of previous depressive episodes, residual depressive symptoms and severity of the last depressive episode. The external predictive performance of this model was poor (C-statistic 0.59; calibration slope 0.56; confidence intervals not reported). None of the identified studies examined the clinical utility (net benefit) of the developed model.

AUTHORS' CONCLUSIONS

Of the 10 prognostic models identified (across 11 studies), only four underwent external validation. Most of the studies (n = 10) were assessed as being at high overall risk of bias, and the one study that was at low risk of bias presented a model with poor predictive performance. There is a need for improved prognostic research in this clinical area, with future studies conforming to current best practice recommendations for prognostic model development/validation and reporting findings in line with the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.

Predictive utilities of lipid traits, lipoprotein subfractions and other risk factors for incident diabetes: a machine learning approach in the Diabetes Prevention Program

INTRODUCTION

Although various lipid and non-lipid analytes measured by nuclear magnetic resonance (NMR) spectroscopy have been associated with type 2 diabetes, a structured comparison of the ability of NMR-derived biomarkers and standard lipids to predict individual diabetes risk has not been undertaken in larger studies nor among individuals at high risk of diabetes.

RESEARCH DESIGN AND METHODS

Cumulative discriminative utilities of various groups of biomarkers including NMR lipoproteins, related non-lipid biomarkers, standard lipids, and demographic and glycemic traits were compared for short-term (3.2 years) and long-term (15 years) diabetes development in the Diabetes Prevention Program, a multiethnic, placebo-controlled, randomized controlled trial of individuals with pre-diabetes in the USA (N=2590). Logistic regression, Cox proportional hazards model and six different hyperparameter-tuned machine learning algorithms were compared. The Matthews Correlation Coefficient (MCC) was used as the primary measure of discriminative utility.

RESULTS

Models with baseline NMR analytes and their changes did not improve the discriminative utility of simpler models including standard lipids or demographic and glycemic traits. Across all algorithms, models with baseline 2-hour glucose performed the best (max MCC=0.36). Sophisticated machine learning algorithms performed similarly to logistic regression in this study.

CONCLUSIONS

NMR lipoproteins and related non-lipid biomarkers were associated but did not augment discrimination of diabetes risk beyond traditional diabetes risk factors except for 2-hour glucose. Machine learning algorithms provided no meaningful improvement for discrimination compared with logistic regression, which suggests a lack of influential latent interactions among the analytes assessed in this study.

TRIAL REGISTRATION NUMBER

Diabetes Prevention Program: NCT00004992; Diabetes Prevention Program Outcomes Study: NCT00038727.

Applicability of Blood Pressure-Lowering Drug Trials to Real-World Patients With Cardiovascular Disease

This study aimed to assess applicability of blood pressure-lowering drug trials to real-world secondary preventive care. We applied the eligibility criteria of the landmark blood pressure-lowering drug trials (EUROPA, PEACE, HOPE-peripheral arterial disease [PAD], PRoFESS, and PROGRESS) to patients with coronary artery disease (CAD; n=5155), peripheral arterial disease (PAD; n=1487), and cerebrovascular disease (n=2515) participating in the UCC-SMART cohort. Baseline differences according to trial eligibility were assessed. Differences in risk of all-cause mortality and a composite of cardiovascular death, myocardial infarction, and stroke (major adverse cardiovascular event) were calculated using Cox proportional hazard models, adjusted for age, sex, and cardiovascular risk factors. Seventy-five percent of UCC-SMART patients with CAD would have been eligible for EUROPA, 84% for PEACE, 59% of patients with PAD for HOPE-PAD, 17% of patients with cerebrovascular disease for PRoFESS, and 100% for PROGRESS. Eligible patients were older (average difference ranging 1.4-14.6 years across trials). Eligible patients with CAD were at lower risk of major adverse cardiovascular event after adjustment for age, sex, and cardiovascular risk factors in PEACE (hazard ratio, 0.65 [95% CI, 0.53-0.79]) and of mortality in both EUROPA (hazard ratio, 0.72 [95% CI, 0.62-0.82]) and PEACE (0.63 [95% CI, 0.51-0.78]). Adjusted mortality and major adverse cardiovascular event risks were not different between eligible and ineligible patients with PAD and cerebrovascular disease in HOPE-PAD, PRoFESS, and PROGRESS. The majority of real-world patients with CAD, PAD, or cerebrovascular disease would be eligible for landmark trials on blood pressure-lowering drugs. Patients with CAD ineligible for the EUROPA and PEACE trials are at higher adjusted mortality and major adverse cardiovascular event risks, which may limit applicability of their results to ineligible patients.

Outpatient physiotherapy versus home-based rehabilitation for patients at risk of poor outcomes after knee arthroplasty: CORKA RCT

BACKGROUND

Over 100,000 primary knee arthroplasty operations are undertaken annually in the UK. Around 15-30% of patients do not report a good outcome. Better rehabilitation strategies may improve patient-reported outcomes.

OBJECTIVES

To compare the outcomes from a traditional outpatient physiotherapy model with those from a home-based rehabilitation programme for people assessed as being at risk of a poor outcome after knee arthroplasty.

DESIGN

An individually randomised, two-arm controlled trial with a blinded outcome assessment, a parallel health economic evaluation and a nested qualitative study.

SETTING

The trial took place in 14 NHS physiotherapy departments.

PARTICIPANTS

People identified as being at high risk of a poor outcome after knee arthroplasty.

INTERVENTIONS

A multicomponent home-based rehabilitation package delivered by rehabilitation assistants with supervision from qualified therapists compared with usual-care outpatient physiotherapy.

MAIN OUTCOME MEASURES

The primary outcome was the Late Life Function and Disability Instrument at 12 months. Secondary outcomes were the Oxford Knee Score (a disease-specific measure of function); Knee injury and Osteoarthritis Outcome Score; Quality of Life subscale; Physical Activity Scale for the Elderly; EuroQol-5 Dimensions, five-level version; and physical function assessed using the Figure-of-8 Walk Test, 30-Second Chair Stand Test and Single Leg Stance. Data on the use of health-care services, time off work and informal care were collected using participant diaries.

RESULTS

In total, 621 participants were randomised. A total of 309 participants were assigned to the COmmunity based Rehabilitation after Knee Arthroplasty (CORKA) home-based rehabilitation programme, receiving a median of five treatment sessions (interquartile range 4-7 sessions). A total of 312 participants were assigned to usual care, receiving a median of four sessions (interquartile range 2-6 sessions). The primary outcome, Late Life Function and Disability Instrument function total score at 12 months, was collected for 279 participants (89%) in the home-based CORKA group and 287 participants (92%) in the usual-care group. No clinically or statistically significant difference was found between the groups (intention-to-treat adjusted difference 0.49 points, 95% confidence interval -0.89 to 1.88 points; p = 0.48). There were no statistically significant differences between the groups in any of the patient-reported or physical secondary outcome measures at 6 or 12 months post randomisation. The health economic analysis found that the CORKA intervention was cheaper to provide than usual care (£66 less per participant). Total societal costs (combining health-care costs and other costs) were lower for the CORKA intervention than usual care (£316 less per participant). Adopting a societal perspective, CORKA had a 75% probability of being cost-effective at a threshold of £30,000 per quality-adjusted life-year. Adopting the narrower health and social care perspective, CORKA had a 43% probability of being cost-effective at the same threshold.

LIMITATIONS

The interventions were of short duration and were set within current commissioning guidance for UK physiotherapy. Participants and treating therapists could not be blinded.

CONCLUSIONS

This randomised controlled trial found no important differences in outcomes when post-arthroplasty rehabilitation was delivered using a home-based, rehabilitation assistant-delivered rehabilitation package or a traditional outpatient model. However, the health economic evaluation found that when adopting a societal perspective, the CORKA home-based intervention was cost-saving and more effective than, and thus dominant over, usual care, owing to reduced time away from paid employment for this group. Further research could look at identifying the risk of poor outcome and further evaluation of a cost-effective treatment, including the workforce model to deliver it.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN13517704.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 24, No. 65. See the NIHR Journals Library website for further project information.

Treatment success of internet-based vestibular rehabilitation in general practice: development and internal validation of a prediction model

OBJECTIVES

To develop and internally validate prediction models to assess treatment success of both stand-alone and blended online vestibular rehabilitation (VR) in patients with chronic vestibular syndrome.

DESIGN

Secondary analysis of a randomised controlled trial.

SETTING

59 general practices in The Netherlands.

PARTICIPANTS

202 adults, aged 50 years and older with a chronic vestibular syndrome who received either stand-alone VR (98) or blended VR (104). Stand-alone VR consisted of a 6-week, internet-based intervention with weekly online sessions and daily exercises. In blended VR, the same intervention was supplemented with physiotherapy support.

MAIN OUTCOME MEASURES

Successful treatment was defined as: clinically relevant improvement of (1) vestibular symptoms (≥3 points improvement Vertigo Symptom Scale-Short Form); (2) vestibular-related disability (>11 points improvement Dizziness Handicap Inventory); and (3) both vestibular symptoms and vestibular-related disability. We assessed performance of the predictive models by applying calibration plots, Hosmer-Lemeshow statistics, area under the receiver operating characteristic curves (AUC) and applied internal validation.

RESULTS

Improvement of vestibular symptoms, vestibular-related disability or both was seen in 121, 81 and 64 participants, respectively. We generated predictive models for each outcome, resulting in different predictors in the final models. Calibration for all models was adequate with non-significant Hosmer-Lemeshow statistics, but the discriminative ability of the final predictive models was poor (AUC 0.54 to 0.61). None of the identified models are therefore suitable for use in daily general practice to predict treatment success of online VR.

CONCLUSION

It is difficult to predict treatment success of internet-based VR and it remains unclear who should be treated with stand-alone VR or blended VR. Because we were unable to develop a useful prediction model, the decision to offer stand-alone or blended VR should for now be based on availability, cost effectiveness and patient preference.

TRIAL REGISTRATION NUMBER

The Netherlands Trial Register NTR5712.

Prognostic models for newly-diagnosed chronic lymphocytic leukaemia in adults: a systematic review and meta-analysis

BACKGROUND

Chronic lymphocytic leukaemia (CLL) is the most common cancer of the lymphatic system in Western countries. Several clinical and biological factors for CLL have been identified. However, it remains unclear which of the available prognostic models combining those factors can be used in clinical practice to predict long-term outcome in people newly-diagnosed with CLL.

OBJECTIVES

To identify, describe and appraise all prognostic models developed to predict overall survival (OS), progression-free survival (PFS) or treatment-free survival (TFS) in newly-diagnosed (previously untreated) adults with CLL, and meta-analyse their predictive performances.

SEARCH METHODS

We searched MEDLINE (from January 1950 to June 2019 via Ovid), Embase (from 1974 to June 2019) and registries of ongoing trials (to 5 March 2020) for development and validation studies of prognostic models for untreated adults with CLL. In addition, we screened the reference lists and citation indices of included studies.

SELECTION CRITERIA

We included all prognostic models developed for CLL which predict OS, PFS, or TFS, provided they combined prognostic factors known before treatment initiation, and any studies that tested the performance of these models in individuals other than the ones included in model development (i.e. 'external model validation studies'). We included studies of adults with confirmed B-cell CLL who had not received treatment prior to the start of the study. We did not restrict the search based on study design.

DATA COLLECTION AND ANALYSIS

We developed a data extraction form to collect information based on the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS). Independent pairs of review authors screened references, extracted data and assessed risk of bias according to the Prediction model Risk Of Bias ASsessment Tool (PROBAST). For models that were externally validated at least three times, we aimed to perform a quantitative meta-analysis of their predictive performance, notably their calibration (proportion of people predicted to experience the outcome who do so) and discrimination (ability to differentiate between people with and without the event) using a random-effects model. When a model categorised individuals into risk categories, we pooled outcome frequencies per risk group (low, intermediate, high and very high). We did not apply GRADE as guidance is not yet available for reviews of prognostic models.

MAIN RESULTS

From 52 eligible studies, we identified 12 externally validated models: six were developed for OS, one for PFS and five for TFS. In general, reporting of the studies was poor, especially predictive performance measures for calibration and discrimination; but also basic information, such as eligibility criteria and the recruitment period of participants was often missing. We rated almost all studies at high or unclear risk of bias according to PROBAST. Overall, the applicability of the models and their validation studies was low or unclear; the most common reasons were inappropriate handling of missing data and serious reporting deficiencies concerning eligibility criteria, recruitment period, observation time and prediction performance measures. We report the results for three models predicting OS, which had available data from more than three external validation studies: CLL International Prognostic Index (CLL-IPI) This score includes five prognostic factors: age, clinical stage, IgHV mutational status, B2-microglobulin and TP53 status. Calibration: for the low-, intermediate- and high-risk groups, the pooled five-year survival per risk group from validation studies corresponded to the frequencies observed in the model development study. In the very high-risk group, predicted survival from CLL-IPI was lower than observed from external validation studies. Discrimination: the pooled c-statistic of seven external validation studies (3307 participants, 917 events) was 0.72 (95% confidence interval (CI) 0.67 to 0.77). The 95% prediction interval (PI) of this model for the c-statistic, which describes the expected interval for the model's discriminative ability in a new external validation study, ranged from 0.59 to 0.83. Barcelona-Brno score Aimed at simplifying the CLL-IPI, this score includes three prognostic factors: IgHV mutational status, del(17p) and del(11q). Calibration: for the low- and intermediate-risk group, the pooled survival per risk group corresponded to the frequencies observed in the model development study, although the score seems to overestimate survival for the high-risk group. Discrimination: the pooled c-statistic of four external validation studies (1755 participants, 416 events) was 0.64 (95% CI 0.60 to 0.67); 95% PI 0.59 to 0.68. MDACC 2007 index score The authors presented two versions of this model including six prognostic factors to predict OS: age, B2-microglobulin, absolute lymphocyte count, gender, clinical stage and number of nodal groups. Only one validation study was available for the more comprehensive version of the model, a formula with a nomogram, while seven studies (5127 participants, 994 events) validated the simplified version of the model, the index score. Calibration: for the low- and intermediate-risk groups, the pooled survival per risk group corresponded to the frequencies observed in the model development study, although the score seems to overestimate survival for the high-risk group. Discrimination: the pooled c-statistic of the seven external validation studies for the index score was 0.65 (95% CI 0.60 to 0.70); 95% PI 0.51 to 0.77.

AUTHORS' CONCLUSIONS

Despite the large number of published studies of prognostic models for OS, PFS or TFS for newly-diagnosed, untreated adults with CLL, only a minority of these (N = 12) have been externally validated for their respective primary outcome. Three models have undergone sufficient external validation to enable meta-analysis of the model's ability to predict survival outcomes. Lack of reporting prevented us from summarising calibration as recommended. Of the three models, the CLL-IPI shows the best discrimination, despite overestimation. However, performance of the models may change for individuals with CLL who receive improved treatment options, as the models included in this review were tested mostly on retrospective cohorts receiving a traditional treatment regimen. In conclusion, this review shows a clear need to improve the conducting and reporting of both prognostic model development and external validation studies. For prognostic models to be used as tools in clinical practice, the development of the models (and their subsequent validation studies) should adapt to include the latest therapy options to accurately predict performance. Adaptations should be timely.