Individual participant data meta-analysis for external validation, recalibration, and updating of a flexible parametric prognostic model

Applying models of care for total hip and knee arthroplasty: External validation of a published predictive model to identify extended stay risk prior to lower-limb arthroplasty

OBJECTIVE

This study aimed to externally validate a reported model for identifying patients requiring extended stay following lower limb arthroplasty in a new setting.

DESIGN

External validation of a previously reported prognostic model, using retrospective data.

SETTING

Medium-sized hospital orthopaedic department, Australia.

PARTICIPANTS

Electronic medical records were accessed for data collection between Sep-2019 and Feb-2020 and retrospective data extracted from 200 randomly selected total hip or knee arthroplasty patients.

INTERVENTION

Participants received total hip or knee replacement between 2-Feb-16 and 4-Apr-19. This study was a non-interventional retrospective study.

MAIN MEASURES

Model validation was assessed with discrimination, calibration on both original and adjusted forms of the candidate model. Decision curve analysis was conducted on the outputs of the adjusted model to determine net benefit at a predetermined decision threshold (0.5).

RESULTS

The original model performed poorly, grossly overestimating length of stay with mean calibration of -3.6 (95% confidence interval -3.9 to -3.2) and calibration slope of 0.52. Performance improved following adjustment of the model intercept and model coefficients (mean calibration 0.48, 95% confidence interval 0.16 to 0.80 and slope of 1.0), but remained poorly calibrated at low and medium risk threshold and net benefit was modest (three additional patients per hundred identified as at-risk) at the a-priori risk threshold.

CONCLUSIONS

External validation demonstrated poor performance when applied to a new patient population and would provide limited benefit for our institution. Implementation of predictive models for arthroplasty should include practical assessment of discrimination, calibration and net benefit at a clinically acceptable threshold.

Multivariable Model to Predict an ACTH Stimulation Test to Diagnose Adrenal Insufficiency Using Previous Test Results

Context

The adrenocorticotropin hormone stimulation test (AST) is used to diagnose adrenal insufficiency, and is often repeated in patients when monitoring recovery of the hypothalamo-pituitary-adrenal axis.

Objective

To develop and validate a prediction model that uses previous AST results with new baseline cortisol to predict the result of a new AST.

Methods

This was a retrospective, longitudinal cohort study in patients who had undergone at least 2 ASTs, using polynomial regression with backwards variable selection, at a Tertiary UK adult endocrinology center. Model was developed from 258 paired ASTs over 5 years in 175 adults (mean age 52.4 years, SD 16.4), then validated on data from 111 patients over 1 year (51.8, 17.5) from the same center, data collected after model development. Candidate prediction variables included previous test baseline adrenocorticotropin hormone (ACTH), previous test baseline and 30-minute cortisol, days between tests, and new baseline ACTH and cortisol used with calculated cortisol/ACTH ratios to assess 8 candidate predictors. The main outcome measure was a new test cortisol measured 30 minutes after Synacthen administration.

Results

Using 258 sequential ASTs from 175 patients for model development and 111 patient tests for model validation, previous baseline cortisol, previous 30-minute cortisol and new baseline cortisol were superior at predicting new 30-minute cortisol (R2 = 0.71 [0.49-0.93], area under the curve [AUC] = 0.97 [0.94-1.0]) than new baseline cortisol alone (R2 = 0.53 [0.22-0.84], AUC = 0.88 [0.81-0.95]).

Conclusion

Results of a previous AST can be objectively combined with new early-morning cortisol to predict the results of a new AST better than new early-morning cortisol alone. An online calculator is available at https://endocrinology.shinyapps.io/sheffield_sst_calculator/ for external validation.

Prediction nomogram development and validation for postoperative radiotherapy in the management of oral squamous cell carcinoma

BACKGROUND

Predictive nomograms are useful tools to guide clinicians in estimating disease course. Oral squamous cell carcinoma (OSCC) patients would benefit from an interactive prediction calculator that defines their levels of survival-risk specific to their tumors to guide the use of postoperative radiotherapy (PORT).

METHODS

Patients with OSCC surgically treated with curative intent at four Head and Neck Cancer Centres were recruited retrospectively for development and validation of nomograms. Predictor variables include PORT, age, T and N classification, surgical margins, perineural invasion, and lymphovascular invasion. Outcomes were disease-free, disease-specific, and overall survivals over 5 years.

RESULTS

1296 patients with OSCC were in training cohort for nomogram analysis. Algorithms were developed to show relative benefit of PORT in survivals for higher-risk patients. External validation on 1212 patients found the nomogram to be robust with favorable discrimination and calibration.

CONCLUSION

The proposed calculator can assist clinicians and patients in the decision-making process for PORT.

Diabetic retinopathy progression in patients under monitoring for treatment or vision loss: external validation and update of a multivariable prediction model

INTRODUCTION

The number of people with diabetes mellitus is increasing globally and consequently so too is diabetic retinopathy (DR). Most patients with diabetes are monitored through the diabetic eye screening programme (DESP) until they have signs of retinopathy and these changes progress, requiring referral into hospital eye services (HES). Here, they continue to be monitored until they require treatment. Due to current pressures on HES, delays can occur, leading to harm. There is a need to triage patients based on their individual risk. At present, patients are stratified according to retinopathy stage alone, yet other risk factors like glycated haemoglobin (HbA1c) may be useful. Therefore, a prediction model that combines multiple prognostic factors to predict progression will be useful for triage in this setting to improve care.We previously developed a Diabetic Retinopathy Progression model to Treatment or Vision Loss (DRPTVL-UK) using a large primary care database. The aim of the present study is to externally validate the DRPTVL-UK model in a secondary care setting, specifically in a population under care by HES. This study will also provide an opportunity to update the model by considering additional predictors not previously available.

METHODS AND ANALYSIS

We will use a retrospective cohort of 2400 patients with diabetes aged 12 years and over, referred from DESP to the NHS hospital trusts with referable DR between 2013 and 2016, with follow-up information recorded until December 2021.We will evaluate the external validity of the DRPTVL-UK model using measures of discrimination, calibration and net benefit. In addition, consensus meetings will be held to agree on acceptable risk thresholds for triage within the HES system.

ETHICS AND DISSEMINATION

This study was approved by REC (ref 22/SC/0425, 05/12/2022, Hampshire A Research Ethics Committee). The results of the study will be published in a peer-reviewed journal, presented at clinical conferences.

TRIAL REGISTRATION NUMBER

ISRCTN 10956293.

Impact of localized fine tuning in the performance of segmentation and classification of lung nodules from computed tomography scans using deep learning

Background

Algorithm malfunction may occur when there is a performance mismatch between the dataset with which it was developed and the dataset on which it was deployed.

Methods

A baseline segmentation algorithm and a baseline classification algorithm were developed using public dataset of Lung Image Database Consortium to detect benign and malignant nodules, and two additional external datasets (i.e., HB and XZ) including 542 cases and 486 cases were involved for the independent validation of these two algorithms. To explore the impact of localized fine tuning on the individual segmentation and classification process, the baseline algorithms were fine tuned with CT scans of HB and XZ datasets, respectively, and the performance of the fine tuned algorithms was tested to compare with the baseline algorithms.

Results

The proposed baseline algorithms of both segmentation and classification experienced a drop when directly deployed in external HB and XZ datasets. Comparing with the baseline validation results in nodule segmentation, the fine tuned segmentation algorithm obtained better performance in Dice coefficient, Intersection over Union, and Average Surface Distance in HB dataset (0.593 vs. 0.444; 0.450 vs. 0.348; 0.283 vs. 0.304) and XZ dataset (0.601 vs. 0.486; 0.482 vs. 0.378; 0.225 vs. 0.358). Similarly, comparing with the baseline validation results in benign and malignant nodule classification, the fine tuned classification algorithm had improved area under the receiver operating characteristic curve value, accuracy, and F1 score in HB dataset (0.851 vs. 0.812; 0.813 vs. 0.769; 0.852 vs. 0.822) and XZ dataset (0.724 vs. 0.668; 0.696 vs. 0.617; 0.737 vs. 0.668).

Conclusions

The external validation performance of localized fine tuned algorithms outperformed the baseline algorithms in both segmentation process and classification process, which showed that localized fine tuning may be an effective way to enable a baseline algorithm generalize to site-specific use.

Quantitative prediction error analysis to investigate predictive performance under predictor measurement heterogeneity at model implementation

BACKGROUND

When a predictor variable is measured in similar ways at the derivation and validation setting of a prognostic prediction model, yet both differ from the intended use of the model in practice (i.e., "predictor measurement heterogeneity"), performance of the model at implementation needs to be inferred. This study proposed an analysis to quantify the impact of anticipated predictor measurement heterogeneity.

METHODS

A simulation study was conducted to assess the impact of predictor measurement heterogeneity across validation and implementation setting in time-to-event outcome data. The use of the quantitative prediction error analysis was illustrated using an example of predicting the 6-year risk of developing type 2 diabetes with heterogeneity in measurement of the predictor body mass index.

RESULTS

In the simulation study, calibration-in-the-large of prediction models was poor and overall accuracy was reduced in all scenarios of predictor measurement heterogeneity. Model discrimination decreased with increasing random predictor measurement heterogeneity.

CONCLUSIONS

Heterogeneity of predictor measurements across settings of validation and implementation reduced predictive performance at implementation of prognostic models with a time-to-event outcome. When validating a prognostic model, the targeted clinical setting needs to be considered and analyses can be conducted to quantify the impact of anticipated predictor measurement heterogeneity on model performance at implementation.

Prognostic models for mortality after cardiac surgery in patients with infective endocarditis: a systematic review and aggregation of prediction models

BACKGROUND

There are several prognostic models to estimate the risk of mortality after surgery for active infective endocarditis (IE). However, these models incorporate different predictors and their performance is uncertain.

OBJECTIVE

We systematically reviewed and critically appraised all available prediction models of postoperative mortality in patients undergoing surgery for IE, and aggregated them into a meta-model.

DATA SOURCES

We searched Medline and EMBASE databases from inception to June 2020.

STUDY ELIGIBILITY CRITERIA

We included studies that developed or updated a prognostic model of postoperative mortality in patient with IE.

METHODS

We assessed the risk of bias of the models using PROBAST (Prediction model Risk Of Bias ASsessment Tool) and we aggregated them into an aggregate meta-model based on stacked regressions and optimized it for a nationwide registry of IE patients. The meta-model performance was assessed using bootstrap validation methods and adjusted for optimism.

RESULTS

We identified 11 prognostic models for postoperative mortality. Eight models had a high risk of bias. The meta-model included weighted predictors from the remaining three models (EndoSCORE, specific ES-I and specific ES-II), which were not rated as high risk of bias and provided full model equations. Additionally, two variables (age and infectious agent) that had been modelled differently across studies, were estimated based on the nationwide registry. The performance of the meta-model was better than the original three models, with the corresponding performance measures: C-statistics 0.79 (95% CI 0.76-0.82), calibration slope 0.98 (95% CI 0.86-1.13) and calibration-in-the-large -0.05 (95% CI -0.20 to 0.11).

CONCLUSIONS

The meta-model outperformed published models and showed a robust predictive capacity for predicting the individualized risk of postoperative mortality in patients with IE.

PROTOCOL REGISTRATION

PROSPERO (registration number CRD42020192602).