Calibration: the Achilles heel of predictive analytics

Machine learning models for predicting blood pressure phenotypes by combining multiple polygenic risk scores

We construct non-linear machine learning (ML) prediction models for systolic and diastolic blood pressure (SBP, DBP) using demographic and clinical variables and polygenic risk scores (PRSs). We developed a two-model ensemble, consisting of a baseline model, where prediction is based on demographic and clinical variables only, and a genetic model, where we also include PRSs. We evaluate the use of a linear versus a non-linear model at both the baseline and the genetic model levels and assess the improvement in performance when incorporating multiple PRSs. We report the ensemble model's performance as percentage variance explained (PVE) on a held-out test dataset. A non-linear baseline model improved the PVEs from 28.1 to 30.1% (SBP) and 14.3% to 17.4% (DBP) compared with a linear baseline model. Including seven PRSs in the genetic model computed based on the largest available GWAS of SBP/DBP improved the genetic model PVE from 4.8 to 5.1% (SBP) and 4.7 to 5% (DBP) compared to using a single PRS. Adding additional 14 PRSs computed based on two independent GWASs further increased the genetic model PVE to 6.3% (SBP) and 5.7% (DBP). PVE differed across self-reported race/ethnicity groups, with primarily all non-White groups benefitting from the inclusion of additional PRSs. In summary, non-linear ML models improves BP prediction in models incorporating diverse populations.

EEG connectivity and network analyses predict outcome in patients with disorders of consciousness - A systematic review and meta-analysis

Outcome prediction in prolonged disorders of consciousness (DOC) remains challenging. This can result in either inappropriate withdrawal of treatment or unnecessary prolongation of treatment. Electroencephalography (EEG) is a cheap, portable, and non-invasive device with various opportunities for complex signal analysis. Computational EEG measures, such as EEG connectivity and network metrics, might be ideal candidates for the investigation of DOC, but their capacity in prognostication is still undisclosed. We conducted a meta-analysis aiming to compare the prognostic power of the widely used clinical scale, Coma Recovery Scale-Revised - CRS-R and EEG connectivity and network metrics. We found that the prognostic power of the CRS-R scale was moderate (AUC: 0.67 (0.60-0.75)), but EEG connectivity and network metrics predicted outcome with significantly (p = 0.0071) higher accuracy (AUC:0.78 (0.70-0.86)). We also estimated the prognostic capacity of EEG spectral power, which was not significantly (p = 0.3943) inferior to that of the EEG connectivity and graph-theory measures (AUC:0.75 (0.70-0.80)). Multivariate automated outcome prediction tools seemed to outperform clinical and EEG markers.

Development and validation of a SCORing systEm for pre-thrombectomy diagnosis of IntraCranial Atherosclerotic Disease (Score-ICAD)

BACKGROUND

Early identification of intracranial atherosclerotic disease (ICAD) may impact the management of patients undergoing mechanical thrombectomy (MT). We sought to develop and validate a scoring system for pre-thrombectomy diagnosis of ICAD in anterior circulation large vessel/distal medium vessel occlusion strokes (LVOs/DMVOs).

METHODS

Retrospective analysis of two prospectively maintained comprehensive stroke center databases including patients with anterior circulation occlusions spanning 2010-22 (development cohort) and 2018-22 (validation cohort). ICAD cases were matched for age and sex (1:1) to non-ICAD controls.

RESULTS

Of 2870 MTs within the study period, 348 patients were included in the development cohort: 174 anterior circulation ICAD (6% of 2870 MTs) and 174 controls. Multivariable analysis β coefficients led to a 20 point scale: absence of atrial fibrillation (5); vascular risk factor burden (1) for each of hypertension, diabetes, smoking, and hyperlipidemia; multifocal single artery stenoses on CT angiography (3); absence of territorial cortical infarct (3); presence of borderzone infarct (3); or ipsilateral carotid siphon calcification (2). The validation cohort comprised 56 ICAD patients (4.1% of 1359 MTs): 56 controls. Area under the receiver operating characteristic curve was 0.88 (0.84-0.91) and 0.82 (0.73-0.89) in the development and validation cohorts, respectively. Calibration slope and intercept showed a good fit for the development cohort although with overestimated risk for the validation cohort. After intercept adjustment, the overestimation was corrected (intercept 0, 95% CI -0.5 to -0.5; slope 0.8, 95% CI 0.5 to 1.1). In the full cohort (n=414), ≥11 points showed the best performance for distinguishing ICAD from non-ICAD, with 0.71 (95% CI 0.65 to 0.78) sensitivity and 0.82 (95% CI 0.77 to 0.87) specificity, and 3.92 (95% CI 2.92 to 5.28) positive and 0.35 (95% CI 0.28 to 0.44) negative likelihood ratio. Scores ≥12 showed 90% specificity and 63% sensitivity.

CONCLUSION

The proposed scoring system for preprocedural diagnosis of ICAD LVOs and DMVOs presented satisfactory discrimination and calibration based on clinical and non-invasive radiological data.

Machine learning models on a web application to predict short-term postoperative outcomes following anterior cervical discectomy and fusion

BACKGROUND

The frequency of anterior cervical discectomy and fusion (ACDF) has increased up to 400% since 2011, underscoring the need to preoperatively anticipate adverse postoperative outcomes given the procedure's expanding use. Our study aims to accomplish two goals: firstly, to develop a suite of explainable machine learning (ML) models capable of predicting adverse postoperative outcomes following ACDF surgery, and secondly, to embed these models in a user-friendly web application, demonstrating their potential utility.

METHODS

We utilized data from the National Surgical Quality Improvement Program database to identify patients who underwent ACDF surgery. The outcomes of interest were four short-term postoperative adverse outcomes: prolonged length of stay (LOS), non-home discharges, 30-day readmissions, and major complications. We utilized five ML algorithms - TabPFN, TabNET, XGBoost, LightGBM, and Random Forest - coupled with the Optuna optimization library for hyperparameter tuning. To bolster the interpretability of our models, we employed SHapley Additive exPlanations (SHAP) for evaluating predictor variables' relative importance and used partial dependence plots to illustrate the impact of individual variables on the predictions generated by our top-performing models. We visualized model performance using receiver operating characteristic (ROC) curves and precision-recall curves (PRC). Quantitative metrics calculated were the area under the ROC curve (AUROC), balanced accuracy, weighted area under the PRC (AUPRC), weighted precision, and weighted recall. Models with the highest AUROC values were selected for inclusion in a web application.

RESULTS

The analysis included 57,760 patients for prolonged LOS [11.1% with prolonged LOS], 57,780 for non-home discharges [3.3% non-home discharges], 57,790 for 30-day readmissions [2.9% readmitted], and 57,800 for major complications [1.4% with major complications]. The top-performing models, which were the ones built with the Random Forest algorithm, yielded mean AUROCs of 0.776, 0.846, 0.775, and 0.747 for predicting prolonged LOS, non-home discharges, readmissions, and complications, respectively.

CONCLUSIONS

Our study employs advanced ML methodologies to enhance the prediction of adverse postoperative outcomes following ACDF. We designed an accessible web application to integrate these models into clinical practice. Our findings affirm that ML tools serve as vital supplements in risk stratification, facilitating the prediction of diverse outcomes and enhancing patient counseling for ACDF.

Generalizability of polygenic prediction models: how is the R2 defined on test data?

BACKGROUND

Polygenic risk scores (PRS) quantify an individual's genetic predisposition for different traits and are expected to play an increasingly important role in personalized medicine. A crucial challenge in clinical practice is the generalizability and transferability of PRS models to populations with different ancestries. When assessing the generalizability of PRS models for continuous traits, the R 2 is a commonly used measure to evaluate prediction accuracy. While the R 2 is a well-defined goodness-of-fit measure for statistical linear models, there exist different definitions for its application on test data, which complicates interpretation and comparison of results.

METHODS

Based on large-scale genotype data from the UK Biobank, we compare three definitions of the R 2 on test data for evaluating the generalizability of PRS models to different populations. Polygenic models for several phenotypes, including height, BMI and lipoprotein A, are derived based on training data with European ancestry using state-of-the-art regression methods and are evaluated on various test populations with different ancestries.

RESULTS

Our analysis shows that the choice of the R 2  definition can lead to considerably different results on test data, making the comparison of R 2  values from the literature problematic. While the definition as the squared correlation between predicted and observed phenotypes solely addresses the discriminative performance and always yields values between 0 and 1, definitions of the R 2 based on the mean squared prediction error (MSPE) with reference to intercept-only models assess both discrimination and calibration. These MSPE-based definitions can yield negative values indicating miscalibrated predictions for out-of-target populations. We argue that the choice of the most appropriate definition depends on the aim of PRS analysis - whether it primarily serves for risk stratification or also for individual phenotype prediction. Moreover, both correlation-based and MSPE-based definitions of R 2 can provide valuable complementary information.

CONCLUSIONS

Awareness of the different definitions of the R 2 on test data is necessary to facilitate the reporting and interpretation of results on PRS generalizability. It is recommended to explicitly state which definition was used when reporting R 2 values on test data. Further research is warranted to develop and evaluate well-calibrated polygenic models for diverse populations.

Machine Learning to Allocate Palliative Care Consultations During Cancer Treatment

PURPOSE

For patients with advanced cancer, early consultations with palliative care (PC) specialists reduce costs, improve quality of life, and prolong survival. However, capacity limitations prevent all patients from receiving PC shortly after diagnosis. We evaluated whether a prognostic machine learning system could promote early PC, given existing capacity.

METHODS

Using population-level administrative data in Ontario, Canada, we assembled a cohort of patients with incurable cancer who received palliative-intent systemic therapy between July 1, 2014, and December 30, 2019. We developed a machine learning system that predicted death within 1 year of each treatment using demographics, cancer characteristics, treatments, symptoms, laboratory values, and history of acute care admissions. We trained the system in patients who started treatment before July 1, 2017, and evaluated the potential impact of the system on PC in subsequent patients.

RESULTS

Among 560,210 treatments received by 54,628 patients, death occurred within 1 year of 45.2% of treatments. The machine learning system recommended the same number of PC consultations observed with usual care at the 60.0% 1-year risk of death, with a first-alarm positive predictive value of 69.7% and an outcome-level sensitivity of 74.9%. Compared with usual care, system-guided care could increase early PC by 8.5% overall (95% CI, 7.5 to 9.5; P < .001) and by 15.3% (95% CI, 13.9 to 16.6; P < .001) among patients who live 6 months beyond their first treatment, without requiring more PC consultations in total or substantially increasing PC among patients with a prognosis exceeding 2 years.

CONCLUSION

Prognostic machine learning systems could increase early PC despite existing resource constraints. These results demonstrate an urgent need to deploy and evaluate prognostic systems in real-time clinical practice to increase access to early PC.

Validation of the European Society of Cardiology pretest probability models for obstructive coronary artery disease in high-risk population

OBJECTIVE

The pre-test probability (PTP) model for obstructive coronary artery disease (CAD) was updated in 2019 by the European Society of Cardiology (ESC). To our knowledge, this model was never externally validated in a population with a high incidence of CAD. The aim of this study is to validate the new PTP ESC model in our population, which has a high CAD incidence, and to compare it with the previous PTP ESC model from 2013.

METHODS

We retrospectively analysed 1294 symptomatic patients with suspected CAD referred to our centre between 2015 and 2019. In all patients, the PTP score was calculated based on age, gender, and symptoms according to the ESC model from 2013 (2013-ESC-PTP) and 2019 (2019-ESC-PTP). All patients underwent invasive coronary angiography (ICA).

RESULTS

Of the 1294 patients, obstructive CAD was diagnosed in 533 patients (41.2%). The 2019-ESC-PTP model categorised significantly more patients into the low probability group (PTP < 15%) than the 2013-ESC-PTP model (39.8% vs. 5.6%, p < 0.001). Obstructive CAD prevalence was underestimated using 2019-ESC-PTP at all PTP levels (calibration intercept 1.15, calibration slope 0.96). The 2013-ESC-PTP overestimated obstructive CAD prevalence (calibration intercept -0.24, calibration slope 0.73). The discrimination measured with an area under the curve was similar for both models, indicating moderate accuracy of the models.

CONCLUSION

In high-risk Serbian population, both the 2013 and 2019 ESC-PTP models had moderate accuracy in diagnosing CAD, with the 2019-ESC-PTP underestimating the prevalence of CAD and the 2013-ESC-PTP overestimating it. Further studies are warranted to establish PTP models for high-risk countries.

External Validation of Updated Prediction Models for Neurological Outcomes at 90 Days in Patients With Out-of-Hospital Cardiac Arrest

BACKGROUND

Few prediction models for individuals with early-stage out-of-hospital cardiac arrest (OHCA) have undergone external validation. This study aimed to externally validate updated prediction models for OHCA outcomes using a large nationwide dataset.

METHODS AND RESULTS

We performed a secondary analysis of the JAAM-OHCA (Comprehensive Registry of In-Hospital Intensive Care for Out-of-Hospital Cardiac Arrest Survival and the Japanese Association for Acute Medicine Out-of-Hospital Cardiac Arrest) registry. Previously developed prediction models for patients with cardiac arrest who achieved the return of spontaneous circulation were updated. External validation was conducted using data from 56 institutions from the JAAM-OHCA registry. The primary outcome was a dichotomized 90-day cerebral performance category score. Two models were updated using the derivation set (n=3337). Model 1 included patient demographics, prehospital information, and the initial rhythm upon hospital admission; Model 2 included information obtained in the hospital immediately after the return of spontaneous circulation. In the validation set (n=4250), Models 1 and 2 exhibited a C-statistic of 0.945 (95% CI, 0.935-0.955) and 0.958 (95% CI, 0.951-0.960), respectively. Both models were well-calibrated to the observed outcomes. The decision curve analysis showed that Model 2 demonstrated higher net benefits at all risk thresholds than Model 1. A web-based calculator was developed to estimate the probability of poor outcomes (https://pcas-prediction.shinyapps.io/90d_lasso/).

CONCLUSIONS

The updated models offer valuable information to medical professionals in the prediction of long-term neurological outcomes for patients with OHCA, potentially playing a vital role in clinical decision-making processes.

Developing and validating prediction models for severe exacerbations and readmissions in patients hospitalised for COPD exacerbation (SERCO) in China: a prospective observational study

BACKGROUND

There is a lack of individualised prediction models for patients hospitalised with chronic obstructive pulmonary disease (COPD) for clinical practice. We developed and validated prediction models of severe exacerbations and readmissions in patients hospitalised for COPD exacerbation (SERCO).

METHODS

Data were obtained from the Acute Exacerbations of Chronic Obstructive Pulmonary Disease Inpatient Registry study (NCT02657525) in China. Cause-specific hazard models were used to estimate coefficients. C-statistic was used to evaluate the discrimination. Slope and intercept were used to evaluate the calibration and used for model adjustment. Models were validated internally by 10-fold cross-validation and externally using data from different regions. Risk-stratified scoring scales and nomograms were provided. The discrimination ability of the SERCO model was compared with the exacerbation history in the previous year.

RESULTS

Two sets with 2196 and 1869 patients from different geographical regions were used for model development and external validation. The 12-month severe exacerbations cumulative incidence rates were 11.55% (95% CI 10.06% to 13.16%) in development cohorts and 12.30% (95% CI 10.67% to 14.05%) in validation cohorts. The COPD-specific readmission incidence rates were 11.31% (95% CI 9.83% to 12.91%) and 12.26% (95% CI 10.63% to 14.02%), respectively. Demographic characteristics, medical history, comorbidities, drug usage, Global Initiative for Chronic Obstructive Lung Disease stage and interactions were included as predictors. C-indexes for severe exacerbations were 77.3 (95% CI 70.7 to 83.9), 76.5 (95% CI 72.6 to 80.4) and 74.7 (95% CI 71.2 to 78.2) at 1, 6 and 12 months. The corresponding values for readmissions were 77.1 (95% CI 70.1 to 84.0), 76.3 (95% CI 72.3 to 80.4) and 74.5 (95% CI 71.0 to 78.0). The SERCO model was consistently discriminative and accurate with C-indexes in the derivation and internal validation groups. In external validation, the C-indexes were relatively lower at 60-70 levels. The SERCO model discriminated outcomes better than prior severe exacerbation history. The slope and intercept after adjustment showed close agreement between predicted and observed risks. However, in external validation, the models may overestimate the risk in higher-risk groups. The model-driven risk groups showed significant disparities in prognosis.

CONCLUSION

The SERCO model provides individual predictions for severe exacerbation and COPD-specific readmission risk, which enables identifying high-risk patients and implementing personalised preventive intervention for patients with COPD.

Biomarkers to improve functional outcome prediction after ischemic stroke: Results from the SICFAIL, STRAWINSKI, and PREDICT studies

BACKGROUND AND AIMS

Acute ischemic stroke (AIS) outcome prognostication remains challenging despite available prognostic models. We investigated whether a biomarker panel improves the predictive performance of established prognostic scores.

METHODS

We investigated the improvement in discrimination, calibration, and overall performance by adding five biomarkers (procalcitonin, copeptin, cortisol, mid-regional pro-atrial natriuretic peptide (MR-proANP), and N-terminal pro-B-type natriuretic peptide (NT-proBNP)) to the Acute Stroke Registry and Analysis of Lausanne (ASTRAL) and age/NIHSS scores using data from two prospective cohort studies (SICFAIL, PREDICT) and one clinical trial (STRAWINSKI). Poor outcome was defined as mRS > 2 at 12 (SICFAIL, derivation dataset) or 3 months (PREDICT/STRAWINSKI, pooled external validation dataset).

RESULTS

Among 412 SICFAIL participants (median age 70 years, quartiles 59-78; 63% male; median NIHSS score 3, quartiles 1-5), 29% had a poor outcome. Area under the curve of the ASTRAL and age/NIHSS were 0.76 (95% CI 0.71-0.81) and 0.77 (95% CI 0.73-0.82), respectively. Copeptin (0.79, 95% CI 0.74-0.84), NT-proBNP (0.80, 95% CI 0.76-0.84), and MR-proANP (0.79, 95% CI 0.75-0.84) significantly improved ASTRAL score's discrimination, calibration, and overall performance. Copeptin improved age/NIHSS model's discrimination, copeptin, MR-proANP, and NT-proBNP improved its calibration and overall performance. In the validation dataset (450 patients, median age 73 years, quartiles 66-81; 54% men; median NIHSS score 8, quartiles 3-14), copeptin was independently associated with various definitions of poor outcome and also mortality. Copeptin did not increase model's discrimination but it did improve calibration and overall model performance.

DISCUSSION

Copeptin, NT-proBNP, and MR-proANP improved modest but consistently the predictive performance of established prognostic scores in patients with mild AIS. Copeptin was most consistently associated with poor outcome in patients with moderate to severe AIS, although its added prognostic value was less obvious.

Reliable prediction of difficult airway for tracheal intubation from patient preoperative photographs by machine learning methods

BACKGROUND

Estimating the risk of a difficult tracheal intubation should help clinicians in better anaesthesia planning, to maximize patient safety. Routine bedside screenings suffer from low sensitivity.

OBJECTIVE

To develop and evaluate machine learning (ML) and deep learning (DL) algorithms for the reliable prediction of intubation risk, using information about airway morphology.

METHODS

Observational, prospective cohort study enrolling n=623 patients who underwent tracheal intubation: 53/623 difficult cases (prevalence 8.51%). First, we used our previously validated deep convolutional neural network (DCNN) to extract 2D image coordinates for 27 + 13 relevant anatomical landmarks in two preoperative photos (frontal and lateral views). Here we propose a method to determine the 3D pose of the camera with respect to the patient and to obtain the 3D world coordinates of these landmarks. Then we compute a novel set of dM=59 morphological features (distances, areas, angles and ratios), engineered with our anaesthesiologists to characterize each individual's airway anatomy towards prediction. Subsequently, here we propose four ad hoc ML pipelines for difficult intubation prognosis, each with four stages: feature scaling, imputation, resampling for imbalanced learning, and binary classification (Logistic Regression, Support Vector Machines, Random Forests and eXtreme Gradient Boosting). These compound ML pipelines were fed with the dM=59 morphological features, alongside dD=7 demographic variables. Here we trained them with automatic hyperparameter tuning (Bayesian search) and probability calibration (Platt scaling). In addition, we developed an ad hoc multi-input DCNN to estimate the intubation risk directly from each pair of photographs, i.e. without any intermediate morphological description. Performance was evaluated using optimal Bayesian decision theory. It was compared against experts' judgement and against state-of-the-art methods (three clinical formulae, four ML, four DL models).

RESULTS

Our four ad hoc ML pipelines with engineered morphological features achieved similar discrimination capabilities: median AUCs between 0.746 and 0.766. They significantly outperformed both expert judgement and all state-of-the-art methods (highest AUC at 0.716). Conversely, our multi-input DCNN yielded low performance due to overfitting. This same behaviour occurred for the state-of-the-art DL algorithms. Overall, the best method was our XGB pipeline, with the fewest false negatives at the optimal Bayesian decision threshold.

CONCLUSIONS

We proposed and validated ML models to assist clinicians in anaesthesia planning, providing a reliable calibrated estimate of airway intubation risk, which outperformed expert assessments and state-of-the-art methods. Our novel set of engineered features succeeded in providing informative descriptions for prognosis.

Harnessing multi-source data for individualized care in Hodgkin Lymphoma

Hodgkin lymphoma is a rare, but highly curative form of cancer, primarily afflicting adolescents and young adults. Despite multiple seminal trials over the past twenty years, there is no single consensus-based treatment approach beyond use of multi-agency chemotherapy with curative intent. The use of radiation continues to be debated in early-stage disease, as part of combined modality treatment, as well as in salvage, as an important form of consolidation. While short-term disease outcomes have varied little across these different approaches across both early and advanced stage disease, the potential risk of severe, longer-term risk has varied considerably. Over the past decade novel therapeutics have been employed in the retrieval setting in preparation to and as consolidation after autologous stem cell transplant. More recently, these novel therapeutics have moved to the frontline setting, initially compared to standard-of-care treatment and later in a direct head-to-head comparison combined with multi-agent chemotherapy. In 2018, we established the HoLISTIC Consortium, bringing together disease and methods experts to develop clinical decision models based on individual patient data to guide providers, patients, and caregivers in decision-making. In this review, we detail the steps we followed to create the master database of individual patient data from patients treated over the past 20 years, using principles of data science. We then describe different methodological approaches we are taking to clinical decision making, beginning with clinical prediction tools at the time of diagnosis, to multi-state models, incorporating treatments and their response. Finally, we describe how simulation modeling can be used to estimate risks of late effects, based on cumulative exposure from frontline and salvage treatment. The resultant database and tools employed are dynamic with the expectation that they will be updated as better and more complete information becomes available.

Applying models of care for total hip and knee arthroplasty: External validation of a published predictive model to identify extended stay risk prior to lower-limb arthroplasty

OBJECTIVE

This study aimed to externally validate a reported model for identifying patients requiring extended stay following lower limb arthroplasty in a new setting.

DESIGN

External validation of a previously reported prognostic model, using retrospective data.

SETTING

Medium-sized hospital orthopaedic department, Australia.

PARTICIPANTS

Electronic medical records were accessed for data collection between Sep-2019 and Feb-2020 and retrospective data extracted from 200 randomly selected total hip or knee arthroplasty patients.

INTERVENTION

Participants received total hip or knee replacement between 2-Feb-16 and 4-Apr-19. This study was a non-interventional retrospective study.

MAIN MEASURES

Model validation was assessed with discrimination, calibration on both original and adjusted forms of the candidate model. Decision curve analysis was conducted on the outputs of the adjusted model to determine net benefit at a predetermined decision threshold (0.5).

RESULTS

The original model performed poorly, grossly overestimating length of stay with mean calibration of -3.6 (95% confidence interval -3.9 to -3.2) and calibration slope of 0.52. Performance improved following adjustment of the model intercept and model coefficients (mean calibration 0.48, 95% confidence interval 0.16 to 0.80 and slope of 1.0), but remained poorly calibrated at low and medium risk threshold and net benefit was modest (three additional patients per hundred identified as at-risk) at the a-priori risk threshold.

CONCLUSIONS

External validation demonstrated poor performance when applied to a new patient population and would provide limited benefit for our institution. Implementation of predictive models for arthroplasty should include practical assessment of discrimination, calibration and net benefit at a clinically acceptable threshold.

Patients' and physicians' beliefs and attitudes towards integrating personalized risk estimates into patient education about left ventricular assist device therapy

BACKGROUND

Personalized risk (PR) estimates may enhance clinical decision making and risk communication by providing individualized estimates of patient outcomes. We explored stakeholder attitudes toward the utility, acceptability, usefulness and best-practices for integrating PR estimates into patient education and decision making about Left Ventricular Assist Device (LVAD).

METHODS AND RESULTS

As part of a 5-year multi-institutional AHRQ project, we conducted 40 interviews with stakeholders (physicians, nurse coordinators, patients, and caregivers), analyzed using Thematic Content Analysis. All stakeholder groups voiced positive views towards integrating PR in decision making. Patients, caregivers and coordinators emphasized that PR can help to better understand a patient's condition and risks, prepare mentally and logistically for likely outcomes, and meaningfully engage in decision making. Physicians felt it can improve their decision making by enhancing insight into outcomes, enhance tailored pre-emptive care, increase confidence in decisions, and reduce bias and subjectivity. All stakeholder groups also raised concerns about accuracy, representativeness and relevance of algorithms; predictive uncertainty; utility in relation to physician's expertise; potential negative reactions among patients; and overreliance.

CONCLUSION

Stakeholders are optimistic about integrating PR into clinical decision making, but acceptability depends on prospectively demonstrating accuracy, relevance and evidence that benefits of PR outweigh potential negative impacts on decision making quality.

Effect of recipient age on prioritisation for liver transplantation in the UK: a population-based modelling study

BACKGROUND

Following the introduction of an algorithm aiming to maximise life-years gained from liver transplantation in the UK (the transplant benefit score [TBS]), donor livers were redirected from younger to older patients, mortality rate equalised across the age range and short-term waiting list mortality reduced. Understanding age-related prioritisation has been challenging, especially for younger patients and clinicians allocating non-TBS-directed livers. We aimed to assess age-related prioritisation within the TBS algorithm by modelling liver transplantation prioritisation based on data from a UK transplant unit and comparing these data with other regions.

METHODS

In this population-based modelling study, serum parameters and age at liver transplantation assessment of patients attending the Scottish Liver Transplant Unit, Edinburgh, UK, between December, 2002, and November, 2023, were combined with representative synthetic data to model TBS survival predictions, which were compared according to age group (25-49 years vs ≥60 years), chronic liver disease severity, and disease cause. Models for end-stage liver disease (UKELD [UK], MELD [Eurotransplant region], and MELD 3.0 [USA]) were used as validated comparators of liver disease severity.

FINDINGS

Of 2093 patients with chronic liver disease, 1808 (86%) had complete datasets and liver disease parameters consistent with eligibility for the liver transplant waiting list in the UK (UKELD ≥49). Disease severity as assessed by UKELD, MELD, and MELD 3.0 did not differ by age (median UKELD scores of 56 for patients aged ≥60 years vs 56 for patients aged 25-49 years; MELD scores of 16 vs 16; and MELD 3.0 scores of 18 vs 18). TBS increased with advancing age (R=0·45, p<0·0001). TBS predicted that transplantation in patients aged 60 years or older would provide a two-fold greater net benefit at 5 years than in patients aged 25-49 years (median TBS 1317 [IQR 1116-1436] in older patients vs 706 [411-1095] in younger patients; p<0·0001). Older patients were predicted to have shorter survival without transplantation than younger patients (263 days [IQR 144-473] in older patients vs 861 days [448-1164] in younger patients; p<0·0001) but similar survival after transplantation (1599 days [1563-1628] vs 1573 days [1525-1614]; p<0·0001). Older patients could reach a TBS for which a liver offer was likely below minimum criteria for transplantation (UKELD <49), whereas many younger patients were required to have high-urgent disease (UKELD >60). US and Eurotransplant programmes did not prioritise according to age.

INTERPRETATION

The UK liver allocation algorithm prioritises older patients for transplantation by predicting that advancing age increases the benefit from liver transplantation. Restricted follow-up and biases in waiting list data might limit the accuracy of these benefit predictions. Measures beyond overall waiting list mortality are required to fully capture the benefits of liver transplantation.

FUNDING

None.

A real-time prognostic model for venous thromboembolic events among hospitalized adults

Background

Hospital-acquired venous thromboembolism (HA-VTE) is a leading cause of morbidity and mortality among hospitalized adults. Guidelines recommend use of a risk-prediction model to estimate HA-VTE risk for individual patients. Extant models do not perform well for broad patient populations and are not conducive to automation in clinical practice.

Objectives

To develop an automated, real-time prognostic model for venous thromboembolism during hospitalization among all adult inpatients using readily available data from the electronic health record.

Methods

The derivation cohort included inpatient hospitalizations ("encounters") for patients ≥16 years old at Vanderbilt University Medical Center between 2018 and 2020 (n = 132,330). HA-VTE events were identified using International Classification of Diseases, 10th Revision, codes. The prognostic model was developed using least absolute shrinkage and selection operator regression. Temporal external validation was performed in a validation cohort of encounters between 2021 and 2022 (n = 62,546). Prediction performance was assessed by discrimination accuracy (C statistic) and calibration (integrated calibration index).

Results

There were 1187 HA-VTEs in the derivation cohort (9.0 per 1000 encounters) and 864 in the validation cohort (13.8 per 1000 encounters). The prognostic model included 25 variables, with placement of a central line among the most important predictors. Prediction performance of the model was excellent (C statistic, 0.891; 95% CI, 0.882-0.900; integrated calibration index, 0.001). The model performed similarly well across subgroups of patients defined by age, sex, race, and type of admission.

Conclusion

This fully automated prognostic model uses readily available data from the electronic health record, exhibits superior prediction performance compared with existing models, and generates granular risk stratification in the form of a predicted probability of HA-VTE for each patient.

Development and validation of risk prediction model for recurrent cardiovascular events among Chinese: the Personalized CARdiovascular DIsease risk Assessment for Chinese model

Aims

Cardiovascular disease (CVD) is a leading cause of mortality, especially in developing countries. This study aimed to develop and validate a CVD risk prediction model, Personalized CARdiovascular DIsease risk Assessment for Chinese (P-CARDIAC), for recurrent cardiovascular events using machine learning technique.

Methods and results

Three cohorts of Chinese patients with established CVD were included if they had used any of the public healthcare services provided by the Hong Kong Hospital Authority (HA) since 2004 and categorized by their geographical locations. The 10-year CVD outcome was a composite of diagnostic or procedure codes with specific International Classification of Diseases, Ninth Revision, Clinical Modification. Multivariate imputation with chained equations and XGBoost were applied for the model development. The comparison with Thrombolysis in Myocardial Infarction Risk Score for Secondary Prevention (TRS-2°P) and Secondary Manifestations of ARTerial disease (SMART2) used the validation cohorts with 1000 bootstrap replicates. A total of 48 799, 119 672 and 140 533 patients were included in the derivation and validation cohorts, respectively. A list of 125 risk variables were used to make predictions on CVD risk, of which 8 classes of CVD-related drugs were considered interactive covariates. Model performance in the derivation cohort showed satisfying discrimination and calibration with a C statistic of 0.69. Internal validation showed good discrimination and calibration performance with C statistic over 0.6. The P-CARDIAC also showed better performance than TRS-2°P and SMART2.

Conclusion

Compared with other risk scores, the P-CARDIAC enables to identify unique patterns of Chinese patients with established CVD. We anticipate that the P-CARDIAC can be applied in various settings to prevent recurrent CVD events, thus reducing the related healthcare burden.

Tutorial: dos and don'ts in clinical prediction research for venous thromboembolism

Clinical prediction modeling has become an increasingly popular domain of venous thromboembolism research in recent years. Prediction models can help healthcare providers make decisions regarding starting or withholding therapeutic interventions, or referrals for further diagnostic workup, and can form a basis for risk stratification in clinical trials. The aim of the current guide is to assist in the practical application of complicated methodological requirements for well-performed prediction research by presenting key dos and don'ts while expanding the understanding of predictive research in general for (clinical) researchers who are not specifically trained in the topic; throughout we will use prognostic venous thromboembolism scores as an exemplar.

Predictive Models to Assess Risk of Persistent Opioid Use, Opioid Use Disorder, and Overdose

BACKGROUND

This systematic review summarizes the development, accuracy, quality, and clinical utility of predictive models to assess the risk of opioid use disorder (OUD), persistent opioid use, and opioid overdose.

METHODS

In accordance with Preferred Reporting Items for a Systematic Review and Meta-analysis guidelines, 8 electronic databases were searched for studies on predictive models and OUD, overdose, or persistent use in adults until June 25, 2023. Study selection and data extraction were completed independently by 2 reviewers. Risk of bias of included studies was assessed independently by 2 reviewers using the Prediction model Risk of Bias ASsessment Tool (PROBAST).

RESULTS

The literature search yielded 3130 reports; after removing 199 duplicates, excluding 2685 studies after abstract review, and excluding 204 studies after full-text review, the final sample consisted of 41 studies that developed more than 160 predictive models. Primary outcomes included opioid overdose (31.6% of studies), OUD (41.4%), and persistent opioid use (17%). The most common modeling approach was regression modeling, and the most common predictors included age, sex, mental health diagnosis history, and substance use disorder history. Most studies reported model performance via the c statistic, ranging from 0.507 to 0.959; gradient boosting tree models and neural network models performed well in the context of their own study. One study deployed a model in real time. Risk of bias was predominantly high; concerns regarding applicability were predominantly low.

CONCLUSIONS

Models to predict opioid-related risks are developed using diverse data sources and predictors, with a wide and heterogenous range of accuracy metrics. There is a need for further research to improve their accuracy and implementation.

Predicting short- to medium-term care home admission risk in older adults: a systematic review of externally validated models

INTRODUCTION

Predicting risk of care home admission could identify older adults for early intervention to support independent living but require external validation in a different dataset before clinical use. We systematically reviewed external validations of care home admission risk prediction models in older adults.

METHODS

We searched Medline, Embase and Cochrane Library until 14 August 2023 for external validations of prediction models for care home admission risk in adults aged ≥65 years with up to 3 years of follow-up. We extracted and narratively synthesised data on study design, model characteristics, and model discrimination and calibration (accuracy of predictions). We assessed risk of bias and applicability using Prediction model Risk Of Bias Assessment Tool.

RESULTS

Five studies reporting validations of nine unique models were included. Model applicability was fair but risk of bias was mostly high due to not reporting model calibration. Morbidities were used as predictors in four models, most commonly neurological or psychiatric diseases. Physical function was also included in four models. For 1-year prediction, three of the six models had acceptable discrimination (area under the receiver operating characteristic curve (AUC)/c statistic 0.70-0.79) and the remaining three had poor discrimination (AUC < 0.70). No model accounted for competing mortality risk. The only study examining model calibration (but ignoring competing mortality) concluded that it was excellent.

CONCLUSIONS

The reporting of models was incomplete. Model discrimination was at best acceptable, and calibration was rarely examined (and ignored competing mortality risk when examined). There is a need to derive better models that account for competing mortality risk and report calibration as well as discrimination.

Prediction of Morbidity and Mortality After Esophagectomy: A Systematic Review

BACKGROUND

Esophagectomy for esophageal cancer has a complication rate of up to 60%. Prediction models could be helpful to preoperatively estimate which patients are at increased risk of morbidity and mortality. The objective of this study was to determine the best prediction models for morbidity and mortality after esophagectomy and to identify commonalities among the models.

PATIENTS AND METHODS

A systematic review was performed in accordance to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement and was prospectively registered in PROSPERO ( https://www.crd.york.ac.uk/prospero/ , study ID CRD42022350846). Pubmed, Embase, and Clarivate Analytics/Web of Science Core Collection were searched for studies published between 2010 and August 2022. The Prediction model Risk of Bias Assessment Tool was used to assess the risk of bias. Extracted data were tabulated and a narrative synthesis was performed.

RESULTS

Of the 15,011 articles identified, 22 studies were included using data from tens of thousands of patients. This systematic review included 33 different models, of which 18 models were newly developed. Many studies showed a high risk of bias. The prognostic accuracy of models differed between 0.51 and 0.85. For most models, variables are readily available. Two models for mortality and one model for pulmonary complications have the potential to be developed further.

CONCLUSIONS

The availability of rigorous prediction models is limited. Several models are promising but need to be further developed. Some models provide information about risk factors for the development of complications. Performance status is a potential modifiable risk factor. None are ready for clinical implementation.

Development and External Validation of Models to Predict Persistent Hypoxemic Respiratory Failure for Clinical Trial Enrichment

OBJECTIVES

Improving the efficiency of clinical trials in acute hypoxemic respiratory failure (HRF) depends on enrichment strategies that minimize enrollment of patients who quickly resolve with existing care and focus on patients at high risk for persistent HRF. We aimed to develop parsimonious models predicting risk of persistent HRF using routine data from ICU admission and select research immune biomarkers.

DESIGN

Prospective cohorts for derivation ( n = 630) and external validation ( n = 511).

SETTING

Medical and surgical ICUs at two U.S. medical centers.

PATIENTS

Adults with acute HRF defined as new invasive mechanical ventilation (IMV) and hypoxemia on the first calendar day after ICU admission.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

We evaluated discrimination, calibration, and practical utility of models predicting persistent HRF risk (defined as ongoing IMV and hypoxemia on the third calendar day after admission): 1) a clinical model with least absolute shrinkage and selection operator (LASSO) selecting Pa o2 /F io2 , vasopressors, mean arterial pressure, bicarbonate, and acute respiratory distress syndrome as predictors; 2) a model adding interleukin-6 (IL-6) to clinical predictors; and 3) a comparator model with Pa o2 /F io2 alone, representing an existing strategy for enrichment. Forty-nine percent and 69% of patients had persistent HRF in derivation and validation sets, respectively. In validation, both LASSO (area under the receiver operating characteristic curve, 0.68; 95% CI, 0.64-0.73) and LASSO + IL-6 (0.71; 95% CI, 0.66-0.76) models had better discrimination than Pa o2 /F io2 (0.64; 95% CI, 0.59-0.69). Both models underestimated risk in lower risk deciles, but exhibited better calibration at relevant risk thresholds. Evaluating practical utility, both LASSO and LASSO + IL-6 models exhibited greater net benefit in decision curve analysis, and greater sample size savings in enrichment analysis, compared with Pa o2 /F io2 . The added utility of LASSO + IL-6 model over LASSO was modest.

CONCLUSIONS

Parsimonious, interpretable models that predict persistent HRF may improve enrichment of trials testing HRF-targeted therapies and warrant future validation.

Integrating molecular and clinical variables to predict myocardial recovery

Mechanical unloading and circulatory support with left ventricular assist devices (LVADs) mediate significant myocardial improvement in a subset of advanced heart failure (HF) patients. The clinical and biological phenomena associated with cardiac recovery are under intensive investigation. Left ventricular (LV) apical tissue, alongside clinical data, were collected from HF patients at the time of LVAD implantation (n=208). RNA was isolated and mRNA transcripts were identified through RNA sequencing and confirmed with RT-qPCR. To our knowledge this is the first study to combine transcriptomic and clinical data to derive predictors of myocardial recovery. We used a bioinformatic approach to integrate 59 clinical variables and 22,373 mRNA transcripts at the time of LVAD implantation for the prediction of post-LVAD myocardial recovery defined as LV ejection fraction (LVEF) ≥40% and LV end-diastolic diameter (LVEDD) ≤5.9cm, as well as functional and structural LV improvement independently by using LVEF and LVEDD as continuous variables, respectively. To substantiate the predicted variables, we used a multi-model approach with logistic and linear regressions. Combining RNA and clinical data resulted in a gradient boosted model with 80 features achieving an AUC of 0.731±0.15 for predicting myocardial recovery. Variables associated with myocardial recovery from a clinical standpoint included HF duration, pre-LVAD LVEF, LVEDD, and HF pharmacologic therapy, and LRRN4CL (ligand binding and programmed cell death) from a biological standpoint. Our findings could have diagnostic, prognostic, and therapeutic implications for advanced HF patients, and inform the care of the broader HF population.

Development and Validation of Two-Step Prediction Models for Postoperative Bedridden Status in Geriatric Intertrochanteric Hip Fractures

Patients with intertrochanteric hip fractures are at an elevated risk of becoming bedridden compared with those with intraarticular hip fractures. Accurate risk assessments can help clinicians select postoperative rehabilitation strategies to mitigate the risk of bedridden status. This study aimed to develop a two-step prediction model to predict bedridden status at 3 months postoperatively: one model (first step) for prediction at the time of admission to help dictate postoperative rehabilitation plans; and another (second step) for prediction at the time before discharge to determine appropriate discharge destinations and home rehabilitation programs. Three-hundred and eighty-four patients were retrospectively reviewed and divided into a development group (n = 291) and external validation group (n = 93). We developed a two-step prediction model to predict the three-month bedridden status of patients with intertrochanteric fractures from the development group. The first (preoperative) model incorporated four simple predictors: age, dementia, American Society of Anesthesiologists physical status classification (ASA), and pre-fracture ambulatory status. The second (predischarge) model used an additional predictor, ambulation status before discharge. Model performances were evaluated using the external validation group. The preoperative model performances were area under ROC curve (AUC) = 0.72 (95%CI 0.61-0.83) and calibration slope = 1.22 (0.40-2.23). The predischarge model performances were AUC = 0.83 (0.74-0.92) and calibration slope = 0.89 (0.51-1.35). A decision curve analysis (DCA) showed a positive net benefit across a threshold probability between 10% and 35%, with a higher positive net benefit for the predischarge model. Our prediction models demonstrated good discrimination, calibration, and net benefit gains. Using readily available predictors for prognostic prediction can assist clinicians in planning individualized postoperative rehabilitation programs, home-based rehabilitation programs, and determining appropriate discharge destinations, especially in environments with limited resources.

A Systematic Review of Predictor Composition, Outcomes, Risk of Bias, and Validation of COVID-19 Prognostic Scores

BACKGROUND

Numerous prognostic scores have been published to support risk stratification for patients with coronavirus disease 2019 (COVID-19).

METHODS

We performed a systematic review to identify the scores for confirmed or clinically assumed COVID-19 cases. An in-depth assessment and risk of bias (ROB) analysis (Prediction model Risk Of Bias ASsessment Tool [PROBAST]) was conducted for scores fulfilling predefined criteria ([I] area under the curve [AUC)] ≥ 0.75; [II] a separate validation cohort present; [III] training data from a multicenter setting [≥2 centers]; [IV] point-scale scoring system).

RESULTS

Out of 1522 studies extracted from MEDLINE/Web of Science (20/02/2023), we identified 242 scores for COVID-19 outcome prognosis (mortality 109, severity 116, hospitalization 14, long-term sequelae 3). Most scores were developed using retrospective (75.2%) or single-center (57.1%) cohorts. Predictor analysis revealed the primary use of laboratory data and sociodemographic information in mortality and severity scores. Forty-nine scores were included in the in-depth analysis. The results indicated heterogeneous quality and predictor selection, with only five scores featuring low ROB. Among those, based on the number and heterogeneity of validation studies, only the 4C Mortality Score can be recommended for clinical application so far.

CONCLUSIONS

The application and translation of most existing COVID scores appear unreliable. Guided development and predictor selection would have improved the generalizability of the scores and may enhance pandemic preparedness in the future.

Should we use nomograms for risk predictions in diffuse large B cell lymphoma patients? A systematic review

Models based on risk stratification are increasingly reported for Diffuse large B cell lymphoma (DLBCL). Due to a rising interest in nomograms for cancer patients, we aimed to review and critically appraise prognostic models based on nomograms in DLBCL patients. A literature search in PubMed/Embase identified 59 articles that proposed prognostic models for DLBCL by combining parameters of interest (e.g., clinical, laboratory, immunohistochemical, and genetic) between January 2000 and 2024. Of them, 40 studies proposed different gene expression signatures and incorporated them into nomogram-based prognostic models. Although most studies assessed discrimination and calibration when developing the model, many lacked external validation. Current nomogram-based models for DLBCL are mainly developed from publicly available databases, lack external validation, and have no applicability in clinical practice. However, they may be helpful in individual patient counseling, although careful considerations should be made regarding model development due to possible limitations when choosing nomograms for prognostication.

External validation of six COVID-19 prognostic models for predicting mortality risk in older populations in a hospital, primary care, and nursing home setting

OBJECTIVES

To systematically evaluate the performance of COVID-19 prognostic models and scores for mortality risk in older populations across three health-care settings: hospitals, primary care, and nursing homes.

STUDY DESIGN AND SETTING

This retrospective external validation study included 14,092 older individuals of ≥70 years of age with a clinical or polymerase chain reaction-confirmed COVID-19 diagnosis from March 2020 to December 2020. The six validation cohorts include three hospital-based (CliniCo, COVID-OLD, COVID-PREDICT), two primary care-based (Julius General Practitioners Network/Academisch network huisartsgeneeskunde/Network of Academic general Practitioners, PHARMO), and one nursing home cohort (YSIS) in the Netherlands. Based on a living systematic review of COVID-19 prediction models using Prediction model Risk Of Bias ASsessment Tool for quality and risk of bias assessment and considering predictor availability in validation cohorts, we selected six prognostic models predicting mortality risk in adults with COVID-19 infection (GAL-COVID-19 mortality, 4C Mortality Score, National Early Warning Score 2-extended model, Xie model, Wang clinical model, and CURB65 score). All six prognostic models were validated in the hospital cohorts and the GAL-COVID-19 mortality model was validated in all three healthcare settings. The primary outcome was in-hospital mortality for hospitals and 28-day mortality for primary care and nursing home settings. Model performance was evaluated in each validation cohort separately in terms of discrimination, calibration, and decision curves. An intercept update was performed in models indicating miscalibration followed by predictive performance re-evaluation.

MAIN OUTCOME MEASURE

In-hospital mortality for hospitals and 28-day mortality for primary care and nursing home setting.

RESULTS

All six prognostic models performed poorly and showed miscalibration in the older population cohorts. In the hospital settings, model performance ranged from calibration-in-the-large -1.45 to 7.46, calibration slopes 0.24-0.81, and C-statistic 0.55-0.71 with 4C Mortality Score performing as the most discriminative and well-calibrated model. Performance across health-care settings was similar for the GAL-COVID-19 model, with a calibration-in-the-large in the range of -2.35 to -0.15 indicating overestimation, calibration slopes of 0.24-0.81 indicating signs of overfitting, and C-statistic of 0.55-0.71.

CONCLUSION

Our results show that most prognostic models for predicting mortality risk performed poorly in the older population with COVID-19, in each health-care setting: hospital, primary care, and nursing home settings. Insights into factors influencing predictive model performance in the older population are needed for pandemic preparedness and reliable prognostication of health-related outcomes in this demographic.

Development and validation of a novel model to predict recurrence-free survival and melanoma-specific survival after sentinel lymph node biopsy in patients with melanoma: an international, retrospective, multicentre analysis

BACKGROUND

The introduction of adjuvant systemic treatment for patients with high-risk melanomas necessitates accurate staging of disease. However, inconsistencies in outcomes exist between disease stages as defined by the American Joint Committee on Cancer (8th edition). We aimed to develop a tool to predict patient-specific outcomes in people with melanoma rather than grouping patients according to disease stage.

METHODS

Patients older than 13 years with confirmed primary melanoma who underwent sentinel lymph node biopsy (SLNB) between Oct 29, 1997, and Nov 11, 2013, at four European melanoma centres (based in Berlin, Germany; Amsterdam and Rotterdam, the Netherlands; and Warsaw, Poland) were included in the development cohort. Potential predictors of recurrence-free and melanoma-specific survival assessed were sex, age, presence of ulceration, primary tumour location, histological subtype, Breslow thickness, sentinel node status, number of sentinel nodes removed, maximum diameter of the largest sentinel node metastasis, and Dewar classification. A prognostic model and nomogram were developed to predict 5-year recurrence-free survival on a continuous scale in patients with stage pT1b or higher melanomas. This model was also calibrated to predict melanoma-specific survival. Model performance was assessed by discrimination (area under the time-dependent receiver operating characteristics curve [AUC]) and calibration. External validation was done in a cohort of patients with primary melanomas who underwent SLNB between Jan 30, 1997, and Dec 12, 2013, at the Melanoma Institute Australia (Sydney, NSW, Australia).

FINDINGS

The development cohort consisted of 4071 patients, of whom 2075 (51%) were female and 1996 (49%) were male. 889 (22%) had sentinel node-positive disease and 3182 (78%) had sentinel node-negative disease. The validation cohort comprised 4822 patients, of whom 1965 (41%) were female and 2857 (59%) were male. 891 (18%) had sentinel node-positive disease and 3931 (82%) had sentinel node-negative disease. Median follow-up was 4·8 years (IQR 2·3-7·8) in the development cohort and 5·0 years (2·2-8·9) in the validation cohort. In the development cohort, 5-year recurrence-free survival was 73·5% (95% CI 72·0-75·1) and 5-year melanoma-specific survival was 86·5% (85·3-87·8). In the validation cohort, the corresponding estimates were 66·1% (64·6-67·7) and 83·3% (82·0-84·6), respectively. The final model contained six prognostic factors: sentinel node status, Breslow thickness, presence of ulceration, age at SLNB, primary tumour location, and maximum diameter of the largest sentinel node metastasis. In the development cohort, for the model's prediction of recurrence-free survival, the AUC was 0·80 (95% CI 0·78-0·81); for prediction of melanoma-specific survival, the AUC was 0·81 (0·79-0·84). External validation showed good calibration for both outcomes, with AUCs of 0·73 (0·71-0·75) and 0·76 (0·74-0·78), respectively.

INTERPRETATION

Our prediction model and nomogram accurately predicted patient-specific risk probabilities for 5-year recurrence-free and melanoma-specific survival. These tools could have important implications for clinical decision making when considering adjuvant treatments in patients with high-risk melanomas.

FUNDING

Erasmus Medical Centre Cancer Institute.

Comparison of the ADNEX and ROMA risk prediction models for the diagnosis of ovarian cancer: a multicentre external validation in patients who underwent surgery

BACKGROUND

Several diagnostic prediction models to help clinicians discriminate between benign and malignant adnexal masses are available. This study is a head-to-head comparison of the performance of the Assessment of Different NEoplasias in the adneXa (ADNEX) model with that of the Risk of Ovarian Malignancy Algorithm (ROMA).

METHODS

This is a retrospective study based on prospectively included consecutive women with an adnexal tumour scheduled for surgery at five oncology centres and one non-oncology centre in four countries between 2015 and 2019. The reference standard was histology. Model performance for ADNEX and ROMA was evaluated regarding discrimination, calibration, and clinical utility.

RESULTS

The primary analysis included 894 patients, of whom 434 (49%) had a malignant tumour. The area under the receiver operating characteristic curve (AUC) was 0.92 (95% CI 0.88-0.95) for ADNEX with CA125, 0.90 (0.84-0.94) for ADNEX without CA125, and 0.85 (0.80-0.89) for ROMA. ROMA, and to a lesser extent ADNEX, underestimated the risk of malignancy. Clinical utility was highest for ADNEX. ROMA had no clinical utility at decision thresholds <27%.

CONCLUSIONS

ADNEX had better ability to discriminate between benign and malignant adnexal tumours and higher clinical utility than ROMA.

CLINICAL TRIAL REGISTRATION

clinicaltrials.gov NCT01698632 and NCT02847832.

A customised down-sampling machine learning approach for sepsis prediction

OBJECTIVE

Sepsis is a life-threatening condition in the ICU and requires treatment in time. Despite the accuracy of existing sepsis prediction models, insufficient focus on reducing alarms could worsen alarm fatigue and desensitisation in ICUs, potentially compromising patient safety. In this retrospective study, we aim to develop an accurate, robust, and readily deployable method in ICUs, only based on the vital signs and laboratory tests.

METHODS

Our method consists of a customised down-sampling process and a specific dynamic sliding window and XGBoost to offer sepsis prediction. The down-sampling process was applied to the retrospective data for training the XGBoost model. During the testing stage, the dynamic sliding window and the trained XGBoost were used to predict sepsis on the retrospective datasets, PhysioNet and FHC.

RESULTS

With the filtered data from PhysioNet, our method achieved 80.74% accuracy (77.90% sensitivity and 84.42% specificity) and 83.95% (84.82% sensitivity and 82.00% specificity) on the test set of PhysioNet-A and PhysioNet-B, respectively. The AUC score was 0.89 for both datasets. On the FHC dataset, our method achieved 92.38% accuracy (88.37% sensitivity and 95.16% specificity) and 0.98 AUC score on the test set of FHC.

CONCLUSION

Our results indicate that the down-sampling process and the dynamic sliding window with XGBoost brought robust and accurate performance to give sepsis prediction under various hospital settings. The localisation and robustness of our method can assist in sepsis diagnosis in different ICU settings.

Development of a simple prediction model for mechanical complication in ST-segment elevation myocardial infarction patients after primary percutaneous coronary intervention

Mechanical complication (MC) is a rare but serious complication in patients with ST-segment elevation myocardial infarction (STEMI). Although several risk factors for MC have been reported, a prediction model for MC has not been established. This study aimed to develop a simple prediction model for MC after STEMI. We included 1717 patients with STEMI who underwent primary percutaneous coronary intervention (PCI). Of 1717 patients, 45 MCs occurred after primary PCI. Prespecified predictors were determined to develop a tentative prediction model for MC using multivariable regression analysis. Then, a simple prediction model for MC was generated. Age ≥ 70, Killip class ≥ 2, white blood cell ≥ 10,000/µl, and onset-to-visit time ≥ 8 h were included in a simple prediction model as "point 1" risk score, whereas initial thrombolysis in myocardial infarction (TIMI) flow grade ≤ 1 and final TIMI flow grade ≤ 2 were included as "point 2" risk score. The simple prediction model for MC showed good discrimination with the optimism-corrected area under the receiver-operating characteristic curve of 0.850 (95% CI: 0.798-0.902). The predicted probability for MC was 0-2% in patients with 0-4 points of risk score, whereas that was 6-50% in patients with 5-8 points. In conclusion, we developed a simple prediction model for MC. We may be able to predict the probability for MC by this simple prediction model.

External validation and update of the International Medical Prevention Registry on Venous Thromboembolism bleeding risk score for predicting bleeding in acutely ill hospitalized medical patients: a retrospective single-center cohort study in Japan

BACKGROUND

The International Medical Prevention Registry for Venous Thromboembolism (IMPROVE) Bleeding Risk Score is the recommended risk assessment model (RAM) for predicting bleeding risk in acutely ill medical inpatients in Western countries. However, few studies have assessed its predictive performance in local Asian settings.

METHODS

We retrospectively identified acutely ill adolescents and adults (aged ≥ 15 years) who were admitted to our general internal medicine department between July 5, 2016 and July 5, 2021, and extracted data from their electronic medical records. The outcome of interest was the cumulative incidence of major and nonmajor but clinically relevant bleeding 14 days after admission. For the two-risk-group model, we estimated sensitivity, specificity, and positive and negative predictive values (PPV and NPV, respectively). For the 11-risk-group model, we estimated C statistic, expected and observed event ratio (E/O), calibration-in-the-large (CITL), and calibration slope. In addition, we recalibrated the intercept using local data to update the RAM.

RESULTS

Among the 3,876 included patients, 998 (26%) were aged ≥ 85 years, while 656 (17%) were hospitalized in the intensive care unit. The median length of hospital stay was 14 days. Clinically relevant bleeding occurred in 58 patients (1.5%), 49 (1.3%) of whom experienced major bleeding. Sensitivity, specificity, NPV, and PPV were 26.1% (95% confidence interval [CI]: 15.8-40.0%), 84.8% (83.6-85.9%), 98.7% (98.2-99.0%), and 2.5% (1.5-4.3%) for any bleeding and 30.9% (95% CI: 18.8-46.3%), 84.9% (83.7-86.0%), 99.0% (98.5-99.3%), and 2.5% (1.5-4.3%) for major bleeding, respectively. The C statistic, E/O, CITL, and calibration slope were 0.64 (95% CI: 0.58-0.71), 1.69 (1.45-2.05), - 0.55 (- 0.81 to - 0.29), and 0.58 (0.29-0.87) for any bleeding and 0.67 (95% CI: 0.60-0.74), 0.76 (0.61-0.87), 0.29 (0.00-0.58), and 0.42 (0.19-0.64) for major bleeding, respectively. Updating the model substantially corrected the poor calibration observed.

CONCLUSIONS

In our Japanese cohort, the IMPROVE bleeding RAM retained the reported moderate discriminative performance. Model recalibration substantially improved the poor calibration obtained using the original RAM. Before its introduction into clinical practice, the updated RAM needs further validation studies and an optimized threshold.

Prognostic factors and prediction models for hospitalisation and all-cause mortality in adults presenting to primary care with a lower respiratory tract infection: a systematic review

OBJECTIVE

To identify and synthesise relevant existing prognostic factors (PF) and prediction models (PM) for hospitalisation and all-cause mortality within 90 days in primary care patients with acute lower respiratory tract infections (LRTI).

DESIGN

Systematic review.

METHODS

Systematic searches of MEDLINE, Embase and the Cochrane Library were performed. All PF and PM studies on the risk of hospitalisation or all-cause mortality within 90 days in adult primary care LRTI patients were included. The risk of bias was assessed using the Quality in Prognostic Studies tool and Prediction Model Risk Of Bias Assessment Tool tools for PF and PM studies, respectively. The results of included PF and PM studies were descriptively summarised.

RESULTS

Of 2799 unique records identified, 16 were included: 9 PF studies, 6 PM studies and 1 combination of both. The risk of bias was judged high for all studies, mainly due to limitations in the analysis domain. Based on reported multivariable associations in PF studies, increasing age, sex, current smoking, diabetes, a history of stroke, cancer or heart failure, previous hospitalisation, influenza vaccination (negative association), current use of systemic corticosteroids, recent antibiotic use, respiratory rate ≥25/min and diagnosis of pneumonia were identified as most promising candidate predictors. One newly developed PM was externally validated (c statistic 0.74, 95% CI 0.71 to 0.78) whereas the previously hospital-derived CRB-65 was externally validated in primary care in five studies (c statistic ranging from 0.72 (95% CI 0.63 to 0.81) to 0.79 (95% CI 0.65 to 0.92)). None of the PM studies reported measures of model calibration.

CONCLUSIONS

Implementation of existing models for individualised risk prediction of 90-day hospitalisation or mortality in primary care LRTI patients in everyday practice is hampered by incomplete assessment of model performance. The identified candidate predictors provide useful information for clinicians and warrant consideration when developing or updating PMs using state-of-the-art development and validation techniques.

PROSPERO REGISTRATION NUMBER

CRD42022341233.

Incorporating social vulnerability in infectious disease mathematical modelling: a scoping review

BACKGROUND

Highlighted by the rise of COVID-19, climate change, and conflict, socially vulnerable populations are least resilient to disaster. In infectious disease management, mathematical models are a commonly used tool. Researchers should include social vulnerability in models to strengthen their utility in reflecting real-world dynamics. We conducted a scoping review to evaluate how researchers have incorporated social vulnerability into infectious disease mathematical models.

METHODS

The methodology followed the Joanna Briggs Institute and updated Arksey and O'Malley frameworks, verified by the PRISMA-ScR checklist. PubMed, Clarivate Web of Science, Scopus, EBSCO Africa Wide Information, and Cochrane Library were systematically searched for peer-reviewed published articles. Screening and extracting data were done by two independent researchers.

RESULTS

Of 4075 results, 89 articles were identified. Two-thirds of articles used a compartmental model (n = 58, 65.2%), with a quarter using agent-based models (n = 24, 27.0%). Overall, routine indicators, namely age and sex, were among the most frequently used measures (n = 42, 12.3%; n = 22, 6.4%, respectively). Only one measure related to culture and social behaviour (0.3%). For compartmental models, researchers commonly constructed distinct models for each level of a social vulnerability measure and included new parameters or influenced standard parameters in model equations (n = 30, 51.7%). For all agent-based models, characteristics were assigned to hosts (n = 24, 100.0%), with most models including age, contact behaviour, and/or sex (n = 18, 75.0%; n = 14, 53.3%; n = 10, 41.7%, respectively).

CONCLUSIONS

Given the importance of equitable and effective infectious disease management, there is potential to further the field. Our findings demonstrate that social vulnerability is not considered holistically. There is a focus on incorporating routine demographic indicators but important cultural and social behaviours that impact health outcomes are excluded. It is crucial to develop models that foreground social vulnerability to not only design more equitable interventions, but also to develop more effective infectious disease control and elimination strategies. Furthermore, this study revealed the lack of transparency around data sources, inconsistent reporting, lack of collaboration with local experts, and limited studies focused on modelling cultural indicators. These challenges are priorities for future research.

Suicide risk assessment tools and prediction models: new evidence, methodological innovations, outdated criticisms

The number of prediction models for suicide-related outcomes has grown substantially in recent years. These models aim to assist in stratifying risk, improve clinical decision-making, and facilitate a personalised medicine approach to the prevention of suicidal behaviour. However, there are contrasting views as to whether prediction models have potential to inform and improve assessment of suicide risk. In this perspective, we discuss common misconceptions that characterise criticisms of suicide risk prediction research. First, we discuss the limitations of a classification approach to risk assessment (eg, categorising individuals as low-risk vs high-risk), and highlight the benefits of probability estimation. Second, we argue that the preoccupation with classification measures (such as positive predictive value) when assessing a model's predictive performance is inappropriate, and discuss the importance of clinical context in determining the most appropriate risk threshold for a given model. Third, we highlight that adequate discriminative ability for a prediction model depends on the clinical area, and emphasise the importance of calibration, which is almost entirely overlooked in the suicide risk prediction literature. Finally, we point out that conclusions about the clinical utility and health-economic value of suicide prediction models should be based on appropriate measures (such as net benefit and decision-analytic modelling), and highlight the role of impact assessment studies. We conclude that the discussion around using suicide prediction models and risk assessment tools requires more nuance and statistical expertise, and that guidelines and suicide prevention strategies should be informed by the new and higher quality evidence in the field.

Development of risk models of incident hypertension using machine learning on the HUNT study data

In this study, we aimed to create an 11-year hypertension risk prediction model using data from the Trøndelag Health (HUNT) Study in Norway, involving 17 852 individuals (20-85 years; 38% male; 24% incidence rate) with blood pressure (BP) below the hypertension threshold at baseline (1995-1997). We assessed 18 clinical, behavioral, and socioeconomic features, employing machine learning models such as eXtreme Gradient Boosting (XGBoost), Elastic regression, K-Nearest Neighbor, Support Vector Machines (SVM) and Random Forest. For comparison, we used logistic regression and a decision rule as reference models and validated six external models, with focus on the Framingham risk model. The top-performing models consistently included XGBoost, Elastic regression and SVM. These models efficiently identified hypertension risk, even among individuals with optimal baseline BP (< 120/80 mmHg), although improvement over reference models was modest. The recalibrated Framingham risk model outperformed the reference models, approaching the best-performing ML models. Important features included age, systolic and diastolic BP, body mass index, height, and family history of hypertension. In conclusion, our study demonstrated that linear effects sufficed for a well-performing model. The best models efficiently predicted hypertension risk, even among those with optimal or normal baseline BP, using few features. The recalibrated Framingham risk model proved effective in our cohort.

Prognostic Models in Nephrology: Where Do We Stand and Where Do We Go from Here? Mapping Out the Evidence in a Scoping Review

Prognostic models can strongly support individualized care provision and well-informed shared decision making. There has been an upsurge of prognostic research in the field of nephrology, but the uptake of prognostic models in clinical practice remains limited. Therefore, we map out the research field of prognostic models for kidney patients and provide directions on how to proceed from here. We performed a scoping review of studies developing, validating, or updating a prognostic model for patients with CKD. We searched all published models in PubMed and Embase and report predicted outcomes, methodological quality, and validation and/or updating efforts. We found 602 studies, of which 30.1% concerned CKD populations, 31.6% dialysis populations, and 38.4% kidney transplantation populations. The most frequently predicted outcomes were mortality ( n =129), kidney disease progression ( n =75), and kidney graft survival ( n =54). Most studies provided discrimination measures (80.4%), but much less showed calibration results (43.4%). Of the 415 development studies, 28.0% did not perform any validation and 57.6% performed only internal validation. Moreover, only 111 models (26.7%) were externally validated either in the development study itself or in an independent external validation study. Finally, in 45.8% of development studies no useable version of the model was reported. To conclude, many prognostic models have been developed for patients with CKD, mainly for outcomes related to kidney disease progression and patient/graft survival. To bridge the gap between prediction research and kidney patient care, patient-reported outcomes, methodological rigor, complete reporting of prognostic models, external validation, updating, and impact assessment urgently need more attention.

Modest performance of text mining to extract health outcomes may be almost sufficient for high-quality prognostic model development

BACKGROUND

Across medicine, prognostic models are used to estimate patient risk of certain future health outcomes (e.g., cardiovascular or mortality risk). To develop (or train) prognostic models, historic patient-level training data is needed containing both the predictive factors (i.e., features) and the relevant health outcomes (i.e., labels). Sometimes, when the health outcomes are not recorded in structured data, these are first extracted from textual notes using text mining techniques. Because there exist many studies utilizing text mining to obtain outcome data for prognostic model development, our aim is to study the impact of the text mining quality on downstream prognostic model performance.

METHODS

We conducted a simulation study charting the relationship between text mining quality and prognostic model performance using an illustrative case study about in-hospital mortality prediction in intensive care unit patients. We repeatedly developed and evaluated a prognostic model for in-hospital mortality, using outcome data extracted by multiple text mining models of varying quality.

RESULTS

Interestingly, we found in our case study that a relatively low-quality text mining model (F1 score ≈ 0.50) could already be used to train a prognostic model with quite good discrimination (area under the receiver operating characteristic curve of around 0.80). The calibration of the risks estimated by the prognostic model seemed unreliable across the majority of settings, even when text mining models were of relatively high quality (F1 ≈ 0.80).

DISCUSSION

Developing prognostic models on text-extracted outcomes using imperfect text mining models seems promising. However, it is likely that prognostic models developed using this approach may not produce well-calibrated risk estimates, and require recalibration in (possibly a smaller amount of) manually extracted outcome data.

Identification of influence factors in overweight population through an interpretable risk model based on machine learning: a large retrospective cohort

BACKGROUND

The identification of associated overweight risk factors is crucial to future health risk predictions and behavioral interventions. Several consensus problems remain in machine learning, such as cross-validation, and the resulting model may suffer from overfitting or poor interpretability.

METHODS

This study employed nine commonly used machine learning methods to construct overweight risk models. The general community are the target of this study, and a total of 10,905 Chinese subjects from Ningde City in Fujian province, southeast China, participated. The best model was selected through appropriate verification and validation and was suitably explained.

RESULTS

The overweight risk models employing machine learning exhibited good performance. It was concluded that CatBoost, which is used in the construction of clinical risk models, may surpass previous machine learning methods. The visual display of the Shapley additive explanation value for the machine model variables accurately represented the influence of each variable in the model.

CONCLUSIONS

The construction of an overweight risk model using machine learning may currently be the best approach. Moreover, CatBoost may be the best machine learning method. Furthermore, combining Shapley's additive explanation and machine learning methods can be effective in identifying disease risk factors for prevention and control.

Decision curve analysis based on summary data

BACKGROUND

To realize the potential of precision medicine, predictive models should be integrated within the framework of decision analysis, such as the decision curve analysis (DCA). To date, its application has required individual patient data (IPD) that are often unavailable. Performing DCA using aggregate data without requiring IPD may advance the goals of precision medicine.

METHODS

We present a statistical framework demonstrating that DCA can be conducted by using only the mean and standard deviation (SD) from the raw probabilities of the predictive model. We tested our theoretical framework by performing extensive simulations and comparing the aggregate-based DCA with IPD DCA. The latter was conducted using IPD from four predictive models that employed logistic regression, Cox or competing risk time-to-event modeling including (a) statins for primary prevention of cardiovascular disease (n = 4859), (b) hospice referral for terminally ill patients (n = 9104), (c) use of thromboprophylaxis for preventing venous thromboembolism in patients with cancer (n = 425) and (d) prevention of sinusoidal obstruction syndrome after hematopoietic cell transplantation (SCT) (n = 80).

RESULTS

Simulations assuming perfect calibration showed that regardless of which probability distributions informed the predictive models, the differences in DCA were negligible. Similarly, for the adequately powered models, the results of DCA based on the summary data were similar to IPD-derived DCA. The inherent instability of the predictive models, based on the smaller sample sizes, resulted in a somewhat larger discrepancy between aggregate and IPD-based DCA.

CONCLUSIONS

DCA informed by adequately powered and well-calibrated models using only summary statistical estimates (mean and SD) approximates well models using IPD. Use of aggregate data will facilitate broader integration of predictive with decision modeling toward the goals of individualized decision-making.

Performance of models for predicting 1-year to 3-year mortality in older adults: a systematic review of externally validated models

Mortality prediction models support identifying older adults with short life expectancy for whom clinical care might need modifications. We systematically reviewed external validations of mortality prediction models in older adults (ie, aged 65 years and older) with up to 3 years of follow-up. In March, 2023, we conducted a literature search resulting in 36 studies reporting 74 validations of 64 unique models. Model applicability was fair but validation risk of bias was mostly high, with 50 (68%) of 74 validations not reporting calibration. Morbidities (most commonly cardiovascular diseases) were used as predictors by 45 (70%) of 64 of models. For 1-year prediction, 31 (67%) of 46 models had acceptable discrimination, but only one had excellent performance. Models with more than 20 predictors were more likely to have acceptable discrimination (risk ratio [RR] vs <10 predictors 1·68, 95% CI 1·06-2·66), as were models including sex (RR 1·75, 95% CI 1·12-2·73) or predicting risk during comprehensive geriatric assessment (RR 1·86, 95% CI 1·12-3·07). Development and validation of better-performing mortality prediction models in older people are needed.

The Improved Kidney Risk Score in ANCA-Associated Vasculitis for Clinical Practice and Trials

SIGNIFICANCE STATEMENT

Reliable prediction tools are needed to personalize treatment in ANCA-associated GN. More than 1500 patients were collated in an international longitudinal study to revise the ANCA kidney risk score. The score showed satisfactory performance, mimicking the original study (Harrell's C=0.779). In the development cohort of 959 patients, no additional parameters aiding the tool were detected, but replacing the GFR with creatinine identified an additional cutoff. The parameter interstitial fibrosis and tubular atrophy was modified to allow wider access, risk points were reweighted, and a fourth risk group was created, improving predictive ability (C=0.831). In the validation, the new model performed similarly well with excellent calibration and discrimination ( n =480, C=0.821). The revised score optimizes prognostication for clinical practice and trials.

BACKGROUND

Reliable prediction tools are needed to personalize treatment in ANCA-associated GN. A retrospective international longitudinal cohort was collated to revise the ANCA renal risk score.

METHODS

The primary end point was ESKD with patients censored at last follow-up. Cox proportional hazards were used to reweight risk factors. Kaplan-Meier curves, Harrell's C statistic, receiver operating characteristics, and calibration plots were used to assess model performance.

RESULTS

Of 1591 patients, 1439 were included in the final analyses, 2:1 randomly allocated per center to development and validation cohorts (52% male, median age 64 years). In the development cohort ( n =959), the ANCA renal risk score was validated and calibrated, and parameters were reinvestigated modifying interstitial fibrosis and tubular atrophy allowing semiquantitative reporting. An additional cutoff for kidney function (K) was identified, and serum creatinine replaced GFR (K0: <250 µ mol/L=0, K1: 250-450 µ mol/L=4, K2: >450 µ mol/L=11 points). The risk points for the percentage of normal glomeruli (N) and interstitial fibrosis and tubular atrophy (T) were reweighted (N0: >25%=0, N1: 10%-25%=4, N2: <10%=7, T0: none/mild or <25%=0, T1: ≥ mild-moderate or ≥25%=3 points), and four risk groups created: low (0-4 points), moderate (5-11), high (12-18), and very high (21). Discrimination was C=0.831, and the 3-year kidney survival was 96%, 79%, 54%, and 19%, respectively. The revised score performed similarly well in the validation cohort with excellent calibration and discrimination ( n =480, C=0.821).

CONCLUSIONS

The updated score optimizes clinicopathologic prognostication for clinical practice and trials.

Population-Based Validation of the MIA and MSKCC Tools for Predicting Sentinel Lymph Node Status

Importance

Patients with melanoma are selected for sentinel lymph node biopsy (SLNB) based on their risk of a positive SLN. To improve selection, the Memorial Sloan Kettering Cancer Center (MSKCC) and Melanoma Institute Australia (MIA) developed predictive models, but the utility of these models remains to be tested.

Objective

To determine the clinical utility of the MIA and MSKCC models.

Design, Setting, and Participants

This was a population-based comparative effectiveness research study including 10 089 consecutive patients with cutaneous melanoma undergoing SLNB from the Swedish Melanoma Registry from January 2007 to December 2021. Data were analyzed from May to August 2023.

Main Outcomes and Measures,

The predicted probability of SLN positivity was calculated using the MSKCC model and a limited MIA model (using mitotic rate as absent/present instead of count/mm2 and excluding the optional variable lymphovascular invasion) for each patient. The operating characteristics of the models were assessed and compared. The clinical utility of each model was assessed using decision curve analysis and compared with a strategy of performing SLNB on all patients.

Results

Among 10 089 included patients, the median (IQR) age was 64.0 (52.0-73.0) years, and 5340 (52.9%) were male. The median Breslow thickness was 1.8 mm, and 1802 patients (17.9%) had a positive SLN. Both models were well calibrated across the full range of predicted probabilities and had similar external area under the receiver operating characteristic curves (AUC; MSKCC: 70.8%; 95% CI, 69.5-72.1 and limited MIA: 69.7%; 95% CI, 68.4-71.1). At a risk threshold of 5%, decision curve analysis indicated no added net benefit for either model compared to performing SLNB for all patients. At risk thresholds of 10% or higher, both models added net benefit compared to SLNB for all patients. The greatest benefit was observed in patients with T2 melanomas using a threshold of 10%; in that setting, the use of the nomograms led to a net reduction of 8 avoidable SLNBs per 100 patients for the MSKCC nomogram and 7 per 100 patients for the limited MIA nomogram compared to a strategy of SLNB for all.

Conclusions and Relevance

This study confirmed the statistical performance of both the MSKCC and limited MIA models in a large, nationally representative data set. However, decision curve analysis demonstrated that using the models only improved selection for SLNB compared to biopsy in all patients when a risk threshold of at least 7% was used, with the greatest benefit seen for T2 melanomas at a threshold of 10%. Care should be taken when using these nomograms to guide selection for SLNB at the lowest thresholds.

Performance of magnetic resonance imaging-based prostate cancer risk calculators and decision strategies in two large European medical centres

OBJECTIVES

To compare the performance of currently available biopsy decision support tools incorporating magnetic resonance imaging (MRI) findings in predicting clinically significant prostate cancer (csPCa).

PATIENTS AND METHODS

We retrospectively included men who underwent prostate MRI and subsequent targeted and/or systematic prostate biopsies in two large European centres. Available decision support tools were identified by a PubMed search. Performance was assessed by calibration, discrimination, decision curve analysis (DCA) and numbers of biopsies avoided vs csPCa cases missed, before and after recalibration, at risk thresholds of 5%-20%.

RESULTS

A total of 940 men were included, 507 (54%) had csPCa. The median (interquartile range) age, prostate-specific antigen (PSA) level, and PSA density (PSAD) were 68 (63-72) years, 9 (7-15) ng/mL, and 0.20 (0.13-0.32) ng/mL2 , respectively. In all, 18 multivariable risk calculators (MRI-RCs) and dichotomous biopsy decision strategies based on MRI findings and PSAD thresholds were assessed. The Van Leeuwen model and the Rotterdam Prostate Cancer Risk Calculator (RPCRC) had the best discriminative ability (area under the receiver operating characteristic curve 0.86) of the MRI-RCs that could be assessed in the whole cohort. DCA showed the highest clinical utility for the Van Leeuwen model, followed by the RPCRC. At the 10% threshold the Van Leeuwen model would avoid 22% of biopsies, missing 1.8% of csPCa, whilst the RPCRC would avoid 20% of biopsies, missing 2.6% of csPCas. These multivariable models outperformed all dichotomous decision strategies based only on MRI-findings and PSAD.

CONCLUSIONS

Even in this high-risk cohort, biopsy decision support tools would avoid many prostate biopsies, whilst missing very few csPCa cases. The Van Leeuwen model had the highest clinical utility, followed by the RPCRC. These multivariable MRI-RCs outperformed and should be favoured over decision strategies based only on MRI and PSAD.