Transparent Reporting of Multivariable Prediction Models in Journal and Conference Abstracts: TRIPOD for Abstracts

SPIN-PM: a consensus framework to evaluate the presence of spin in studies on prediction models

OBJECTIVES

To develop a framework to identify and evaluate spin practices and its facilitators in studies on clinical prediction model regardless of the modeling technique.

STUDY DESIGN AND SETTING

We followed a three-phase consensus process: (1) premeeting literature review to generate items to be included; (2) a series of structured meetings to provide comments discussed and exchanged viewpoints on items to be included with a panel of experienced researchers; and (3) postmeeting review on final list of items and examples to be included. Through this iterative consensus process, a framework was derived after all panel's researchers agreed.

RESULTS

This consensus process involved a panel of eight researchers and resulted in SPIN-Prediction Models which consists of two categories of spin (misleading interpretation and misleading transportability), and within these categories, two forms of spin (spin practices and facilitators of spin). We provide criteria and examples.

CONCLUSION

We proposed this guidance aiming to facilitate not only the accurate reporting but also an accurate interpretation and extrapolation of clinical prediction models which will likely improve the reporting quality of subsequent research, as well as reduce research waste.

Clinical Value of the Diagonal Earlobe Crease in Patients with Chest Pain for Diagnosing Coronary Heart Disease

Purpose

To investigate the clinical application value of diagonal earlobe crease (DELC) in patients with chest pain for the diagnosis of coronary heart disease (CHD) and to construct a risk model by multivariate logistic regression.

Patients and Methods

Our trial enrolled prospectively and consecutively 706 chest pain patients with suspected CHD between January 2021 to June 2023 from Chengde Central Hospital. According to coronary angiography results, they were categorized into the CHD (n=457) and non-CHD groups (n=249).

Results

The trial demonstrated a significant positive relationship between DELC and CHD. Independent risk factors were sex, age, hypertension, diabetes mellitus, LP (a), Cys C, and DELC, whilst HDL-C was a protective factor, for CHD. Patients with-DELC were older than those in the without-DELC arm (P<0.001) and had a higher proportion of males than females (61.6% vs 50.0%, P=0.026). After multifactorial correction, independent risk factors for CHD included DELC (OR=1.660, 95% CI:1.153 to 2.388, P=0.006), age (OR=1.024, 95% CI:1.002 to 1.045, P=0.030), gender (OR=1.702, 95% CI:1.141 to 2.539, P=0.009), hypertension (OR=1.744, 95% CI:1.226 to 2.482, P=0.002), diabetes mellitus (OR=2.113, 95% CI:1.404 to 3.179, P<0.001), LP(a) (OR=1.010, 95% CI:1.003 to 1.017, P=0.005), Cys C (OR=3.549, 95% CI:1.605 to 7.846, P=0.002). The Hosmer and Lemeshow (H-L) test (P=0.818) suggests a high goodness of fit, and the area under the ROC curve was calculated to be 0.721 (95% CI:0.682 to 0.760, P<0.001), which demonstrates that the model has a superior diagnostic value for CHD.

Conclusion

DELC is an independent risk factor for CHD after adjusting for sex, age, hypertension, diabetes mellitus, smoking index, LP (a), Cys C, and HDL-C. Our model can be used clinically for assessing the risk of CHD.

Development and validation of a noninvasive prediction model of autoimmune hepatitis in patients with liver diseases

BACKGROUND AND AIMS

There is no golden standard for the diagnosis of autoimmune hepatitis which still dependent on liver biopsy currently. So, we developed a noninvasive prediction model to help optimize the diagnosis of autoimmune hepatitis.

METHODS

From January 2017 to December 2019, 1739 patients who had undergone liver biopsy were seen in the second hospital of Nanjing, of which 128 were here for consultation. Clinical, laboratory, and histologic data were obtained retrospectively. Multivariable logistic regression analysis was employed to create a nomogram model that predicting the risk of autoimmune hepatitis. Internal and external validation was both performed to evaluate the model.

RESULTS

A total of 1288 patients with liver biopsy were enrolled (1184 from the second hospital of Nanjing, the remaining 104 from other centers). After the univariate and multivariate logistic regression analysis, nine variables including ALT, IgG, ALP/AST, ALB, ANA, AMA, HBsAg, age, and gender were selected to establish the noninvasive prediction model. The nomogram model exhibits good prediction in diagnosing autoimmune hepatitis with AUROC of 0.967 (95% CI: 0.776-0.891) in internal validation and 0.835 (95% CI: 0.752-0.919) in external validation.

CONCLUSIONS

ALT, IgG, ALP/AST, ALB, ANA, AMA, HBsAg, age, and gender are predictive factors for the diagnosis of autoimmune hepatitis in patients with unexplained liver diseases. The predictive nomogram model built by the nine predictors achieved good prediction for diagnosing autoimmune hepatitis.

Explainable machine learning model for predicting furosemide responsiveness in patients with oliguric acute kidney injury

BACKGROUND

Although current guidelines didn't support the routine use of furosemide in oliguric acute kidney injury (AKI) management, some patients may benefit from furosemide administration at an early stage. We aimed to develop an explainable machine learning (ML) model to differentiate between furosemide-responsive (FR) and furosemide-unresponsive (FU) oliguric AKI.

METHODS

From Medical Information Mart for Intensive Care-IV (MIMIC-IV) and eICU Collaborative Research Database (eICU-CRD), oliguric AKI patients with urine output (UO) < 0.5 ml/kg/h for the first 6 h after ICU admission and furosemide infusion ≥ 40 mg in the following 6 h were retrospectively selected. The MIMIC-IV cohort was used in training a XGBoost model to predict UO > 0.65 ml/kg/h during 6-24 h succeeding the initial 6 h for assessing oliguria, and it was validated in the eICU-CRD cohort. We compared the predictive performance of the XGBoost model with the traditional logistic regression and other ML models.

RESULTS

6897 patients were included in the MIMIC-IV training cohort, with 2235 patients in the eICU-CRD validation cohort. The XGBoost model showed an AUC of 0.97 (95% CI: 0.96-0.98) for differentiating FR and FU oliguric AKI. It outperformed the logistic regression and other ML models in correctly predicting furosemide diuretic response, achieved 92.43% sensitivity (95% CI: 90.88-93.73%) and 95.12% specificity (95% CI: 93.51-96.3%).

CONCLUSION

A boosted ensemble algorithm can be used to accurately differentiate between patients who would and would not respond to furosemide in oliguric AKI. By making the model explainable, clinicians would be able to better understand the reasoning behind the prediction outcome and make individualized treatment.

Association between neutrophil-to-albumin ratio and long-term mortality of aneurysmal subarachnoid hemorrhage

OBJECTIVE

The prognosis of aneurysmal subarachnoid hemorrhage (aSAH) survivors is concerning. The goal of this study was to investigate and demonstrate the relationship between the neutrophil-to-albumin ratio (NAR) and long-term mortality of aSAH survivors.

METHODS

A retrospective observational cohort study was conducted at Sichuan University West China Hospital between January 2009 and June 2019. The investigation of relationship between NAR and long-term mortality was conducted using univariable and multivariable Cox regression models. To demonstrate the predictive performance of different biomarkers over time, time-dependent receiver operating characteristic curve (ROC) analysis and decision curve analysis (DCA) were created.

RESULTS

In total, 3173 aSAH patients were included in this study. There was a strong and continuous relationship between NAR levels and long-term mortality (HR 3.23 95% CI 2.75-3.79, p < 0.001). After adjustment, the result was still significant (adjusted HR 1.78 95% CI 1.49-2.12). Compared with patients with the lowest quartile (< 0.15) of NAR levels, the risk of long-term mortality in the other groups was higher (0.15-0.20: adjusted HR 1.30 95% CI 0.97-1.73; 0.20-0.28: adjusted HR 1.37 95% CI 1.03-1.82; >0.28: adjusted HR 1.74 95% CI 1.30-2.32). Results in survivors were found to be still robust. Moreover, out of all the inflammatory markers studied, NAR demonstrated the highest correlation with long-term mortality.

CONCLUSIONS

A high level of NAR was associated with increased long-term mortality among patients with aSAH. NAR was a promising inflammatory marker for long-term mortality of aSAH.

Determination of Factors Related to the Reversal and Perioperative Outcomes of Defunctioning Ileostomies in Patients Undergoing Rectal Cancer Surgery: A Regression Analysis Model

PURPOSE

Defunctioning ileostomies are often performed during rectal cancer surgery. However, stomas are sometimes associated with complications, while 20-30% of them are never reversed. Additionally, ileostomy closure can have associated morbidity, with rates as high as 45%, with the respective literature evidence being scarce and conflicting. Thus, we evaluated the stoma reversal outcomes and the risk factors for non-closure after rectal cancer surgery.

METHODS

This is a retrospective analysis of a prospectively collected database of all patients who had a defunctioning ileostomy at the time of resection for rectal cancer. All operations were performed by the same surgical team. A multivariable regression model was implemented.

RESULTS

In this study, 129 patients (male: 68.2%, female: 31.8%) were included. Ileostomy formation was associated with a total of 31% complication rate. Eventually 73.6% of the stomas were reversed at a mean time to closure of 26.6 weeks, with a morbidity of 13.7%. Non-reversal of ileostomy was correlated with neoadjuvant CRT (OR: 0.093, 95% CI: 0.012-0.735), anastomotic leakage (OR: 0.107, 95% CI: 0.019-0.610), and lymph node yield (OR: 0.946, 95% CI: 0.897-0.998). Time to reversal was affected by the N status, the LNR, the need for adjuvant chemotherapy, and the histologic grade.

CONCLUSION

In patients with rectal cancer resections, defunctioning stoma closure rate and time to closure were associated with several perioperative and pathological outcomes.

Easily Applicable Predictive Score for Differential Diagnosis of Prefibrotic Primary Myelofibrosis from Essential Thrombocythemia

Essential thrombocythemia (ET) and prefibrotic primary myelofibrosis (prePMF) initially have a similar phenotypic presentation with thrombocytosis. The aim of our study was to determine significant clinical-laboratory parameters at presentation to differentiate prePMF from ET as well as to develop and validate a predictive diagnostic prePMF model. This retrospective study included 464 patients divided into ET (289 pts) and prePMF (175 pts) groups. The model was built using data from a development cohort (229 pts; 143 ET, 86 prePMF), which was then tested in an internal validation cohort (235 pts; 146 ET, 89 prePMF). The most important prePMF predictors in the multivariate logistic model were age ≥ 60 years (RR = 2.2), splenomegaly (RR = 13.2), and increased lactat-dehidrogenase (RR = 2.8). Risk scores were assigned according to derived relative risk (RR) for age ≥ 60 years (1 point), splenomegaly (2 points), and increased lactat-dehidrogenase (1 point). Positive predictive value (PPV) for pre-PMF diagnosis with a score of ≥points was 69.8%, while for a score of ≥3 it was 88.2%. Diagnostic performance had similar values in the validation cohort. In MPN patients with thrombocytosis at presentation, the application of the new model enables differentiation of pre-PMF from ET, which is clinically relevant considering that these diseases have different prognoses and treatments.

Understanding the contribution of lifestyle in breast cancer risk prediction: a systematic review of models applicable to Europe

BACKGROUND

Breast cancer (BC) is a significant health concern among European women, with the highest prevalence rates among all cancers. Existing BC prediction models account for major risks such as hereditary, hormonal and reproductive factors, but research suggests that adherence to a healthy lifestyle can reduce the risk of developing BC to some extent. Understanding the influence and predictive role of lifestyle variables in current risk prediction models could help identify actionable, modifiable, targets among high-risk population groups.

PURPOSE

To systematically review population-based BC risk prediction models applicable to European populations and identify lifestyle predictors and their corresponding parameter values for a better understanding of their relative contribution to the prediction of incident BC.

METHODS

A systematic review was conducted in PubMed, Embase and Web of Science from January 2000 to August 2021. Risk prediction models were included if (i) developed and/or validated in adult cancer-free women in Europe, (ii) based on easily ascertained information, and (iii) reported models' final predictors. To investigate further the comparability of lifestyle predictors across models, estimates were standardised into risk ratios and visualised using forest plots.

RESULTS

From a total of 49 studies, 33 models were developed and 22 different existing models, mostly from Gail (22 studies) and Tyrer-Cuzick and co-workers (12 studies) were validated or modified for European populations. Family history of BC was the most frequently included predictor (31 models), while body mass index (BMI) and alcohol consumption (26 and 21 models, respectively) were the lifestyle predictors most often included, followed by smoking and physical activity (7 and 6 models respectively). Overall, for lifestyle predictors, their modest predictive contribution was greater for riskier lifestyle levels, though highly variable model estimates across different models.

CONCLUSIONS

Given the increasing BC incidence rates in Europe, risk models utilising readily available risk factors could greatly aid in widening the population coverage of screening efforts, while the addition of lifestyle factors could help improving model performance and serve as intervention targets of prevention programmes.

Systematic review finds "spin" practices and poor reporting standards in studies on machine learning-based prediction models

OBJECTIVES

We evaluated the presence and frequency of spin practices and poor reporting standards in studies that developed and/or validated clinical prediction models using supervised machine learning techniques.

STUDY DESIGN AND SETTING

We systematically searched PubMed from 01/2018 to 12/2019 to identify diagnostic and prognostic prediction model studies using supervised machine learning. No restrictions were placed on data source, outcome, or clinical specialty.

RESULTS

We included 152 studies: 38% reported diagnostic models and 62% prognostic models. When reported, discrimination was described without precision estimates in 53/71 abstracts (74.6% [95% CI 63.4-83.3]) and 53/81 main texts (65.4% [95% CI 54.6-74.9]). Of the 21 abstracts that recommended the model to be used in daily practice, 20 (95.2% [95% CI 77.3-99.8]) lacked any external validation of the developed models. Likewise, 74/133 (55.6% [95% CI 47.2-63.8]) studies made recommendations for clinical use in their main text without any external validation. Reporting guidelines were cited in 13/152 (8.6% [95% CI 5.1-14.1]) studies.

CONCLUSION

Spin practices and poor reporting standards are also present in studies on prediction models using machine learning techniques. A tailored framework for the identification of spin will enhance the sound reporting of prediction model studies.

Show Your Work: Responsible Model Reporting in Health Care Artificial Intelligence

Standardized and thorough model reporting is an integral component in the development and deployment of machine learning models in health care. Model reporting includes sharing multiple model performance metrics and incorporating metadata to provide the necessary context for model evaluation. Thorough model reporting addresses common concerns about artificial intelligence in health care including model explainability, transparency, fairness, and generalizability. All stages in the model development lifecycle, from initial design to data capture to model deployment, can be communicated openly to stakeholders with responsible model reporting. Physician involvement throughout these processes can ensure clinical concerns and potential consequences are considered.

Overinterpretation of findings in machine learning prediction model studies in oncology: a systematic review

OBJECTIVES

In biomedical research, spin is the overinterpretation of findings, and it is a growing concern. To date, the presence of spin has not been evaluated in prognostic model research in oncology, including studies developing and validating models for individualized risk prediction.

STUDY DESIGN AND SETTING

We conducted a systematic review, searching MEDLINE and EMBASE for oncology-related studies that developed and validated a prognostic model using machine learning published between 1st January, 2019, and 5th September, 2019. We used existing spin frameworks and described areas of highly suggestive spin practices.

RESULTS

We included 62 publications (including 152 developed models; 37 validated models). Reporting was inconsistent between methods and the results in 27% of studies due to additional analysis and selective reporting. Thirty-two studies (out of 36 applicable studies) reported comparisons between developed models in their discussion and predominantly used discrimination measures to support their claims (78%). Thirty-five studies (56%) used an overly strong or leading word in their title, abstract, results, discussion, or conclusion.

CONCLUSION

The potential for spin needs to be considered when reading, interpreting, and using studies that developed and validated prognostic models in oncology. Researchers should carefully report their prognostic model research using words that reflect their actual results and strength of evidence.

Systematic review identifies the design and methodological conduct of studies on machine learning-based prediction models

BACKGROUND AND OBJECTIVES

We sought to summarize the study design, modelling strategies, and performance measures reported in studies on clinical prediction models developed using machine learning techniques.

METHODS

We search PubMed for articles published between 01/01/2018 and 31/12/2019, describing the development or the development with external validation of a multivariable prediction model using any supervised machine learning technique. No restrictions were made based on study design, data source, or predicted patient-related health outcomes.

RESULTS

We included 152 studies, 58 (38.2% [95% CI 30.8-46.1]) were diagnostic and 94 (61.8% [95% CI 53.9-69.2]) prognostic studies. Most studies reported only the development of prediction models (n = 133, 87.5% [95% CI 81.3-91.8]), focused on binary outcomes (n = 131, 86.2% [95% CI 79.8-90.8), and did not report a sample size calculation (n = 125, 82.2% [95% CI 75.4-87.5]). The most common algorithms used were support vector machine (n = 86/522, 16.5% [95% CI 13.5-19.9]) and random forest (n = 73/522, 14% [95% CI 11.3-17.2]). Values for area under the Receiver Operating Characteristic curve ranged from 0.45 to 1.00. Calibration metrics were often missed (n = 494/522, 94.6% [95% CI 92.4-96.3]).

CONCLUSION

Our review revealed that focus is required on handling of missing values, methods for internal validation, and reporting of calibration to improve the methodological conduct of studies on machine learning-based prediction models.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO, CRD42019161764.

Machine learning model successfully identifies important clinical features for predicting outpatients with rotator cuff tears

PURPOSE

The aim of this study is to develop a machine learning model to identify important clinical features related to rotator cuff tears (RCTs) using explainable artificial intelligence (XAI) for efficiently predicting outpatients with RCTs.

METHODS

A retrospective review of a local clinical registry dataset was performed to include patients with shoulder pain and dysfunction who underwent questionnaires and physical examinations between 2019 and 2022. RCTs were diagnosed by shoulder arthroscopy. Six machine-learning algorithms (Stacking, Gradient Boosting Machine, Bagging, Random Forest, Extreme Gradient Boost (XGBoost), and Adaptive Boosting) were developed for the prediction. The performance of the models was assessed by the area under the receiver operating characteristic curve (AUC), Brier scores, and Decision curve. The interpretability of the predicted outcomes was evaluated using Shapley additive explanation (SHAP) values.

RESULTS

A total of 1684 patients who completed questionnaires and clinical tests were included, and 417 patients with RCTs underwent shoulder arthroscopy. In six machining learning algorithms for predicting RCTs, the accuracy, AUC values, and Brier scores were in the range of 0.81-0.86, 0.75-0.92, and 0.15-0.19, respectively. The XGBoost model showed superior performance with accuracy, AUC, and Brier scores of 0.85(95% confidence interval, 0.82-0.87), 0.92 (95% confidence interval,0.90-0.94), and 0.15 (95% confidence interval,0.14-0.16), respectively. The Shapley plot showed the impact of the clinical features on predicting RCTs. The most important variables were Jobe test, Bear hug test, and age for prediction, with mean SHAP values of 1.458, 0.950, and 0.790, respectively.

CONCLUSION

The machine learning model successfully identified important clinical variables for predicting patients with RCTs. In addition, the best algorithm was also integrated into a digital application to provide predictions in outpatient settings. This tool may assist patients in reducing their pain experience and providing prompt treatments.

LEVEL OF EVIDENCE

Level III.

Evolution of Making Clinical Predictions in Neurosurgery

Prediction of clinical outcomes is an essential task for every physician. Physicians may base their clinical prediction of an individual patient on their intuition and on scientific material such as studies presenting population risks and studies reporting on risk factors (prognostic factors). A relatively new and more informative approach for making clinical predictions relies on the use of statistical models that simultaneously consider multiple predictors that provide an estimate of the patient's absolute risk of an outcome. There is a growing body of literature in the neurosurgical field reporting on clinical prediction models. These tools have high potential in supporting (not replacing) neurosurgeons with their prediction of a patient's outcome. If used sensibly, these tools pave the way for more informed decision-making with or for individual patients. Patients and their significant others want to know their risk of the anticipated outcome, how it is derived, and the uncertainty associated with it. Learning from these prediction models and communicating the output to others has become an increasingly important skill neurosurgeons have to master. This article describes the evolution of making clinical predictions in neurosurgery, synopsizes key phases for the generation of a useful clinical prediction model, and addresses some considerations when deploying and communicating the results of a prediction model. The paper is illustrated with multiple examples from the neurosurgical literature, including predicting arachnoid cyst rupture, predicting rebleeding in patients suffering from aneurysmal subarachnoid hemorrhage, and predicting survival in glioblastoma patients.

Development and validation of a nomogram for esophagogastric variceal bleeding in liver cirrhosis: A cohort study in 1099 cases

OBJECTIVES

Esophagogastric variceal bleeding (EVB) is a catastrophic complication of decompensated liver cirrhosis. We aimed to establish a nomogram based on noninvasive clinical and imaging variables to predict the risk of EVB.

METHODS

The cut-off value of each variable was determined through univariate regression analysis. The least absolute shrinkage and selection operator (LASSO) regression and multivariate logistic regression analyses were used to determine the risk factors and establish predictive models. The nomogram was established and validated using the calibration discrimination across different groups.

RESULTS

Six indicators, including platelet count, hemoglobin, albumin to globulin ratio, fasting blood glucose, serum chloride, and computed tomography portal vein diameter (CTPD), were found to be related to the risk of EVB. Two models, with or without CTPD, were established and compared. Model 1 with CTPD had better discrimination than model 2 with C-index of 0.893 (95% confidence interval [CI] 0.872-0.915) and 0.862 (95% CI 0.837-0.887) in the primary cohort, respectively (Z = 2.027, P = 0.043). While the C-index of the two models in the validation cohort was 0.878 (95% CI 0.838-0.919) and 0.810 (95% CI 0.757-0.863). Moreover, the clinical decision analysis curve and clinical impact curve showed that these models might confer a significant net benefit on patients and provide a reference threshold for clinicians.

CONCLUSION

A nomogram using routine clinical indicators was established to predict the risk of EVB in patients with liver cirrhosis, which was verified in an independent cohort and demonstrated a great consistency.

Development and external validation of a clinical prediction model for survival in patients with IDH wild-type glioblastoma

OBJECTIVE

Prognostication of glioblastoma survival has become more refined due to the molecular reclassification of these tumors into isocitrate dehydrogenase (IDH) wild-type and IDH mutant. Since this molecular stratification, however, robust clinical prediction models relevant to the entire IDH wild-type glioblastoma patient population are lacking. This study aimed to provide an updated model that predicts individual survival prognosis in patients with IDH wild-type glioblastoma.

METHODS

Databases from Germany and the Netherlands provided data on 1036 newly diagnosed glioblastoma patients treated between 2012 and 2018. A clinical prediction model for all-cause mortality was developed with Cox proportional hazards regression. This model included recent glioblastoma-associated molecular markers in addition to well-known classic prognostic variables, which were updated and refined with additional categories. Model performance was evaluated according to calibration (using calibration plots and calibration slope) and discrimination (using a C-statistic) in a cross-validation procedure by country to assess external validity.

RESULTS

The German and Dutch patient cohorts consisted of 710 and 326 patients, respectively, of whom 511 (72%) and 308 (95%) had died. Three models were developed, each with increasing complexity. The final model considering age, sex, preoperative Karnofsky Performance Status, extent of resection, O6-methylguanine DNA methyltransferase (MGMT) promoter methylation status, and adjuvant therapeutic regimen showed an optimism-corrected C-statistic of 0.73 (95% confidence interval 0.71-0.75). Cross-validation between the national cohorts yielded comparable results.

CONCLUSIONS

This prediction model reliably predicts individual survival prognosis in patients with newly diagnosed IDH wild-type glioblastoma, although additional validation, especially for long-term survival, may be desired. The nomogram and web application of this model may support shared decision-making if used properly.

Multivariable Prediction Model for Futile Recanalization Therapies in Patients With Acute Ischemic Stroke

BACKGROUND AND OBJECTIVES

Very poor outcome despite IV thrombolysis (IVT) and mechanical thrombectomy (MT) occurs in approximately 1 of 4 patients with ischemic stroke and is associated with a high logistic and economic burden. We aimed to develop and validate a multivariable prognostic model to identify futile recanalization therapies (FRTs) in patients undergoing those therapies.

METHODS

Patients from a prospectively collected observational registry of a single academic stroke center treated with MT and/or IVT were included. The data set was split into a training (N = 1,808, 80%) and internal validation (N = 453, 20%) cohort. We used gradient boosted decision tree machine learning models after k-nearest neighbor imputation of 32 variables available at admission to predict FRT defined as modified Rankin scale 5-6 at 3 months. We report feature importance, ability for discrimination, calibration, and decision curve analysis.

RESULTS

A total of 2,261 patients with a median (interquartile range) age of 75 years (64-83 years), 46% female, median NIH Stroke Scale 9 (4-17), 34% IVT alone, 41% MT alone, and 25% bridging were included. Overall, 539 (24%) had FRT, more often in MT alone (34%) as compared with IVT alone (11%). Feature importance identified clinical variables (stroke severity, age, active cancer, prestroke disability), laboratory values (glucose, C-reactive protein, creatinine), imaging biomarkers (white matter hyperintensities), and onset-to-admission time as the most important predictors. The final model was discriminatory for predicting 3-month FRT (area under the curve 0.87, 95% CI 0.87-0.88) and had good calibration (Brier 0.12, 0.11-0.12). Overall performance was moderate (F1-score 0.63 ± 0.004), and decision curve analyses suggested higher mean net benefit at lower thresholds of treatment (up to 0.8).

CONCLUSIONS

This FRT prediction model can help inform shared decision making and identify the most relevant features in the emergency setting. Although it might be particularly useful in low resource healthcare settings, incorporation of further multifaceted variables is necessary to further increase the predictive performance.

Development and Internal Validation of the ARISE Prediction Models for Rebleeding After Aneurysmal Subarachnoid Hemorrhage

BACKGROUND

Aneurysmal rerupture is one of the most important determents for outcome after aneurysmal subarachnoid hemorrhage and still occurs frequently because individual risk assessment is challenging given the heterogeneity in patient characteristics and aneurysm morphology.

OBJECTIVE

To develop and internally validate a practical prediction model to estimate the risk of aneurysmal rerupture before aneurysm closure.

METHODS

We designed a multinational cohort study of 2 prospective hospital registries and 3 retrospective observational studies to predict the risk of computed tomography confirmed rebleeding within 24 and 72 hours after ictus. We assessed predictors with Cox proportional hazard regression analysis.

RESULTS

Rerupture occurred in 269 of 2075 patients. The cumulative incidence equaled 7% and 11% at 24 and 72 hours, respectively. Our base model included hypertension, World Federation of Neurosurgical Societies scale, Fisher grade, aneurysm size, and cerebrospinal fluid drainage before aneurysm closure and showed good discrimination with an optimism corrected c-statistic of 0.77. When we extend the base model with aneurysm irregularity, the optimism-corrected c-statistic increased to 0.79.

CONCLUSION

Our prediction models reliably estimate the risk of aneurysm rerupture after aneurysmal subarachnoid hemorrhage using predictor variables available upon hospital admission. An online prognostic calculator is accessible at https://www.evidencio.com/models/show/2626 .

Development and Internal Validation of a New Prognostic Model Powered to Predict 28-Day All-Cause Mortality in ICU COVID-19 Patients-The COVID-SOFA Score

Background: The sequential organ failure assessment (SOFA) score has poor discriminative ability for death in severely or critically ill patients with Coronavirus disease 2019 (COVID-19) requiring intensive care unit (ICU) admission. Our aim was to create a new score powered to predict 28-day mortality. Methods: Retrospective, observational, bicentric cohort study including 425 patients with COVID-19 pneumonia, acute respiratory failure and SOFA score ≥ 2 requiring ICU admission for ≥72 h. Factors with independent predictive value for 28-day mortality were identified after stepwise Cox proportional hazards (PH) regression. Based on the regression coefficients, an equation was computed representing the COVID-SOFA score. Discriminative ability was tested using receiver operating characteristic (ROC) analysis, concordance statistics and precision-recall curves. This score was internally validated. Results: Median (Q1−Q3) age for the whole sample was 64 [55−72], with 290 (68.2%) of patients being male. The 28-day mortality was 54.58%. After stepwise Cox PH regression, age, neutrophil-to-lymphocyte ratio (NLR) and SOFA score remained in the final model. The following equation was computed: COVID-SOFA score = 10 × [0.037 × Age + 0.347 × ln(NLR) + 0.16 × SOFA]. Harrell’s C-index for the COVID-SOFA score was higher than the SOFA score alone for 28-day mortality (0.697 [95% CI; 0.662−0.731] versus 0.639 [95% CI: 0.605−0.672]). Subsequently, the prediction error rate was improved up to 16.06%. Area under the ROC (AUROC) was significantly higher for the COVID-SOFA score compared with the SOFA score for 28-day mortality: 0.796 [95% CI: 0.755−0.833] versus 0.699 [95% CI: 0.653−0.742, p < 0.001]. Better predictive value was observed with repeated measurement at 48 h after ICU admission. Conclusions: The COVID-SOFA score is better than the SOFA score alone for 28-day mortality prediction. Improvement in predictive value seen with measurements at 48 h after ICU admission suggests that the COVID-SOFA score can be used in a repetitive manner. External validation is required to support these results.

TAGE Score for Symptomatic Intracranial Hemorrhage Prediction After Successful Endovascular Treatment in Acute Ischemic Stroke

BACKGROUND

Determine if early venous filling (EVF) after complete successful recanalization with mechanical thrombectomy in acute ischemic stroke is an independent predictor of symptomatic intracranial hemorrhage (sICH) and integrate EVF into a risk score for sICH prediction.

METHODS

Consecutive patients with anterior acute ischemic stroke treated by mechanical thrombectomy issued from patients enrolled in the THRACE trial (Thrombectomie des Artères Cérébrales) and from 2 prospective registries were included and divided into a derivation (Center I; n=402) and validation cohorts (THRACE and center 2; n=507). EVF was evaluated by 2 blinded readers. sICH was defined according to the modified European cooperative acute stroke study II. Clinical and radiological data were analyzed in the derivation cohort (C1) to identify independent predictors of sICH and construct a predictive score test on the validation cohort (THRACE + C2).

RESULTS

Symptomatic ICH rate was similar between the two cohorts (9.9% and 8.9% respectively, P=0.9). Time from onset-to-successful recanalization >270 minutes (odds ratio [OR], 7.8 [95% CI, 2.5-24]), Alberta Stroke Program Early CT Score (≤5 [OR, 2.49 (95% CI, 1.8-8.1) or 6-7 [OR, 1.15 (95% CI, 1.03-4.46)]), glucose blood level >7 mmol/L (OR, 2.92 [95% CI, 1.26-6.7]), and EVF presence (OR, 11.9 [95% CI, 3.8-37.5]) were independent predictors of sICH and constituted the Time-Alberta Stroke Program Early CT-Glycemia-EVF score. Time-Alberta Stroke Program Early CT-Glycemia-EVF score was associated with an increased risk of sICH in the derivation cohort (OR increase per unit, 1.99 [95% CI, 1.53-2.59]; P<0.001) with area under the curve, 0.832 [95% CI, 0.767-0.898]. The score had good performance in the validation cohort (area under the curve, 0.801 [95% CI, 0.69-0.91]).

CONCLUSIONS

Time-Alberta Stroke Program Early CT-Glycemia-EVF score is a simple tool with readily available clinical variables with good performances for sICH prediction after mechanical thrombectomy.

REGISTRATION

URL: https://www.

CLINICALTRIALS

gov; Unique identifier: NCT01062698.

External validation of two prediction models for adequate bowel preparation in Asia: a prospective study

BACKGROUND

Several models for predicting adequate bowel preparation are available but have never been externally validated. The aim of this study is to compare the available models in an independent population.

METHODS

This study prospectively recruited 500 consecutive patients from August to December 2020 from the Endoscopy Center of a tertiary hospital. All patients underwent the same bowel preparation regimen. The discrimination of the prediction models was quantified with the area under the receiver operating characteristic curve (AUC), and the 95% confidence interval (CI) was calculated for each AUC.

RESULTS

Finally, 461 patients were eligible for this study. A total of 110 (23.9%) patients were deemed to show inadequate bowel preparation during colonoscopy. There were significant differences between patients with and without adequate bowel preparation in terms of current hospitalization, procedure time, comorbidities (including diabetes and constipation), American Society of Anesthesiologists Physical Status Classification System score (ASA) ≥ 3, medication usage, and abdominal/pelvic surgery. The prediction models performed as follows: the Dik ≥ 2 model, the Dik ≥ 3 model, and the Antonio > 1.225 model had AUCs of 0.660 (95% CI = 0.604-0.717), 0.691 (95% CI = 0.646-0.733), and 0.645 (95% CI = 0.615-0.704), respectively. Comparison of the two prediction models showed no significant improvement (Antonio > 1.225 vs. Dik ≥ 3, 1.801, 95% CI =  -0.004-0.096, P = 0.072).

CONCLUSIONS

Both models are potentially helpful. However, it is necessary to develop or improve a prediction model to obtain a more suitable and detailed model.

TRIAL REGISTRATION

ClinicalTrials.gov, Number NCT04607161.

Reporting Quality of Studies Developing and Validating Melanoma Prediction Models: An Assessment Based on the TRIPOD Statement

Transparent and accurate reporting is essential to evaluate the validity and applicability of risk prediction models. Our aim was to evaluate the reporting quality of studies developing and validating risk prediction models for melanoma according to the TRIPOD (Transparent Reporting of a multivariate prediction model for Individual Prognosis Or Diagnosis) checklist. We included studies that were identified by a recent systematic review and updated the literature search to ensure that our TRIPOD rating included all relevant studies. Six reviewers assessed compliance with all 37 TRIPOD components for each study using the published “TRIPOD Adherence Assessment Form”. We further examined a potential temporal effect of the reporting quality. Altogether 42 studies were assessed including 35 studies reporting the development of a prediction model and seven studies reporting both development and validation. The median adherence to TRIPOD was 57% (range 29% to 78%). Study components that were least likely to be fully reported were related to model specification, title and abstract. Although the reporting quality has slightly increased over the past 35 years, there is still much room for improvement. Adherence to reporting guidelines such as TRIPOD in the publication of study results must be adopted as a matter of course to achieve a sufficient level of reporting quality necessary to foster the use of the prediction models in applications.

Inpatient Delirium in Guillain-Barré Syndrome: Frequency and Clinical Characteristics in a Mexican Hospital

Background

Delirium has a prevalence of 14%-56% in hospitalized patients. Risk factors include advanced age, invasive mechanical ventilation (IMV), and prolonged intensive care unit stay. Neuropsychiatric symptoms have been reported to be related to autoimmune responses secondary to Guillain-Barré syndrome (GBS) with direct involvement of the central nervous system (CNS) or to delirium. There are few reports of the frequency of delirium in patients with Guillain-Barré syndrome (GBS).

Objective

To describe the frequency of and the characteristics associated with delirium in patients with GBS.

Material and methods

A single-center, cross-sectional study was conducted in patients with GBS diagnosis between 2015 and 2019. The diagnosis of delirium was made using the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5) criteria. We compared patients with and without delirium. We performed both univariate and multivariate analyses to identify factors associated with delirium.

Results

A total of 154 patients with GBS were included, of which 20 (12.9%) fulfilled the DSM-5 criteria for delirium. The mean age was 48 ± 18.2 years, the median Medical Research Council (MRC) sum score was 29.3 ± 21.9 points, 65% had bulbar cranial nerve involvement, 80% presented autonomic dysfunction, 85% had ICU stay, and 90% had mechanical ventilation requirement. In the multivariate analysis, the following were the independent factors for the development of delirium: age ≥ 60 (odds ratio (OR): 5.7; 95% confidence interval (CI): 1.3-23.5), time from symptom onset to admission ≤ 3 days (OR: 4.3; 95% CI: 1.1-16.8), autonomic dysfunction (OR: 13.1; 95% CI: 3-56), and intensive care unit stay (OR: 9.5; 95% CI: 2.1-42.6).

Conclusion

Delirium is not frequent in patients with Guillain-Barré syndrome. Patients with advanced age, rapid motor progression, bulbar cranial nerve involvement, prolonged intensive care unit stay, and mechanical ventilation need are more likely to present delirium.

Expert Panel Consensus on Management of Advanced Cancer-Related Pain in Individuals With Opioid Use Disorder

IMPORTANCE

Opioid use disorder (OUD) is an important comorbidity in individuals with advanced cancer, in whom pain is common. Full-agonist opioid medications are the cornerstone of cancer pain management, but the existing literature does not address how to manage cancer pain in patients with OUD.

OBJECTIVE

To conduct an expert panel to develop consensus on the appropriateness of management of cancer pain in individuals with co-occurring advanced cancer and OUD.

EVIDENCE REVIEW

A 3-round modified Delphi process was completed from August to October 2020 with 2 cases: patient with advanced cancer, pain, and OUD treated with buprenorphine-naloxone or methadone. Participants rated management strategies in round 1, discussed results in round 2, and provided final responses in round 3. ExpertLens, an online approach to conducting modified Delphi panels, was used. Participants were experts in palliative care, addiction, or both, recruited by email from palliative care and addiction-focused professional groups, lists from prior studies, and snowball sampling. Data analysis was performed from November 2020 to July 2021.

FINDINGS

Of 120 experts (median age, 40-49 years), most were White (78 participants [94%]), female (74 participants [62%]), and held MD or DO degrees (115 participants [96%]); 84 (70%) participated in all rounds. For a patient with OUD taking buprenorphine-naloxone, it was deemed appropriate to continue buprenorphine-naloxone with thrice-daily dosing. Continuing buprenorphine-naloxone and adding a full-agonist opioid was deemed to be appropriate for patients with a prognosis of weeks to months and of uncertain appropriateness for patients with a prognosis of months to years. For a patient with OUD taking methadone dispensed at a methadone clinic, it was deemed appropriate to take over prescribing and dose twice or thrice daily. Continuing methadone daily while adding another full-agonist opioid was deemed appropriate for patients with a prognosis of weeks to months and of uncertain appropriateness for those with a prognosis of months to years.

CONCLUSIONS AND RELEVANCE

The findings of this qualitative study provide urgently needed, consensus-based guidance for clinicians and highlight critical research and policy gaps needed to facilitate implementation.

Reporting of prognostic clinical prediction models based on machine learning methods in oncology needs to be improved

OBJECTIVE

Evaluate the completeness of reporting of prognostic prediction models developed using machine learning methods in the field of oncology.

STUDY DESIGN AND SETTING

We conducted a systematic review, searching the MEDLINE and Embase databases between 01/01/2019 and 05/09/2019, for non-imaging studies developing a prognostic clinical prediction model using machine learning methods (as defined by primary study authors) in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement to assess the reporting quality of included publications. We described overall reporting adherence of included publications and by each section of TRIPOD.

RESULTS

Sixty-two publications met the inclusion criteria. 48 were development studies and 14 were development with validation studies. 152 models were developed across all publications. Median adherence to TRIPOD reporting items was 41% [range: 10%-67%] and at least 50% adherence was found in 19% (n=12/62) of publications. Adherence was lower in development only studies (median: 38% [range: 10%-67%]); and higher in development with validation studies (median: 49% [range: 33%-59%]).

CONCLUSION

Reporting of clinical prediction models using machine learning in oncology is poor and needs urgent improvement, so readers and stakeholders can appraise the study methods, understand study findings, and reduce research waste.

Development and Validation of a Practical Model to Identify Patients at Risk of Bleeding After TAVR

OBJECTIVES

No standardized algorithm exists to identify patients at risk of bleeding after transcatheter aortic valve replacement (TAVR). The aim of this study was to generate and validate a useful predictive model.

BACKGROUND

Bleeding events after TAVR influence prognosis and quality of life and may be preventable.

METHODS

Using machine learning and multivariate regression, more than 100 clinical variables from 5,185 consecutive patients undergoing TAVR in the prospective multicenter RISPEVA (Registro Italiano GISE sull'Impianto di Valvola Aortica Percutanea; NCT02713932) registry were analyzed in relation to Valve Academic Research Consortium-2 bleeding episodes at 1 month. The model's performance was externally validated in 5,043 TAVR patients from the prospective multicenter POL-TAVI (Polish Registry of Transcatheter Aortic Valve Implantation) database.

RESULTS

Derivation analyses generated a 6-item score (PREDICT-TAVR) comprising blood hemoglobin and serum iron concentrations, oral anticoagulation and dual antiplatelet therapy, common femoral artery diameter, and creatinine clearance. The 30-day area under the receiver-operating characteristic curve (AUC) was 0.80 (95% confidence interval [CI]: 0.75-0.83). Internal validation by optimism bootstrap-corrected AUC was 0.79 (95% CI: 0.75-0.83). Score quartiles were in graded relation to 30-day events (0.8%, 1.1%, 2.5%, and 8.5%; overall p <0.001). External validation produced a 30-day AUC of 0.78 (95% CI: 0.72-0.82). A simple nomogram and a web-based calculator were developed to predict individual patient probabilities. Landmark cumulative event analysis showed greatest bleeding risk differences for top versus lower score quartiles in the first 30 days, when most events occurred. Predictivity was maintained when omitting serum iron values.

CONCLUSIONS

PREDICT-TAVR is a practical, validated, 6-item tool to identify patients at risk of bleeding post-TAVR that can assist in decision making and event prevention.

Preferred reporting items for journal and conference abstracts of systematic reviews and meta-analyses of diagnostic test accuracy studies (PRISMA-DTA for Abstracts): checklist, explanation, and elaboration

For many users of the biomedical literature, abstracts may be the only source of information about a study. Hence, abstracts should allow readers to evaluate the objectives, key design features, and main results of the study. Several evaluations have shown deficiencies in the reporting of journal and conference abstracts across study designs and research fields, including systematic reviews of diagnostic test accuracy studies. Incomplete reporting compromises the value of research to key stakeholders. The authors of this article have developed a 12 item checklist of preferred reporting items for journal and conference abstracts of systematic reviews and meta-analyses of diagnostic test accuracy studies (PRISMA-DTA for Abstracts). This article presents the checklist, examples of complete reporting, and explanations for each item of PRISMA-DTA for Abstracts.

Development and validation of an online model to predict critical COVID-19 with immune-inflammatory parameters

BACKGROUND

Immune and inflammatory dysfunction was reported to underpin critical COVID-19(coronavirus disease 2019). We aim to develop a machine learning model that enables accurate prediction of critical COVID-19 using immune-inflammatory features at admission.

METHODS

We retrospectively collected 2076 consecutive COVID-19 patients with definite outcomes (discharge or death) between January 27, 2020 and March 30, 2020 from two hospitals in China. Critical illness was defined as admission to intensive care unit, receiving invasive ventilation, or death. Least Absolute Shrinkage and Selection Operator (LASSO) was applied for feature selection. Five machine learning algorithms, including Logistic Regression (LR), Support Vector Machine (SVM), Gradient Boosted Decision Tree (GBDT), K-Nearest Neighbor (KNN), and Neural Network (NN) were built in a training dataset, and assessed in an internal validation dataset and an external validation dataset.

RESULTS

Six features (procalcitonin, [T + B + NK cell] count, interleukin 6, C reactive protein, interleukin 2 receptor, T-helper lymphocyte/T-suppressor lymphocyte) were finally used for model development. Five models displayed varying but all promising predictive performance. Notably, the ensemble model, SPMCIIP (severity prediction model for COVID-19 by immune-inflammatory parameters), derived from three contributive algorithms (SVM, GBDT, and NN) achieved the best performance with an area under the curve (AUC) of 0.991 (95% confidence interval [CI] 0.979-1.000) in internal validation cohort and 0.999 (95% CI 0.998-1.000) in external validation cohort to identify patients with critical COVID-19. SPMCIIP could accurately and expeditiously predict the occurrence of critical COVID-19 approximately 20 days in advance.

CONCLUSIONS

The developed online prediction model SPMCIIP is hopeful to facilitate intensive monitoring and early intervention of high risk of critical illness in COVID-19 patients.

TRIAL REGISTRATION

This study was retrospectively registered in the Chinese Clinical Trial Registry ( ChiCTR2000032161 ). vv.

Validation of manufacturers' laryngeal mask airway size selection standard: a large retrospective study

Background

Laryngeal mask airway (LMA) is a prominent supraglottic airway device, widely used especially in difficult airway management. However, the LMA sizes recommended by the manufacturers are not always well matched in clinical practice, which leads to complications. To date, there are rare models to validate whether the manufacturers’ standard is suitable for use in clinical practice.

Methods

A total of 58,956 patients undergoing general anesthesia using LMA device were included in the study between January 1, 2011 and December 31, 2018, to validate the adherence rate of LMA sizes according to the manufacturers’ recommendations. A logistic regression analysis was performed based on the actual LMA size used in clinical practice to establish separately size selection guidelines with gender, weight, and age as variables in adults, adolescents, and children.

Results

LMA insertions were analyzed in 50,776 (86.1%) adults, 3,548 (6%) adolescents, and 4,632 (7.9%) children. Suitability of manufacturers’ recommendations was higher in children [male: 86.02%; female: 85.09%] than adults [male: 72.75%; female: 78.13%] or adolescents [male: 73.4%; female: 70.79%]. For adults and adolescents, LMA size was better predicted using the regression model rather than the manufacturers’ recommendations [male adults: 82.4% (81.16–83.57%) vs. 73.21% (71.79–74.59%), P<0.05; female adults: 87.82% (86.65–88.9%) vs. 77.07% (75.6–78.48%), P<0.05; male adolescents: 79.45% (74.86–83.4%) vs. 72.05% (67.09–76.53%), P<0.05; female adolescents: 78.4% (71.11–84.31%) vs. 72.22% (64.54–78.82%), P<0.05]. For children, there was equal performance suitability using the regression model and the manufacturers’ recommendations.

Conclusions

The model-based guidelines may provide more accurate directions for LMA size selection for adolescents and adults than the manufacturers’ weight-based recommendations, whereas the manufacturers’ recommendation in children is consistent with clinical practice.

Application of clinical prediction modeling in pediatric neurosurgery: a case study

There has been an increasing interest in articles reporting on clinical prediction models in pediatric neurosurgery. Clinical prediction models are mathematical equations that combine patient-related risk factors for the estimation of an individual's risk of an outcome. If used sensibly, these evidence-based tools may help pediatric neurosurgeons in medical decision-making processes. Furthermore, they may help to communicate anticipated future events of diseases to children and their parents and facilitate shared decision-making accordingly. A basic understanding of this methodology is incumbent when developing or applying a prediction model. This paper addresses this methodology tailored to pediatric neurosurgery. For illustration, we use original pediatric data from our institution to illustrate this methodology with a case study. The developed model is however not externally validated, and clinical impact has not been assessed; therefore, the model cannot be recommended for clinical use in its current form.

Development and Validation of a New Clinical Prediction Model of Catheter-Related Thrombosis Based on Vascular Ultrasound Diagnosis in Cancer Patients

Background: Central venous catheters are convenient for drug delivery and improved comfort for cancer patients, but they also cause serious complications. The most common complication is catheter-related thrombosis (CRT). Objectives: This study aimed to evaluate the incidence and risk factors for CRT in cancer patients and develop an effective prediction model for CRT in cancer patients. Methods: The development of our prediction model was based on a retrospective cohort (n = 3,131) from the National Cancer Center. Our prediction model was confirmed in a prospective cohort from the National Cancer Center (n = 685) and a retrospective cohort from the Hunan Cancer Hospital (n = 61). The predictive accuracy and discriminative ability were determined by receiver operating characteristic (ROC) curves and calibration plots. Results: Multivariate analysis demonstrated that sex, cancer type, catheter type, position of the catheter tip, chemotherapy status, and antiplatelet/anticoagulation status at baseline were independent risk factors for CRT. The area under the ROC curve of our prediction model was 0.741 (CI: 0.715-0.766) in the primary cohort and 0.754 (CI: 0.704-0.803) and 0.658 (CI: 0.470-0.845) in validation cohorts 1 and 2, respectively. The model also showed good calibration and clinical impact in the primary and validation cohorts. Conclusions: Our model is a novel prediction tool for CRT risk that accurately assigns cancer patients into high- and low-risk groups. Our model will be valuable for clinicians when making decisions regarding thromboprophylaxis.