Prediction of survival with alternative modeling techniques using pseudo values

Predicting the risk of pancreatic cancer in adults with new-onset diabetes: development and internal-external validation of a clinical risk prediction model

BACKGROUND

The National Institute for Health and Care Excellence (NICE) recommends that people aged 60+ years with newly diagnosed diabetes and weight loss undergo abdominal imaging to assess for pancreatic cancer. More nuanced stratification could lead to enrichment of these referral pathways.

METHODS

Population-based cohort study of adults aged 30-85 years at type 2 diabetes diagnosis (2010-2021) using the QResearch primary care database in England linked to secondary care data, the national cancer registry and mortality registers. Clinical prediction models were developed to estimate risks of pancreatic cancer diagnosis within 2 years and evaluated using internal-external cross-validation.

RESULTS

Seven hundred and sixty-seven of 253,766 individuals were diagnosed with pancreatic cancer within 2 years. Models included age, sex, BMI, prior venous thromboembolism, digoxin prescription, HbA1c, ALT, creatinine, haemoglobin, platelet count; and the presence of abdominal pain, weight loss, jaundice, heartburn, indigestion or nausea (previous 6 months). The Cox model had the highest discrimination (Harrell's C-index 0.802 (95% CI: 0.797-0.817)), the highest clinical utility, and was well calibrated. The model's highest 1% of predicted risks captured 12.51% of pancreatic cancer cases. NICE guidance had 3.95% sensitivity.

DISCUSSION

A new prediction model could have clinical utility in identifying individuals with recent onset diabetes suitable for fast-track abdominal imaging.

Pseudo-value regression trees

This paper presents a semi-parametric modeling technique for estimating the survival function from a set of right-censored time-to-event data. Our method, named pseudo-value regression trees (PRT), is based on the pseudo-value regression framework, modeling individual-specific survival probabilities by computing pseudo-values and relating them to a set of covariates. The standard approach to pseudo-value regression is to fit a main-effects model using generalized estimating equations (GEE). PRT extend this approach by building a multivariate regression tree with pseudo-value outcome and by successively fitting a set of regularized additive models to the data in the nodes of the tree. Due to the combination of tree learning and additive modeling, PRT are able to perform variable selection and to identify relevant interactions between the covariates, thereby addressing several limitations of the standard GEE approach. In addition, PRT include time-dependent effects in the node-wise models. Interpretability of the PRT fits is ensured by controlling the tree depth. Based on the results of two simulation studies, we investigate the properties of the PRT method and compare it to several alternative modeling techniques. Furthermore, we illustrate PRT by analyzing survival in 3,652 patients enrolled for a randomized study on primary invasive breast cancer.

Predicting 10-year breast cancer mortality risk in the general female population in England: a model development and validation study

BACKGROUND

Identifying female individuals at highest risk of developing life-threatening breast cancers could inform novel stratified early detection and prevention strategies to reduce breast cancer mortality, rather than only considering cancer incidence. We aimed to develop a prognostic model that accurately predicts the 10-year risk of breast cancer mortality in female individuals without breast cancer at baseline.

METHODS

In this model development and validation study, we used an open cohort study from the QResearch primary care database, which was linked to secondary care and national cancer and mortality registers in England, UK. The data extracted were from female individuals aged 20-90 years without previous breast cancer or ductal carcinoma in situ who entered the cohort between Jan 1, 2000, and Dec 31, 2020. The primary outcome was breast cancer-related death, which was assessed in the full dataset. Cox proportional hazards, competing risks regression, XGBoost, and neural network modelling approaches were used to predict the risk of breast cancer death within 10 years using routinely collected health-care data. Death due to causes other than breast cancer was the competing risk. Internal-external validation was used to evaluate prognostic model performance (using Harrell's C, calibration slope, and calibration in the large), performance heterogeneity, and transportability. Internal-external validation involved dataset partitioning by time period and geographical region. Decision curve analysis was used to assess clinical utility.

FINDINGS

We identified data for 11 626 969 female individuals, with 70 095 574 person-years of follow-up. There were 142 712 (1·2%) diagnoses of breast cancer, 24 043 (0·2%) breast cancer-related deaths, and 696 106 (6·0%) deaths from other causes. Meta-analysis pooled estimates of Harrell's C were highest for the competing risks model (0·932, 95% CI 0·917-0·946). The competing risks model was well calibrated overall (slope 1·011, 95% CI 0·978-1·044), and across different ethnic groups. Decision curve analysis suggested favourable clinical utility across all age groups. The XGBoost and neural network models had variable performance across age and ethnic groups.

INTERPRETATION

A model that predicts the combined risk of developing and then dying from breast cancer at the population level could inform stratified screening or chemoprevention strategies. Further evaluation of the competing risks model should comprise effect and health economic assessment of model-informed strategies.

FUNDING

Cancer Research UK.

Development and internal-external validation of statistical and machine learning models for breast cancer prognostication: cohort study

OBJECTIVE

To develop a clinically useful model that estimates the 10 year risk of breast cancer related mortality in women (self-reported female sex) with breast cancer of any stage, comparing results from regression and machine learning approaches.

DESIGN

Population based cohort study.

SETTING

QResearch primary care database in England, with individual level linkage to the national cancer registry, Hospital Episodes Statistics, and national mortality registers.

PARTICIPANTS

141 765 women aged 20 years and older with a diagnosis of invasive breast cancer between 1 January 2000 and 31 December 2020.

MAIN OUTCOME MEASURES

Four model building strategies comprising two regression (Cox proportional hazards and competing risks regression) and two machine learning (XGBoost and an artificial neural network) approaches. Internal-external cross validation was used for model evaluation. Random effects meta-analysis that pooled estimates of discrimination and calibration metrics, calibration plots, and decision curve analysis were used to assess model performance, transportability, and clinical utility.

RESULTS

During a median 4.16 years (interquartile range 1.76-8.26) of follow-up, 21 688 breast cancer related deaths and 11 454 deaths from other causes occurred. Restricting to 10 years maximum follow-up from breast cancer diagnosis, 20 367 breast cancer related deaths occurred during a total of 688 564.81 person years. The crude breast cancer mortality rate was 295.79 per 10 000 person years (95% confidence interval 291.75 to 299.88). Predictors varied for each regression model, but both Cox and competing risks models included age at diagnosis, body mass index, smoking status, route to diagnosis, hormone receptor status, cancer stage, and grade of breast cancer. The Cox model's random effects meta-analysis pooled estimate for Harrell's C index was the highest of any model at 0.858 (95% confidence interval 0.853 to 0.864, and 95% prediction interval 0.843 to 0.873). It appeared acceptably calibrated on calibration plots. The competing risks regression model had good discrimination: pooled Harrell's C index 0.849 (0.839 to 0.859, and 0.821 to 0.876, and evidence of systematic miscalibration on summary metrics was lacking. The machine learning models had acceptable discrimination overall (Harrell's C index: XGBoost 0.821 (0.813 to 0.828, and 0.805 to 0.837); neural network 0.847 (0.835 to 0.858, and 0.816 to 0.878)), but had more complex patterns of miscalibration and more variable regional and stage specific performance. Decision curve analysis suggested that the Cox and competing risks regression models tested may have higher clinical utility than the two machine learning approaches.

CONCLUSION

In women with breast cancer of any stage, using the predictors available in this dataset, regression based methods had better and more consistent performance compared with machine learning approaches and may be worthy of further evaluation for potential clinical use, such as for stratified follow-up.

Rational Design of a Self-Assembling High Performance Organic Nanofluorophore for Intraoperative NIR-II Image-Guided Tumor Resection of Oral Cancer

The first line of treatment for most solid tumors is surgical resection of the primary tumor with adequate negative margins. Incomplete tumor resections with positive margins account for over 75% of local recurrences and the development of distant metastases. In cases of oral cavity squamous cell carcinoma (OSCC), the rate of successful tumor removal with adequate margins is just 50-75%. Advanced real-time imaging methods that improve the detection of tumor margins can help improve success rates,overall safety, and reduce the cost. Fluorescence imaging in the second near-infrared (NIR-II) window has the potential to revolutionize the field due to its high spatial resolution, low background signal, and deep tissue penetration properties, but NIR-II dyes with adequate in vivo performance and safety profiles are scarce. A novel NIR-II fluorophore, XW-03-66, with a fluorescence quantum yield (QY) of 6.0% in aqueous media is reported. XW-03-66 self-assembles into nanoparticles (≈80 nm) and has a systemic circulation half-life (t_1/2 ) of 11.3 h. In mouse models of human papillomavirus (HPV)+ and HPV- OSCC, XW-03-66 outperformed indocyanine green (ICG), a clinically available NIR dye, and enabled intraoperative NIR-II image-guided resection of the tumor and adjacent draining lymph node with negative margins. In vitro and in vivo toxicity assessments revealed minimal safety concerns for in vivo applications.

The Utility of Oncology Information Systems for Prognostic Modelling in Head and Neck Cancer

Cancer centres rely on electronic information in oncology information systems (OIS) to guide patient care. We investigated the completeness and accuracy of routinely collected head and neck cancer (HNC) data sourced from an OIS for suitability in prognostic modelling and other research. Three hundred and fifty-three adults diagnosed from 2000 to 2017 with head and neck squamous cell carcinoma, treated with radiotherapy, were eligible. Thirteen clinically relevant variables in HNC prognosis were extracted from a single-centre OIS and compared to that compiled separately in a research dataset. These two datasets were compared for agreement using Cohen's kappa coefficient for categorical variables, and intraclass correlation coefficients for continuous variables. Research data was 96% complete compared to 84% for OIS data. Agreement was perfect for gender (κ = 1.000), high for age (κ = 0.993), site (κ = 0.992), T (κ = 0.851) and N (κ = 0.812) stage, radiotherapy dose (κ = 0.889), fractions (κ = 0.856), and duration (κ = 0.818), and chemotherapy treatment (κ = 0.871), substantial for overall stage (κ = 0.791) and vital status (κ = 0.689), moderate for grade (κ = 0.547), and poor for performance status (κ = 0.110). Thirty-one other variables were poorly captured and could not be statistically compared. Documentation of clinical information within the OIS for HNC patients is routine practice; however, OIS data was less correct and complete than data collected for research purposes. Substandard collection of routine data may hinder advancements in patient care. Improved data entry, integration with clinical activities and workflows, system usability, data dictionaries, and training are necessary for OIS data to generate robust research. Data mining from clinical documents may supplement structured data collection.

Machine Learning and Nomogram Prognostic Modeling for 2-Year Head and Neck Cancer-Specific Survival Using Electronic Health Record Data: A Multisite Study

PURPOSE

There is limited knowledge of the prediction of 2-year cancer-specific survival (CSS) in the head and neck cancer (HNC) population. The aim of this study is to develop and validate machine learning models and a nomogram for the prediction of 2-year CSS in patients with HNC using real-world data collected by major teaching and tertiary referral hospitals in New South Wales (NSW), Australia.

MATERIALS AND METHODS

Data collected in oncology information systems at multiple NSW Cancer Centres were extracted for 2,953 eligible adults diagnosed between 2000 and 2017 with squamous cell carcinoma of the head and neck. Death data were sourced from the National Death Index using record linkage. Machine learning and Cox regression/nomogram models were developed and internally validated in Python and R, respectively.

RESULTS

Machine learning models demonstrated highest performance (C-index) in the larynx and nasopharynx cohorts (0.82), followed by the oropharynx (0.79) and the hypopharynx and oral cavity cohorts (0.73). In the whole HNC population, C-indexes of 0.79 and 0.70 and Brier scores of 0.10 and 0.27 were reported for the machine learning and nomogram model, respectively. Cox regression analysis identified age, T and N classification, and time-corrected biologic equivalent dose in two gray fractions as independent prognostic factors for 2-year CSS. N classification was the most important feature used for prediction in the machine learning model followed by age.

CONCLUSION

Machine learning and nomogram analysis predicted 2-year CSS with high performance using routinely collected and complete clinical information extracted from oncology information systems. These models function as visual decision-making tools to guide radiotherapy treatment decisions and provide insight into the prediction of survival outcomes in patients with HNC.

Simulations of fluorescence imaging in the oral cavity

We describe an end-to-end image systems simulation that models a device capable of measuring fluorescence in the oral cavity. Our software includes a 3D model of the oral cavity and excitation-emission matrices of endogenous fluorophores that predict the spectral radiance of oral mucosal tissue. The predicted radiance is transformed by a model of the optics and image sensor to generate expected sensor image values. We compare simulated and real camera data from tongues in healthy individuals and show that the camera sensor chromaticity values can be used to quantify the fluorescence from porphyrins relative to the bulk fluorescence from multiple fluorophores (elastin, NADH, FAD, and collagen). Validation of the simulations supports the use of soft-prototyping in guiding system design for fluorescence imaging.

A Comparison of Different Modelling Techniques in Predicting Mortality with the Tilburg Frailty Indicator (Preprint)

Background

Objective

Methods

Results

Conclusions

Performance of Intraoperative Assessment of Resection Margins in Oral Cancer Surgery: A Review of Literature

Introduction

Achieving adequate resection margins during oral cancer surgery is important to improve patient prognosis. Surgeons have the delicate task of achieving an adequate resection and safeguarding satisfactory remaining function and acceptable physical appearance, while relying on visual inspection, palpation, and preoperative imaging. Intraoperative assessment of resection margins (IOARM) is a multidisciplinary effort, which can guide towards adequate resections. Different forms of IOARM are currently used, but it is unknown how accurate these methods are in predicting margin status. Therefore, this review aims to investigate: 1) the IOARM methods currently used during oral cancer surgery, 2) their performance, and 3) their clinical relevance.

Methods

A literature search was performed in the following databases: Embase, Medline, Web of Science Core Collection, Cochrane Central Register of Controlled Trials, and Google Scholar (from inception to January 23, 2020). IOARM performance was assessed in terms of accuracy, sensitivity, and specificity in predicting margin status, and the reduction of inadequate margins. Clinical relevance (i.e., overall survival, local recurrence, regional recurrence, local recurrence-free survival, disease-specific survival, adjuvant therapy) was recorded if available.

Results

Eighteen studies were included in the review, of which 10 for soft tissue and 8 for bone. For soft tissue, defect-driven IOARM-studies showed the average accuracy, sensitivity, and specificity of 90.9%, 47.6%, and 84.4%, and specimen-driven IOARM-studies showed, 91.5%, 68.4%, and 96.7%, respectively. For bone, specimen-driven IOARM-studies performed better than defect-driven, with an average accuracy, sensitivity, and specificity of 96.6%, 81.8%, and 98%, respectively. For both, soft tissue and bone, IOARM positively impacts patient outcome.

Conclusion

IOARM improves margin-status, especially the specimen-driven IOARM has higher performance compared to defect-driven IOARM. However, this conclusion is limited by the low number of studies reporting performance results for defect-driven IOARM. The current methods suffer from inherent disadvantages, namely their subjective character and the fact that only a small part of the resection surface can be assessed in a short time span, causing sampling errors. Therefore, a solution should be sought in the field of objective techniques that can rapidly assess the whole resection surface.

Specimen-driven intraoperative assessment of resection margins should be standard of care for oral cancer patients

With an incidence of 350.000 new cases per year, cancer of the oral cavity ranks among the 10 most common solid organ cancers. Most of these cancers are squamous cell carcinomas. Five-year survival is about 50%. It has been shown that clear resection margins (>5 mm healthy tissue surrounding the resected tumor) have a significant positive effect on locoregional control and survival. It is not uncommon that the resection margins of oral tumors are inadequate. However, when providing the surgeon with intraoperative feedback on the resection margin status, it is expected that obtaining adequate resection margins is improved. In this respect, it has been shown that specimen-driven intraoperative assessment of resection margins is superior to defect-driven intraoperative assessment of resection margins. In this concise report, it is described how a specimen-driven approach can increase the rate of adequate resections of oral cavity squamous cell carcinoma as well as that it is discussed how intraoperative assessment can be further improved with regard to the surgical treatment of oral cavity squamous cell carcinoma.

The Differences Between Individuals Engaging in Nonsuicidal Self-Injury and Suicide Attempt Are Complex (vs. Complicated or Simple)

BACKGROUND

Why do some people engage in nonsuicidal self-injury (NSSI) while others attempt suicide? One way to advance knowledge about this question is to shed light on the differences between people who engage in NSSI and people who attempt suicide. These groups could differ in three broad ways. First, these two groups may differ in a simple way, such that one or a small set of factors is both necessary and sufficient to accurately distinguish the two groups. Second, they might differ in a complicated way, meaning that a specific set of a large number of factors is both necessary and sufficient to accurately classify them. Third, they might differ in a complex way, with no necessary factor combinations and potentially no sufficient factor combinations. In this scenario, at the group level, complicated algorithms would either be insufficient (i.e., no complicated algorithm produces good accuracy) or unnecessary (i.e., many complicated algorithms produce good accuracy) to distinguish between groups. This study directly tested these three possibilities in a sample of people with a history of NSSI and/or suicide attempt.

METHOD

A total of 954 participants who have either engaged in NSSI and/or suicide attempt in their lifetime were recruited from online forums. Participants completed a series of measures on factors commonly associated with NSSI and suicide attempt. To test for simple differences, univariate logistic regressions were conducted. One theoretically informed multiple logistic regression model with suicidal desire, capability for suicide, and their interaction term was considered as well. To examine complicated and complex differences, multiple logistic regression and machine learning analyses were conducted.

RESULTS

No simple algorithm (i.e., single factor or small set of factors) accurately distinguished between groups. Complicated algorithms constructed with cross-validation methods produced fair accuracy; complicated algorithms constructed with bootstrap optimism methods produced good accuracy, but multiple different algorithms with this method produced similar results.

CONCLUSIONS

Findings were consistent with complex differences between people who engage in NSSI and suicide attempts. Specific complicated algorithms were either insufficient (cross-validation results) or unnecessary (bootstrap optimism results) to distinguish between these groups with high accuracy.

Combining statistical techniques to predict postsurgical risk of 1-year mortality for patients with colon cancer

Introduction

Colorectal cancer is one of the most frequently diagnosed malignancies and a common cause of cancer-related mortality. The aim of this study was to develop and validate a clinical predictive model for 1-year mortality among patients with colon cancer who survive for at least 30 days after surgery.

Methods

Patients diagnosed with colon cancer who had surgery for the first time and who survived 30 days after the surgery were selected prospectively. The outcome was mortality within 1 year. Random forest, genetic algorithms and classification and regression trees were combined in order to identify the variables and partition points that optimally classify patients by risk of mortality. The resulting decision tree was categorized into four risk categories. Split-sample and bootstrap validation were performed. ClinicalTrials.gov Identifier: NCT02488161.

Results

A total of 1945 patients were enrolled in the study. The variables identified as the main predictors of 1-year mortality were presence of residual tumor, American Society of Anesthesiologists Physical Status Classification System risk score, pathologic tumor staging, Charlson Comorbidity Index, intraoperative complications, adjuvant chemotherapy and recurrence of tumor. The model was internally validated; area under the receiver operating characteristic curve (AUC) was 0.896 in the derivation sample and 0.835 in the validation sample. Risk categorization leads to AUC values of 0.875 and 0.832 in the derivation and validation samples, respectively. Optimal cut-off point of estimated risk had a sensitivity of 0.889 and a specificity of 0.758.

Conclusion

The decision tree was a simple, interpretable, valid and accurate prediction rule of 1-year mortality among colon cancer patients who survived for at least 30 days after surgery.

Raman spectroscopy for assessment of bone resection margins in mandibulectomy for oral cavity squamous cell carcinoma

OBJECTIVES

The aim of this study was to investigate the potential of Raman spectroscopy for detection of oral cavity squamous cell carcinoma (OCSCC) in bone resection surfaces during mandibulectomy.

MATERIALS & METHODS

Raman mapping experiments were performed on fresh mandible resection specimens from patients treated with mandibulectomy for OCSCC. A tumour detection algorithm was created based on water concentration and the high-wavenumber range (2800 cm^-1-3050 cm^-1) of the Raman spectra.

RESULTS

Twenty-six ex vivo Raman mapping experiments were performed on 26 fresh mandible resection specimens obtained from 22 patients. The algorithm was applied on an independent test set and showed an accuracy of 95%, a sensitivity of 95%, and a specificity of 87%.

CONCLUSION

These results form the basis for further development of a Raman spectroscopy tool as an objective method for intraoperative assessment of bone resection margins.

Modern modeling techniques had limited external validity in predicting mortality from traumatic brain injury

BACKGROUND AND OBJECTIVE

Prediction of medical outcomes may potentially benefit from using modern statistical modeling techniques. We aimed to externally validate modeling strategies for prediction of 6-month mortality of patients suffering from traumatic brain injury (TBI) with predictor sets of increasing complexity.

METHODS

We analyzed individual patient data from 15 different studies including 11,026 TBI patients. We consecutively considered a core set of predictors (age, motor score, and pupillary reactivity), an extended set with computed tomography scan characteristics, and a further extension with two laboratory measurements (glucose and hemoglobin). With each of these sets, we predicted 6-month mortality using default settings with five statistical modeling techniques: logistic regression (LR), classification and regression trees, random forests (RFs), support vector machines (SVM) and neural nets. For external validation, a model developed on one of the 15 data sets was applied to each of the 14 remaining sets. This process was repeated 15 times for a total of 630 validations. The area under the receiver operating characteristic curve (AUC) was used to assess the discriminative ability of the models.

RESULTS

For the most complex predictor set, the LR models performed best (median validated AUC value, 0.757), followed by RF and support vector machine models (median validated AUC value, 0.735 and 0.732, respectively). With each predictor set, the classification and regression trees models showed poor performance (median validated AUC value, <0.7). The variability in performance across the studies was smallest for the RF- and LR-based models (inter quartile range for validated AUC values from 0.07 to 0.10).

CONCLUSION

In the area of predicting mortality from TBI, nonlinear and nonadditive effects are not pronounced enough to make modern prediction methods beneficial.

Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints

BACKGROUND

Modern modelling techniques may potentially provide more accurate predictions of binary outcomes than classical techniques. We aimed to study the predictive performance of different modelling techniques in relation to the effective sample size ("data hungriness").

METHODS

We performed simulation studies based on three clinical cohorts: 1282 patients with head and neck cancer (with 46.9% 5 year survival), 1731 patients with traumatic brain injury (22.3% 6 month mortality) and 3181 patients with minor head injury (7.6% with CT scan abnormalities). We compared three relatively modern modelling techniques: support vector machines (SVM), neural nets (NN), and random forests (RF) and two classical techniques: logistic regression (LR) and classification and regression trees (CART). We created three large artificial databases with 20 fold, 10 fold and 6 fold replication of subjects, where we generated dichotomous outcomes according to different underlying models. We applied each modelling technique to increasingly larger development parts (100 repetitions). The area under the ROC-curve (AUC) indicated the performance of each model in the development part and in an independent validation part. Data hungriness was defined by plateauing of AUC and small optimism (difference between the mean apparent AUC and the mean validated AUC <0.01).

RESULTS

We found that a stable AUC was reached by LR at approximately 20 to 50 events per variable, followed by CART, SVM, NN and RF models. Optimism decreased with increasing sample sizes and the same ranking of techniques. The RF, SVM and NN models showed instability and a high optimism even with >200 events per variable.

CONCLUSIONS

Modern modelling techniques such as SVM, NN and RF may need over 10 times as many events per variable to achieve a stable AUC and a small optimism than classical modelling techniques such as LR. This implies that such modern techniques should only be used in medical prediction problems if very large data sets are available.