Harnessing repeated measurements of predictor variables for clinical risk prediction: a review of existing methods

Prioritising deteriorating patients using time-to-event analysis: prediction model development and internal-external validation

BACKGROUND

Binary classification models are frequently used to predict clinical deterioration, however they ignore information on the timing of events. An alternative is to apply time-to-event models, augmenting clinical workflows by ranking patients by predicted risks. This study examines how and why time-to-event modelling of vital signs data can help prioritise deterioration assessments using lift curves, and develops a prediction model to stratify acute care inpatients by risk of clinical deterioration.

METHODS

We developed and validated a Cox regression for time to in-hospital mortality. The model used time-varying covariates to estimate the risk of clinical deterioration. Adult inpatient medical records from 5 Australian hospitals between 1 January 2019 and 31 December 2020 were used for model development and validation. Model discrimination and calibration were assessed using internal-external cross validation. A discrete-time logistic regression model predicting death within 24 h with the same covariates was used as a comparator to the Cox regression model to estimate differences in predictive performance between the binary and time-to-event outcome modelling approaches.

RESULTS

Our data contained 150,342 admissions and 1016 deaths. Model discrimination was higher for Cox regression than for discrete-time logistic regression, with cross-validated AUCs of 0.96 and 0.93, respectively, for mortality predictions within 24 h, declining to 0.93 and 0.88, respectively, for mortality predictions within 1 week. Calibration plots showed that calibration varied by hospital, but this can be mitigated by ranking patients by predicted risks.

CONCLUSION

Time-varying covariate Cox models can be powerful tools for triaging patients, which may lead to more efficient and effective care in time-poor environments when the times between observations are highly variable.

Clinical prediction models in children that use repeated measurements with time-varying covariates: a scoping review

BACKGROUND

Emerging evidence suggests that clinical prediction models that use repeated (time-varying) measurements within each patient may have higher predictive accuracy than models that use patient information from a single measurement.

OBJECTIVE

To determine the breadth of the published literature reporting the development of clinical prediction models in children that use time-varying predictors.

DATA SOURCES

MEDLINE, EMBASE and Cochrane databases.

ELIGIBILITY CRITERIA

We included studies reporting the development of a multivariable clinical prediction model in children, with or without validation, to predict a repeatedly measured binary or time-to-event outcome and utilizing at least one repeatedly measured predictor.

SYNTHESIS METHODS

We categorized included studies by the method used to model time-varying predictors.

RESULTS

Of 99 clinical prediction model studies that had a repeated measurements data structure, only 27 (27%) used methods that incorporated the repeated measurements as time-varying predictors in a single model. Among these 27 time-varying prediction model studies, we grouped model types into nine categories: time-dependent Cox regression, generalized estimating equations, random effects model, landmark model, joint model, neural network, K-nearest neighbor, support vector machine and tree-based algorithms. Where there was comparison of time-varying models to single measurement models, using time-varying predictors improved predictive accuracy.

CONCLUSIONS

Various methods have been used to develop time-varying prediction models in children, but there is a paucity of pediatric time-varying models in the literature. Incorporating time-varying covariates in pediatric prediction models may improve predictive accuracy. Future research in pediatric prediction model development should further investigate whether incorporation of time-varying covariates improves predictive accuracy.

The Association between Blood Test Trends and Undiagnosed Cancer: A Systematic Review and Critical Appraisal

Clinical guidelines include monitoring blood test abnormalities to identify patients at increased risk of undiagnosed cancer. Noting blood test changes over time may improve cancer risk stratification by considering a patient's individual baseline and important changes within the normal range. We aimed to review the published literature to understand the association between blood test trends and undiagnosed cancer. MEDLINE and EMBASE were searched until 15 May 2023 for studies assessing the association between blood test trends and undiagnosed cancer. We used descriptive summaries and narratively synthesised studies. We included 29 articles. Common blood tests were haemoglobin (24%, n = 7), C-reactive protein (17%, n = 5), and fasting blood glucose (17%, n = 5), and common cancers were pancreatic (29%, n = 8) and colorectal (17%, n = 5). Of the 30 blood tests studied, an increasing trend in eight (27%) was associated with eight cancer types, and a decreasing trend in 17 (57%) with 10 cancer types. No association was reported between trends in 11 (37%) tests and breast, bile duct, glioma, haematological combined, liver, prostate, or thyroid cancers. Our review highlights trends in blood tests that could facilitate the identification of individuals at increased risk of undiagnosed cancer. For most possible combinations of tests and cancers, there was limited or no evidence.

Speech markers to predict and prevent recurrent episodes of psychosis: A narrative overview and emerging opportunities

Preventing relapse in schizophrenia improves long-term health outcomes. Repeated episodes of psychotic symptoms shape the trajectory of this illness and can be a detriment to functional recovery. Despite early intervention programs, high relapse rates persist, calling for alternative approaches in relapse prevention. Predicting imminent relapse at an individual level is critical for effective intervention. While clinical profiles are often used to foresee relapse, they lack the specificity and sensitivity needed for timely prediction. Here, we review the use of speech through Natural Language Processing (NLP) to predict a recurrent psychotic episode. Recent advancements in NLP of speech have shown the ability to detect linguistic markers related to thought disorder and other language disruptions within 2-4 weeks preceding a relapse. This approach has shown to be able to capture individual speech patterns, showing promise in its use as a prediction tool. We outline current developments in remote monitoring for psychotic relapses, discuss the challenges and limitations and present the speech-NLP based approach as an alternative to detect relapses with sufficient accuracy, construct validity and lead time to generate clinical actions towards prevention.

Clinical perspectives on serum tumor marker use in predicting prognosis and treatment response in advanced non-small cell lung cancer

The optimal positioning and usage of serum tumor markers (STMs) in advanced non-small cell lung cancer (NSCLC) care is still unclear. This review aimed to provide an overview of the potential use and value of STMs in routine advanced NSCLC care for the prediction of prognosis and treatment response. Radiological imaging and clinical symptoms have shown not to capture a patient's entire disease status in daily clinical practice. Since STM measurements allow for a rapid, minimally invasive, and safe evaluation of the patient's tumor status in real time, STMs can be used as companion decision-making support tools before start and during treatment. To overcome the limited sensitivity and specificity associated with the use of STMs, tests should only be applied in specific subgroups of patients and different test characteristics should be defined per clinical context in order to answer different clinical questions. The same approach can similarly be relevant when developing clinical applications for other (circulating) biomarkers. Future research should focus on the approaches described in this review to achieve STM test implementation in advanced NSCLC care.

Empowerment and enablement and their associations with change in health-related quality of life after a supported osteoarthritis self-management programme - a prospective observational study

BACKGROUND

Osteoarthritis is a leading cause of disability worldwide. Current treatment supports coping strategies to improve health-related quality of life (HRQoL). The need to predict response to treatment has been raised to personalise care. This study aims to examine change in HRQoL from baseline to three and nine months follow-up after participating in a Supported Osteoarthritis Self-Management Programme (SOASP) and to examine if empowerment and/or enablement were associated with change in HRQoL after a SOASP.

METHODS

Patients participating in a SOASP were recruited consecutively between April 2016 and June 2018. The EQ-5D was used to measure HRQoL, the Swedish Rheumatic Disease Empowerment Scale (SWE-RES-23) (score range 1-5) to measure empowerment and the Patient Enablement Instrument (PEI) (score range 0-12) to measure enablement. The instruments were answered before (EQ-5D, SWE-RES-23) and after (EQ-5D, SWE-RES-23, PEI) the SOASP. A patient partner was involved in the research process to enhance the patient perspective. Changes in outcome were examined with paired sample t-test and standardized effect sizes (Cohen´s d). Multiple linear regression analysis was performed to assess potential associations.

RESULTS

One hundred forty-three patients participated in baseline measurement. Mean EQ-5D-5 L index score increased significantly from baseline to three months corresponding to a standardised effect size (Cohen´s d) of d = 0.43, 95% CI [0.24, 0.63] (n = 109), and from baseline to nine months d = 0.19, 95% CI [0.01, 0.37] (n = 119). The average EQ VAS score increased significantly from baseline to three months corresponding to a standardised effect size of d = 0.26, 95% CI [0.07, 0.45] (n = 109), and from baseline to nine months d = 0.18, 95% CI [0.00, 0.36] (n = 119). Neither SWE-RES-23 nor PEI at three months follow-up nor the change in the SWE-RES-23 score from baseline to three months follow-up were associated with change in either EQ-5D-5 L index (p > 0.05) or the EQ VAS (p > 0.05).

CONCLUSIONS

Health-related quality of life increased after participating in a SOASP. Empowerment and enablement as measured with the SWE-RES-23 and the PEI were not associated with change in HRQoL among patients participating in a SOASP.

TRIAL REGISTRATION

ClinicalTrials.gov. Identification number: NCT02974036. First registration 28/11/2016, retrospectively registered.

Characterization of direct and/or indirect genetic associations for multiple traits in longitudinal studies of disease progression

When quantitative longitudinal traits are risk factors for disease progression and subject to random biological variation, joint model analysis of time-to-event and longitudinal traits can effectively identify direct and/or indirect genetic association of single nucleotide polymorphisms (SNPs) with time-to-event. We present a joint model that integrates: (1) a multivariate linear mixed model describing trajectories of multiple longitudinal traits as a function of time, SNP effects, and subject-specific random effects and (2) a frailty Cox survival model that depends on SNPs, longitudinal trajectory effects, and subject-specific frailty accounting for dependence among multiple time-to-event traits. Motivated by complex genetic architecture of type 1 diabetes complications (T1DC) observed in the Diabetes Control and Complications Trial (DCCT), we implement a 2-stage approach to inference with bootstrap joint covariance estimation and develop a hypothesis testing procedure to classify direct and/or indirect SNP association with each time-to-event trait. By realistic simulation study, we show that joint modeling of 2 time-to-T1DC (retinopathy and nephropathy) and 2 longitudinal risk factors (HbA1c and systolic blood pressure) reduces estimation bias in genetic effects and improves classification accuracy of direct and/or indirect SNP associations, compared to methods that ignore within-subject risk factor variability and dependence among longitudinal and time-to-event traits. Through DCCT data analysis, we demonstrate feasibility for candidate SNP modeling and quantify effects of sample size and Winner's curse bias on classification for 2 SNPs identified as having indirect associations with time-to-T1DC traits. Joint analysis of multiple longitudinal and multiple time-to-event traits provides insight into complex traits architecture.

Vital signs-based deterioration prediction model assumptions can lead to losses in prediction performance

OBJECTIVE

Vital signs-based models are complicated by repeated measures per patient and frequently missing data. This paper investigated the impacts of common vital signs modeling assumptions during clinical deterioration prediction model development.

STUDY DESIGN AND SETTING

Electronic medical record (EMR) data from five Australian hospitals (1 January 2019-31 December 2020) were used. Summary statistics for each observation's prior vital signs were created. Missing data patterns were investigated using boosted decision trees, then imputed with common methods. Two example models predicting in-hospital mortality were developed, as follows: logistic regression and eXtreme Gradient Boosting. Model discrimination and calibration were assessed using the C-statistic and nonparametric calibration plots.

RESULTS

The data contained 5,620,641 observations from 342,149 admissions. Missing vitals were associated with observation frequency, vital sign variability, and patient consciousness. Summary statistics improved discrimination slightly for logistic regression and markedly for eXtreme Gradient Boosting. Imputation method led to notable differences in model discrimination and calibration. Model calibration was generally poor.

CONCLUSION

Summary statistics and imputation methods can improve model discrimination and reduce bias during model development, but it is questionable whether these differences are clinically significant. Researchers should consider why data are missing during model development and how this may impact clinical utility.

Development and Internal Validation of a Risk Prediction Model to Identify Myeloma Based on Routine Blood Tests: A Case-Control Study

Myeloma is one of the hardest cancers to diagnose in primary care due to its rarity and non-specific symptoms. A rate-limiting step in diagnosing myeloma is the clinician considering myeloma and initiating appropriate investigations. We developed and internally validated a risk prediction model to identify those with a high risk of having undiagnosed myeloma based on results from routine blood tests taken for other reasons. A case-control study, based on 367 myeloma cases and 1488 age- and sex-matched controls, was used to develop a risk prediction model including results from 15 blood tests. The model had excellent discrimination (C-statistic 0.85 (95%CI 0.83, 0.89)) and good calibration (calibration slope 0.87 (95%CI 0.75, 0.90)). At a prevalence of 15 per 100,000 population and a probability threshold of 0.4, approximately 600 patients would need additional reflex testing to detect one case. We showed that it is possible to combine signals and abnormalities from several routine blood test parameters to identify individuals at high-risk of having undiagnosed myeloma who may benefit from additional reflex testing. Further work is needed to explore the full potential of such a strategy, including whether it is clinically useful and cost-effective and how to make it ethically acceptable.

Multivariate longitudinal data for survival analysis of cardiovascular event prediction in young adults: insights from a comparative explainable study

BACKGROUND

Multivariate longitudinal data are under-utilized for survival analysis compared to cross-sectional data (CS - data collected once across cohort). Particularly in cardiovascular risk prediction, despite available methods of longitudinal data analysis, the value of longitudinal information has not been established in terms of improved predictive accuracy and clinical applicability.

METHODS

We investigated the value of longitudinal data over and above the use of cross-sectional data via 6 distinct modeling strategies from statistics, machine learning, and deep learning that incorporate repeated measures for survival analysis of the time-to-cardiovascular event in the Coronary Artery Risk Development in Young Adults (CARDIA) cohort. We then examined and compared the use of model-specific interpretability methods (Random Survival Forest Variable Importance) and model-agnostic methods (SHapley Additive exPlanation (SHAP) and Temporal Importance Model Explanation (TIME)) in cardiovascular risk prediction using the top-performing models.

RESULTS

In a cohort of 3539 participants, longitudinal information from 35 variables that were repeatedly collected in 6 exam visits over 15 years improved subsequent long-term (17 years after) risk prediction by up to 8.3% in C-index compared to using baseline data (0.78 vs. 0.72), and up to approximately 4% compared to using the last observed CS data (0.75). Time-varying AUC was also higher in models using longitudinal data (0.86-0.87 at 5 years, 0.79-0.81 at 10 years) than using baseline or last observed CS data (0.80-0.86 at 5 years, 0.73-0.77 at 10 years). Comparative model interpretability analysis revealed the impact of longitudinal variables on model prediction on both the individual and global scales among different modeling strategies, as well as identifying the best time windows and best timing within that window for event prediction. The best strategy to incorporate longitudinal data for accuracy was time series massive feature extraction, and the easiest interpretable strategy was trajectory clustering.

CONCLUSION

Our analysis demonstrates the added value of longitudinal data in predictive accuracy and epidemiological utility in cardiovascular risk survival analysis in young adults via a unified, scalable framework that compares model performance and explainability. The framework can be extended to a larger number of variables and other longitudinal modeling methods.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT00005130, Registration Date: 26/05/2000.

BLOod Test Trend for cancEr Detection (BLOTTED): protocol for an observational and prediction model development study using English primary care electronic health record data

BACKGROUND

Simple blood tests can play an important role in identifying patients for cancer investigation. The current evidence base is limited almost entirely to tests used in isolation. However, recent evidence suggests combining multiple types of blood tests and investigating trends in blood test results over time could be more useful to select patients for further cancer investigation. Such trends could increase cancer yield and reduce unnecessary referrals. We aim to explore whether trends in blood test results are more useful than symptoms or single blood test results in selecting primary care patients for cancer investigation. We aim to develop clinical prediction models that incorporate trends in blood tests to identify the risk of cancer.

METHODS

Primary care electronic health record data from the English Clinical Practice Research Datalink Aurum primary care database will be accessed and linked to cancer registrations and secondary care datasets. Using a cohort study design, we will describe patterns in blood testing (aim 1) and explore associations between covariates and trends in blood tests with cancer using mixed-effects, Cox, and dynamic models (aim 2). To build the predictive models for the risk of cancer, we will use dynamic risk modelling (such as multivariate joint modelling) and machine learning, incorporating simultaneous trends in multiple blood tests, together with other covariates (aim 3). Model performance will be assessed using various performance measures, including c-statistic and calibration plots.

DISCUSSION

These models will form decision rules to help general practitioners find patients who need a referral for further investigation of cancer. This could increase cancer yield, reduce unnecessary referrals, and give more patients the opportunity for treatment and improved outcomes.

Risk prediction of mortality for patients with heart failure in England: observational study in primary care

AIMS

Many risk prediction models have been proposed for heart failure (HF), but few studies have used only information available to general practitioners (GPs) in primary care electronic health records (EHRs). We describe the predictors and performance of models built from GP-based EHRs in two cohorts of patients 10 years apart.

METHODS AND RESULTS

Linked primary and secondary care data for incident HF cases in England were extracted from the Clinical Practice Research Datalink for 2001-02 and 2011-12. Time-to-event models for all-cause mortality were developed using a long list of potential baseline predictors. Discrimination and calibration were calculated. A total of 5966 patients in 156 general practices were diagnosed in 2001-02, and 12 827 patients in 331 practices were diagnosed in 2011-12. The 5-year survival rate was 40.0% in 2001-02 and 40.2% in 2011-12, though the latter population were older, frailer, and more comorbid; for 2001-02, the 10-year survival was 20.8% and 15-year survival 11.1%. Consistent predictors included age, male sex, systolic blood pressure, body mass index, GP domiciliary visits before diagnosis, and some comorbidities. Model performance for both time windows was modest (c = 0.70), but calibration was generally excellent in both time periods.

CONCLUSIONS

Information routinely available to UK GPs at the time of diagnosis of HF gives only modest predictive accuracy of all-cause mortality, making it hard to decide on the type, place, and urgency of follow-up. More consistent recording of data relevant to HF (such as echocardiography and natriuretic peptide results) in GP EHRs is needed to support accurate prediction of healthcare needs in individuals with HF.

Modeling strategies to analyse longitudinal biomarker data: An illustration on predicting immunotherapy non-response in non-small cell lung cancer

Serum tumor markers acquired through a blood draw are known to reflect tumor activity. Their non-invasive nature allows for more frequent testing compared to traditional imaging methods used for response evaluations. Our study aims to compare nine prediction methods to accurately, and with a low false positive rate, predict progressive disease despite treatment (i.e. non-response) using longitudinal tumor biomarker data. Bi-weekly measurements of CYFRA, CA-125, CEA, NSE, and SCC were available from a cohort of 412 advanced stage non-small cell lung cancer (NSCLC) patients treated up to two years with immune checkpoint inhibitors. Serum tumor marker measurements from the first six weeks after treatment initiation were used to predict treatment response at 6 months. Nine models with varying complexity were evaluated in this study, showing how longitudinal biomarker data can be used to predict non-response to immunotherapy in NSCLC patients.

Modelling of longitudinal data to predict cardiovascular disease risk: a methodological review

OBJECTIVE

The identification of methodology for modelling cardiovascular disease (CVD) risk using longitudinal data and risk factor trajectories.

METHODS

We screened MEDLINE-Ovid from inception until 3 June 2020. MeSH and text search terms covered three areas: data type, modelling type and disease area including search terms such as "longitudinal", "trajector*" and "cardiovasc*" respectively. Studies were filtered to meet the following inclusion criteria: longitudinal individual patient data in adult patients with ≥3 time-points and a CVD or mortality outcome. Studies were screened and analyzed by one author. Any queries were discussed with the other authors. Comparisons were made between the methods identified looking at assumptions, flexibility and software availability.

RESULTS

From the initial 2601 studies returned by the searches 80 studies were included. Four statistical approaches were identified for modelling the longitudinal data: 3 (4%) studies compared time points with simple statistical tests, 40 (50%) used single-stage approaches, such as including single time points or summary measures in survival models, 29 (36%) used two-stage approaches including an estimated longitudinal parameter in survival models, and 8 (10%) used joint models which modelled the longitudinal and survival data together. The proportion of CVD risk prediction models created using longitudinal data using two-stage and joint models increased over time.

CONCLUSIONS

Single stage models are still heavily utilized by many CVD risk prediction studies for modelling longitudinal data. Future studies should fully utilize available longitudinal data when analyzing CVD risk by employing two-stage and joint approaches which can often better utilize the available data.

Informative presence and observation in routine health data: A review of methodology for clinical risk prediction

Objective

Informative presence (IP) is the phenomenon whereby the presence or absence of patient data is potentially informative with respect to their health condition, with informative observation (IO) being the longitudinal equivalent. These phenomena predominantly exist within routinely collected healthcare data, in which data collection is driven by the clinical requirements of patients and clinicians. The extent to which IP and IO are considered when using such data to develop clinical prediction models (CPMs) is unknown, as is the existing methodology aiming at handling these issues. This review aims to synthesize such existing methodology, thereby helping identify an agenda for future methodological work.

Materials and Methods

A systematic literature search was conducted by 2 independent reviewers using prespecified keywords.

Results

Thirty-six articles were included. We categorized the methods presented within as derived predictors (including some representation of the measurement process as a predictor in the model), modeling under IP, and latent structures. Including missing indicators or summary measures as predictors is the most commonly presented approach amongst the included studies (24 of 36 articles).

Discussion

This is the first review to collate the literature in this area under a prediction framework. A considerable body relevant of literature exists, and we present ways in which the described methods could be developed further. Guidance is required for specifying the conditions under which each method should be used to enable applied prediction modelers to use these methods.

Conclusions

A growing recognition of IP and IO exists within the literature, and methodology is increasingly becoming available to leverage these phenomena for prediction purposes. IP and IO should be approached differently in a prediction context than when the primary goal is explanation. The work included in this review has demonstrated theoretical and empirical benefits of incorporating IP and IO, and therefore we recommend that applied health researchers consider incorporating these methods in their work.

Toward a framework for the design, implementation, and reporting of methodology scoping reviews

BACKGROUND AND OBJECTIVE

In view of the growth of published articles, there is an increasing need for studies that summarize scientific research. An increasingly common review is a "methodology scoping review," which provides a summary of existing analytical methods, techniques and software that have been proposed or applied in research articles to address an analytical problem or further an analytical approach. However, guidelines for their design, implementation, and reporting are limited.

METHODS

Drawing on the experiences of the authors, which were consolidated through a series of face-to-face workshops, we summarize the challenges inherent in conducting a methodology scoping review and offer suggestions of best practice to promote future guideline development.

RESULTS

We identified three challenges of conducting a methodology scoping review. First, identification of search terms; one cannot usually define the search terms a priori, and the language used for a particular method can vary across the literature. Second, the scope of the review requires careful consideration because new methodology is often not described (in full) within abstracts. Third, many new methods are motivated by a specific clinical question, where the methodology may only be documented in supplementary materials. We formulated several recommendations that build upon existing review guidelines. These recommendations ranged from an iterative approach to defining search terms through to screening and data extraction processes.

CONCLUSION

Although methodology scoping reviews are an important aspect of research, there is currently a lack of guidelines to standardize their design, implementation, and reporting. We recommend a wider discussion on this topic.