Harnessing repeated measurements of predictor variables for clinical risk prediction: a review of existing methods

Translational Algorithms for Technological Dietary Quality Assessment Integrating Nutrimetabolic Data with Machine Learning Methods

Recent advances in machine learning technologies and omics methodologies are revolutionizing dietary assessment by integrating phenotypical, clinical, and metabolic biomarkers, which are crucial for personalized precision nutrition. This investigation aims to evaluate the feasibility and efficacy of artificial intelligence tools, particularly machine learning (ML) methods, in analyzing these biomarkers to characterize food and nutrient intake and to predict dietary patterns.

METHODS

We analyzed data from 138 subjects from the European Dietary Deal project through comprehensive examinations, lifestyle questionnaires, and fasting blood samples. Clustering was based on 72 h dietary recall, considering sex, age, and BMI. Exploratory factor analysis (EFA) assigned nomenclature to clusters based on food consumption patterns and nutritional indices from food frequency questionnaires. Elastic net regression identified biomarkers linked to these patterns, helping construct algorithms.

RESULTS

Clustering and EFA identified two dietary patterns linked to biochemical markers, distinguishing pro-Mediterranean (pro-MP) and pro-Western (pro-WP) patterns. Analysis revealed differences between pro-MP and pro-WP clusters, such as vegetables, pulses, cereals, drinks, meats, dairy, fish, and sweets. Markers related to lipid metabolism, liver function, blood coagulation, and metabolic factors were pivotal in discriminating clusters. Three computational algorithms were created to predict the probabilities of being classified into the pro-WP pattern. The first is the main algorithm, followed by a supervised algorithm, which is a simplified version of the main model that focuses on clinically feasible biochemical parameters and practical scientific criteria, demonstrating good predictive capabilities (ROC curve = 0.91, precision-recall curve = 0.80). Lastly, a reduced biochemical-based algorithm is presented, derived from the supervised algorithm.

CONCLUSIONS

This study highlights the potential of biochemical markers in predicting nutritional patterns and the development of algorithms for classifying dietary clusters, advancing dietary intake assessment technologies.

Construction and validation of a clinical prediction model for sepsis using peripheral perfusion index to predict in-hospital and 28-day mortality risk

Sepsis is a clinical syndrome caused by infection, leading to organ dysfunction due to a dysregulated host response. In recent years, its high mortality rate has made it a significant cause of death and disability worldwide. The pathophysiological process of sepsis is related to the body's dysregulated response to infection, with microcirculatory changes serving as early warning signals that guide clinical treatment. The Peripheral Perfusion Index (PI), as an indicator of peripheral microcirculation, can effectively evaluate patient prognosis. This study aims to develop two new prediction models using PI and other common clinical indicators to assess the mortality risk of sepsis patients during hospitalization and within 28 days post-ICU admission. This retrospective study analyzed data from sepsis patients treated in the Intensive Care Unit of Peking Union Medical College Hospital between December 2019 and June 2023, ultimately including 645 patients. LASSO regression and logistic regression analyses were used to select predictive factors from 35 clinical indicators, and two clinical prediction models were constructed to predict in-hospital mortality and 28-day mortality. The models' performance was then evaluated using ROC curve, calibration curve, and decision curve analyses. The two prediction models performed excellently in distinguishing patient mortality risk. The AUC for the in-hospital mortality prediction model was 0.82 in the training set and 0.73 in the validation set; for the 28-day mortality prediction model, the AUC was 0.79 in the training set and 0.73 in the validation set. The calibration curves closely aligned with the ideal line, indicating consistency between predicted and actual outcomes. Decision curve analysis also demonstrated high net benefits for the clinical utility of both models. The study shows that these two prediction models not only perform excellently statistically but also hold high practical value in clinical applications. The models can help physicians accurately assess the mortality risk of sepsis patients, providing a scientific basis for personalized treatment.

Dynamic prediction of pregnancy outcome after previous stillbirth or perinatal death: pilot study to establish proof-of-concept and explore method feasibility

OBJECTIVE

To establish proof-of-concept for the dynamic prediction of adverse pregnancy outcome in women with a history of stillbirth or perinatal death, repeatedly throughout the pregnancy.

METHODS

A retrospective cohort study of women in a subsequent pregnancy following previous perinatal loss, who received antenatal care at a tertiary hospital between January 2014 and December 2017, was used as the basis for exploratory prognostic model development. Models were developed to repeatedly predict a composite adverse outcome (stillbirth or neonatal death, 5-min Apgar score < 7, umbilical artery pH ≤ 7.05, admission to the neonatal intensive care unit for longer than 24 h, preterm birth (< 37 completed weeks) or birth weight < 10th centile) using the findings of sequential ultrasound scans for fetal biometry and umbilical and uterine artery Doppler.

RESULTS

In total, 506 participants were eligible, of whom 504 were included in the analysis. An adverse pregnancy outcome was experienced by 110 (22%) participants. The ability to predict the composite outcome using repeated head circumference and estimated fetal weight measurements improved as the pregnancy progressed (e.g. area under the receiver-operating-characteristics curve improved from 0.59 at 24 weeks' gestation to 0.74 at 36 weeks' gestation), supporting proof-of-concept. Predictors to include in dynamic prediction models were identified, including ultrasound measurements of fetal biometry, umbilical and uterine artery Doppler and placental size and shape.

CONCLUSION

The present study supports proof-of-concept for dynamic prediction of adverse outcome in pregnancy following prior stillbirth or perinatal death, which could be used to identify risks earlier in pregnancy, while highlighting methodological challenges and requirements for subsequent large-scale model development studies. © 2024 The Author(s). Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Delayed kernels for longitudinal survival analysis and dynamic prediction

Predicting patient survival probabilities based on observed covariates is an important assessment in clinical practice. These patient-specific covariates are often measured over multiple follow-up appointments. It is then of interest to predict survival based on the history of these longitudinal measurements, and to update predictions as more observations become available. The standard approaches to these so-called 'dynamic prediction' assessments are joint models and landmark analysis. Joint models involve high-dimensional parameterizations, and their computational complexity often prohibits including multiple longitudinal covariates. Landmark analysis is simpler, but discards a proportion of the available data at each 'landmark time'. In this work, we propose a 'delayed kernel' approach to dynamic prediction that sits somewhere in between the two standard methods in terms of complexity. By conditioning hazard rates directly on the covariate measurements over the observation time frame, we define a model that takes into account the full history of covariate measurements but is more practical and parsimonious than joint modelling. Time-dependent association kernels describe the impact of covariate changes at earlier times on the patient's hazard rate at later times. Under the constraints that our model (a) reduces to the standard Cox model for time-independent covariates, and (b) contains the instantaneous Cox model as a special case, we derive two natural kernel parameterizations. Upon application to three clinical data sets, we find that the predictive accuracy of the delayed kernel approach is comparable to that of the two existing standard methods.

A review of methods for the analysis of diagnostic tests performed in sequence

BACKGROUND

Many clinical pathways for the diagnosis of disease are based on diagnostic tests that are performed in sequence. The performance of the full diagnostic sequence is dictated by the diagnostic performance of each test in the sequence as well as the conditional dependence between them, given true disease status. Resulting estimates of performance, such as the sensitivity and specificity of the test sequence, are key parameters in health-economic evaluations. We conducted a methodological review of statistical methods for assessing the performance of diagnostic tests performed in sequence, with the aim of guiding data analysts towards classes of methods that may be suitable given the design and objectives of the testing sequence.

METHODS

We searched PubMed, Scopus and Web of Science for relevant papers describing methodology for analysing sequences of diagnostic tests. Papers were classified by the characteristics of the method used, and these were used to group methods into themes. We illustrate some of the methods using data from a cohort study of repeat faecal immunochemical testing for colorectal cancer in symptomatic patients, to highlight the importance of allowing for conditional dependence in test sequences and adjustment for an imperfect reference standard.

RESULTS

Five overall themes were identified, detailing methods for combining multiple tests in sequence, estimating conditional dependence, analysing sequences of diagnostic tests used for risk assessment, analysing test sequences in conjunction with an imperfect or incomplete reference standard, and meta-analysis of test sequences.

CONCLUSIONS

This methodological review can be used to help researchers identify suitable analytic methods for studies that use diagnostic tests performed in sequence.

Towards an early warning system for monitoring of cancer patients using hybrid interactive machine learning

Background

The use of smartphone apps in cancer patients undergoing systemic treatment can promote the early detection of symptoms and therapy side effects and may be supported by machine learning (ML) for timely adaptation of therapies and reduction of adverse events and unplanned admissions.

Objective

We aimed to create an Early Warning System (EWS) to predict situations where supportive interventions become necessary to prevent unplanned visits. For this, dynamically collected standardized electronic patient reported outcome (ePRO) data were analyzed in context with the patient's individual journey. Information on well-being, vital parameters, medication, and free text were also considered for establishing a hybrid ML model. The goal was to integrate both the strengths of ML in sifting through large amounts of data and the long-standing experience of human experts. Given the limitations of highly imbalanced datasets (where only very few adverse events are present) and the limitations of humans in overseeing all possible cause of such events, we hypothesize that it should be possible to combine both in order to partially overcome these limitations.

Methods

The prediction of unplanned visits was achieved by employing a white-box ML algorithm (i.e., rule learner), which learned rules from patient data (i.e., ePROs, vital parameters, free text) that were captured via a medical device smartphone app. Those rules indicated situations where patients experienced unplanned visits and, hence, were captured as alert triggers in the EWS. Each rule was evaluated based on a cost matrix, where false negatives (FNs) have higher costs than false positives (FPs, i.e., false alarms). Rules were then ranked according to the costs and priority was given to the least expensive ones. Finally, the rules with higher priority were reviewed by two oncological experts for plausibility check and for extending them with additional conditions. This hybrid approach comprised the application of a sensitive ML algorithm producing several potentially unreliable, but fully human-interpretable and -modifiable rules, which could then be adjusted by human experts.

Results

From a cohort of 214 patients and more than 16'000 available data entries, the machine-learned rule set achieved a recall of 19% on the entire dataset and a precision of 5%. We compared this performance to a set of conditions that a human expert had defined to predict adverse events. This "human baseline" did not discover any of the adverse events recorded in our dataset, i.e., it came with a recall and precision of 0%. Despite more plentiful results were expected by our machine learning approach, the involved medical experts a) had understood and were able to make sense of the rules and b) felt capable to suggest modification to the rules, some of which could potentially increase their precision. Suggested modifications of rules included e.g., adding or tightening certain conditions to make them less sensitive or changing the rule consequences: sometimes further monitoring the situation, applying certain test (such as a CRP test) or applying some simple pain-relieving measures was deemed sufficient, making a costly consultation with the physician unnecessary. We can thus conclude that it is possible to apply machine learning as an inspirational tool that can help human experts to formulate rules for an EWS. While humans seem to lack the ability to define such rules without such support, they are capable of modifying the rules to increase their precision and generalizability.

Conclusions

Learning rules from dynamic ePRO datasets may be used to assist human experts in establishing an early warning system for cancer patients in outpatient settings.

Prioritising deteriorating patients using time-to-event analysis: prediction model development and internal-external validation

BACKGROUND

Binary classification models are frequently used to predict clinical deterioration, however they ignore information on the timing of events. An alternative is to apply time-to-event models, augmenting clinical workflows by ranking patients by predicted risks. This study examines how and why time-to-event modelling of vital signs data can help prioritise deterioration assessments using lift curves, and develops a prediction model to stratify acute care inpatients by risk of clinical deterioration.

METHODS

We developed and validated a Cox regression for time to in-hospital mortality. The model used time-varying covariates to estimate the risk of clinical deterioration. Adult inpatient medical records from 5 Australian hospitals between 1 January 2019 and 31 December 2020 were used for model development and validation. Model discrimination and calibration were assessed using internal-external cross validation. A discrete-time logistic regression model predicting death within 24 h with the same covariates was used as a comparator to the Cox regression model to estimate differences in predictive performance between the binary and time-to-event outcome modelling approaches.

RESULTS

Our data contained 150,342 admissions and 1016 deaths. Model discrimination was higher for Cox regression than for discrete-time logistic regression, with cross-validated AUCs of 0.96 and 0.93, respectively, for mortality predictions within 24 h, declining to 0.93 and 0.88, respectively, for mortality predictions within 1 week. Calibration plots showed that calibration varied by hospital, but this can be mitigated by ranking patients by predicted risks.

CONCLUSION

Time-varying covariate Cox models can be powerful tools for triaging patients, which may lead to more efficient and effective care in time-poor environments when the times between observations are highly variable.

Clinical prediction models in children that use repeated measurements with time-varying covariates: a scoping review

BACKGROUND

Emerging evidence suggests that clinical prediction models that use repeated (time-varying) measurements within each patient may have higher predictive accuracy than models that use patient information from a single measurement.

OBJECTIVE

To determine the breadth of the published literature reporting the development of clinical prediction models in children that use time-varying predictors.

DATA SOURCES

MEDLINE, EMBASE and Cochrane databases.

ELIGIBILITY CRITERIA

We included studies reporting the development of a multivariable clinical prediction model in children, with or without validation, to predict a repeatedly measured binary or time-to-event outcome and utilizing at least one repeatedly measured predictor.

SYNTHESIS METHODS

We categorized included studies by the method used to model time-varying predictors.

RESULTS

Of 99 clinical prediction model studies that had a repeated measurements data structure, only 27 (27%) used methods that incorporated the repeated measurements as time-varying predictors in a single model. Among these 27 time-varying prediction model studies, we grouped model types into nine categories: time-dependent Cox regression, generalized estimating equations, random effects model, landmark model, joint model, neural network, K-nearest neighbor, support vector machine and tree-based algorithms. Where there was comparison of time-varying models to single measurement models, using time-varying predictors improved predictive accuracy.

CONCLUSIONS

Various methods have been used to develop time-varying prediction models in children, but there is a paucity of pediatric time-varying models in the literature. Incorporating time-varying covariates in pediatric prediction models may improve predictive accuracy. Future research in pediatric prediction model development should further investigate whether incorporation of time-varying covariates improves predictive accuracy.

The Association between Blood Test Trends and Undiagnosed Cancer: A Systematic Review and Critical Appraisal

Clinical guidelines include monitoring blood test abnormalities to identify patients at increased risk of undiagnosed cancer. Noting blood test changes over time may improve cancer risk stratification by considering a patient's individual baseline and important changes within the normal range. We aimed to review the published literature to understand the association between blood test trends and undiagnosed cancer. MEDLINE and EMBASE were searched until 15 May 2023 for studies assessing the association between blood test trends and undiagnosed cancer. We used descriptive summaries and narratively synthesised studies. We included 29 articles. Common blood tests were haemoglobin (24%, n = 7), C-reactive protein (17%, n = 5), and fasting blood glucose (17%, n = 5), and common cancers were pancreatic (29%, n = 8) and colorectal (17%, n = 5). Of the 30 blood tests studied, an increasing trend in eight (27%) was associated with eight cancer types, and a decreasing trend in 17 (57%) with 10 cancer types. No association was reported between trends in 11 (37%) tests and breast, bile duct, glioma, haematological combined, liver, prostate, or thyroid cancers. Our review highlights trends in blood tests that could facilitate the identification of individuals at increased risk of undiagnosed cancer. For most possible combinations of tests and cancers, there was limited or no evidence.

Speech markers to predict and prevent recurrent episodes of psychosis: A narrative overview and emerging opportunities

Preventing relapse in schizophrenia improves long-term health outcomes. Repeated episodes of psychotic symptoms shape the trajectory of this illness and can be a detriment to functional recovery. Despite early intervention programs, high relapse rates persist, calling for alternative approaches in relapse prevention. Predicting imminent relapse at an individual level is critical for effective intervention. While clinical profiles are often used to foresee relapse, they lack the specificity and sensitivity needed for timely prediction. Here, we review the use of speech through Natural Language Processing (NLP) to predict a recurrent psychotic episode. Recent advancements in NLP of speech have shown the ability to detect linguistic markers related to thought disorder and other language disruptions within 2-4 weeks preceding a relapse. This approach has shown to be able to capture individual speech patterns, showing promise in its use as a prediction tool. We outline current developments in remote monitoring for psychotic relapses, discuss the challenges and limitations and present the speech-NLP based approach as an alternative to detect relapses with sufficient accuracy, construct validity and lead time to generate clinical actions towards prevention.

Clinical perspectives on serum tumor marker use in predicting prognosis and treatment response in advanced non-small cell lung cancer

The optimal positioning and usage of serum tumor markers (STMs) in advanced non-small cell lung cancer (NSCLC) care is still unclear. This review aimed to provide an overview of the potential use and value of STMs in routine advanced NSCLC care for the prediction of prognosis and treatment response. Radiological imaging and clinical symptoms have shown not to capture a patient's entire disease status in daily clinical practice. Since STM measurements allow for a rapid, minimally invasive, and safe evaluation of the patient's tumor status in real time, STMs can be used as companion decision-making support tools before start and during treatment. To overcome the limited sensitivity and specificity associated with the use of STMs, tests should only be applied in specific subgroups of patients and different test characteristics should be defined per clinical context in order to answer different clinical questions. The same approach can similarly be relevant when developing clinical applications for other (circulating) biomarkers. Future research should focus on the approaches described in this review to achieve STM test implementation in advanced NSCLC care.

Empowerment and enablement and their associations with change in health-related quality of life after a supported osteoarthritis self-management programme - a prospective observational study

BACKGROUND

Osteoarthritis is a leading cause of disability worldwide. Current treatment supports coping strategies to improve health-related quality of life (HRQoL). The need to predict response to treatment has been raised to personalise care. This study aims to examine change in HRQoL from baseline to three and nine months follow-up after participating in a Supported Osteoarthritis Self-Management Programme (SOASP) and to examine if empowerment and/or enablement were associated with change in HRQoL after a SOASP.

METHODS

Patients participating in a SOASP were recruited consecutively between April 2016 and June 2018. The EQ-5D was used to measure HRQoL, the Swedish Rheumatic Disease Empowerment Scale (SWE-RES-23) (score range 1-5) to measure empowerment and the Patient Enablement Instrument (PEI) (score range 0-12) to measure enablement. The instruments were answered before (EQ-5D, SWE-RES-23) and after (EQ-5D, SWE-RES-23, PEI) the SOASP. A patient partner was involved in the research process to enhance the patient perspective. Changes in outcome were examined with paired sample t-test and standardized effect sizes (Cohen´s d). Multiple linear regression analysis was performed to assess potential associations.

RESULTS

One hundred forty-three patients participated in baseline measurement. Mean EQ-5D-5 L index score increased significantly from baseline to three months corresponding to a standardised effect size (Cohen´s d) of d = 0.43, 95% CI [0.24, 0.63] (n = 109), and from baseline to nine months d = 0.19, 95% CI [0.01, 0.37] (n = 119). The average EQ VAS score increased significantly from baseline to three months corresponding to a standardised effect size of d = 0.26, 95% CI [0.07, 0.45] (n = 109), and from baseline to nine months d = 0.18, 95% CI [0.00, 0.36] (n = 119). Neither SWE-RES-23 nor PEI at three months follow-up nor the change in the SWE-RES-23 score from baseline to three months follow-up were associated with change in either EQ-5D-5 L index (p > 0.05) or the EQ VAS (p > 0.05).

CONCLUSIONS

Health-related quality of life increased after participating in a SOASP. Empowerment and enablement as measured with the SWE-RES-23 and the PEI were not associated with change in HRQoL among patients participating in a SOASP.

TRIAL REGISTRATION

ClinicalTrials.gov. Identification number: NCT02974036. First registration 28/11/2016, retrospectively registered.

Characterization of direct and/or indirect genetic associations for multiple traits in longitudinal studies of disease progression

When quantitative longitudinal traits are risk factors for disease progression and subject to random biological variation, joint model analysis of time-to-event and longitudinal traits can effectively identify direct and/or indirect genetic association of single nucleotide polymorphisms (SNPs) with time-to-event. We present a joint model that integrates: (1) a multivariate linear mixed model describing trajectories of multiple longitudinal traits as a function of time, SNP effects, and subject-specific random effects and (2) a frailty Cox survival model that depends on SNPs, longitudinal trajectory effects, and subject-specific frailty accounting for dependence among multiple time-to-event traits. Motivated by complex genetic architecture of type 1 diabetes complications (T1DC) observed in the Diabetes Control and Complications Trial (DCCT), we implement a 2-stage approach to inference with bootstrap joint covariance estimation and develop a hypothesis testing procedure to classify direct and/or indirect SNP association with each time-to-event trait. By realistic simulation study, we show that joint modeling of 2 time-to-T1DC (retinopathy and nephropathy) and 2 longitudinal risk factors (HbA1c and systolic blood pressure) reduces estimation bias in genetic effects and improves classification accuracy of direct and/or indirect SNP associations, compared to methods that ignore within-subject risk factor variability and dependence among longitudinal and time-to-event traits. Through DCCT data analysis, we demonstrate feasibility for candidate SNP modeling and quantify effects of sample size and Winner's curse bias on classification for 2 SNPs identified as having indirect associations with time-to-T1DC traits. Joint analysis of multiple longitudinal and multiple time-to-event traits provides insight into complex traits architecture.

Vital signs-based deterioration prediction model assumptions can lead to losses in prediction performance

OBJECTIVE

Vital signs-based models are complicated by repeated measures per patient and frequently missing data. This paper investigated the impacts of common vital signs modeling assumptions during clinical deterioration prediction model development.

STUDY DESIGN AND SETTING

Electronic medical record (EMR) data from five Australian hospitals (1 January 2019-31 December 2020) were used. Summary statistics for each observation's prior vital signs were created. Missing data patterns were investigated using boosted decision trees, then imputed with common methods. Two example models predicting in-hospital mortality were developed, as follows: logistic regression and eXtreme Gradient Boosting. Model discrimination and calibration were assessed using the C-statistic and nonparametric calibration plots.

RESULTS

The data contained 5,620,641 observations from 342,149 admissions. Missing vitals were associated with observation frequency, vital sign variability, and patient consciousness. Summary statistics improved discrimination slightly for logistic regression and markedly for eXtreme Gradient Boosting. Imputation method led to notable differences in model discrimination and calibration. Model calibration was generally poor.

CONCLUSION

Summary statistics and imputation methods can improve model discrimination and reduce bias during model development, but it is questionable whether these differences are clinically significant. Researchers should consider why data are missing during model development and how this may impact clinical utility.

Development and Internal Validation of a Risk Prediction Model to Identify Myeloma Based on Routine Blood Tests: A Case-Control Study

Myeloma is one of the hardest cancers to diagnose in primary care due to its rarity and non-specific symptoms. A rate-limiting step in diagnosing myeloma is the clinician considering myeloma and initiating appropriate investigations. We developed and internally validated a risk prediction model to identify those with a high risk of having undiagnosed myeloma based on results from routine blood tests taken for other reasons. A case-control study, based on 367 myeloma cases and 1488 age- and sex-matched controls, was used to develop a risk prediction model including results from 15 blood tests. The model had excellent discrimination (C-statistic 0.85 (95%CI 0.83, 0.89)) and good calibration (calibration slope 0.87 (95%CI 0.75, 0.90)). At a prevalence of 15 per 100,000 population and a probability threshold of 0.4, approximately 600 patients would need additional reflex testing to detect one case. We showed that it is possible to combine signals and abnormalities from several routine blood test parameters to identify individuals at high-risk of having undiagnosed myeloma who may benefit from additional reflex testing. Further work is needed to explore the full potential of such a strategy, including whether it is clinically useful and cost-effective and how to make it ethically acceptable.

Multivariate longitudinal data for survival analysis of cardiovascular event prediction in young adults: insights from a comparative explainable study

BACKGROUND

Multivariate longitudinal data are under-utilized for survival analysis compared to cross-sectional data (CS - data collected once across cohort). Particularly in cardiovascular risk prediction, despite available methods of longitudinal data analysis, the value of longitudinal information has not been established in terms of improved predictive accuracy and clinical applicability.

METHODS

We investigated the value of longitudinal data over and above the use of cross-sectional data via 6 distinct modeling strategies from statistics, machine learning, and deep learning that incorporate repeated measures for survival analysis of the time-to-cardiovascular event in the Coronary Artery Risk Development in Young Adults (CARDIA) cohort. We then examined and compared the use of model-specific interpretability methods (Random Survival Forest Variable Importance) and model-agnostic methods (SHapley Additive exPlanation (SHAP) and Temporal Importance Model Explanation (TIME)) in cardiovascular risk prediction using the top-performing models.

RESULTS

In a cohort of 3539 participants, longitudinal information from 35 variables that were repeatedly collected in 6 exam visits over 15 years improved subsequent long-term (17 years after) risk prediction by up to 8.3% in C-index compared to using baseline data (0.78 vs. 0.72), and up to approximately 4% compared to using the last observed CS data (0.75). Time-varying AUC was also higher in models using longitudinal data (0.86-0.87 at 5 years, 0.79-0.81 at 10 years) than using baseline or last observed CS data (0.80-0.86 at 5 years, 0.73-0.77 at 10 years). Comparative model interpretability analysis revealed the impact of longitudinal variables on model prediction on both the individual and global scales among different modeling strategies, as well as identifying the best time windows and best timing within that window for event prediction. The best strategy to incorporate longitudinal data for accuracy was time series massive feature extraction, and the easiest interpretable strategy was trajectory clustering.

CONCLUSION

Our analysis demonstrates the added value of longitudinal data in predictive accuracy and epidemiological utility in cardiovascular risk survival analysis in young adults via a unified, scalable framework that compares model performance and explainability. The framework can be extended to a larger number of variables and other longitudinal modeling methods.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT00005130, Registration Date: 26/05/2000.

BLOod Test Trend for cancEr Detection (BLOTTED): protocol for an observational and prediction model development study using English primary care electronic health record data

BACKGROUND

Simple blood tests can play an important role in identifying patients for cancer investigation. The current evidence base is limited almost entirely to tests used in isolation. However, recent evidence suggests combining multiple types of blood tests and investigating trends in blood test results over time could be more useful to select patients for further cancer investigation. Such trends could increase cancer yield and reduce unnecessary referrals. We aim to explore whether trends in blood test results are more useful than symptoms or single blood test results in selecting primary care patients for cancer investigation. We aim to develop clinical prediction models that incorporate trends in blood tests to identify the risk of cancer.

METHODS

Primary care electronic health record data from the English Clinical Practice Research Datalink Aurum primary care database will be accessed and linked to cancer registrations and secondary care datasets. Using a cohort study design, we will describe patterns in blood testing (aim 1) and explore associations between covariates and trends in blood tests with cancer using mixed-effects, Cox, and dynamic models (aim 2). To build the predictive models for the risk of cancer, we will use dynamic risk modelling (such as multivariate joint modelling) and machine learning, incorporating simultaneous trends in multiple blood tests, together with other covariates (aim 3). Model performance will be assessed using various performance measures, including c-statistic and calibration plots.

DISCUSSION

These models will form decision rules to help general practitioners find patients who need a referral for further investigation of cancer. This could increase cancer yield, reduce unnecessary referrals, and give more patients the opportunity for treatment and improved outcomes.

Risk prediction of mortality for patients with heart failure in England: observational study in primary care

AIMS

Many risk prediction models have been proposed for heart failure (HF), but few studies have used only information available to general practitioners (GPs) in primary care electronic health records (EHRs). We describe the predictors and performance of models built from GP-based EHRs in two cohorts of patients 10 years apart.

METHODS AND RESULTS

Linked primary and secondary care data for incident HF cases in England were extracted from the Clinical Practice Research Datalink for 2001-02 and 2011-12. Time-to-event models for all-cause mortality were developed using a long list of potential baseline predictors. Discrimination and calibration were calculated. A total of 5966 patients in 156 general practices were diagnosed in 2001-02, and 12 827 patients in 331 practices were diagnosed in 2011-12. The 5-year survival rate was 40.0% in 2001-02 and 40.2% in 2011-12, though the latter population were older, frailer, and more comorbid; for 2001-02, the 10-year survival was 20.8% and 15-year survival 11.1%. Consistent predictors included age, male sex, systolic blood pressure, body mass index, GP domiciliary visits before diagnosis, and some comorbidities. Model performance for both time windows was modest (c = 0.70), but calibration was generally excellent in both time periods.

CONCLUSIONS

Information routinely available to UK GPs at the time of diagnosis of HF gives only modest predictive accuracy of all-cause mortality, making it hard to decide on the type, place, and urgency of follow-up. More consistent recording of data relevant to HF (such as echocardiography and natriuretic peptide results) in GP EHRs is needed to support accurate prediction of healthcare needs in individuals with HF.

Modeling strategies to analyse longitudinal biomarker data: An illustration on predicting immunotherapy non-response in non-small cell lung cancer

Serum tumor markers acquired through a blood draw are known to reflect tumor activity. Their non-invasive nature allows for more frequent testing compared to traditional imaging methods used for response evaluations. Our study aims to compare nine prediction methods to accurately, and with a low false positive rate, predict progressive disease despite treatment (i.e. non-response) using longitudinal tumor biomarker data. Bi-weekly measurements of CYFRA, CA-125, CEA, NSE, and SCC were available from a cohort of 412 advanced stage non-small cell lung cancer (NSCLC) patients treated up to two years with immune checkpoint inhibitors. Serum tumor marker measurements from the first six weeks after treatment initiation were used to predict treatment response at 6 months. Nine models with varying complexity were evaluated in this study, showing how longitudinal biomarker data can be used to predict non-response to immunotherapy in NSCLC patients.

Modelling of longitudinal data to predict cardiovascular disease risk: a methodological review

OBJECTIVE

The identification of methodology for modelling cardiovascular disease (CVD) risk using longitudinal data and risk factor trajectories.

METHODS

We screened MEDLINE-Ovid from inception until 3 June 2020. MeSH and text search terms covered three areas: data type, modelling type and disease area including search terms such as "longitudinal", "trajector*" and "cardiovasc*" respectively. Studies were filtered to meet the following inclusion criteria: longitudinal individual patient data in adult patients with ≥3 time-points and a CVD or mortality outcome. Studies were screened and analyzed by one author. Any queries were discussed with the other authors. Comparisons were made between the methods identified looking at assumptions, flexibility and software availability.

RESULTS

From the initial 2601 studies returned by the searches 80 studies were included. Four statistical approaches were identified for modelling the longitudinal data: 3 (4%) studies compared time points with simple statistical tests, 40 (50%) used single-stage approaches, such as including single time points or summary measures in survival models, 29 (36%) used two-stage approaches including an estimated longitudinal parameter in survival models, and 8 (10%) used joint models which modelled the longitudinal and survival data together. The proportion of CVD risk prediction models created using longitudinal data using two-stage and joint models increased over time.

CONCLUSIONS

Single stage models are still heavily utilized by many CVD risk prediction studies for modelling longitudinal data. Future studies should fully utilize available longitudinal data when analyzing CVD risk by employing two-stage and joint approaches which can often better utilize the available data.

Informative presence and observation in routine health data: A review of methodology for clinical risk prediction

Objective

Informative presence (IP) is the phenomenon whereby the presence or absence of patient data is potentially informative with respect to their health condition, with informative observation (IO) being the longitudinal equivalent. These phenomena predominantly exist within routinely collected healthcare data, in which data collection is driven by the clinical requirements of patients and clinicians. The extent to which IP and IO are considered when using such data to develop clinical prediction models (CPMs) is unknown, as is the existing methodology aiming at handling these issues. This review aims to synthesize such existing methodology, thereby helping identify an agenda for future methodological work.

Materials and Methods

A systematic literature search was conducted by 2 independent reviewers using prespecified keywords.

Results

Thirty-six articles were included. We categorized the methods presented within as derived predictors (including some representation of the measurement process as a predictor in the model), modeling under IP, and latent structures. Including missing indicators or summary measures as predictors is the most commonly presented approach amongst the included studies (24 of 36 articles).

Discussion

This is the first review to collate the literature in this area under a prediction framework. A considerable body relevant of literature exists, and we present ways in which the described methods could be developed further. Guidance is required for specifying the conditions under which each method should be used to enable applied prediction modelers to use these methods.

Conclusions

A growing recognition of IP and IO exists within the literature, and methodology is increasingly becoming available to leverage these phenomena for prediction purposes. IP and IO should be approached differently in a prediction context than when the primary goal is explanation. The work included in this review has demonstrated theoretical and empirical benefits of incorporating IP and IO, and therefore we recommend that applied health researchers consider incorporating these methods in their work.

Toward a framework for the design, implementation, and reporting of methodology scoping reviews

BACKGROUND AND OBJECTIVE

In view of the growth of published articles, there is an increasing need for studies that summarize scientific research. An increasingly common review is a "methodology scoping review," which provides a summary of existing analytical methods, techniques and software that have been proposed or applied in research articles to address an analytical problem or further an analytical approach. However, guidelines for their design, implementation, and reporting are limited.

METHODS

Drawing on the experiences of the authors, which were consolidated through a series of face-to-face workshops, we summarize the challenges inherent in conducting a methodology scoping review and offer suggestions of best practice to promote future guideline development.

RESULTS

We identified three challenges of conducting a methodology scoping review. First, identification of search terms; one cannot usually define the search terms a priori, and the language used for a particular method can vary across the literature. Second, the scope of the review requires careful consideration because new methodology is often not described (in full) within abstracts. Third, many new methods are motivated by a specific clinical question, where the methodology may only be documented in supplementary materials. We formulated several recommendations that build upon existing review guidelines. These recommendations ranged from an iterative approach to defining search terms through to screening and data extraction processes.

CONCLUSION

Although methodology scoping reviews are an important aspect of research, there is currently a lack of guidelines to standardize their design, implementation, and reporting. We recommend a wider discussion on this topic.