Predictive risk algorithms in a population setting: an overview

The impact of different imputation methods on estimates and model performance: an example using a risk prediction model for premature mortality

OBJECTIVE

To compare how different imputation methods affect the estimates and performance of a prediction model for premature mortality.

STUDY DESIGN AND SETTING

Sex-specific Weibull accelerated failure time survival models were run on four separate datasets using complete case, mode, single and multiple imputation to impute missing values. Six performance measures were compared to access predictive accuracy (Nagelkerke R2, integrated brier score), discrimination (Harrell's c-index, discrimination slope) and calibration (calibration in the large, calibration slope).

RESULTS

The highest proportion of missingness for a single variable was 10.86% for the female model and 8.24% for the male model. Comparing the performance measures for complete case, mode, single and multiple imputation: the Nagelkerke R2 values for the female model was 0.1084, 0.1116, 0.1120 and 0.111-0.1120 with the male model exhibited similar variation of 0.1050, 0.1078, 0.1078 and 0.1078-0.1081. Harrell's c-index also demonstrated small variation with values of 0.8666, 0.8719, 0.8719 and 0.8711-0.8719 for the female model and 0.8549, 0.8548, 0.8550 and 0.8550-0.8553 for the male model.

CONCLUSION

In the scenarios examined in this study, mode imputation performed well when using a population health survey compared to single and multiple imputation when predictive performance measures is the main model goal. To generate unbiased hazard ratios, multiple imputation methods were superior. This study shows the need to consider the best imputation approach for a predictive model development given the conditions of missing data and the goals of the analysis.

External validation and application of the Diabetes Population Risk Tool (DPoRT) for prediction of type 2 diabetes onset in the US population

INTRODUCTION

Characterizing diabetes risk in the population is important for population health assessment and diabetes prevention planning. We aimed to externally validate an existing 10-year population risk model for type 2 diabetes in the USA and model the population benefit of diabetes prevention approaches using population survey data.

RESEARCH DESIGN AND METHODS

The Diabetes Population Risk Tool (DPoRT), originally derived and validated in Canada, was applied to an external validation cohort of 23 477 adults from the 2009 National Health Interview Survey (NHIS). We assessed predictive performance for discrimination (C-statistic) and calibration plots against observed incident diabetes cases identified from the NHIS 2009-2018 cycles. We applied DPoRT to the 2018 NHIS cohort (n=21 187) to generate 10-year risk prediction estimates and characterize the preventive benefit of three diabetes prevention scenarios: (1) community-wide strategy; (2) high-risk strategy and (3) combined approach.

RESULTS

DPoRT demonstrated good discrimination (C-statistic=0.778 (males); 0.787 (females)) and good calibration across the range of risk. We predicted a baseline risk of 10.2% and 21 076 000 new cases of diabetes in the USA from 2018 to 2028. The community-wide strategy and high-risk strategy estimated diabetes risk reductions of 0.2% and 0.3%, respectively. The combined approach estimated a 0.4% risk reduction and 843 000 diabetes cases averted in 10 years.

CONCLUSIONS

DPoRT has transportability for predicting population-level diabetes risk in the USA using routinely collected survey data. We demonstrate the model's applicability for population health assessment and diabetes prevention planning. Our modeling predicted that the combination of community-wide and targeted prevention approaches for those at highest risk are needed to reduce diabetes burden in the USA.

Development and validation of cardiovascular risk prediction equations in 76 000 people with known cardiovascular disease

AIMS

Multiple health administrative databases can be individually linked in Aotearoa New Zealand, using encrypted identifiers. These databases were used to develop cardiovascular risk prediction equations for patients with known cardiovascular disease (CVD).

METHODS AND RESULTS

Administrative health databases were linked to identify all people aged 18-84 years with known CVD, living in Auckland and Northland, Aotearoa New Zealand, on 1 January 2014. The cohort was followed until study outcome, death, or 5 years. The study outcome was death or hospitalization due to ischaemic heart disease, stroke, heart failure, or peripheral vascular disease. Sex-specific 5-year CVD risk prediction equations were developed using multivariable Fine and Gray models. A total of 43 862 men {median age: 67 years [interquartile range (IQR): 59-75]} and 32 724 women [median age: 70 years (IQR: 60-77)] had 14 252 and 9551 cardiovascular events, respectively. Equations were well calibrated with good discrimination. Increasing age and deprivation, recent cardiovascular hospitalization, Mori ethnicity, smoking history, heart failure, diabetes, chronic renal disease, atrial fibrillation, use of blood pressure lowering and anti-thrombotic drugs, haemoglobin A1c, total cholesterol/HDL cholesterol, and creatinine were statistically significant independent predictors of the study outcome. Fourteen per cent of men and 23% of women had predicted 5-year cardiovascular risk <15%, while 28 and 24% had ≥40% risk.

CONCLUSION

Robust cardiovascular risk prediction equations were developed from linked routine health databases, a currently underutilized resource worldwide. The marked heterogeneity demonstrated in predicted risk suggests that preventive therapy in people with known CVD would be better informed by risk stratification beyond a one-size-fits-all high-risk categorization.

Modelling population-level and targeted interventions of weight loss on chronic disease prevention in the Canadian population

Obesity is a known risk factor for major chronic diseases. Prevention of chronic disease is a top global priority. The study aimed to model scenarios of population-level and targeted weight loss interventions on 10-year projected risk of chronic disease in Canada using a population-level risk prediction algorithm. The validated Chronic Disease Population Risk Tool (CDPoRT) forecasts 10-year risk of chronic disease in the adult population. We applied CDPoRT to the 2013/14 Canadian Community Health Survey to generate prospective chronic disease estimates for adults 20 years and older in Canada (n = 83,220). CDPoRT was used to model the following scenarios: British Columbia's (BC) and Quebec's (QC) provincial population-level weight reduction targets, a population-level intervention that could achieve weight loss, targeted weight loss interventions for overweight and obese groups, and the combination of a population-level and targeted weight loss intervention. We estimated chronic disease risk reductions and number of cases prevented in each scenario compared with the baseline. At baseline, we predicted an 18.4% risk and 4,151,929 new cases of chronic disease in Canada over the 10-year period. Provincial weight loss targets applied to the Canadian population estimated chronic disease reductions of 0.6% (BC) and 0.1% (QC). The population-level intervention estimated a greater reduction in risk (0.2%), compared to the targeted interventions (0.1%). The combined approach estimated a 0.3% reduction in chronic disease risk. Our modelling predicted that population-level approaches that achieve weight loss in combination with targeted weight loss interventions can substantially decrease the chronic disease burden in Canada.

Predicting cardiovascular risk from national administrative databases using a combined survival analysis and deep learning approach

Background

Machine learning-based risk prediction models may outperform traditional statistical models in large datasets with many variables, by identifying both novel predictors and the complex interactions between them. This study compared deep learning extensions of survival analysis models with Cox proportional hazards models for predicting cardiovascular disease (CVD) risk in national health administrative datasets.

Methods

Using individual person linkage of administrative datasets, we constructed a cohort of all New Zealanders aged 30–74 who interacted with public health services during 2012. After excluding people with prior CVD, we developed sex-specific deep learning and Cox proportional hazards models to estimate the risk of CVD events within 5 years. Models were compared based on the proportion of explained variance, model calibration and discrimination, and hazard ratios for predictor variables.

Results

First CVD events occurred in 61 927 of 2 164 872 people. Within the reference group, the largest hazard ratios estimated by the deep learning models were for tobacco use in women (2.04, 95% CI: 1.99, 2.10) and chronic obstructive pulmonary disease with acute lower respiratory infection in men (1.56, 95% CI: 1.50, 1.62). Other identified predictors (e.g. hypertension, chest pain, diabetes) aligned with current knowledge about CVD risk factors. Deep learning outperformed Cox proportional hazards models on the basis of proportion of explained variance (R²: 0.468 vs 0.425 in women and 0.383 vs 0.348 in men), calibration and discrimination (all P <0.0001).

Conclusions

Deep learning extensions of survival analysis models can be applied to large health administrative datasets to derive interpretable CVD risk prediction equations that are more accurate than traditional Cox proportional hazards models.

Downstream health impacts of employment losses during the COVID-19 pandemic

Objectives

The Canadian workforce has experienced significant employment losses during the COVID-19 pandemic, in part as a result of non-pharmaceutical interventions to slow COVID-19 transmission. Health consequences are likely to result from these job losses, but without historical precedent for the current economic shutdown they are challenging to plan for. Our study aimed to use population risk models to quantify potential downstream health impacts of the COVID-19 pandemic and inform public health planning to minimize future health burden.

Methods

The impact of COVID-19 job losses on future premature mortality and high-resource health care utilization (HRU) was estimated using an economic model of Canadian COVID-19 lockdowns and validated population risk models. Five-year excess premature mortality and HRU were estimated by age and sex to describe employment-related health consequences of COVID-19 lockdowns in the Canadian population.

Results

With federal income supplementation like the Canadian Emergency Response Benefit, we estimate that each month of economic lockdown will result in 5.6 new high-resource health care system users (HRUs), and 4.1 excess premature deaths, per 100,000, over the next 5 years. These effects were concentrated in ages 45–64, and among males 18–34. Without income supplementation, the health consequences were approximately twice as great in terms of both HRUs and premature deaths.

Conclusion

Employment losses associated with COVID-19 countermeasures may have downstream implications for health. Public health responses should consider financially vulnerable populations at high risk of downstream health outcomes.

Supplementary Information

The online version contains supplementary material available at 10.17269/s41997-021-00588-3.

Self-help behaviors partially mediate the relationship between personalized depression risk disclosure and psychological distress: A mediation analysis using data from a randomized controlled trial

Major depressive disorder (MDD) is one of the most prevalent forms of mental illness. Multivariate risk predictive (MVRP) algorithms provide a new means of identifying high-risk individuals for mental health disorders. Self-help behaviors may provide accessible methods to mitigate depression risk. The objective of this study is to investigate the mediating effects of self-help behavior on the relationship between depression risk disclosure and psychological distress. A sample (n = 556) of high-risk Canadians for a major depressive episode (MDE) were randomized into risk-disclosure or control groups and followed-up at 6 and 12 months. Mediation analysis using repeated measure mixed effects models was used to investigate the mediating effects of self-help behaviors on the relationship between depression risk disclosure and psychological distress over time. Self-help behavior was found to partially mediate the relationship between risk disclosure and psychological distress at month 12. Both unadjusted and adjusted associations were found to be negative and significant (ß_unadj = -0.16 [-0.30, -0.03]) (ß_adj = -0.15[-0.29, -0.02]). Self-help plays a partial mediating role in the relationship between depression risk disclosure and psychological distress over time. More research is required in this field to increase knowledge about the role of self-help in mental health treatment.

Forecasting Diabetes Cases Prevented and Cost Savings Associated with Population Increases of Walking in the Greater Toronto and Hamilton Area, Canada

Promoting adequate levels of physical activity in the population is important for diabetes prevention. However, the scale needed to achieve tangible population benefits is unclear. We aimed to estimate the public health impact of increases in walking as a means of diabetes prevention and health care cost savings attributable to diabetes. We applied the validated Diabetes Population Risk Tool (DPoRT) to the 2015/16 Canadian Community Health Survey for adults aged 18–64, living in the Greater Toronto and Hamilton area, Ontario, Canada. DPoRT was used to generate three population-level scenarios involving increases in walking among individuals with low physical activity levels, low daily step counts and high dependency on non-active forms of travel, compared to a baseline scenario (no change in walking rates). We estimated number of diabetes cases prevented and health care costs saved in each scenario compared with the baseline. Each of the three scenarios predicted a considerable reduction in diabetes and related health care cost savings. In order of impact, the largest population benefits were predicted from targeting populations with low physical activity levels, low daily step counts, and non active transport use. Population increases of walking by 25 min each week was predicted to prevent up to 10.4 thousand diabetes cases and generate CAD 74.4 million in health care cost savings in 10 years. Diabetes reductions and cost savings were projected to be higher if increases of 150 min of walking per week could be achieved at the population-level (up to 54.3 thousand diabetes cases prevented and CAD 386.9 million in health care cost savings). Policy, programming, and community designs that achieve modest increases in population walking could translate to meaningful reductions in the diabetes burden and cost savings to the health care system.

Development and validation of a predictive algorithm for risk of dementia in the community setting

Background

Most dementia algorithms are unsuitable for population-level assessment and planning as they are designed for use in the clinical setting. A predictive risk algorithm to estimate 5-year dementia risk in the community setting was developed.

Methods

The Dementia Population Risk Tool (DemPoRT) was derived using Ontario respondents to the Canadian Community Health Survey (survey years 2001 to 2012). Five-year incidence of physician-diagnosed dementia was ascertained by individual linkage to administrative healthcare databases and using a validated case ascertainment definition with follow-up to March 2017. Sex-specific proportional hazards regression models considering competing risk of death were developed using self-reported risk factors including information on socio-demographic characteristics, general and chronic health conditions, health behaviours and physical function.

Results

Among 75 460 respondents included in the combined derivation and validation cohorts, there were 8448 cases of incident dementia in 348 677 person-years of follow-up (5-year cumulative incidence, men: 0.044, 95% CI: 0.042 to 0.047; women: 0.057, 95% CI: 0.055 to 0.060). The final full models each include 90 df (65 main effects and 25 interactions) and 28 predictors (8 continuous). The DemPoRT algorithm is discriminating (C-statistic in validation data: men 0.83 (95% CI: 0.81 to 0.85); women 0.83 (95% CI: 0.81 to 0.85)) and well-calibrated in a wide range of subgroups including behavioural risk exposure categories, socio-demographic groups and by diabetes and hypertension status.

Conclusions

This algorithm will support the development and evaluation of population-level dementia prevention strategies, support decision-making for population health and can be used by individuals or their clinicians for individual risk assessment.

Antibiotic Prescribing Choices and Their Comparative C. Difficile Infection Risks: A Longitudinal Case-Cohort Study

BACKGROUND

Antibiotic use is the strongest modifiable risk factor for the development of Clostridioides difficile infection, but prescribers lack quantitative information on comparative risks of specific antibiotic courses. Our objective was to estimate risks of C. difficile infection associated with receipt of specific antibiotic courses.

METHODS

We conducted a longitudinal case-cohort analysis representing over 90% of Ontario nursing home residents, between 2012 and 2017. Our primary exposure was days of antibiotic receipt in the prior 90 days. Adjustment covariates included: age, sex, prior emergency department or acute care stay, Charlson comorbidity index, prior C. difficile infection, acid suppressant use, device use, and functional status. We examined incident C. difficile infection, including cases identified within the nursing home, and those identified during subsequent hospital admissions. Adjusted and unadjusted regression models were used to measure risk associated with 5- to 14-day courses of 18 different antibiotics.

RESULTS

We identified 1708 cases of C. difficile infection (1.27 per 100 000 resident-days). Longer antibiotic duration was associated with increased risk: 10- and 14-day courses incurred 12% (adjusted relative risk [ARR] = 1.12, 95% confidence interval [CI]: 1.09, 1.14) and 27% (ARR = 1.27, 95% CI: 1.21,1.30) more risk compared to 7-day courses. Among 7-day courses with similar indications: moxifloxacin resulted in 121% more risk than amoxicillin (ARR = 2.21, 95% CI: 1.67, 3.08), ciprofloxacin engendered 89% more risk than nitrofurantoin (ARR = 1.89, 95% CI: 1.45, 2.68), and clindamycin resulted in 112% (ARR = 2.12, 95% CI: 1.32, 3.78) more risk than cloxacillin.

CONCLUSIONS

C. difficile infection risk increases with antibiotic duration, and there are wide disparities in risks associated with antibiotic courses used for similar indications.

Prognosis tools for short-term adverse events in older emergency department users: result of a Québec observational prospective cohort

Background

The “Program of Research on the Integration of Services for the Maintenance of Autonomy” (PRISMA-7) and “Emergency room evaluation and recommendations” (ER²) are both clinical tools used in Québec Emergency Departments (EDs) for screening of older ED users at higher risk of poor outcomes, such as prolonged length of stay (LOS) in EDs and in hospital. The study aimed to: 1) examine whether the PRISMA-7 and ER² risk levels were associated with length of stays in ED and hospital, as well as hospital admission; and 2) compare the criteria performance (i.e., sensitivity, specificity, positive predictive value, negative predictive value, likelihood ratios and area under receiver operating characteristic curve) of the PRISMA-7 and ER2 high-risk levels for these three ED adverse events in Québec older patients visiting ED on a stretcher.

Methods

A total of 1905 older patients who visited the ED of the Jewish General Hospital (Montreal, Québec, Canada) on stretchers were recruited in this prospective observational cohort. Upon their ED arrival, PRISMA-7 and ER² were performed. The outcomes were LOS in ED and in hospital, and hospital admission.

Results

The PRISMA-7 and ER² risk levels were associated with length of stay in ED and hospital as well as with hospital admission. Prolonged stays and higher hospitalization rates were associated with high-risk levels, whereas those in low-risk level groups had significantly shorter LOS and a lower rate of hospital admission (P < 0.006). While performance measures were poor for both assessment tools, ER² had a greater prognostic testing accuracy compared with PRISMA-7.

Conclusion

PRISMA-7 and ER² were both associated with incidental short-term ED adverse events but their overall prognostic testing accuracy was low, suggesting that they cannot be used as prognostic tools for this purpose.

The Canadian Health Clock and health calculators

Setting

This paper documents a participatory process of Health Portfolio staff in the design of a clock, and announces the 2020 Canadian Health Clock, with links to numerous online health calculators. The clock is part of the Health Portfolio’s celebration activities in 2019 of “100 Years of Health”, as the Department of Health was established in Canada in 1919.

Intervention

The intervention was the development of a clock on the Government of Canada website with linkage to calculators as a health promotion tool. The clock was built on the concept of the 2004 Chronic Disease Clock, which shows the number of deaths so far today, and so far this year. The clock was developed using a consultative approach, following a review of the original clock.

Outcomes

The 2020 clock incorporates new data visualization concepts. New features, facilitated by improved technology, include: expansion to all causes of death; blinking red dots to enhance visual impact; and three clock versions (analogue, featuring a moving circle; digital, table format; and graphical, bar chart format). The clock also provides links to a number of health calculators, to allow people to seek personalized information to improve their health.

Implications

The online health clock and health calculators are good examples of innovation in health risk communication tools for effective knowledge translation and dissemination. They inform people about health statistics (clock) and their health (calculators). The clock engages people in the context of the Canadian population, whereas the calculators provide personalized information about improving an individual’s future health.

The future burden of obesity in Canada: a modelling study

OBJECTIVES

We applied the validated Obesity Population Risk Tool (OPoRT) to estimate the future burden of obesity in Canada using baseline risk factors attained through routinely collected survey data.

METHODS

OPoRT was developed using logistic regression with sex-specific generalized estimating equations to predict the 10-year prevalence of obesity (outcome BMI ≥ 30.0) among adults 18 and older. The algorithm includes 17 predictive factors, including socio-demographic and health behavioural characteristics. OPoRT demonstrated excellent discrimination (C-statistic ≥ 0.89) and achieved calibration. We applied OPoRT to Canadian Community Health Survey (2013/14) data to predict the future prevalence of obesity in Canada for a variety of population subgroups.

RESULTS

The predicted burden of obesity grew from 261 cases per 1000 in 2013/14 to 326 cases per 1000 in 2023/24 corresponding to a total of 8.54 million individuals with obesity. The burden is expected to be higher among males (347 cases per 1000) than females (305 cases per 1000). Individuals aged 35-49 had the highest predicted burden of obesity (374 cases per 1000) and the largest number of predicted cases (2.42 million), while individuals in the ≥ 65 age group had the lowest predicted burden (236 cases per 1000). The number of individuals with obesity per 1000 is highest among those severely food insecure (452 cases per 1000), compared with food secure individuals (324 cases per 1000).

CONCLUSIONS

OPoRT can be used to estimate the future population burden of obesity, to identify priority subgroups at an elevated risk. Burden estimates should be reflected in approaches to curb the future burden of obesity.

Development and validation of a cardiovascular disease risk-prediction model using population health surveys: the Cardiovascular Disease Population Risk Tool (CVDPoRT)

BACKGROUND

Routinely collected data from large population health surveys linked to chronic disease outcomes create an opportunity to develop more complex risk-prediction algorithms. We developed a predictive algorithm to estimate 5-year risk of incident cardiovascular disease in the community setting.

METHODS

We derived the Cardiovascular Disease Population Risk Tool (CVDPoRT) using prospectively collected data from Ontario respondents of the Canadian Community Health Surveys, representing 98% of the Ontario population (survey years 2001 to 2007; follow-up from 2001 to 2012) linked to hospital admission and vital statistics databases. Predictors included body mass index, hypertension, diabetes, and multiple behavioural, demographic and general health risk factors. The primary outcome was the first major cardiovascular event resulting in hospital admission or death. Death from a noncardiovascular cause was considered a competing risk.

RESULTS

We included 104 219 respondents aged 20 to 105 years. There were 3709 cardiovascular events and 818 478 person-years follow-up in the combined derivation and validation cohorts (5-year cumulative incidence function, men: 0.026, 95% confidence interval [CI] 0.025-0.028; women: 0.018, 95% 0.017-0.019). The final CVDPoRT algorithm contained 12 variables, was discriminating (men: C statistic 0.82, 95% CI 0.81-0.83; women: 0.86, 95% CI 0.85-0.87) and was well-calibrated in the overall population (5-year observed cumulative incidence function v. predicted risk, men: 0.28%; women: 0.38%) and in nearly all predefined policy-relevant subgroups (206 of 208 groups).

INTERPRETATION

The CVDPoRT algorithm can accurately discriminate cardiovascular disease risk for a wide range of health profiles without the aid of clinical measures. Such algorithms hold potential to support precision medicine for individual or population uses. Study registration: ClinicalTrials.gov, no. NCT02267447.

Predicting High Health Care Resource Utilization in a Single-payer Public Health Care System: Development and Validation of the High Resource User Population Risk Tool

Supplemental Digital Content is available in the text.

Background:

A large proportion of health care spending is incurred by a small proportion of the population. Population-based health planning tools that consider both the clinical and upstream determinants of high resource users (HRU) of the health system are lacking.

Objective:

To develop and validate the High Resource User Population Risk Tool (HRUPoRT), a predictive model of adults that will become the top 5% of health care users over a 5-year period, based on self-reported clinical, sociodemographic, and health behavioral predictors in population survey data.

Research Design:

The HRUPoRT model was developed in a prospective cohort design using the combined 2005 and 2007/2008 Canadian Community Health Surveys (CCHS) (N=58,617), and validated using the external 2009/2010 CCHS cohort (N=28,721). Health care utilization for each of the 5 years following CCHS interview date were determined by applying a person-centered costing algorithm to the linked health administrative databases. Discrimination and calibration of the model were assessed using c-statistic and Hosmer-Lemeshow (HL) χ² statistic.

Results:

The best prediction model for 5-year transition to HRU status included 12 predictors and had good discrimination (c-statistic=0.8213) and calibration (HL χ²=18.71) in the development cohort. The model performed similarly in the validation cohort (c-statistic=0.8171; HL χ²=19.95). The strongest predictors in the HRUPoRT model were age, perceived general health, and body mass index.

Conclusions:

HRUPoRT can accurately project the proportion of individuals in the population that will become a HRU over 5 years. HRUPoRT can be applied to inform health resource planning and prevention strategies at the community level.

Development and validation of a population based risk algorithm for obesity: The Obesity Population Risk Tool (OPoRT)

Background

Given the dramatic rise in the prevalence of obesity, greater focus on prevention is necessary. We sought to develop and validate a population risk tool for obesity to inform prevention efforts.

Methods

We developed the Obesity Population Risk Tool (OPoRT) using the longitudinal National Population Health Survey and sex-specific Generalized Estimating Equations to predict the 10-year risk of obesity among adults 18 and older. The model was validated using a bootstrap approach accounting for the survey design. Model performance was measured by the Brier statistic, discrimination was measured by the C-statistic, and calibration was assessed using the Hosmer-Lemeshow Goodness of Fit Chi Square (HL χ²).

Results

Predictive factors included baseline body mass index, age, time and their interactions, smoking status, living arrangements, education, alcohol consumption, physical activity, and ethnicity. OPoRT showed good performance for males and females (Brier 0.118 and 0.095, respectively), excellent discrimination (C statistic ≥ 0.89) and achieved calibration (HL χ² <20).

Conclusion

OPoRT is a valid and reliable algorithm that can be applied to routinely collected survey data to estimate the risk of obesity and identify groups at increased risk of obesity. These results can guide prevention efforts aimed at reducing the population burden of obesity.

The current application of the Royston-Parmar model for prognostic modeling in health research: a scoping review

BACKGROUND

Prognostic models incorporating survival analysis predict the risk (i.e., probability) of experiencing a future event over a specific time period. In 2002, Royston and Parmar described a type of flexible parametric survival model called the Royston-Parmar model in Statistics in Medicine, a model which fits a restricted cubic spline to flexibly model the baseline log cumulative hazard on the proportional hazards scale. This feature permits absolute measures of effect (e.g., hazard rates) to be estimated at all time points, an important feature when using the model. The Royston-Parmar model can also incorporate time-dependent effects and be used on different scales (e.g., proportional odds, probit). These features make the Royston-Parmar model attractive for prediction, yet their current uptake for prognostic modeling is unknown. Thus, the objectives were to conduct a scoping review of how the Royston-Parmar model has been applied to prognostic models in health research, to raise awareness of the model, to identify gaps in current reporting, and to offer model building considerations and reporting suggestions for other researchers.

METHODS

Five electronic databases and gray literature indexed in web sources from 2001 to 2016 were searched to identify articles for inclusion in the scoping review. Two reviewers independently screened 1429 articles, and after applying exclusion criteria through a two-step screening process, data from 12 studies were abstracted.

RESULTS

Since 2001, only 12 studies were identified that used the Royston-Parmar model in some capacity for prognostic modeling, 10 of which used the model as the basis for their prognostic model. The restricted cubic spline varied across studies in the number of interior knots (range 1 to 6), and only three studies reported knot placement. Three studies provided details about the baseline function, with two studies using a figure and the third providing coefficients. However, no studies provided adequate information on their restricted cubic spline to permit others to validate or completely use the model.

CONCLUSIONS

Despite the advantages of the Royston-Parmar model for prognostic models, they are not widely used in health research. Better reporting of details about the restricted cubic spline is needed, so the prognostic model can be used and validated by others.

REGISTRATION

The protocol was registered with Open Science Framework (https://osf.io/r3232/).

Tools, Clinical Prediction Rules, and Algorithms for the Insertion of Peripheral Intravenous Catheters in Adult Hospitalized Patients: A Systematic Scoping Review of Literature

BACKGROUND

First-time peripheral intravenous catheter (PIVC) insertion success is dependent on patient, clinician, and product factors. Failed PIVC insertion are an under-recognized clinical phenomenon.

OBJECTIVE

To provide a scoping review of decision aids for PIVC insertion including tools, clinical prediction rules, and algorithms (TRAs) and their findings on factors associated with insertion success.

METHODS

In June 2016, a systematic literature search was performed using the medical subject heading of peripheral catheterization and tool* or rule* or algorithm*. Data extraction included clinician, patient, and/or product variables associated with PIVC insertion success. Information about TRA reliability, validity, responsiveness, and utility was also extracted.

RESULTS

We screened 36 studies, and included 13 for review. Seven papers reported insertion success ranging from 61%-90% (4030 insertion attempts), 6 on validity, and 5 on reliability, with none reporting on responsiveness and utility. Failed insertions were associated with obesity (odds ratio [OR], 0.71-1.7; 2 studies) and smaller gauge PIVCs (OR, 6.4; 95% Confidence Interval [CI}, 3.4-11.9). Successful inser tions were associated with visible veins (OR, 0.87-3.63; 3 studies) or palpable veins (OR, 0.79-5.05; 3 studies) and inserters with greater procedural volume (OR, 4.4; 95% CI, 1.6-12.1) or who predicted that insertion would be successful (OR, 1.06; 95% CI, 1.04-1.07). Definitions of insertion difficulty are heterogeneous such as time to insert to a number of failed attempts.

CONCLUSIONS

Few well-validated reliable TRAs exist for PIVC insertion. Patients would benefit from a validated, clinically pragmatic TRA that matches insertion difficulty with clinician competency.

Measuring Burden of Unhealthy Behaviours Using a Multivariable Predictive Approach: Life Expectancy Lost in Canada Attributable to Smoking, Alcohol, Physical Inactivity, and Diet

BACKGROUND

Behaviours such as smoking, poor diet, physical inactivity, and unhealthy alcohol consumption are leading risk factors for death. We assessed the Canadian burden attributable to these behaviours by developing, validating, and applying a multivariable predictive model for risk of all-cause death.

METHODS

A predictive algorithm for 5 y risk of death-the Mortality Population Risk Tool (MPoRT)-was developed and validated using the 2001 to 2008 Canadian Community Health Surveys. There were approximately 1 million person-years of follow-up and 9,900 deaths in the development and validation datasets. After validation, MPoRT was used to predict future mortality and estimate the burden of smoking, alcohol, physical inactivity, and poor diet in the presence of sociodemographic and other risk factors using the 2010 national survey (approximately 90,000 respondents). Canadian period life tables were generated using predicted risk of death from MPoRT. The burden of behavioural risk factors attributable to life expectancy was estimated using hazard ratios from the MPoRT risk model.

FINDINGS

The MPoRT 5 y mortality risk algorithms were discriminating (C-statistic: males 0.874 [95% CI: 0.867-0.881]; females 0.875 [0.868-0.882]) and well calibrated in all 58 predefined subgroups. Discrimination was maintained or improved in the validation cohorts. For the 2010 Canadian population, unhealthy behaviour attributable life expectancy lost was 6.0 years for both men and women (for men 95% CI: 5.8 to 6.3 for women 5.8 to 6.2). The Canadian life expectancy associated with health behaviour recommendations was 17.9 years (95% CI: 17.7 to 18.1) greater for people with the most favourable risk profile compared to those with the least favourable risk profile (88.2 years versus 70.3 years). Smoking, by itself, was associated with 32% to 39% of the difference in life expectancy across social groups (by education achieved or neighbourhood deprivation).

CONCLUSIONS

Multivariable predictive algorithms such as MPoRT can be used to assess health burdens for sociodemographic groups or for small changes in population exposure to risks, thereby addressing some limitations of more commonly used measurement approaches. Unhealthy behaviours have a substantial collective burden on the life expectancy of the Canadian population.

Predicting Stroke Risk Based on Health Behaviours: Development of the Stroke Population Risk Tool (SPoRT)

Background

Health behaviours, important factors in cardiovascular disease, are increasingly a focus of prevention. We appraised whether stroke risk can be accurately assessed using self-reported information focused on health behaviours.

Methods

Behavioural, sociodemographic and other risk factors were assessed in a population-based survey of 82 259 Ontarians who were followed for a median of 8.6 years (688 000 person-years follow-up) starting in 2001. Predictive algorithms for 5-year incident stroke resulting in hospitalization were created and then validated in a similar 2007 survey of 28 605 respondents (median 4.2 years follow-up).

Results

We observed 3 236 incident stroke events (1 551 resulting in hospitalization; 1 685 in the community setting without hospital admission). The final algorithms were discriminating (C-stat: 0.85, men; 0.87, women) and well-calibrated (in 65 of 67 subgroups for men; 61 of 65 for women). An index was developed to summarize cumulative relative risk of incident stroke from health behaviours and stress. For men, each point on the index corresponded to a 12% relative risk increase (180% risk difference, lowest (0) to highest (9) scores). For women, each point corresponded to a 14% relative risk increase (340% difference, lowest (0) to highest (11) scores). Algorithms for secondary stroke outcomes (stroke resulting in death; classified as ischemic; excluding transient ischemic attack; and in the community setting) had similar health behaviour risk hazards.

Conclusion

Incident stroke can be accurately predicted using self-reported information focused on health behaviours. Risk assessment can be performed with population health surveys to support population health planning or outside of clinical settings to support patient-focused prevention.

The influence of socioeconomic status on future risk for developing Type 2 diabetes in the Canadian population between 2011 and 2022: differential associations by sex

BACKGROUND

Articulating future risk of diabetes at the population level can inform prevention strategies. While previous studies have characterized diabetes burden according to socioeconomic status (SES), none have studied future risk.

METHODS

We quantified the influence of multiple constructs of SES on future diabetes risk using the Diabetes Population Risk Tool (DPoRT), a validated risk prediction algorithm that generates 10-year rates of new diabetes cases. We applied DPoRT to adults aged 30-64 in the 2011-2012 Canadian Community Health Survey (n = 65,372) and calculated risk for 2021-22. A multi-category outcome was created classifying risk as low (≤5%), moderate (greater than 5% and less than 20%), and high (≥20%), then assessed the impact of individual-level SES indicators, and area-level measures of marginalization on being moderate or high risk using multinomial logistic regression, stratified by sex.

RESULTS

We found nuanced profiles of social determinants by sex, where women are more sensitive to social context. Women living in households where highest educational attainment was less than secondary school were at greater risk [odds ratio (OR) of high compared to low diabetes risk 3.10, 95% confidence interval (CI) 2.19-4.40, p < 0.0001). The same relationship was less pronounced for males (OR 2.17, 95% CI 1.42-3.32, p = 0.0004). Lower household income and being food insecure predicted high future diabetes risk for women (OR 1.37, 95% CI 1.01-1.86, p = 0.0418 comparing quintile 1 to quintile 5; OR 2.64, 95% CI 1.78-3.92, p < 0.0001 comparing severely food insecure to food secure), but not men (OR 1.15, 95% CI 0.84-1.57, p = 0.3818 and OR 1.22, 95% CI 0.71-2.10, p = 0.4815). At the area-level, material deprivation was significantly associated with increased future risk comparing the most to the least deprived (OR females 2.39, 95% CI 1.77-3.23; OR males 1.61, 95% CI 1.22-2.14). Additionally, a strong protective effect was observed for women living in ethnically dense areas (OR 0.75, 95% CI 0.63-0.89, p = 0.0011) which was not as pronounced for men (OR 0.95, 95% CI 0.76-1.18, p = 0.6351).

CONCLUSIONS

This study characterized socio-contextual predictors for future diabetes risk, showing sex-specific effects. Diabetes prevention must consider factors beyond individual-level behavioral lifestyle change and actively take steps to mitigate the adverse impacts of socio-contextual factors.

Derivation and validation of the detection of indicators and vulnerabilities for emergency room trips scale for classifying the risk of emergency department use in frail community-dwelling older adults

OBJECTIVES

To develop and validate a prognostic case finding tool that classifies the risk of emergency department (ED) use in an older home care population.

DESIGN

Population-based retrospective cohort study using routinely collected data from home care clinical assessments linked prospectively to ED records.

SETTING

Ontario and the Winnipeg Regional Health Authority, Canada.

PARTICIPANTS

Older adults living at home and expected to receive in-home services for at least 60 days (N = 361,942).

MEASUREMENTS

One or more ED visits within 6 months after an in-home clinical assessment was used as the main dependent measure. Ninety-five person-level risk measures from a clinical assessment instrument were selected as potential independent variables. The Detection of Indicators and Vulnerabilities for Emergency Room Trips (DIVERT) Scale was derived using recursive partitioning analyses informed by a multinational clinical panel.

RESULTS

Overall, 41.2% had one or more ED visits within 6 months of their in-home assessment. Previous ED use and cardiorespiratory symptoms, cardiac conditions, and specific geriatric syndromes were predictors within the six-level DIVERT Scale. The scale provided adequate risk differentiation for case finding, with an area under the receiver operating characteristic curve of 0.62 (95% confidence interval = 0.61-0.62) and distinct risk gradients between risk scores. The multilevel validation demonstrated consistent performance across geographic and participant clusters.

CONCLUSION

The DIVERT Scale is a valid case-finding tool for ED use in older home care clients. It may be suitable for preemptively and systematically risk-stratifying individuals or groups for additional assessment, case management, and preventative interventions. It may also be suitable for the stratification and adjustment of performance metrics.

Navigating knowledge to action: a conceptual map for facilitating translation of population health risk planning tools into practice

A population health risk tool was created that estimates future diabetes risk and provides outputs that can inform practical and meaningful diabetes prevention strategies and support local decision making and planning. A project was designed to inform and understand knowledge translation and application of this novel tool in multiple health-related settings. Lacking published studies in this area, the authors conceived a conceptual map to guide the project that integrates and adapts elements from several planned action theories. This paper describes the rationale and basis for constructing the Population Health Planning Knowledge-to-Action Model and elaborates on the 2 connected structures of the framework: the Tool Creation Path and the Action Cycle. Although created for an express purpose, this model has the potential to inform application of other tools. This work demonstrates how a research team can adapt and integrate existing frameworks to better align with novel real-world knowledge translation issues. Furthermore, the integration of a population risk tool to support health decision making highlights the interaction between continuing education and knowledge translation.

Cardiovascular Disease Population Risk Tool (CVDPoRT): predictive algorithm for assessing CVD risk in the community setting. A study protocol

INTRODUCTION

Recent publications have called for substantial improvements in the design, conduct, analysis and reporting of prediction models. Publication of study protocols, with prespecification of key aspects of the analysis plan, can help to improve transparency, increase quality and protect against increased type I error. Valid population-based risk algorithms are essential for population health planning and policy decision-making. The purpose of this study is to develop, evaluate and apply cardiovascular disease (CVD) risk algorithms for the population setting.

METHODS AND ANALYSIS

The Ontario sample of the Canadian Community Health Survey (2001, 2003, 2005; 77,251 respondents) will be used to assess risk factors focusing on health behaviours (physical activity, diet, smoking and alcohol use). Incident CVD outcomes will be assessed through linkage to administrative healthcare databases (619,886 person-years of follow-up until 31 December 2011). Sociodemographic factors (age, sex, immigrant status, education) and mediating factors such as presence of diabetes and hypertension will be included as predictors. Algorithms will be developed using competing risks survival analysis. The analysis plan adheres to published recommendations for the development of valid prediction models to limit the risk of overfitting and improve the quality of predictions. Key considerations are fully prespecifying the predictor variables; appropriate handling of missing data; use of flexible functions for continuous predictors; and avoiding data-driven variable selection procedures. The 2007 and 2009 surveys (approximately 50,000 respondents) will be used for validation. Calibration will be assessed overall and in predefined subgroups of importance to clinicians and policymakers.

ETHICS AND DISSEMINATION

This study has been approved by the Ottawa Health Science Network Research Ethics Board. The findings will be disseminated through professional and scientific conferences, and in peer-reviewed journals. The algorithm will be accessible electronically for population and individual uses.

TRIAL REGISTRATION NUMBER

ClinicalTrials.gov NCT02267447.

Supporting collaborative use of the Diabetes Population Risk Tool (DPoRT) in health-related practice: a multiple case study research protocol

Background

Health policy makers have stated that diabetes prevention is a priority; however, the type, intensity, and target of interventions or policy changes that will achieve the greatest impact remains uncertain. In response to this uncertainty, the Diabetes Population Risk Tool (DPoRT) was developed and validated to estimate future diabetes risk based on routinely collected population data. To facilitate use of DPoRT, we partnered with regional and provincial health-related decision makers in Ontario and Manitoba, Canada. Primary objectives include: i) evaluate the effectiveness of partnerships between the research team and DPoRT users; ii) explore strategies that facilitate uptake and overcome barriers to DPoRT use; and iii) implement and evaluate the knowledge translation approach.

Methods

This protocol reflects an integrated knowledge translation (IKT) approach and corresponds to the action phase of the Knowledge-to-Action (KtoA) framework. Our IKT approach includes: employing a knowledge brokering team to facilitate relationships with DPoRT users (objective 1); tailored training for DPoRT users; assessment of barriers and facilitators to DPoRT use; and customized dissemination strategies to present DPoRT outputs to decision maker audiences (objective 2). Finally, a utilization-focused evaluation will assess the effectiveness and impact of the proposed KtoA process for DPoRT application (objective 3). This research design utilizes a multiple case study approach. Units of analyses consist of two public health units, one provincial health organization, and one provincial knowledge dissemination team whereby we will connect with multiple regional health authorities. Evaluation will be based on analysis of both quantitative and qualitative data collected from passive (e.g., observer notes) and active (e.g., surveys and interviews) methods.

Discussion

DPoRT offers an innovative way to make routinely collected population health data practical and meaningful for diabetes prevention planning and decision making. Importantly, we will evaluate the utility of the KtoA cycle for a novel purpose – the application of a tool. Additionally, we will evaluate this approach in multiple diverse settings, thus considering contextual factors. This research will offer insights into how knowledge translation strategies can support the use of population-based risk assessment tools to promote informed decision making in health-related settings.

Validation of a population coronary disease predictive system: the CASSANDRA model

BACKGROUND

The use of validated multivariate cardiovascular predictive models in a population setting is of interest for public health policy makers. We aimed to validate the estimations of the CASSANDRA model (coronary heart disease (CHD) incidence and CHD risk distribution), considering the population changes in age, sex and CHD risk factors prevalence in a 10-year period.

METHODS

We compared the projected CHD incidence estimated with CASSANDRA with that observed in the Girona Heart Registry (REGICOR) for 1995-2004 and 2000-2009 in the population of Girona (Spain) aged 35-74 years. We used official age and sex distributions for this population. Baseline cardiovascular risk factors prevalence and the distribution of cardiovascular risk were obtained from three cross-sectional studies performed in 1995, 2000 and 2005. To validate the future distribution of cardiovascular risk, we tested the yearly CHD risk variance over the study period.

RESULTS

No significant differences between the estimated and observed annual CHD incidence per 100 000 men were found in 1995-2004 (CASSANDRA=457.8 and REGICOR=420.3, incidence rate ratio (IRR) (95% CI)=0.92 (0.89 to 0.96)) and in 2000-2009 (441.4 and 409.6, respectively, IRR=0.93 (0.90 to 0.96)). However, overpredictions of 18% and 22%, respectively, were observed in women (198.8 and 160.4, IRR=0.82 (0.77 to 0.86), and 197.1 and 152.8, IRR=0.78 (0.74 to 0.83), respectively). No significant differences were found in the CHD risk variance in the three different cross-sectional studies.

CONCLUSIONS

The CASSANDRA model produces valid estimates, particularly in men, of the future burden of disease and in the distribution of cardiovascular risk in individuals aged 35-74 years.

Risk distribution and its influence on the population targets for diabetes prevention

OBJECTIVE

To quantify the influence of type 2 diabetes risk distribution on prevention benefit and apply a method to optimally identify population targets.

METHODS

We used data from the 2011 Canadian Community Health Survey (N=45,040) and the validated Diabetes Population Risk Tool to calculate 10-year diabetes risk. We calculated the Gini coefficient as a measure of risk dispersion. Intervention benefit was estimated using absolute risk reduction (ARR), number-needed-to-treat (NNT), and number of cases prevented.

RESULTS

There is a wide variation of diabetes risk in Canada (Gini=0.48) and with an inverse relation to risk (r=-0.99). Risk dispersion is lower among individuals meeting an empirically derived risk cut-off (Gini=0.18). Targeting prevention based on a risk cut-off (10-year risk ≥ 16.5%) resulted in a greater number of cases prevented (340 thousand), higher ARR (7.7%) and lower NNT (13) compared to targeting individuals based on risk factor targets.

CONCLUSIONS

This study provides empirical evidence to demonstrate that risk variability is an important consideration for estimating the prevention benefit. Prioritizing target populations using an empirically derived cut-off based on a multivariate risk score will result in greater benefit and efficiency compared to risk factor targets.

Development and validation of prediction algorithms for major depressive episode in the general population

BACKGROUND

To develop and validate sex specific prediction algorithms for 4-year risk of major depressive episode (MDE) using data from a population-based longitudinal cohort.

METHODS

Household residents from 10 provinces were randomly recruited and interviewed by Statistics Canada. 10,601 participants who were aged 18 years and older and who did not meet the criteria for MDE in the 12 months prior to a baseline interview in 2000/01 were included in algorithm development; data from 7902 participants who were aged 18 and older and who were free of MDE in 2004/05 were used for validation. Validation was also conducted in sub-populations that are of practice and policy importance. MDE was assessed using the World Health Organization's Composite International Diagnostic Interview(CIDI)-Short Form for Major Depression (CIDI-SFMD).

RESULTS

In the training data, the C statistics for algorithms in men was 0.7953 and was 0.7667 for algorithm in women. The algorithms had good predictive power and calibrated well in the development and validation data.

LIMITATIONS

The data relied on self-report. MDE was assessed with CIDI-SFMD. It was not feasible to validate the algorithms in different populations from different countries.

CONCLUSIONS

More studies are needed to further validate and refine these algorithms. However, the ability of a small number of easily assessed variables to predict MDE risk indicates that algorithms are a promising strategy for identifying individuals in need of enhanced monitoring and preventive interventions. Ultimately, application of algorithms may lead to increased personalization of treatment, and better clinical outcomes.