Participation in the Global Corporate Challenge®, a Four-Month Workplace Pedometer Program, Reduces Psychological Distress

BACKGROUND

Psychological distress (stress) has been linked to an increased risk of chronic diseases and is exacerbated by a range of workplace factors. Physical activity has been shown to alleviate psychological distress. Previous pedometer-based intervention evaluations have tended to focus on physical health outcomes. This study aimed to investigate the immediate and long-term changes in psychological distress in employees based in Melbourne, Australia after their participation in a four-month pedometer-based program in sedentary workplaces.

METHODS

At baseline, 716 adults (aged 40 ± 10 years, 40% male) employed in primarily sedentary occupations, voluntarily enrolled in the Global Corporate Challenge© (GCC©), recruited from 10 Australian workplaces to participate in the GCC^® Evaluation Study, completed the Kessler 10 Psychological Distress Scale (K10). Of these, 422 completed the K10 at baseline, 4 months and 12 months.

RESULTS

Psychological distress reduced after participation in a four-month workplace pedometer-based program, which was sustained eight months after the program ended. Participants achieving the program goal of 10,000 steps per day or with higher baseline psychological distress had the greatest immediate and sustained reductions in psychological distress. Demographic predictors of immediate reduced psychological distress (n = 489) was having an associate professional occupation, younger age, and being 'widowed, separated or divorced'.

CONCLUSIONS

Participation in a workplace pedometer-based program is associated with a sustained reduction in psychological distress. Low-impact physical health programs conducted in groups or teams that integrate a social component may be an avenue to improve both physical and psychological health in the workplace.

Using randomization tests to assess treatment effects in multiple-group interrupted time series analysis

RATIONALE, AIMS, AND OBJECTIVES

Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. However, multiple-group ITSA typically has small sample sizes, and parametric methods for multiple-group ITSA require strong assumptions that are unlikely to be met, possibly resulting in misleading P values. In this paper, randomization tests are introduced as a non-parametric, distribution-free option for computing exact P values.

METHOD

The effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales is evaluated by comparing California (CA) to Montana (MT) and Idaho (ID)-the two best matched control states not exposed to any smoking reduction initiatives. Results from randomization tests are contrasted to those of interrupted time series analysis regression (ITSAREG)-a commonly used parametric approach for evaluating treatment effects in ITSA studies.

RESULTS

Both approaches found ID and MT to be comparable to CA on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the postintervention period (P < 0.01).

CONCLUSIONS

In these data, randomization tests computed P values comparable with ITSAREG, bolstering confidence in the intervention effect. Routinely including randomization tests as a complement, or alternative, to parametric methods is therefore beneficial because randomization tests are free of assumptions regarding sample size and distribution and are extremely flexible in the choice of test statistic.

Using forecast modelling to evaluate treatment effects in single-group interrupted time series analysis

RATIONALE, AIMS, AND OBJECTIVES

Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied serially over time and the intervention is expected to "interrupt" the level and/or trend of that outcome. ITSA is commonly evaluated using methods which may produce biased results if model assumptions are violated. In this paper, treatment effects are alternatively assessed by using forecasting methods to closely fit the preintervention observations and then forecast the post-intervention trend. A treatment effect may be inferred if the actual post-intervention observations diverge from the forecasts by some specified amount.

METHOD

The forecasting approach is demonstrated using the effect of California's Proposition 99 for reducing cigarette sales. Three forecast models are fit to the preintervention series-linear regression (REG), Holt-Winters (HW) non-seasonal smoothing, and autoregressive moving average (ARIMA)-and forecasts are generated into the post-intervention period. The actual observations are then compared with the forecasts to assess intervention effects.

RESULTS

The preintervention data were fit best by HW, followed closely by ARIMA. REG fit the data poorly. The actual post-intervention observations were above the forecasts in HW and ARIMA, suggesting no intervention effect, but below the forecasts in the REG (suggesting a treatment effect), thereby raising doubts about any definitive conclusion of a treatment effect.

CONCLUSIONS

In a single-group ITSA, treatment effects are likely to be biased if the model is misspecified. Therefore, evaluators should consider using forecast models to accurately fit the preintervention data and generate plausible counterfactual forecasts, thereby improving causal inference of treatment effects in single-group ITSA studies.

Health Care Cost Analyses for Exploring Cost Savings Opportunities in Older Patients: Longitudinal Retrospective Study

BACKGROUND

Half of Medicare reimbursement goes toward caring for the top 5% of the most expensive patients. However, little is known about these patients prior to reaching the top or how their costs change annually. To address these gaps, we analyzed patient flow and associated health care cost trends over 5 years.

OBJECTIVE

To evaluate the cost of health care utilization in older patients by analyzing changes in their long-term expenditures.

METHODS

This was a retrospective, longitudinal, multicenter study to evaluate health care costs of 2643 older patients from 2011 to 2015. All patients had at least one episode of home health care during the study period and used a personal emergency response service (PERS) at home for any length of time during the observation period. We segmented all patients into top (5%), middle (6%-50%), and bottom (51%-100%) segments by their annual expenditures and built cost pyramids based thereon. The longitudinal health care expenditure trends of the complete study population and each segment were assessed by linear regression models. Patient flows throughout the segments of the cost acuity pyramids from year to year were modeled by Markov chains.

RESULTS

Total health care costs of the study population nearly doubled from US $17.7M in 2011 to US $33.0M in 2015 with an expected annual cost increase of US $3.6M (P=.003). This growth was primarily driven by a significantly higher cost increases in the middle segment (US $2.3M, P=.003). The expected annual cost increases in the top and bottom segments were US $1.2M (P=.008) and US $0.1M (P=.004), respectively. Patient and cost flow analyses showed that 18% of patients moved up the cost acuity pyramid yearly, and their costs increased by 672%. This was in contrast to 22% of patients that moved down with a cost decrease of 86%. The remaining 60% of patients stayed in the same segment from year to year, though their costs also increased by 18%.

CONCLUSIONS

Although many health care organizations target intensive and costly interventions to their most expensive patients, this analysis unveiled potential cost savings opportunities by managing the patients in the lower cost segments that are at risk of moving up the cost acuity pyramid. To achieve this, data analytics integrating longitudinal data from electronic health records and home monitoring devices may help health care organizations optimize resources by enabling clinicians to proactively manage patients in their home or community environments beyond institutional settings and 30- and 60-day telehealth services.

Using machine learning to evaluate treatment effects in multiple-group interrupted time series analysis

RATIONALE, AIMS, AND OBJECTIVES

Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time, and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. In this paper, we introduce a novel machine learning approach using optimal discriminant analysis (ODA) to evaluate treatment effects in multiple-group ITSA.

METHOD

We evaluate the effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales, by comparing California (CA) to Montana (MT)-the best matching control state not exposed to any smoking reduction initiatives. We contrast results from ODA to those of ITSA regression (ITSAREG)-a commonly used approach for evaluating treatment effects in ITSA studies.

RESULTS

Both approaches found CA and MT to be comparable on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the post-intervention period (P < 0.0001). The ODA model achieved very high effect strength of sensitivity (a measure of classification accuracy) of 91.67%, which remained high (75.00%) after conducting leave-one-out analysis to assess generalizability.

CONCLUSIONS

The ODA framework achieved results comparable to ITSAREG, bolstering confidence in the intervention effect. In addition, ODA confers several advantages over conventional approaches that may make it a better approach to use in multiple group ITSA studies: insensitivity to skewed data, model-free permutation tests to derive P values, identification of the threshold value which best discriminates intervention and control groups, a chance- and maximum-corrected index of classification accuracy, and cross-validation to assess generalizability.

Using group-based trajectory modelling to enhance causal inference in interrupted time series analysis

RATIONALE, AIMS, AND OBJECTIVES

Several enhancements have been proposed for interrupted time series analysis (ITSA) to improve causal inference. Presently, group-based trajectory modelling (GBTM) is introduced as a complement to ITSA. GBTM assumes a certain number of discrete groups in the sample have unique trajectories of the outcome. GBTM is used herein for 2 purposes: (1) to compare outcomes across all trajectory groups via a stand-alone GBTM and (2) to identify comparable non-treated units to serve as controls in the ITSA outcome model. Examples of each are offered.

METHOD

The effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales is evaluated by comparing California to other states not exposed to smoking reduction initiatives. In the stand-alone GBTM, distinct trajectory groups are identified based on cigarette sales for the entire observation period (1970-2000). In the second approach, a GBTM is generated using only baseline period observations (1970-1988), and treatment effects (difference in post-intervention trends) are then estimated for the treatment unit versus non-treated units in the treated unit's trajectory group.

RESULTS

In the stand-alone GBTM, 3 distinct trajectory groups were identified: low-decreasing, medium-decreasing, and high-decreasing (California and 26 other states were in the low-decreasing group). When using baseline data for matching, California and 19 non-treated states comprised the low group. California had a significantly larger decrease in post-intervention cigarette sales than these controls (P < 0.01).

CONCLUSIONS

GBTM enhances ITSA by providing perspective for the outcome trajectory in the treated unit's group versus all others and can identify non-treated units to be used for estimating treatment effects.

Using permutation tests to enhance causal inference in interrupted time series analysis

RATIONALE, AIMS AND OBJECTIVES

Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied serially over time and the intervention is expected to "interrupt" the level and/or trend of that outcome. The internal validity is strengthened considerably when the treated unit is contrasted with a comparable control group. In this paper, we introduce a robustness check based on permutation tests to further improve causal inference.

METHOD

We evaluate the effect of California's Proposition 99 for reducing cigarette sales by iteratively casting each nontreated state into the role of "treated," creating a comparable control group using the ITSAMATCH package in Stata, and then evaluating treatment effects using ITSA regression. If statistically significant "treatment effects" are estimated for pseudotreated states, then any significant changes in the outcome of the actual treatment unit (California) cannot be attributed to the intervention. We perform these analyses setting the cutpoint significance level to P > .40 for identifying balanced matches (the highest threshold possible for which controls could still be found for California) and use the difference in differences of trends as the treatment effect estimator.

RESULTS

Only California attained a statistically significant treatment effect, strengthening confidence in the conclusion that Proposition 99 reduced cigarette sales.

CONCLUSIONS

The proposed permutation testing framework provides an additional robustness check to either support or refute a treatment effect identified in for the true treated unit in ITSA. Given its value and ease of implementation, this framework should be considered as a standard robustness test in all multiple group interrupted time series analyses.

Combining synthetic controls and interrupted time series analysis to improve causal inference in program evaluation

RATIONALE, AIMS AND OBJECTIVES

Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome. The internal validity is strengthened considerably when the treated unit is contrasted with a comparable control group. In this paper, we introduce a robust evaluation framework that combines the synthetic controls method (SYNTH) to generate a comparable control group and ITSA regression to assess covariate balance and estimate treatment effects.

METHODS

We evaluate the effect of California's Proposition 99 for reducing cigarette sales, by comparing California to other states not exposed to smoking reduction initiatives. SYNTH is used to reweight nontreated units to make them comparable to the treated unit. These weights are then used in ITSA regression models to assess covariate balance and estimate treatment effects.

RESULTS

Covariate balance was achieved for all but one covariate. While California experienced a significant decrease in the annual trend of cigarette sales after Proposition 99, there was no statistically significant treatment effect when compared to synthetic controls.

CONCLUSIONS

The advantage of using this framework over regression alone is that it ensures that a comparable control group is generated. Additionally, it offers a common set of statistical measures familiar to investigators, the capability for assessing covariate balance, and enhancement of the evaluation with a comprehensive set of postestimation measures. Therefore, this robust framework should be considered as a primary approach for evaluating treatment effects in multiple group time series analysis.

Identifying causal mechanisms in health care interventions using classification tree analysis

RATIONALE, AIMS, AND OBJECTIVES

Mediation analysis identifies causal pathways by testing the relationships between the treatment, the outcome, and an intermediate variable that mediates the relationship between the treatment and outcome. This paper introduces classification tree analysis (CTA), a machine-learning procedure, as an alternative to conventional methods for analysing mediation effects.

METHOD

Using data from the JOBS II study, we compare CTA to structural equation models (SEMs) by assessing their consistency in revealing mediation effects on 2 outcomes; reemployment (a binary variable) and depressive symptoms (a continuous variable). Because study participants were not randomized sequentially to both treatment and mediator, an additional model was generated including baseline covariates to strengthen the validity of some key identifying assumptions required of all mediation analyses.

RESULTS

Using SEM, no statistically significant treatment or mediated effects were found for either outcome. In contrast, CTA found a significant treatment effect for reemployment (P = .047) and a mediated pathway for individuals in the treatment group (P = .014). No CTA model could be generated for the reemployment outcome. When covariates were added to the model, CTA found numerous interactions, while SEM found no effects.

CONCLUSIONS

CTA may uncover mediation effects where conventional approaches do not, because CTA does not require any assumptions about the distribution of variables nor of the functional form of the model, and CTA will systematically identify all statistically viable interactions. The versatility of CTA enables the investigator to explore the theorized underlying causal mechanism of an intervention in a much more comprehensive manner than conventional mediation analytic approaches.

A matching framework to improve causal inference in interrupted time-series analysis

RATIONALE, AIMS, AND OBJECTIVES

Interrupted time-series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. When ITSA is implemented without a comparison group, the internal validity may be quite poor. Therefore, adding a comparable control group to serve as the counterfactual is always preferred. This paper introduces a novel matching framework, ITSAMATCH, to create a comparable control group by matching directly on covariates and then use these matches in the outcomes model.

METHOD

We evaluate the effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales, by comparing California to other states not exposed to smoking reduction initiatives. We compare ITSAMATCH results to 2 commonly used matching approaches, synthetic controls (SYNTH), and regression adjustment; SYNTH reweights nontreated units to make them comparable to the treated unit, and regression adjusts covariates directly. Methods are compared by assessing covariate balance and treatment effects.

RESULTS

Both ITSAMATCH and SYNTH achieved covariate balance and estimated similar treatment effects. The regression model found no treatment effect and produced inconsistent covariate adjustment.

CONCLUSIONS

While the matching framework achieved results comparable to SYNTH, it has the advantage of being technically less complicated, while producing statistical estimates that are straightforward to interpret. Conversely, regression adjustment may "adjust away" a treatment effect. Given its advantages, ITSAMATCH should be considered as a primary approach for evaluating treatment effects in multiple-group time-series analysis.

Estimating causal effects for survival (time-to-event) outcomes by combining classification tree analysis and propensity score weighting

RATIONALE, AIMS AND OBJECTIVES

A common approach to assessing treatment effects in nonrandomized studies with time-to-event outcomes is to estimate propensity scores and compute weights using logistic regression, test for covariate balance, and then estimate treatment effects using Cox regression. A machine-learning alternative-classification tree analysis (CTA)-used to generate propensity scores and to estimate treatment effects in time-to-event data may identify complex relationships between covariates not found using conventional regression-based approaches.

METHOD

Using empirical data, we identify all statistically valid CTA propensity score models and then use them to compute strata-specific, observation-level propensity score weights that are subsequently applied in outcomes analyses. We compare findings obtained using this framework to the conventional method, by evaluating covariate balance and treatment effect estimates obtained using Cox regression and a weighted CTA outcomes model.

RESULTS

All models had some imbalanced covariates. Nevertheless, treatment effect estimates were generally consistent across all weighted models.

CONCLUSIONS

In the study sample, given that all approaches elicited similar results, using CTA increased confidence that bias could not be reduced any further. Because the CTA algorithm identifies all statistically valid propensity score models for a sample, it is most likely to identify a correctly specified propensity score model-and therefore should be used either to confirm results using traditional methods, or to reveal biases that may be missed by traditional methods. Moreover, given that the true treatment effect is never known in observational data, CTA should be considered for estimating outcomes because no statistical assumptions are required.

Minimizing imbalances on patient characteristics between treatment groups in randomized trials using classification tree analysis

RATIONALE, AIMS, AND OBJECTIVES

Randomization ensures that treatment groups do not differ systematically in their characteristics, thereby reducing threats to validity that may otherwise explain differences in outcomes. Large observed imbalances in patient characteristics may indicate that selection bias is being introduced into the treatment allocation process. We introduce classification tree analysis (CTA) as a novel algorithmic approach for identifying potential imbalances in characteristics and their interactions when provisionally assigning each new participant to one or the other treatment group. The participant is then permanently assigned to the treatment group that elicits either no or less imbalance than if assigned to the alternate group.

METHOD

Using data on participant characteristics from a clinical trial, we compare 3 different treatment allocation approaches: permuted block randomization (the original allocation method), minimization, and CTA. Treatment allocation performance is assessed by examining balance of all 17 patient characteristics between study groups for each of the allocation techniques.

RESULTS

While all 3 treatment allocation techniques achieved excellent balance on main effect variables, Classification tree analysis further identified imbalances on interactions and in the distributions of some of the continuous variables.

CONCLUSIONS

Classification tree analysis offers an algorithmic procedure that may be used with any randomization methodology to identify and then minimize linear, nonlinear, and interactive effects that induce covariate imbalance between groups. Investigators should consider using the CTA approach as a real-time complement to randomization for any clinical trial to safeguard the treatment allocation process against bias.

Modeling time-to-event (survival) data using classification tree analysis

RATIONALE, AIMS, AND OBJECTIVES

Time to the occurrence of an event is often studied in health research. Survival analysis differs from other designs in that follow-up times for individuals who do not experience the event by the end of the study (called censored) are accounted for in the analysis. Cox regression is the standard method for analysing censored data, but the assumptions required of these models are easily violated. In this paper, we introduce classification tree analysis (CTA) as a flexible alternative for modelling censored data. Classification tree analysis is a "decision-tree"-like classification model that provides parsimonious, transparent (ie, easy to visually display and interpret) decision rules that maximize predictive accuracy, derives exact P values via permutation tests, and evaluates model cross-generalizability.

METHOD

Using empirical data, we identify all statistically valid, reproducible, longitudinally consistent, and cross-generalizable CTA survival models and then compare their predictive accuracy to estimates derived via Cox regression and an unadjusted naïve model. Model performance is assessed using integrated Brier scores and a comparison between estimated survival curves.

RESULTS

The Cox regression model best predicts average incidence of the outcome over time, whereas CTA survival models best predict either relatively high, or low, incidence of the outcome over time.

CONCLUSIONS

Classification tree analysis survival models offer many advantages over Cox regression, such as explicit maximization of predictive accuracy, parsimony, statistical robustness, and transparency. Therefore, researchers interested in accurate prognoses and clear decision rules should consider developing models using the CTA-survival framework.

Persistent threats to validity in single-group interrupted time series analysis with a cross over design

RATIONALE, AIMS AND OBJECTIVES

The basic single-group interrupted time series analysis (ITSA) design has been shown to be susceptible to the most common threat to validity-history-the possibility that some other event caused the observed effect in the time series. A single-group ITSA with a crossover design (in which the intervention is introduced and withdrawn 1 or more times) should be more robust. In this paper, we describe and empirically assess the susceptibility of this design to bias from history.

METHOD

Time series data from 2 natural experiments (the effect of multiple repeals and reinstatements of Louisiana's motorcycle helmet law on motorcycle fatalities and the association between the implementation and withdrawal of Gorbachev's antialcohol campaign with Russia's mortality crisis) are used to illustrate that history remains a threat to ITSA validity, even in a crossover design.

RESULTS

Both empirical examples reveal that the single-group ITSA with a crossover design may be biased because of history. In the case of motorcycle fatalities, helmet laws appeared effective in reducing mortality (while repealing the law increased mortality), but when a control group was added, it was shown that this trend was similar in both groups. In the case of Gorbachev's antialcohol campaign, only when contrasting the results against those of a control group was the withdrawal of the campaign found to be the more likely culprit in explaining the Russian mortality crisis than the collapse of the Soviet Union.

CONCLUSIONS

Even with a robust crossover design, single-group ITSA models remain susceptible to bias from history. Therefore, a comparable control group design should be included, whenever possible.

Challenges to validity in single-group interrupted time series analysis

RATIONALE, AIMS AND OBJECTIVES

Single-group interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single unit of observation is studied; the outcome variable is serially ordered as a time series, and the intervention is expected to "interrupt" the level and/or trend of the time series, subsequent to its introduction. The most common threat to validity is history-the possibility that some other event caused the observed effect in the time series. Although history limits the ability to draw causal inferences from single ITSA models, it can be controlled for by using a comparable control group to serve as the counterfactual.

METHOD

Time series data from 2 natural experiments (effect of Florida's 2000 repeal of its motorcycle helmet law on motorcycle fatalities and California's 1988 Proposition 99 to reduce cigarette sales) are used to illustrate how history biases results of single-group ITSA results-as opposed to when that group's results are contrasted to those of a comparable control group.

RESULTS

In the first example, an external event occurring at the same time as the helmet repeal appeared to be the cause of a rise in motorcycle deaths, but was only revealed when Florida was contrasted with comparable control states. Conversely, in the second example, a decreasing trend in cigarette sales prior to the intervention raised question about a treatment effect attributed to Proposition 99, but was reinforced when California was contrasted with comparable control states.

CONCLUSIONS

Results of single-group ITSA should be considered preliminary, and interpreted with caution, until a more robust study design can be implemented.

Using machine learning to identify structural breaks in single-group interrupted time series designs

RATIONALE, AIMS AND OBJECTIVES

Single-group interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single unit of observation is being studied, the outcome variable is serially ordered as a time series and the intervention is expected to 'interrupt' the level and/or trend of the time series, subsequent to its introduction. Given that the internal validity of the design rests on the premise that the interruption in the time series is associated with the introduction of the treatment, treatment effects may seem less plausible if a parallel trend already exists in the time series prior to the actual intervention. Thus, sensitivity analyses should focus on detecting structural breaks in the time series before the intervention.

METHOD

In this paper, we introduce a machine-learning algorithm called optimal discriminant analysis (ODA) as an approach to determine if structural breaks can be identified in years prior to the initiation of the intervention, using data from California's 1988 voter-initiated Proposition 99 to reduce smoking rates.

RESULTS

The ODA analysis indicates that numerous structural breaks occurred prior to the actual initiation of Proposition 99 in 1989, including perfect structural breaks in 1983 and 1985, thereby casting doubt on the validity of treatment effects estimated for the actual intervention when using a single-group ITSA design.

CONCLUSIONS

Given the widespread use of ITSA for evaluating observational data and the increasing use of machine-learning techniques in traditional research, we recommend that structural break sensitivity analysis is routinely incorporated in all research using the single-group ITSA design.

Combining machine learning and propensity score weighting to estimate causal effects in multivalued treatments

RATIONALE, AIMS AND OBJECTIVES

Interventions with multivalued treatments are common in medical and health research; examples include comparing the efficacy of competing interventions and contrasting various doses of a drug. In recent years, there has been growing interest in the development of methods that estimate multivalued treatment effects using observational data. This paper extends a previously described analytic framework for evaluating binary treatments to studies involving multivalued treatments utilizing a machine learning algorithm called optimal discriminant analysis (ODA).

METHOD

We describe the differences between regression-based treatment effect estimators and effects estimated using the ODA framework. We then present an empirical example using data from an intervention including three study groups to compare corresponding effects.

RESULTS

The regression-based estimators produced statistically significant mean differences between the two intervention groups, and between one of the treatment groups and controls. In contrast, ODA was unable to discriminate between distributions of any of the three study groups.

CONCLUSIONS

Optimal discriminant analysis offers an appealing alternative to conventional regression-based models for estimating effects in multivalued treatment studies because of its insensitivity to skewed data and use of accuracy measures applicable to all prognostic analyses. If these analytic approaches produce consistent treatment effect P values, this bolsters confidence in the validity of the results. If the approaches produce conflicting treatment effect P values, as they do in our empirical example, the investigator should consider the ODA-derived estimates to be most robust, given that ODA uses permutation P values that require no distributional assumptions and are thus, always valid.

Combining machine learning and matching techniques to improve causal inference in program evaluation

RATIONALE, AIMS AND OBJECTIVES

Program evaluations often utilize various matching approaches to emulate the randomization process for group assignment in experimental studies. Typically, the matching strategy is implemented, and then covariate balance is assessed before estimating treatment effects. This paper introduces a novel analytic framework utilizing a machine learning algorithm called optimal discriminant analysis (ODA) for assessing covariate balance and estimating treatment effects, once the matching strategy has been implemented. This framework holds several key advantages over the conventional approach: application to any variable metric and number of groups; insensitivity to skewed data or outliers; and use of accuracy measures applicable to all prognostic analyses. Moreover, ODA accepts analytic weights, thereby extending the methodology to any study design where weights are used for covariate adjustment or more precise (differential) outcome measurement.

METHOD

One-to-one matching on the propensity score was used as the matching strategy. Covariate balance was assessed using standardized difference in means (conventional approach) and measures of classification accuracy (ODA). Treatment effects were estimated using ordinary least squares regression and ODA.

RESULTS

Using empirical data, ODA produced results highly consistent with those obtained via the conventional methodology for assessing covariate balance and estimating treatment effects.

CONCLUSIONS

When ODA is combined with matching techniques within a treatment effects framework, the results are consistent with conventional approaches. However, given that it provides additional dimensions and robustness to the analysis versus what can currently be achieved using conventional approaches, ODA offers an appealing alternative.

The Immediate and Sustained long-Term Changes in Daytime Sleepiness After Participation in a Workplace Pedometer Program: A Prospective Cohort Study

OBJECTIVE

To assess the potential benefit of a workplace physical activity program on daytime sleepiness.

METHODS

A total of 685 participants of a 4-month workplace physical activity program were assessed for daytime sleepiness (Epworth Sleepiness Scale [ESS]) at baseline, 4 months (postprogram), and 12 months. Changes in ESS were analyzed using multilevel mixed linear regression.

RESULTS

In the total population, no changes in ESS scores were observed; 0 to 4 months: -0.2 (95% CI: -0.5 to 0.0), 4 to 12 months: 0.1 (95% CI: -0.2 to 0.4). In participants with baseline excessive daytime sleepiness (ESS > 10, n = 109), ESS scores improved significantly by -2.2 (95% CI: -3.0 to -1.4) at 4 months, sustained at 12 months; and almost half no longer had excessive daytime sleepiness by end of program.

CONCLUSIONS

This study suggests that for employees with excessive daytime sleepiness, short- and long-term improvement in daytime sleepiness may be an unforeseen benefit of workplace physical activity programs.

Clinical effects of an optimised care program with telehealth in heart failure patients in a community hospital in the Netherlands

Background

Our hypothesis was that telehealth in combination with an optimised care program coordinated amongst care professionals in primary, secondary and tertiary care can achieve beneficial outcomes in heart failure. The objective was to evaluate the clinical effects of introduction of telehealth in an optimised care program in a community hospital in the north of the Netherlands.

Methods

We compared the number of unplanned admissions for heart failure in the year before and after adding telehealth to the optimised care program. Furthermore, blood pressure and N-terminal pro-B-type natriuretic peptide (NT-proBNP) levels were evaluated at baseline and 3, 6 and 12 months after telehealth. Quality of life and knowledge about the disease were regularly evaluated via surveys on the telehealth system.

Findings

The number of unplanned admissions for heart failure decreased from on average 1.29 to 0.31 admissions per year after telehealth introduction. Blood pressure decreased independent of medication and NT-proBNP levels improved as well. Quality of life increased during the telehealth intervention and disease knowledge remained high throughout the follow-up period. Unplanned admissions that remained after telehealth introduction could be accurately predicted at baseline by a multivariate regression model.

Change in well-being amongst participants in a four-month pedometer-based workplace health program

BACKGROUND

There is increasing uptake of workplace physical activity programs to prevent chronic disease. While they are frequently evaluated for improvement in biomedical risk factors there has been little evaluation of additional benefits for psychosocial health. We aimed to evaluate whether participation in a four-month, team-based, pedometer-based workplace health program known to improve biomedical risk factors is associated with an improvement in well-being, immediately after the program and eight-months after program completion.

METHODS

At baseline (2008), 762 adults (aged 40 ± 10 SD years, 42% male) employed in primarily sedentary occupations and voluntarily enrolled in a physical activity program were recruited from ten Australian worksites. Data was collected at baseline, at the completion of the four-month program and eight-months after program completion. The outcome was the WHO-Five Well-being Index (WHO-5), a self-administered five-item scale that can be dichotomised as 'poor' (less than 52%) or 'positive' (more than or equal to 52%) well-being.

RESULTS

At baseline, 75% of participants had positive well-being (mean: 60 ± 19 SD WHO-5 units). On average, well-being improved immediately after the health program (+3.5 units, p < 0.001) and was sustained eight-months later (+3.4 units from baseline, p < 0.001). In the 25% with poor well-being at baseline, 49.5% moved into the positive well-being category immediately after program completion, sustained eight-months later (p < 0.001).

CONCLUSIONS

Clinically relevant immediate and sustained improvements in well-being were observed after participation in the health program. These results suggest that participation in workplace programs, such as the one evaluated here, also has the potential to improve well-being.

Assessing regression to the mean effects in health care initiatives

Background

Interventions targeting individuals classified as “high-risk” have become common-place in health care. High-risk may represent outlier values on utilization, cost, or clinical measures. Typically, such individuals are invited to participate in an intervention intended to reduce their level of risk, and after a period of time, a follow-up measurement is taken. However, individuals initially identified by their outlier values will likely have lower values on re-measurement in the absence of an intervention. This statistical phenomenon is known as “regression to the mean” (RTM) and often leads to an inaccurate conclusion that the intervention caused the effect. Concerns about RTM are rarely raised in connection with most health care interventions, and it is uncommon to find evaluators who estimate its effect. This may be due to lack of awareness, cognitive biases that may cause people to systematically misinterpret RTM effects by creating (erroneous) explanations to account for it, or by design.

Methods

In this paper, the author fully describes the RTM phenomenon, and tests the accuracy of the traditional approach in calculating RTM assuming normality, using normally distributed data from a Monte Carlo simulation and skewed data from a control group in a pre-post evaluation of a health intervention. Confidence intervals are generated around the traditional RTM calculation to provide more insight into the potential magnitude of the bias introduced by RTM. Finally, suggestions are offered for designing interventions and evaluations to mitigate the effects of RTM.

Results

On multivariate normal data, the calculated RTM estimates are identical to true estimates. As expected, when using skewed data the calculated method underestimated the true RTM effect. Confidence intervals provide helpful guidance on the magnitude of the RTM effect.

Conclusion

Decision-makers should always consider RTM to be a viable explanation of the observed change in an outcome in a pre-post study, and evaluators of health care initiatives should always take the appropriate steps to estimate the magnitude of the effect and control for it when possible. Regardless of the cause, failure to address RTM may result in wasteful pursuit of ineffective interventions, both at the organizational level and at the policy level.

Patients with type 2 diabetes benefit from primary care-based disease management: a propensity score matched survival time analysis

This study aimed to assess the impact of a nationwide German diabetes mellitus disease management program (DMP) on survival time and costs in comparison to routine care. The authors conducted a retrospective observational cohort study using routine administration data from Germany's largest sickness fund to identify insured suffering from diabetes in 2002. A total of 95,443 insured with type 2 diabetes mellitus who were born before January 1, 1962 met the defined inclusion criteria, resulting in 19,888 pairs of DMP participants and nonparticipants matched for socioeconomic and health status using propensity score matching methods. This is the first time propensity score matching has been used to evaluate a survival benefit of DMPs. In the time frame analyzed (3 years), mean survival time for the DMP group was 1045 days vs. 985 days for the routine care group (P<0.001). Mean daily hospital and total costs (including DMP administration and medical costs) were lower for the DMP group in the case of deceased insureds (92€ vs. 139€ and 122€ vs. 169€, respectively) as well as for censored observations (6€ vs. 7€ and 12.9€ vs. 13.4€, respectively). Mean daily drug costs were slightly lower for deceased insured in the DMP group (difference 0.6€), while no identifiable difference was found for censored observations. In this study, insured who were enrolled in a DMP for diabetes mellitus in the German Statutory Health Insurance showed a significant benefit in survival time. They also incurred lower costs compared to propensity score matched insured in routine care.

Applying a propensity score-based weighting model to interrupted time series data: improving causal inference in programme evaluation

Often, when conducting programme evaluations or studying the effects of policy changes, researchers may only have access to aggregated time series data, presented as observations spanning both the pre- and post-intervention periods. The most basic analytic model using these data requires only a single group and models the intervention effect using repeated measurements of the dependent variable. This model controls for regression to the mean and is likely to detect a treatment effect if it is sufficiently large. However, many potential sources of bias still remain. Adding one or more control groups to this model could strengthen causal inference if the groups are comparable on pre-intervention covariates and level and trend of the dependent variable. If this condition is not met, the validity of the study findings could be called into question. In this paper we describe a propensity score-based weighted regression model, which overcomes these limitations by weighting the control groups to represent the average outcome that the treatment group would have exhibited in the absence of the intervention. We illustrate this technique studying cigarette sales in California before and after the passage of Proposition 99 in California in 1989. While our results were similar to those of the Synthetic Control method, the weighting approach has the advantage of being technically less complicated, rooted in regression techniques familiar to most researchers, easy to implement using any basic statistical software, may accommodate any number of treatment units, and allows for greater flexibility in the choice of treatment effect estimators.

Using propensity score-based weighting in the evaluation of health management programme effectiveness

When the randomized controlled trial is unfeasible, programme evaluators attempt to emulate the randomization process in observational studies by creating a control group that is essentially equivalent to the treatment group on known characteristics and trust that the remaining unknown characteristics are inconsequential and will not bias the results. In recent years, adjustment procedures based on the propensity score, such as matching and subclassification, have become increasingly popular. A new technique that has particular appeal for evaluating health management programmes uses the propensity score to create a weight based on the subject's inverse probability of receiving treatment. This weighting mechanism removes imbalances of pre-intervention characteristics between treated and non-treated individuals, and is then used within a regression framework to provide unbiased estimates of treatment effects. This paper presents a non-technical introduction of this technique by illustrating its implementation with data from a recent study estimating the impact of a motivational interviewing-based health coaching on patient activation measure scores in a chronically ill group of individuals. Because of its relative simplicity and tremendous utility, propensity-score weighting adjustment should be considered as an alternative procedure for use with observational data to evaluate health management programme effectiveness.

Improving participant selection in disease management programmes: insights gained from propensity score stratification

While the randomized controlled trial (RCT) remains the gold-standard study design for evaluating treatment effect, outcomes researchers turn to powerful quasi-experimental designs when only observational studies can be conducted. Within these designs, propensity score matching is one of the most popular to evaluate disease management (DM) programme effectiveness. Given that DM programmes generally have a much smaller number of participants than non-participants in the population, propensity score matching will typically result in all or nearly all participants finding successful matches, while most of the non-participants in the population remain unmatched and thereby excluded from the analysis. By excluding data from the unmatched population, the effect of non-treatment in the remaining population with the disease is not captured. In the present study, we examine changes in hospitalization rates stratified by propensity score quintiles across the entire population allowing us to gain insight as to how well the programme chose its participants, or if the programme could have been effective on those individuals not explicitly targeted for the intervention. These data indicate the presence of regression to the mean, and suggest that the DM programme may be overly limited to only the highest strata when there is evidence of a potential benefit for those in all the lower strata as well.

Regression to the mean: a limited issue in disease management programs for chronic obstructive pulmonary disease

Our objective was to test for evidence of regression to the mean in chronic obstructive pulmonary disease (COPD)-related health care utilization in a Colorado Medicaid population that met the criteria for, but were not participating in, a COPD disease management (DM) program. National Jewish Medical and Research Center had enrolled individuals who (1) had a diagnosis of COPD for at least 1 year and (2) were active participants in Colorado Medicaid's 1-year DM program called breatheWise; the present study sought a comparator group for that population. In order to test for evidence of regression to the mean (ie, high utilization from the recruitment period reducing without active intervention) in this case management model, we conducted a case-controlled analysis of total spending for a comparator population that would have met the inclusion criteria for the DM program. The present study assessed health care utilization for fiscal years 2002 and 2003 in terms of total rates of emergency room (ER) visits and hospitalizations for all causes in the comparator group of COPD patients. In addition, total costs related to both ER visits and hospitalizations were compiled. In total, 354 individuals met the inclusion criteria and were identified as the comparator group. ER visits and hospitalizations were consistent for 2002 and 2003. ER visits totaled 314 and 315 in 2002 and 2003, respectively, indicating a 0.3% increase that was not significant. Hospitalizations decreased from 0.53 admissions per patient in 2002 to 0.48 in 2003-a 9.4% reduction that was not significant. With comparable rates of ER visits and hospitalizations, total costs for health care utilization remained virtually unchanged between 2002 and 2003. There is minimal evidence of regression to the mean over 2 consecutive years in the Colorado Medicaid patients with moderate to severe COPD.

Disease management in chronic kidney disease

Chronic kidney disease (CKD) is a growing health problem of epidemic proportions both in the United States and worldwide. The care of CKD patients, before and after starting dialysis, remains highly fragmented resulting in suboptimal clinical outcomes and high costs, creating a high burden of disease on patients and the health care system. Disease management (DM) is an approach to coordinating care for this complex population of patients that has the promise of improving outcomes and constraining costs. For CKD patients not yet on dialysis, the major goals of a DM program are (1) early identification of CKD patients and therapy to slow the progression of CKD, (2) identification and management of the complications of CKD per se, (3) identification and management of the complications of comorbid conditions, and (4) smooth transition to renal replacement therapy. For those CKD patients on dialysis, focused attention on avoidable hospitalizations is a key to a successful DM program. Multidisciplinary collaboration among physicians (nephrologist, primary care physician, cardiologist, endocrinologist, vascular surgeons, and transplant physicians) and participating caregivers (nurse, pharmacist, social worker, and dietician) is critical as well. There are several potential barriers to the successful implementation of a CKD/end-stage renal disease DM program, including lack of awareness of the disease state among patients and health care providers, late identification and referrals to a nephrologist, complex fragmented care delivered by multiple providers in many different sites of care, and reimbursement that does not align incentives for all involved. Recent experience suggests that these barriers can be overcome, with DM becoming a promising approach for improving outcomes for this vulnerable population.

The case-mix of chronic illness hospitalization rates in a managed care population: implications for health management programmes

OBJECTIVE

This paper reports on the case-mix of hospitalized patients based on their health plan enrollment and utilization experience, absent a health management (HM) programme. The implications for achieving targeted reductions in admissions within the context of implementing a population hm programme are discussed.

STUDY DESIGN

descriptive.

METHODS

Members were identified with asthma, coronary artery disease, congestive heart failure and diabetes. These cohorts were then mapped to disease-specific hospitalizations across a 2-year period (2004-2005). Four distinct case-mix categories were developed. Group 1 comprised members hospitalized for the specific condition in both years. Group 2 comprised all identified members of a disease cohort in 2004 that were not hospitalized in that year but were admitted for the condition in 2005. Members were assigned to Group 3 if they were hospitalized in 2005, did not appear in the 2004 identified cohort but were, in fact, enrolled in the health plan. Group 4 comprised new health plan enrollees in 2005 and were subsequently hospitalized during that year.

RESULTS

Of the total admissions in 2005, on average 6.4% came from Group 1, 62.4% came from group 2, 10% from group 3 and 21.2% from Group 4.

CONCLUSIONS

If an HM programme was to be implemented in this population, the typical identification methods currently used by the industry would have resulted in most hospitalized patients either being initially classified as low risk or going undetected. Improving identification and stratification methods will allow HM programmes to better tailor interventions to impact hospitalization rates for the chronically ill.