A comparison between traditional methods and multilevel regression for the analysis of multicenter intervention studies

This article reviews three traditional methods for the analysis of multicenter trials with persons nested within clusters, i.e., centers, namely naïve regression (persons as units of analysis), fixed effects regression, and the use of summary measures (clusters as units of analysis), and compares these methods with multilevel regression. The comparison is made for continuous (quantitative) outcomes, and is based on the estimator of the treatment effect and its standard error, because these usually are of main interest in intervention studies. When the results of the experiment have to be valid for some larger population of centers, the centers in the intervention study have to present a random sample from this population and multilevel regression may be used. It is shown that the treatment effect and especially its standard error, are generally incorrectly estimated by the traditional methods, which should, therefore, not in general be used as an alternative to multilevel regression.

A simulation study of sample size demonstrated the importance of the number of events per variable to develop prediction models in clustered data

OBJECTIVES

This study aims to investigate the influence of the amount of clustering [intraclass correlation (ICC) = 0%, 5%, or 20%], the number of events per variable (EPV) or candidate predictor (EPV = 5, 10, 20, or 50), and backward variable selection on the performance of prediction models.

STUDY DESIGN AND SETTING

Researchers frequently combine data from several centers to develop clinical prediction models. In our simulation study, we developed models from clustered training data using multilevel logistic regression and validated them in external data.

RESULTS

The amount of clustering was not meaningfully associated with the models' predictive performance. The median calibration slope of models built in samples with EPV = 5 and strong clustering (ICC = 20%) was 0.71. With EPV = 5 and ICC = 0%, it was 0.72. A higher EPV related to an increased performance: the calibration slope was 0.85 at EPV = 10 and ICC = 20% and 0.96 at EPV = 50 and ICC = 20%. Variable selection sometimes led to a substantial relative bias in the estimated predictor effects (up to 118% at EPV = 5), but this had little influence on the model's performance in our simulations.

CONCLUSION

We recommend at least 10 EPV to fit prediction models in clustered data using logistic regression. Up to 50 EPV may be needed when variable selection is performed.

The Consequence of Ignoring a Level of Nesting in Multilevel Analysis

Multilevel analysis is an appropriate tool for the analysis of hierarchically structured data. There may, however, be reasons to ignore one of the levels of nesting in the data analysis. In this article a three level model with one predictor variable is used as a reference model and the top or intermediate level is ignored in the data analysis. Analytical results show that this has an effect on the estimated variance components and that standard errors of regression coefficients estimators may be overestimated, leading to a lower power of the test of the effect of the predictor variable. The magnitude of these results depends on the ignored level and the level at which the predictor variable varies, and on the values of the variance components and the sample sizes.

A Statistical Evaluation of Toxicity Study Designs for the Estimation of the Benchmark Dose in Continuous Endpoints

The benchmark approach is gaining attention as an alternative to the No-Observed-Adverse-Effect-Level (NOAEL) approach. However, current guidelines for the design of toxicity tests are based on assessing a NOAEL. It has been suggested that the current study design may not be optimal for assessing a Benchmark Dose (BMD). To further investigate this we performed three simulation studies in which a large number of designs were compared, focusing on continuous endpoints. Four fictitious endpoints were considered, their underlying dose-response curves having a linear, sublinear, supralinear, or sigmoidal shape. In each simulation run the BMD was derived from a model fitted to the generated data, where the selection of the model was based on that particular data set (according to a formal likelihood ratio test procedure). Thus, the model used for deriving the BMD in a single generated data set may not be the same as the one used for generating the data. In this way, model uncertainty is taken into account as well. The results show that the performance of a design is, first of all, determined by the total number of animals used. Distributing them over more dose groups does not result in a poorer performance of the study, despite the smaller number of animals per dose group. Dose placement is another crucial factor, and to minimize the risk of inadequate dose placement, the use of multiple dose studies is favorable. As a concomitant advantage, the use of multiple doses mitigates the disturbing effect of potential systematic errors in single dose groups. However, for endpoints with large residual variation (CV > or = 18%) there is a substantial probability of not detecting the overall dose-response, and this probability increases in designs with increasing number of dose groups. In such situations, six dose groups may be used as a compromise. Designs with high dose levels (i.e., associated with relatively high effects) are helpful in estimating doses with smaller effects (such as the benchmark dose), and it appears bad practice to omit higher dose groups to improve the fit at lower doses. The typical 28-day study design of four dose groups with five animals (per sex) may not be adequate to assess endpoints with large residual variation (CV > or = 18%), both in assessing a benchmark dose and in assessing a NOAEL.

Effects of anger and anger regulation styles on pain in daily life of women with fibromyalgia: a diary study

BACKGROUND

Fibromyalgia is characterized by an amplified pain response to various physical stimuli. Through biological and behavioural mechanisms, patients with fibromyalgia may also show an increase of pain in response to emotions. Anger, and how it is regulated, may be particularly important in chronic pain.

AIM

To examine, among patients with fibromyalgia, whether anger during everyday life amplifies pain and whether general and situational anger inhibition and anger expression modulate the anger-pain link.

METHODS

For 28 consecutive days, 333 women with fibromyalgia (mean age 47+/-12years) reported their transient anger and state anger inhibition (anger-in) and expression (anger-out) responses regarding a significant emotional event during the day as well as end-of-day pain. Trait anger inhibition and expression were assessed by questionnaire. Multilevel regression analyses were performed.

RESULTS

State anger predicted higher end-of-day pain (p<.001) in half of the patients, but lower pain in one-quarter of patients. State anger inhibition was unrelated to pain. Trait anger inhibition was related to more pain (p=.02). The lowest pain level was observed among patients with high trait anger expression who actually expressed their anger in an anger-arousing situation (p=.02).

CONCLUSIONS

Our study suggests that anger and a general tendency to inhibit anger predicts heightened pain in the everyday life of female patients with fibromyalgia. Psychological intervention could focus on healthy anger expression to try to mitigate the symptoms of fibromyalgia.

A comparison of three methods for calculating confidence intervals for the benchmark dose

Various methods exist to calculate confidence intervals for the benchmark dose in risk analysis. This study compares the performance of three such methods in fitting nonlinear dose-response models: the delta method, the likelihood-ratio method, and the bootstrap method. A data set from a developmental toxicity test with continuous, ordinal, and quantal dose-response data is used for the comparison of these methods. Nonlinear dose-response models, with various shapes, were fitted to these data. The results indicate that a few thousand runs are generally needed to get stable confidence limits when using the bootstrap method. Further, the bootstrap and the likelihood-ratio method were found to give fairly similar results. The delta method, however, resulted in some cases in different (usually narrower) intervals, and appears unreliable for nonlinear dose-response models. Since the bootstrap method is more time consuming than the likelihood-ratio method, the latter is more attractive for routine dose-response analysis. In the context of a probabilistic risk assessment the bootstrap method has the advantage that it directly links to Monte Carlo analysis.

PTSD after childbirth: A predictive ethological model for symptom development

BACKGROUND

Childbirth can be a traumatic experience occasionally leading to posttraumatic stress disorder (PTSD). This study aimed to assess childbirth-related PTSD risk-factors using an etiological model inspired by the transactional model of stress and coping.

METHODS

348 out of 505 (70%) Dutch women completed questionnaires during pregnancy, one week postpartum, and three months postpartum. A further 284 (56%) also completed questionnaires ten months postpartum. The model was tested using path analysis.

RESULTS

Antenatal depressive symptoms (β=.15, p<.05), state anxiety (β=.17, p<.01), and perinatal psychoform (β=.17, p<.01) and somatoform (β=.17, p<.01) dissociation were identified as PTSD symptom risk factors three months postpartum. Antenatal depressive symptoms (β=.31, p<.001) and perinatal somatoform dissociation (β=.14, p<.05) predicted symptoms ten months postpartum.

LIMITATIONS

Almost a third of our sample was lost at three months postpartum, and 44% at ten months. The sample size was relatively small. The present study did not control for prior PTSD. The PTSD A criterion was not considered an exclusion criteria for model testing, and the fit index of the ten months model was just below suggested cut-off values.

CONCLUSIONS

Screening for high risk pregnant women should focus on antenatal depression, anxiety and dissociative tendencies. Hospital staff and midwives are advised to be vigilant for perinatal dissociation after intense negative emotions. To help regulate perinatal negative emotional responses, hospital staff and midwifes are recommended to provide information about birth procedures and be attentive to women's birth-related needs.

Sample size calculations for 3-level cluster randomized trials

Background The first applications of cluster randomized trials with three instead of two levels are beginning to appear in health research, for instance, in trials where different strategies to implement best-practice guidelines are compared. In such trials, the strategy is implemented in health care units (`clusters') and aims at changing the behavior of health care professionals working in this unit (`subjects'), while the effects are measured at patient level (`evaluations').

Purpose To guide the choice of number of clusters, number of subjects per cluster, and number of evaluations per subject.

Methods We derive a sample size formula and investigate the influence of sample allocation on power or number of clusters required.

Results The required sample size is the product of the sample size in absence of correlation and two variance inflation factors (VIFs) that describe the clustering of evaluations within subjects and of subjects within cluster, respectively. Because each VIF is expressed in terms of an interpretable Pearson correlation, subject matter knowledge can be incorporated. Moreover, these Pearson's correlations are related to intracluster correlations (ICCs) from comparable, but 2-level cluster randomized trials. Formulas are obtained to guide the sample allocation (number of clusters, subjects, and evaluations) for minimizing total sample size, minimizing the number of clusters, or maximizing power given a budget constraint.

Limitations Empirical estimates of variance components or ICCs from 3-level cluster trials are scarce which limits reliably powering.

Conclusions When parameterized in terms of Pearson correlations, the two variance inflation factors give quantitative insight into the impact of the number of clusters, subjects and evaluations on power. Moreover, subject matter knowledge as well as ICCs from 2-level cluster randomized trials can be incorporated in the sample size calculation, when empirical estimates of variance components or ICCs from a pilot or comparable 3-level study are lacking. Clinical Trials 2008; 5: 486—495. http://ctj.sagepub.com

Power and money in cluster randomized trials: when is it worth measuring a covariate?

The power to detect a treatment effect in cluster randomized trials can be increased by increasing the number of clusters. An alternative is to include covariates into the regression model that relates treatment condition to outcome. In this paper, formulae are derived in order to evaluate both strategies on basis of their costs. It is shown that the strategy that uses covariates is more cost-efficient in detecting a treatment effect when the costs to measure these covariates are small and the correlation between the covariates and outcome is sufficiently large. The minimum required correlation depends on the cluster size, and the costs to recruit a cluster and to measure the covariate, relative to the costs to recruit a person. Measuring a covariate that varies at the person level only is recommended when cluster sizes are small and the costs to recruit and measure a cluster are large. Measuring a cluster level covariate is recommended when cluster sizes are large and the costs to recruit and measure a cluster are small. An illustrative example shows the use of the formulae in a practical setting.

A priori power analysis in longitudinal three-level multilevel models: an example with therapist effects

Over the last few years, three-level longitudinal models have become more common in psychotherapy research, particularly in therapist-effect or group-effect studies. Thus far, limited attention has been paid to power analysis in these models. This article demonstrates the effects of intraclass correlation, level of randomization, sample size, covariates and drop-out on power, using data from a routine outcome monitoring study. Results indicate that randomization at the patient level is the most efficient, and that increasing the number of measurements does not increase power much. Adding a covariate or having a 25% drop-out rate had limited effects on study power in our data. In addition, the results demonstrate that sufficient power can be reached with small sample sizes, but that larger sample sizes are needed to prevent estimation bias for the model parameters and standard errors.

Influence of motivational interviewing on explicit and implicit alcohol-related cognition and alcohol use in at-risk adolescents

Both implicit and explicit cognitions play an important role in the development of addictive behavior. This study investigated the influence of a single-session motivational interview (MI) on implicit and explicit alcohol-related cognition and whether this intervention was successful in consequently decreasing alcohol use in at-risk adolescents. Implicit and explicit alcohol-related cognitions were assessed at pretest and one month posttest in 125 Dutch at-risk adolescents ranging in age from 15 to 23 (51 males) with adapted versions of the Implicit Association Test (IAT) and an expectancy questionnaire. Motivation to change, alcohol use and alcohol-related problems were measured with self-report questionnaires, at pretest, at posttest after one month, and at the six-month follow-up. Although the quality of the intervention was rated positively, the results did not yield support for any differential effects of the intervention on drinking behavior or readiness to change at posttest and six-month follow-up. There were indications of changes in implicit and explicit alcohol-related cognitions between pretest and posttest. Our findings raise questions regarding the use of MI in this particular at-risk adolescent population and the mechanisms through which MI is effective. (PsycINFO Database Record (c) 2009 APA, all rights reserved).

The "best balance" allocation led to optimal balance in cluster-controlled trials

OBJECTIVE

Balance of prognostic factors between treatment groups is desirable because it improves the accuracy, precision, and credibility of the results. In cluster-controlled trials, imbalance can easily occur by chance when the number of cluster is small. If all clusters are known at the start of the study, the "best balance" allocation method (BB) can be used to obtain optimal balance. This method will be compared with other allocation methods.

STUDY DESIGN AND SETTING

We carried out a simulation study to compare the balance obtained with BB, minimization, unrestricted randomization, and matching for four to 20 clusters and one to five categorical prognostic factors at cluster level.

RESULTS

BB resulted in a better balance than randomization in 13-100% of the situations, in 0-61% for minimization, and in 0-88% for matching. The superior performance of BB increased as the number of clusters and/or the number of factors increased.

CONCLUSION

BB results in a better balance of prognostic factors than randomization, minimization, stratification, and matching in most situations. Furthermore, BB cannot result in a worse balance of prognostic factors than the other methods.

Changes in heart rate and skin conductance in the 30 min preceding aggressive behavior

Aggressive behavior of inpatients threatens the safety and well-being of both mental health staff members and fellow patients. It was investigated whether heart rate and electrodermal activity can be used to signal imminent aggression. A naturalistic study was conducted in which 100 inpatients wore sensor wristbands during 5 days to monitor their heart rate and electrodermal activity while staff members recorded patients' aggressive incidents on the ward. Of the 100 patients, 36 displayed at least one aggressive incident. Longitudinal multilevel models indicated that heart rate, skin conductance level, and the number of nonspecific skin conductance responses per minute rose significantly in the 20 min preceding aggressive incidents. Although psychopathy was modestly correlated with displaying aggression, it was not a significant predictor of heart rate and skin conductance preceding aggression. The current findings may provide opportunities for the development of individual prediction models to aid acute risk assessment and to predict aggressive incidents in an earlier stage. The current results on the physiological indicators of aggression are promising for reducing aggression and improving both staff as well as patient safety in psychiatric mental health institutions.

Analyzing indirect effects in cluster randomized trials. The effect of estimation method, number of groups and group sizes on accuracy and power

Cluster randomized trials assess the effect of an intervention that is carried out at the group or cluster level. Ajzen's theory of planned behavior is often used to model the effect of the intervention as an indirect effect mediated in turn by attitude, norms and behavioral intention. Structural equation modeling (SEM) is the technique of choice to estimate indirect effects and their significance. However, this is a large sample technique, and its application in a cluster randomized trial assumes a relatively large number of clusters. In practice, the number of clusters in these studies tends to be relatively small, e.g., much less than fifty. This study uses simulation methods to find the lowest number of clusters needed when multilevel SEM is used to estimate the indirect effect. Maximum likelihood estimation is compared to Bayesian analysis, with the central quality criteria being accuracy of the point estimate and the confidence interval. We also investigate the power of the test for the indirect effect. We conclude that Bayes estimation works well with much smaller cluster level sample sizes such as 20 cases than maximum likelihood estimation; although the bias is larger the coverage is much better. When only 5–10 clusters are available per treatment condition even with Bayesian estimation problems occur.

Randomised controlled trial comparing narrative exposure therapy with present-centred therapy for older patients with post-traumatic stress disorder

BACKGROUND

Evidence-based treatment and age-specific services are required to address the needs of trauma-affected older populations. Narrative exposure therapy (NET) may present an appropriate treatment approach for this population since it provides prolonged exposure in a lifespan perspective. As yet, however, no trial on this intervention has been conducted with older adults from Western Europe.AimsExamining the efficacy of NET in a sample of older adults.

METHOD

Out-patients with post-traumatic stress disorder (PTSD), aged 55 years and over, were randomly assigned to either 11 sessions of NET (n = 18) or 11 sessions of present-centred therapy (PCT) (n = 15) and assessed on the Clinician-Administered PTSD Scale (CAPS) pre-treatment, post-treatment and at follow-up. Total scores as well as symptom scores (re-experience, avoidance and hyperarousal) were evaluated.

RESULTS

Using a piecewise mixed-effects growth model, at post-treatment a medium between-treatment effect size for CAPS total score (Cohen's d = 0.44) was found, favouring PCT. At follow-up, however, the between-treatment differences were non-significant. Drop-out rates were low (NET 6.7%, PCT 14.3%) and no participant dropped out of the study because of increased distress.

CONCLUSIONS

Both NET and PCT appear to be safe and efficacious treatments with older adults: PCT is non-intrusive and NET allows for imaginal exposure in a lifespan perspective. By selectively providing these approaches in clinical practice, patient matching can be optimised.Declaration of interestNone.

How large are the consequences of covariate imbalance in cluster randomized trials: a simulation study with a continuous outcome and a binary covariate at the cluster level

BACKGROUND

The number of clusters in a cluster randomized trial is often low. It is therefore likely random assignment of clusters to treatment conditions results in covariate imbalance. There are no studies that quantify the consequences of covariate imbalance in cluster randomized trials on parameter and standard error bias and on power to detect treatment effects.

METHODS

The consequences of covariance imbalance in unadjusted and adjusted linear mixed models are investigated by means of a simulation study. The factors in this study are the degree of imbalance, the covariate effect size, the cluster size and the intraclass correlation coefficient. The covariate is binary and measured at the cluster level; the outcome is continuous and measured at the individual level.

RESULTS

The results show covariate imbalance results in negligible parameter bias and small standard error bias in adjusted linear mixed models. Ignoring the possibility of covariate imbalance while calculating the sample size at the cluster level may result in a loss in power of at most 25 % in the adjusted linear mixed model. The results are more severe for the unadjusted linear mixed model: parameter biases up to 100 % and standard error biases up to 200 % may be observed. Power levels based on the unadjusted linear mixed model are often too low. The consequences are most severe for large clusters and/or small intraclass correlation coefficients since then the required number of clusters to achieve a desired power level is smallest.

CONCLUSIONS

The possibility of covariate imbalance should be taken into account while calculating the sample size of a cluster randomized trial. Otherwise more sophisticated methods to randomize clusters to treatments should be used, such as stratification or balance algorithms. All relevant covariates should be carefully identified, be actually measured and included in the statistical model to avoid severe levels of parameter and standard error bias and insufficient power levels.

Enhancing the effect of psychotherapy through systematic client feedback in outpatient mental healthcare: A cluster randomized trial

Objective: Systematic client feedback (SCF), the regular monitoring and informing of patients' progress during therapy to patient and therapist, has been found to have effects on treatment outcomes varying from very positive to slightly negative. Several prior studies have been biased by researcher allegiance or lack of an independent outcome measure. The current study has taken this into account and aims to clarify the effects of SCF in outpatient psychological treatment.

Method: Outpatients (n = 1733) of four centers offering brief psychological treatments were cluster randomized to either treatment as usual (TAU) or TAU with SCF based on the Partners for Change Outcome Management System (PCOMS). Primary outcome measure was the Outcome Questionnaire (OQ-45). Effects of the two treatment conditions on treatment outcome, patient satisfaction, dropout rate, costs, and treatment duration were assessed using a three-level multilevel analysis. DSM-classification, sex, and age of each patient were included as covariates.

Results: In both analyses, SCF significantly improved treatment outcome, particularly in the first three months. No significant effects were found on the other outcome variables.

Conclusions: Addition of systematic client feedback to treatment as usual, is likely to have a beneficial impact in outpatient psychological treatment. Implementation requires a careful plan of action.

Clinical or methodological significance of this article: This study, with large sample size and several independent outcome measures, provides strong evidence that addition of systematic client feedback to outpatient psychological treatment can have a beneficial effect on treatment outcome (symptoms and wellbeing), particularly in the first three months. However, implementation requires a careful plan of action.

Contamination: How much can an individually randomized trial tolerate?

Cluster randomization results in an increase in sample size compared to individual randomization, referred to as an efficiency loss. This efficiency loss is typically presented under an assumption of no contamination in the individually randomized trial. An alternative comparator is the sample size needed under individual randomization to detect the attenuated treatment effect due to contamination. A general framework is provided for determining the extent of contamination that can be tolerated in an individually randomized trial before a cluster randomized design yields a larger sample size. Results are presented for a variety of cluster trial designs including parallel arm, stepped-wedge and cluster crossover trials. Results reinforce what is expected: individually randomized trials can tolerate a surprisingly large amount of contamination before they become less efficient than cluster designs. We determine the point at which the contamination means an individual randomized design to detect an attenuated effect requires a larger sample size than cluster randomization under a nonattenuated effect. This critical rate is a simple function of the design effect for clustering and the design effect for multiple periods as well as design effects for stratification or repeated measures under individual randomization. These findings are important for pragmatic comparisons between a novel treatment and usual care as any bias due to contamination will only attenuate the true treatment effect. This is a bias that operates in a predictable direction. Yet, cluster randomized designs with post-randomization recruitment without blinding, are at high risk of bias due to the differential recruitment across treatment arms. This sort of bias operates in an unpredictable direction. Thus, with knowledge that cluster randomized trials are generally at a greater risk of biases that can operate in a nonpredictable direction, results presented here suggest that even in situations where there is a risk of contamination, individual randomization might still be the design of choice.

A randomized clinical trial of a targeted intervention to moderate alcohol use and alcohol-related problems in at-risk adolescents

This study investigated the effectiveness of a targeted intervention program aimed at at-risk adolescents in a randomized clinical trial design (N=107). This program combined intervention methods which have been proven effective in reducing drinking in young adults, such as an expectancy challenge, cognitive behavioral skill training and brief motivational feedback. Additionally, this intervention contained the new element of discussing biological, cognitive and social risk factors for developing alcohol problems. We investigated whether this seven session program was successful in changing cognitive determinants of drinking behavior and consequently in moderating alcohol use and the development of alcohol-related problems in at-risk adolescents. The intervention was effective in changing several of the targeted cognitive determinants. However, despite the changes in these cognitive determinants of drinking, the experimental group did not show a significant difference in decrease of drinking at posttest compared with the control group. The results did not yield support for any differential long term effects of the intervention. We concluded that although the present intervention successfully changed important cognitive determinants of drinking more is needed to change subsequent drinking behavior in at-risk adolescents.

The Effects of the Number of Cohorts, Degree of Overlap Among Cohorts, and Frequency of Observation on Power in Accelerated Longitudinal Designs

With the accelerated longitudinal design data of different age cohorts are used to study individual development over a broad age span during a period of shorter duration. When planning an accelerated longitudinal study one must decide on the number of cohorts, the degree of overlap among cohorts, and the frequency of observation. This paper provides a framework to study the effects of these three design factors on the statistical power to detect a linear change. As no simple mathematical formulae for these relations exist, an example is used to illustrate how the effects of these three design factors can be evaluated. It is shown that the optimal number of cohorts, the optimal degree of overlap among cohorts, and the optimal frequency of observation depend on the total number of subjects and the total number of measurements. R code for evaluating the power of longitudinal designs is provided.