Bias and inference from misspecified mixed-effect models in stepped wedge trial analysis

Effectiveness of a Population Health Intervention on Disparities in Hypertension Control: A Stepped Wedge Cluster Randomized Clinical Trial

BACKGROUND

Disparities in hypertension control across race, ethnicity, and language have been a long-standing problem in the United States.

OBJECTIVE

To assess whether a multi-pronged intervention can improve hypertension control for a target population and reduce disparities.

DESIGN

This stepped wedge cluster randomized trial was conducted at 15 adult primary care clinics affiliated with Massachusetts General Hospital. PCPs were randomized to receive the intervention in twelve groups.

PARTICIPANTS

The target population was patients who met one of the following criteria based on self-identification: (1) Asian, Black, Indigenous, multi-racial, or other race; (2) Hispanic ethnicity; or (3) preferred language other than English. Reference population was White, English-speaking patients.

INTERVENTIONS

PCPs were given access to an online equity dashboard that displays disparities in chronic disease management and completed an equity huddle with population health coordinators (PHCs), which involved reviewing target patients whose hypertension was not well controlled. In addition, community health workers (CHWs) were available in some practices to offer additional support.

MAIN MEASURES

The primary outcome was change in the proportion of target patients meeting the hypertension control goal when comparing intervention and control periods.

KEY RESULTS

Of the 365 PCPs who were randomized, 311 PCPs and their 10,865 target patients were included in the analysis. The intervention led to an increase in hypertension control in the target population (RD 0.9%; 95% CI [0.3,1.5]) and there was a higher intervention effect in the target population compared to the reference population (DiD 2.1%; 95% CI [1.1, 3.1]).

CONCLUSIONS

Utilizing data on disparities in quality outcome measures in routine clinical practice augmented by clinical support provided by PHCs and CHWs led to modest, but statistically significant, improvement in hypertension control among BIPOC, Hispanic, and LEP patients.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT05278806.

Robust analysis of stepped wedge trials using composite likelihood models

Stepped wedge trials (SWTs) are a type of cluster randomized trial that involve repeated measures on clusters and design-induced confounding between time and treatment. Although mixed models are commonly used to analyze SWTs, they are susceptible to misspecification particularly for cluster-longitudinal designs such as SWTs. Mixed model estimation leverages both "horizontal" or within-cluster information and "vertical" or between-cluster information. To use horizontal information in a mixed model, both the mean model and correlation structure must be correctly specified or accounted for, since time is confounded with treatment and measurements are likely correlated within clusters. Alternative non-parametric methods have been proposed that use only vertical information; these are more robust because between-cluster comparisons in a SWT preserve randomization, but these non-parametric methods are not very efficient. We propose a composite likelihood method that focuses on vertical information, but has the flexibility to recover efficiency by using additional horizontal information. We compare the properties and performance of various methods, using simulations based on COVID-19 data and a demonstration of application to the LIRE trial. We found that a vertical composite likelihood model that leverages baseline data is more robust than traditional methods, and more efficient than methods that use only vertical information. We hope that these results demonstrate the potential value of model-based vertical methods for SWTs with a large number of clusters, and that these new tools are useful to researchers who are concerned about misspecification of traditional models.

Maintaining the validity of inference from linear mixed models in stepped-wedge cluster randomized trials under misspecified random-effects structures

Linear mixed models are commonly used in analyzing stepped-wedge cluster randomized trials. A key consideration for analyzing a stepped-wedge cluster randomized trial is accounting for the potentially complex correlation structure, which can be achieved by specifying random-effects. The simplest random effects structure is random intercept but more complex structures such as random cluster-by-period, discrete-time decay, and more recently, the random intervention structure, have been proposed. Specifying appropriate random effects in practice can be challenging: assuming more complex correlation structures may be reasonable but they are vulnerable to computational challenges. To circumvent these challenges, robust variance estimators may be applied to linear mixed models to provide consistent estimators of standard errors of fixed effect parameters in the presence of random-effects misspecification. However, there has been no empirical investigation of robust variance estimators for stepped-wedge cluster randomized trials. In this article, we review six robust variance estimators (both standard and small-sample bias-corrected robust variance estimators) that are available for linear mixed models in R, and then describe a comprehensive simulation study to examine the performance of these robust variance estimators for stepped-wedge cluster randomized trials with a continuous outcome under different data generators. For each data generator, we investigate whether the use of a robust variance estimator with either the random intercept model or the random cluster-by-period model is sufficient to provide valid statistical inference for fixed effect parameters, when these working models are subject to random-effect misspecification. Our results indicate that the random intercept and random cluster-by-period models with robust variance estimators performed adequately. The CR3 robust variance estimator (approximate jackknife) estimator, coupled with the number of clusters minus two degrees of freedom correction, consistently gave the best coverage results, but could be slightly conservative when the number of clusters was below 16. We summarize the implications of our results for the linear mixed model analysis of stepped-wedge cluster randomized trials and offer some practical recommendations on the choice of the analytic model.

Digital Medication Management in Polypharmacy

BACKGROUND

Inappropriate drug prescriptions for patients with polypharmacy can have avoidable adverse consequences. We studied the effects of a clinical decision-support system (CDSS) for medication management on hospitalizations and mortality.

METHODS

This stepped-wedge, cluster-randomized, controlled trial involved an open cohort of adult patients with polypharmacy in primary care practices (=clusters) in Westphalia-Lippe, Germany. During the period of the intervention, their medication lists were checked annually using the CDSS. The CDSS warns against inappropriate prescriptions on the basis of patient-related health insurance data. The combined primary endpoint consisted of overall mortality and hospitalization for any reason. The secondary endpoints were mortality, hospitalizations, and high-risk prescription. We analyzed the quarterly health insurance data of the intention- to-treat population with a mixed logistic model taking account of clustering and repeated measurements. Sensitivity analyses addressed effects of the COVID-19 pandemic and other effects.

RESULTS

688 primary care practices were randomized, and data were obtained on 42 700 patients over 391 994 quarter years. No significant reduction was found in either the primary endpoint (odds ratio [OR] 1.00; 95% confidence interval [0.95; 1.04]; p = 0.8716) or the secondary endpoints (hospitalizations: OR 1.00 [0.95; 1.05]; mortality: OR 1.04 [0.92; 1.17]; high-risk prescription: OR 0.98 [0.92; 1.04]).

CONCLUSION

The planned analyses did not reveal any significant effect of the intervention. Pandemicadjusted analyses yielded evidence that the mortality of adult patients with polypharmacy might potentially be lowered by the CDSS. Controlled trials with appropriate follow-up are needed to prove that a CDSS has significant effects on mortality in patients with polypharmacy.

glmmPen: High Dimensional Penalized Generalized Linear Mixed Models

Generalized linear mixed models (GLMMs) are widely used in research for their ability to model correlated outcomes with non-Gaussian conditional distributions. The proper selection of fixed and random effects is a critical part of the modeling process, where model misspecification may lead to significant bias. However, the joint selection of fixed and random effects has historically been limited to lower dimensional GLMMs, largely due to the use of criterion-based model selection strategies. Here we present the R package glmmPen, one of the first to select fixed and random effects in higher dimension using a penalized GLMM modeling framework. Model parameters are estimated using a Monte Carlo expectation conditional minimization (MCECM) algorithm, which leverages Stan and RcppArmadillo for increased computational efficiency. Our package supports the Binomial, Gaussian, and Poisson families and multiple penalty functions. In this manuscript we discuss the modeling procedure, estimation scheme, and software implementation through application to a pancreatic cancer subtyping study. Simulation results show our method has good performance in selecting both the fixed and random effects in high dimensional GLMMs.

Restrictive versus Liberal Rate of Extracorporeal Volume Removal Evaluation in Acute Kidney Injury (RELIEVE-AKI): a pilot clinical trial protocol

INTRODUCTION

Observational studies have linked slower and faster net ultrafiltration (UFNET) rates during kidney replacement therapy (KRT) with mortality in critically ill patients with acute kidney injury (AKI) and fluid overload. To inform the design of a larger randomised trial of patient-centered outcomes, we conduct a feasibility study to examine restrictive and liberal approaches to UFNET during continuous KRT (CKRT).

METHODS AND ANALYSIS

This study is an investigator-initiated, unblinded, 2-arm, comparative-effectiveness, stepped-wedged, cluster randomised trial among 112 critically ill patients with AKI treated with CKRT in 10 intensive care units (ICUs) across 2 hospital systems. In the first 6 months, all ICUs started with a liberal UFNET rate strategy. Thereafter, one ICU is randomised to the restrictive UFNET rate strategy every 2 months. In the liberal group, the UFNET rate is maintained between 2.0 and 5.0 mL/kg/hour; in the restrictive group, the UFNET rate is maintained between 0.5 and 1.5 mL/kg/hour. The three coprimary feasibility outcomes are (1) between-group separation in mean delivered UFNET rates; (2) protocol adherence; and (3) patient recruitment rate. Secondary outcomes include daily and cumulative fluid balance, KRT and mechanical ventilation duration, organ failure-free days, ICU and hospital length of stay, hospital mortality and KRT dependence at hospital discharge. Safety endpoints include haemodynamics, electrolyte imbalance, CKRT circuit issues, organ dysfunction related to fluid overload, secondary infections and thrombotic and haematological complications.

ETHICS AND DISSEMINATION

The University of Pittsburgh Human Research Protection Office approved the study, and an independent Data and Safety Monitoring Board monitors the study. A grant from the United States National Institute of Diabetes and Digestive and Kidney Diseases sponsors the study. The trial results will be submitted for publication in peer-reviewed journals and presented at scientific conferences.

TRIAL REGISTRATION NUMBER

This trial has been prospectively registered with clinicaltrials.gov (NCT05306964). Protocol version identifier and date: 1.5; 13 June 2023.

Accounting for complex intracluster correlations in longitudinal cluster randomized trials: a case study in malaria vector control

BACKGROUND

The effectiveness of malaria vector control interventions is often evaluated using cluster randomized trials (CRT) with outcomes assessed using repeated cross-sectional surveys. A key requirement for appropriate design and analysis of longitudinal CRTs is accounting for the intra-cluster correlation coefficient (ICC). In addition to exchangeable correlation (constant ICC over time), correlation structures proposed for longitudinal CRT are block exchangeable (allows a different within- and between-period ICC) and exponential decay (allows between-period ICC to decay exponentially). More flexible correlation structures are available in statistical software packages and, although not formally proposed for longitudinal CRTs, may offer some advantages. Our objectives were to empirically explore the impact of these correlation structures on treatment effect inferences, identify gaps in the methodological literature, and make practical recommendations.

METHODS

We obtained data from a parallel-arm CRT conducted in Tanzania to compare four different types of insecticide-treated bed-nets. Malaria prevalence was assessed in cross-sectional surveys of 45 households in each of 84 villages at baseline, 12-, 18- and 24-months post-randomization. We re-analyzed the data using mixed-effects logistic regression according to a prespecified analysis plan but under five different correlation structures as well as a robust variance estimator under exchangeable correlation and compared the estimated correlations and treatment effects. A proof-of-concept simulation was conducted to explore general conclusions.

RESULTS

The estimated correlation structures varied substantially across different models. The unstructured model was the best-fitting model based on information criteria. Although point estimates and confidence intervals for the treatment effect were similar, allowing for more flexible correlation structures led to different conclusions based on statistical significance. Use of robust variance estimators generally led to wider confidence intervals. Simulation results showed that under-specification can lead to coverage probabilities much lower than nominal levels, but over-specification is more likely to maintain nominal coverage.

CONCLUSION

More flexible correlation structures should not be ruled out in longitudinal CRTs. This may be particularly important in malaria trials where outcomes may fluctuate over time. In the absence of robust methods for selecting the best-fitting correlation structure, researchers should examine sensitivity of results to different assumptions about the ICC and consider robust variance estimators.

Tighter or less tight glycaemic targets for women with gestational diabetes mellitus for reducing maternal and perinatal morbidity: A stepped-wedge, cluster-randomised trial

BACKGROUND

Treatment for gestational diabetes mellitus (GDM) aims to reduce maternal hyperglycaemia. The TARGET Trial assessed whether tighter compared with less tight glycaemic control reduced maternal and perinatal morbidity.

METHODS AND FINDINGS

In this stepped-wedge, cluster-randomised trial, identification number ACTRN12615000282583, 10 hospitals in New Zealand were randomised to 1 of 5 implementation dates. The trial was registered before the first participant was enrolled. All hospitals initially used less tight targets (fasting plasma glucose (FPG) <5.5 mmol/L (<99 mg/dL), 1-hour <8.0 mmol/L (<144 mg/dL), 2 hour postprandial <7.0 mmol/L (<126 mg/dL)) and every 4 months, 2 hospitals moved to use tighter targets (FPG ≤5.0 mmol/L (≤90 mg/dL), 1-hour ≤7.4 mmol/L (≤133 mg/dL), 2 hour postprandial ≤6.7 mmol/L) (≤121 mg/dL). Women with GDM, blinded to the targets in use, were eligible. The primary outcome was large for gestational age. Secondary outcomes assessed maternal and infant health. Analyses were by intention to treat. Between May 2015 and November 2017, data were collected from 1,100 women with GDM (1,108 infants); 598 women (602 infants) used the tighter targets and 502 women (506 infants) used the less tight targets. The rate of large for gestational age was similar between the treatment target groups (88/599, 14.7% versus 76/502, 15.1%; adjusted relative risk [adjRR] 0.96, 95% confidence interval [CI] 0.66 to 1.40, P = 0.839). The composite serious health outcome for the infant of perinatal death, birth trauma, or shoulder dystocia was apparently reduced in the tighter group when adjusted for gestational age at diagnosis of GDM, BMI, ethnicity, and history of GDM compared with the less tight group (8/599, 1.3% versus 13/505, 2.6%, adjRR 0.23, 95% CI 0.06 to 0.88, P = 0.032). No differences were seen for the other infant secondary outcomes apart from a shorter stay in intensive care (P = 0.041). Secondary outcomes for the woman showed an apparent increase for the composite serious health outcome that included major haemorrhage, coagulopathy, embolism, and obstetric complications in the tighter group (35/595, 5.9% versus 15/501, 3.0%, adjRR 2.29, 95% CI 1.14 to 4.59, P = 0.020). There were no differences between the target groups in the risk for pre-eclampsia, induction of labour, or cesarean birth, but more women using tighter targets required pharmacological treatment (404/595, 67.9% versus 293/501, 58.5%, adjRR 1.20, 95% CI 1.00 to 1.44, P = 0.047). The main study limitation is that the treatment targets used may vary to those in use in some countries.

CONCLUSIONS

Tighter glycaemic targets in women with GDM compared to less tight targets did not reduce the risk of a large for gestational age infant, but did reduce serious infant morbidity, although serious maternal morbidity was increased. These findings can be used to aid decisions on the glycaemic targets women with GDM should use.

TRIAL REGISTRATION

The Australian New Zealand Clinical Trials Registry (ANZCTR). ACTRN12615000282583.

Linking Primary Care Patients to Mental Health Care via Behavioral Health Social Workers: A Stepped-Wedge Study

OBJECTIVE

Demand for systematic linkage of patients to behavioral health care has increased because of the widespread implementation of depression screening. This study assessed the impact of deploying behavioral health social workers (BHSWs) in primary care on behavioral health visits for depression or anxiety.

METHODS

This quasi-experimental, stepped-wedge study included adults with a primary care visit between 2016 and 2019 at Cleveland Clinic, a large integrated health system. BHSWs were deployed in 40 practices between 2017 and 2019. Patients were allocated to a control group (diagnosed before BHSW deployment) and an intervention group (diagnosed after deployment). Data were collected on behavioral health visits (i.e., to therapists and psychiatrists) within 30 days of the diagnosis. Multilevel logistic regression models identified associations between BHSW deployment period and behavioral health visit, adjusted for demographic variables and clustering within each group.

RESULTS

Of 68,659 persons with a diagnosis, 21% had a depression diagnosis, 49% an anxiety diagnosis, and 31% both diagnoses. In the period after BHSW deployment, the proportion of patients with depression who had a behavioral health visit increased by 10 percentage points, of patients with anxiety by 9 percentage points, and of patients with both disorders by 11 percentage points. The adjusted odds of having a behavioral health visit was higher in the postdeployment period for patients with depression (adjusted odds ratio [AOR]=4.35, 95% confidence interval [CI]=3.50-5.41), anxiety (AOR=4.27, 95% CI=3.57-5.11), and both (AOR= 3.26, 95% CI=2.77-3.84).

CONCLUSIONS

Integration of BHSWs in primary care was associated with increased behavioral health visits.

Divergent confidence intervals among pre-specified analyses in the HiSTORIC stepped wedge trial: An exploratory post-hoc investigation

BACKGROUND

The high-sensitivity cardiac troponin on presentation to rule out myocardial infarction (HiSTORIC) study was a stepped-wedge cluster randomised trial with long before-and-after periods, involving seven hospitals across Scotland. Results were divergent for the binary safety endpoint (type 1 or type 4b myocardial infarction or cardiac death) across certain pre-specified analyses, which warranted further investigation. In particular, the calendar-matched analysis produced an odds ratio in the opposite direction to the primary logistic mixed-effects model analysis.

METHODS

Several post-hoc statistical models were fitted to each of the co-primary outcomes of length of hospital stay and safety events, which included adjusting for exposure time, incorporating splines, and fitting a random time effect. We improved control of patient characteristics over time by adjusting for multiple additional covariates using different methods: direct inclusion, regression adjustment for propensity score, and weighting. A data augmentation approach was also conducted aiming to reduce the effect of sparse data bias. Finally, the raw data was examined.

RESULTS

The new statistical models confirmed the results of the pre-specified trial analysis. In particular, the observed divergence between the calendar-matched and other analyses remained, even after performing the covariate adjustment methods, and after using data augmentation. Divergence was particularly acute for the safety endpoint, which had an event rate of 0.36% overall. Examining the raw data was particularly helpful to assess the sensitivity of the results to small changes in event rates and identify patterns in the data.

CONCLUSIONS

Our experience reveals the importance of conducting multiple pre-specified sensitivity analyses and examining the raw data, particularly for stepped wedge trials with low event rates or with a small number of sites. Before-and-after analytical approaches that adjust for differences in patient populations but avoid direct modelling of the time trend should be considered in future stepped wedge trials with similar designs.

Considerations Before Selecting a Stepped-Wedge Cluster Randomized Trial Design for a Practice Improvement Study

PURPOSE

Despite the growing popularity of stepped-wedge cluster randomized trials (SW-CRTs) for practice-based research, the design's advantages and challenges are not well documented. The objective of this study was to identify the advantages and challenges of the SW-CRT design for large-scale intervention implementations in primary care settings.

METHODS

The EvidenceNOW: Advancing Heart Health initiative, funded by the Agency for Healthcare Research and Quality, included a large collection of SW-CRTs. We conducted qualitative interviews with 17 key informants from EvidenceNOW grantees to identify the advantages and challenges of using SW-CRT design.

RESULTS

All interviewees reported that SW-CRT can be an effective study design for large-scale intervention implementations. Advantages included (1) incentivized recruitment, (2) staggered resource allocation, and (3) statistical power. Challenges included (1) time-sensitive recruitment, (2) retention, (3) randomization requirements and practice preferences, (4) achieving treatment schedule fidelity, (5) intensive data collection, (6) the Hawthorne effect, and (7) temporal trends.

CONCLUSIONS

The challenges experienced by EvidenceNOW grantees suggest that certain favorable real-world conditions constitute a context that increases the odds of a successful SW-CRT. An existing infrastructure can support the recruitment of many practices. Strong retention plans are needed to continue to engage sites waiting to start the intervention. Finally, study outcomes should be ones already captured in routine practice; otherwise, funders and investigators should assess the feasibility and cost of data collection.VISUAL ABSTRACT.

Stepped Wedge Cluster Randomized Trials: A Methodological Overview

BACKGROUND

Stepped wedge cluster randomized trials enable rigorous evaluations of health intervention programs in pragmatic settings. In the present study, we aimed to update neurosurgeon scientists on the design of stepped wedge randomized trials.

METHODS

We have presented an overview of recent methodological developments for stepped wedge designs and included an update on the newer associated methodological tools to aid with future study designs.

RESULTS

We defined the stepped wedge trial design and reviewed the indications for the design in depth. In addition, key considerations, including mainstream methods of analysis and sample size determination, were discussed.

CONCLUSIONS

Stepped wedge designs can be attractive for study intervention programs aiming to improve the delivery of patient care, especially when examining a small number of heterogeneous clusters.

Generalized quasi-linear mixed-effects model

The generalized linear mixed model (GLMM) is one of the most common method in the analysis of longitudinal and clustered data in biological sciences. However, issues of model complexity and misspecification can occur when applying the GLMM. To address these issues, we extend the standard GLMM to a nonlinear mixed-effects model based on quasi-linear modeling. An estimation algorithm for the proposed model is provided by extending the penalized quasi-likelihood and the restricted maximum likelihood which are known in the GLMM inference. Also, the conditional AIC is formulated for the proposed model. The proposed model should provide a more flexible fit than the GLMM when there is a nonlinear relation between fixed and random effects. Otherwise, the proposed model is reduced to the GLMM. The performance of the proposed model under model misspecification is evaluated in several simulation studies. In the analysis of respiratory illness data from a randomized controlled trial, we observe the proposed model can capture heterogeneity; that is, it can detect a patient subgroup with specific clinical character in which the treatment is effective.

Random effect misspecification in stepped wedge designs

Stepped wedge cluster randomized trials are often analysed using linear mixed effects models that may include random effects for cluster, time and/or treatment. We investigate the impact of misspecification of the random effects structure of the model. Specifically, we considered two cases of misspecification of the random effects in a cross-sectional stepped wedge cluster randomized trials model - fit a linear mixed effects model with random time effects but the true model includes random treatment effects (case 1) or fit a linear mixed effects model with random treatment effect but the true model includes random time effects (case 2) - and derived the variance of the estimated treatment effect under misspecification. We defined two measures of the effect of misspecification: validity and efficiency. Validity is the ratio of the model-based variance of the treatment effect from the mis-specified model divided by the true variance of the treatment effect from the mis-specified model (based on a sandwich estimate of the variance). Efficiency is the ratio of the model-based variance of the treatment effect from the correctly specified model divided by the true variance of the treatment effect from the mis-specified model. We found that validity is less than 1.0 (anti-conservative) in almost all situations investigated with the exception of case 1 with two sequences, when validity could be greater than 1.0. Efficiency is less than 1 in all cases and depends on the intracluster correlation coefficient, the relative magnitude of the variance of the misclassified variance component, and the number of sequences. In general, there is no universal recommendation as to the most robust approach except for the case of a classic stepped wedge cluster randomized trial with only 2 sequences, where fitting a random time model is less likely to lead to anti-conservative inference compared with fitting a random intervention model.

Model misspecification in stepped wedge trials: Random effects for time or treatment

Mixed models are commonly used to analyze stepped wedge trials (SWTs) to account for clustering and repeated measures on clusters. One critical issue researchers face is whether to include a random time effect or a random treatment effect. When the wrong model is chosen, inference on the treatment effect may be invalid. We explore asymptotic and finite-sample convergence of variance component estimates when the model is misspecified and how misspecification affects the estimated variance of the treatment effect. For asymptotic results, we rely on analytical solutions rather than simulation studies, which allow us to succinctly describe the convergence of misspecified estimates, even though there are multiple roots for each misspecified model. We found that both direction and magnitude of the bias associated with model-based standard errors depends on the study design and magnitude of the true variance components. We identify some scenarios in which choosing the wrong random effect has a large impact on model-based inference. However, many trends depend on trial design and assumptions about the true correlation structure, so we provide tools for researchers to investigate specific scenarios of interest. We use data from an SWT on disinvesting from weekend services in hospital wards to demonstrate how these results can be applied as a sensitivity analysis, which quantifies the impact of misspecification under a variety of settings and directly compares the potential consequences of different modeling choices. Our results will provide guidance for prespecified model choices and supplement sensitivity analyses to inform confidence in the validity of results.

Influential methods reports for group-randomized trials and related designs

BACKGROUND

This article identifies the most influential methods reports for group-randomized trials and related designs published through 2020. Many interventions are delivered to participants in real or virtual groups or in groups defined by a shared interventionist so that there is an expectation for positive correlation among observations taken on participants in the same group. These interventions are typically evaluated using a group- or cluster-randomized trial, an individually randomized group treatment trial, or a stepped wedge group- or cluster-randomized trial. These trials face methodological issues beyond those encountered in the more familiar individually randomized controlled trial.

METHODS

PubMed was searched to identify candidate methods reports; that search was supplemented by reports known to the author. Candidate reports were reviewed by the author to include only those focused on the designs of interest. Citation counts and the relative citation ratio, a new bibliometric tool developed at the National Institutes of Health, were used to identify influential reports. The relative citation ratio measures influence at the article level by comparing the citation rate of the reference article to the citation rates of the articles cited by other articles that also cite the reference article.

RESULTS

In total, 1043 reports were identified that were published through 2020. However, 55 were deemed to be the most influential based on their relative citation ratio or their citation count using criteria specific to each of the three designs, with 32 group-randomized trial reports, 7 individually randomized group treatment trial reports, and 16 stepped wedge group-randomized trial reports. Many of the influential reports were early publications that drew attention to the issues that distinguish these designs from the more familiar individually randomized controlled trial. Others were textbooks that covered a wide range of issues for these designs. Others were "first reports" on analytic methods appropriate for a specific type of data (e.g. binary data, ordinal data), for features commonly encountered in these studies (e.g. unequal cluster size, attrition), or for important variations in study design (e.g. repeated measures, cohort versus cross-section). Many presented methods for sample size calculations. Others described how these designs could be applied to a new area (e.g. dissemination and implementation research). Among the reports with the highest relative citation ratios were the CONSORT statements for each design.

CONCLUSIONS

Collectively, the influential reports address topics of great interest to investigators who might consider using one of these designs and need guidance on selecting the most appropriate design for their research question and on the best methods for design, analysis, and sample size.

Inference for the treatment effect in longitudinal cluster randomized trials when treatment effect heterogeneity is ignored

In cluster-randomized trials, sometimes the effect of the intervention being studied differs between clusters, commonly referred to as treatment effect heterogeneity. In the analysis of stepped wedge and cluster-randomized crossover trials, it is possible to include terms in outcome regression models to allow for such treatment effect heterogeneity yet this is not frequently considered. Outside of some simulation studies of specific cases where the outcome is binary, the impact of failing to include terms for treatment effect heterogeneity on the variance of the treatment effect estimator is unknown. We analytically examine the impact of failing to include terms for treatment effect heterogeneity on the variance of the treatment effect estimator, when outcomes are continuous. Using analysis of variance and feasible generalized least squares we provide expressions for this variance. For both the cluster-randomized crossover design and the stepped wedge design, our analytic derivations indicate that failing to include treatment effect heterogeneity results in the estimates for variance of the treatment effect that are too small, leading to inflation of type I error rates. We therefore recommend assessing the sensitivity of sample size calculations and conclusions drawn from the analysis of cluster randomized trials to the inclusion of treatment effect heterogeneity.

Evaluation of a covariate-constrained randomization procedure in stepped wedge cluster randomized trials

In stepped wedge (SW) designs, differing cluster-level characteristics or individual-level covariate distributions that differ by cluster can lead to imbalance by treatment arm and potential confounding of the treatment effect. Adapting a method used in cluster-randomized trials, we propose a covariate-constrained randomization method to be used in SW designs. First, we define a balance metric to be calculated for all possible randomizations of cluster order for a given SW design. The resulting distribution of this balance metric across all possible randomizations is used to select a candidate set of randomizations with acceptable covariate balance. One cluster order is selected at random from this candidate set to be used as the cluster order for treatment implementation. In a simulation study, we implement the covariate-constrained randomization procedure and compare treatment effect estimation, type I error, and power under varying SW design and confounding settings, and using multiple analysis methods. We observed optimal statistical properties when the balance metric was used to exclude a small set of potential randomizations with the highest level of imbalance, and when analysis methods were adjusted for the potential confounders. The covariate-constrained randomization was most beneficial in settings with a small number of clusters and in the presence of cluster-level confounding.

Reflection on modern methods: when is a stepped-wedge cluster randomized trial a good study design choice?

The stepped-wedge cluster randomized trial (SW-CRT) involves the sequential transition of clusters (such as hospitals, public health units or communities) from control to intervention conditions in a randomized order. The use of the SW-CRT is growing rapidly. Yet the SW-CRT is at greater risks of bias compared with the conventional parallel cluster randomized trial (parallel-CRT). For this reason, the CONSORT extension for SW-CRTs requires that investigators provide a clear justification for the choice of study design. In this paper, we argue that all other things being equal, the SW-CRT is at greater risk of bias due to misspecification of the secular trends at the analysis stage. This is particularly problematic for studies randomizing a small number of heterogeneous clusters. We outline the potential conditions under which an SW-CRT might be an appropriate choice. Potentially appropriate and often overlapping justifications for conducting an SW-CRT include: (i) the SW-CRT provides a means to conduct a randomized evaluation which otherwise would not be possible; (ii) the SW-CRT facilitates cluster recruitment as it enhances the acceptability of a randomized evaluation either to cluster gatekeepers or other stakeholders; (iii) the SW-CRT is the only feasible design due to pragmatic and logistical constraints (for example the roll-out of a scare resource); and (iv) the SW-CRT has increased statistical power over other study designs (which will include situations with a limited number of clusters). As the number of arguments in favour of an SW-CRT increases, the likelihood that the benefits of using the SW-CRT, as opposed to a parallel-CRT, outweigh its risks also increases. We argue that the mere popularity and novelty of the SW-CRT should not be a factor in its adoption. In situations when a conventional parallel-CRT is feasible, it is likely to be the preferred design.

Accounting for confounding by time, early intervention adoption, and time-varying effect modification in the design and analysis of stepped-wedge designs: application to a proposed study design to reduce opioid-related mortality

BACKGROUND

Beginning in 2019, stepped-wedge designs (SWDs) were being used in the investigation of interventions to reduce opioid-related deaths in communities across the United States. However, these interventions are competing with external factors such as newly initiated public policies limiting opioid prescriptions, media awareness campaigns, and the COVID-19 pandemic. Furthermore, control communities may prematurely adopt components of the intervention as they become available. The presence of time-varying external factors that impact study outcomes is a well-known limitation of SWDs; common approaches to adjusting for them make use of a mixed effects modeling framework. However, these models have several shortcomings when external factors differentially impact intervention and control clusters.

METHODS

We discuss limitations of commonly used mixed effects models in the context of proposed SWDs to investigate interventions intended to reduce opioid-related mortality, and propose extensions of these models to address these limitations. We conduct an extensive simulation study of anticipated data from SWD trials targeting the current opioid epidemic in order to examine the performance of these models in the presence of external factors. We consider confounding by time, premature adoption of intervention components, and time-varying effect modification- in which external factors differentially impact intervention and control clusters.

RESULTS

In the presence of confounding by time, commonly used mixed effects models yield unbiased intervention effect estimates, but can have inflated Type 1 error and result in under coverage of confidence intervals. These models yield biased intervention effect estimates when premature intervention adoption or effect modification are present. In such scenarios, models incorporating fixed intervention-by-time interactions with an unstructured covariance for intervention-by-cluster-by-time random effects result in unbiased intervention effect estimates, reach nominal confidence interval coverage, and preserve Type 1 error.

CONCLUSIONS

Mixed effects models can adjust for different combinations of external factors through correct specification of fixed and random time effects. Since model choice has considerable impact on validity of results and study power, careful consideration must be given to how these external factors impact study endpoints and what estimands are most appropriate in the presence of such factors.

Mixed-effects models for the design and analysis of stepped wedge cluster randomized trials: An overview

The stepped wedge cluster randomized design has received increasing attention in pragmatic clinical trials and implementation science research. The key feature of the design is the unidirectional crossover of clusters from the control to intervention conditions on a staggered schedule, which induces confounding of the intervention effect by time. The stepped wedge design first appeared in the Gambia hepatitis study in the 1980s. However, the statistical model used for the design and analysis was not formally introduced until 2007 in an article by Hussey and Hughes. Since then, a variety of mixed-effects model extensions have been proposed for the design and analysis of these trials. In this article, we explore these extensions under a unified perspective. We provide a general model representation and regard various model extensions as alternative ways to characterize the secular trend, intervention effect, as well as sources of heterogeneity. We review the key model ingredients and clarify their implications for the design and analysis. The article serves as an entry point to the evolving statistical literatures on stepped wedge designs.

Accounting for Confounding by Time, Early Intervention Adoption, and Time-Varying Effect Modification in the Design and Analysis of Stepped-Wedge Designs: Application to a Proposed Study Design to Reduce Opioid-Related Mortality

Background: Stepped-wedge designs (SWDs) are currently being used in the investigation of interventions to reduce opioid-related deaths in communities located in several states. However, these interventions are competing with external factors such as newly initiated public policies limiting opioid prescriptions, media awareness campaigns, and COVID-19 social distancing mandates. Furthermore, control communities may prematurely adopt components of the intervention as they become available. The presence of time-varying external factors that impact study outcomes is a well-known limitation of SWDs; common approaches to adjusting for them make use of a mixed effects modeling framework. However, these models have several shortcomings when external factors differentially impact intervention and control clusters. Methods: We discuss limitations of commonly used mixed effects models in the context of proposed SWDs to investigate interventions intended to reduce opioid-related mortality, and propose extensions of these models to address these limitations. We conduct an extensive simulation study of anticipated data from SWD trials targeting the current opioid epidemic in order to examine the performance of these models in the presence of external factors. We consider confounding by time, premature adoption of components of the intervention, and time-varying effect modificationâ€" in which external factors differentially impact intervention and control clusters. Results: In the presence of confounding by time, commonly used mixed effects models yield unbiased intervention effect estimates, but can have inflated Type 1 error and result in under coverage of confidence intervals. These models yield biased intervention effect estimates when premature intervention adoption or effect modification are present. In such scenarios, models incorporating fixed intervention-by-time interactions with an unstructured covariance for intervention-by-cluster-by-time random effects result in unbiased intervention effect estimates, reach nominal confidence interval coverage, and preserve Type 1 error. Conclusions: Mixed effects models can adjust for different combinations of external factors through correct specification of fixed and random time effects; misspecification can result in bias of the intervention effect estimate, under coverage of confidence intervals, and Type 1 error inflation. Since model choice has considerable impact on validity of results and study power, careful consideration must be given to choosing appropriate models that account for potential external factors.

Statistical Properties of Stepped Wedge Cluster-Randomized Trials in Infectious Disease Outbreaks

Randomized controlled trials are crucial for the evaluation of interventions such as vaccinations, but the design and analysis of these studies during infectious disease outbreaks is complicated by statistical, ethical, and logistical factors. Attempts to resolve these complexities have led to the proposal of a variety of trial designs, including individual randomization and several types of cluster randomization designs: parallel-arm, ring vaccination, and stepped wedge designs. Because of the strong time trends present in infectious disease incidence, however, methods generally used to analyze stepped wedge trials might not perform well in these settings. Using simulated outbreaks, we evaluated various designs and analysis methods, including recently proposed methods for analyzing stepped wedge trials, to determine the statistical properties of these methods. While new methods for analyzing stepped wedge trials can provide some improvement over previous methods, we find that they still lag behind parallel-arm cluster-randomized trials and individually randomized trials in achieving adequate power to detect intervention effects. We also find that these methods are highly sensitive to the weighting of effect estimates across time periods. Despite the value of new methods, stepped wedge trials still have statistical disadvantages compared with other trial designs in epidemic settings.

Extending the I-squared statistic to describe treatment effect heterogeneity in cluster, multi-centre randomized trials and individual patient data meta-analysis

Treatment effect heterogeneity is commonly investigated in meta-analyses to identify if treatment effects vary across studies. When conducting an aggregate level data meta-analysis it is common to describe the magnitude of any treatment effect heterogeneity using the I-squared statistic, which is an intuitive and easily understood concept. The effect of a treatment might also vary across clusters in a cluster randomized trial, or across centres in multi-centre randomized trial, and it can be of interest to explore this at the analysis stage. In cross-over trials and other randomized designs, in which clusters or centres are exposed to both treatment and control conditions, this treatment effect heterogeneity can be identified. Here we derive and evaluate a comparable I-squared measure to describe the magnitude of heterogeneity in treatment effects across clusters or centres in randomized trials. We further show how this methodology can be used to estimate treatment effect heterogeneity in an individual patient data meta-analysis.

A stepped-wedge randomised trial on the impact of early ART initiation on HIV-patients' economic outcomes in Eswatini

Background

Since 2015, the World Health Organisation (WHO) recommends immediate initiation of antiretroviral therapy (ART) for all HIV-positive patients. Epidemiological evidence points to important health benefits of immediate ART initiation; however, the policy’s impact on the economic aspects of patients' lives remains unknown.

Methods

We conducted a stepped-wedge cluster-randomised controlled trial in Eswatini to determine the causal impact of immediate ART initiation on patients’ individual- and household-level economic outcomes. Fourteen healthcare facilities were non-randomly matched into pairs and then randomly allocated to transition from the standard of care (ART eligibility at CD4 counts of <350 cells/mm3 until September 2016 and <500 cells/mm3 thereafter) to the ‘Early Initiation of ART for All’ (EAAA) intervention at one of seven timepoints. Patients, healthcare personnel, and outcome assessors remained unblinded. Data were collected via standardised paper-based surveys with HIV-positive adults who were neither pregnant nor breastfeeding. Outcomes were patients’ time use, employment status, household expenditures, and household living standards.

Results

A total sample of 3019 participants were interviewed over the duration of the study. The mean number of participants approached at each facility per time step varied from 4 to 112 participants. Using mixed-effects negative binomial regressions accounting for time trends and clustering at the level of the healthcare facility, we found no significant difference between study arms for any economic outcome. Specifically, the EAAA intervention had no significant effect on non-resting time use (RR = 1.00 [CI: 0.96, 1.05, p=0.93]) or income-generating time use (RR = 0.94, [CI: 0.73,1.20, p=0.61]). Employment and household expenditures decreased slightly but not significantly in the EAAA group, with risk ratios of 0.93 [CI: 0.82, 1.04, p=0.21] and 0.92 [CI: 0.79, 1.06, p=0.26], respectively. We also found no significant treatment effect on households’ asset ownership and living standards (RR = 0.96, [CI 0.92, 1.00, p=0.253]). Lastly, there was no evidence of heterogeneity in effect estimates by patients’ sex, age, education, timing of HIV diagnosis and ART initiation.

Conclusions

Our findings do not provide evidence that should discourage further investments into scaling up immediate ART for all HIV patients.

Funding

Funded by the Dutch Postcode Lottery in the Netherlands, Alexander von Humboldt-Stiftung (Humboldt-Stiftung), the Embassy of the Kingdom of the Netherlands in South Africa/Mozambique, British Columbia Centre of Excellence in Canada, Doctors Without Borders (MSF USA), National Center for Advancing Translational Sciences of the National Institutes of Health and Joachim Herz Foundation.

Clinical trial number

NCT02909218 and NCT03789448.

Accounting for external factors and early intervention adoption in the design and analysis of stepped-wedge designs: Application to a proposed study design to reduce opioid-related mortality

BACKGROUND

Stepped-wedge designs (SWDs) are currently being used to investigate interventions to reduce opioid overdose deaths in communities located in several states. However, these interventions are competing with external factors such as newly initiated public policies limiting opioid prescriptions, media awareness campaigns, and social distancing orders due to the COVID-19 pandemic. Furthermore, control communities may prematurely adopt components of the proposed intervention as they become widely available. These types of events induce confounding of the intervention effect by time. Such confounding is a well-known limitation of SWDs; a common approach to adjusting for it makes use of a mixed effects modeling framework that includes both fixed and random effects for time. However, these models have several shortcomings when multiple confounding factors are present.

METHODS

We discuss the limitations of existing methods based on mixed effects models in the context of proposed SWDs to investigate interventions intended to reduce mortality associated with the opioid epidemic, and propose solutions to accommodate deviations from assumptions that underlie these models. We conduct an extensive simulation study of anticipated data from SWD trials targeting the current opioid epidemic in order to examine the performance of these models under different sources of confounding. We specifically examine the impact of factors external to the study and premature adoption of intervention components.

RESULTS

When only external factors are present, our simulation studies show that commonly used mixed effects models can result in unbiased estimates of the intervention effect, but have inflated Type 1 error and result in under coverage of confidence intervals. These models are severely biased when confounding factors differentially impact intervention and control clusters; premature adoption of intervention components is an example of this scenario. In these scenarios, models that incorporate fixed intervention-by-time interaction terms and an unstructured covariance for the intervention-by-cluster-by-time random effects result in unbiased estimates of the intervention effect, reach nominal confidence interval coverage, and preserve Type 1 error, but may reduce power.

CONCLUSIONS

The incorporation of fixed and random time effects in mixed effects models require certain assumptions about the impact of confounding by time in SWD. Violations of these assumptions can result in severe bias of the intervention effect estimate, under coverage of confidence intervals, and inflated Type 1 error. Since model choice has considerable impact on study power as well as validity of results, careful consideration needs to be given to choosing an appropriate model that takes into account potential confounding factors.

Maintaining the validity of inference in small-sample stepped wedge cluster randomized trials with binary outcomes when using generalized estimating equations

Stepped wedge cluster trials are an increasingly popular alternative to traditional parallel cluster randomized trials. Such trials often utilize a small number of clusters and numerous time intervals, and these components must be considered when choosing an analysis method. A generalized linear mixed model containing a random intercept and fixed time and intervention covariates is the most common analysis approach. However, the sole use of a random intercept applies a constant intraclass correlation coefficient structure, which is an assumption that is likely to be violated given stepped wedge trials (SWTs) have multiple time intervals. Alternatively, generalized estimating equations (GEE) are robust to the misspecification of the working correlation structure, although it has been shown that small-sample adjustments to standard error estimates and the use of appropriate degrees of freedom are required to maintain the validity of inference when the number of clusters is small. In this article, we show, using an extensive simulation study based on a motivating example and a more general design, the use of GEE can maintain the validity of inference in small-sample SWTs with binary outcomes. Furthermore, we show which combinations of bias corrections to standard error estimates and degrees of freedom work best in terms of attaining nominal type I error rates.

Statistical Properties of Stepped Wedge Cluster-Randomized Trials in Infectious Disease Outbreaks

Randomized controlled trials are crucial for the evaluation of interventions such as vaccinations, but the design and analysis of these studies during infectious disease outbreaks is complicated by statistical, ethical, and logistical factors. Attempts to resolve these complexities have led to the proposal of a variety of trial designs, including individual randomization and several types of cluster randomization designs: parallel-arm, ring vaccination, and stepped wedge designs. Because of the strong time trends present in infectious disease incidence, however, methods generally used to analyze stepped wedge trials may not perform well in these settings. Using simulated outbreaks, we evaluate various designs and analysis methods, including recently proposed methods for analyzing stepped wedge trials, to determine the statistical properties of these methods. While new methods for analyzing stepped wedge trials can provide some improvement over previous methods, we find that they still lag behind parallel-arm cluster-randomized trials and individually-randomized trials in achieving adequate power to detect intervention effects. We also find that these methods are highly sensitive to the weighting of effect estimates across time periods. Despite the value of new methods, stepped wedge trials still have statistical disadvantages compared to other trial designs in epidemic settings.

Novel methods for the analysis of stepped wedge cluster randomized trials

Stepped wedge cluster randomized trials (SW-CRTs) have become increasingly popular and are used for a variety of interventions and outcomes, often chosen for their feasibility advantages. SW-CRTs must account for time trends in the outcome because of the staggered rollout of the intervention. Robust inference procedures and nonparametric analysis methods have recently been proposed to handle such trends without requiring strong parametric modeling assumptions, but these are less powerful than model-based approaches. We propose several novel analysis methods that reduce reliance on modeling assumptions while preserving some of the increased power provided by the use of mixed effects models. In one method, we use the synthetic control approach to find the best matching clusters for a given intervention cluster. Another method makes use of within-cluster crossover information to construct an overall estimator. We also consider methods that combine these approaches to further improve power. We test these methods on simulated SW-CRTs, describing scenarios in which these methods have increased power compared with existing nonparametric methods while preserving nominal validity when mixed effects models are misspecified. We also demonstrate theoretical properties of these estimators with less restrictive assumptions than mixed effects models. Finally, we propose avenues for future research on the use of these methods; motivation for such research arises from their flexibility, which allows the identification of specific causal contrasts of interest, their robustness, and the potential for incorporating covariates to further increase power. Investigators conducting SW-CRTs might well consider such methods when common modeling assumptions may not hold.

Protocol for a randomised trial of an interprofessional team-delivered intervention to support surrogate decision-makers in ICUs

INTRODUCTION

Although shortcomings in clinician-family communication and decision making for incapacitated, critically ill patients are common, there are few rigorously tested interventions to improve outcomes. In this manuscript, we present our methodology for the Pairing Re-engineered Intensive Care Unit Teams with Nurse-Driven Emotional support and Relationship Building (PARTNER 2) trial, and discuss design challenges and their resolution.

METHODS AND ANALYSIS

This is a pragmatic, stepped-wedge, cluster randomised controlled trial comparing the PARTNER 2 intervention to usual care among 690 incapacitated, critically ill patients and their surrogates in five ICUs in Pennsylvania. Eligible subjects will include critically ill patients at high risk of death and/or severe long-term functional impairment, their main surrogate decision-maker and their clinicians. The PARTNER intervention is delivered by the interprofessional ICU team and overseen by 4-6 nurses from each ICU. It involves: (1) advanced communication skills training for nurses to deliver support to surrogates throughout the ICU stay; (2) deploying a structured family support pathway; (3) enacting strategies to foster collaboration between ICU and palliative care services and (4) providing intensive implementation support to each ICU to incorporate the family support pathway into clinicians' workflow. The primary outcome is surrogates' ratings of the quality of communication during the ICU stay as assessed by telephone at 6-month follow-up. Prespecified secondary outcomes include surrogates' scores on the Hospital Anxiety and Depression Scale, the Impact of Event Scale, the modified Patient Perception of Patient Centredness scale, the Decision Regret Scale, nurses' scores on the Maslach Burnout Inventory, and length of stay during and costs of the index hospitalisation.We also discuss key methodological challenges, including determining the optimal level of randomisation, using existing staff to deploy the intervention and maximising long-term follow-up of participants.

ETHICS AND DISSEMINATION

We obtained ethics approval through the University of Pittsburgh, Human Research Protection Office. The findings will be published in peer-reviewed journals.

TRIAL REGISTRATION NUMBER

NCT02445937.

Information growth for sequential monitoring of clinical trials with a stepped wedge cluster randomized design and unknown intracluster correlation

Background/aims

In a stepped wedge study design, study clusters usually start with the baseline treatment and then cross over to the intervention at randomly determined times. Such designs are useful when the intervention must be delivered at the cluster level and are becoming increasingly common in practice. In these trials, if the outcome is death or serious morbidity, one may have an ethical imperative to monitor the trial and stop before maximum enrollment if the new therapy is proven to be beneficial. In addition, because formal monitoring allows for the stoppage of trials when a significant benefit for new therapy has been ruled out, their use can make a research program more efficient. However, use of the stepped wedge cluster randomized study design complicates the implementation of standard group sequential monitoring methods. Both the correlation of observations introduced by the clustered randomization and the timing of crossover from one treatment to the other impact the rate of information growth, an important component of an interim analysis.

Methods

We simulated cross-sectional stepped wedge study data in order to evaluate the impact of sequential monitoring on the Type I error and power when the true intracluster correlation is unknown. We studied the impact of varying intracluster correlations, treatment effects, methods of estimating the information growth, and boundary shapes.

Results

While misspecified information growth can impact both the Type I error and power of a study in some settings, we observed little inflation of the Type I error and only moderate reductions in power across a range of misspecified information growth patterns in our simulations.

Conclusion

Taking the study design into account and using either an estimate of the intracluster correlation from the ongoing study or other data in the same clusters should allow for easy implementation of group sequential methods in future stepped wedge designs.

Breastfeeding peer counselling for mothers of preterm neonates: protocol of a stepped-wedge cluster randomised controlled trial

INTRODUCTION

Among preterm infants, mother's own milk feeding reduces neonatal morbidity and decreases the length of hospital stay. However, breastfeeding rates and duration are lower than among term infants. It is reported that peer counselling is effective in increasing breast feeding in term infants in low-income and middle-income countries, but results are mixed in high-income countries. We aim to investigate herein whether peer counselling may be a feasible and effective breastfeeding support among preterm infants in French-speaking high-income countries.

METHODS AND ANALYSIS

Eight European centres will participate in this stepped-wedge cluster randomised controlled trial. We plan to include 2400 hospitalised neonates born before 35 gestational weeks. Each centre will begin with an observational period. Every 3 months, a randomised cluster (centre) will begin the interventional period with peer counsellors until the end of the study. The counsellors will be trained and supervised by the trained nurses. They will have a weekly contact with participating mothers, with a face-to-face meeting at least once every fortnight. During these meetings, peer counsellors will listen to mothers' concerns, share experiences and help the mother with their own knowledge of breast feeding. The main outcome is breastfeeding rate at 2 months corrected age. Secondary outcomes are breastfeeding rates at hospital discharge and at 6 months, breastfeeding duration and severe neonatal morbidity and mortality. The mental health of the mother, mother-infant bonding and infant behaviour will be assessed using self-report questionnaires. A neurodevelopmental follow-up, a cost-effectiveness analysis and a cost-consequence at 2 years corrected age will be performed among infants in a French subgroup.

ETHICS AND DISSEMINATION

French, Belgian and Swiss ethics committees gave their agreement. Publications in peer-reviewed journals are planned on breast feeding, mental health and economic outcomes.

TRIAL REGISTRATION NUMBER

NCT03156946.

Advancing successful implementation of task-shifted mental health care in low-resource settings (BASIC): protocol for a stepped wedge cluster randomized trial

BACKGROUND

The mental health treatment gap-the difference between those with mental health need and those who receive treatment-is high in low- and middle-income countries. Task-shifting has been used to address the shortage of mental health professionals, with a growing body of research demonstrating the effectiveness of mental health interventions delivered through task-shifting. However, very little research has focused on how to embed, support, and sustain task-shifting in government-funded systems with potential for scale up. The goal of the Building and Sustaining Interventions for Children (BASIC) study is to examine implementation policies and practices that predict adoption, fidelity, and sustainment of a mental health intervention in the education sector via teacher delivery and the health sector via community health volunteer delivery.

METHODS

BASIC is a Hybrid Type II Implementation-Effectiveness trial. The study design is a stepped wedge, cluster randomized trial involving 7 sequences of 40 schools and 40 communities surrounding the schools. Enrollment consists of 120 teachers, 120 community health volunteers, up to 80 site leaders, and up to 1280 youth and one of their primary guardians. The evidence-based mental health intervention is a locally adapted version of Trauma-focused Cognitive Behavioral Therapy, called Pamoja Tunaweza. Lay counselors are trained and supervised in Pamoja Tunaweza by local trainers who are experienced in delivering the intervention and who participated in a Train-the-Trainer model of skills transfer. After the first sequence completes implementation, in-depth interviews are conducted with initial implementing sites' counselors and leaders. Findings are used to inform delivery of implementation facilitation for subsequent sequences' sites. We use a mixed methods approach including qualitative comparative analysis to identify necessary and sufficient implementation policies and practices that predict 3 implementation outcomes of interest: adoption, fidelity, and sustainment. We also examine child mental health outcomes and cost of the intervention in both the education and health sectors.

DISCUSSION

The BASIC study will provide knowledge about how implementation of task-shifted mental health care can be supported in government systems that already serve children and adolescents. Knowledge about implementation policies and practices from BASIC can advance the science of implementation in low-resource contexts.

TRIAL REGISTRATION

Trial Registration: ClinicalTrials.gov Identifier: NCT03243396. Registered 9th August 2017, https://clinicaltrials.gov/ct2/show/NCT03243396.

swpermute: Permutation tests for Stepped-Wedge Cluster-Randomised Trials

Permutation tests are useful in stepped-wedge trials to provide robust statistical tests of intervention-effect estimates. However, the Stata command permute does not produce valid tests in this setting because individual observations are not exchangeable. We introduce the swpermute command that permutes clusters to sequences to maintain exchangeability. The command provides additional functionality to aid users in performing analyses of stepped-wedge trials. In particular, we include the option "withinperiod" that performs the specified analysis separately in each period of the study with the resulting period-specific intervention-effect estimates combined as a weighted average. We also include functionality to test non-zero null hypotheses to aid the construction of confidence intervals. Examples of the application of swpermute are given using data from a trial testing the impact of a new tuberculosis diagnostic test on bacterial confirmation of a tuberculosis diagnosis.

Robust inference for the stepped wedge design

Stepped wedge designed trials are a type of cluster-randomized study in which the intervention is introduced to each cluster in a random order over time. This design is often used to assess the effect of a new intervention as it is rolled out across a series of clinics or communities. Based on a permutation argument, we derive a closed-form expression for an estimate of the intervention effect, along with its standard error, for a stepped wedge design trial. We show that these estimates are robust to misspecification of both the mean and covariance structure of the underlying data-generating mechanism, thereby providing a robust approach to inference for the intervention effect in stepped wedge designs. We use simulations to evaluate the type 1 error and power of the proposed estimate and to compare the performance of the proposed estimate to the optimal estimate when the correct model specification is known. The limitations, possible extensions, and open problems regarding the method are discussed.

Using simulation studies to evaluate statistical methods

Simulation studies are computer experiments that involve creating data by pseudo-random sampling. A key strength of simulation studies is the ability to understand the behavior of statistical methods because some "truth" (usually some parameter/s of interest) is known from the process of generating the data. This allows us to consider properties of methods, such as bias. While widely used, simulation studies are often poorly designed, analyzed, and reported. This tutorial outlines the rationale for using simulation studies and offers guidance for design, execution, analysis, reporting, and presentation. In particular, this tutorial provides a structured approach for planning and reporting simulation studies, which involves defining aims, data-generating mechanisms, estimands, methods, and performance measures ("ADEMP"); coherent terminology for simulation studies; guidance on coding simulation studies; a critical discussion of key performance measures and their estimation; guidance on structuring tabular and graphical presentation of results; and new graphical presentations. With a view to describing recent practice, we review 100 articles taken from Volume 34 of Statistics in Medicine, which included at least one simulation study and identify areas for improvement.

Quality of stepped-wedge trial reporting can be reliably assessed using an updated CONSORT: crowd-sourcing systematic review

OBJECTIVES

The Consolidated Standards of Reporting Trials extension for the stepped-wedge cluster randomized trial (SW-CRT) is a recently published reporting guideline for SW-CRTs. We assess the quality of reporting of a recent sample of SW-CRTs.

STUDY DESIGN AND SETTING

Quality of reporting was asssessed according to the 26 items in the new guideline using a novel crowd sourcing methodology conducted independently and in duplicate, with random assignment, by 50 reviewers. We assessed reliability of the quality assessments, proposing this as a novel way to assess robustness of items in reporting guidelines.

RESULTS

Several items were well reported. Some items were very poorly reported, including several items that have unique requirements for the SW-CRT, such as the rationale for use of the design, description of the design, identification and recruitment of participants within clusters, and concealment of cluster allocation (not reported in more than 50% of the reports). Agreement across items was moderate (median percentage agreement was 76% [IQR 64 to 86]). Agreement was low for several items including the description of the trial design and why trial ended or stopped for example.

CONCLUSIONS

When reporting SW-CRTs, authors should pay particular attention to ensure clear reporting on the exact format of the design with justification, as well as how clusters and individuals were identified for inclusion in the study, and whether this was done before or after randomization of the clusters, which are crucial for risk of bias assessments. Some items, including why the trial ended, might either not be relevant to SW-CRTs or might be unclearly described in the statement.

A Randomized Trial of a Family-Support Intervention in Intensive Care Units

BACKGROUND

Surrogate decision makers for incapacitated, critically ill patients often struggle with decisions related to goals of care. Such decisions cause psychological distress in surrogates and may lead to treatment that does not align with patients' preferences.

METHODS

We conducted a stepped-wedge, cluster-randomized trial involving patients with a high risk of death and their surrogates in five intensive care units (ICUs) to compare a multicomponent family-support intervention delivered by the interprofessional ICU team with usual care. The primary outcome was the surrogates' mean score on the Hospital Anxiety and Depression Scale (HADS) at 6 months (scores range from 0 to 42, with higher scores indicating worse symptoms). Prespecified secondary outcomes were the surrogates' mean scores on the Impact of Event Scale (IES; scores range from 0 to 88, with higher scores indicating worse symptoms), the Quality of Communication (QOC) scale (scores range from 0 to 100, with higher scores indicating better clinician-family communication), and a modified Patient Perception of Patient Centeredness (PPPC) scale (scores range from 1 to 4, with lower scores indicating more patient- and family-centered care), as well as the mean length of ICU stay.

RESULTS

A total of 1420 patients were enrolled in the trial. There was no significant difference between the intervention group and the control group in the surrogates' mean HADS score at 6 months (11.7 and 12.0, respectively; beta coefficient, -0.34; 95% confidence interval [CI], -1.67 to 0.99; P=0.61) or mean IES score (21.2 and 20.3; beta coefficient, 0.90; 95% CI, -1.66 to 3.47; P=0.49). The surrogates' mean QOC score was better in the intervention group than in the control group (69.1 vs. 62.7; beta coefficient, 6.39; 95% CI, 2.57 to 10.20; P=0.001), as was the mean modified PPPC score (1.7 vs. 1.8; beta coefficient, -0.15; 95% CI, -0.26 to -0.04; P=0.006). The mean length of stay in the ICU was shorter in the intervention group than in the control group (6.7 days vs. 7.4 days; incidence rate ratio, 0.90; 95% CI, 0.81 to 1.00; P=0.045), a finding mediated by the shortened mean length of stay in the ICU among patients who died (4.4 days vs. 6.8 days; incidence rate ratio, 0.64; 95% CI, 0.52 to 0.78; P<0.001).

CONCLUSIONS

Among critically ill patients and their surrogates, a family-support intervention delivered by the interprofessional ICU team did not significantly affect the surrogates' burden of psychological symptoms, but the surrogates' ratings of the quality of communication and the patient- and family-centeredness of care were better and the length of stay in the ICU was shorter with the intervention than with usual care. (Funded by the UPMC Health System and the Greenwall Foundation; PARTNER ClinicalTrials.gov number, NCT01844492 .).

Robust analysis of stepped wedge trials using cluster-level summaries within periods

In stepped‐wedge trials (SWTs), the intervention is rolled out in a random order over more than 1 time‐period. SWTs are often analysed using mixed‐effects models that require strong assumptions and may be inappropriate when the number of clusters is small.

We propose a non‐parametric within‐period method to analyse SWTs. This method estimates the intervention effect by comparing intervention and control conditions in a given period using cluster‐level data corresponding to exposure. The within‐period intervention effects are combined with an inverse‐variance‐weighted average, and permutation tests are used. We present an example and, using simulated data, compared the method to (1) a parametric cluster‐level within‐period method, (2) the most commonly used mixed‐effects model, and (3) a more flexible mixed‐effects model. We simulated scenarios where period effects were common to all clusters, and when they varied according to a distribution informed by routinely collected health data.

The non‐parametric within‐period method provided unbiased intervention effect estimates with correct confidence‐interval coverage for all scenarios. The parametric within‐period method produced confidence intervals with low coverage for most scenarios. The mixed‐effects models' confidence intervals had low coverage when period effects varied between clusters but had greater power than the non‐parametric within‐period method when period effects were common to all clusters.

The non‐parametric within‐period method is a robust method for analysing SWT. The method could be used by trial statisticians who want to emphasise that the SWT is a randomised trial, in the common position of being uncertain about whether data will meet the assumptions necessary for mixed‐effect models.

Modeling clustering and treatment effect heterogeneity in parallel and stepped-wedge cluster randomized trials

Cluster randomized trials are frequently used in health service evaluation. It is common practice to use an analysis model with a random effect to allow for clustering at the analysis stage. In designs where clusters are exposed to both control and treatment conditions, it may be of interest to examine treatment effect heterogeneity across clusters. In designs where clusters are not exposed to both control and treatment conditions, it can also be of interest to allow heterogeneity in the degree of clustering between arms. These two types of heterogeneity are related. It has been proposed in both parallel cluster trials, stepped-wedge, and other cross-over designs that this heterogeneity can be allowed for by incorporating additional random effect(s) into the model. Here, we show that the choice of model parameterization needs careful consideration as some parameterizations for additional heterogeneity induce unnecessary or implausible assumptions. We suggest more appropriate parameterizations, discuss their relative advantages, and demonstrate the implications of these model choices using a real example of a parallel cluster trial and a simulated stepped-wedge trial.

Bias and inference from misspecified mixed-effect models in stepped wedge trial analysis

Many stepped wedge trials (SWTs) are analysed by using a mixed‐effect model with a random intercept and fixed effects for the intervention and time periods (referred to here as the standard model). However, it is not known whether this model is robust to misspecification.

We simulated SWTs with three groups of clusters and two time periods; one group received the intervention during the first period and two groups in the second period. We simulated period and intervention effects that were either common‐to‐all or varied‐between clusters. Data were analysed with the standard model or with additional random effects for period effect or intervention effect. In a second simulation study, we explored the weight given to within‐cluster comparisons by simulating a larger intervention effect in the group of the trial that experienced both the control and intervention conditions and applying the three analysis models described previously.

Across 500 simulations, we computed bias and confidence interval coverage of the estimated intervention effect.

We found up to 50% bias in intervention effect estimates when period or intervention effects varied between clusters and were treated as fixed effects in the analysis. All misspecified models showed undercoverage of 95% confidence intervals, particularly the standard model. A large weight was given to within‐cluster comparisons in the standard model.

In the SWTs simulated here, mixed‐effect models were highly sensitive to departures from the model assumptions, which can be explained by the high dependence on within‐cluster comparisons. Trialists should consider including a random effect for time period in their SWT analysis model. © 2017 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.