Designing studies that would address the multilayered nature of health care

The Importance of Evaluating Health Disparities Research

Health disparity populations are socially disadvantaged, and the multiple levels of discrimination they often experience mean that their characteristics and attributes differ from those of the mainstream. Programs and policies targeted at reducing health disparities or improving minority health must consider these differences. Despite the importance of evaluating health disparities research to produce high-quality data that can guide decision-making, it is not yet a customary practice. Although health disparities evaluations incorporate the same scientific methods as all evaluations, they have unique components such as population characteristics, sociocultural context, and the lack of health disparity common indicators and metrics that must be considered in every phase of the research. This article describes evaluation strategies grouped into 3 components: formative (needs assessments and process), design and methodology (multilevel designs used in real-world settings), and summative (outcomes, impacts, and cost). Each section will describe the standards for each component, discuss the unique health disparity aspects, and provide strategies from the National Institute on Minority Health and Health Disparities Metrics and Measures Visioning Workshop (April 2016) to advance the evaluation of health disparities research.

Essential Ingredients and Innovations in the Design and Analysis of Group-Randomized Trials

This article reviews the essential ingredients and innovations in the design and analysis of group-randomized trials. The methods literature for these trials has grown steadily since they were introduced to the biomedical research community in the late 1970s, and we summarize those developments. We review, in addition to the group-randomized trial, methods for two closely related designs, the individually randomized group treatment trial and the stepped-wedge group-randomized trial. After describing the essential ingredients for these designs, we review the most important developments in the evolution of their methods using a new bibliometric tool developed at the National Institutes of Health. We then discuss the questions to be considered when selecting from among these designs or selecting the traditional randomized controlled trial. We close with a review of current methods for the analysis of data from these designs, a case study to illustrate each design, and a brief summary.

Evaluating moderators of beneficial effects of severity-based assignment to substance use treatments in impaired drivers

Remedial programs for impaired driving offenders have proved valuable in reducing subsequent alcohol and other drug use and preventing recidivism in this population. Many of these programs are based on a severity-based assignment scheme, where individuals assessed to have greater problems or be at higher risk are assigned to longer, more intensive interventions. Recent research, using regression discontinuity analyses, provided support for severity-based assignment schemes in demonstrating that those with higher problem or risk levels assigned to longer and more intensive programming showed a significant reduction in drinking days over a follow-up interval, attributable to program assignment. Regression discontinuity analyses can also be used to assess moderators of this assignment benefit. We report an assessment of the impact of eight potential moderators of assignment benefit, derived from a factor analysis of the Research on Addictions Self-Inventory screening instrument. Five of the eight factors were found to moderate the assignment benefit: Negative Affect, Sensation Seeking, High Risk Lifestyle, Alcohol Problems, and Family History. The significance of these results for developing more effective program assignment procedures is discussed.

Design and analysis of group-randomized trials in cancer: A review of current practices

The purpose of this paper is to summarize current practices for the design and analysis of group-randomized trials involving cancer-related risk factors or outcomes and to offer recommendations to improve future trials. We searched for group-randomized trials involving cancer-related risk factors or outcomes that were published or online in peer-reviewed journals in 2011-15. During 2016-17, in Bethesda MD, we reviewed 123 articles from 76 journals to characterize their design and their methods for sample size estimation and data analysis. Only 66 (53.7%) of the articles reported appropriate methods for sample size estimation. Only 63 (51.2%) reported exclusively appropriate methods for analysis. These findings suggest that many investigators do not adequately attend to the methodological challenges inherent in group-randomized trials. These practices can lead to underpowered studies, to an inflated type 1 error rate, and to inferences that mislead readers. Investigators should work with biostatisticians or other methodologists familiar with these issues. Funders and editors should ensure careful methodological review of applications and manuscripts. Reviewers should ensure that studies are properly planned and analyzed. These steps are needed to improve the rigor and reproducibility of group-randomized trials. The Office of Disease Prevention (ODP) at the National Institutes of Health (NIH) has taken several steps to address these issues. ODP offers an online course on the design and analysis of group-randomized trials. ODP is working to increase the number of methodologists who serve on grant review panels. ODP has developed standard language for the Application Guide and the Review Criteria to draw investigators' attention to these issues. Finally, ODP has created a new Research Methods Resources website to help investigators, reviewers, and NIH staff better understand these issues.

Selecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research

Interventional researchers face many design challenges when assessing intervention implementation in real-world settings. Intervention implementation requires holding fast on internal validity needs while incorporating external validity considerations (such as uptake by diverse subpopulations, acceptability, cost, and sustainability). Quasi-experimental designs (QEDs) are increasingly employed to achieve a balance between internal and external validity. Although these designs are often referred to and summarized in terms of logistical benefits, there is still uncertainty about (a) selecting from among various QEDs and (b) developing strategies to strengthen the internal and external validity of QEDs. We focus here on commonly used QEDs (prepost designs with nonequivalent control groups, interrupted time series, and stepped-wedge designs) and discuss several variants that maximize internal and external validity at the design, execution and implementation, and analysis stages.

Sugarsquare, a Web-Based Patient Portal for Parents of a Child With Type 1 Diabetes: Multicenter Randomized Controlled Feasibility Trial

Background

Raising a child with type 1 diabetes (T1D) means combining the demands of the disease management with everyday parenting, which is associated with increased levels of distress. A Web-based patient portal, Sugarsquare, was developed to support parents, by providing online parent-professional communication, online peer support and online disease information.

Objective

The first aim of this study was to assess the feasibility of conducting a multicenter, randomized controlled trial in Dutch parents of a child with T1D. The second aim was to assess the feasibility of implementing Sugarsquare in clinical practice.

Methods

The parents of 105 children (N=105) with T1D below the age of 13 participated in a 6-month multicenter randomized controlled feasibility trial. They were randomly assigned to an experimental (n=54, usual care and Sugarsquare) or a control group (n=51, usual care). Attrition rates and user statistics were gathered to evaluate feasibility of the trial and implementation. To determine potential efficacy, the parenting stress index (PSI-SF) was assessed at baseline (T0) and after 6 months (T1).

Results

Of a potential population of parents of 445 children, 189 were willing to participate (enrollment refusal=57.5%, n=256), 142 filled in the baseline questionnaire (baseline attrition rate=25%, n=47), and 105 also filled in the questionnaire at T1 (post randomization attrition rate during follow-up=26%, n=32). As such, 24% of the potential population participated. Analysis in the experimental group (n=54) revealed a total of 32 (59%) unique users, divided into 12 (38%) frequent users, 9 (28%) incidental users, and 11 (34%) low-frequent users. Of the total of 44 professionals, 34 (77%) logged in, and 32 (73%) logged in repeatedly. Analysis of the user statistics in the experimental group further showed high practicability and integration in all users, moderate acceptability and demand in parents, and high acceptability and demand in health care professionals. Baseline parenting stress index scores were related to the parents’ frequency of logging on (ρ=.282, P=.03) and page-views (ρ=.304, P=.01). No significant differences in change in parenting stress between experimental and control group were found (F_3,101=.49, P=.49).

Conclusions

The trial can be considered feasible, considering the average enrollment refusal rate, baseline attrition rate and postrandomization attrition rate, compared to other eHealth studies, although lower than hypothesized. Implementing Sugarsquare in clinical practice was partly feasible, given moderate demand and acceptability in parent users and lack of potential efficacy. Parents who reported higher levels of parenting stress used Sugarsquare more often than other parents, although Sugarsquare did not reduce parenting stress. These results indicate that Web-based interventions are a suitable way of providing parents of children with T1D with additional support. Future studies should determine how Sugarsquare could reduce parenting stress, for instance by adding targeted interventions. Factors potentially contributing to successful implementation are suggested.

Trial Registration

Nederlands Trial Register Number: NTR3643; http://www.trialregister.nl/trialreg/admin/rctview.asp?TC=3643 (Archived by WebCite at http://www.webcitation.org/6qihOVCi6)

Review of Recent Methodological Developments in Group-Randomized Trials: Part 1-Design

In 2004, Murray et al. reviewed methodological developments in the design and analysis of group-randomized trials (GRTs). We have highlighted the developments of the past 13 years in design with a companion article to focus on developments in analysis. As a pair, these articles update the 2004 review. We have discussed developments in the topics of the earlier review (e.g., clustering, matching, and individually randomized group-treatment trials) and in new topics, including constrained randomization and a range of randomized designs that are alternatives to the standard parallel-arm GRT. These include the stepped-wedge GRT, the pseudocluster randomized trial, and the network-randomized GRT, which, like the parallel-arm GRT, require clustering to be accounted for in both their design and analysis.

Multilevel Interventions Targeting Obesity: Research Recommendations for Vulnerable Populations

INTRODUCTION

The origins of obesity are complex and multifaceted. To be successful, an intervention aiming to prevent or treat obesity may need to address multiple layers of biological, social, and environmental influences.

METHODS

NIH recognizes the importance of identifying effective strategies to combat obesity, particularly in high-risk and disadvantaged populations with heightened susceptibility to obesity and subsequent metabolic sequelae. To move this work forward, the National Heart, Lung, and Blood Institute, in collaboration with the NIH Office of Behavioral and Social Science Research and NIH Office of Disease Prevention convened a working group to inform research on multilevel obesity interventions in vulnerable populations. The working group reviewed relevant aspects of intervention planning, recruitment, retention, implementation, evaluation, and analysis, and then made recommendations.

RESULTS

Recruitment and retention techniques used in multilevel research must be culturally appropriate and suited to both individuals and organizations. Adequate time and resources for preliminary work are essential. Collaborative projects can benefit from complementary areas of expertise and shared investigations rigorously pretesting specific aspects of approaches. Study designs need to accommodate the social and environmental levels under study, and include appropriate attention given to statistical power. Projects should monitor implementation in the multiple venues and include a priori estimation of the magnitude of change expected within and across levels.

CONCLUSIONS

The complexity and challenges of delivering interventions at several levels of the social-ecologic model require careful planning and implementation, but hold promise for successful reduction of obesity in vulnerable populations.

Measuring improvement in populations: implementing and evaluating successful change in lung cancer care

Improving quality of care in lung cancer, the leading cause of cancer death worldwide and in the United States, is a major public health challenge. Such improvement requires accurate and meaningful measurement of quality of care. Preliminary indicators have been derived from clinical practice guidelines and expert opinions, but there are few standard sets of quality of care measures for lung cancer in the United States or elsewhere. Research to develop validated evidence-based quality of care measures is critical in promoting population improvement initiatives in lung cancer. Furthermore, novel research designs beyond the traditional randomized controlled trials (RCTs) are needed for wide-scale applications of quality improvement and should extend into alternative designs such as quasi-experimental designs, rigorous observational studies, population modeling, and other pragmatic study designs. We discuss several study design options to aid the development of practical, actionable, and measurable quality standards for lung cancer care. We also provide examples of ongoing pragmatic studies for the dissemination and implementation of lung cancer quality improvement interventions in community settings.

Aquatic Exercises were Associated with an Acceptable State of Symptoms in Patients with Inflammatory Rheumatic Diseases: Results from a Study with Interrupted Time-Series Design

AIM OF STUDY

The aim of this study was two-fold: to compare symptoms and daily activity in patients with inflammatory rheumatic diseases across periods with and without aquatic exercises, and to examine whether the patients reached an acceptable state of symptoms during the periods with aquatic exercises.

METHODS

Thirty-six patients reported pain, fatigue, stiffness and ability to carry out daily activities across periods with and without aquatic exercises. The study has an interrupted time-series design and variables were collected with text messages on mobile phones twice a week over a period of 35 weeks.

RESULTS

There was a significant reduction in pain, fatigue, stiffness and enhanced level of daily activity (p > 0.05) during periods of aquatic exercises compared to periods without. Further, a significantly higher proportion of patients reached an acceptable state for both pain and fatigue during periods with aquatic exercises.

CONCLUSIONS

Living with an inflammatory rheumatic disease is a lifelong challenge. Pain and fatigue are considered major obstacles for daily functioning and adequate self-management strategies are requested. Based on the high proportion of patients reporting to be in an acceptable state of both pain and fatigue during periods with aquatic exercises, the intervention should be regarded as an important self-management tool rather than a treatment option assuming long-lasting effects. Copyright © 2014 John Wiley & Sons, Ltd.

Expanding research to provide an evidence base for nutritional interventions for the management of inborn errors of metabolism

A trans-National Institutes of Health initiative, Nutrition and Dietary Supplement Interventions for Inborn Errors of Metabolism (NDSI-IEM), was launched in 2010 to identify gaps in knowledge regarding the safety and utility of nutritional interventions for the management of inborn errors of metabolism (IEM) that need to be filled with evidence-based research. IEM include inherited biochemical disorders in which specific enzyme defects interfere with the normal metabolism of exogenous (dietary) or endogenous protein, carbohydrate, or fat. For some of these IEM, effective management depends primarily on nutritional interventions. Further research is needed to demonstrate the impact of nutritional interventions on individual health outcomes and on the psychosocial issues identified by patients and their families. A series of meetings and discussions were convened to explore the current United States' funding and regulatory infrastructure and the challenges to the conduct of research for nutritional interventions for the management of IEM. Although the research and regulatory infrastructure are well-established, a collaborative pathway that includes the professional and advocacy rare disease community and federal regulatory and research agencies will be needed to overcome current barriers.

Developing a research agenda for cardiovascular disease prevention in high-risk rural communities

The National Institutes of Health convened a workshop to engage researchers and practitioners in dialogue on research issues viewed as either unique or of particular relevance to rural areas, key content areas needed to inform policy and practice in rural settings, and ways rural contexts may influence study design, implementation, assessment of outcomes, and dissemination. Our purpose was to develop a research agenda to address the disproportionate burden of cardiovascular disease (CVD) and related risk factors among populations living in rural areas. Complementary presentations used theoretical and methodological principles to describe research and practice examples from rural settings. Participants created a comprehensive CVD research agenda that identified themes and challenges, and provided 21 recommendations to guide research, practice, and programs in rural areas.

Evidence from regression-discontinuity analyses for beneficial effects of a criterion-based increase in alcohol treatment

Brief interventions effectively reduce alcohol problems; however, it is controversial whether longer interventions result in greater improvement. This study aims to determine whether an increase in treatment for people with more severe problems resulted in better outcome. We employed regression-discontinuity analyses to determine if drinking driver clients (n = 22,277) in Ontario benefited when they were assigned to a longer treatment program (8-hour versus 16-hour) based on assessed addiction severity criteria. Assignment to the longer16-hour program was based on two addiction severity measures derived from the Research Institute on Addictions Self-inventory (RIASI) (meeting criteria for assignment based on either the total RIASI score or the score on the recidivism subscale). The main outcome measure was self-reported number of days of alcohol use during the 90 days preceding the six month follow-up interview. We found significant reductions of one or two self-reported drinking days at the point of assignment, depending on the severity criterion used. These data suggest that more intensive treatment for alcohol problems may improve results for individuals with more severe problems.

Increasing the degrees of freedom in future group randomized trials: the df * method revisited

BACKGROUND

This article revisits an article published in Evaluation Review in 2005 on sample size estimation and power analysis for group-randomized trials. With help from a careful reader, we learned of an important error in the spreadsheet used to perform the calculations and generate the results presented in that article. As we studied the spreadsheet, we discovered other minor errors. When we corrected the errors, we found that the results were substantially different and that the conclusions reported in the original article were not always appropriate.

OBJECTIVE

This article corrects the errors and reports the results as they should have been reported originally.

METHOD

Using a random-effects meta-analytic model, estimates of intraclass correlation were combined from two studies to guide sample size calculations for a new study.

RESULTS

The df * method can result in improved power or smaller studies when used a priori to plan future group-randomized trials, though the improvements will be modest in larger studies and will likely be insufficient to provide adequate power to small studies.

CONCLUSION

Smaller group-randomized trials are often desirable, for example, as pilot studies to help plan for a full-scale efficacy trial, as replication studies, or in situations in which resource constraints prohibit a larger trial. We discuss the circumstances under which the df * method will be most helpful and the risks associated with conducting smaller studies.

A multilevel research perspective on cancer care delivery: the example of follow-up to an abnormal mammogram

In 1999, researchers and policy makers recognized the challenge of creating an integrated patient-centered cancer care process across the many types of care from risk assessment through end of life. More than a decade later, there has been limited progress toward that goal even though the standard reductionist approach to health services and medical research has resulted in major advances in tests, procedures, and individualized patient approaches to care. In this commentary, we propose that considering an entire care process within its multilevel context may increase progress toward an integrated experience and improvements in the quality of care. As an illustrative case, we describe the multilevel context of care delivery for the process of follow-up to an abnormal screening mammogram. By taking a multilevel perspective on this process, we identify a rich set of options for intervening and improving follow-up to abnormalities and, therefore, outcomes of screening. We propose that taking this multilevel perspective when designing interventions may improve the quality of cancer care in an effective and sustainable way.

Multilevel interventions: study design and analysis issues

Multilevel interventions, implemented at the individual, physician, clinic, health-care organization, and/or community level, increasingly are proposed and used in the belief that they will lead to more substantial and sustained changes in behaviors related to cancer prevention, detection, and treatment than would single-level interventions. It is important to understand how intervention components are related to patient outcomes and identify barriers to implementation. Designs that permit such assessments are uncommon, however. Thus, an important way of expanding our knowledge about multilevel interventions would be to assess the impact of interventions at different levels on patients as well as the independent and synergistic effects of influences from different levels. It also would be useful to assess the impact of interventions on outcomes at different levels. Multilevel interventions are much more expensive and complicated to implement and evaluate than are single-level interventions. Given how little evidence there is about the value of multilevel interventions, however, it is incumbent upon those arguing for this approach to do multilevel research that explicates the contributions that interventions at different levels make to the desired outcomes. Only then will we know whether multilevel interventions are better than more focused interventions and gain greater insights into the kinds of interventions that can be implemented effectively and efficiently to improve health and health care for individuals with cancer. This chapter reviews designs for assessing multilevel interventions and analytic ways of controlling for potentially confounding variables that can account for the complex structure of multilevel data.

Multilevel interventions and racial/ethnic health disparities

To examine the impact of multilevel interventions (with three or more levels of influence) designed to reduce health disparities, we conducted a systematic review and meta-analysis of interventions for ethnic/racial minorities (all except non-Hispanic whites) that were published between January 2000 and July 2011. The primary aims were to synthesize the findings of studies evaluating multilevel interventions (three or more levels of influence) targeted at ethnic and racial minorities to reduce disparities in their health care and obtain a quantitative estimate of the effect of multilevel interventions on health outcomes among these subgroups. The electronic database PubMed was searched using Medical Subject Heading terms and key words. After initial review of abstracts, 26 published studies were systematically reviewed by at least two independent coders. Those with sufficient data (n = 12) were assessed by meta-analysis and examined for quality using a modified nine-item Physiotherapy Evidence Database coding scheme. The findings from this descriptive review suggest that multilevel interventions have positive effects on several health behavior outcomes, including cancer prevention and screening, as well improving the quality of health-care system processes. The weighted average effect size across studies for all health behavior outcomes reported at the individual participant level (k = 17) was odds ratio (OR) = 1.27 (95% confidence interval [CI] = 1.11 to 1.44); for the outcomes reported by providers or organizations, the weighted average effect size (k = 3) was OR = 2.53 (95% CI = 0.82 to 7.81). Enhanced application of theories to multiple levels of change, novel design approaches, and use of cultural leveraging in intervention design and implementation are proposed for this nascent field.

Factors in quality care--the case of follow-up to abnormal cancer screening tests--problems in the steps and interfaces of care

To achieve the benefit of cancer screening, appropriate follow-up of abnormal screening test results must occur. Such follow-up requires traversing the transition between screening detection and diagnosis, including several steps and interfaces in care. This article reviews factors and interventions associated with follow-up of abnormal tests for cervical, breast and colorectal cancers. We synthesized 12 reviews of descriptive and intervention studies published between 1980 and 2008. There was wide variability in definition of follow-up, setting, study population, and reported prevalence rates. Correlates of follow-up included patient characteristics (eg, knowledge and age), social support, provider characteristics, practice (eg, having reminders systems), community and professional norms (eg, quality measures), and policy (eg, federal programs). Effective interventions included patient education and support; delivery systems design changes, such as navigation; and information system changes, most notably patient tracking and physician reminders. Few studies focused explicitly on interfaces and steps of care, such as communication between primary care and specialists, or simultaneously targeted the multilevel factors that affect care. Future practice and research priorities should include development of clear operational definitions of the steps and interfaces related to patients, providers, and organizations; reflect evolving guidelines and new technologies; determine priorities for intervention testing; and improve measures and apply appropriate study designs.