Using process evaluation for program improvement in dose, fidelity and reach: the ACT trial experience

From theory to practice: intervention fidelity in a randomized controlled trial aiming to optimize weight development during pregnancy

BACKGROUND

Intervention fidelity is an increasingly important methodological concept in process evaluations. In this article, the authors investigated the intervention fidelity in a randomized controlled trial on excessive weight gain prevention in pregnancy.

METHOD

A sample of 109 audiotaped counseling sessions, linked to 65 women in the intervention group of the New Life(style) trial, was drawn. The following criteria were quantitatively evaluated using a fidelity checklist: (a) reach, (b) dose, (c) adherence to study objectives, (d) adherence to underlying problem-solving treatment (PST) theory, and (e) counselor competence.

RESULTS

A total of 60.4% received all counseling sessions. The dose of intervention components was generally moderate (50.9% to 60.4%), and the dose of PST components was low (17.3%). Adherence to study objectives was moderate (64.2%) and adherence to PST theory was low (43.2%). The counselors sufficiently stimulated the participant to optimize lifestyle (54.2% of the sessions), provided positive feedback (50.5%), and left the initiative regarding problem solving to the participant (71%). One of the two counselors performed significantly better on all measured criteria (p < .001).

CONCLUSIONS

Intervention fidelity in the New Life(style) trial was generally low to moderate. In future interventions, it is recommended to put more emphasis on counselor recruitment, training, and intervention protocol contents. Fellow researchers are encouraged to embed a process evaluation into all study stages, taking into account all essential process elements, and to link process outcomes to more distal, health outcomes.

Intervention description is not enough: evidence from an in-depth multiple case study on the untold role and impact of context in randomised controlled trials of seven complex interventions

BACKGROUND

A number of single case reports have suggested that the context within which intervention studies take place may challenge the assumptions that underpin randomised controlled trials (RCTs). However, the diverse ways in which context may challenge the central tenets of the RCT, and the degree to which this information is known to researchers or subsequently reported, has received much less attention. In this paper, we explore these issues by focusing on seven RCTs of interventions varying in type and degree of complexity, and across diverse contexts.

METHODS

This in-depth multiple case study using interviews, focus groups and documentary analysis was conducted in two phases. In phase one, a RCT of a nurse-led intervention provided a single exploratory case and informed the design, sampling and data collection within the main study. Phase two consisted of a multiple explanatory case study covering a spectrum of trials of different types of complex intervention. A total of eighty-four data sources across the seven trials were accessed.

RESULTS

We present consistent empirical evidence across all trials to indicate that four key elements of context (personal, organisational, trial and problem context) are crucial to understanding how a complex intervention works and to enable both assessments of internal validity and likely generalisability to other settings. The ways in which context challenged trial operation was often complex, idiosyncratic, and subtle; often falling outside of current trial reporting formats. However, information on such issues appeared to be available via first hand 'insider accounts' of each trial suggesting that improved reporting on the role of context is possible.

CONCLUSIONS

Sufficient detail about context needs to be understood and reported in RCTs of complex interventions, in order for the transferability of complex interventions to be assessed. Improved reporting formats that require and encourage the clarification of both general and project-specific threats to the likely internal and external validity need to be developed. In addition, a cultural change is required in which the open and honest reporting of such issues is seen as an indicator of study strength and researcher integrity, rather than a symbol of a poor quality study or investigator ability.

Process evaluation of a positive youth development program in Hong Kong based on different cohorts

There are only a few process evaluation studies on positive youth development programs, particularly in the Chinese context. This study aims to examine the quality of implementation of a positive youth development program (the Project P.A.T.H.S.: Positive Adolescent Training through Holistic Social Programmes) and investigate the relationships among program adherence, process factors, implementation quality, and perceived program success. Process evaluation of 97 classroom-based teaching units was conducted in 62 schools from 2005 to 2009. Findings based on different cohorts generally showed that there were high overall program adherence and implementation quality. Program adherence and implementation process were highly correlated with quality and success of the program. Multiple regression analyses further showed that both implementation process and program adherence are significant predictors of program quality and success. Theoretical and practical implications of the findings are discussed.

The examination of process evaluation use in church-based health interventions: a systematic review

Churches have been a popular site for the implementation of health promotion interventions. Although the efficacy of church-based health programs have been established, it is unknown which aspects of church-based health promotion drive health behavior change. Process evaluation is a way to increase our understanding of key components of church-based health promotion and to move the field forward. Thus, a systematic review of the utilization of process evaluation in church-based health programs was conducted. Articles from 1990 to 2008 were screened for eligibility, resulting in the analysis of 67 articles. The majority of church-based health programs assessed recruitment (88.1%) and reach (80.6%). About 28.4% assessed dose delivered, and 27.3% measured dose received. Context and fidelity was assessed by 34.3% and 20.9%, respectively, of church-based interventions. Approximately 9% of church-based programs measured fidelity. On average, only three of seven possible components of process evaluation were measured among the studies reviewed. The number of process evaluation components assessed did not differ by program feature (e.g., target population, target health condition, program objective, etc.). Consistency in the conceptualization and measurement of process evaluation may facilitate the implementation of a comprehensive process evaluation effort in church-based and other health promotion interventions.

Neighborhood and parental supports for physical activity in minority adolescents

BACKGROUND

Few studies have examined environmental, home, and parental supports for physical activity in underserved adolescents (low income, ethnic minority). Given the increasing incidence of obesity in minority adolescents, it is important to better understand ecologic determinants of physical activity in these youth. This study used an ecologic model to evaluate the significance of neighborhood, home, and parental supports for physical activity on moderate-to-vigorous (MV) physical activity in underserved adolescents.

DESIGN

The study was a secondary data analysis of a randomized controlled school-based trial "Active by Choice Today" (ACT) for increasing physical activity in underserved sixth-graders. Schools were matched on school size, percentage minorities, percentage entitled to free or reduced-price lunch, and urban or rural setting prior to randomization. This study used a randomly selected sample of parents (n=280) from the intervention and control schools whose adolescent was enrolled in the larger trial.

SETTING/PARTICIPANTS

A total of 679 6th-grade students (mean age=11.4 years, 70% African-American, 76% free or reduced-price lunch, 52% female) participated in the larger trial. Parents of 280 youth were contacted to participate in a telephone survey and 198 (71%) took part in the study.

INTERVENTIONS

The ACT trial was designed to test the efficacy of a 17-week (1 academic year) motivational plus behavioral skills intervention versus comparison after-school programs on increasing physical activity. A telephone survey was developed and was administered within 6 months after the trial began on parents of 198 adolescents from the ACT randomized school-based trial during 2005-2007.

MAIN OUTCOME MEASURES

The primary outcome measure was adolescent MVPA based on 7-day accelerometry estimates from baseline to mid-intervention. The data were analyzed in 2010-2011 and included both parent and adolescent self-reports of environmental, home, and family supports for physical activity.

RESULTS

Regression analyses indicated a significant effect of parental and neighborhood supports for physical activity on adolescent MVPA. Adolescents who perceived higher (vs lower) levels of parental support for physical activity engaged in more minutes of MVPA (B=3.01, SE=1.38, p<0.05) at mid-intervention. Adolescents who lived in neighborhoods with more (vs fewer) supports for physical activity (parks, lighting), also engaged in more minutes MVPA (B=4.27, SE=2.15, p<0.05).

CONCLUSIONS

Support from parents and neighborhood quality are both associated with increased physical activity in underserved adolescents.

Results of the "Active by Choice Today" (ACT) randomized trial for increasing physical activity in low-income and minority adolescents

OBJECTIVE

This study reports the results of the "Active by Choice Today" (ACT) trial for increasing moderate-to-vigorous physical activity (MVPA) in low-income and minority adolescents.

DESIGN

The ACT program was a randomized controlled school-based trial testing the efficacy of a motivational plus behavioral skills intervention on increasing MVPA in underserved adolescents. Twenty-four middle schools were matched on school size, percentage minorities, percentage free or reduce lunch, and urban or rural setting before randomization. A total of 1,563 6th grade students (mean age, 11.3 years, 73% African American, 71% free or reduced lunch, 55% female) participated in either a 17-week (over one academic year) intervention or comparison after-school program.

MAIN OUTCOME MEASURE

The primary outcome measure was MVPA based on 7-day accelerometry estimates at 2-weeks postintervention and an intermediate outcome was MVPA at midintervention.

RESULTS

At midintervention students in the intervention condition engaged in 4.87 greater minutes of MVPA per day (95% CI: 1.18 to 8.57) than control students. Students in intervention schools engaged in 9.11 min (95% CI: 5.73 to 12.48) more of MVPA per day than those in control schools during the program time periods; indicating a 27 min per week increase in MVPA. No significant effect of the ACT intervention was found outside of school times or for MVPA at 2-weeks postintervention.

CONCLUSIONS

Motivational and behavioral skills programs are effective at increasing MVPA in low-income and minority adolescents during program hours, but further research is needed to address home barriers to youth MVPA.