1
|
Ostermeier E, Burke SM, Gilliland J, Tucker P. Implementation models and frameworks used to guide community-based physical activity programs for children: a scoping review. BMC Public Health 2023; 23:1604. [PMID: 37612686 PMCID: PMC10463798 DOI: 10.1186/s12889-023-16465-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2023] [Accepted: 08/06/2023] [Indexed: 08/25/2023] Open
Abstract
BACKGROUND The implementation of community-based programs is key to effective, sustainable initiatives that can support population-level changes in children's physical activity. The purpose of this scoping review was to explore the implementation models and frameworks used to develop (process models), explore (determinant frameworks), and/or evaluate (evaluation frameworks) community-based physical activity programs for children. Also, the foundational components of the implementation models and frameworks and practical application in real-world settings were described. METHODS The methodological framework developed by Arksey and O'Malley (2005) and the updated recommendations from Levac, Colquhoun and O'Brien (2010) were used to search, identify, and summarize applicable studies. This review also met the requirements in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Scoping Reviews Checklist (PRISMA-ScR). A detailed search of six databases and three academic journals was conducted. Information about the article, the program, and the implementation model/framework were extracted and summarized. RESULTS The search retrieved 42,202 articles, of which 27 met the inclusion criteria. Eleven process models, one determinant framework, and two evaluation frameworks were identified. Nineteen components were developed from the models and frameworks. Tailoring, situational analysis, and element identification were common components among the identified models and frameworks. CONCLUSIONS Since the execution of interventions is vital for creating successful health-promoting initiatives, researchers and program developers should consider using implementation models and frameworks to guide their community-based physical activity programs. Further research examining the application of new and existing implementation models and frameworks in developing, exploring, and evaluating community-level programs is warranted.
Collapse
Affiliation(s)
- Emma Ostermeier
- Health and Rehabilitation Sciences, Faculty of Health Sciences, Western University, London, ON, Canada
| | - Shauna M Burke
- School of Health Studies, Western University, London, ON, Canada
- Children's Health Research Institute, London, ON, Canada
| | - Jason Gilliland
- School of Health Studies, Western University, London, ON, Canada
- Children's Health Research Institute, London, ON, Canada
- Human Environments Analysis Laboratory, Department of Geography and Environment, Western University, London, ON, Canada
- Department of Geography & Environment, Western University, London, ON, Canada
- Department of Epidemiology & Biostatistics, Western University, London, ON, Canada
- Department of Paediatrics, Western University, London, ON, Canada
- Lawson Health Research Institute, London, ON, Canada
| | - Patricia Tucker
- Children's Health Research Institute, London, ON, Canada.
- Lawson Health Research Institute, London, ON, Canada.
- School of Occupational Therapy, Faculty of Health Sciences, Elborn College, Western University, 1201 Western Rd, ON, N6G 1H1, London, Canada.
| |
Collapse
|
2
|
Rose CL, Owen KB, Foley BC, Reece LJ. "We Don't Have to Do Things the Way They've Been Done Before"; Mixed-Method Evaluation of a National Grant Program Tackling Physical Inactivity through Sport. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2022; 19:7931. [PMID: 35805601 PMCID: PMC9265654 DOI: 10.3390/ijerph19137931] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/27/2022] [Revised: 06/24/2022] [Accepted: 06/25/2022] [Indexed: 11/16/2022]
Abstract
National strategies are needed to continue to promote the broader benefits of participating in sport and organised physical activity to reduce physical inactivity and related disease burden. This paper employs the RE-AIM framework to evaluate the impact of the federally funded $150 million Move it AUS program in engaging inactive people in sport and physical activity through the Participation (all ages) and Better Ageing (over 65 years) funding streams. A pragmatic, mixed-methods evaluation was conducted to understand the impact of the grant on both the participants, and the funded organisations. This included participant surveys, case studies, and qualitative interviews with funded program leaders. A total of 75% of participants in the Participation stream, and 65% in the Better Ageing stream, were classified as inactive. The largest changes in overall physical activity behaviour were seen among socioeconomically disadvantaged participants and culturally and linguistically diverse participants. Seven key insights were gained from the qualitative interviews: Clarity of who, Partnerships, Communication, Program delivery, Environmental impacts, Governance, and that Physical inactivity must be a priority. The Move It AUS program successfully engaged physically inactive participants. Additional work is needed to better engage inactive people that identify as culturally and linguistically diverse, Aboriginal and/or Torres Strait Islander and those that live in disadvantaged communities in sport and physical activities. Tangible actions from the seven key insights should be adopted into workforce capability planning for the sport sector to effectively engage physically inactive communities.
Collapse
Affiliation(s)
- Catriona L. Rose
- SPRINTER Group, Prevention Research Collaboration, Charles Perkins Centre, Sydney School of Public Health, Faculty of Medicine and Health, The University of Sydney, Camperdown, NSW 2006, Australia; (K.B.O.); (B.C.F.); (L.J.R.)
| | | | | | | |
Collapse
|
3
|
Koorts H, Salmon PM, Swain CTV, Cassar S, Strickland D, Salmon J. A systems thinking approach to understanding youth active recreation. Int J Behav Nutr Phys Act 2022; 19:53. [PMID: 35549726 PMCID: PMC9097093 DOI: 10.1186/s12966-022-01292-2] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2021] [Accepted: 04/05/2022] [Indexed: 11/30/2022] Open
Abstract
Background Active recreation contributes to child and adolescent physical activity, however, factors affecting uptake are poorly understood at the systems level. The aims of this study were: (1) to use systems analysis methods to understand youth active recreation in Victoria, Australia, (ii) identify potential system leverage points to enhance active recreation, and (iii) explore stakeholder views of systems analysis methods for informing practice and policy decision-making. Methods Phase 1: Umbrella review of systematic reviews (2013–2018), synthesising evidence for correlates, determinants and intervention evidence for promoting active recreation. Phase 2: Development of three systems models (ActorMap and two ActivMaps), depicting active recreation actors/organisations, correlates, determinants and intervention evidence. Phase 3: Development of causal loop diagrams (CLDs) and identification of leverage points based on the Action Scales Model. Phase 4: Model feedback via stakeholder interviews (n = 23; 16 organisations). Results From the literature, 93 correlates and determinants, and 49 intervention strategies were associated with child and adolescent active recreation; the majority located at a social or individual level. Ten potential system leverage points were identified in the CLDs, which differed for pre-schoolers versus children and adolescents. Only time outdoors (an event leverage point) emerged for all age groups. Changes to the built and natural environment (i.e., land use planning, urban design) as a complete domain was a key structural leverage point for influencing active recreation in children and adolescents. Subject matter experts and stakeholder interviews identified 125 actors operating across seven hierarchical active recreation system levels in Victoria. Stakeholder interviews identified 12 areas for future consideration and recommendations for practice/policy influence. Conclusions Our findings underscore the need for dynamic models of system behaviour in active recreation, and to capture stakeholder influence as more than a transactional role in evidence generation and use. Effective responses to youth inactivity require a network of interventions that target specific leverage points across the system. Our models illustrate areas that may have the greatest system-level impact, such as changes to the built and natural environment, and they provide a tool for policy, appraisal, advocacy, and decision-making within and outside of government. Supplementary Information The online version contains supplementary material available at 10.1186/s12966-022-01292-2.
Collapse
Affiliation(s)
- Harriet Koorts
- School of Exercise and Nutrition Sciences, Institute for Physical Activity and Nutrition (IPAN), Deakin University, 221 Burwood Highway, Burwood, Geelong, VIC, 3125, Australia.
| | - Paul M Salmon
- Centre for Human Factors and Sociotechnical Systems, University of the Sunshine Coast, Sunshine Coast, Queensland, Australia
| | - Christopher T V Swain
- School of Exercise and Nutrition Sciences, Institute for Physical Activity and Nutrition (IPAN), Deakin University, 221 Burwood Highway, Burwood, Geelong, VIC, 3125, Australia.,Cancer Epidemiology Division, Cancer Council Victoria, Melbourne, Australia
| | - Samuel Cassar
- School of Exercise and Nutrition Sciences, Institute for Physical Activity and Nutrition (IPAN), Deakin University, 221 Burwood Highway, Burwood, Geelong, VIC, 3125, Australia
| | - David Strickland
- Sport and Recreation Victoria, Department of Jobs Precincts and Regions, Melbourne, Victoria, Australia
| | - Jo Salmon
- School of Exercise and Nutrition Sciences, Institute for Physical Activity and Nutrition (IPAN), Deakin University, 221 Burwood Highway, Burwood, Geelong, VIC, 3125, Australia
| |
Collapse
|
4
|
Fynn JF, Hardeman W, Milton K, Murphy J, Jones A. A systematic review of the use and reporting of evaluation frameworks within evaluations of physical activity interventions. Int J Behav Nutr Phys Act 2020; 17:107. [PMID: 32831111 PMCID: PMC7444034 DOI: 10.1186/s12966-020-01013-7] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2020] [Accepted: 08/12/2020] [Indexed: 01/06/2023] Open
Abstract
BACKGROUND Evaluation of physical activity interventions is vital to inform, and justify, evidence-based policy and practice to support population-wide changes in physical activity. Several evaluation frameworks and guidance documents have been developed to facilitate the evaluation and reporting of evaluation studies in public health. However, there is a lack of evidence about whether frameworks are being used to guide evaluation. There continues to be claims of poor and inconsistent reporting in evaluation studies. The aim of this review was to assess the use of evaluation frameworks and the quality of reporting of how they were applied within evaluation studies of physical activity interventions. OBJECTIVES 1. To identify whether evaluation frameworks are reported to have been used within evaluation studies of physical activity interventions, and which frameworks have been used. 2. To appraise the quality of reporting with regards to how evaluation frameworks have been used. METHOD We developed a checklist of indicators to enable a critical appraisal of the use and reporting of different evaluation frameworks in evaluation studies. We conducted a systematic search and review of evaluation studies published between 2015 and the date of the search to appraise the use and reporting of evaluation frameworks. A narrative synthesis is provided. RESULTS The review identified 292 evaluation studies of physical activity interventions, only 69 (23%) of these mentioned using an evaluation framework, and only 16 different frameworks were referred to. There was variation in the quality of reporting of framework use. 51 (74%) studies were identified as being explicitly based on the stated framework, however only 26 (38%) provided detailed descriptions consistently across all the checklist indicators. Details of adaptations and limitations in how frameworks were applied were less frequently reported. The review also highlighted variability in the reporting of intervention components. More consistent and precise reporting of framework and intervention components is needed. CONCLUSION Evaluation frameworks can facilitate a more systematic evaluation report and we argue their limited use suggests missed opportunities to apply frameworks to guide evaluation and reporting in evaluation studies. Variability in the quality of reporting of framework use limits the comparability and transferability of evidence. Where a framework has been used, the checklist of indicators can be employed to facilitate the reporting of an evaluation study and to review the quality of an evaluation report.
Collapse
Affiliation(s)
- Judith F Fynn
- UKCRC Centre for Diet and Activity Research (CEDAR) and Norwich Medical School, University of East Anglia, Norwich, UK.
| | - Wendy Hardeman
- School of Health Sciences, University of East Anglia, Norwich, UK
| | - Karen Milton
- Norwich Medical School, University of East Anglia, Norwich, UK
| | - Joseph Murphy
- Physical Activity for Health Research Cluster, Physical Education and Sport Sciences Department, University of Limerick, Limerick, Ireland
| | - Andy Jones
- UKCRC Centre for Diet and Activity Research (CEDAR) and Norwich Medical School, University of East Anglia, Norwich, UK
| |
Collapse
|
5
|
Lawrason S, Turnnidge J, Tomasone J, Allan V, Côté J, Dawson K, Martin LJ. Employing the RE-AIM Framework to Evaluate Multisport Service Organization Initiatives. JOURNAL OF SPORT PSYCHOLOGY IN ACTION 2020. [DOI: 10.1080/21520704.2020.1773592] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
Affiliation(s)
- Sarah Lawrason
- University of British Columbia Okanagan, Kelowna, BC, Canada
| | | | | | | | - Jean Côté
- Queen’s University, Kingston, ON, Canada
| | | | | |
Collapse
|
6
|
Pawlowski CS, Schmidt T, Nielsen JV, Troelsen J, Schipperijn J. Will the children use it?-A RE-AIM evaluation of a local public open space intervention involving children from a deprived neighbourhood. EVALUATION AND PROGRAM PLANNING 2019; 77:101706. [PMID: 31472381 DOI: 10.1016/j.evalprogplan.2019.101706] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/25/2018] [Revised: 08/22/2019] [Accepted: 08/22/2019] [Indexed: 06/10/2023]
Abstract
Knowledge on how to improve public open spaces in deprived neighbourhoods to increase active living among children is scarce and comprehensively evaluated public open space interventions are needed. Firstly, the aim was to explore if involving 39 local fifth-grade children (10-11 years old) from a deprived neighbourhood in creating playable installations in a public open space influenced their use of this space. Secondly, we wanted to explore if the Reach, Effectiveness, Adoption, Implementation and Maintenance framework (RE-AIM) was useful for evaluating different stages in the intervention project. RE-AIM was applied through a convergent mixed-methods triangulation design using survey, accelerometer, GPS, and interviews as data sources. Effectiveness outcomes revealed that on average the space was used less by the 39 children after the intervention. The implementation and maintenance dimensions revealed aspects of why most children involved in the project did not use the space after intervention. The evaluation cast light on children's perceptions of their role, and importance of maintenance when the intervention was completed. In future, all dimensions of built environmental projects would benefit from being planned and evaluated in a collaboration with all project partners using an evaluation framework integrated and applied from the beginning of the project.
Collapse
Affiliation(s)
- Charlotte Skau Pawlowski
- Research unit for Active Living, Department of Sports Science and Clinical Biomechanics, University of Southern Denmark, Campusvej 55, 5230 Odense M, Denmark.
| | - Tanja Schmidt
- Research unit for Active Living, Department of Sports Science and Clinical Biomechanics, University of Southern Denmark, Campusvej 55, 5230 Odense M, Denmark.
| | - Jonas Vestergaard Nielsen
- Research unit for Active Living, Department of Sports Science and Clinical Biomechanics, University of Southern Denmark, Campusvej 55, 5230 Odense M, Denmark.
| | - Jens Troelsen
- Research unit for Active Living, Department of Sports Science and Clinical Biomechanics, University of Southern Denmark, Campusvej 55, 5230 Odense M, Denmark.
| | - Jasper Schipperijn
- Research unit for Active Living, Department of Sports Science and Clinical Biomechanics, University of Southern Denmark, Campusvej 55, 5230 Odense M, Denmark.
| |
Collapse
|
7
|
Bird EL, Biddle MSY, Powell JE. General practice referral of 'at risk' populations to community leisure services: applying the RE-AIM framework to evaluate the impact of a community-based physical activity programme for inactive adults with long-term conditions. BMC Public Health 2019; 19:1308. [PMID: 31623584 PMCID: PMC6798368 DOI: 10.1186/s12889-019-7701-5] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2018] [Accepted: 09/27/2019] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND In the UK a high proportion of adults with long-term conditions do not engage in regular physical activity. General practice (GP) referral to community-based physical activity is one strategy that has gained traction in recent years. However, evidence for the real-world effectiveness and translation of such programmes is limited. This study aimed to evaluate the individual and organisational impacts of the 'CLICK into Activity' programme - GP referral of inactive adults living with (or at risk of) long-term conditions to community-based physical activity. METHODS A mixed methods evaluation using the RE-AIM framework was conducted with data obtained from a range of sources: follow-up questionnaires, qualitative interviews, and programme-related documentation, including programme cost data. Triangulation methods were used to analyse data, with findings synthesised across each dimension of the RE-AIM framework. RESULTS A total of 602 individuals were referred to CLICK into Activity physical activity sessions. Of those referred, 326 individuals participated in at least one session; the programme therefore reached 30.2% of the 1080 recruitment target. A range of individual-, social-, and environmental-level factors contributed to initial physical activity participation. Positive changes over time in physical activity and other outcomes assessed were observed among participants. Programme adoption at GP surgeries was successful, but the GP referral process was not consistently implemented across sites. Physical activity sessions were successfully implemented, with programme deliverers and group-based delivery identified as having an influential effect on programme outcomes. Changes to physical activity session content were made in response to participant feedback. CLICK into Activity cost £175,000 over 3 years, with an average cost per person attending at least one programme session of £535. CONCLUSIONS Despite not reaching its recruitment target, CLICK into Activity was successfully adopted. Positive outcomes were associated with participation, although low 6- and 12-month follow-up response rates limit understanding of longer-term programme effects. Contextual and individual factors, which may facilitate successful implementation with the target population, were identified. Findings highlight strategies to be explored in future development and implementation of GP referral to community-based physical activity programmes targeting inactive adults living with (or at risk of) long-term conditions.
Collapse
Affiliation(s)
- E L Bird
- Centre for Public Health and Wellbeing, University of the West of England, Frenchay Campus, Bristol, BS16 1QY, UK.
| | - M S Y Biddle
- Centre for Public Health and Wellbeing, University of the West of England, Frenchay Campus, Bristol, BS16 1QY, UK
| | - J E Powell
- Centre for Public Health and Wellbeing, University of the West of England, Frenchay Campus, Bristol, BS16 1QY, UK
| |
Collapse
|
8
|
If You Make it Free, Will They Come? Using a Physical Activity Accessibility Model to Understand the Use of a Free Children's Recreation Pass. J Phys Act Health 2019; 16:493-503. [PMID: 31138095 DOI: 10.1123/jpah.2018-0364] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2018] [Revised: 01/02/2019] [Accepted: 03/02/2019] [Indexed: 11/18/2022]
Abstract
BACKGROUND Children's sedentary lifestyles and low physical activity levels may be countered using population-level interventions. This study examines factors influencing the use of a free community-wide physical activity access pass for grade 5 students (G5AP). METHODS A natural experiment with longitudinal data collection. A sample of 881 children completed the 9-month follow-up survey self-reporting where they used the G5AP. Two analyses were conducted: Getis-Ord GI* geographic cluster analysis of the spatial distribution of users, and logistic regression examining the relationship between use and accessibility (informational, economic, and geographic) and mobility options, while accounting for intrapersonal and interpersonal factors. RESULTS Overall, 44.9% of children used the G5AP with clusters of high use in urban areas and low use in the suburbs. Other factors significantly related to G5AP included gender (girls), informational accessibility (active recruitment), economic accessibility (median household income), geographic accessibility (facilities within 1.6 km of home), and mobility options (access to Boys & Girls Club bus). CONCLUSIONS This study found that a diverse population of children used the G5AP. To continue being successful, community-based physical activity interventions need to ensure that the intervention increases geographic, economic, and informational accessibility and provides mobility options that are available to the target population.
Collapse
|
9
|
Shaw RB, Sweet SN, McBride CB, Adair WK, Martin Ginis KA. Operationalizing the reach, effectiveness, adoption, implementation, maintenance (RE-AIM) framework to evaluate the collective impact of autonomous community programs that promote health and well-being. BMC Public Health 2019; 19:803. [PMID: 31234804 PMCID: PMC6591988 DOI: 10.1186/s12889-019-7131-4] [Citation(s) in RCA: 30] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2019] [Accepted: 06/09/2019] [Indexed: 12/13/2022] Open
Abstract
Background The RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework is a useful tool for evaluating the impact of programs in community settings. RE-AIM has been applied to evaluate individual programs but seldom used to evaluate the collective impact of community-based, public health programming developed and delivered by multiple autonomous organizations. The purposes of this paper were to (a) demonstrate how RE-AIM can be operationalized and applied to evaluate the collective impact of similar autonomous programs that promote health and well-being and (b) provide preliminary data on the collective impact of Canadian spinal cord injury (SCI) peer mentorship programs on the delivery of peer mentorship services. Methods Criteria from all five RE-AIM dimensions were operationalized to evaluate multiple similar community-based programs. For this study, nine provincial organizations that serve people with SCI were recruited from across Canada. Organizations completed a structured self-report questionnaire and participated in a qualitative telephone interview to examine different elements of their peer mentorship program. Data were analyzed using summary statistics. Results Having multiple indicators to assess RE-AIM dimensions provided a broad evaluation of the impact of Canadian SCI peer mentorship programs. Peer mentorship programs reached 1.63% of the estimated Canadian SCI population. The majority (67%) of organizations tracked the effectiveness of peer mentorship through testimonials and reports. Setting-level adoption rates were high with 100% of organizations offering peer mentorship in community and hospital settings. On average, organizations allocated 10.4% of their operating budget and 9.8% of their staff to implement peer mentorship and 89% had maintained their programming for over 10 years. Full interpretation of the collective impact of peer mentorship programs was limited as complete data were only collected for 52% of survey questions. Conclusions The lack of available organizational data highlights a significant challenge when using RE-AIM to evaluate the collective impact of multiple programs that promote health and well-being. Although researchers are encouraged to use RE-AIM to evaluate the collective impact of programs delivered by different organizations, documenting limitations and providing recommendations should be done to further the understanding of how best to operationalize RE-AIM in these contexts. Electronic supplementary material The online version of this article (10.1186/s12889-019-7131-4) contains supplementary material, which is available to authorized users.
Collapse
Affiliation(s)
- Robert B Shaw
- School of Health & Exercise Sciences, University of British Columbia, 1147 Research Rd, Kelowna, British Columbia, V1V 1V7, Canada. .,International Collaboration on Repair Discoveries (ICORD), University of British Columbia, 818 West 10th Avenue, Vancouver, British Columbia, V5Z 1M9, Canada.
| | - Shane N Sweet
- Department of Kinesiology and Physical Education, McGill University, 475 Pine Ave W, Montreal, Quebec, H2W 1S4, Canada.,Center for Interdisciplinary Research in Rehabilitation of Greater Montreal, 6363 Chemin Hudson (Pavillon Lindsay) suite 061, Montreal, Quebec, H3S 1M9, Canada
| | - Christopher B McBride
- Spinal Cord Injury BC, 780 SW Marine Drive, Vancouver, British Columbia, V6P 5Y7, Canada
| | - William K Adair
- Spinal Cord Injury Canada, 520 Sutherland Drive, Toronto, Ontario, M4G 3V9, Canada
| | - Kathleen A Martin Ginis
- School of Health & Exercise Sciences, University of British Columbia, 1147 Research Rd, Kelowna, British Columbia, V1V 1V7, Canada.,International Collaboration on Repair Discoveries (ICORD), University of British Columbia, 818 West 10th Avenue, Vancouver, British Columbia, V5Z 1M9, Canada.,Department of Medicine, Division of Physical Medicine & Rehabilitation, University of British Columbia, 2329 West Mall, Vancouver, British Columbia, V6T1Z4, Canada
| |
Collapse
|
10
|
Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, Ory MG, Estabrooks PA. RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. Front Public Health 2019; 7:64. [PMID: 30984733 PMCID: PMC6450067 DOI: 10.3389/fpubh.2019.00064] [Citation(s) in RCA: 1082] [Impact Index Per Article: 180.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2019] [Accepted: 03/05/2019] [Indexed: 01/07/2023] Open
Abstract
The RE-AIM planning and evaluation framework was conceptualized two decades ago. As one of the most frequently applied implementation frameworks, RE-AIM has now been cited in over 2,800 publications. This paper describes the application and evolution of RE-AIM as well as lessons learned from its use. RE-AIM has been applied most often in public health and health behavior change research, but increasingly in more diverse content areas and within clinical, community, and corporate settings. We discuss challenges of using RE-AIM while encouraging a more pragmatic use of key dimensions rather than comprehensive applications of all elements. Current foci of RE-AIM include increasing the emphasis on cost and adaptations to programs and expanding the use of qualitative methods to understand "how" and "why" results came about. The framework will continue to evolve to focus on contextual and explanatory factors related to RE-AIM outcomes, package RE-AIM for use by non-researchers, and integrate RE-AIM with other pragmatic and reporting frameworks.
Collapse
Affiliation(s)
- Russell E. Glasgow
- Dissemination and Implementation Science Program of ACCORDS, Department of Family Medicine, School of Medicine, University of Colorado, Aurora, CO, United States,*Correspondence: Russell E. Glasgow
| | - Samantha M. Harden
- Physical Activity Research and Community Implementation, Human Nutrition, Foods, and Exercise, Virginia Tech, Blacksburg, VA, United States
| | - Bridget Gaglio
- Patient-Centered Outcomes Research Institute, Washington, DC, United States
| | - Borsika Rabin
- Dissemination and Implementation Science Program of ACCORDS, Department of Family Medicine, School of Medicine, University of Colorado, Aurora, CO, United States,Department of Family Medicine and Public Health, School of Medicine, University of California, San Diego, La Jolla, CA, United States
| | - Matthew Lee Smith
- Center for Population Health and Aging, Texas A&M University, College Station, TX, United States,Department of Environmental and Occupational Health, School of Public Health, Texas A&M University, College Station, TX, United States,Department of Health Promotion and Behavior, College of Public Health, The University of Georgia, Athens, GA, United States
| | - Gwenndolyn C. Porter
- Department of Health Promotion, College of Public Health, University of Nebraska Medical Center, Omaha, NE, United States
| | - Marcia G. Ory
- Center for Population Health and Aging, Texas A&M University, College Station, TX, United States,Department of Environmental and Occupational Health, School of Public Health, Texas A&M University, College Station, TX, United States
| | - Paul A. Estabrooks
- Department of Health Promotion, College of Public Health, University of Nebraska Medical Center, Omaha, NE, United States
| |
Collapse
|
11
|
Norris E, Dunsmuir S, Duke-Williams O, Stamatakis E, Shelton N. Mixed method evaluation of the Virtual Traveller physically active lesson intervention: An analysis using the RE-AIM framework. EVALUATION AND PROGRAM PLANNING 2018; 70:107-114. [PMID: 29499839 DOI: 10.1016/j.evalprogplan.2018.01.007] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/08/2017] [Revised: 12/20/2017] [Accepted: 01/30/2018] [Indexed: 06/08/2023]
Abstract
BACKGROUND Physically active lessons integrating movement into academic content are a way to increase children's physical activity levels. Virtual Traveller was a physically active lesson intervention set in Year 4 (aged 8-9) primary school classes in Greater London, UK. Implemented by classroom teachers, it was a six-week intervention providing 10-min physically active Virtual Field Trips three times a week. The aim of this paper is to report the process evaluation of the Virtual Traveller randomized controlled trial according to RE-AIM framework criteria (Reach, Effectiveness, Adoption, Implementation and Maintenance). METHODS A mixed methods approach to evaluation was conducted with five intervention group classes. Six sources of data were collected via informed consent logs, teacher session logs, teacher and pupil questionnaires, teacher interviews and pupil focus groups. RESULTS High participation and low attrition rates were identified (Reach) alongside positive evaluations of Virtual Traveller sessions from pupil and teachers (Effectiveness). Participants were from more deprived and ethnic backgrounds than local and national averages, with Virtual Traveller having the potential to be a free intervention (Adoption). 70% of sessions were delivered overall (Implementation) but no maintenance of the programme was evident at three month follow-up (Maintenance). CONCLUSIONS Mixed method evaluation of Virtual Traveller showed potential for it to be implemented as a low-cost physically active lesson intervention in UK primary schools.
Collapse
Affiliation(s)
- E Norris
- Centre for Behaviour Change, University College London, 1-19 Torrington Place, London, WC1E 7HB, UK; Department of Epidemiology & Public Health, University College London, 1-19 Torrington Place, London, WC1E 7HB, UK.
| | - S Dunsmuir
- Department of Clinical, Educational and Health Psychology, University College London, 26 Bedford Way, London, WC1H 0AP, UK
| | - O Duke-Williams
- Department of Information Studies, University College London, Foster Court, London, WC1E 6BT, UK
| | - E Stamatakis
- Charles Perkins Centre, Prevention Research Collaboration, School of Public Health, Sydney Medical School, University of Sydney, Australia
| | - N Shelton
- Department of Epidemiology & Public Health, University College London, 1-19 Torrington Place, London, WC1E 7HB, UK
| |
Collapse
|
12
|
Jung ME, Bourne JE, Gainforth HL. Evaluation of a community-based, family focused healthy weights initiative using the RE-AIM framework. Int J Behav Nutr Phys Act 2018; 15:13. [PMID: 29373975 PMCID: PMC5787319 DOI: 10.1186/s12966-017-0638-0] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2017] [Accepted: 12/18/2017] [Indexed: 12/12/2022] Open
Abstract
Background Childhood overweight and obesity is a major public health concern. Community-based interventions have the potential to reach caregivers and children. However, the overall health impact of these programs is rarely comprehensively assessed. This study evaluated a physical activity and healthy eating family program (Healthy Together; HT) using the RE-AIM framework. Methods Ten sites implemented the 5-week program. Thirty-nine staff members and 277 program participants (126 caregivers [Mage = 35.6] and 151 children [Mage = 13]) participated in the evaluation. Each RE-AIM dimension was assessed independently using a mixed-methods approach. Sources of data included archival records, interviews and surveys. Effectiveness outcome variables were assessed at pre- and post-intervention and 6-month follow-up. Results Reach: HT participants were almost entirely recruited from existing programs within sites. Effectiveness: Caregivers’ nutrition related efficacy beliefs increased following HT (ps < .03). Participation in HT was not associated with significant changes in physical activity or nutrition behaviour or perceived social support (ps > .05). Knowledge surrounding healthy diets and physical activity increased in children and caregivers (ps < .05). Adoption: Thirty-five percent of sites approached to implement HT expressed interest. The 10 sites selected recruited existing staff members to implement HT. Implementation: Program objectives were met 72.8% of the time and 71 adaptations were made. HT was finance- and time-dependent. Maintenance: Two sites fully implemented HT in the follow-up year and 5 sites incorporated aspects of HT into other programs. Conclusions Working alongside organizations that develop community programs to conduct comprehensive, arms-length evaluations can systematically highlight areas of success and challenges. Overall HT represents a feasible community-based intervention; however further support is required in order to ensure the program is effective at positively targeting the desired outcomes. As a result of this evaluation, modifications are currently being implemented to HT. Electronic supplementary material The online version of this article (10.1186/s12966-017-0638-0) contains supplementary material, which is available to authorized users.
Collapse
Affiliation(s)
- Mary E Jung
- School of Health and Exercise Sciences, Faculty of Health and Social Development, University of British Columbia, Kelowna, Canada.
| | - Jessica E Bourne
- Centre for Exercise, Nutrition and health Sciences, School for Policy Studies, University of Bristol, Bristol, UK
| | - Heather L Gainforth
- School of Health and Exercise Sciences, Faculty of Health and Social Development, University of British Columbia, Kelowna, Canada
| |
Collapse
|
13
|
Kumar P, Dhand A, Tabak RG, Brownson RC, Yadama GN. Adoption and sustained use of cleaner cooking fuels in rural India: a case control study protocol to understand household, network, and organizational drivers. ACTA ACUST UNITED AC 2017; 75:70. [PMID: 29255604 PMCID: PMC5729269 DOI: 10.1186/s13690-017-0244-2] [Citation(s) in RCA: 19] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2017] [Accepted: 11/05/2017] [Indexed: 01/16/2023]
Abstract
Background Implementing efficient stoves and clean fuels in low and middle-income countries are critical for improving health of poor women and children and improve the environment. Cleaner biomass stoves, however, perform poorly against the World Health Organization’s indoor air quality guidelines. This has shifted the focus to systematic dissemination and implementation of cleaner cooking systems such as liquefied petroleum gas (LPG) among poor communities. Even when there is some uptake of LPG by poor communities, its sustained use has been low. Concurrent use of LPG with traditional biomass cookstoves compromises reductions in household air pollution and limits health and environmental dividends. Therefore understanding key drivers of adoption and sustained implementation of clean fuels among the poor is critical. There is a significant gap, however, in the research to understand determinants and sustained exclusive use of clean fuels in rural poor communities. Methods/design Using a case control study design, this study will explore the impact of affordability, accessibility, and awareness on adoption and sustained use of LPG among rural poor communities of India. The study uses a multistage random sampling to collect primary data from 510 households. Case group or LPG adopters constitute 255 households while control group or non-LPG adopters constitute the remaining 255 households. The study will deploy sophisticated stove use monitoring sensors in each of the stoves in 100 case group households to monitor stove use and stacking behavior (using clean and traditional systems of cooking) of participants for 12 months. Moreover, this will be the first study to explore the impact of personal social networks striated by gender on LPG adoption. This study is guided by the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) implementation science evaluation framework. Discussion Lessons from this study will feed into a larger discussion on developing a pro-poor strategy to foster uptake and sustained use of cleaner cooking systems such as LPG. Understanding the determinants of adoption and sustained use of cleaner cooking systems through the RE-AIM framework will expand our insights on implementation of cleaner cooking systems among poor communities and will advance implementation science in the clean cooking sector. A thorough study of such implementation strategies is crucial to realize multiple UN Sustainable Development Goals on global health, climate change, and energy security.
Collapse
Affiliation(s)
- Praveen Kumar
- Boston College School of Social Work, Boston College, 125 McGuinn Hall, 140 Commonwealth Avenue, Chestnut Hill, MA 02467 USA
| | - Amar Dhand
- Department of Neurology, Harvard Medical School/Brigham and Women's Hospital, 75 Francis Street, Boston, MA 02115 USA.,Network Science Institute, Northeastern University, 177 Huntington Street, Boston, MA 02115 USA
| | - Rachel G Tabak
- Brown School, Washington University in St. Louis, Campus Box 1196, One Brookings Drive, St. Louis, MO 63130 USA
| | - Ross C Brownson
- Brown School, Washington University in St. Louis, Campus Box 1196, One Brookings Drive, St. Louis, MO 63130 USA
| | - Gautam N Yadama
- Boston College School of Social Work, McGuinn Hall 132, 140 Commonwealth Avenue, Chestnut Hill, MA 02467 USA
| |
Collapse
|
14
|
Brinkley A, McDermott H, Munir F. Team Sport in the Workplace? A RE-AIM Process Evaluation of 'Changing the Game'. AIMS Public Health 2017; 4:466-489. [PMID: 30155499 PMCID: PMC6111271 DOI: 10.3934/publichealth.2017.5.466] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2017] [Accepted: 10/25/2017] [Indexed: 11/24/2022] Open
Abstract
Background The workplace is a priority setting to promote health. Team sports can be an effective way to promote both physical and social health. This study evaluated the potential enablers and barriers for outcomes of a workplace team sports intervention programme ‘Changing the Game’ (CTG). This study was conducted in a FTSE 100 services organisation. This process evaluation was conducted using the RE-AIM framework. Methods A mixed methods approach was used. Data were collected from the participants in the intervention group prior to, during and at the end of the intervention using interviews (n = 12), a focus group (n = 5), and questionnaires (n = 17). Organisational documentation was collected, and a research diary was recorded by the lead author. The evidence collected was triangulated to examine the reach, efficacy, adoption, implementation and maintenance of the programme. Data was assessed through template analysis, and questionnaire data were analysed using multiple regression and a series of univariate ANOVAs. Results CTG improved VO2 Max, interpersonal communication, and physical activity behaviour (efficacy) over 12-weeks. This may be attributed to the supportive approach adopted within the design and delivery of the programme (implementation). Individual and organisational factors challenged the adoption and maintenance of the intervention. The recruitment and communication strategy limited the number of employees the programme could reach. Conclusion The process evaluation suggests addressing the culture within workplaces may better support the reach, adoption and maintenance of workplace team sport programmes. Future research should consider investigating and applying these findings across a range of industries and sectors.
Collapse
Affiliation(s)
- Andrew Brinkley
- School of Sport, Exercise and Health Sciences, National Centre for Sport and Exercise Medicine, Loughborough University, Loughborough, Leicestershire LE11 3TU, UK
| | - Hilary McDermott
- School of Sport, Exercise and Health Sciences, National Centre for Sport and Exercise Medicine, Loughborough University, Loughborough, Leicestershire LE11 3TU, UK
| | - Fehmidah Munir
- School of Sport, Exercise and Health Sciences, National Centre for Sport and Exercise Medicine, Loughborough University, Loughborough, Leicestershire LE11 3TU, UK
| |
Collapse
|
15
|
Torres A, Díaz MP, Hayat MJ, Lyn R, Pratt M, Salvo D, Sarmiento OL. Assessing the effect of physical activity classes in public spaces on leisure-time physical activity: "Al Ritmo de las Comunidades" A natural experiment in Bogota, Colombia. Prev Med 2017; 103S:S51-S58. [PMID: 27847217 DOI: 10.1016/j.ypmed.2016.11.005] [Citation(s) in RCA: 22] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/25/2016] [Revised: 11/10/2016] [Accepted: 11/11/2016] [Indexed: 10/20/2022]
Abstract
The Recreovia program provides free physical activity (PA) classes in public spaces in Bogota, Colombia. The purpose of this study was to assess the effectiveness of the Recreovia program in increasing PA among users of nine parks in Bogota. This study was a natural experiment conducted between 2013 and 2015 in Bogota. Community members and park users living nearby three groups of parks were compared: Group 1 were parks implementing new Recreovias (n=3), Group 2 were control parks (n=3) without Recreovias, and Group 3 were parks with existing Recreovías. Individuals in the "intervention" group were exposed to newly implemented Recreovia programs in parks near their homes. Measurements were collected at baseline and 6-8months after the intervention started. A total of 1533 participants were enrolled in the study: 501 for the existing Recreovias (included in a cross-sectional assessment) and 1032 participants (from the new Recreovias and control parks) included in the cross-sectional and pre-post study. Most participants were low income females. Twenty-three percent of the intervention group started participating in the program. Users of existing Recreovias were significantly more active and less likely to be overweight/obese compared to new Recreovia users at baseline. No changes on PA were found when comparing the intervention and control groups. Recreovias may have potential for increasing PA at the population level in urban areas given their rapid scalability, the higher levels of PA observed among program users, and its potential to reach women, low-income, less educated populations, and the overweight and obese.
Collapse
Affiliation(s)
| | | | - Matthew J Hayat
- School of Public Health, Georgia State University, Atlanta, GA, United States
| | - Rodney Lyn
- School of Public Health, Georgia State University, Atlanta, GA, United States
| | - Michael Pratt
- Institute for Public Health, Department of Family Medicine and Public Health, University of California, San Diego School of Medicine, La Jolla, CA, United States
| | - Deborah Salvo
- School of Public Health, University of Texas, Austin, TX, United States
| | | |
Collapse
|
16
|
Clark AF, Wilk P, Mitchell CA, Smith C, Archer J, Gilliland JA. Examining How Neighborhood Socioeconomic Status, Geographic Accessibility, and Informational Accessibility Influence the Uptake of a Free Population-Level Physical Activity Intervention for Children. Am J Health Promot 2017; 32:315-324. [PMID: 28677402 DOI: 10.1177/0890117117718433] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
PURPOSE To evaluate the uptake of ACT-i-Pass (G5AP), a physical activity (PA) intervention that provides free access to PA opportunities, and to understand the extent to which the intervention provides equitable access to children. DESIGN This study evaluates the differences in uptake (ie, enrollment) by comparing postal codes of registrants with the postal codes of all eligible children. SETTING Children were provided the opportunity to register for the G5AP during the 2014 to 2015 school year in London, Canada. PARTICIPANTS The population of grade 5 students in London who registered for the G5AP (n = 1484) and did not register (n = 1589). INTERVENTION The G5AP offered grade 5 students free access to select PA facilities/programs during 2014 to 2015 school year. MEASURES Measures included G5AP registration status, method of recruitment, distance between home and the nearest facility, and neighborhood socioeconomic status. ANALYSIS Getis-Ord Gi* and multilevel logistic regression were used to analyze these data. RESULTS There were significant differences in the uptake of the G5AP: residing in neighborhoods of high income (odds ratio [OR] = 1.062, P = .029) and high proportion of recent immigrants (OR = 1.036, P = .001) increased the likelihood of G5AP registration. Children who were recruited actively were significantly more likely to register for the G5AP (OR = 2.444, P < .001). CONCLUSION To increase the uptake of a PA intervention, children need to be actively recruited. Interactive presentations provide children with increased access to information about both the program and its nuances that cannot be communicated as effectively through passive methods.
Collapse
Affiliation(s)
- Andrew F Clark
- 1 Human Environments Analysis Laboratory, University of Western Ontario, London, Ontario, Canada.,2 Department of Geography, University of Western Ontario, London, Ontario, Canada
| | - Piotr Wilk
- 1 Human Environments Analysis Laboratory, University of Western Ontario, London, Ontario, Canada.,3 Department of Epidemiology & Biostatistics, University of Western Ontario, London, Ontario, Canada.,4 Department of Paediatrics, University of Western Ontario, London, Ontario, Canada
| | - Christine A Mitchell
- 1 Human Environments Analysis Laboratory, University of Western Ontario, London, Ontario, Canada.,2 Department of Geography, University of Western Ontario, London, Ontario, Canada
| | - Christine Smith
- 1 Human Environments Analysis Laboratory, University of Western Ontario, London, Ontario, Canada.,2 Department of Geography, University of Western Ontario, London, Ontario, Canada
| | - Josh Archer
- 5 Healthy Eating-Healthy Physical Activity, Child & Youth Network, London, Ontario, Canada
| | - Jason A Gilliland
- 1 Human Environments Analysis Laboratory, University of Western Ontario, London, Ontario, Canada.,2 Department of Geography, University of Western Ontario, London, Ontario, Canada.,3 Department of Epidemiology & Biostatistics, University of Western Ontario, London, Ontario, Canada.,4 Department of Paediatrics, University of Western Ontario, London, Ontario, Canada.,6 School of Health Studies, University of Western Ontario, London, Ontario, Canada
| |
Collapse
|
17
|
Arora PG, Connors EH, Blizzard A, Coble K, Gloff N, Pruitt D. Dissemination and implementation science in program evaluation: A telemental health clinical consultation case example. EVALUATION AND PROGRAM PLANNING 2017; 60:56-63. [PMID: 27685163 DOI: 10.1016/j.evalprogplan.2016.09.003] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/27/2015] [Revised: 09/16/2016] [Accepted: 09/19/2016] [Indexed: 06/06/2023]
Abstract
Increased attention has been placed on evaluating the extent to which clinical programs that support the behavioral health needs of youth have effective processes and result in improved patient outcomes. Several theoretical frameworks from dissemination and implementation (D&I) science have been put forth to guide the evaluation of behavioral health program implemented in the context of real-world settings. Although a strong rationale for the integration of D&I science in program evaluation exists, few examples exist available to guide the evaluator in integrating D&I science in the planning and execution of evaluation activities. This paper seeks to inform program evaluation efforts by outlining two D&I frameworks and describing their integration in program evaluation design. Specifically, this paper seeks to support evaluation efforts by illustrating the use of these frameworks via a case example of a telemental health consultation program in pediatric primary care designed to improve access to behavioral health care for children and adolescents in rural settings. Lessons learned from this effort, as well as recommendations regarding the future evaluation of programs using D&I science to support behavioral health care in community-based settings are discussed.
Collapse
Affiliation(s)
- Prerna G Arora
- Pace University, Department of Psychology, 41 Park Row, 13th Floor, New York, NY, 10038, United States; University of Maryland, School of Medicine, 737 W Lombard St, Baltimore, MD 21201, United States.
| | - Elizabeth H Connors
- University of Maryland, School of Medicine, 737 W Lombard St, Baltimore, MD 21201, United States.
| | - Angela Blizzard
- Florida International University, Department of Psychology, 11200 S.W. 8th Street, Miama, FL 33199, United States.
| | - Kelly Coble
- University of Maryland, School of Medicine, 737 W Lombard St, Baltimore, MD 21201, United States.
| | - Nicole Gloff
- University of Maryland, School of Medicine, 737 W Lombard St, Baltimore, MD 21201, United States.
| | - David Pruitt
- University of Maryland, School of Medicine, 737 W Lombard St, Baltimore, MD 21201, United States.
| |
Collapse
|
18
|
Schwingel A, Gálvez P, Linares D, Sebastião E. Using a Mixed-Methods RE-AIM Framework to Evaluate Community Health Programs for Older Latinas. J Aging Health 2016; 29:551-593. [PMID: 27079919 DOI: 10.1177/0898264316641075] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/31/2022]
Abstract
OBJECTIVE This study used the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework to evaluate a promotora-led community health program designed for Latinas ages 50 and older that sought to improve physical activity, nutrition, and stress management. METHOD A mixed-methods evaluation approach was administered at participant and organizational levels with a focus on the efficacy, adoption, implementation, and maintenance components of the RE-AIM theoretical model. RESULTS The program was shown to be effective at improving participants' eating behaviors, increasing their physical activity levels, and lowering their depressive symptoms. Promotoras felt motivated and sufficiently prepared to deliver the program. Some implementation challenges were reported. More child care opportunities and an increased focus on mental well-being were suggested. DISCUSSION The promotora delivery model has promise for program sustainability with both promotoras and participants alike expressing interest in leading future programs.
Collapse
|