A scoping review of evaluation frameworks and their applicability to real-world physical activity and dietary change programme evaluation

Thirty Years of SNAP-Ed: The Transition of the Nation's Largest Nutrition Education Program Into a Pillar of the Public Health Infrastructure

This paper describes the 30-year evolution of Supplemental Nutrition Assistance Program-Education (SNAP-Ed) to provide evidence to support our perspective that SNAP-Ed has earned its position as a pillar of the public health infrastructure in the US. Legislatively designated as a nutrition education and obesity prevention program, its focus is the nearly 90 million Americans with limited income. This audience experiences ongoing health disparities and is disproportionately affected by public health crises. The SNAP-Ed program works to reduce nutrition-related health disparities at all levels of the Social-Ecological Model, follows a robust evaluation framework, and leverages strong partnerships between state-based practitioners, state agencies, and the US Department of Agriculture. The expansion of SNAP-Ed would enable the program to reach more Americans so that our nation can end hunger and reduce diet-related health disparities.

Adapting the Donabedian model in undergraduate nursing education: a modified Delphi study

BACKGROUND

Donabedian conceptual and multidimensional framework has been used in several studies in an educational context. In this study, we aimed to adapt the Donabedian three-component model (structure, process, and outcome) in undergraduate nursing education. This conceptual framework provides a comprehensive image of nursing education institutions and can help to evaluate institutions by measuring different aspects of their performance. A comprehensive understanding of the various elements of an educational institution helps to develop a complete, appropriate relevant set of performance indicators.

METHODS

This was a modified Delphi study. It had three rounds. The expert panel consisted of nursing faculty members and nursing Ph.D. students. In the first round, a questionnaire was designed based on interviews, focus groups, and a literature review. Experts rated their agreement with each element on a 5-point Likert scale in rounds two and three. The consensus level was set as 75%. The stability between rounds was also determined by calculating kappa coefficients. One Sample T-Test was also calculated for new items in round three.

RESULTS

All 55 items of the questionnaire were confirmed in the second round based on the consensus percentage of 75. Five new items were added to the third round based on comments in round two. Eventually, all elements except one were confirmed according to the consensus level, kappa values, means, and One-Sample T-Test in round three. The structure's key elements include staff (academic and non-academic); equipment; guidelines; resources and facilities; and students' demographics and characteristics. Process key elements include communication; education; evaluation; cooperation; and consultation. Outcome key elements include knowledge development; nursing image; alumni's outcome; students' outcome; related medical centers' performance; accreditation and evaluation results; and satisfaction.

CONCLUSIONS

Different elements of a nursing education institution at the bachelor's level were determined. The results of this study can help related bodies to develop and implement a comprehensive and systematic evaluation. These results can also be a basis for making this model useful in other nursing courses or education in other fields.

The Collaborative Service Design Playbook to plan, design, and implement sustainable health services for impact

This paper sets out the Collaborative Service Design Playbook, to guide planning, design, and implementation of co-created health services. Successful health service development and implementation is best guided by theoretically informed approaches; however, organisations often lack design and implementation know-how and have difficulty applying it. This study seeks to improve health service design and potential for scale-up by proposing a tool to guide an end-to-end process, drawing together service design, co-design, and implementation science; and exploring the tool's feasibility to establish a sustainable service solution developed with participants and experts that is scalable and sustainable. The Collaborative Service Design Playbook phases include, (1) Define the opportunity and initiatives, (2) Design the concept and prototype, (3) Deliver to scale and evaluate; and (4) Optimise to transform and sustain. This paper has implications for health marketing through providing an end-to-end approach with phased guidance for health service development, implementation, and scale up.

An assessment of the move your way program among hispanic adults in Las Vegas, Nevada

BACKGROUND

Physical activity (PA) is important because of its associated health benefits. However, many Americans remain inactive. The 2018 guidelines recognize that PA bouts of less than 10 min are beneficial. The U.S. Department of Health and Human Services' Office of Disease Prevention and Health Promotion created the Move Your Way® (MYW) campaign to provide PA resources for communities and healthcare providers. This study aims to describe the reach of the MYW campaign, and assess whether having seen, heard, or read (SHR) about MYW, or having seen the MYW logo, was associated with PA knowledge, self-efficacy, and behavior among Hispanics residing in the Las Vegas, Nevada area.

METHODS

Hispanics, aged 18-74, were surveyed at community events (n = 481) or via an online survey (n = 123). Respondents were asked if they had SHR of MYW, were aware of 2018 PA recommendations, and to report their personal PA behavior, PA knowledge, PA intentions, and demographics.

RESULTS

Respondents (71% female) completed a 38-42-item survey. Approximately (12.4%) had SHR of the MYW campaign and 16.1% had seen the MYW logo. Only 3.4% and 15% identified, "150 minutes per week" and "75 minutes per week" as the 2018 guidelines for moderate physical activity (MPA) and vigorous physical activity (VPA), respectively. Those who had SHR of the MYW campaign were more likely to correctly identify 150 min per week as the MPA guideline and had greater confidence in overcoming selected PA barriers. Regression analysis showed that those who had seen the MYW logo were more likely to meet strength guidelines, and that those who had SHR of MYW or saw the logo were more likely to know the recommended minutes of MPA.

CONCLUSIONS

Knowledge of the MYW program positively associates with some PA behaviors and PA correlates.

Multicomponent interventions to improve routine childhood vaccine uptake in low and middle-income countries: a scoping review protocol

INTRODUCTION

Vaccine coverage remains inequitable globally. Many systematic reviews have looked at the effectiveness of strategies to improve vaccine uptake; however, these reviews frequently lack data from low and middle-income countries (LMICs), where evidence of cost-effective strategies is most valuable. This is partly because reviews often exclude non-randomised, observational or unpublished evaluations that are common in LMICs. Many reviews also exclude multicomponent interventions due to challenges isolating the effect of each component. A comprehensive mapping of multicomponent interventions implemented in LMICs would increase the visibility of studies excluded from systematic reviews and improve comparability of future evaluations by providing guidance for researchers on evaluation frameworks. This scoping review aims to identify, compare and summarise the properties and evaluation methods of multicomponent interventions to improve uptake of routine childhood vaccines in LMICs, and to assess the strengths and limitations of evaluation frameworks applied.

METHODS AND ANALYSIS

This review will be conducted using the Joanna Briggs Institute methodology for scoping reviews and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses Extension for Scoping Reviews guidelines. We will search the following databases: MEDLINE, Embase, PubMed, Cochrane, Eldis and Global Health (CAB Direct), Global Index Medicus, 3ie Portal, Google Scholar, COnnecting REpositories, and reference lists. One author will screen titles and abstracts and extract data from included articles using a pretested data extraction template. Uncertainties will be resolved through discussion with another author. Only studies published in English will be included for full review. We will assess the practicability, applicability, sensitivity and specificity of the evaluation frameworks used and present results using descriptive statistics, summary tables and charts.

ETHICS AND DISSEMINATION

Ethics approval is not required. The review will be submitted as part of a doctoral thesis, presented at conferences and published in peer-reviewed journals.

STUDY REGISTRATION

https://osf.io/7r84g.

Health action process approach: promoting physical activity, and fruit and vegetable intake among Australian adults

Based on the health action process approach (HAPA) this study examined whether changes in social cognition constructs could predict change in physical activity and fruit and vegetable intake for adult participants in My health for life, an Australian health promotion behaviour change program. Variance-based structural equation modelling was used to analyse data obtained from Australian adult program participants (n = 167) at baseline (T1), week 14 (T2), week 26 (T2), and 6-month post-program (T4). Change scores were calculated for the social cognition constructs and behaviour. Changes in action self-efficacy and outcome expectancies positively predicted changes in intentions. Action self-efficacy changes also predicted changes in maintenance self-efficacy which, in turn, mediated the effect of action self-efficacy on recovery self-efficacy and planning. Planning was predicted by changes in intentions and maintenance self-efficacy. Findings support the use of the HAPA model in designing complex health behaviour change interventions to achieve sustained behaviour change.

Good practice recommendations on implementation evaluation for policies targeting diet, physical activity, and sedentary behaviour

Health policies aim to achieve specific health goals through system-level changes, unlike common health interventions that focus on promoting specific health behaviors on individual level. However, reliable data on the feasibility and implementation of policy actions across Europe are lacking. Moreover, no practice-oriented guidance exists for policy makers and implementers on how to evaluate policy implementation.As part of the Policy Evaluation Network, we aimed to synthesise knowledge on how to evaluate the implementation of policies promoting healthy diets, physical activity, and reducing sedentary behaviours. The multidisciplinary working group comprised 16 researchers and conducted two scoping reviews, three systematic reviews, two meta-reviews, two qualitative case studies and one quantitative case study over three years. The target populations included the general population, those at risk for obesity, and school children. Based on these reviews and case studies, this article summarises and presents the findings and lessons learned regarding the implementation evaluation of policies in nine case reports.Drawing on these experiences, three critical requirements for policy implementation evaluation were set: 1) conduct a comprehensive policy implementation evaluation from a multi-level perspective, 2) use implementation frameworks to address processes, determinants, and outcomes, and 3) engage relevant stakeholders in policy implementation evaluation. Finally, the consensus process resulted in 10 steps for the implementation evaluation of policies to promote physical activity and a healthy diet and to reduce sedentary behaviours, which adhere to the requirements and resources of the targeted policy.The findings of an implementation evaluation can lead to a better understanding of why policies work or not and can serve as a basis for developing solutions. This practice-oriented guidance outlines factors that should be considered in policy implementation evaluation to address its complexity. In this way, involved researchers and practitioners are empowered to engage in the evaluation process to close the knowledge gap regarding policy implementation.

Concept and study protocol of the process evaluation of a pragmatic randomized controlled trial to promote physical activity in outpatients with heterogeneous mental disorders-the ImPuls study

BACKGROUND

Evidence suggests that patients suffering from different mental disorders benefit from exercise programs combined with behavior change techniques. Based on this evidence, we have developed an exercise program (ImPuls) specifically designed to provide an additional treatment option in the outpatient mental health care system. The implementation of such complex programs into the outpatient context requires research that goes beyond the evaluation of effectiveness, and includes process evaluation. So far, process evaluation related to exercise interventions has rarely been conducted. As part of a current pragmatic randomized controlled trial evaluating ImPuls treatment effects, we are therefore carrying out comprehensive process evaluation according to the Medical Research Council (MRC) framework. The central aim of our process evaluation is to support the findings of the ongoing randomized controlled trial.

METHODS

The process evaluation follows a mixed-methods approach. We collect quantitative data via online-questionnaires from patients, exercise therapists, referring healthcare professionals and managers of outpatient rehabilitative and medical care facilities before, during, and after the intervention. In addition, documentation data as well as data from the ImPuls smartphone application are collected. Quantitative data is complemented by qualitative interviews with exercise therapists as well as a focus-group interview with managers. Treatment fidelity will be assessed through the rating of video-recorded sessions. Quantitative data analysis includes descriptive as well as mediation and moderation analyses. Qualitative data will be analyzed via qualitative content analysis.

DISCUSSION

The results of our process evaluation will complement the evaluation of effectiveness and cost-effectiveness and will, for example, provide important information about mechanisms of impact, structural prerequisites, or provider qualification that may support the decision-making process of health policy stakeholders. It might contribute to paving the way for exercise programs like ImPuls to be made successively available for patients with heterogeneous mental disorders in the German outpatient mental health care system.

TRIAL REGISTRATION

The parent clinical study was registered in the German Clinical Trials Register (ID: DRKS00024152, registered 05/02/2021, https://drks.de/search/en/trial/DRKS00024152 ).

First Analysis of Nationwide Trends in the Use of the SNAP-Ed Evaluation Framework

OBJECTIVE

To understand Supplemental Nutrition Assistance Program-Education (SNAP-Ed) Implementing Agencies'(SIAs) use of the SNAP-Ed Evaluation Framework (Framework), which is a tool that includes 51 indicators that SNAP-Ed programs can use to measure the success of their programs in the first 5 years after its release.

METHODS

A repeated cross-sectional study design was utilized to administer electronic surveys to between 124 and 154 SIAs who received SNAP-Ed funding in fiscal years 2017, 2019, and 2021. Analyses included descriptive statistics and tests of proportions.

RESULTS

Most SIAs indicated that they used the Framework to inform both data collection instruments and program planning decisions and the rates remained relatively constant over the 3 time points (> 80%). The most common specific use of the Framework across all 3 time points was to define, count, or measure the work accomplished, but this statistically decreased from 2017 (76%) to 2021 (57%) (z-score = 3.31; P < 0.001).

CONCLUSIONS AND IMPLICATIONS

The results of this analysis confirmed that 5 years after its introduction, uptake and use of the Framework was high and that, as a whole, SIAs focused on priority indicators set by the US Department of Agriculture, with no notable increases in addressing and measuring longer-term, multisector, and population-wide outcomes. The systematic study of the Framework's usability over time has a broader application to other national health promotion initiatives with shared frameworks.

Approaches to enabling rapid evaluation of innovations in health and social care: a scoping review of evidence from high-income countries

OBJECTIVE

The COVID-19 pandemic increased the demand for rapid evaluation of innovation in health and social care. Assessment of rapid methodologies is lacking although challenges in ensuring rigour and effective use of resources are known. We mapped reports of rapid evaluations of health and social care innovations, categorised different approaches to rapid evaluation, explored comparative benefits of rapid evaluation, and identified knowledge gaps.

DESIGN

Scoping review.

DATA SOURCES

MEDLINE, EMBASE and Health Management Information Consortium (HMIC) databases were searched through 13 September 2022.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

We included publications reporting primary research or methods for rapid evaluation of interventions or services in health and social care in high-income countries.

DATA EXTRACTION AND SYNTHESIS

Two reviewers developed and piloted a data extraction form. One reviewer extracted data, a second reviewer checked 10% of the studies; disagreements and uncertainty were resolved through consensus. We used narrative synthesis to map different approaches to conducting rapid evaluation.

RESULTS

We identified 16 759 records and included 162 which met inclusion criteria.We identified four main approaches for rapid evaluation: (1) Using methodology designed specifically for rapid evaluation; (2) Increasing rapidity by doing less or using less time-intensive methodology; (3) Using alternative technologies and/or data to increase speed of existing evaluation method; (4) Adapting part of non-rapid evaluation.The COVID-19 pandemic resulted in an increase in publications and some limited changes in identified methods. We found little research comparing rapid and non-rapid evaluation.

CONCLUSIONS

We found a lack of clarity about what 'rapid evaluation' means but identified some useful preliminary categories. There is a need for clarity and consistency about what constitutes rapid evaluation; consistent terminology in reporting evaluations as rapid; development of specific methodologies for making evaluation more rapid; and assessment of advantages and disadvantages of rapid methodology in terms of rigour, cost and impact.

Evaluation Methods in Community Microgrant Programs for Health Promotion: A Scoping Review

CONTEXT

Evaluating community-based health promotion programs is an essential aspect of public health programming to ensure evidence-based practices, justify limited resource allocation, and share knowledge about promising strategies. Community microgrant schemes represent a viable strategy to strengthen local health promotion, yet limited guidance exists surrounding the evaluation of such programs.

OBJECTIVE

This research aimed to map methods used in evaluating community microgrants programs for health promotion.

DESIGN

A scoping review in accordance with Joanna Briggs Institute methodology.

ELIGIBILITY CRITERIA

Peer-reviewed academic and gray literature, which evaluated processes and/or outcomes of nonprescriptive community-based microgrant program for health promotion in high-income countries.

STUDY SELECTION

Literature was identified through MEDLINE, Scopus, CINAHL, and EMBASE databases, Google and DuckDuckGo browsers, and hand-searching key reference lists. Articles were screened by title, abstract, and then full-text.

MAIN OUTCOME MEASURES

Review findings were mapped according to included sources' evaluation methodologies.

RESULTS

Eighteen academic publications describing 14 microgrant programs met the inclusion criteria. These spanned the years 1996 to 2020 and were located across 5 countries. Evaluation methods were very similar in terms of design (mixed-methods case studies) and focus (both processes and outcomes) and yet varied widely with regard to measurements and data collection. Few made explicit the use of an evaluation framework, chose to involve stakeholders in the evaluation process, or discussed application and dissemination of evaluation findings.

DISCUSSION

Community-based microgrant programs for health promotion are a promising model in diverse settings and with various health priorities and targets, yet robust evaluations of their implementation and effectiveness are lacking. The review highlights the need to further develop and support evaluation practices in public health, even in small-scale programs with limited budgets.

The Better By Moving study: A multifaceted intervention to improve physical activity in adults during hospital stay

Objective

‘Better By Moving’ is a multifaceted intervention developed and implemented in collaboration with patients and healthcare professionals to improve physical activity in hospitalized adults. This study aimed to understand if, how and why ‘Better By Moving’ resulted in higher levels of physical activity by evaluating both outcomes and implementation process.

Design

Mixed-methods study informed by the Medical Research Council guidance.

Setting

Tertiary hospital.

Participants

Adult patients admitted to surgery, haematology, infectious diseases and cardiology wards, and healthcare professionals.

Measures

Physical activity was evaluated before and after implementation using the Physical Activity Monitor AM400 on one random day during hospital stay between 8 am and 8 pm. Furthermore, the time spent lying on bed, length of stay and discharge destination was investigated. The implementation process was evaluated using an audit trail, surveys and interviews.

Results

There was no significant difference observed in physical activity (median [IQR] 23 [12–51] vs 27 [17–55] minutes, P = 0.107) and secondary outcomes before-after implementation. The intervention components’ reach was moderate and adoption was low among patients and healthcare professionals. Patients indicated they perceived more encouragement from the environment and performed exercises more frequently, and healthcare professionals signalled increased awareness and confidence among colleagues. Support (priority, resources and involvement) was perceived a key contextual factor influencing the implementation and outcomes.

Conclusion

Although implementing ‘Better By Moving’ did not result in significant improvements in outcomes at our centre, the process evaluation yielded important insights that may improve the effectiveness of implementing multifaceted interventions aiming to improve physical activity during hospital stay.

Social, economic, political, and geographical context that counts: meta-review of implementation determinants for policies promoting healthy diet and physical activity

Background

This meta-review investigated the context-related implementation determinants from seven domains (geographical, epidemiological, sociocultural, economic, ethics-related, political, and legal) that were systematically indicated as occurring during the implementation of obesity prevention policies targeting a healthy diet and a physically active lifestyle.

Methods

Data from nine databases and documentation of nine major stakeholders were searched for the purpose of this preregistered meta-review (#CRD42019133341). Context-related determinants were considered strongly supported if they were indicated in ≥60% of the reviews/stakeholder documents. The ROBIS tool and the Methodological Quality Checklist-SP were used to assess the quality-related risk of bias.

Results

Published reviews (k = 25) and stakeholder documents that reviewed the evidence of policy implementation (k = 17) were included. Across documents, the following six determinants from three context domains received strong support: economic resources at the macro (66.7% of analyzed documents) and meso/micro levels (71.4%); sociocultural context determinants at the meso/micro level, references to knowledge/beliefs/abilities of target groups (69.0%) and implementers (73.8%); political context determinants (interrelated policies supported in 71.4% of analyzed reviews/documents; policies within organizations, 69.0%).

Conclusions

These findings indicate that sociocultural, economic, and political contexts need to be accounted for when formulating plans for the implementation of a healthy diet and physical activity/sedentary behavior policies.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12889-022-13340-4.

"The More We Know, the More We're Able to Help": Participatory Development of an Evaluation Framework for Community Health Worker Programs

CONTEXT

It is critical to evaluate community health worker (CHW) programs to maximize effectiveness. However, there is little consensus, and structure, for how to evaluate such programs.

OBJECTIVE

The purpose of this study was to develop a flexible framework for evaluating components of CHW programs in community and clinical settings.

DESIGN

Semistructured interviews were conducted with CHWs and their supervisors to determine evaluation categories for the framework. A survey was then administered to pilot the framework at federally qualified health centers and community-based organizations with existing CHW programs.

SETTING

Eighteen federally qualified health centers and community-based organizations in Louisiana, Mississippi, Alabama, and Florida.

PARTICIPANTS

Participants consisted of 23 CHWs and 19 CHW supervisors.

RESULTS

Interview participants directly informed the development of an evaluation framework for CHW programs. The framework consists of 7 evaluation categories: client relations, intraorganizational relations, interorganizational relations, capacity development, program effectiveness, cost-efficiency, and sustainability. Survey respondents specified best practices for program implementation for each evaluation category. Recommendations for CHW program evaluation include tailoring evaluation efforts and data collection methods to program context, using mixed-methods approaches for collecting evaluation data, and streamlining evaluation efforts with an organization's existing evaluation systems.

CONCLUSIONS

The evaluation framework is a flexible and practical model for collecting information needed for monitoring and evaluating CHW programs. By relying on practitioners' perspectives, this study contributes to an evidence base for implementing and evaluating CHW programs.

Evaluation of a systems ergonomics-based incident reporting system

INTRODUCTION

Incident reporting and learning systems are a fundamental component of safety management. The purpose of this study was to evaluate a novel incident reporting system specific to the Australian Led Outdoor Activity (LOA) sector. The Understanding and Preventing Led Outdoor Accidents Data System (UPLOADS), is a recently introduced systems thinking-based incident reporting and learning system that utilises contemporary safety theory and methods.

METHOD

The implementation of UPLOADS was evaluated using the RE-AIM framework based on the following dimensions: Reach, Efficacy, Adoption, Implementation, and Maintenance. A pragmatic evaluation approach was used in which evaluation data were collected through the triangulation of multiple sources including different LOA stakeholders from both the individual organisational level (LOA providers) and LOA sector governance level (LOA industry representatives), incident data collected through the UPLOADS National Incident Dataset, and the online and physical presence of UPLOADS.

RESULTS

The findings show that a key strength of UPLOADS is its effectiveness as incident reporting tool for improving safety in the LOA sector. However, a weakness of UPLOADS is that it is not being implemented appropriately by the LOA providers.

CONCLUSION

Overall, the current findings suggest that UPLOADS incident reporting tool is perceived by SMEs as an effective tool for improving safety in LOA. However, further work is required for UPLOADS to have a greater impact on the LOA sector. Specifically, Implementation of the UPLOADS system requires improvement, as well as additional training and education may be required to upskill and empower LOA providers to improve reporting and enhance the value placed on safety by LOA stakeholders. The RE-AIM framework was an appropriate evaluation framework for understanding the effectiveness of UPLOADS as a LOA sector specific incident reporting and learning system.

PRACTICAL APPLICATIONS

The current findings have practical implications for ergonomics researchers applying evaluation frameworks in the real world, and LOA providers for implementing safety interventions. Lastly, contemporary systems-based incident reporting and learning systems have the capability to enhance the safety practices of the LOA sector.

Evaluation of Digital Interventions for Physical Activity Promotion: Protocol for a Scoping Review

Background

Digital interventions (DIs) could support physical activity (PA) promotion, according to recent reviews. However, it remains unclear if and how DIs for PA promotion are evaluated; thus, it is unclear if they support behavior change in real-world settings. A mapping of evidence from published reviews is required to focus on the evaluation of DIs for PA promotion.

Objective

The aim of our study is to investigate evaluation strategies for any outcome in the context of DIs for PA promotion by conducting a scoping review of published reviews.

Methods

Our scoping review adheres to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. The information sources include bibliographic databases (MEDLINE, PsycINFO, and CINAHL) and the bibliographies of the selected studies. The electronic search strategy was developed and conducted in collaboration with an experienced database specialist. The electronic search was conducted in English with no limits up to March 19, 2021, for sources with the terms digital intervention AND evaluation AND physical activity in titles or abstracts. After deduplication, 300 reviews selected from 4912 search results were assessed for eligibility by 2 authors working independently. The inclusion criteria were (1) healthy or clinical samples (population), (2) DIs for PA promotion (intervention), (3) comparisons to any other intervention or no intervention (comparison), (4) evaluation strategies (methods, results, or frameworks) for any outcome in the context of DIs for PA promotion (outcome), and (5) any published review (study type). According to the consensus reached during a discussion, 40 reviews met the inclusion criteria—36 from the electronic search and 4 from the manual search of the bibliographies of the 36 reviews. All reviews reported the evaluation strategies for any outcomes in the context of DIs for PA promotion in healthy or clinical samples. Data coding and the quality appraisal of systematic reviews are currently being performed independently by 2 authors.

Results

Our scoping review includes data from 40 published reviews (1 rapid review, 9 scoping reviews, and 30 systematic reviews). The focus of data coding is on evaluation strategies in the context of DIs for PA promotion and on the critical appraisal of the included systematic reviews. The final consensus regarding all data is expected in early 2022.

Conclusions

Interventions for PA promotion that are supported by digital technologies require evaluation to ensure their efficacy in real-world settings. Our scoping review is needed because it addresses novel objectives that focus on such evaluations and are not answered in the published reviews identified in our search. The evaluation strategies addressing DIs for PA promotion will be mapped to synthesize the results that have been reported in published reviews so far.

International Registered Report Identifier (IRRID)

DERR1-10.2196/35332

Frameworks for implementation of policies promoting healthy nutrition and physically active lifestyle: systematic review

Background

Policy frameworks focusing on policy implementation may vary in terms of their scope, included constructs, relationships between the constructs, and context factors. Although multiple policy implementation frameworks exist, the overarching synthesis characterizing differences between the frameworks is missing. This study investigated frameworks guiding implementation of policies aiming at healthy nutrition, physical activity promotion, and a reduction of sedentary behavior. In particular, we aimed at examining the scope of the frameworks and the content of included constructs (e.g., referring to implementation processes, determinants, or implementation evaluation), the level at which these constructs operate (e.g., the individual level, the organizational/community level), relationships between the constructs, and the inclusion of equity factors.

Methods

A systematic review (the PROSPERO registration no. CRD42019133251) was conducted using 9 databases and 8 stakeholder websites. The content of 38 policy implementation frameworks was coded and analyzed.

Results

Across the frameworks, 47.4% (18 in 38) addressed three aims: description of the process, determinants, and the evaluation of implementation. The majority of frameworks (65.8%; 25 in 38) accounted for constructs from three levels: individual, organizational/community, and the system level. System-level constructs were included less often (76.3%; 29 in 38) than individual-level or organizational/community-level constructs (86.8% [33 in 38 frameworks] and 94.7% [36 in 38 frameworks] respectively). The majority of frameworks (84.2%, 32 in 38) included at least some sections that were solely of descriptive character (a list of unassociated constructs); 50.0% (19 in 38) included sections of prescriptive character (general steps of implementation); 60.5% (23 in 38) accounted for explanatory sections (assuming bi- or uni-directorial associations). The complex system approach was accounted for only in 21.1% (8 in 38) of frameworks. More than half (55.3%; 21 in 38) of frameworks did not account for any equity constructs (e.g., socioeconomic status, culture).

Conclusions

The majority of policy implementation frameworks have two or three aims (combining processes, determinants and/or the evaluation of implementation), include multi-level constructs (although the system-level determinants are less frequently included than those from the individual- or organizational/community-level), combine sections of purely descriptive character with sections accounting for prescriptive and/or explanatory associations, and are likely to include a little or no equity constructs.

Registration

PROSPERO, #CRD42019133251.

Using social networks to scale up and sustain community-based programmes to improve physical activity and diet in low-income and middle-income countries: a scoping review protocol

INTRODUCTION

The World Health Organisation endorses community-based programmes as a cost-effective, feasible and a 'best buy' in the prevention and management of non-communicable diseases (NCDs). These programmes are particularly successful when the community actively participates in its design, implementation and evaluation. However, they may be only useful insofar as they can be scaled up and sustained in some meaningful way. Social network research may serve as an important tool for determining the underlying mechanisms that contribute to this process. The aim of this planned scoping review is to map and collate literature on the role of social networks in scaling-up and sustaining community-based physical activity and diet programmes in low-income and middle-income countries.

METHODS AND ANALYSIS

This scoping review protocol has been planned around the Arksey and O'Malley framework and its enhancement. Inclusion criteria are peer-reviewed articles and grey literature exploring the role of social networks in the scale-up and/or sustainability of NCD prevention community-based programmes in adult populations. Studies must have been published since 2000, in English, and be based in a low-income or middle-income country. The following databases will be used for this review: PubMed, Cochrane, Scopus, Web of Science, CINAHL, SocIndex, the International Bibliography of the Social Sciences, Google and Google Scholar. Books, conference abstracts and research focused only on children will be excluded. Two reviewers will independently select and extract eligible studies. Included publications will be thematically analysed using the Framework Approach.

ETHICS AND DISSEMINATION

Ethical approval will not be sought for this review as no individual-level data or human participants will be involved. This protocol is registered on the Open Science Framework (https://doi.org/10.17605/OSF.IO/KG7TX). The findings from the review will be published in an accredited journal. The Preferred Reporting Items for Systematic Reviews and Meta-analyses extension for Scoping Reviews checklist will be used to support transparency and guide translation of the review.

Evaluation frameworks for digital nursing technologies: analysis, assessment, and guidance. An overview of the literature

Background

The evaluation of digital nursing technologies (DNT) plays a major role in gaining knowledge about certain aspects of a technology such as acceptance, effectiveness, or efficiency. Evaluation frameworks can help to classify the success or failure of a DNT or to further develop the technology. In general, there are many different evaluation frameworks in the literature that provide overviews of a wide variety of aspects, which makes this a highly diverse field and raises the question how to select a suitable framework. The aim of this article is to provide orientation in the field of comprehensive evaluation frameworks that can be applied to the field of DNT and to conduct a detailed analysis and assessment of these frameworks to guide field researchers.

Methods

This overview was conducted using a three-component search process to identify relevant frameworks. These components were (1) a systematized literature search in PubMed; (2) a narrative review and (3) expert consultations. Data relating to the frameworks’ evaluation areas, purpose, perspectives, and success definitions were extracted. Quality criteria were developed in an expert workshop and a strength and weakness assessment was carried out.

Results

Eighteen relevant comprehensive evaluation frameworks for DNT were identified. Nine overarching evaluation areas, seven categories of purposes, five evaluation perspectives and three categories of success definitions could be identified. Eleven quality criteria for the strengths and weaknesses of DNT-related evaluation frameworks were developed and the included frameworks were assessed against them.

Conclusion

Evaluators can use the concise information and quality criteria of this article as a starting point to select and apply appropriate DNT evaluation frameworks for their research projects or to assess the quality of an evaluation framework for DNT, as well as a basis for exploring the questions raised in this article. Future research could address gaps and weaknesses in existing evaluation frameworks, which could improve the quality of future DNT evaluations.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12912-021-00654-8.

A systematic review of the use and reporting of evaluation frameworks within evaluations of physical activity interventions

BACKGROUND

Evaluation of physical activity interventions is vital to inform, and justify, evidence-based policy and practice to support population-wide changes in physical activity. Several evaluation frameworks and guidance documents have been developed to facilitate the evaluation and reporting of evaluation studies in public health. However, there is a lack of evidence about whether frameworks are being used to guide evaluation. There continues to be claims of poor and inconsistent reporting in evaluation studies. The aim of this review was to assess the use of evaluation frameworks and the quality of reporting of how they were applied within evaluation studies of physical activity interventions.

OBJECTIVES

1. To identify whether evaluation frameworks are reported to have been used within evaluation studies of physical activity interventions, and which frameworks have been used. 2. To appraise the quality of reporting with regards to how evaluation frameworks have been used.

METHOD

We developed a checklist of indicators to enable a critical appraisal of the use and reporting of different evaluation frameworks in evaluation studies. We conducted a systematic search and review of evaluation studies published between 2015 and the date of the search to appraise the use and reporting of evaluation frameworks. A narrative synthesis is provided.

RESULTS

The review identified 292 evaluation studies of physical activity interventions, only 69 (23%) of these mentioned using an evaluation framework, and only 16 different frameworks were referred to. There was variation in the quality of reporting of framework use. 51 (74%) studies were identified as being explicitly based on the stated framework, however only 26 (38%) provided detailed descriptions consistently across all the checklist indicators. Details of adaptations and limitations in how frameworks were applied were less frequently reported. The review also highlighted variability in the reporting of intervention components. More consistent and precise reporting of framework and intervention components is needed.

CONCLUSION

Evaluation frameworks can facilitate a more systematic evaluation report and we argue their limited use suggests missed opportunities to apply frameworks to guide evaluation and reporting in evaluation studies. Variability in the quality of reporting of framework use limits the comparability and transferability of evidence. Where a framework has been used, the checklist of indicators can be employed to facilitate the reporting of an evaluation study and to review the quality of an evaluation report.