Criteria for evaluating evidence on public health interventions

Evaluation of a Community-Led Intervention in South London: How Much Standardization Is Possible?

It is widely recognized that public health interventions benefit from community engagement and leadership, yet there are challenges to evaluating complex, community-led interventions assuming hierarchies of evidence derived from laboratory experimentation and clinical trials. Particular challenges include, first, the inconsistency of the intervention across sites and, second, the absence of researcher control over the sampling frame and methodology. This report highlights these challenges as they played out in the evaluation of a community-organized health project in South London. The project aimed to benefit maternal mental health, health literacy, and social capital, and especially to engage local populations known to have reduced contact with statutory services. We evaluated the project using two studies with different designs, sampling frames, and methodologies. In one, the sampling frame and methodology were under community control, permitting a comparison of change in outcomes before and after participation in the project. In the other, the sampling frame and methodology were under researcher control, permitting a case-control design. The two evaluations led to different results, however: participants in the community-controlled study showed benefits, while participants in the researcher-controlled study did not. The principal conclusions are that while there are severe challenges to evaluating a community-led health intervention using a controlled design, the measurement of pre-/post-participation changes in well-defined health outcomes should typically be a minimum evaluation requirement, and confidence in attributing causation of any positive changes to participation can be increased by use of interventions in the project and in the engagement process itself that have a credible theoretical and empirical basis.

Communication's role in overcoming challenges for obesity prevention partnerships

With obesity rates growing worldwide calls have increased to coordinate efforts in nationwide partnership models. Yet, the necessity for professional partnerships has brought into sharper focus a problem in jointly communicating obesity and its prevention. Following a social representation framework and using episodic interviewing of members from different professions directly or indirectly involved in obesity prevention, this report critically analysed concepts and communicative approaches held by those professional areas. Key findings included different perspectives about communication orientations, namely distribution of responsibility, control and communication goals. This suggests that the profession-specific world-views prevented respondents from communicating with members from other professions. We conclude that health communication experts bring unique skills to bring together professional teams to not only develop a common language and conceptual framework but also facilitate a sense of inclusion and personal accomplishment.

RE-AIM Evaluation Plan for Washington State Innovation Models Project

The State of Washington received a State Innovation Models (SIM) $65 million award from the federal Centers for Medicare & Medicaid Services to improve population health and quality of care and reduce the growth of health care costs in the entire state, which has over 7 million residents. SIM is a "complex intervention" that implements several interacting components in a complex, decentralized health system to achieve goals, which poses challenges for evaluation. Our purpose is to present the state-level evaluation methods for Washington's SIM, a 3-year intervention (2016-2018). We apply the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) evaluation framework to structure our evaluation. We create a conceptual model and a plan to use multiple and mixed methods to study SIM performance in the RE-AIM components from a statewide, population-based perspective.

Development and content validity of the Abilitator: a self-report questionnaire on work ability and functioning aimed at the population in a weak labour market position

BACKGROUND

The unemployed have lower work ability and poorer health than the employed. This situation deteriorates when unemployment continues. The long-term unemployed often have co-morbidities and face many other challenges. This increases the need for a multidimensional assessment of work ability and functioning in different service settings. In this study, we describe the development and analyse the content validity of the Abilitator, a self-report questionnaire on work ability and functioning for those in a weak labour market position.

METHODS

The Abilitator was developed in 2014-2017. Its construct was assessed by members of academic expert panels (n = 30), practical expert panels of professionals (n = 700) and target group clients (n = 28). The structure and the content of the questionnaire was co-developed in 29 workshops and adjusted twice based on the expert panels' feedback. The Abilitator was also implemented among target group clients (n = 3360) in different services and projects. During its development the Abilitator was linked to the International Classification of Functioning, Disability and Health (ICF). The content validation process followed the guidelines recommended by the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) panel.

RESULTS

The construct of the Abilitator combines the multidimensional and biopsychosocial models of work ability and functioning. It also includes aspects of social inclusion and employability. It evaluates social, psychological, cognitive and physical functioning, and the ability to cope with everyday life. The content of these concepts was validated by the academic and practical expert panels. The Abilitator's 79 ICF codes covered 57% of the Generic, 77% of the Brief Vocational Rehabilitation, and 8% of the Minimal Environmental ICF Core Sets. When compared with the Work Ability Index (WAI) and the World Health Organization Disability Assessment Schedule (WHODAS 2.0), the direct equivalences of the ICF codes were 36 and 44%, respectively.

CONCLUSION

The Abilitator sufficiently comprehensively covers the relevant aspects to enable the assessment of the overall work ability and functioning of the population in a weak labour market position.

The smoking cessation in pregnancy incentives trial (CPIT): study protocol for a phase III randomised controlled trial

BACKGROUND

Eighty per cent of UK women have at least one baby, making pregnancy an opportunity to help women stop smoking before their health is irreparably compromised. Smoking cessation during pregnancy helps protect infants from miscarriage, still birth, low birth weight, asthma, attention deficit disorder and adult cardiovascular disease. UK national guidelines highlight lack of evidence for effectiveness of financial incentives to help pregnant smokers quit. This includes a research recommendation: within a UK context, are incentives an acceptable, effective and cost-effective way to help pregnant women who smoke to quit?

METHODS

The Cessation in Pregnancy Incentives Trial (CPIT) III is a pragmatic, 42-month, multi-centre, parallel-group, individually randomised controlled superiority trial of the effect on smoking status of adding to usual Stop Smoking Services (SSS) support, the offer of up to £400 of financial voucher incentives, compared with usual support alone, to quit smoking during pregnancy. Participants (n = 940) are pregnant smokers (age > 16 years, < 24 weeks pregnant, English speaking), who consent via telephone to take part and are willing to be followed-up in late pregnancy and 6 months after birth. The primary outcome is cotinine/anabasine-validated abstinence from smoking in late pregnancy. Secondary outcomes include engagement with SSS, quit rates at 4 weeks from agreed quit date and 6 months after birth, and birth weight. Outcomes will be analysed by intention to treat, and regression models will be used to compare treatment effects on outcomes. A meta-analysis will include data from the feasibility study in Glasgow. An economic evaluation will assess cost-effectiveness from a UK NHS perspective. Process evaluation using a case-study approach will identify opportunities to improve recruitment and learning for future implementation. Research questions include: what is the therapeutic efficacy of incentives; are incentives cost-effective; and what are the potential facilitators and barriers to implementing incentives in different parts of the UK?

DISCUSSION

This phase III trial in Scotland, England and Northern Ireland follows a successful phase II trial in Glasgow, UK. The participating sites have diverse SSS that represent most cessation services in the UK and serve demographically varied populations. If found to be acceptable and cost-effective, this trial could demonstrate that financial incentives are effective and transferable to most UK SSS for pregnant women.

TRIAL REGISTRATION

Current Controlled Trials, ISRCTN15236311. Registered on 9 October 2017.

Evaluating sustainability in the Childhood Obesity Research Demonstration project: the model and process

Background

In the context of health-related interventions, sustainability is the capacity to maintain the changes resulting from the intervention. These can be improved policies, practices or trends intended to improve population health. The Childhood Obesity Research Demonstration (CORD) project was a multi-site, multi-intervention collaboration testing the Obesity Chronic Care Model with interventions for childhood obesity prevention and management. We present the model, definitions and methodology used for the cross-site sustainability evaluation of CORD.

Methods

We applied the Ecologic Model of Obesity to childhood obesity interventions to operationalize four sustainability constructs: replicability, continuation of benefits, institutionalization, and community capacity. We used a triangulation approach and employed mixed methods to assess sustainability constructs at each level of the Ecologic Model of Obesity: Micro, Meso, Exo and Macro. We constructed checklists to count and code intervention activities, use of evidence-based practices among providers, and environmental factors and policies hypothesized to influence intervention sustainability. We developed in-depth interviews for principal investigators and project leads. We applied the Wilder Collaboration Factors Inventory with key stakeholders.

Results

Lessons learned suggested that sustainability constructs should be clearly identified and operationalized a priori. Constructs must be flexible to account for differences between intervention plans and implementation to obtain robust and informative data.

Conclusion

Strong links are needed among researchers, program implementers and communities to accomplish consistent, robust and valuable data collection efforts to assure sustainable and healthy communities.

Interventions to improve the health and wellbeing of older people living alone: a mixed-methods systematic review of effectiveness and accessibility

The global population is ageing and the likelihood of living alone increases with age. Services are necessary to help older people living alone to optimise health and wellbeing. This systematic review aimed to summarise the effectiveness and accessibility of interventions to improve the health and wellbeing of older people living alone. Relevant electronic databases (CINAHL, MEDLINE, PsycINFO and Scopus) were searched for all years up to August 2018. Studies were included if they involved older people (aged ⩾55 years) living alone, and an intervention with measured health and wellbeing outcomes. All study types were included. The Theory of Access was used to assess interventions across dimensions of accessibility, availability, acceptability, affordability, adequacy and awareness. Twenty-eight studies met the eligibility criteria; 17 studies focused on ageing safely in place and 11 on psychological and social wellbeing. Studies comprised quantitative (N = 19), qualitative (N = 4) and mixed-methods (N = 5) approaches. Dimensions from the Theory of Access were poorly addressed in the studies, particularly those of higher-quality methodology. Studies were heterogeneous, preliminary in scope and lacked consistent study design, methodology or measurement. Services that do not address user accessibility in design or evaluation may be limited in their uptake and impact. It is recommended that dimensions of access and co-creation principles be integrated into service design processes and be evaluated alongside clinical effectiveness.

Caregiver involvement in interventions for improving children's dietary intake and physical activity behaviors

BACKGROUND

Poor diet and insufficient physical activity are major risk factors for non-communicable diseases. Developing healthy diet and physical activity behaviors early in life is important as these behaviors track between childhood and adulthood. Parents and other adult caregivers have important influences on children's health behaviors, but whether their involvement in children's nutrition and physical activity interventions contributes to intervention effectiveness is not known.

OBJECTIVES

• To assess effects of caregiver involvement in interventions for improving children's dietary intake and physical activity behaviors, including those intended to prevent overweight and obesity • To describe intervention content and behavior change techniques employed, drawing from a behavior change technique taxonomy developed and advanced by Abraham, Michie, and colleagues (Abraham 2008; Michie 2011; Michie 2013; Michie 2015) • To identify content and techniques related to reported outcomes when such information was reported in included studies SEARCH METHODS: In January 2019, we searched CENTRAL, MEDLINE, Embase, 11 other databases, and three trials registers. We also searched the references lists of relevant reports and systematic reviews.

SELECTION CRITERIA

Randomised controlled trials (RCTs) and quasi-RCTs evaluating the effects of interventions to improve children's dietary intake or physical activity behavior, or both, with children aged 2 to 18 years as active participants and at least one component involving caregivers versus the same interventions but without the caregiver component(s). We excluded interventions meant as treatment or targeting children with pre-existing conditions, as well as caregiver-child units residing in orphanages and school hostel environments.

DATA COLLECTION AND ANALYSIS

We used standard methodological procedures outlined by Cochrane.

MAIN RESULTS

We included 23 trials with approximately 12,192 children in eligible intervention arms. With the exception of two studies, all were conducted in high-income countries, with more than half performed in North America. Most studies were school-based and involved the addition of healthy eating or physical education classes, or both, sometimes in tandem with other changes to the school environment. The specific intervention strategies used were not always reported completely. However, based on available reports, the behavior change techniques used most commonly in the child-only arm were "shaping knowledge," "comparison of behavior," "feedback and monitoring," and "repetition and substitution." In the child + caregiver arm, the strategies used most commonly included additional "shaping knowledge" or "feedback and monitoring" techniques, as well as "social support" and "natural consequences." We considered all trials to be at high risk of bias for at least one design factor. Seven trials did not contribute any data to analyses. The quality of reporting of intervention content varied between studies, and there was limited scope for meta-analysis. Both validated and non-validated instruments were used to measure outcomes of interest. Outcomes measured and reported differed between studies, with 16 studies contributing data to the meta-analyses. About three-quarters of studies reported their funding sources; no studies reported industry funding. We assessed the quality of evidence to be low or very low. Dietary behavior change interventions with a caregiver component versus interventions without a caregiver component Seven studies compared dietary behavior change interventions with and without a caregiver component. At the end of the intervention, we did not detect a difference between intervention arms in children's percentage of total energy intake from saturated fat (mean difference [MD] -0.42%, 95% confidence interval [CI] -1.25 to 0.41, 1 study, n = 207; low-quality evidence) or from sodium intake (MD -0.12 g/d, 95% CI -0.36 to 0.12, 1 study, n = 207; low-quality evidence). No trial in this comparison reported data for children's combined fruit and vegetable intake, sugar-sweetened beverage (SSB) intake, or physical activity levels, nor for adverse effects of interventions. Physical activity interventions with a caregiver component versus interventions without a caregiver component Six studies compared physical activity interventions with and without a caregiver component. At the end of the intervention, we did not detect a difference between intervention arms in children's total physical activity (MD 0.20 min/h, 95% CI -1.19 to 1.59, 1 study, n = 54; low-quality evidence) or moderate to vigorous physical activity (MVPA) (standard mean difference [SMD] 0.04, 95% CI -0.41 to 0.49, 2 studies, n = 80; moderate-quality evidence). No trial in this comparison reported data for percentage of children's total energy intake from saturated fat, sodium intake, fruit and vegetable intake, or SSB intake, nor for adverse effects of interventions. Combined dietary and physical activity interventions with a caregiver component versus interventions without a caregiver component Ten studies compared dietary and physical activity interventions with and without a caregiver component. At the end of the intervention, we detected a small positive impact of a caregiver component on children's SSB intake (SMD -0.28, 95% CI -0.44 to -0.12, 3 studies, n = 651; moderate-quality evidence). We did not detect a difference between intervention arms in children's percentage of total energy intake from saturated fat (MD 0.06%, 95% CI -0.67 to 0.80, 2 studies, n = 216; very low-quality evidence), sodium intake (MD 35.94 mg/d, 95% CI -322.60 to 394.47, 2 studies, n = 315; very low-quality evidence), fruit and vegetable intake (MD 0.38 servings/d, 95% CI -0.51 to 1.27, 1 study, n = 134; very low-quality evidence), total physical activity (MD 1.81 min/d, 95% CI -15.18 to 18.80, 2 studies, n = 573; low-quality evidence), or MVPA (MD -0.05 min/d, 95% CI -18.57 to 18.47, 1 study, n = 622; very low-quality evidence). One trial indicated that no adverse events were reported by study participants but did not provide data.

AUTHORS' CONCLUSIONS

Current evidence is insufficient to support the inclusion of caregiver involvement in interventions to improve children's dietary intake or physical activity behavior, or both. For most outcomes, the quality of the evidence is adversely impacted by the small number of studies with available data, limited effective sample sizes, risk of bias, and imprecision. To establish the value of caregiver involvement, additional studies measuring clinically important outcomes using valid and reliable measures, employing appropriate design and power, and following established reporting guidelines are needed, as is evidence on how such interventions might contribute to health equity.

Oral language comprehension interventions in 1-8-year-old children with language disorders or difficulties: A systematic scoping review

BACKGROUND AND AIMS

The most severe problems in language manifest as difficulties in comprehending oral language. These difficulties are persistent and expose individuals to several risk factors. There is a lack of intervention research in the area of oral language comprehension, and no reviews have focused solely on oral language comprehension interventions in young children. The aim of this review was to identify interventions targeting oral language comprehension in children 8 years or younger with language disorders or difficulties. The review also examined the possible intervention foci, efficacy, and level of evidence of these interventions.

METHODS

A systematic scoping review of eight databases was carried out. Twenty of 2399 articles met the inclusion criteria and a further six articles were identified through reference lists of sourced articles. These 26 articles described 25 studies. Altogether 2460 children aged 1-8 years participated in the 25 studies. The data from these studies were extracted and analysed, and the intervention foci, efficacy, and level of evidence were evaluated.Main contribution: The reviewed interventions focused on three aspects: modifying the communicative environment of the child; targeting aspects of the child's language; or targeting the child's language processing. Of the included studies, 80% indicated positive effects on participants' oral language comprehension. The level of evidence of the included studies varied. With few exceptions, researchers and practitioners can have moderate confidence in the results of the included studies indicating that it is possible to ameliorate difficulties in oral language comprehension.

CONCLUSIONS

This review summarises the existing evidence on oral language comprehension interventions in young children with language disorders or difficulties. The evidence base is still limited, and more research is urgently needed. The results suggest that though not all interventions seem to provide desired outcomes, there are several interventions indicating efficacy to target problems in oral language comprehension in 1-8-year-old children with language disorders or difficulties. A careful choice of therapy technique and collaboration with people in the child's environment is required to maximize outcomes.Implications: The results suggest that young children's oral language comprehension skills can be improved by guiding parents and clinicians in their communication strategies, and by clinician-implemented interventions targeting aspects of the child's language. The research on interventions targeting children's language processing is limited, and the results mixed. The present study provides information on different oral language comprehension interventions and their outcomes. The findings are readily applicable for clinical use.

Process Evaluation of Animal-Assisted Therapy: Feasibility and Relevance of a Dog-Assisted Therapy Program in Adults with Autism Spectrum Disorder

Simple Summary

An explorative study on the effects of animal-assisted therapy showed reduced stress and improved social communication in adults with autism spectrum disorder (ASD). To examine whether this intervention is suitable for a broader scope of application in clinical practice, we conducted a process evaluation alongside the effect study. The aim of this process evaluation was to gain insight into the quality of the study, the relevance and feasibility of the intervention, and the barriers and facilitators to the implementation of the intervention. Questionnaires, semi-structured interviews, and treatment reports were used to analyze process data in 27 participants who were randomized into the intervention condition and in three therapists. Both the participants and therapists evaluated the animal-assisted therapy program as satisfying, feasible, and relevant for adults with ASD. They reported the following arguments for their positive appraisal of the therapy: the therapy helped improve self-insight, joy, relaxation, and physical contact. The participants’ attitudes, personal skills for generalization, and severity of contextual problems were named as potential barriers to the feasibility of the therapy program. Females and dog owners were over-represented in this study, and therefore, generalization of the previously established effects should be made with caution. However, given the intervention’s effects and the positive evaluation of the program by the participants and therapists alike, animal-assisted therapy can be considered a valuable addition to treatment possibilities for reducing stress and improving social communication in adults with autism spectrum disorder.

Abstract

(1) Background: Randomized controlled trials (RCTs) are considered the gold standard for evaluating a treatment. However, the results of an RCT may remain meaningless for clinical practice in cases of poor intervention feasibility or fidelity (the extent to which the protocol was executed), or when health care professionals or patients experience the intervention as irrelevant or unpleasant. Feasibility and relevance of psychosocial interventions are highly understudied in adults with autism spectrum disorder (ASD). In order to put the effects revealed in an RCT on an animal-assisted therapy (AAT) program in adults with autism spectrum disorder (ASD) into the context of clinical practice and to formulate guidelines for potential improvements and further implementation of the therapy, the aim of this process evaluation was to gain insight into the relevance and feasibility of the intervention and barriers and facilitators to its implementation. (2) Methods: Data were collected from 27 participants with ASD and three therapists using questionnaires, semi-structured interviews, and treatment reports. Reach, adherence, program fidelity, and program appraisal were evaluated, and barriers and facilitators to recruitment and implementation of the AAT program were explored. (3) Results: The participants were satisfied with the program and evaluated it as feasible and relevant for adults with ASD. The participants documented improving self-insight, joy, relaxation, and physical contact with a therapy dog as the reason of their positive appraisal of the therapy. Documented aspects that may influence feasibility and appraised relevance were the participants’ therapy attitude, skills for generalization, and severity of contextual problems (e.g., problems at work, relationship problems). Regarding the sample quality, females and dog owners were slightly over-represented in the RCT. (4) Discussion: Considering the positive evaluation of the intervention and its positive effects revealed in the RCT, the AAT program can be added to the treatment repertoire to reduce stress and improve social communication in adults with ASD. More research in larger samples is needed for better understanding the generalization of the intervention effects, especially in male patients and those who do not have a dog at home.

Will the children use it?-A RE-AIM evaluation of a local public open space intervention involving children from a deprived neighbourhood

Knowledge on how to improve public open spaces in deprived neighbourhoods to increase active living among children is scarce and comprehensively evaluated public open space interventions are needed. Firstly, the aim was to explore if involving 39 local fifth-grade children (10-11 years old) from a deprived neighbourhood in creating playable installations in a public open space influenced their use of this space. Secondly, we wanted to explore if the Reach, Effectiveness, Adoption, Implementation and Maintenance framework (RE-AIM) was useful for evaluating different stages in the intervention project. RE-AIM was applied through a convergent mixed-methods triangulation design using survey, accelerometer, GPS, and interviews as data sources. Effectiveness outcomes revealed that on average the space was used less by the 39 children after the intervention. The implementation and maintenance dimensions revealed aspects of why most children involved in the project did not use the space after intervention. The evaluation cast light on children's perceptions of their role, and importance of maintenance when the intervention was completed. In future, all dimensions of built environmental projects would benefit from being planned and evaluated in a collaboration with all project partners using an evaluation framework integrated and applied from the beginning of the project.

Is evidence-informed urban health planning a myth or reality? Lessons from a qualitative assessment in three Asian cities

City governments are well-positioned to effectively address urban health challenges in the context of rapid urbanization in Asia. They require good quality and timely evidence to inform their planning decisions. In this article, we report our analyses of degree of data-informed urban health planning from three Asian cities: Dhaka, Hanoi and Pokhara. Our theoretical framework stems from conceptualizations of evidence-informed policymaking, health planning and policy analysis, and includes: (1) key actors, (2) approaches to developing and implementing urban health plans, (3) characteristics of the data itself. We collected qualitative data between August 2017 and October 2018 using: in-depth interviews with key actors, document review and observations of planning events. Framework approach guided the data analysis. Health is one of competing priorities with multiple plans being produced within each city, using combinations of top-down, bottom-up and fragmented planning approaches. Mostly data from government information systems are used, which were perceived as good quality though often omits the urban poor and migrants. Key common influences on data use include constrained resources and limitations of current planning approaches, alongside data duplication and limited co-ordination within Dhaka's pluralistic system, limited opportunities for data use in Hanoi and inadequate and incomplete data in Pokhara. City governments have the potential to act as a hub for multi-sectoral planning. Our results highlight the tensions this brings, with health receiving less attention than other sector priorities. A key emerging issue is that data on the most marginalized urban poor and migrants are largely unavailable. Feasible improvements to evidence-informed urban health planning include increasing availability and quality of data particularly on the urban poor, aligning different planning processes, introducing clearer mechanisms for data use, working within the current systemic opportunities and enhancing participation of local communities in urban health planning.

Can implementation failure or intervention failure explain the result of the 3D multimorbidity trial in general practice: mixed-methods process evaluation

OBJECTIVES

During a cluster randomised trial, (the 3D study) of an intervention enacting recommended care for people with multimorbidity, including continuity of care and comprehensive biennial reviews, we examined implementation fidelity to interpret the trial outcome and inform future implementation decisions.

DESIGN

Mixed-methods process evaluation using cross-trial data and a sample of practices, clinicians, administrators and patients. Interviews, focus groups and review observations were analysed thematically and integrated with quantitative data about implementation. Analysis was blind to trial outcomes and examined context, intervention adoption, reach and maintenance, and delivery of reviews to patients.

SETTING

Thirty-three UK general practices in three areas.

PARTICIPANTS

The trial included 1546 people with multimorbidity. 11 general practitioners, 14 nurses, 7 administrators and 38 patients from 9 of 16 intervention practices were sampled for an interview.

RESULTS

Staff loss, practice size and different administrative strategies influenced implementation fidelity. Practices with whole administrative team involvement and good alignment between the intervention and usual care generally implemented better. Fewer reviews than intended were delivered (49% of patients receiving both intended reviews, 30% partially reviewed). In completed reviews >90% of intended components were delivered, but review observations and interviews with patients and clinicians found variation in style of component delivery, from 'tick-box' to patient-centred approaches. Implementation barriers included inadequate skills training to implement patient-centred care planning, but patients reported increased patient-centredness due to comprehensive reviews, extra time and being asked about their health concerns.

CONCLUSIONS

Implementation failure contributed to lack of impact of the 3D intervention on the trial primary outcome (quality of life), but so did intervention failure since modifiable elements of intervention design were partially responsible. When a decisive distinction between implementation failure and intervention failure cannot be made, identifying potentially modifiable reasons for suboptimal implementation is important to enhance potential for impact and effectiveness of a redesigned intervention.

TRIAL REGISTRATION NUMBER

ISRCTN06180958.

Protocol for a systematic review of the use of qualitative comparative analysis for evaluative questions in public health research

BACKGROUND

There is an increasing recognition that health intervention research requires methods and approaches that can engage with the complexity of systems, interventions, and the relations between systems and interventions. One approach which shows promise to this end is qualitative comparative analysis (QCA), which examines casual complexity across a medium to large number of cases (between 10 and 60+), whilst also being able to generalise across those cases. Increasingly, QCA is being adopted in public health intervention research. However, there is a limited understanding of how it is being adopted. This systematic review will address this gap, examining how it is being used to understand complex causation; for what settings, populations and interventions; and with which datasets to describe cases.

METHODS

We will include published and peer-reviewed studies of any public health intervention where the effects on population health, health equity, or intervention uptake are being evaluated. Electronic searches of PubMed, Scopus, Web of Science (incorporating Social Sciences Citation Index and Arts & Humanities Citation Index), Microsoft Academic, and Google Scholar will be performed. This will be supplemented with reference citation tracking and personal contact with experts to identify any additional published studies. Search results will be single screened, with machine learning used to check these results, acting as a 'second screener'. Any disagreement will be resolved through discussion. Data will be extracted from full texts of eligible studies, which will be assessed against inclusion criteria, and synthesised narratively, using thematic synthesis methods.

DISCUSSION

This systematic review will provide an important map of the increasing use of QCA in public health intervention literature. This review will identify the current scope of research in this area, as well as assessing claims about the utility of the method for addressing complex causation in public health research. We will identify implications for better reporting of QCA methods in public health research and for reporting of case studies such that they can be used in future QCA studies.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO, CRD42019131910.

Evaluating case management as a complex intervention: Lessons for the future

The methodological challenges to effectiveness evaluation of complex interventions has been widely discussed. Bottom-up case management for frail older person was implemented in Belgium, and indeed, it was evaluated as a complex intervention. This paper presents the methodological approach we developed to respond to four main methodological challenges regarding the evaluation of case management: (1) the standardization of the interventions, (2) stratification of the frail older population that was used to test various modalities of case management with different risks groups, (3) the building of a control group, and (4) the use of multiple outcomes in evaluating case management. To address these challenges, we developed a mixed-methods approach that (1) used multiple embedded case studies to classify case management types according to their characteristics and implementation conditions; and (2) compared subgroups of beneficiaries with specific needs (defined by Principal Component Analysis prior to cluster analysis) and a control group receiving 'usual care', to evaluate the effectiveness of case management. The beneficiaries' subgroups were matched using propensity scores and compared using generalized pairwise comparison and the hurdle model with the control group. Our results suggest that the impact of case management on patient health and the services used varies according to specific needs and categories of case management. However, these equivocal results question our methodological approach. We suggest to reconsider the evaluation approach by moving away from a viewing case management as an intervention. Rather, it should be considered as a process of interconnected actions taking place within a complex system.

Mobile Health for First Nations Populations: Systematic Review

Background

The ubiquitous presence and functionality of mobile devices offers the potential for mobile health (mHealth) to create equitable health opportunities. While mHealth is used among First Nations populations to respond to health challenges, the characteristics, uptake, and effectiveness of these interventions are unclear.

Objective

This review aimed to identify the characteristics of mHealth interventions (eg, study locations, health topic, and modality) evaluated with First Nations populations and to summarize the outcomes reported for intervention use, user perspectives including cultural responsiveness, and clinical effectiveness. In addition, the review sought to identify the presence of First Nations expertise in the design and evaluation of mHealth interventions with First Nations populations.

Methods

The methods of this systematic review were detailed in a registered protocol with the International Prospective Register of Systematic Reviews (PROSPERO, CRD42019123276). Systematic searches of peer-reviewed, scientific papers were conducted across 7 databases in October 2018. Eligible studies had a primary focus on mHealth interventions with experimental or quasi-experimental design to respond to a health challenge with First Nations people from Canada, Australia, New Zealand, and the United States. Two authors independently screened records for eligibility and assessed risk of bias using the Joanna Briggs Institute checklists. Data were synthesized narratively owing to the mix of study designs, interventions, and outcomes. The review was reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.

Results

Searches yielded 1053 unique records, after review and screening, 13 studies (5 randomized controlled trials and 8 quasi-experimental designs) were included in the final analysis. Studies were conducted in Australia (n=9), the United States (n=2), and New Zealand (n=2). The most common health challenge addressed was mental health and suicide (n=5). Intervention modalities included text messaging (n=5), apps (n=4), multimedia messaging (n=1), tablet software (n=1), or a combination of short messaging service (SMS) and apps (n=1). Results showed mixed engagement with the intervention (n=3); favorable user perspectives, including acceptability and cultural appropriateness (n=6); and mixed outcomes for clinical effectiveness (n=10). A diverse range of risks of bias were identified, the most common of which included a lack of clarity about allocation and blinding protocols and group treatment for randomized controlled trials and a lack of control group and single outcome measures for quasi-experimental designs. First Nations expertise informed all mHealth studies, through authorship (n=8), affiliation with First Nations bodies (n=3), participatory study design (n=5), First Nations reference groups (n=5), or a combination of these.

Conclusions

mHealth modalities, including SMS and apps, appear favorable for delivery of health interventions with First Nations populations, particularly in the area of mental health and suicide prevention. Importantly, First Nations expertise was strongly embedded within the studies, augmenting favorable use and user engagement. However, evidence of efficacy is limited.

Noncomprehensive and Intermittent Obesity-Related School Programs and Policies May Not Work: Evidence from the COMPASS Study

BACKGROUND

The school environment has the potential to influence student body mass index (BMI) through programs and policies. Our objective was to examine the effect of modifying obesity-related school policies and programs on youths' BMI trajectories.

METHODS

Obesity-related school policies and programs related to physical activity and healthy eating were collected from 41 schools across Ontario at baseline (2012-2013) and year 2 (2013-2014) of the COMPASS study. Self-reported height and weight were collected from the 4951 grades 9 and 10 students who attended those schools for 3 years. Linear mixed effects regression models examined the effect of modifying obesity-related school policies and programs on youths' BMI trajectories.

RESULTS

Between Y₁ and Y₂ , 26 of the 41 schools implemented distinct new obesity-related programs or policies related to physical activity or healthy eating. Five of the interventions were associated with BMI trajectories of students attending those schools compared to students attending a pooled sample of control schools, predicting a higher BMI trajectory.

CONCLUSIONS

Isolated programs and policies may not successfully improve youths' BMI trajectories. Further research is required to test the implementation of policies and programs that follow a comprehensive school health (CSH) approach, targeting BMI and associated behaviors.

A systematic review of dyadic approaches to reminiscence and life review among older adults

Objective: This paper describes a systematic review of quantitative and qualitative studies that examine the effectiveness of dyadic approaches to life review and reminiscence. Methods: The search process began with 1,056 potentially eligible articles and, after a multi-step screening process, led to 20 full-text articles for review. Results: The resulting studies (ten quantitative, nine qualitative, and one mixed-method) comprised six different kinds of dyadic approaches. There are a combination of benefits and problems associated with dyadic life review and reminiscence approaches. Conclusions: This review provides initial understanding of the impact of life review and reminiscence when care recipients and caregivers participate in the intervention together. Future clinical studies on dyadic approaches to life review and reminiscence should provide clearer descriptions of their protocols, include more consistent measures (incorporating in-the-moment measures), and involve populations of healthy older adults and their partners as well as those with cognitive impairment. The inclusion of qualitative studies helps interpret the diverse results reported in the quantitative studies and highlights key themes related to participating in dyadic interventions.

An assessment of program evaluation methods and quality in Australian prevention agencies

ISSUE ADDRESSED

This study aimed to examine evaluation methods and quality in Australian health promotion agencies and the factors associated with this. The evidence base for prevention strategies is limited, with the evidence generated through program evaluation by health promotion and disease prevention agencies lacking rigour. Despite the need to improve the quality of evaluation, there is limited evidence of what influences evaluation quality in the prevention field.

METHODS

Data were collected using the Evaluation Practice Analysis Survey and an audit and appraisal of evaluation reports. Descriptive analysis was used to examine evaluation characteristics and multivariable regression was used to explore the association between evaluation and organisational attributes and evaluation quality.

RESULTS

In total, 392 evaluation reports were reviewed from 78 government and non-government agencies. Process evaluation was conducted most frequently, followed by impact evaluation. Overall evaluation quality was low (median 24.5%). In multivariable regression analysis, only two factors were associated with evaluation quality: health promotion budget (ratio of geometric means 1.53 [95% CI 1.02-2.29]); and, conducting statewide or national prevention programs (1.38 [95% CI 1.05-1.82]).

CONCLUSIONS

The findings show that the potential to improve evaluation quality is greatest in smaller organisations that deliver health promotion at a local or regional scale. SO WHAT?: By improving the rigour of existing evaluation, there is opportunity to build the evidence base for prevention strategies, which highlights the importance of embedding the enablers of program learning and evidence generation within health promotion and prevention organisations.

Rights to social determinants of flourishing? A paradigm for disability and public health research and policy

BACKGROUND

The term evidence based medicine was introduced in the early 1990s in clinical medicine to educate clinicians about how to assess the 'credibility' of research to ensure best treatments for their patients. The evidence based medicine paradigm has become more diffuse in times of austerity and randomised controlled designs are being used to address complex issues in public health and disability research. This research is not addressing inequalities in terms of disability nor how people can live well with disabilities.

MAIN TEXT

We argue that there are four ways that public health research needs to change if it wants to address inequalities linked to disability: 1) rethinking theoretical connections between public health and disability; 2) building ethics and equity into interventions through a human rights approach; 3) ensuring ethical inclusion through intersectionality; and 4) evaluating policy and other social impacts to ensure they capture diversity. We argue that these are key issues to building a social determinants of flourishing.

CONCLUSIONS

We need to understand how disability might have an accumulative impact across the life course, as well as how to ensure equity for people living with disabilities. This means conceptualising a social determinants of flourishing where we evaluate how exactly randomised controlled trials and public health interventions, not only lead to greater equality but also ensure rights to health and wellbeing.

Process evaluation of complex interventions tested in randomised controlled trials in musculoskeletal disorders: a systematic review protocol

INTRODUCTION

The effectiveness of complex interventions for the management of musculoskeletal disorders has been estimated in many randomised clinical trials (RCTs). These trials inform which interventions are the most effective, however they do not always inform how an intervention achieved its clinical outcomes, nor how and what elements of an intervention were delivered to patients. Such information is useful for translating findings into clinical practice. A few process evaluation studies have been conducted alongside RCTs and a variety of methods have been used. To gain a better understanding of current practices of process evaluation in RCTs in musculoskeletal disorders, this systematic review is designed to answer the following research question: How are process evaluation of complex interventions tested in RCTs in musculoskeletal disorders being conducted?

METHODS AND ANALYSIS

We will systematically search seven electronic databases (MEDLINE, SCOPUS, CINAHL, PsycINFO, EMBASE, Web of Science and Cochrane database) from the date of inception to August 2018 for studies on process evaluation of RCTs on non-surgical and non-pharmacological management of musculoskeletal disorders. We will include qualitative and quantitative studies conducted alongside RCTs, reported with the RCTs or separate studies that assessed interventions for musculoskeletal disorders. Two reviewers will screen abstracts and apply prespecified inclusion criteria to identify relevant studies, extract the data and assess the risk of bias within included studies. We will follow recommendations from the 'Cochrane Qualitative and Implementation Methods Group Guidance Series' when assessing methodological strengths and limitations of included studies. We will use a narrative synthesis to describe findings.

ETHICS AND DISSEMINATION

Ethical approval is not required as this review will not collect original data. Findings from this systematic review will be presented at a scientific conference and published in a peer reviewed journal.

PROSPERO REGISTRATION NUMBER

CRD42018109600.

Subnational health management and the advancement of health equity: a case study of Ethiopia

BACKGROUND

Health equity is a cross-cutting theme in the United Nations 2030 Agenda for Sustainable Development, and a priority in health sector planning in countries including Ethiopia. Subnational health managers in Ethiopia are uniquely positioned to advance health equity, given the coordination, planning, budgetary, and administration tasks that they are assigned. Yet, the nature of efforts to advance health equity by subnational levels of the health sector is poorly understood and rarely researched. This study assesses how subnational health managers in Ethiopia understand health equity issues and their role in promoting health equity and offers insight into how these roles can be harnessed to advance health equity.

METHODS

A descriptive case study assessed perspectives and experiences of health equity among subnational health managers at regional, zonal, district and Primary Health Care Unit administrative levels. Twelve in-depth interviews were conducted with directors, vice-directors, coordinators and technical experts. Data were analyzed using thematic analysis.

RESULTS

Subnational managers perceived geographical factors as a predominant concern in health service delivery inequities, especially when they intersected with poor infrastructure, patriarchal gender norms, unequal support from non-governmental organizations or challenging topography. Participants used ad hoc, context-specific strategies (such as resource-pooling with other sectors or groups and shaming-as-motivation) to improve health service delivery to remote populations and strengthen health system operations. Collaboration with other groups facilitated cost sharing and access to resources; however, the opportunities afforded by these collaborations, were not realized equally in all areas. Subnational health managers' efforts in promoting health equity are affected by inadequate resource availability, which restricts their ability to enact long-term and sustainable solutions.

CONCLUSIONS

Advancing health equity in Ethiopia requires: extra support to communities in hard-to-reach areas; addressing patriarchal norms; and strategic aligning of the subnational health system with non-health government sectors, community groups, and non-governmental organizations. The findings call attention to the unrealized potential of effectively coordinating governance actors and processes to better align national priorities and resources with subnational governance actions to achieve health equity, and offer potentially useful knowledge for subnational health system administrators working in conditions similar to those in our Ethiopian case study.

Workplace Interventions to Promote Sleep Health and an Alert, Healthy Workforce

STUDY OBJECTIVES

The purpose of this review is to synthesize the published literature that addresses employer-initiated interventions to improve the sleep of workers and in turn improve health, productivity, absenteeism, and other outcomes that have been associated with sleep disorders or sleep deficiency.

METHODS

We conducted a systematic search and a selective narrative review of publications in PubMed from 1966 to December 2017. We extracted study characteristics, including the workers' professions, workplace settings and shift work, and workplace interventions focused on worker sleep. Because of the high degree of heterogeneity in design and outcomes, we conducted a narrative review.

RESULTS

We identified 219 publications. After restriction to publications with studies of workplace interventions that evaluated the outcomes of sleep duration or quality, we focused on 47 articles. An additional 13 articles were accepted in the pearling process. Most studies employed non-randomized or controlled pretest and posttest designs and self-reported measures of sleep. The most common workplace interventions were educational programs stressing sleep hygiene or fatigue management. Other interventions included timed napping before or after work, urging increased daytime activity levels, modifying workplace environmental characteristics such as lighting, and screening, and referral for sleep disorders treatment. Overall, most reports indicated that employer efforts to encourage improved sleep hygiene and healthier habits result in improvements in sleep duration, sleep quality, and self-reported sleepiness complaints.

CONCLUSIONS

These studies suggest employer-sponsored efforts can improve sleep and sleep-related outcomes. The existing evidence, although weak, suggests efforts by employers to encourage better sleep habits and general fitness result in self-reported improvements in sleep-related outcomes, and may be associated with reduced absenteeism and better overall quality of life. Candidate workplace strategies to promote sleep health are provided.

Drug consumption rooms: A systematic review of evaluation methodologies

ISSUES

Drug consumptions rooms (DCR) and supervised injecting facilities (SIF) are expanding internationally. Previous reviews have not systematically addressed evaluation methodologies.

APPROACH

Results from systematic searches of scientific databases in English until June 2017 were coded for paper type, country and year of publication. For evaluation papers, study outcome, methodology/study design and main indicators of DCR/SIF 'exposure' were recorded.

KEY FINDINGS

Two hundred and nineteen eligible peer-reviewed papers were published since 1999: the majority from Canada (n = 117 papers), Europe (n = 36) and Australia (n = 32). Fifty-six papers reported evaluation outcomes. Ecological study designs (n = 10) were used to assess the impact on overdose, public nuisance and crime; modelling techniques (n = 6) estimated impact on blood-borne diseases, overdose deaths and costs. Papers using individual-level data included four prospective cohorts (n = 28), cross-sectional surveys (n = 7) and service records (n = 5). Individual-level data were used to assess safer injecting practice, uptake into health and social services and all the other above outcomes except for impact on crime and costs. Four different indicators of DCR/SIF attendance were used to measure service 'exposure'.

IMPLICATIONS

Research around DCRs/SIFs has used ecological, modelling, cross-sectional and cohort study designs. Further research could involve systematic inclusion of a control group of people who are eligible but do not access SIFs, validation of self-reported proportion of injections at SIFs or a stepped-wedge or a cluster trial comparing localities.

CONCLUSIONS

Methodologies appropriate for DCR/SIF evaluation have been established and can be readily replicated from the existing literature. Research on operational aspects, implementation and transferability is also warranted.

Applying pragmatic approaches to complex program evaluation: A case study of implementation of the New South Wales Get Healthy at Work program

ISSUE ADDRESSED

Complex health promotion programs, which can have multilevels of implementation and multi-components with nonlinear causal pathways, present many evaluation challenges. Traditional evaluation methods often fail to account for the complexity inherent in assessing these programs. In real-world settings, evaluations of complex programs are often beset by additional constraints of limited budgets and short timeframes. Determining whether a complex program is successful and how a program worked requires evaluators of complex programs to adopt a level of pragmatism.

METHODS

This paper describes a pragmatic evaluation approach used to evaluate the Get Healthy at Work workplace health promotion program, implemented in New South Wales, Australia. Using the program as a case study, we describe some key principles for applying a pragmatic evaluation approach and use these principles to develop an appropriate evaluation strategy.

RESULTS

The evaluation includes multiple research methods to assess program outputs and implementation; and identify emergent program impacts, within constrained resources. The evaluation was guided by epistemological flexibility, methodological comprehensiveness and operational practicality.

CONCLUSION

Health promotion programs, such as state-wide obesity prevention programs, require appropriate evaluation methods which address their inherent complexity amidst the real-world evaluation constraints, and focuses on the essential evaluation needs.

SO WHAT

The main complex program evaluation principles are applicable to other multilevel health promotion programs, challenged by methodological and practical or political constraints.

Effectiveness of a skin care programme for the prevention of contact dermatitis in healthcare workers (the Healthy Hands Project): A single-centre, cluster randomized controlled trial

BACKGROUND

Healthcare workers (HCWs) are at risk of developing hand dermatitis (HD). Guidelines recommend moisturizers to prevent HD, but in practice their effectiveness has been poorly investigated.

OBJECTIVES

To assess whether an intervention aimed at improving skin care leads to a reduction in HD severity.

METHODS

In this 1-year randomized controlled trial, 9 wards (285 HCWs) were allocated to an intervention group (IG), and 10 wards (216 HCWs) were allocated to the control group (CG). The intervention included provision of cream dispensers with electronic monitoring of use, regularly communicated to the HCWs. The primary and secondary outcomes were change from baseline in Hand Eczema Severity Index (HECSI) score (ΔHECSI) and change in natural moisturizing factor (NMF) level (ΔNMF).

RESULTS

At 12 months, the rates of loss to follow-up were 41% and 39% in the IG and the CG, respectively. The HECSI score was reduced in the IG by -6.2 points (95%CI: -7.7 to -4.7) and in the CG by -4.2 points (95%CI: -6.0 to -2.4). There was no significant difference in ΔHECSI or ΔNMF between the groups. Relative improvement in the HECSI score was significantly higher in the IG than in the CG (56% vs 44%). In a subgroup of HCWs with mild HD, the IG showed a larger HECSI score decrease than the CG (P < 0.001).

CONCLUSION

Although there was no significant effect on the primary outcomes, the intervention showed overall positive effects on the HECSI score.

Planning for Community Scale-Up of Project HEAL: Insights From the SPRINT Initiative

Project HEAL (Health through Early Awareness and Learning) is an evidence-based intervention rooted in health behavior change theory and aims to increase cancer awareness and early detection through African American faith-based organizations. This study explored the potential for broader scale-up and dissemination of Project HEAL with the team's participation in a training program called Speeding Research-Tested INTerventions (SPRINT). The SPRINT training was framed using tools from the Business Model Canvas and the Value Proposition Canvas to guide trainees in designing (1) compelling value propositions, (2) a minimal viable product, and (3) questions to gain critical insight from various stakeholders during a process called Customer Discovery. We report on our experiences and insights on intervention scale-up that we gained from the training, including key findings from 41 discovery interviews conducted with various stakeholders of the church. We learned several valuable lessons from the discovery interviews such as scale-up will likely be more incremental than immediate. Additional refinement is needed to scale up the intervention for "real-world" application, such as making our technology more user-friendly and including additional health topics beyond cancer. We discuss how insights from the training refined our plans for future scale-up and dissemination in a constituent-informed way.

Systematic mapping of checklists for assessing transferability

BACKGROUND

Systematic reviews of research evidence have become an expected basis for decisions about practice guidelines and policy decisions in the health and welfare sectors. Review authors define inclusion criteria to help them determine which studies to search for and include in their reviews. However, these studies may still vary in the extent to which they reflect the context of interest in the review question. While most review authors would agree that systematic reviews should be relevant and useful for decision makers, there appears to be few well known, if any, established methods for supporting review authors to assess the transferability of review findings to the context of interest in the review. With this systematic mapping and content analysis, we aim to identify whether there exists checklists to support review authors in considering transferability early in the systematic review process. The secondary aim was to develop a comprehensive list of factors that influence transferability as discussed in existing checklists.

METHODS

We conducted a systematic mapping of checklists and performed a content analysis of the checklist criteria included in the identified checklists. In June 2016, we conducted a systematic search of eight databases to identify checklists to assess transferability of findings from primary or secondary research, without limitations related to publication type, status, language, or date. We also conducted a gray literature search and searched the EQUATOR repository of checklists for any relevant document. We used search terms such as modified versions of the terms "transferability," "applicability," "generalizability," etc. and "checklist," "guideline," "tool," "criteria," etc. We did not include papers that discussed transferability at a theoretical level or checklists to assess the transferability of guidelines to local contexts.

RESULTS

Our search resulted in 11,752 titles which were screened independently by two review authors. The 101 articles which were considered potentially relevant were subsequently read by two authors, independently in full text and assessed for inclusion. We identified 31 relevant checklists. Six of these examined transferability of economic evaluations, and 25 examined transferability of primary or secondary research findings in health (n = 23) or social welfare (n = 2). The content analysis is based on the 25 health and social welfare checklists. We identified seven themes under which we grouped categories of checklist criteria: population, intervention, implementation context (immediate), comparison intervention, outcomes, environmental context, and researcher conduct.

CONCLUSIONS

We identified a variety of checklists intended to support end users (researchers, review authors, practitioners, etc.) to assess transferability or related concepts. While four of these checklists are intended for use in systematic reviews of effectiveness, we found no checklists for qualitative evidence syntheses or for the field of social welfare practice or policy. Furthermore, none of the identified checklists for review authors included guidance to on how to assess transferability, or present assessments in a systematic review. The results of the content analysis can serve as the basis for developing a comprehensive list of factors to be used in an approach to support review authors in systematically and transparently considering transferability from the beginning of the review process.

Communicating the benefits of population health interventions: The health effects can be on par with those of medication

How can we communicate to the public that population level health interventions are effective at improving health? Perhaps the most familiar "currency" of effect is that which can be brought about via medication. Comparisons of effect sizes may be effective ways of communicating the benefits of population health interventions if they are seen and understood in the same way that medications are. We developed a series of comparisons to communicate benefits of population health interventions in terms of similar gains to be obtained from statins, metformin and antihypertensive medications for prevention of cardiovascular events, type 2 diabetes, obesity and hypertension. A purposive search identified evidence of population health intervention-related benefits. This evidence ranged from meta-analyses of RCTs to that from observational cohort studies. Population health interventions included implementation of national smoke free legislation, enhanced neighbourhood walkability, increased opportunities for active travel and protection of urban green space. In some cases, the benefits of population health interventions were found to be equivalent to, or even outweighed those of the medications to which they were compared. For example, RCT-based evidence suggested that exercise taken with a view of a green space was associated with 12 mmHg and 6 mmHg reductions in systolic and diastolic blood pressure, respectively, which was at least on par with the reductions associated with antihypertensive medications. Future work will test the effectiveness of these comparisons for increasing the familiarity, credibility and acceptability of population health interventions and, in particular, examine the importance of communicating putative mechanisms and potential co-benefits.

Examining the wider context of formal youth mentoring programme development, delivery and maintenance: A qualitative study with mentoring managers and experts in the United Kingdom

Mentoring programmes are commonplace, yet little is known about the circumstances in which they operate. This study aimed to gain insight into the context surrounding youth mentoring programmes by asking programme managers and experts in the United Kingdom about their experiences. Telephone interviews with twenty-three programme managers and five experts were undertaken. Interviews were recorded, transcribed verbatim and analysed iteratively using thematic analysis. Contextual influences at the individual-, interpersonal-, organisational-, community-, policy-, and societal-level were identified to impact on programme's development, delivery and maintenance and were summarised in a model. This study further found that youth mentoring programmes operate within a complex context. This context provides challenges and opportunities that impact on programme's sustainability; resulting in many externally-funded programmes to 'hang by a thread'. It is important for service providers, commissioners and academics to recognise the complexity surrounding mentoring programmes to ensure that programmes are delivered as intended and evaluated appropriately.

How can additional secondary data analysis of observational data enhance the generalisability of meta-analytic evidence for local public health decision making?

This paper critically explores how survey and routinely collected data could aid in assessing the generalisability of public health evidence. We propose developing approaches that could be employed in understanding the relevance of public health evidence, and investigate ways of producing meta-analytic estimates tailored to reflect local circumstances, based on analyses of secondary data. Currently, public health decision makers face challenges in interpreting global review evidence to assess its meaning in local contexts. A lack of clarity on the definition and scope of generalisability, and the absence of consensus on its measurement, has stunted methodological progress. The consequence of failing to tackle generalisability means that systematic review evidence often fails to fulfil its potential contribution in public health decision making. Three approaches to address these problems are considered and emerging challenges discussed: (1) purposeful exploration after a review has been conducted, and we present a framework of potential avenues of enquiry and a worked example; (2) recalibration of the results to weight studies differentially based on their similarity to conditions in an inference population, and we provide a worked example using UK Census data to understand potential differences in the effectiveness of community engagement interventions among sites in England and Wales; (3) purposeful exploration before starting a review to ensure that the findings are relevant to an inference population. The paper aims to demonstrate how a more nuanced treatment of context in reviews of public health interventions could be achieved through greater engagement with existing large sources of secondary data.

Exploration of the experiences, practices and needs of health promotion professionals when evaluating their interventions and programmes

INTRODUCTION

The aim of this study was to describe the practices of health promotion professionals when evaluating interventions and their transferability and to identify these professionals' needs in relation to a tool that will guide them during monitoring and evaluation.

METHOD

A survey was carried out among health promotion actors, which focussed on the interventions they had carried out.

RESULTS

Of the 1017 organisations (covering the whole of France) approached, 246 responded to the survey. More than 60% of the respondents had faced difficulties during the construction of the process evaluation, and more than 50% had faced difficulties during its implementation. These difficulties related to data collection, the conception or choice of indicators, criteria and relevant evaluation tools, and lack of time and personnel. Eighty per cent reported that they had carried out the same action in different places and had been faced with two main obstacles, namely budget and environment. Recognition, positive evaluation, existing collaborations and willingness to work in partnership plus the capacity to adapt to local contexts were identified as factors fostering transferability.

DISCUSSION

Despite the many recommendations available, constructing and carrying out evaluations is not systematic. Actors are confronted with many difficulties, which could be overcome with an IT tool that incorporates the actors' suggestions during its development.

30 Days Wild: who benefits most?

Purpose

There is a need to provide interventions to improve well-being that are accessible and cost-effective. Interventions to increase engagement with nature are coming to the fore. The Wildlife Trusts 30 Days Wild campaign shows promise as a large-scale intervention for improving public engagement with nature for well-being. The paper aims to discuss this issue.

Design/methodology/approach

In total, 273 people fully participated in a repeated measures evaluation comparing baseline measures of nature connection, health, happiness and conservation behaviours with measures post-30 days and 3 months.

Findings

There were sustained and significant increases for scores in nature connection, health, happiness and conservation behaviours. Those with lower scores at baseline in nature connection, conservation behaviours and happiness showed the most benefit. Older participants and those with higher baseline scores in conservation behaviours were the most likely to sustain their engagement with the campaign.

Research limitations/implications

Although the design and defined outcomes meet criteria for public health interventions, the self-reported measures, self-selecting sample and attrition are limitations.

Originality/value

The significant and sustained effects of the campaign on health, happiness and nature connection and conservation make this a promising intervention for improving human’s and nature’s well-being. The large community sample and naturalistic setting for the intervention make these data relevant to future interventions and policy.

30 Days Wild and the Relationships Between Engagement With Nature's Beauty, Nature Connectedness and Well-Being

Recent research suggests that engagement with natural beauty (EWNB) is key to the well-being benefits of nature connectedness. The Wildlife Trust's 30 Days Wild campaign provides a large-scale intervention for improving public engagement with nature and its beauty. The effect of 30 Days Wild participation on levels of EWNB and the relationship between EWNB, nature connectedness and happiness was evaluated during the 2017 campaign. Of the 49,000 people who signed up to the campaign, 308 people fully completed measures of EWNB, nature connection, health, happiness, and conservation behaviors at baseline, post-30 days and post-2 months. There were sustained and significant increases for scores in nature connection, health, happiness, and conservation behaviors. In addition, 30 Days Wild was the first intervention found to increase EWNB. Further, the significant increase in EWNB mediated the relationship between the increases in nature connectedness and happiness. In a supplementary study to understand the well-being benefits further (n = 153), emotional regulation was found to mediate the relationship between nature connectedness and happiness, but EWNB and emotional regulation were not related. The links between nature's beauty, nature connectedness and well-being are discussed within an account of affect-regulation.

How is it best to deliver care in acute medical units? A systematic review

The majority of medical patients presenting to hospital in the UK are cared for in acute medical units (AMUs). Such units are also increasingly present internationally. Care delivery varies across units: this review aims to examine the evidence for how best to deliver AMU care.Six electronic databases and grey literature were searched. Inclusion criteria comprised interventions applied to undifferentiated patients in AMU settings. All studies were quality assessed. A narrative approach was undertaken.Nine studies, all conducted in the UK or Ireland, evaluated 1.3 million episodes, 3617 patients and 49 staff. There was single study evidence for beneficial effects of: enhanced pharmacy care, a dedicated occupational therapy service, an all-inclusive consultant work pattern, a rapid-access medical clinic and formalized handovers. Two studies found increased consultant presence was associated with reduced mortality; one of these studies found an association with a reduction in 28-day readmissions; and the other found an association with an increased proportion of patients discharged on the day they were admitted. Three studies provide evidence of the beneficial effects of multiple interventions developed from local service reviews.Overall, the quality of the evidence was limited. This review has identified operationally relevant evidence that increased consultant presence is associated with improved outcomes of care; has highlighted the potential to improve outcomes locally through service reviews; and has demonstrated an important knowledge gap of how best to deliver AMU care. These findings have importance given the challenges acute services currently face.

Comparison of registered and published intervention fidelity assessment in cluster randomised trials of public health interventions in low- and middle-income countries: systematic review

BACKGROUND

Cluster randomised trials (CRTs) are a key instrument to evaluate public health interventions. Fidelity assessment examines study processes to gauge whether an intervention was delivered as initially planned. Evaluation of implementation fidelity (IF) is required to establish whether the measured effects of a trial are due to the intervention itself and may be particularly important for CRTs of complex interventions conducted in low- and middle-income countries (LMICs). However, current CRT reporting guidelines offer no guidance on IF assessment. The objective of this review was to study current practices concerning the assessment of IF in CRTs of public health interventions in LMICs.

METHODS

CRTs of public health interventions in LMICs that planned or reported IF assessment in either the trial protocol or the main trial report were included. The MEDLINE/PubMed, CINAHL and EMBASE databases were queried from January 2012 to May 2016. To ensure availability of a study protocol, CRTs reporting a registration number in the abstract were included. Relevant data were extracted from each study protocol and trial report by two researchers using a predefined screening sheet. Risk of bias for individual studies was assessed.

RESULTS

We identified 90 CRTs of public health interventions in LMICs with a study protocol in a publicly available trial registry published from January 2012 to May 2016. Among these 90 studies, 25 (28%) did not plan or report assessing IF; the remaining 65 studies (72%) addressed at least one IF dimension. IF assessment was planned in 40% (36/90) of trial protocols and reported in 71.1% (64/90) of trial reports. The proportion of overall agreement between the trial protocol and trial report concerning occurrence of IF assessment was 66.7% (60/90). Most studies had low to moderate risk of bias.

CONCLUSIONS

IF assessment is not currently a systematic practice in CRTs of public health interventions carried out in LMICs. In the absence of IF assessment, it may be difficult to determine if CRT results are due to the intervention design, to its implementation, or to unknown or external factors that may influence results. CRT reporting guidelines should promote IF assessment.

TRIAL REGISTRATION

Protocol published and available at: https://doi.org/10.1186/s13643-016-0351-0.

Population-level interventions in government jurisdictions for dietary sodium reduction: a Cochrane Review

Background

Worldwide, excessive salt consumption is common and is a leading cause of high blood pressure. Our objectives were to assess the overall and differential impact (by social and economic indicators) of population-level interventions for dietary sodium reduction in government jurisdictions worldwide.

Methods

This is a Cochrane systematic review. We searched nine peer-reviewed databases, seven grey literature resources and contacted national programme leaders. We appraised studies using an adapted version of the Cochrane risk of bias tool. To assess impact, we computed the mean change in salt intake (g/day) from before to after intervention.

Results

Fifteen initiatives met the inclusion criteria and 10 provided sufficient data for quantitative analysis of impact. Of these, five showed a mean decrease in salt intake from before to after intervention including: China, Finland (Kuopio area), France, Ireland and the UK. When the sample was constrained to the seven initiatives that were multicomponent and incorporated activities of a structural nature (e.g. procurement policy), most (4/7) showed a mean decrease in salt intake. A reduction in salt intake was more apparent among men than women. There was insufficient information to assess differential impact by other social and economic axes. Although many initiatives had methodological strengths, all scored as having a high risk of bias reflecting the observational design. Study heterogeneity was high, reflecting different contexts and initiative characteristics.

Conclusions

Population-level dietary sodium reduction initiatives have the potential to reduce dietary salt intake, especially if they are multicomponent and incorporate intervention activities of a structural nature. It is important to consider data infrastructure to permit monitoring of these initiatives.

WASH Benefits Bangladesh trial: system for monitoring coverage and quality in an efficacy trial

BACKGROUND

Researchers typically report more on the impact of public health interventions and less on the degree to which interventions were followed implementation fidelity. We developed and measured fidelity indicators for the WASH Benefits Bangladesh study, a large-scale efficacy trial, in order to identify gaps between intended and actual implementation.

METHODS

Community health workers (CHWs) delivered individual and combined water, sanitation, handwashing (WSH) and child nutrition interventions to 4169 enrolled households in geographically matched clusters. Households received free enabling technologies (insulated water storage container; sani-scoop, potty, double-pit, pour-flush latrine; handwashing station, soapy-water storage bottle), and supplies (chlorine tablets, lipid-based nutrient supplements, laundry detergent sachets) integrated with parallel behavior-change promotion. Behavioral objectives were drinking treated, safely stored water, safe feces disposal, handwashing with soap at key times, and age-appropriate nutrition behaviors. We administered monthly surveys and spot-checks to households from randomly selected clusters for 6 months early in the trial. If any fidelity measures fell below set benchmarks, a rapid response mechanism was triggered.

RESULTS

In the first 3 months, functional water seals were detected in 33% (14/42) of latrines in the sanitation only arm; 35% (14/40) for the combined WSH arm; and 60% (34/57) for the combined WSH and Nutrition arm, all falling below the pre-set benchmark of 80%. Other fidelity indicators met the 65 to 80% uptake benchmarks. Rapid qualitative investigations determined that households concurrently used their own latrines with broken water seals in parallel with those provided by the trial. In consultation with the households, we closed pre-existing latrines without water seals, increased the CHWs' visit frequency to encourage correct maintenance of latrines with water seals, and discouraged water-seal removal or breakage. At the sixth assessment, 86% (51/59) of households were in sanitation only; 92% (72/78) in the combined WSH; and 93% (71/76) in the combined WSH and Nutrition arms had latrines with functional water seals.

CONCLUSIONS

An intensive implementation fidelity monitoring and rapid response system proved beneficial for this efficacy trial. To implement a routine program at scale requires further research into an adaptation of fidelity monitoring that supports program effectiveness.

TRIAL REGISTRATION

WASH Benefits Bangladesh: ClinicalTrials.gov, ID: NCT01590095 . Registered on 30 April 2012.

Goal-setting to Promote a Healthier Lifestyle in Later Life: Qualitative Evaluation of the AgeWell Trial

OBJECTIVE

We report a mixed method evaluation of the feasibility and implementation of the AgeWell goal-setting intervention to promote healthy ageing later life.

METHOD

Researcher field notes, goal-setting interview content, and semi-structured interviews with participants were content analysed to review trial implementation and participants' perspective on the goal-setting and mentoring intervention.

RESULTS

75 people were recruited: 21 in the goal-setting and 22 in the goal-setting with mentoring arms of the intervention. Goal-setting was feasible in the main domains of interest. Adherence to the protocol was good and the mentoring schedule was adhered to. Participants reported satisfaction with their goal attainment, but barriers for non-achievement were also identified. Recommendations for small changes to the intervention included reducing the number of goals.

CONCLUSIONS

Participants understood the goal-setting process, and were able to set realistic and achievable lifestyle goals. The intervention and the procedures were acceptable but changes in how goal-setting is both introduced and monitored are needed for wider implementation.

CLINICAL IMPLICATIONS

Goal-setting can be a useful process to help people alter their lifestyle to allow them to age more successfully and reduce risk factors associated with dementia.

Criteria for evaluating transferability of health interventions: a systematic review and thematic synthesis

BACKGROUND

Improving the public's health in different countries requires the consideration of diverse health care systems and settings. For evidence-based public health, decision-makers need to consider the transferability of effective health interventions from a primary context to their specific target context. The aim of this systematic review was to develop a model for the assessment of transferability of health interventions through identification and systematization of influencing criteria, including facilitators and barriers.

METHODS

A systematic literature search was performed in the databases PubMed, Embase, CINAHL, and PsycINFO. Articles were eligible if they were published in English or German and provided a description of transferability criteria. Included articles were ranked based on their thematic relevance and methodological support of transferability criteria. Using a qualitative approach, a thematic synthesis was conducted.

RESULTS

Thirty-seven articles were included in the review. The thematic synthesis revealed 44 criteria, covered by 4 overarching themes, which influence transferability of health interventions: The population (P), the intervention (I), and the environment (E) represent 30 conditional transferability criteria, and the transfer of the intervention (T) represents 14 process criteria for transferring the intervention to the target context. Transferability (-T) depends on the dynamic interaction of conditional criteria in the primary and target context as well as on the process of transfer. The description of facilitators and barriers deepens the understanding of the criteria. The synthesis resulted in two related models: the conceptual PIET-T model explains the underlying mechanism of transferability of health interventions and the PIET-T process model provides practical guidance for a transferability assessment.

CONCLUSIONS

Transferability of health interventions is a complex concept, which needs systematic consideration of the primary and target context. It should be anticipated before and evaluated after an intervention is implemented in the target context. Therefore, decision-makers need systematic and practically relevant knowledge on transferability. The synthesized PIET-T conceptual and process models with systematized criteria, facilitators, and barriers are intended as a theoretical basis to determine transferability of health interventions. Further research is needed to develop a practical tool for the PIET-T models and to evaluate the tool's usefulness for decision-making processes and intervention transfer.

Criteria for evaluating transferability of health interventions: a systematic review and thematic synthesis

BACKGROUND

Improving the public's health in different countries requires the consideration of diverse health care systems and settings. For evidence-based public health, decision-makers need to consider the transferability of effective health interventions from a primary context to their specific target context. The aim of this systematic review was to develop a model for the assessment of transferability of health interventions through identification and systematization of influencing criteria, including facilitators and barriers.

METHODS

A systematic literature search was performed in the databases PubMed, Embase, CINAHL, and PsycINFO. Articles were eligible if they were published in English or German and provided a description of transferability criteria. Included articles were ranked based on their thematic relevance and methodological support of transferability criteria. Using a qualitative approach, a thematic synthesis was conducted.

RESULTS

Thirty-seven articles were included in the review. The thematic synthesis revealed 44 criteria, covered by 4 overarching themes, which influence transferability of health interventions: The population (P), the intervention (I), and the environment (E) represent 30 conditional transferability criteria, and the transfer of the intervention (T) represents 14 process criteria for transferring the intervention to the target context. Transferability (-T) depends on the dynamic interaction of conditional criteria in the primary and target context as well as on the process of transfer. The description of facilitators and barriers deepens the understanding of the criteria. The synthesis resulted in two related models: the conceptual PIET-T model explains the underlying mechanism of transferability of health interventions and the PIET-T process model provides practical guidance for a transferability assessment.

CONCLUSIONS

Transferability of health interventions is a complex concept, which needs systematic consideration of the primary and target context. It should be anticipated before and evaluated after an intervention is implemented in the target context. Therefore, decision-makers need systematic and practically relevant knowledge on transferability. The synthesized PIET-T conceptual and process models with systematized criteria, facilitators, and barriers are intended as a theoretical basis to determine transferability of health interventions. Further research is needed to develop a practical tool for the PIET-T models and to evaluate the tool's usefulness for decision-making processes and intervention transfer.

Hubris, humility and humanity: expanding evidence approaches for improving and sustaining community health programmes

Community-based approaches are a critical foundation for many health outcomes, including reproductive, maternal, newborn and child health (RMNCH). Evidence is a vital part of strengthening that foundation, but largely focuses on the technical content of what must be done, rather than on how disparate community actors continuously interpret, implement and adapt interventions in dynamic and varied community health systems. We argue that efforts to strengthen evidence for community programmes must guard against the hubris of relying on a single approach or hierarchy of evidence for the range of research questions that arise when sustaining community programmes at scale. Moving forward we need a broader evidence agenda that better addresses the implementation realities influencing the scale and sustainability of community programmes and the partnerships underpinning them if future gains in community RMNCH are to be realised. This will require humility in understanding communities as social systems, the complexity of the interventions they engage with and the heterogeneity of evidence needs that address the implementation challenges faced. It also entails building common ground across epistemological word views to strengthen the robustness of implementation research by improving the use of conceptual frameworks, addressing uncertainty and fostering collaboration. Given the complexity of scaling up and sustaining community RMNCH, ensuring that evidence translates into action will require the ongoing brokering of relationships to support the human creativity, scepticism and scaffolding that together build layers of evidence, critical thinking and collaborative learning to effect change.

Evaluation of cash transfer programs in sub-Saharan Africa: A methodological review

Cash transfer programs (CTs) have been rigorously evaluated since their inception in the 1990s. However, to date, there has been no study critically examining the utility of the methodological approaches used to evaluate CTs. This article reviews the approaches used to evaluate CTs in sub-Saharan Africa (SSA) to provide recommendations for improving future CTs evaluations. We conducted searches for CTs evaluation studies in SSA in the peer-reviewed and grey literature using electronic databases, hand searching of selected journals, organisational websites, Google Scholar and Scirus Internet search engines. The review included 53 evaluation studies which were largely outcome-focused evaluations (95%; n = 50). The studies were undertaken within 24 CT programs comprising 11 unconditional CTs (UCTs), eight conditional CTs (CCTs) and five combined UCTs and CCTs. The review finds that while there is evidence of CTs impacts on a broad range of outcomes, the current evaluation approaches have primarily been experimental designs and have largely failed to provide explanations for mechanisms of change. To improve CTs policy and practice, there is the need to consider theory-based evaluation approaches such as realist evaluation that provide insights about the contexts and mechanisms through which programs generate outcomes in different circumstances.

The Ontario Food and Nutrition Strategy: identifying indicators of food access and food literacy for early monitoring of the food environment

INTRODUCTION

To address challenges Canadians face within their food environments, a comprehensive, multistakeholder, intergovernmental approach to policy development is essential. Food environment indicators are needed to assess population status and change. The Ontario Food and Nutrition Strategy (OFNS) integrates the food, agriculture and nutrition sectors, and aims to improve the health of Ontarians through actions that promote healthy food systems and environments. This report describes the process of identifying indicators for 11 OFNS action areas in two strategic directions (SDs): Healthy Food Access, and Food Literacy and Skills.

METHODS

The OFNS Indicators Advisory Group used a five-step process to select indicators: (1) potential indicators from national and provincial data sources were identified; (2) indicators were organized by SD, action area and data type; (3) selection criteria were identified, pilot tested and finalized; (4) final criteria were applied to refine the indicator list; and (5) indicators were prioritized after reapplication of selection criteria.

RESULTS

Sixty-nine potential indicators were initially identified; however, many were individual-level rather than system-level measures. After final application of the selection criteria, one individual-level indicator and six system-level indicators were prioritized in five action areas; for six of the action areas, no indicators were available.

CONCLUSION

Data limitations suggest that available data may not measure important aspects of the food environment, highlighting the need for action and resources to improve system-level indicators and support monitoring of the food environment and health in Ontario and across Canada.

Interventions to improve physical activity among socioeconomically disadvantaged groups: an umbrella review

Background

People from socioeconomically disadvantaged population groups are less likely to be physically active and more likely to experience adverse health outcomes than those who are less disadvantaged. In this umbrella review we examined across all age groups, (1) the effectiveness of interventions to improve physical activity among socioeconomically disadvantaged groups, (2) the characteristics of effective interventions, and (3) directions for future research.

Methods

PubMed/MEDLINE and Scopus were searched up to May 2017 to identify systematic reviews reporting physical activity interventions in socioeconomically disadvantaged populations or sub-groups. Two authors independently conducted study screening and selection, data extraction (one author, with data checked by two others) and assessment of methodological quality using the ‘Assessment of Multiple Systematic Reviews’ scale. Results were synthesized narratively.

Results

Seventeen reviews met our inclusion criteria, with only 5 (30%) reviews being assessed as high quality. Seven (41%) reviews focused on obesity prevention and an additional four focused on multiple behavioural outcomes. For pre school children, parent-focused, group-based interventions were effective in improving physical activity. For children, school-based interventions and policies were effective; few studies focused on adolescents and those that did were generally not effective; for adults, there was mixed evidence of effectiveness but characteristics such as group-based interventions and those that focused on physical activity only were associated with effectiveness. Few studies focused on older adults. Across all ages, interventions that were more intensive tended to be more effective. Most studies reported short-term, rather than longer-term, outcomes and common methodological limitations included high probability of selection bias, low response rates, and high attrition.

Conclusions

Interventions can be successful at improving physical activity among children from socioeconomically disadvantaged groups, with evidence for other age groups weak or inconclusive. More high-quality studies in this population group are needed, which adopt strategies to increase recruitment rates and reduce attrition, report longer term outcomes, and provide adequate intervention details, to allow determination of the characteristics of effective interventions. We recommend that the benefits of physical activity be recognised more broadly than obesity prevention in future studies, as this may have implications for the design and appeal of interventions.

Electronic supplementary material

The online version of this article (10.1186/s12966-018-0676-2) contains supplementary material, which is available to authorized users.