It might work in Oklahoma but will it work in Oakhampton? Context and implementation in the effectiveness literature on domestic smoke detectors

E-health interventions targeting STIs, sexual risk, substance use and mental health among men who have sex with men: four systematic reviews

Background

Human immunodeficiency virus/sexually transmitted infections, sexual risk, substance (alcohol and other legal and illegal drugs) use and mental ill health constitute a ‘syndemic’ of mutually reinforcing epidemics among men who have sex with men. Electronic health (e-health) interventions addressing these epidemics among men who have sex with men might have multiplicative effects. To our knowledge, no systematic review has examined the effectiveness of such interventions on these epidemics among men who have sex with men.

Objective

The objective was to synthesise evidence addressing the following: (1) What approaches and theories of change do existing e-health interventions employ to prevent human immunodeficiency virus/sexually transmitted infections, sexual risk, alcohol/drug use or mental ill health among men who have sex with men? (2) What factors influence implementation? (3) What are the effects of such interventions on the aforementioned epidemics? (4) Are such interventions cost-effective?

Data sources

A total of 24 information sources were searched initially (October–November 2018) [the following sources were searched: ProQuest Applied Social Sciences Index and Abstracts; Campbell Library; EBSCO Cumulative Index to Nursing and Allied Health Literature Plus, Wiley Online Library The Cochrane Library; Centre for Reviews and Dissemination databases (the Database of Abstracts of Reviews of Effects and the NHS Economic Evaluation Database); the Health Technology Assessment database; Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) database of health promotion research (Bibliomap); ProQuest Dissertations & Theses Global; OvidSP EconLit; OvidSP EMBASE; OvidSP Global Health; OvidSP Health Management Information Consortium; ProQuest International Bibliography of the Social Sciences; Ovid MEDLINE ALL; OvidSP PsycINFO; Web of Science Science Citation Index Expanded; Elsevier Scopus; OvidSP Social Policy & Practice; Web of Science Social Sciences Citation Index Expanded; ProQuest Sociological Abstracts; ClinicalTrials.gov; World Health Organization International Clinical Trials Registry Platform; EPPI-Centre Trials Register of Promoting Health Interventions; and the OpenGrey database], and an updated search of 19 of these was conducted in April 2020. Reference lists of included reports were searched and experts were contacted.

Review methods

Eligible reports presented theories of change and/or process, outcome and/or economic evaluations of e-health interventions offering ongoing support to men who have sex with men to prevent human immunodeficiency virus/sexually transmitted infections, sexual risk behaviour, alcohol/drug use and/or common mental illnesses. References were screened by title/abstract, then by full text. Data extraction and quality assessments used existing tools. Theory and process reports were synthesised using qualitative methods. Outcome and economic data were synthesised narratively; outcome data were meta-analysed.

Results

Original searches retrieved 27 eligible reports. Updated searches retrieved 10 eligible reports. Thirty-seven reports on 28 studies of 23 interventions were included: 33 on theories of change, 12 on process evaluations, 16 on outcome evaluations and one on an economic evaluation. Research question 1: five intervention types were identified – ‘online modular’, ‘computer games’ and ‘non-interactive’ time-limited/modular interventions, and open-ended interventions with ‘content organised by assessment’ and ‘general content’. Three broad types of intervention theories of change were identified, focusing on ‘cognitive/skills’, ‘self-monitoring’ and ‘cognitive therapy’. Research question 2: individual tailoring based on participant characteristics was particularly acceptable, and participants valued intervention content reflecting their experiences. Research question 3: little evidence was available of effects on human immunodeficiency virus or sexually transmitted infections. The analysis did not suggest that interventions were effective in reducing instances of human immunodeficiency virus or sexually transmitted infections. The overall meta-analysis for sexually transmitted infections reported a small non-significant increase in sexually transmitted infections in the intervention group, compared with the control group. Meta-analyses found a significant impact on sexual risk behaviour. The findings for drug use could not be meta-analysed because of study heterogeneity. Studies addressing this outcome did not present consistent evidence of effectiveness. Trials did not report effects on alcohol use or mental health. Research question 4: evidence on cost-effectiveness was limited.

Limitations

The quality of the eligible reports was variable and the economic synthesis was limited to one eligible study.

Conclusions

There is commonality in intervention theories of change and factors affecting receipt of e-health interventions. Evidence on effectiveness is limited.

Future work

Future trials should assess the impact of interventions on multiple syndemic factors, among them sexual risk, substance use and mental health; incorporate sufficient follow-up and sample sizes to detect the impact on human immunodeficiency virus/sexually transmitted infections; and incorporate rigorous process and economic evaluations.

Study registration

This study is registered as PROSPERO CRD42018110317.

Funding

This project was funded by the National Institute for Health Research (NIHR) Public Health Research programme and will be published in full in Public Health Research; Vol. 10, No. 4. See the NIHR Journals Library website for further project information.

eHealth Interventions to Address Sexual Health, Substance Use, and Mental Health Among Men Who Have Sex With Men: Systematic Review and Synthesis of Process Evaluations

Background

Men who have sex with men (MSM) face disproportionate risks concerning HIV and other sexually transmitted infections, substance use, and mental health. These outcomes constitute an interacting syndemic among MSM; interventions addressing all 3 together could have multiplicative effects. eHealth interventions can be accessed privately, and evidence from general populations suggests these can effectively address all 3 health outcomes. However, it is unclear how useable, accessible, or acceptable eHealth interventions are for MSM and what factors affect this.

Objective

We undertook a systematic review of eHealth interventions addressing sexual risk, substance use, and common mental illnesses among MSM and synthesized evidence from process evaluations.

Methods

We searched 19 databases, 3 trials registers, OpenGrey, and Google, and supplemented this by reference checks and requests to experts. Eligible reports were those that discussed eHealth interventions offering ongoing support to MSM aiming to prevent sexual risk, substance use, anxiety or depression; and assessed how intervention delivery or receipt varied with characteristics of interventions, providers, participants, or context. Reviewers screened citations on titles, abstracts, and then full text. Reviewers assessed quality of eligible studies, and extracted data on intervention, study characteristics, and process evaluation findings. The analysis used thematic synthesis.

Results

A total of 12 reports, addressing 10 studies of 8 interventions, were eligible for process synthesis. Most addressed sexual risk alone or with other outcomes. Studies were assessed as medium and high reliability (reflecting the trustworthiness of overall findings) but tended to lack depth and breadth in terms of the process issues explored. Intervention acceptability was enhanced by ease of use; privacy protection; use of diverse media; opportunities for self-reflection and to gain knowledge and skills; and content that was clear, interactive, tailored, reflective of MSM’s experiences, and affirming of sexual-minority identity. Technical issues and interventions that were too long detracted from acceptability. Some evidence suggested that acceptability varied by race or ethnicity and educational level; findings on variation by socioeconomic status were mixed. No studies explored how intervention delivery or receipt varied by provider characteristics.

Conclusions

Findings suggest that eHealth interventions targeting sexual risk, substance use, and mental health are acceptable for MSM across sociodemographic groups. We identified the factors shaping MSM’s receipt of such interventions, highlighting the importance of tailored content reflecting MSM’s experiences and of language affirming sexual-minority identities. Intervention developers can draw on these findings to increase the usability and acceptability of integrated eHealth interventions to address the syndemic of sexual risk, substance use, and mental ill health among MSM. Evaluators of these interventions can draw on our findings to plan evaluations that explore the factors shaping usability and acceptability.

'It is surprising how much nonsense you hear': How residents experience and react to living in a stigmatised place. A narrative synthesis of the qualitative evidence

There are significant geographical inequalities in health. Spatial stigma - negative representations of particular localities - could be an important mechanism through which place influences population health. To explore this, we undertook a narrative synthesis of studies reporting residents' perspectives of living in stigmatised localities. Qualitative research (38 studies) was reviewed to identify how spatial stigma manifested in residents' lives, their strategies to cope with stigma and the health consequences. The review found residents internalised stigma, but also resisted it differently. Although relatively few studies purposefully investigated health, living somewhere stigmatised had psychological effects and constrained life opportunities that have implications for health.

To the bone: Comment on "I wanted a skeleton … they brought a prince": A qualitative investigation of factors mediating the implementation of a Performance Based Incentive program in Malawi

Recently, McMahon and colleagues set out to build on a widely-used fidelity framework, assessing the role of moderating factors during the implementation of performance-based financing programs in Malawi. Their attempt draws again the attention to the importance of approaching real word implementation issues from a theoretical perspective. It also highlights the importance of fidelity assessment within process evaluation of health programs. In this comment we argue that theoretical developments in the field of implementation science in global health would benefit from an accurate understanding of existing conceptual frameworks as well as from taking into account all contemporary contributions.

A mixed-methods systematic review protocol to examine the use of physical restraint with critically ill adults and strategies for minimizing their use

BACKGROUND

Critically ill patients frequently experience severe agitation placing them at risk of harm. Physical restraint is common in intensive care units (ICUs) for clinician concerns about safety. However, physical restraint may not prevent medical device removal and has been associated with negative physical and psychological consequences. While professional society guidelines, legislation, and accreditation standards recommend physical restraint minimization, guidelines for critically ill patients are over a decade old, with recommendations that are non-specific. Our systematic review will synthesize evidence on physical restraint in critically ill adults with the primary objective of identifying effective minimization strategies.

METHODS

Two authors will independently search from inception to July 2016 the following: Ovid MEDLINE, CINAHL, Embase, Web of Science, Cochrane Library, PROSPERO, Joanna Briggs Institute, grey literature, professional society websites, and the International Clinical Trials Registry Platform. We will include quantitative and qualitative study designs, clinical practice guidelines, policy documents, and professional society recommendations relevant to physical restraint of critically ill adults. Authors will independently perform data extraction in duplicate and complete risk of bias and quality assessment using recommended tools. We will assess evidence quality for quantitative studies using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach and for qualitative studies using the Confidence in the Evidence from Reviews of Qualitative Research (CERQual) guidelines. Outcomes of interest include (1) efficacy/effectiveness of physical restraint minimization strategies; (2) adverse events (unintentional device removal, psychological impact, physical injury) and associated benefits including harm prevention; (3) ICU outcomes (ventilation duration, length of stay, and mortality); (4) prevalence, incidence, patterns of use including patient and treatment characteristics and chemical restraint; (5) barriers and facilitators to minimization; (6) patient, family, and healthcare professional perspectives; (7) professional society-endorsed recommendations; and (8) evidence gaps and research priorities.

DISCUSSION

We will use our systematic review findings to produce updated guidelines on physical restraint use for critically ill adults and to develop a professional society-endorsed position statement. This will foster patient and clinician safety by providing clinicians, administrators, and policy makers with a tool to promote minimal and safe use of physical restraint for critically ill adults.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42015027860.

How Do Contextual Factors Influence Implementation and Receipt of Positive Youth Development Programs Addressing Substance Use and Violence? A Qualitative Meta-Synthesis of Process Evaluations

OBJECTIVE

Positive youth development (PYD) often aims to prevent tobacco, alcohol, and drugs use and violence. We systematically reviewed PYD interventions, synthesizing process, and outcomes evidence. Synthesis of outcomes, published elsewhere, found no overall evidence of reducing substance use or violence but notable variability of fidelity. Our synthesis of process evaluations examined how implementation varied and was influenced by context.

DATA SOURCE

Process evaluations of PYD aiming to reduce substance use and violence. Study Inclusion Criteria: Overall review published since 1985; written in English; focused on youth aged 11 to 18 years; focused on interventions addressing multiple positive assets; reported on theory, process, or outcomes; and concerned with reducing substance use or violence. Synthesis of process evaluations examined how implementation varies with or is influenced by context.

DATA EXTRACTION

Two reviewers in parallel.

DATA SYNTHESIS

Thematic synthesis.

RESULTS

We identified 12 reports. Community engagement enhanced program appeal. Collaboration with other agencies could broaden the activities offered. Calm but authoritative staff increased acceptability. Staff continuity underpinned diverse activities and durable relationships. Empowering participants were sometimes in tension with requiring them to engage in diverse activities.

CONCLUSION

Our systematic review identified factors that might help improve the fidelity and acceptability of PYD interventions. Addressing these might enable PYD to fulfill its potential as a means of promoting health.

What is the effectiveness of the support worker role for people with dementia and their carers? A systematic review

BACKGROUND

Dementia is progressive in nature and the associated functional decline inevitably leads to increasing dependence on others in areas of daily living. Models of support have been developed and implemented to assist with adjusting to living with memory loss and functional decline; to navigate the health and aged care system; and to access services. We undertook a systematic review of international literature on key worker type support roles to identify essential components and ascertain how the role can be best utilised to assist community-dwelling people with dementia and their carers. This review of support roles is the first to our knowledge to include both quantitative and qualitative studies and all models of support.

METHOD

A systematic review of studies written in English and published between January 2003 and December 2014. Data sources were Medline, PsychInfo and CINAHL, internet, expert consultation and reference lists of included studies. After screening articles to ensure that they reported on a key worker type support role, involved carers and or people with dementia living at home and removing duplicates, eligible papers were appraised and evaluated.

RESULTS

Thirty six studies were eligible for inclusion in the review. Eligible studies were divided into type of support roles and study type. The heterogeneity of included studies and high risk of bias made a meta-analysis inappropriate and it was therefore difficult to draw overall conclusions. However, essential components shared across support worker models that demonstrated a positive impact on carer burden and improved quality of life included: long term intervention, face to face contact, individualised education and support based on needs, multi-disciplinary teams, collaborative input, health/clinical background of support workers, ongoing follow up and inter professional and inter-sectoral collaborations. There was a lack of studies assessing cost-effectiveness.

CONCLUSIONS

Studies that include a high quality evaluation of holistic, tailored models of support that identify which components of support produce the most valuable outcomes to assist people with dementia and their carers and families to continue to live meaningful lives are needed. There is also a need for a cost effectiveness evaluation of support worker roles.

TRIAL REGISTRATION

PROSPERO international prospective register of systematic reviews: PROSPERO 2014 CRD42014013992 .

A modified theoretical framework to assess implementation fidelity of adaptive public health interventions

BACKGROUND

One of the major debates in implementation research turns around fidelity and adaptation. Fidelity is the degree to which an intervention is implemented as intended by its developers. It is meant to ensure that the intervention maintains its intended effects. Adaptation is the process of implementers or users bringing changes to the original design of an intervention. Depending on the nature of the modifications brought, adaptation could either be potentially positive or could carry the risk of threatening the theoretical basis of the intervention, resulting in a negative effect on expected outcomes. Adaptive interventions are those for which adaptation is allowed or even encouraged. Classical fidelity dimensions and conceptual frameworks do not address the issue of how to adapt an intervention while still maintaining its effectiveness.

DISCUSSION

We support the idea that fidelity and adaptation co-exist and that adaptations can impact either positively or negatively on the intervention's effectiveness. For adaptive interventions, research should answer the question how an adequate fidelity-adaptation balance can be reached. One way to address this issue is by looking systematically at the aspects of an intervention that are being adapted. We conducted fidelity research on the implementation of an empowerment strategy for dengue prevention in Cuba. In view of the adaptive nature of the strategy, we anticipated that the classical fidelity dimensions would be of limited use for assessing adaptations. The typology we used in the assessment-implemented, not-implemented, modified, or added components of the strategy-also had limitations. It did not allow us to answer the question which of the modifications introduced in the strategy contributed to or distracted from outcomes. We confronted our empirical research with existing literature on fidelity, and as a result, considered that the framework for implementation fidelity proposed by Carroll et al. in 2007 could potentially meet our concerns. We propose modifications to the framework to assess both fidelity and adaptation. The modified Carroll et al.'s framework we propose may permit a comprehensive assessment of the implementation fidelity-adaptation balance required when implementing adaptive interventions, but more empirical research is needed to validate it.

The effects of Positive Youth Development interventions on substance use, violence and inequalities: systematic review of theories of change, processes and outcomes

Background

Positive Youth Development (PYD) delivered outside school aims to enable young people to develop positive assets such as relationships and confidence, rather than to merely address risk. Existing reviews of PYD effects on substance use or violence are old and unsystematic.

Objectives

To systematically review evidence to answer the following questions: what theories of change inform PYD interventions addressing substance use and violence? What characteristics of participants and contexts are identified as barriers to and facilitators of implementation and receipt in process evaluations of PYD? What is the effectiveness and cost-effectiveness of PYD in reducing substance use and violence? What characteristics of participants and contexts appear to moderate, or are necessary and sufficient for, PYD effectiveness?

Data sources

A total of 21 bibliographic databases; 35 websites and contacting authors.

Review methods

We included reports published in English since 1985 and reporting on theories of change, as well as process, outcome and economic evaluations of PYD targeting 11- to 18-year-olds and addressing substance use or violence. References were screened on title/abstract and, where appropriate, on full report. Data extraction and quality assessment used Critical Appraisal Skills Programme, Evidence for Policy and Practice Information and Co-ordinating Centre and Cochrane tools. Theories of change and process evaluations were qualitatively metasynthesised. Outcome evaluations were synthesised narratively and meta-analytically.

Results

32,394 unique references were identified and 48 were included. A total of 16 reports described theories, 13 (10 studies) evaluated processes and 25 (10 studies) evaluated outcomes.

Theories of change

PYD interventions aim to offer opportunities for young people to develop positive ‘assets’ such as skills and confidence. These are theorised to promote and be promoted by young people’s ‘intentional self-regulation’, which involves reflecting on behaviour; determining goals; using existing resources to pursue these; and redirecting effort when thwarted. This enables ‘developmental regulation’, namely individuals capitalising on other opportunities to promote personal development. Positive assets thus accrued reduce health risks by reducing the impact on individuals of environmental risk or by ameliorating the impact of such risks. The literature offers limited insights beyond these general ideas.

Process evaluations

Community engagement ensured that programmes were accessible and appealing. Staff capacity and continuity were crucial factors but often challenging when programmes could not offer full-time jobs. Tensions arose between a desire to empower participants to choose activities and a requirement for them to undertake a breadth of activities.

Outcome evaluations

Meta-analyses of all combined outcomes and of short-term alcohol use, illicit drug use and smoking found no significant effects. There were small, statistically significant, short-term effects for an omnibus measure of substance use and for violence. We could not undertake metaregression to assess sociodemographic moderators but narrative synthesis suggested no clear pattern of effects by sex. We found no economic evaluations.

Limitations

Insufficient studies precluded qualitative comparative analyses.

Conclusions

How PYD might promote health is currently undertheorised. Implementation can be challenging. We found little evidence that current PYD interventions delivered outside school reduce substance use or violence. However, these may not constitute a test of the effectiveness of the PYD model, as some included interventions that, although meeting our inclusion criteria, were not exemplars of PYD.

Future work

Further evaluations should assess interventions employing PYD theory of change.

Study registration

This study is registered as PROSPERO CRD42013005439.

Funding

The National Institute for Health Research Public Health Research programme.

Development, inter-rater reliability and feasibility of a checklist to assess implementation (Ch-IMP) in systematic reviews: the case of provider-based prevention and treatment programs targeting children and youth

BACKGROUND

Several papers report deficiencies in the reporting of information about the implementation of interventions in clinical trials. Information about implementation is also required in systematic reviews of complex interventions to facilitate the translation and uptake of evidence of provider-based prevention and treatment programs. To capture whether and how implementation is assessed within systematic effectiveness reviews, we developed a checklist for implementation (Ch-IMP) and piloted it in a cohort of reviews on provider-based prevention and treatment interventions for children and young people. This paper reports on the inter-rater reliability, feasibility and reasons for discrepant ratings.

METHODS

Checklist domains were informed by a framework for program theory; items within domains were generated from a literature review. The checklist was pilot-tested on a cohort of 27 effectiveness reviews targeting children and youth. Two raters independently extracted information on 47 items. Inter-rater reliability was evaluated using percentage agreement and unweighted kappa coefficients. Reasons for discrepant ratings were content analysed.

RESULTS

Kappa coefficients ranged from 0.37 to 1.00 and were not influenced by one-sided bias. Most kappa values were classified as excellent (n = 20) or good (n = 17) with a few items categorised as fair (n = 7) or poor (n = 1). Prevalence-adjusted kappa coefficients indicate good or excellent agreement for all but one item. Four areas contributed to scoring discrepancies: 1) clarity or sufficiency of information provided in the review; 2) information missed in the review; 3) issues encountered with the tool; and 4) issues encountered at the review level. Use of the tool demands time investment and it requires adjustment to improve its feasibility for wider use.

CONCLUSIONS

The case of provider-based prevention and treatment interventions showed relevancy in developing and piloting the Ch-IMP as a useful tool for assessing the extent to which systematic reviews assess the quality of implementation. The checklist could be used by authors and editors to improve the quality of systematic reviews, and shows promise as a pedagogical tool to facilitate the extraction and reporting of implementation characteristics.

Lessons from complex interventions to improve health

Complexity-resulting from interactions among many component parts-is a property of both the intervention and the context (or system) into which it is placed. Complexity increases the unpredictability of effects. Complexity invites new approaches to logic modeling, definitions of integrity and means of standardization, and evaluation. New metaphors and terminology are needed to capture the recognition that knowledge generation comes from the hands of practitioners/implementers as much as it comes from those usually playing the role of intervention researcher. Failure to acknowledge this may blind us to the very mechanisms we seek to understand. Researchers in clinical settings are documenting health improvement gains made as a consequence of complex systems thinking. Improvement science in clinical settings has much to offer researchers in population health.

Using process data to understand outcomes in sexual health promotion: an example from a review of school-based programmes to prevent sexually transmitted infections

This article discusses how process indicators can complement outcomes as part of a comprehensive explanatory evaluation framework, using the example of skills-based behavioural interventions to prevent sexually transmitted infections and promote sexual health among young people in schools. A systematic review was conducted, yielding 12 eligible outcome evaluations, 9 of which included a process evaluation. There were few statistically significant effects in terms of changes in sexual behaviour outcomes, but statistically significant effects were more common for knowledge and self-efficacy. Synthesis of the findings of the process evaluations identified a range of factors that might explain outcomes, and these were organized into two overarching categories: the implementation of interventions, and student engagement and intervention acceptability. Factors which supported implementation and engagement and acceptability included good quality teacher training, involvement and motivation of key school stakeholders and relevance and appeal to young people. Factors which had a negative impact included teachers' failure to comprehend the theoretical basis for behaviour change, school logistical problems and omission of topics that young people considered important. It is recommended that process indicators such as these be assessed in future evaluations of school-based sexual health behavioural interventions, as part of a logic model.

What works in practice: user and provider perspectives on the acceptability, affordability, implementation, and impact of a family-based intervention for child overweight and obesity delivered at scale

BACKGROUND

As part of a study considering the impact of a child weight management programme when rolled out at scale following an RCT, this qualitative study focused on acceptability and implementation for providers and for families taking part.

METHODS

Participants were selected on the basis of a maximum variation sample providing a range of experiences and social contexts. Qualitative interviews were conducted with 29 professionals who commissioned or delivered the programme, and 64 individuals from 23 families in 3 English regions. Topic guides were used as a tool rather than a rule, enabling participants to construct a narrative about their experiences. Transcripts were analysed using framework analysis.

RESULTS

Practical problems such as transport, work schedules and competing demands on family time were common barriers to participation. Delivery partners often put considerable efforts into recruiting, retaining and motivating families, which increased uptake but also increased cost. Parents and providers valued skilled delivery staff. Some providers made adaptations to meet local social and cultural needs. Both providers and parents expressed concerns about long term outcomes, and how this was compromised by an obesogenic environment. Concerns about funding together with barriers to uptake and engagement could translate into barriers to commissioning. Where these barriers were not experienced, commissioners were enthusiastic about continuing the programme.

CONCLUSIONS

Most families felt that they had gained something from the programme, but few felt that it had 'worked' for them. The demands on families including time and emotional work were experienced as difficult. For commissioners, an RCT with positive results was an important driver, but family barriers, alongside concerns about recruitment and retention, a desire for local adaptability with qualified motivated staff, and funding changes discouraged some from planning to use the intervention in future.

Celebrating methodological challenges and changes: reflecting on the emergence and importance of the role of qualitative evidence in Cochrane reviews

Cochrane systematic reviews have proven to be beneficial for decision making processes, both on a practitioner and a policy level, and there are current initiatives to extend the types of evidence used by them, including qualitative research. In this article we outline the major achievements of the Cochrane Qualitative and Implementation Methods Group. Although the Group has encountered numerous challenges in dealing with the evolution of qualitative evidence synthesis, both outside and within the Cochrane Collaboration, it has successfully responded to the challenges posed in terms of incorporating qualitative evidence in systematic reviews. The Methods Group will continue to advocate for more flexible and inclusive approaches to evidence synthesis in order to meet the exciting challenges and opportunities presented by mixed methods systematic reviews and reviews of complex interventions.

Towards a methodology for cluster searching to provide conceptual and contextual "richness" for systematic reviews of complex interventions: case study (CLUSTER)

Background

Systematic review methodologies can be harnessed to help researchers to understand and explain how complex interventions may work. Typically, when reviewing complex interventions, a review team will seek to understand the theories that underpin an intervention and the specific context for that intervention. A single published report from a research project does not typically contain this required level of detail. A review team may find it more useful to examine a “study cluster”; a group of related papers that explore and explain various features of a single project and thus supply necessary detail relating to theory and/or context.

We sought to conduct a preliminary investigation, from a single case study review, of techniques required to identify a cluster of related research reports, to document the yield from such methods, and to outline a systematic methodology for cluster searching.

Methods

In a systematic review of community engagement we identified a relevant project – the Gay Men’s Task Force. From a single “key pearl citation” we conducted a series of related searches to find contextually or theoretically proximate documents. We followed up Citations, traced Lead authors, identified Unpublished materials, searched Google Scholar, tracked Theories, undertook ancestry searching for Early examples and followed up Related projects (embodied in the CLUSTER mnemonic).

Results

Our structured, formalised procedure for cluster searching identified useful reports that are not typically identified from topic-based searches on bibliographic databases. Items previously rejected by an initial sift were subsequently found to inform our understanding of underpinning theory (for example Diffusion of Innovations Theory), context or both. Relevant material included book chapters, a Web-based process evaluation, and peer reviewed reports of projects sharing a common ancestry. We used these reports to understand the context for the intervention and to explore explanations for its relative lack of success. Additional data helped us to challenge simplistic assumptions on the homogeneity of the target population.

Conclusions

A single case study suggests the potential utility of cluster searching, particularly for reviews that depend on an understanding of context, e.g. realist synthesis. The methodology is transparent, explicit and reproducible. There is no reason to believe that cluster searching is not generalizable to other review topics. Further research should examine the contribution of the methodology beyond improved yield, to the final synthesis and interpretation, possibly by utilizing qualitative sensitivity analysis.

The Oxford Implementation Index: a new tool for incorporating implementation data into systematic reviews and meta-analyses

OBJECTIVES

This article presents a new tool that helps systematic reviewers to extract and compare implementation data across primary trials. Currently, systematic review guidance does not provide guidelines for the identification and extraction of data related to the implementation of the underlying interventions.

STUDY DESIGN AND SETTING

A team of systematic reviewers used a multistaged consensus development approach to develop this tool. First, a systematic literature search on the implementation and synthesis of clinical trial evidence was performed. The team then met in a series of subcommittees to develop an initial draft index. Drafts were presented at several research conferences and circulated to methodological experts in various health-related disciplines for feedback. The team systematically recorded, discussed, and incorporated all feedback into further revisions. A penultimate draft was discussed at the 2010 Cochrane-Campbell Collaboration Colloquium to finalize its content.

RESULTS

The Oxford Implementation Index provides a checklist of implementation data to extract from primary trials. Checklist items are organized into four domains: intervention design, actual delivery by trial practitioners, uptake of the intervention by participants, and contextual factors. Systematic reviewers piloting the index at the Cochrane-Campbell Colloquium reported that the index was helpful for the identification of implementation data.

CONCLUSION

The Oxford Implementation Index provides a framework to help reviewers assess implementation data across trials. Reviewers can use this tool to identify implementation data, extract relevant information, and compare features of implementation across primary trials in a systematic review. The index is a work-in-progress, and future efforts will focus on refining the index, improving usability, and integrating the index with other guidance on systematic reviewing.

Preventing deaths and injuries from house fires: an outcome evaluation of a community-based smoke alarm installation programme

BACKGROUND

Few studies have examined the impact of community-based smoke alarm (SA) distribution programmes on the occurrence of house fire-related deaths and injuries (HF-D/I).

OBJECTIVE

To determine whether the rate of HF-D/I differed for programme houses that had a SA installed through a community-based programme called Operation Installation, versus non-programme houses in the same census tracts that had not received such a SA.

METHODS

Teams of volunteers and firefighters canvassed houses in 36 high-risk target census tracts in Dallas, TX, between April 2001 and April 2011, and installed lithium-powered SAs in houses where residents were present and gave permission. We then followed incidence of HF-D/I among residents of the 8134 programme houses versus the 24 346 non-programme houses.

RESULTS

After a mean of 5.2 years of follow-up, the unadjusted HF-D/I rate was 68% lower among residents of programme houses versus non-programme houses (3.1 vs 9.6 per 100 000 population, respectively; rate ratio, 0.32; 95% CI 0.10 to 0.84). Multivariate analysis including several demographic variables showed that the adjusted HF-D/I rate in programme houses was 63% lower than non-programme houses. The programme was most effective in the first 5 years after SA installation, with declining difference in rates after the 6th year, probably due to SAs becoming non-functional during that time.

CONCLUSIONS

This collaborative, community-based SA installation programme was effective at preventing deaths and injuries from house fires, but the duration of effectiveness was less than 10 years.

Organizational issues in the implementation and adoption of health information technology innovations: an interpretative review

PURPOSE

Implementations of health information technologies are notoriously difficult, which is due to a range of inter-related technical, social and organizational factors that need to be considered. In the light of an apparent lack of empirically based integrated accounts surrounding these issues, this interpretative review aims to provide an overview and extract potentially generalizable findings across settings.

METHODS

We conducted a systematic search and critique of the empirical literature published between 1997 and 2010. In doing so, we searched a range of medical databases to identify review papers that related to the implementation and adoption of eHealth applications in organizational settings. We qualitatively synthesized this literature extracting data relating to technologies, contexts, stakeholders, and their inter-relationships.

RESULTS

From a total body of 121 systematic reviews, we identified 13 systematic reviews encompassing organizational issues surrounding health information technology implementations. By and large, the evidence indicates that there are a range of technical, social and organizational considerations that need to be deliberated when attempting to ensure that technological innovations are useful for both individuals and organizational processes. However, these dimensions are inter-related, requiring a careful balancing act of strategic implementation decisions in order to ensure that unintended consequences resulting from technology introduction do not pose a threat to patients.

CONCLUSIONS

Organizational issues surrounding technology implementations in healthcare settings are crucially important, but have as yet not received adequate research attention. This may in part be due to the subjective nature of factors, but also due to a lack of coordinated efforts toward more theoretically-informed work. Our findings may be used as the basis for the development of best practice guidelines in this area.

Identifying facilitators and barriers for home injury prevention interventions for pre-school children: a systematic review of the quantitative literature

Injuries are the leading cause of childhood death internationally; steep social gradients exist in mortality and morbidity. The majority of pre-school injuries occur in the home, but implementing research into practice for injury prevention has received little attention. This systematic review describes key facilitators and barriers when implementing injury prevention interventions. The review used articles included in a Cochrane systematic review of the effectiveness of home safety education, with or without the provision of safety equipment. Each paper was screened to ensure that children under 5 years, intervention details and process measures and/or barriers and facilitators were included. Two authors independently reviewed each paper and extracted data. Themes were identified and framework analysis used in an iterative process. Ninety-nine papers were identified, 42 excluded and 57 included in the analysis. Seven facilitators and six barriers were identified. Facilitators related to the approach used, focused messages, minimal changes, deliverer characteristics, equipment accessibility, behaviour change and including incentives. The barriers included complex interventions, cultural, socio-economic, physical and behavioural barriers and deliverer constraints. Barriers and facilitators should be addressed when implementing injury prevention interventions and studies should explicitly explore factors that help or hinder the process.

How do we know when research from one setting can be useful in another? A review of external validity, applicability and transferability frameworks

OBJECTIVE

To review published frameworks that included criteria for the assessment of external validity, applicability and transferability in their assessment of health research.

METHODS

Five databases were searched for articles relating to the assessment of external validity or applicability and transferability in health research. A coding framework was developed inductively and used to assess which types of criteria were included in the frameworks.

RESULTS

Thirty-eight articles describing 25 frameworks were identified. Eleven focused solely on the assessment of applicability and transferability; 14 presented more general decision-making or evidence appraisal frameworks. The criteria were synthesized into four main categories: setting, intervention, outcomes and evidence. None of the frameworks covered all the criteria identified. A major limitation was the lack of empirical data used to develop many frameworks and the apparent lack of assessment of their perceived utility.

CONCLUSION

A validated framework of applicability and transferability would help those aiming to encourage research use, as well as those conducting research. Greater understanding of applicability and transferability could help to encourage the appropriate use of research and the development of research that is more useful.

Protocol for a systematic review of the effects of schools and school-environment interventions on health: evidence mapping and syntheses

BACKGROUND

Schools may have important effects on students' and staff's health. Rather than treating schools merely as sites for health education, 'school-environment' interventions treat schools as settings which influence health. Evidence concerning the effects of such interventions has not been recently synthesised.

METHODS/DESIGN

Systematic review aiming to map and synthesise evidence on what theories and conceptual frameworks are most commonly used to inform school-environment interventions or explain school-level influences on health; what effects school-environment interventions have on health/health inequalities; how feasible and acceptable are school-environment interventions; what effects other school-level factors have on health; and through what processes school-level influences affect health.We will examine interventions aiming to promote health by modifying schools' physical, social or cultural environment via actions focused on school policies and practices relating to education, pastoral care and other aspects of schools beyond merely providing health education. Participants are staff and students age 4-18 years.We will review published research unrestricted by language, year or source. Searching will involve electronic databases including Embase, ERIC, PubMed, PsycInfo and Social Science Citation Index using natural-language phrases plus reference/citation checking.Stage 1 will map studies descriptively by focus and methods. Stage 2 will involve additional inclusion criteria, quality assessment and data extraction undertaken by two reviewers in parallel. Evidence will be synthesised narratively and statistically where appropriate (undertaking subgroup analyses and meta-regression and where no significant heterogeneity of effect sizes is found, pooling these to calculate a final effect size).

DISCUSSION

We anticipate: finding a large number of studies missed by previous reviews; that non-intervention studies of school effects examine a greater breadth of determinants than are addressed by intervention studies; and that intervention effect estimates are greater than for school-based health curriculum interventions without school-environment components.

Implementing successful intimate partner violence screening programs in health care settings: evidence generated from a realist-informed systematic review

We undertook a synthesis of existing studies to re-evaluate the evidence on program mechanisms of intimate partner violence (IPV) universal screening and disclosure within a health care context by addressing how, for whom, and in what circumstances these programs work. Our review is informed by a realist review approach, which focuses on program mechanisms. Systematic, realist reviews can help reveal why and how interventions work and can yield information to inform policies and programs. A review of the scholarly literature from January 1990 to July 2010 identified 5046 articles, 23 of which were included in our study. We identified studies on 17 programs that evaluated IPV screening. We found that programs that took a comprehensive approach (i.e., incorporated multiple program components, including institutional support) were successful in increasing IPV screening and disclosure/identification rates. Four program components appeared to increase provider self-efficacy for screening, including institutional support, effective screening protocols, thorough initial and ongoing training, and immediate access/referrals to onsite and/or offsite support services. These findings support a multi-component comprehensive IPV screening program approach that seeks to build provider self-efficacy for screening. Further implications for IPV screening intervention planning and implementation in health care settings are discussed.

Why isn't more injury prevention evidence-based?

The focus on evidence-based practice is critical to addressing the issue of injuries, yet advances in the science of injury prevention have not always led to advances in practice. Effective approaches are not always adopted, or when adopted and transferred from one setting to another, they do not always achieve expected results. These challenges were the basis of two breakout sessions at the second European Injury Control and Safety Promotion Conference in Paris, France (October 2008). In summarising the key issues raised during those sessions, this article describes what is meant by evidence-based practice, discusses why evidence-based practice tends not to occur and considers approaches that may facilitate the adoption and implementation of evidence-based strategies. To address the challenge, specific action is required, both on the part of the research community and those responsible for developing and implementing injury prevention policies and programmes.

Community-based services for homeless adults experiencing concurrent mental health and substance use disorders: a realist approach to synthesizing evidence

Consultations with community-based service providers in Toronto identified a lack of strong research evidence about successful community-based interventions that address the needs of homeless clients experiencing concurrent mental health and substance use disorders. We undertook a collaborative research effort between academic-based and community-based partners to conduct a systematic evidence synthesis drawing heavily from Pawson's realist review methodology to focus on both whether programs are successful and why and how they lead to improved outcomes. We examined scholarly and nonscholarly literature to explore program approaches and program elements that lead to improvements in mental health and substance use disorders among homeless individuals with concurrent disorders (CD). Information related to program contexts, elements, and successes and failures were extracted and further supplemented by key informant interviews and author communication regarding reviewed published studies. From the ten programs that we reviewed, we identified six important and promising program strategies that reduce mental health and, to a far lesser degree, substance use problems: client choice in treatment decision-making, positive interpersonal relationships between client and provider, assertive community treatment approaches, providing supportive housing, providing supports for instrumental needs, and nonrestrictive program approaches. These promising program strategies function, in part, by promoting and supporting autonomy among homeless adults experiencing CD. Our realist informed review is a useful methodology for synthesizing complex programming information on community-based interventions.

Case management for long-term conditions: implementation and processes

This paper presents a structured literature review that focused on comprehensive case management by nurses for adults with long-term conditions living in the community. The emphases of the review are the implementation of case-management approaches, including its roles, core tasks and components, and the coverage and quality of the reported implementation data. Twenty-nine studies were included: the majority were concerned with case management for frail older people, and others focused on people with multiple chronic diseases, high-cost patients, or those at high risk of hospital admissions. All the studies reported that case managers undertook the core tasks of assessment, care planning and the implementation of the care plan, but there was more variation in who carried out case finding, monitoring, review and case closure. Few studies provided adequate implementation information. On the basis of the reviewed evidence, three issues were identified as key to the coherent and sustainable implementation of case management for people with long-term conditions: fidelity to the core elements of case management; size of caseload; and case-management practice, incorporating matters relating to the continuity of care, the intensity and breadth of involvement, and control over resources. It is recommended that future evaluations of case-management interventions include a comprehensive process component or, at the very least, that interventions should be more fully described.

A translational framework for public health research

BACKGROUND

The paradigm of translational medicine that underpins frameworks such as the Cooksey report on the funding of health research does not adequately reflect the complex reality of the public health environment. We therefore outline a translational framework for public health research.

DISCUSSION

Our framework redefines the objective of translation from that of institutionalising effective interventions to that of improving population health by influencing both individual and collective determinants of health. It incorporates epidemiological perspectives with those of the social sciences, recognising that many types of research may contribute to the shaping of policy, practice and future research. It also identifies a pivotal role for evidence synthesis and the importance of non-linear and intersectoral interfaces with the public realm.

SUMMARY

We propose a research agenda to advance the field and argue that resources for 'applied' or 'translational' public health research should be deployed across the framework, not reserved for 'dissemination' or 'implementation'.

Population tobacco control interventions and their effects on social inequalities in smoking: systematic review

OBJECTIVE

To assess the effects of population tobacco control interventions on social inequalities in smoking.

DATA SOURCES

Medical, nursing, psychological, social science and grey literature databases, bibliographies, hand-searches and contact with authors.

STUDY SELECTION

Studies were included (n = 84) if they reported the effects of any population-level tobacco control intervention on smoking behaviour or attitudes in individuals or groups with different demographic or socioeconomic characteristics.

DATA EXTRACTION

Data extraction and quality assessment for each study were conducted by one reviewer and checked by a second.

DATA SYNTHESIS

Data were synthesised using graphical ("harvest plot") and narrative methods. No strong evidence of differential effects was found for smoking restrictions in workplaces and public places, although those in higher occupational groups may be more likely to change their attitudes or behaviour. Smoking restrictions in schools may be more effective in girls. Restrictions on sales to minors may be more effective in girls and younger children. Increasing the price of tobacco products may be more effective in reducing smoking among lower-income adults and those in manual occupations, although there was also some evidence to suggest that adults with higher levels of education may be more price-sensitive. Young people aged under 25 are also affected by price increases, with some evidence that boys and non-white young people may be more sensitive to price.

CONCLUSIONS

Population-level tobacco control interventions have the potential to benefit more disadvantaged groups and thereby contribute to reducing health inequalities.

A conceptual framework for implementation fidelity

BACKGROUND

Implementation fidelity refers to the degree to which an intervention or programme is delivered as intended. Only by understanding and measuring whether an intervention has been implemented with fidelity can researchers and practitioners gain a better understanding of how and why an intervention works, and the extent to which outcomes can be improved.

DISCUSSION

The authors undertook a critical review of existing conceptualisations of implementation fidelity and developed a new conceptual framework for understanding and measuring the process. The resulting theoretical framework requires testing by empirical research.

SUMMARY

Implementation fidelity is an important source of variation affecting the credibility and utility of research. The conceptual framework presented here offers a means for measuring this variable and understanding its place in the process of intervention implementation.

Evidence into practice: combining the art and science of injury prevention

OBJECTIVES

To bring together scientific evidence of what works in injury prevention with the knowledge and experience of practitioners, using a case study of smoke alarm installation from England.

DESIGN

There is good evidence of strategies to reduce injuries but less is known about the art of translating those strategies to implementation in real-world settings. England's Health Development Agency developed a structured process applicable to many public health fields, which integrates practitioner knowledge into the evidence base and reflects local contexts. The multistep process includes convening structured field meetings with local practitioners and policy makers, which focus on a mapping exercise of strategies, policies, targets, and funding streams related to childhood injury prevention, and barriers and facilitators relating to implementation of specific interventions.

SETTING

Meetings were held in six venues across England with 98 participants from a range of professional backgrounds and sectors.

RESULTS

The collective knowledge of participants provided many local insights unlikely to emerge in conventional research. Discussion topics covered key partners and sectors to include when planning a program; national policies and programs that could be used to drive the agenda; potential sources of funding; the importance of providing and installing appropriate smoke alarms; targeting of programs; and suggestions for gaining access to hard-to-reach populations.

CONCLUSION

This methodology represents an efficient way of gaining insight necessary for successful implementation of evidence based programs. It may be particularly useful in lower and middle income countries, serving to translate evidence into the local contexts and circumstances within which practitioners operate.

Reporting implementation in randomized trials: proposed additions to the consolidated standards of reporting trials statement

Randomized controlled trials of public health interventions are often complex: practitioners may not deliver interventions as researchers intended, participants may not initiate interventions and may not behave as expected, and interventions and their effects may vary with environmental and social context. Reports of randomized controlled trials can be misleading when they omit information about the implementation of interventions, yet such data are frequently absent in trial reports, even in journals that endorse current reporting guidelines. Particularly for complex interventions, the Consolidated Standards of Reporting Trials (CONSORT) statement does not include all types of information needed to understand the results of randomized controlled trials. CONSORT should be expanded to include more information about the implementation of interventions in all trial arms.

A systematic review of lay views about infant size and growth

OBJECTIVES

To understand lay views on infant size and growth and their implications for a British population.

METHODS

A systematic review of parental and other lay views about the meanings and importance of infant size and growth using Medline, PsycInfo, CINAHL, Sociological Abstracts, IBSS, ASSIA, British Nursing Index ChildData, Caredata, SIGLE, Dissertation Abstracts (US), Index to Theses. 19 studies, most of which reported the views of mothers, from the US, Canada, the UK and Finland were reviewed.

RESULTS

Notions of healthy size and growth were dominated by the concept of normality. Participants created norms by assessing and comparing size and growth against several reference points. When size or growth differed from these norms, explanations were sought for factors that would account for this difference. When no plausible explanation could be found, growth or size became a worry for parents.

CONCLUSIONS

Parents consider the importance of contextual factors when judging what is appropriate or healthy growth. For public health advice to be effective, lay, as well as scientific, findings and values need to be considered.

Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews

BACKGROUND

The inclusion of qualitative studies in systematic reviews poses methodological challenges. This paper presents worked examples of two methods of data synthesis (textual narrative and thematic), used in relation to one review, with the aim of enabling researchers to consider the strength of different approaches.

METHODS

A systematic review of lay perspectives of infant size and growth was conducted, locating 19 studies (including both qualitative and quantitative). The data extracted from these were synthesised using both a textual narrative and a thematic synthesis.

RESULTS

The processes of both methods are presented, showing a stepwise progression to the final synthesis. Both methods led us to similar conclusions about lay views toward infant size and growth. Differences between methods lie in the way they dealt with study quality and heterogeneity.

CONCLUSION

On the basis of the work reported here, we consider textual narrative and thematic synthesis have strengths and weaknesses in relation to different research questions. Thematic synthesis holds most potential for hypothesis generation, but may obscure heterogeneity and quality appraisal. Textual narrative synthesis is better able to describe the scope of existing research and account for the strength of evidence, but is less good at identifying commonality.

The role and theoretical evolution of knowledge translation and exchange in public health

BACKGROUND

There is an increased emphasis in public health research on effective models and strategies to support knowledge translation (KT), the exchange, synthesis and ethically sound application of research findings within a complex set of interactions among researchers and knowledge users. In other words, KT can be seen as an acceleration of the knowledge cycle-an acceleration of the natural transformation of knowledge into use (Canadian Institutes of Health Services Research. Knowledge Translation Strategy, 2004). The most recent conceptualizations consider the complexities of public health decision-making. The role of practitioners and communities is increasingly considered.

METHODS

We identify, describe and discuss the theoretical underpinnings of KT and recommend a way forward to build the evidence for more effective practice.

RESULTS

Theoretical perspectives increasingly influence research on KT in public health. A range of innovative work is being conducted to explore methods for KT using practical tools, often with the support of government.

CONCLUSIONS

KT describes a crucial and to date under-developed element of the research process. There is an important gap in theoretically informed empirical studies of effectiveness of proposed approaches in public health, health promotion and preventive medicine, and thus much of the debate remains abstract. There is clearly an urgent policy need to establish the effectiveness of KT models in a range of contexts. This must include both the consideration of development and the utilization of knowledge.

Extending systematic reviews to include evidence on implementation: Methodological work on a review of community-based initiatives to prevent injuries

Unintentional injury is a leading cause of mortality and disability among young and old. While evidence about the effectiveness of interventions in reducing injuries is accumulating, reviews of this evidence frequently fail to include details of implementation processes. Our research, of which the work reported here formed a part, had two main objectives: (1) to identify evidence about the implementation of interventions aimed at reducing unintentional injuries amongst children and young people; and (2) to explore methods for systematically reviewing evidence on implementation. Existing systematic reviews of the effectiveness of interventions aiming to reduce unintentional injuries in children and young people formed the starting point for the work reported here. In summary, many of the published papers we identified contained little information on implementation processes and, even when these were discussed, the extent to which authors' claims were based on research evidence was unclear. On the basis of the studies we reviewed implementation data were insufficiently strong to provide a sound evidence base for practitioners and policymakers. Notwithstanding this, we identified valuable data about the context in which such initiatives are implemented and the type of factors that might impinge on implementation. This work has implications in three areas: (1) researchers with an interest in evidence-based public health could be encouraged to consider implementation issues in the design of intervention studies; (2) funding bodies could be encouraged to prioritise intervention studies using mixed methods that will enable researchers to consider effectiveness and implementation; (3) journal editors could work towards increasing the quality of reporting on implementation issues through the development of guidelines.