Using logic model methods in systematic review synthesis: describing complex pathways in referral management interventions

Cochrane Qualitative and Implementation Methods Group guidance series-paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews

The Cochrane Qualitative and Implementation Methods Group develops and publishes guidance on the synthesis of qualitative and mixed-method evidence from process evaluations. Despite a proliferation of methods for the synthesis of qualitative research, less attention has focused on how to integrate these syntheses within intervention effectiveness reviews. In this article, we report updated guidance from the group on approaches, methods, and tools, which can be used to integrate the findings from quantitative studies evaluating intervention effectiveness with those from qualitative studies and process evaluations. We draw on conceptual analyses of mixed methods systematic review designs and the range of methods and tools that have been used in published reviews that have successfully integrated different types of evidence. We outline five key methods and tools as devices for integration which vary in terms of the levels at which integration takes place; the specialist skills and expertise required within the review team; and their appropriateness in the context of limited evidence. In situations where the requirement is the integration of qualitative and process evidence within intervention effectiveness reviews, we recommend the use of a sequential approach. Here, evidence from each tradition is synthesized separately using methods consistent with each tradition before integration takes place using a common framework. Reviews which integrate qualitative and process evaluation evidence alongside quantitative evidence on intervention effectiveness in a systematic way are rare. This guidance aims to support review teams to achieve integration and we encourage further development through reflection and formal testing.

Increasing value and reducing waste by optimizing the development of complex interventions: Enriching the development phase of the Medical Research Council (MRC) Framework

BACKGROUND

In recent years there has been much emphasis on 'research waste' caused by poor question selection, insufficient attention to previous research results, and avoidable weakness in research design, conduct and analysis. Little attention has been paid to the effect of inadequate development of interventions before proceeding to a full clinical trial.

OBJECTIVE

We therefore propose to enrich the development phase of the MRC Framework by adding crucial elements to improve the likelihood of success and enhance the fit with clinical practice METHODS: Based on existing intervention development guidance and synthesis, a comprehensive iterative intervention development approach is proposed. Examples from published reports are presented to illustrate the methodology that can be applied within each element to enhance the intervention design.

RESULTS

A comprehensive iterative approach is presented by combining the elements of the MRC Framework development phase with essential elements from existing guidance including: problem identification, the systematic identification of evidence, identification or development of theory, determination of needs, the examination of current practice and context, modelling the process and expected outcomes leading to final element: the intervention design. All elements are drawn from existing models to provide intervention developers with a greater chance of producing an intervention that is well adopted, effective and fitted to the context.

CONCLUSION

This comprehensive approach of developing interventions will strengthen the internal and external validity, minimize research waste and add value to health care research. In complex interventions in health care research, flaws in the development process immediately impact the chances of success. Knowledge regarding the causal mechanisms and interactions within the intended clinical context is needed to develop interventions that fit daily practice and are beneficial for the end-user.

Intervention planning for a digital intervention for self-management of hypertension: a theory-, evidence- and person-based approach

Background

This paper describes the intervention planning process for the Home and Online Management and Evaluation of Blood Pressure (HOME BP), a digital intervention to promote hypertension self-management. It illustrates how a Person-Based Approach can be integrated with theory- and evidence-based approaches. The Person-Based Approach to intervention development emphasises the use of qualitative research to ensure that the intervention is acceptable, persuasive, engaging and easy to implement.

Methods

Our intervention planning process comprised two parallel, integrated work streams, which combined theory-, evidence- and person-based elements. The first work stream involved collating evidence from a mixed methods feasibility study, a systematic review and a synthesis of qualitative research. This evidence was analysed to identify likely barriers and facilitators to uptake and implementation as well as design features that should be incorporated in the HOME BP intervention. The second work stream used three complementary approaches to theoretical modelling: developing brief guiding principles for intervention design, causal modelling to map behaviour change techniques in the intervention onto the Behaviour Change Wheel and Normalisation Process Theory frameworks, and developing a logic model.

Results

The different elements of our integrated approach to intervention planning yielded important, complementary insights into how to design the intervention to maximise acceptability and ease of implementation by both patients and health professionals. From the primary and secondary evidence, we identified key barriers to overcome (such as patient and health professional concerns about side effects of escalating medication) and effective intervention ingredients (such as providing in-person support for making healthy behaviour changes). Our guiding principles highlighted unique design features that could address these issues (such as online reassurance and procedures for managing concerns). Causal modelling ensured that all relevant behavioural determinants had been addressed, and provided a complete description of the intervention. Our logic model linked the hypothesised mechanisms of action of our intervention to existing psychological theory.

Conclusion

Our integrated approach to intervention development, combining theory-, evidence- and person-based approaches, increased the clarity, comprehensiveness and confidence of our theoretical modelling and enabled us to ground our intervention in an in-depth understanding of the barriers and facilitators most relevant to this specific intervention and user population.

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-017-0553-4) contains supplementary material, which is available to authorized users.

Factors that impact on the use of mechanical ventilation weaning protocols in critically ill adults and children: a qualitative evidence-synthesis

BACKGROUND

Prolonged mechanical ventilation is associated with a longer intensive care unit (ICU) length of stay and higher mortality. Consequently, methods to improve ventilator weaning processes have been sought. Two recent Cochrane systematic reviews in ICU adult and paediatric populations concluded that protocols can be effective in reducing the duration of mechanical ventilation, but there was significant heterogeneity in study findings. Growing awareness of the benefits of understanding the contextual factors impacting on effectiveness has encouraged the integration of qualitative evidence syntheses with effectiveness reviews, which has delivered important insights into the reasons underpinning (differential) effectiveness of healthcare interventions.

OBJECTIVES

1. To locate, appraise and synthesize qualitative evidence concerning the barriers and facilitators of the use of protocols for weaning critically-ill adults and children from mechanical ventilation;2. To integrate this synthesis with two Cochrane effectiveness reviews of protocolized weaning to help explain observed heterogeneity by identifying contextual factors that impact on the use of protocols for weaning critically-ill adults and children from mechanical ventilation;3. To use the integrated body of evidence to suggest the circumstances in which weaning protocols are most likely to be used.

SEARCH METHODS

We used a range of search terms identified with the help of the SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) mnemonic. Where available, we used appropriate methodological filters for specific databases. We searched the following databases: Ovid MEDLINE, Embase, OVID, PsycINFO, CINAHL Plus, EBSCOHost, Web of Science Core Collection, ASSIA, IBSS, Sociological Abstracts, ProQuest and LILACS on the 26th February 2015. In addition, we searched: the grey literature; the websites of professional associations for relevant publications; and the reference lists of all publications reviewed. We also contacted authors of the trials included in the effectiveness reviews as well as of studies (potentially) included in the qualitative synthesis, conducted citation searches of the publications reporting these studies, and contacted content experts.We reran the search on 3rd July 2016 and found three studies, which are awaiting classification.

SELECTION CRITERIA

We included qualitative studies that described: the circumstances in which protocols are designed, implemented or used, or both, and the views and experiences of healthcare professionals either involved in the design, implementation or use of weaning protocols or involved in the weaning of critically-ill adults and children from mechanical ventilation not using protocols. We included studies that: reflected on any aspect of the use of protocols, explored contextual factors relevant to the development, implementation or use of weaning protocols, and reported contextual phenomena and outcomes identified as relevant to the effectiveness of protocolized weaning from mechanical ventilation.

DATA COLLECTION AND ANALYSIS

At each stage, two review authors undertook designated tasks, with the results shared amongst the wider team for discussion and final development. We independently reviewed all retrieved titles, abstracts and full papers for inclusion, and independently extracted selected data from included studies. We used the findings of the included studies to develop a new set of analytic themes focused on the barriers and facilitators to the use of protocols, and further refined them to produce a set of summary statements. We used the Confidence in the Evidence from Reviews of Qualitative Research (CERQual) framework to arrive at a final assessment of the overall confidence of the evidence used in the synthesis. We included all studies but undertook two sensitivity analyses to determine how the removal of certain bodies of evidence impacted on the content and confidence of the synthesis. We deployed a logic model to integrate the findings of the qualitative evidence synthesis with those of the Cochrane effectiveness reviews.

MAIN RESULTS

We included 11 studies in our synthesis, involving 267 participants (one study did not report the number of participants). Five more studies are awaiting classification and will be dealt with when we update the review.The quality of the evidence was mixed; of the 35 summary statements, we assessed 17 as 'low', 13 as 'moderate' and five as 'high' confidence. Our synthesis produced nine analytical themes, which report potential barriers and facilitators to the use of protocols. The themes are: the need for continual staff training and development; clinical experience as this promotes felt and perceived competence and confidence to wean; the vulnerability of weaning to disparate interprofessional working; an understanding of protocols as militating against a necessary proactivity in clinical practice; perceived nursing scope of practice and professional risk; ICU structure and processes of care; the ability of protocols to act as a prompt for shared care and consistency in weaning practice; maximizing the use of protocols through visibility and ease of implementation; and the ability of protocols to act as a framework for communication with parents.

AUTHORS' CONCLUSIONS

There is a clear need for weaning protocols to take account of the social and cultural environment in which they are to be implemented. Irrespective of its inherent strengths, a protocol will not be used if it does not accommodate these complexities. In terms of protocol development, comprehensive interprofessional input will help to ensure broad-based understanding and a sense of 'ownership'. In terms of implementation, all relevant ICU staff will benefit from general weaning as well as protocol-specific training; not only will this help secure a relevant clinical knowledge base and operational understanding, but will also demonstrate to others that this knowledge and understanding is in place. In order to maximize relevance and acceptability, protocols should be designed with the patient profile and requirements of the target ICU in mind. Predictably, an under-resourced ICU will impact adversely on protocol implementation, as staff will prioritize management of acutely deteriorating and critically-ill patients.

Health Information Technologies-Academic and Commercial Evaluation (HIT-ACE) methodology: description and application to clinical feedback systems

Background

Health information technologies (HIT) have become nearly ubiquitous in the contemporary healthcare landscape, but information about HIT development, functionality, and implementation readiness is frequently siloed. Theory-driven methods of compiling, evaluating, and integrating information from the academic and commercial sectors are necessary to guide stakeholder decision-making surrounding HIT adoption and to develop pragmatic HIT research agendas. This article presents the Health Information Technologies—Academic and Commercial Evaluation (HIT-ACE) methodology, a structured, theory-driven method for compiling and evaluating information from multiple sectors. As an example demonstration of the methodology, we apply HIT-ACE to mental and behavioral health measurement feedback systems (MFS). MFS are a specific class of HIT that support the implementation of routine outcome monitoring, an evidence-based practice.

Results

HIT-ACE is guided by theories and frameworks related to user-centered design and implementation science. The methodology involves four phases: (1) coding academic and commercial materials, (2) developer/purveyor interviews, (3) linking putative implementation mechanisms to hit capabilities, and (4) experimental testing of capabilities and mechanisms. In the current demonstration, phase 1 included a systematic process to identify MFS in mental and behavioral health using academic literature and commercial websites. Using user-centered design, implementation science, and feedback frameworks, the HIT-ACE coding system was developed, piloted, and used to review each identified system for the presence of 38 capabilities and 18 additional characteristics via a consensus coding process. Bibliometic data were also collected to examine the representation of the systems in the scientific literature. As an example, results are presented for the application of HIT-ACE phase 1 to MFS wherein 49 separate MFS were identified, reflecting a diverse array of characteristics and capabilities.

Conclusions

Preliminary findings demonstrate the utility of HIT-ACE to represent the scope and diversity of a given class of HIT beyond what can be identified in the academic literature. Phase 2 data collection is expected to confirm and expand the information presented and phases 3 and 4 will provide more nuanced information about the impact of specific HIT capabilities. In all, HIT-ACE is expected to support adoption decisions and additional HIT development and implementation research.

To what extent do site-based training, mentoring, and operational research improve district health system management and leadership in low- and middle-income countries: a systematic review protocol

BACKGROUND

District health managers play a key role in the effectiveness of decentralized health systems in low- and middle-income countries. Inadequate management and leadership skills often hamper their ability to improve quality of care and effectiveness of health service delivery. Nevertheless, significant investments have been made in capacity-building programmes based on site-based training, mentoring, and operational research. This systematic review aims to review the effectiveness of site-based training, mentoring, and operational research (or action research) on the improvement of district health system management and leadership. Our secondary objectives are to assess whether variations in composition or intensity of the intervention influence its effectiveness and to identify enabling and constraining contexts and underlying mechanisms.

METHODS

We will search the following databases: MEDLINE, PsycInfo, Cochrane Library, CRD database (DARE), Cochrane Effective Practice and Organisation of Care (EPOC) group, ISI Web of Science, Health Evidence.org, PDQ-Evidence, ERIC, EMBASE, and TRIP. Complementary search will be performed (hand-searching journals and citation and reference tracking). Studies that meet the following PICO (Population, Intervention, Comparison, Outcome) criteria will be included: P: professionals working at district health management level; I: site-based training with or without mentoring, or operational research; C: normal institutional arrangements; and O: district health management functions. We will include cluster randomized controlled trials, controlled before-and-after studies, interrupted time series analysis, quasi-experimental designs, and cohort and longitudinal studies. Qualitative research will be included to contextualize findings and identify barriers and facilitators. Primary outcomes that will be reported are district health management and leadership functions. We will assess risk of bias with the Cochrane Collaboration's tools for randomized controlled trials (RCT) and non RCT studies and Critical Appraisal Skills Programme checklists for qualitative studies. We will assess strength of recommendations with the GRADE tool for quantitative studies, and the CERQual approach for qualitative studies. Synthesis of quantitative studies will be performed through meta-analysis when appropriate. Best fit framework synthesis will be used to synthesize qualitative studies.

DISCUSSION

This protocol paper describes a systematic review assessing the effectiveness of site-based training (with or without mentoring programmes or operational research) on the improvement of district health system management and leadership.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42015032351.

Developing and Optimising the Use of Logic Models in Systematic Reviews: Exploring Practice and Good Practice in the Use of Programme Theory in Reviews

BACKGROUND

Logic models are becoming an increasingly common feature of systematic reviews, as is the use of programme theory more generally in systematic reviewing. Logic models offer a framework to help reviewers to 'think' conceptually at various points during the review, and can be a useful tool in defining study inclusion and exclusion criteria, guiding the search strategy, identifying relevant outcomes, identifying mediating and moderating factors, and communicating review findings.

METHODS AND FINDINGS

In this paper we critique the use of logic models in systematic reviews and protocols drawn from two databases representing reviews of health interventions and international development interventions. Programme theory featured only in a minority of the reviews and protocols included. Despite drawing from different disciplinary traditions, reviews and protocols from both sources shared several limitations in their use of logic models and theories of change, and these were used almost unanimously to solely depict pictorially the way in which the intervention worked. Logic models and theories of change were consequently rarely used to communicate the findings of the review.

CONCLUSIONS

Logic models have the potential to be an aid integral throughout the systematic reviewing process. The absence of good practice around their use and development may be one reason for the apparent limited utility of logic models in many existing systematic reviews. These concerns are addressed in the second half of this paper, where we offer a set of principles in the use of logic models and an example of how we constructed a logic model for a review of school-based asthma interventions.

Intervention Component Analysis (ICA): a pragmatic approach for identifying the critical features of complex interventions

BACKGROUND

In order to enable replication of effective complex interventions, systematic reviews need to provide evidence about their critical features and clear procedural details for their implementation. Currently, few systematic reviews provide sufficient guidance of this sort.

METHODS

Through a worked example, this paper reports on a methodological approach, Intervention Component Analysis (ICA), specifically developed to bridge the gap between evidence of effectiveness and practical implementation of interventions. By (a) using an inductive approach to explore the nature of intervention features and (b) making use of trialists' informally reported experience-based evidence, the approach is designed to overcome the deficiencies of poor reporting which often hinders knowledge translation work whilst also avoiding the need to invest significant amounts of time and resources in following up details with authors.

RESULTS

A key strength of the approach is its ability to reveal hidden or overlooked intervention features and barriers and facilitators only identified in practical application of interventions. It is thus especially useful where hypothesised mechanisms in an existing programme theory have failed. A further benefit of the approach is its ability to identify potentially new configurations of components that have not yet been evaluated.

CONCLUSIONS

ICA is a formal and rigorous yet relatively streamlined approach to identify key intervention content and implementation processes. ICA addresses a critical need for knowledge translation around complex interventions to support policy decisions and evidence implementation.

Conceptual decomposition of complex health care interventions for evidence synthesis: a literature review

RATIONALE, AIMS AND OBJECTIVES

The clarity of pivotal concepts is an important prerequisite for the development, evaluation and exchange of scientific ideas. The term 'complex intervention' is increasingly used in the health care literature, although it often remains unclear what is actually meant by this concept. Therefore, our aim was to analyse the literature regarding definitions of the terms 'complex intervention' and 'components' of such interventions.

METHOD

To identify the methodological publications, systematic and snowballing techniques were combined for the literature search. Relevant units of meaning were extracted from 68 included publications. Afterwards, we deduced categories and related frequencies by inductive and quantitative content analysis techniques.

RESULTS

Several types of complexity were distinguished in the literature. Most authors viewed complex interventions as multicomponent interventions that are characterized by interactions between the components themselves, with the context or as systemic interventions. Components of complex interventions were described in the publications as having the potential to causally influence outcomes, thus being essential for achieving an effect. Other definitions and inconsistencies among the definitions are highlighted and discussed.

CONCLUSIONS

From our synthesis, we derived definitions of the central health care-related concepts 'complex intervention' and 'components' of an intervention. Although we found numerous diverse definitions, they could be reduced to a defined number of core characteristics. These characteristics may facilitate communication regarding complex interventions and enable the deduction of methodological approaches for evidence synthesis.

How to build up the actionable knowledge base: the role of 'best fit' framework synthesis for studies of improvement in healthcare

Increasing recognition of the role and value of theory in improvement work in healthcare offers the prospect of capitalising upon, and consolidating, actionable lessons from synthesis of improvement projects and initiatives. We propose that informed use of theory can (i) provide a mechanism by which to collect and organise data from a body of improvement work, (ii) offer a framework for analysis and identification of lessons learnt and (iii) facilitate an evaluation of the feasibility, effectiveness and acceptability of improvement programmes. Improvement practitioners can benefit from using an underpinning external structure as a lens by which to examine the specific achievements of their own projects alongside comparable initiatives led by others. We demonstrate the utility of a method known as ‘best fit framework synthesis’ (BFFS) in offering a ubiquitous and versatile means by which to collect, analyse and evaluate improvement work in healthcare. First reported in 2011, BFFS represents a pragmatic, flexible approach to integrating theory with findings from practice. A deductive phase, where a review team seeks to accommodate a substantial part of the data, is followed by an inductive phase, in which the team explores data not accommodated by the framework. We explore the potential for BFFS within improvement work by drawing upon the evidence synthesis methodology literature and practical examples of improvement work reported in BMJ Quality and Safety (2011–2015). We suggest four variants of BFFS that may have particular value in synthesising a body of improvement work. We conclude that BFFS, alongside other approaches that seek to optimise the contribution of theory to improvement work, represents one important enabling mechanism by which to establish the rigour and scientific credentials of the emerging discipline of ‘improvement science’.

Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable?

BACKGROUND

In recognising the potential value of theory in understanding how interventions work comes a challenge - how to make identification of theory less haphazard?

OBJECTIVES

To explore the feasibility of systematic identification of theory.

METHOD

We searched PubMed for published reviews (1998-2012) that had explicitly sought to identify theory. Systematic searching may be characterised by a structured question, methodological filters and an itemised search procedure. We constructed a template (BeHEMoTh - Behaviour of interest; Health context; Exclusions; Models or Theories) for use when systematically identifying theory. The authors tested the template within two systematic reviews.

RESULTS

Of 34 systematic reviews, only 12 reviews (35%) reported a method for identifying theory. Nineteen did not specify how they identified studies containing theory. Data were unavailable for three reviews. Candidate terms include concept(s)/conceptual, framework(s), model(s), and theory/theories/theoretical. Information professionals must overcome inadequate reporting and the use of theory out of context. The review team faces an additional concern in lack of 'theory fidelity'.

CONCLUSIONS

Based on experience with two systematic reviews, the BeHEMoTh template and procedure offers a feasible and useful approach for identification of theory. Applications include realist synthesis, framework synthesis or review of complex interventions. The procedure requires rigorous evaluation.

New Approaches for Moving Upstream

Growing evidence shows that unequal distribution of wealth and power across race, class, and gender produces the differences in living conditions that are “upstream” drivers of health inequalities. Health educators and other public health professionals, however, still develop interventions that focus mainly on “downstream” behavioral risks. Three factors explain the difficulty in translating this knowledge into practice. First, in their allegiance to the status quo, powerful elites often resist upstream policies and programs that redistribute wealth and power. Second, public health practice is often grounded in dominant biomedical and behavioral paradigms, and health departments also face legal and political limits on expanding their scope of activities. Finally, the evidence for the impact of upstream interventions is limited, in part because methodologies for evaluating upstream interventions are less developed. To illustrate strategies to overcome these obstacles, we profile recent campaigns in the United States to enact living wages, prevent mortgage foreclosures, and reduce exposure to air pollution. We then examine how health educators working in state and local health departments can transform their practice to contribute to campaigns that reallocate the wealth and power that shape the living conditions that determine health and health inequalities. We also consider health educators’ role in producing the evidence that can guide transformative expansion of upstream interventions to reduce health inequalities.

Referral interventions from primary to specialist care: a systematic review of international evidence

BACKGROUND

Demand management defines any method used to monitor, direct, or regulate patient referrals. Strategies have been developed to manage the referral of patients to secondary care, with interventions that target primary care, specialist services, or infrastructure.

AIM

To review the international evidence on interventions to manage referral from primary to specialist care.

DESIGN AND SETTING

Systematic review.

METHOD

Iterative, systematic searches of published and unpublished sources public health, health management, management, and grey literature databases from health care and other industries were undertaken to identify recent, relevant studies. A narrative synthesis of the data was completed to structure the evidence into groups of similar interventions.

RESULTS

The searches generated 8327 unique results, of which 140 studies were included. Interventions were grouped into four intervention categories: GP education (n = 50); process change (n = 49); system change (n = 38); and patient-focused (n = 3). It is clear that there is no 'magic bullet' to managing demand for secondary care services: although some groups of interventions may have greater potential for development, given the existing evidence that they can be effective in specific contexts.

CONCLUSIONS

To tackle demand management of primary care services, the focus cannot be on primary care alone; a whole-systems approach is needed because the introduction of interventions in primary care is often just the starting point of the referral process. In addition, more research is needed to develop and evaluate interventions that acknowledge the role of the patient in the referral decision.