Assessing the applicability of public health intervention evaluations from one setting to another: a methodological study of the usability and usefulness of assessment tools and frameworks

A Systematic Review of Tools to Assess Coeliac Disease-Related Knowledge

Background: Coeliac disease (CD) is an immune-mediated disorder, with dietary exclusion of gluten the only current treatment. A good knowledge of CD and gluten-free diet (GFD) is essential for those with CD to support effective self-management. Knowledge assessment with a validated tool helps evaluate understanding and knowledge gaps to better tailor educational resources. This study's aim was to perform a systematic review to identify validated CD knowledge assessment tools. Methods: PRISMA guidelines were followed, and searches were carried out in five literature databases. Papers were reviewed for tool development and testing process and assessed against pre-defined criteria for feasibility, validity, and reliability. Results: Twenty-five papers were included in the final analysis. Studies were from 16 countries, with a range of target populations, study designs, and development processes. Eleven reported pilot testing, and five assessed readability. Content validity was assessed in ten papers and formal content validity testing in one. Many tools contained items affecting generalisability outside the region developed. Conclusions: For a CD knowledge assessment tool to be suitable for use, it needs to be well designed, tested, and generalisable. No papers identified satisfied all requirements, thus highlighting a need to develop an appropriate tool.

The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations

Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of "evidence-based" psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts.

Assessing Healthcare Simulation Facilitation: A Scoping Review of Available Tools, Validity Evidence, and Context Suitability for Faculty Development in Low-Resource Settings

SUMMARY STATEMENT

Assessment tools support simulation facilitation skill development by guiding practice, structuring feedback, and promoting reflective learning among educators. This scoping review followed a systematic process to identify facilitation assessment tools used in postlicensure healthcare simulation. Secondary objectives included mapping of the validity evidence to support their use and a critical appraisal of their suitability for simulation faculty development in low-resource settings. Database searching, gray literature searching, and stakeholder engagement identified 11,568 sources for screening, of which 72 met criteria for full text review. Thirty sources met inclusion; 16 unique tools were identified. Tools exclusively originated from simulation practice in high-resource settings and predominantly focused on debriefing. Many tools have limited validity evidence supporting their use. In particular, the validity evidence supporting the extrapolation and implications of assessment is lacking. No current tool has high context suitability for use in low-resource settings.

A qualitative study of a recovery capital assessment tool in alcohol and drug treatment facilities: Perspectives from social work professionals

The ambition to translate the concept of recovery capital (RC) from research to practice has entailed operationalisations of RC to measure progress and outcomes of alcohol and other drug (AOD) treatments.At the same time, the role played by standardisation in social work organisations is known to be a complex matter, and research suggests that the implementation of standardisation should be carried out with caution. By examining the need for a RC assessment tool in a Swedish AOD treatment context, this paper contributes to the discussion on the use of assessment tools for measuring treatment progress and outcome in AOD treatments, as well as interrelations between standardisation aims and social work professionals. Four group interviews with employees at AOD treatment facilities were conducted in Stockholm, Sweden. The interviewees were presented with an example of an RC-based assessment tool and asked to review the applicability of the tool in their daily work. The findings provide insights on the applicability of specific assessment list items, as well as general observations on the complex relationship between standardisation and discretion in social work. The findings suggest that professionals will ultimately rely on their knowledge and experience, and act accordingly to support the service user, regardless of any manual or standardisation that regulates their work.

Underreporting contextual factors preclude the applicability appraisal in primary care randomized controlled trials

OBJECTIVES

To assess applicability reporting in randomized controlled trials (RCTs) conducted in primary care (PC).

STUDY DESIGN AND SETTING

We used a random sample of PC RCTs published between 2000 and 2020 to assess applicability. We extracted data related to setting, population, intervention (including implementation), comparator, outcomes, and context. Based on data availability, we assessed whether the five predefined applicability questions were adequately addressed by each PC RCT.

RESULTS

Adequately described elements that were reported frequently (>50%) included the responsible organization for intervention provision (97, 93.3%), study population characteristics (94, 90.4%), intervention implementation including monitoring and evaluation (92, 88.5%), intervention components (89, 85.6%), time frame (82, 78.8%), baseline prevalence (58, 55.8%), and the type of setting and location (53, 51%). Elements that were often underreported included contextual factors, that is, evidence of differential effects across sociodemographic or other groupings (2, 1.9%), intervention components tailored for specific settings (7, 6.7%), health system structure (32, 30.8%), factors affecting implementation (40, 38.5%) and organization structure (50, 48.1%). The proportion of trials that adequately addressed each applicability question ranged between 1% and 20.2%, while none RCT could address all of them.

CONCLUSION

Underreporting contextual factors jeopardize the appraisal of applicability in PC RCTs.

Characteristics of the sources, evaluation, and grading of the certainty of evidence in systematic reviews in public health: A methodological study

Objectives

To systematically explore how the sources of evidence, types of primary studies, and tools used to assess the quality of the primary studies vary across systematic reviews (SRs) in public health.

Methods

We conducted a methodological survey of SRs in public health by searching the of literature in selected journals from electronic bibliographic databases. We selected a 10% random sample of the SRs that met the explicit inclusion criteria. Two researchers independently extracted data for analysis.

Results

We selected 301 SRs for analysis: 94 (31.2%) of these were pre-registered, and 211 (70.1%) declared to have followed published reporting standard. All SRs searched for evidence in electronic bibliographic databases, and more than half (n = 180, 60.0%) searched also the references of the included studies. The common types of primary studies included in the SRs were primarily cross-sectional studies (n = 132, 43.8%), cohort studies (n = 126, 41.9%), randomized controlled trials (RCTs, n = 89, 29.6%), quasi-experimental studies (n = 83, 27.6%), case-control studies (n = 58, 19.3%) qualitative studies (n = 38, 12.6%) and mixed-methods studies (n = 32, 10.6%). The most frequently used quality assessment tools were the Newcastle-Ottawa Scale (used for 50.0% of cohort studies and 55.6% of case-control studies), Cochrane Collaboration's Risk of Bias tool (50.7% of RCTs) and Critical Appraisal Skills Program (38.5% of qualitative studies). Only 20 (6.6%) of the SRs assessed the certainty of the body of evidence, of which 19 (95.0%) used the GRADE approach. More than 65% of the evidence in the SRs using GRADE was of low or very low certainty.

Conclusions

SRs should always assess the quality both at the individual study level and the body of evidence for outcomes, which will benefit patients, health care practitioners, and policymakers.

Use of existing systematic reviews for the development of evidence-based vaccination recommendations: Guidance from the SYSVAC expert panel

National immunization technical advisory groups (NITAGs) develop immunization-related recommendations and assist policy-makers in making evidence informed decisions. Systematic reviews (SRs) that summarize the available evidence on a specific topic are a valuable source of evidence in the development of such recommendations. However, conducting SRs requires significant human, time, and financial resources, which many NITAGs lack. Given that SRs already exist for many immunization-related topics, and to prevent duplication and overlap of reviews, a more practical approach may be for NITAGs to use existing SRs. Nevertheless, it can be challenging to identify relevant SRs, to select one SR from among multiple SRs, or to critically assess and effectively use them. To support NITAGs, the London School of Hygiene and Tropical Medicine, Robert Koch Institute and collaborators developed the SYSVAC project, which consists of an online registry of systematic reviews on immunization-related topics and an e-learning course, that supports the use of them (both freely accessible at https://www.nitag-resource.org/sysvac-systematic-reviews). Drawing from the e-learning course and recommendations from an expert panel, this paper outlines methods for using existing systematic reviews when making immunization-related recommendations. With specific examples and reference to the SYSVAC registry and other resources, it offers guidance on locating existing systematic reviews; assessing their relevance to a research question, up-to-dateness, and methodological quality and/or risk of bias; and considering the transferability and applicability of their findings to other populations or settings.

A Framework for Applying Global Learning to Improve Primary Health Care in the United States

Background and objectives

Lessons from global health have long informed efforts to improve primary health care (PHC) in the United States (US). Despite this history, no generalizable framework exists to guide US stakeholders in the identification and application of ideas from abroad related to the key PHC components of community engagement and service delivery. We sought to develop such a framework.

Methods

We reviewed the experience of Global to Local, a community-based organization (CBO) founded with a mission to apply global health strategies to improve the health in vulnerable populations in the US, and examined the experience of care delivery organizations in the US that have successfully implemented global-to-local solutions. Based on that experience, and supported by the advice of an expert panel, we developed a framework for applying global learning to improve US PHC.

Findings

The framework includes six change concepts under three broad categories. The first category focuses on the need to actively and intentionally incorporate a global perspective in organizational program design and improvement activities. The second category addresses approaches to identifying global solutions related to community engagement and to health service delivery. The third category focuses on adaptation and implementation of lessons from global health in domestic contexts by applying relevant insights from dissemination and implementation science and diffusion of innovation theory.

Conclusions

In the absence of a robust literature providing implementation guidance to US health systems and CBOs open to adopting or adapting PHC strategies and practices from other countries, the proposed framework synthesizing the experience of organizations that have done so can inform efforts to apply lessons from global health to improve PHC in the US.

User experiences of structured stakeholder engagement to consider transferability: The TRANSFER approach

BACKGROUND

Systematic reviews are increasingly used to inform decision-making in health, education, social care and environmental protection. However, decision makers still experience barriers to using reviews, including not knowing how findings might translate to their own contexts, and lack of collaboration with systematic review authors. The TRANSFER approach is a novel method that aims to support review authors to systematically and transparently collaborate with stakeholders to consider context and the transferability of review findings from the beginning of the review process. Such collaboration is intended to improve the usefulness and relevance of review findings for decision makers.

OBJECTIVES

We aim to explore the user experience of the TRANSFER approach conversation guide, and in doing so gain a better understanding of the role and perceived value of stakeholder engagement in systematic reviews for informed decision-making.

METHODS

We conducted four user tests of groups using the guide, organized around simulated meetings between review authors and stakeholders. Review authors led the meeting using the TRANSFER approach conversation guide. We audio-recorded and observed the meetings, collected feedback forms and conducted semi-structured interviews with review authors following the meeting. We analysed the data using framework analysis to examine the user experience of the TRANSFER approach conversation guide and of stakeholder engagement more generally.

RESULTS

Seventeen participants in four user groups participated in the user tests. Most participants were generally positive toward the structured approach using the conversation guide, and felt it would be useful in systematic review projects. We observed examples of misunderstanding of the terminology included in the guide, and received multiple suggestions for how to make the conversation guide more user friendly. We observed numerous challenges related to the hypothetical nature of a user test, including lack of familiarity with the review question/topic among participants and lack of preparation for the meeting.

CONCLUSIONS

Review authors and stakeholders are positive toward using a structured approach to guide collaboration within the context of a systematic review. The TRANSFER conversation guide helps participants to discuss the review question and context in a structured way. Such structured collaboration could help to improve the usefulness and relevance of systematic reviews for decision making by improving the review question, inclusion criteria and consideration of transferability of review findings. The conversation guide needs to be modified to improve user experience. Further research is needed to explore stakeholder collaboration and the use of the TRANSFER conversation guide in systematic review processes.

Program adaptation by health departments

Introduction

The dissemination of evidence-based interventions (i.e., programs, practices, and policies) is a core function of US state health departments (SHDs). However, interventions are originally designed and tested with a specific population and context. Hence, adapting the intervention to meet the real-world circumstances and population's needs can increase the likelihood of achieving the expected health outcomes for the target population from the implemented intervention. This study identified how SHD employees decide to adapt public health programs and what influences decisions on how to adapt them.

Materials and methods

SHD employees (n = 45) were interviewed using a qualitative semi-structured interview guide. Telephone interviews were audio-recorded and transcribed verbatim. The transcripts were consensus-coded and themes were identified using thematic analysis. Several themes aligned with the Model for Adaptation Design and Impact.

Results

Data, outcomes, and health department evaluations influenced decisions to adapt a program (pre-adaptation), and reasons to adapt a program included organizational and sociopolitical contextual factors. SHD middle-level managers, program managers and staff, and local agencies were involved in the decisions to adapt the programs. Finally, the goals for adapting a program included enhancing effectiveness/outcomes, reach and satisfaction with the program; funding; and partner engagement. After SHD employees decided to adapt a program, data and evidence guided the changes. Program staff and evaluators were engaged in the adaptation process. Program managers consulted partners to gather ideas on how best to adapt a program based on partners' experiences implementing the program and obtaining community input. Lastly, program managers also received input on adapting content and context from coalition meetings and periodic technical assistance calls.

Discussion

The findings related to decisions to adapt public health programs provide practitioners with considerations for adapting them. Findings reaffirm the importance of promoting public health competencies in program evaluation and adaptation, as well as systematically documenting and evaluating the adaptation processes. In addition, the themes could be studied in future research as mechanisms, mediators, and moderators to implementation outcomes.

Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties

Background

Internal and external validity are the most relevant components when critically appraising randomized controlled trials (RCTs) for systematic reviews. However, there is no gold standard to assess external validity. This might be related to the heterogeneity of the terminology as well as to unclear evidence of the measurement properties of available tools. The aim of this review was to identify tools to assess the external validity of RCTs. It was further, to evaluate the quality of identified tools and to recommend the use of individual tools to assess the external validity of RCTs in future systematic reviews.

Methods

A two-phase systematic literature search was performed in four databases: PubMed, Scopus, PsycINFO via OVID, and CINAHL via EBSCO. First, tools to assess the external validity of RCTs were identified. Second, studies investigating the measurement properties of these tools were selected. The measurement properties of each included tool were appraised using an adapted version of the COnsensus based Standards for the selection of health Measurement INstruments (COSMIN) guidelines.

Results

38 publications reporting on the development or validation of 28 included tools were included. For 61% (17/28) of the included tools, there was no evidence for measurement properties. For the remaining tools, reliability was the most frequently assessed property. Reliability was judged as “sufficient” for three tools (very low certainty of evidence). Content validity was rated as “sufficient” for one tool (moderate certainty of evidence).

Conclusions

Based on these results, no available tool can be fully recommended to assess the external validity of RCTs in systematic reviews. Several steps are required to overcome the identified difficulties to either adapt and validate available tools or to develop a better suitable tool.

Trial registration

Prospective registration at Open Science Framework (OSF): 10.17605/OSF.IO/PTG4D.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01561-5.

Interventions to increase linkage to care and adherence to treatment for hepatitis C among people who inject drugs: A systematic review and practical considerations from an expert panel consultation

•

Integrated care and cooperation between service providers optimize the HCV care continuum among people who inject drugs.

•

Results suggest that people who inject drugs with HCV infection can be effectively linked and treated with direct-acting antivirals regimens in settings outside of hospital.

•

Interventions that facilitate HCV care must be implemented at settings where people who inject drugs are already accessing services.

•

The experts' reflections complement the findings of the literature review and inform public health practice by considering the heterogeneity of health systems and national regulatory frameworks.

•

Higher quality studies investigating interventions addressing the entire care cascade from testing to cure and prevention of reinfections among highly vulnerable populations are urgently needed.

Background

Methods

Results

Conclusions

Appraising clinical applicability of studies: mapping and synthesis of current frameworks, and proposal of the FrACAS framework and VICORT checklist

BACKGROUND

Not all research findings are translated to clinical practice. Reasons for lack of applicability are varied, and multiple frameworks and criteria exist to appraise the general applicability of epidemiological and clinical research. In this two-part study, we identify, map, and synthesize frameworks and criteria; we develop a framework to assist clinicians to appraise applicability specifically from a clinical perspective.

METHODS

We conducted a literature search in PubMed and Embase to identify frameworks appraising applicability of study results. Conceptual thematic analysis was used to synthesize frameworks and criteria. We carried out a framework development process integrating contemporary debates in epidemiology, findings from the literature search and synthesis, iterative pilot-testing, and brainstorming and consensus discussions to propose a concise framework to appraise clinical applicability.

RESULTS

Of the 4622 references retrieved, we identified 26 unique frameworks featuring 21 criteria. Frameworks and criteria varied by scope and level of aggregation of the evidence appraised, target user, and specific area of applicability (internal validity, clinical applicability, external validity, and system applicability). Our proposed Framework Appraising the Clinical Applicability of Studies (FrACAS) classifies studies in three domains (research, practice informing, and practice changing) by examining six criteria sequentially: Validity, Indication-informativeness, Clinical relevance, Originality, Risk-benefit comprehensiveness, and Transposability (VICORT checklist).

CONCLUSIONS

Existing frameworks to applicability vary by scope, target user, and area of applicability. We introduce FrACAS to specifically assess applicability from a clinical perspective. Our framework can be used as a tool for the design, appraisal, and interpretation of epidemiological and clinical studies.

A Rapid Review of Interventions to Prevent First Pregnancy among Adolescents and Its Applicability to Latin America

STUDY OBJECTIVE

To summarize recent literature on the effectiveness of interventions to prevent adolescent pregnancy and to explore the applicability of these interventions to Latin America (LA). Design, Setting, Participants, Interventions, and Main Outcome Measures: We carried out a rapid review of the literature (2005-2019). Studies were included if: they evaluated interventions targeting adolescents and prevention of pregnancy; they used a randomized controlled design; and pregnancy was measured as an outcome. Applicability of the interventions to LA was assessed using the following information: target population; intervention design and resources; type, skills, and training of providers; system arrangements; and acceptability and social context.

RESULTS

Nine studies were included, 5 described interventions in African countries, 2 in the United Kingdom, and 2 in the United States. Interventions were rated as highly applicable to LA in the context of target population, profile of the providers, and design; however, variations arose when assessing system arrangements and social context. Incentive-based interventions showed significant effects in the prevention of adolescent pregnancy and were rated as highly applicable.

CONCLUSION

This review provides professionals, policymakers, researchers, and educators potential criteria to consider when adapting successful evidence-based interventions to prevent adolescent pregnancy in LA.

Cultural adaptation of two school-based smoking prevention programs in Bogotá, Colombia

Smoking prevention among adolescents is a public health challenge that is even more significant in low- and middle-income countries where local evidence is limited and smoking rates remain high. Evidence-based interventions could be transferred to low- and middle-income country settings but only after appropriate cultural adaptation. This paper aims to describe the process of the cultural adaptation of two school-based smoking prevention interventions, A Stop Smoking in Schools Trial and Dead Cool, to be implemented in Bogotá, Colombia. A recognized heuristic framework guided the cultural adaptation through five stages. We conducted a concurrent nested mixed-methods study consisting of a qualitative descriptive case study and a quantitative pre- and post quasi-experiment without a control. Contextual, content, training, and implementation modifications were made to the programs to address cultural factors, to maintain the fidelity of implementation, and to increase the pupils' engagement with the programs. Modifications incorporated the suggestions of stakeholders, the original developers, and local community members, whilst considering the feasibility of delivering the programs. Involving stakeholders, original program developers, and community members in the cultural adaptation of evidence-based interventions is essential to properly adapt them to the local context, and to maintain the fidelity of program implementation.

Implementation of evidence-based health promotion and disease prevention interventions: theoretical and practical implications of the concept of transferability for decision-making and the transfer process

Evidence-based health promotion and disease prevention require incorporating evidence of the effectiveness of interventions into policy and practice. With the entry into force of the German Act to Strengthen Health Promotion and Prevention (PrävG), interventions that take place in people’s everyday living environments have gained in importance. Decision-makers need to assess whether an evidence-based intervention is transferable to their specific target context. The Federal Centre for Health Education (BZgA) recommends that transferability of an intervention should be clarified before any decision to implement it. Furthermore, transferability needs to be finally determined after an evaluation in the target context. In this article, we elaborate on theoretical and practical implications of the concept of transferability for health promotion and disease prevention based on the Population–Intervention–Environment–Transfer Models of Transferability (PIET-T). We discuss how decision-makers can anticipate transferability prior to the intervention transfer with the help of transferability criteria and how they can take transferability into account in the further process. This includes the steps of the analysis of a health problem and identification of effective interventions, the steps of the initial transferability assessment and identification of the need for adaptation, and the steps of the implementation and evaluation. Considering transferability is a complex task that comes with challenges. But it offers opportunities to select a suitable intervention for a target context and, in the transfer process, to understand the conditions under which the intervention works in this context. This knowledge helps to establish an evidence base, which is practically relevant.

Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

BACKGROUND

Usability - the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction - may be a key determinant of implementation success. However, few instruments have been developed to measure the design quality of complex health interventions (i.e., those with several interacting components). This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale (SUS) for digital technologies, to measure the usability of a leading complex psychosocial intervention, Motivational Interviewing (MI), for behavioral health service delivery in primary care. Prior SUS studies have found both one- and two-factor solutions, both of which were examined in the current study of the IUS.

METHOD

A survey administered to 136 medical professionals from 11 primary care sites collected demographic information and IUS ratings for MI, the evidence-based psychosocial intervention that primary care providers reported using most often for behavioral health service delivery. Factor analyses replicated procedures used in prior research on the SUS.

RESULTS

Analyses indicated that a two-factor solution (with "usable" and "learnable" subscales) best fit the data, accounting for 54.1% of the variance. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .84, and α = .67, respectively.

CONCLUSIONS

This study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability. Implications for implementation research evaluating the usability of complex health interventions are discussed, including the potential for future comparisons across multiple interventions and provider types, as well as the use of the IUS to evaluate the relationship between usability and implementation outcomes such as feasibility.

How can we adapt complex population health interventions for new contexts? Progressing debates and research priorities

INTRODUCTION

The UK Medical Research Council and National Institute for Health Research have funded the ADAPT study (2018-2020), to develop methodological guidance for the adaptation of complex population health interventions for new contexts. While there have been advances in frameworks, there are key theoretical and methodological debates to progress. The ADAPT study convened a panel meeting to identify and enrich these debates. This paper presents the panel's discussions and suggests directions for future research.

METHODS

Sixteen researchers and one policymaker convened for a 1-day meeting in July 2019. The aim was to reflect on emerging study findings (systematic review of adaptation guidance; scoping review of case examples; and qualitative interviews with funders, journal editors, researchers and policymakers), progress theoretical and methodological debates, and consider where innovation may be required to address research gaps.

DISCUSSION

Despite the proliferation of adaptation frameworks, questions remain over the definition of basic concepts (eg, adaptation). The rationale for adaptation, which often focuses on differences between contexts, may lead to adaptation hyperactivity. Equal emphasis should be placed on similarities. Decision-making about intervention modification currently privileges the concept of 'core components', and work is needed to progress the use and operationalisation of 'functional fidelity'. Language and methods must advance to ensure meaningful engagement with diverse stakeholders in adaptation processes. Further guidance is required to assess the extent of re-evaluation required in the new context. A better understanding of different theoretical perspectives, notably complex systems thinking, implementation science and realist evaluation may help in enhancing research on adaptation.

Assessing context suitability (generalizability, external validity, applicability or transferability) of findings in evidence syntheses in healthcare-An integrative review of methodological guidance

BACKGROUND

Evidence syntheses provide the basis for evidence-based decision making in healthcare. To judge the certainty of findings for the specific decision context evidence syntheses should consider context suitability (ie, generalizability, external validity, applicability or transferability). Our objective was to determine the status quo and to provide a comprehensive overview of existing methodological recommendations of Health Technology Assessment (HTA) and Systematic Review (SR) producing organizations in assessing context suitability of evidence on effectiveness of health care interventions. Additionally, we analyzed similarities and differences between the recommendations.

METHODS

In this Integrative Review we performed a structured search for methods documents from evidence synthesis producing organizations that include recommendations on appraising context suitability in effectiveness assessments. Two reviewers independently selected documents according to predefined eligibility criteria. Data were extracted in standardized and piloted tables by one reviewer and verified by a second reviewer. We performed a thematic analysis to identify and summarize the main themes and categories regarding recommended context suitability assessments.

RESULTS

We included 14 methods documents of 12 organizations in our synthesis. Assessment approaches are very heterogeneous both regarding the general concepts (eg, integration in the evidence synthesis preparation process) and the content of assessments (eg, assessment criteria).

CONCLUSION

Some heterogeneity seems to be justified because of the need to tailor the assessment to different settings and medical areas. However, most differences were inexplicable. More harmonization is desirable and appears possible.

Evaluating Public Health Interventions: A Neglected Area in Health Technology Assessment

Introduction: Public health (PH) interventions are crucial for ensuring sustainable healthcare services. Nevertheless, they represent a neglected area in the field of health technology assessment (HTA) due to various methodological issues and their complex design that goes beyond clinical setting. The present study provides an environmental scan of HTA initiatives related to the assessment of PH technologies on a global level.

Methods: We conducted a cross-sectional survey among 85 HTA-related European and international societies, health bodies, and networks from September 2018 to January 2019. The questionnaire contained four sections and 18 questions regarding activities related to the evaluation of PH technologies, information on existing PH technologies, and methodologies of assessment as well as barriers and facilitators to reaching a decision and implementing a PH technology.

Results: Among 52 survey responses, the majority of the respondents came from European countries (35%), followed by North American (27%), and South American (19%) countries. The main type of organizations covered by our survey included HTA agencies, public administrations, and research institutes. Seventy-one % of the institutions reported engagement in any aspect of HTA in the area of PH (N = 37). Among those, 81% evaluated less than 5 PH technologies from 2013 to 2018. The most common barriers for reaching a decision on PH technologies were lack of data, conflicting stakeholder priorities, and methodological issues. A total of 76 PH interventions were reported, and most cited initiatives were related to chronic disease screening, prevention of infectious diseases, and maternal, prenatal, and neonatal screening.

Conclusion: Our survey reported a rather limited involvement of HTA in the evaluation of PH technologies. In particular, an evaluation of behavioral and lifestyle interventions remains extremely rare. The implementation of collaborative HTA approaches in the setting of PH practice and policy needs to be prioritized and further strengthened. Moreover, ensuring reliable data structures and consolidation of HTA methods for the evaluation of PH technologies will be crucial for tackling the enormous burden of non-communicable diseases in societies.

When assessing generalisability, focusing on differences in population or setting alone is insufficient

Generalisability is typically only briefly mentioned in discussion sections of evaluation articles, which are unhelpful in judging whether an intervention could be implemented elsewhere, with similar effects. Several tools to assess generalisability exist, but they are difficult to operationalise and are rarely used. We believe a different approach is needed. Instead of focusing on similarities (or more likely, differences) in generic population and setting characteristics, generalisability assessments should focus on understanding an intervention’s mechanism of action - why or how an intervention was effective. We believe changes are needed to four types of research. First, outcome evaluations should draw on programme theory. Second, process evaluations should aim to understand interventions’ mechanism of action, rather than simply ‘what happened’. Third, small scoping studies should be conducted in new settings, to explore how to enact identified mechanisms. Finally, innovative synthesis methods are required, in order to identify mechanisms of action where there is a lack of existing process evaluations.

The TRANSFER Approach for assessing the transferability of systematic review findings

BACKGROUND

Systematic reviews are a key input to health and social welfare decisions. Studies included in systematic reviews often vary with respect to contextual factors that may impact on how transferable review findings are to the review context. However, many review authors do not consider the transferability of review findings until the end of the review process, for example when assessing confidence in the evidence using GRADE or GRADE-CERQual. This paper describes the TRANSFER Approach, a novel approach for supporting collaboration between review authors and stakeholders from the beginning of the review process to systematically and transparently consider factors that may influence the transferability of systematic review findings.

METHODS

We developed the TRANSFER Approach in three stages: (1) discussions with stakeholders to identify current practices and needs regarding the use of methods to consider transferability, (2) systematic search for and mapping of 25 existing checklists related to transferability, and (3) using the results of stage two to develop a structured conversation format which was applied in three systematic review processes.

RESULTS

None of the identified existing checklists related to transferability provided detailed guidance for review authors on how to assess transferability in systematic reviews, in collaboration with decision makers. The content analysis uncovered seven categories of factors to consider when discussing transferability. We used these to develop a structured conversation guide for discussing potential transferability factors with stakeholders at the beginning of the review process. In response to feedback and trial and error, the TRANSFER Approach has developed, expanding beyond the initial conversation guide, and is now made up of seven stages which are described in this article.

CONCLUSIONS

The TRANSFER Approach supports review authors in collaborating with decision makers to ensure an informed consideration, from the beginning of the review process, of the transferability of the review findings to the review context. Further testing of TRANSFER is needed.

Adapting evidence-informed complex population health interventions for new contexts: a systematic review of guidance

BACKGROUND

Adapting interventions that have worked elsewhere can save resources associated with developing new interventions for each specific context. While a developing body of evidence shows benefits of adapted interventions compared with interventions transported without adaptation, there are also examples of interventions which have been extensively adapted, yet have not worked in the new context. Decisions on when, to what extent, and how to adapt interventions therefore are not straightforward, particularly when conceptualising intervention effects as contingent upon contextual interactions in complex systems. No guidance currently addresses these questions comprehensively. To inform development of an overarching guidance on adaptation of complex population health interventions, this systematic review synthesises the content of the existing guidance papers.

METHODS

We searched for papers published between January 2000 and October 2018 in 7 bibliographic databases. We used citation tracking and contacted authors and experts to locate further papers. We double screened all the identified records. We extracted data into the following categories: descriptive information, key concepts and definitions, rationale for adaptation, aspects of adaptation, process of adaptation, evaluating and reporting adapted interventions. Data extraction was conducted independently by two reviewers, and retrieved data were synthesised thematically within pre-specified and emergent categories.

RESULTS

We retrieved 6694 unique records. Thirty-eight papers were included in the review representing 35 sources of guidance. Most papers were developed in the USA in the context of implementing evidence-informed interventions among different population groups within the country, such as minority populations. We found much agreement on how the papers defined key concepts, aims, and procedures of adaptation, including involvement of key stakeholders, but also identified gaps in scope, conceptualisation, and operationalisation in several categories.

CONCLUSIONS

Our review found limitations that should be addressed in future guidance on adaptation. Specifically, future guidance needs to be reflective of adaptations in the context of transferring interventions across countries, including macro- (e.g. national-) level interventions, better theorise the role of intervention mechanisms and contextual interactions in the replicability of effects and accordingly conceptualise key concepts, such as fidelity to intervention functions, and finally, suggest evidence-informed strategies for adaptation re-evaluation and reporting.

TRIAL REGISTRATION

PROSPERO 2018, CRD42018112714.

Influence of external contextual factors on the implementation of health and social care interventions into practice within or across countries-a protocol for a 'best fit' framework synthesis

BACKGROUND

The widespread implementation of interventions is often hindered by a decline and variability in effectiveness across implementation sites. It is anticipated that variations in the characteristics of the external context in different sites, such as the political and funding environment, socio-cultural context, physical environment or population demographics can influence implementation outcome. However, there is only a limited understanding about which and how external contextual factors influence implementation. We aim to develop a comprehensive framework conceptualising the influence of external contextual factors on implementation, particularly when spreading health and social care interventions within or across countries.

METHODS

The review will use the 'best fit' framework synthesis approach. In the first stage of the review, we will examine existing frameworks, models, concepts and theories on external contextual factors and their influence on implementation from a variety of sectors and disciplines including health and social care, education, environmental studies and international development fields. The resulting a priori meta-framework will be tested and refined in the second review stage by analysing evidence from empirical studies focusing on the implementation of health and social care interventions within or across countries. Searches will be conducted in bibliographic databases such as MEDLINE, ERIC, HMIC and IBSS, grey literature sources and on relevant websites. We will also search reference lists, relevant journals, perform citation searches and ask experts in the field. There is no restriction to study type, setting, intervention type or implementation strategy to enable obtaining a broad and in-depth knowledge from various sources of evidence.

DISCUSSION

The review will lead to a comprehensive framework for understanding the influence of external contextual factors on implementation, particularly when spreading health and social care interventions within or across countries. The framework is anticipated to help identify factors explaining the decline and variability in effectiveness of interventions and assessing the prospects of implementation effectiveness, when spreading interventions. We do not intend to only develop another stand-alone implementation framework but one that can be used in conjunction with existing frameworks. The framework can be honed and validated in future empirical research.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42018084485.

Health professional-delivered obesity prevention interventions during the first 1,000 days: A systematic review of external validity reporting

Background: Childhood obesity prevention interventions delivered by health professionals during the first 1,000 days show some evidence of effectiveness, particularly in relation to behavioural outcomes. External validity refers to how generalisable interventions are to populations or settings beyond those in the original study. The degree to which external validity elements are reported in such studies is unclear however. This systematic review aimed to determine the extent to which childhood obesity interventions delivered by health professionals during the first 1,000 days report on elements that can be used to inform generalizability across settings and populations. Methods: Eligible studies meeting study inclusion and exclusion criteria were identified through a systematic review of 11 databases and three trial registers. An assessment tool based on the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework was used to assess the external validity of included studies. It comprised five dimensions: reach and representativeness of individuals, reach and representativeness of settings, implementation and adaptation, outcomes for decision making maintenance and/or institutionalisation. Two authors independently assessed the external validity of 20% of included studies; discrepancies were resolved, and then one author completed assessments of the remaining studies. Results: In total, 39 trials involving 46 interventions published between 1999 and 2019 were identified. The majority of studies were randomized controlled trials (n=24). Reporting varied within and between dimensions. External validity elements that were poorly described included: representativeness of individuals and settings, treatment receipt, intervention mechanisms and moderators, cost effectiveness, and intervention sustainability and acceptability. Conclusions: Our review suggests that more emphasis is needed on research designs that consider generalisability, and the reporting of external validity elements in early life childhood obesity prevention interventions. Important gaps in external validity reporting were identified that could facilitate decisions around the translation and scale-up of interventions from research to practice. Registration: PROSPERO CRD42016050793 03/11/16.

Health professional-delivered obesity prevention interventions during the first 1,000 days: A systematic review of external validity reporting

Background: Childhood obesity prevention interventions delivered by health professionals during the first 1,000 days show some evidence of effectiveness, particularly in relation to behavioural outcomes. External validity refers to how generalisable interventions are to populations or settings beyond those in the original study. The degree to which external validity elements are reported in such studies is unclear however. This systematic review aimed to determine the extent to which childhood obesity interventions delivered by health professionals during the first 1,000 days report on elements that can be used to inform generalizability across settings and populations. Methods: Eligible studies meeting study inclusion and exclusion criteria were identified through a systematic review of 11 databases and three trial registers. An assessment tool based on the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework was used to assess the external validity of included studies. It comprised five dimensions: reach and representativeness of individuals, reach and representativeness of settings, implementation and adaptation, outcomes for decision making maintenance and/or institutionalisation. Two authors independently assessed the external validity of 20% of included studies; discrepancies were resolved, and then one author completed assessments of the remaining studies. Results: In total, 39 trials involving 46 interventions published between 1999 and 2019 were identified. The majority of studies were randomized controlled trials (n=24). Reporting varied within and between dimensions. External validity elements that were poorly described included: representativeness of individuals and settings, treatment receipt, intervention mechanisms and moderators, cost effectiveness, and intervention sustainability and acceptability. Conclusions: Our review suggests that more emphasis is needed on research designs that consider generalisability, and the reporting of external validity elements in early life childhood obesity prevention interventions. Important gaps in external validity reporting were identified that could facilitate decisions around the translation and scale-up of interventions from research to practice. Registration: PROSPERO CRD42016050793 03/11/16.

Towards greater understanding of implementation during systematic reviews of complex healthcare interventions: the framework for implementation transferability applicability reporting (FITAR)

Background

There have been calls for greater consideration of applicability and transferability in systematic reviews, to improve their usefulness in informing policy and practice. Understanding how evidence is, or is not applicable and transferable to varying local situations and contexts, is a key challenge for systematic review synthesis in healthcare. Assessing applicability and transferability in systematic reviews is reported to be difficult, particularly in reviews of complex interventions. There is a need for exploration of factors perceived to be important by policy-makers, and for further guidance on which items should be reported. In this paper we focus on the process of development of a framework that can be used by systematic reviewers to identify and report data across studies relating to applicability and transferability.

Methods

The framework was developed by scrutinising existing literature on applicability and transferability, examining data during a systematic review of highly complex changes to health service delivery, and was informed by stakeholder engagement. The items of the framework were thus grounded in both data identified during a real review, and stakeholder input. The paper describes examples of data identified using the framework during a review of integrated care interventions, and outlines how it informed analysis and reporting of the review findings.

Results

The Framework for Implementation Transferability Applicability Reporting (FITAR) comprises 44 items which can be used to structure analysis and reporting across studies during systematic reviews of complex interventions. The framework prompts detailed consideration of contextual data during extraction and reporting, within areas of: patient type and populations; type of organisations and systems; financial and commissioning processes; systems leadership elements; features of services; features of the workforce; and finally elements of the interventions/initiatives.

Conclusions

Use of the framework during our review of complex healthcare interventions helped the review team to surface contextual data, which may not be commonly extracted, analysed and reported. Further exploration and evaluation of systems for identifying and reporting these factors during reviews is required.

Electronic supplementary material

The online version of this article (10.1186/s12874-019-0723-y) contains supplementary material, which is available to authorized users.

[Study Designs for Effect Evaluation in Disease Prevention and Health Promotion - Meaning of Internal and External Validity in Intervention Studies]

BACKGROUND

Health promotion and disease prevention in Germany have become more important as a result of the Prevention Act that gives special attention to the setting. So far, however, there is neither a common understanding of the terms of this approach, nor are its potentials empirically derived. Therefore, a discussion about suitable evaluation and study concepts is required.

AIMS AND METHODS

The aims of this study were to address the meaning of internal and external validity of intervention studies in health promotion and disease prevention. We provide an overview of the achievable bias control for different study designs and provide examples for the evaluation of setting-based approaches.

RESULTS

Interventions in settings are often characterized by a multitude of measures, actors and intervention contexts. Methods of analysis from evidence-based medicine are viewed critically for health promotion and disease prevention in Germany. Such studies are considered to provide a high degree of internal validity, but the extrapolation of the results to "reality" is viewed as low. In contrast, the extrapolation of study results is not more of a challenge for setting projects than for any other research area. It is not limited by different contexts, but rather by different causal relationships. Impact assessment aims at causally attributing an observed outcome to the intervention. Thus, the epistemological requirements do not differ between studies that are designed for internal or external validity. The international discussion focuses on the refinement of (quasi)-experimental study designs. Examples for the evaluation of setting projects, mainly from Germany, illustrate that those alternative methods have already been used in evaluation practice.

CONCLUSION

A challenge for health promotion and disease prevention in settings is to systematically assess the different needs for evidence-basing. At present, there is a wealth of findings in a large number of intervention fields. These findings must be compiled and analyzed to determine whether and to what extent further evaluations need to be initiated and by which methods.

How can additional secondary data analysis of observational data enhance the generalisability of meta-analytic evidence for local public health decision making?

This paper critically explores how survey and routinely collected data could aid in assessing the generalisability of public health evidence. We propose developing approaches that could be employed in understanding the relevance of public health evidence, and investigate ways of producing meta-analytic estimates tailored to reflect local circumstances, based on analyses of secondary data. Currently, public health decision makers face challenges in interpreting global review evidence to assess its meaning in local contexts. A lack of clarity on the definition and scope of generalisability, and the absence of consensus on its measurement, has stunted methodological progress. The consequence of failing to tackle generalisability means that systematic review evidence often fails to fulfil its potential contribution in public health decision making. Three approaches to address these problems are considered and emerging challenges discussed: (1) purposeful exploration after a review has been conducted, and we present a framework of potential avenues of enquiry and a worked example; (2) recalibration of the results to weight studies differentially based on their similarity to conditions in an inference population, and we provide a worked example using UK Census data to understand potential differences in the effectiveness of community engagement interventions among sites in England and Wales; (3) purposeful exploration before starting a review to ensure that the findings are relevant to an inference population. The paper aims to demonstrate how a more nuanced treatment of context in reviews of public health interventions could be achieved through greater engagement with existing large sources of secondary data.