Assessing the applicability of findings in systematic reviews of complex interventions can enhance the utility of reviews for decision making

The prevalence and effects of treatments of rapidly progressive interstitial lung disease of dermatomyositis/polymyositis adults: A systematic review and meta-analysis

Rapidly progressive interstitial lung disease (RP-ILD) clearly harms the prognoses of dermatomyositis/polymyositis (DM/PM) patients, however there is a dearth of numerical prevalence and therapy comparison in this field. Therefore, the purpose of this study was to determine the prevalence of RP-ILD in DM/PM patients and compare prognoses, including remission rate and survival data, between treatments. Studies with reports of RP-ILD in DM/PM patients and studies with definite remission and/or survival data of DM/PM-RP-ILD were included in the study. Data sources were Pubmed, Embase, and Cochrane Library without language restrictions. Two authors (WHL and WWQ) extracted independently the data. Estimates of the pooled effects were calculated using the Mantel-Haenszel technique (random effects). The prevalence meta-analysis included 18 papers with 6058 DM/PM patients, and 31 papers were analyzed for treatment effects, including remission rate, 6-month survival rate, 1-year survival rate, and 5-year survival rate. Database search yielded 1816 articles. In the DM/PM population, the combined prevalence of RP-ILD was 8.9% (95% CI, 5.8% to 12.1%). Patients with RP-ILD have a remission rate of 58.4% (95% CI, 47.3% to 69.4%), with biologic treatment with the highest remission rate, followed by triple therapy (defined as adding a third intravenous medication, including cyclophosphamide and immunoglobulin). Biologics therapy had the highest overall survival rate at six months (95% CI, 49.8% to 73.9%), followed by cDMARDs, plasma exchange, and triple therapy. The 1-year survival rate was 77.4% (95% CI, 66.7% to 88.1%), and triple therapy and cDMARDs had the best survival rates. The 5-year survival rate was 40.0% (95% CI, 10.0% to 69.9%). The prevalence of RP-ILD in DM/PM was approximately 8.9%, with a poor long-term prognosis. The use of biological agents appears to provide the best therapeutic outcomes, providing RP-ILD management with a novel evidence-based therapy. The use of strong immunosuppressive treatments may result in life-threatening side effects, thus clinicians must closely monitor the condition.

Underreporting contextual factors preclude the applicability appraisal in primary care randomized controlled trials

OBJECTIVES

To assess applicability reporting in randomized controlled trials (RCTs) conducted in primary care (PC).

STUDY DESIGN AND SETTING

We used a random sample of PC RCTs published between 2000 and 2020 to assess applicability. We extracted data related to setting, population, intervention (including implementation), comparator, outcomes, and context. Based on data availability, we assessed whether the five predefined applicability questions were adequately addressed by each PC RCT.

RESULTS

Adequately described elements that were reported frequently (>50%) included the responsible organization for intervention provision (97, 93.3%), study population characteristics (94, 90.4%), intervention implementation including monitoring and evaluation (92, 88.5%), intervention components (89, 85.6%), time frame (82, 78.8%), baseline prevalence (58, 55.8%), and the type of setting and location (53, 51%). Elements that were often underreported included contextual factors, that is, evidence of differential effects across sociodemographic or other groupings (2, 1.9%), intervention components tailored for specific settings (7, 6.7%), health system structure (32, 30.8%), factors affecting implementation (40, 38.5%) and organization structure (50, 48.1%). The proportion of trials that adequately addressed each applicability question ranged between 1% and 20.2%, while none RCT could address all of them.

CONCLUSION

Underreporting contextual factors jeopardize the appraisal of applicability in PC RCTs.

User experiences of structured stakeholder engagement to consider transferability: The TRANSFER approach

Background

Systematic reviews are increasingly used to inform decision-making in health, education, social care and environmental protection. However, decision makers still experience barriers to using reviews, including not knowing how findings might translate to their own contexts, and lack of collaboration with systematic review authors. The TRANSFER approach is a novel method that aims to support review authors to systematically and transparently collaborate with stakeholders to consider context and the transferability of review findings from the beginning of the review process. Such collaboration is intended to improve the usefulness and relevance of review findings for decision makers.

Objectives

We aim to explore the user experience of the TRANSFER approach conversation guide, and in doing so gain a better understanding of the role and perceived value of stakeholder engagement in systematic reviews for informed decision-making.

Methods

We conducted four user tests of groups using the guide, organized around simulated meetings between review authors and stakeholders. Review authors led the meeting using the TRANSFER approach conversation guide. We audio-recorded and observed the meetings, collected feedback forms and conducted semi-structured interviews with review authors following the meeting. We analysed the data using framework analysis to examine the user experience of the TRANSFER approach conversation guide and of stakeholder engagement more generally.

Results

Seventeen participants in four user groups participated in the user tests. Most participants were generally positive toward the structured approach using the conversation guide, and felt it would be useful in systematic review projects. We observed examples of misunderstanding of the terminology included in the guide, and received multiple suggestions for how to make the conversation guide more user friendly. We observed numerous challenges related to the hypothetical nature of a user test, including lack of familiarity with the review question/topic among participants and lack of preparation for the meeting.

Conclusions

Review authors and stakeholders are positive toward using a structured approach to guide collaboration within the context of a systematic review. The TRANSFER conversation guide helps participants to discuss the review question and context in a structured way. Such structured collaboration could help to improve the usefulness and relevance of systematic reviews for decision making by improving the review question, inclusion criteria and consideration of transferability of review findings. The conversation guide needs to be modified to improve user experience. Further research is needed to explore stakeholder collaboration and the use of the TRANSFER conversation guide in systematic review processes.

Considerations for transferability of health technology assessments: a scoping review of tools, methods, and practices

Health technology assessment (HTA) is commonly used to guide evidence-informed decisions to optimize resource use, prioritize policies, and support countries to achieve universal health coverage. Producing HTAs requires time, scientific expertise, and political commitment, but these are not available in all settings - especially in low- and middle-income countries (LMIC) where HTA processes may be less institutionalized. Transferring and adapting existing HTAs to local settings may offer a solution while reducing duplication efforts. This scoping review aims to provide an overview of tools, methods, approaches, and considerations which can aid HTA transfers. We systematically searched (from 2005 to 2020) six databases and, using predefined inclusion criteria, included twenty-two studies. Data extraction followed a structured process, while synthesis was more iterative. We identified a common approach for HTA transfers. It follows the de novo process of undertaking original HTAs, but with additional steps to assess relevance (applicability), quality, and transferability, as well as steps to adapt parameters where necessary. The EUnetHTA Adaptation Toolkit was the only tool that provided guidance for adapting multiple HTA domains. Other tools were specific to systematic reviews (n = 1) or economic evaluations (n = 12), where one provided guidance for systematic reviews of economic evaluations. Eight papers reported transferring an HTA, with only one transferring to an LMIC. Finally, we reported issues that may facilitate or hinder transferability. In conclusion, we identified fourteen transfer approaches in the form of guidance or checklists, but harmonized and pragmatic guidance for HTA transfers to suit settings with limited HTA capacity seems warranted.

Reaching a Tipping Point for Neurorehabilitation Research: Obstacles and Opportunities in Trial Design, Description, and Pooled Analysis

The record-breaking pace of COVID-19 vaccine development and implementation depended heavily on collaboration among academic, government, and commercial stakeholders, especially through data-sharing and robust multicenter trials. Collaborative efforts have not been as fruitful in fields such as neurorehabilitation, where non-pharmacological interventions play a much larger role. Barriers to translating scientific advancements into clinical practice in neurorehabilitation include pervasively small study sizes, exacerbated by limited funding for non-pharmacological multicenter clinical trials; difficulty standardizing—and adequately describing—non-pharmacological interventions; and a lack of incentives for individual patient-level data-sharing. These barriers prevent reliable meta-analysis of non-pharmacological clinical studies in neurorehabilitation. This point-of-view will highlight these challenges as well as suggest practical steps that may be taken to improve the neurorehabilitation pipeline between evidence and implementation.

Algorithm-based pain management for people with dementia in nursing homes

BACKGROUND

People with dementia in nursing homes often experience pain, but often do not receive adequate pain therapy. The experience of pain has a significant impact on quality of life in people with dementia, and is associated with negative health outcomes. Untreated pain is also considered to be one of the causes of challenging behaviour, such as agitation or aggression, in this population. One approach to reducing pain in people with dementia in nursing homes is an algorithm-based pain management strategy, i.e. the use of a structured protocol that involves pain assessment and a series of predefined treatment steps consisting of various non-pharmacological and pharmacological pain management interventions.

OBJECTIVES

To assess the effects of algorithm-based pain management interventions to reduce pain and challenging behaviour in people with dementia living in nursing homes. To describe the components of the interventions and the content of the algorithms.

SEARCH METHODS

We searched ALOIS, the Cochrane Dementia and Cognitive Improvement Group's register, MEDLINE, Embase, PsycINFO, CINAHL (Cumulative Index to Nursing and Allied Health Literature), Web of Science Core Collection (ISI Web of Science), LILACS (Latin American and Caribbean Health Science Information database), ClinicalTrials.gov and the World Health Organization's meta-register the International Clinical Trials Registry Portal on 30 June 2021.

SELECTION CRITERIA

We included randomised controlled trials investigating the effects of algorithm-based pain management interventions for people with dementia living in nursing homes. All interventions had to include an initial pain assessment, a treatment algorithm (a treatment plan consisting of at least two different non-pharmacological or pharmacological treatment steps to reduce pain), and criteria to assess the success of each treatment step. The control groups could receive usual care or an active control intervention. Primary outcomes for this review were pain-related outcomes, e.g. the number of participants with pain (self- or proxy-rated), challenging behaviour (we used a broad definition that could also include agitation or behavioural and psychological symptoms assessed with any validated instrument), and serious adverse events.

DATA COLLECTION AND ANALYSIS

Two authors independently selected the articles for inclusion, extracted data and assessed the risk of bias of all included studies. We reported results narratively as there were too few studies for a meta-analysis. We used GRADE methods to rate the certainty of the results.

MAIN RESULTS

We included three cluster-randomised controlled trials with a total of 808 participants (mean age 82 to 89 years). In two studies, participants had severe cognitive impairment and in one study mild to moderate impairment. The algorithms used in the studies varied in the number of treatment steps. The comparator was pain education for nursing staff in two studies and usual care in one study. We judged the risk of detection bias to be high in one study. The risk of selection bias and performance bias was unclear in all studies. Self-rated pain (i.e. pain rated by participants themselves) was reported in two studies. In one study, all residents in the nursing homes were included, but fewer than half of the participants experienced pain at baseline, and the mean values of self-rated and proxy-rated pain at baseline and follow-up in both study groups were below the threshold of pain that may require treatment. We considered the evidence from this study to be very low-certainty and therefore are uncertain whether the algorithm-based pain management intervention had an effect on self-rated pain intensity compared with pain education (MD -0.27, 95% CI -0.49 to -0.05, 170 participants; Verbal Descriptor Scale, range 0 to 3). In the other study, all participants had mild to moderate pain at baseline. Here, we found low-certainty evidence that an algorithm-based pain management intervention may have little to no effect on self-rated pain intensity compared with pain education (MD 0.4, 95% CI -0.58 to 1.38, 246 participants; Iowa Pain Thermometer, range 0 to 12). Pain was rated by proxy in all three studies. Again, we considered the evidence from the study in which mean pain scores indicated no pain, or almost no pain, at baseline to be very low-certainty and were uncertain whether the algorithm-based pain management intervention had an effect on proxy-rated pain intensity compared with pain education. For participants with mild to moderate pain at baseline, we found low-certainty evidence that an algorithm-based pain management intervention may reduce proxy-rated pain intensity in comparison with usual care (MD -1.49, 95% CI -2.11 to -0.87, 1 study, 128 participants; Pain Assessment in Advanced Dementia Scale-Chinese version, range 0 to 10), but may not be more effective than pain education (MD -0.2, 95% CI -0.79 to 0.39, 1 study, 383 participants; Iowa Pain Thermometer, range 0 to 12). For challenging behaviour, we found very low-certainty evidence from one study in which mean pain scores indicated no pain, or almost no pain, at baseline. We were uncertain whether the algorithm-based pain management intervention had any more effect than education for nursing staff on challenging behaviour of participants (MD -0.21, 95% CI -1.88 to 1.46, 1 study, 170 participants; Cohen-Mansfield Agitation Inventory-Chinese version, range 7 to 203). None of the studies systematically assessed adverse effects or serious adverse effects and no study reported information about the occurrence of any adverse effect. None of the studies assessed any of the other outcomes of this review.

AUTHORS' CONCLUSIONS

There is no clear evidence for a benefit of an algorithm-based pain management intervention in comparison with pain education for reducing pain intensity or challenging behaviour in people with dementia in nursing homes. We found that the intervention may reduce proxy-rated pain compared with usual care. However, the certainty of evidence is low because of the small number of studies, small sample sizes, methodological limitations, and the clinical heterogeneity of the study populations (e.g. pain level and cognitive status). The results should be interpreted with caution. Future studies should also focus on the implementation of algorithms and their impact in clinical practice.

Assessing transferability in systematic reviews of health economic evaluations – a review of methodological guidance

Objective

For assessing cost-effectiveness, Health Technology Assessment (HTA) organisations may use primary economic evaluations (P-HEs) or Systematic Reviews of Health Economic evaluations (SR-HEs). A prerequisite for meaningful results of SR-HEs is that the results from existing P-HEs are transferable to the decision context (e.g, HTA jurisdiction). A particularly pertinent issue is the high variability of costs and resource needs across jurisdictions. Our objective was to review the methods documents of HTA organisations and compare their recommendations on considering transferability in SR-HE.

Methods

We systematically hand searched the webpages of 158 HTA organisations for relevant methods documents from 8th January to 31st March 2019. Two independent reviewers performed searches and selected documents according to pre-defined criteria. One reviewer extracted data in standardised and piloted tables and a second reviewer checked them for accuracy. We synthesised data using tabulations and in a narrative way.

Results

We identified 155 potentially relevant documents from 63 HTA organisations. Of these, 7 were included in the synthesis. The included organisations have different aims when preparing a SR-HE (e.g. to determine the need for conducting their own P-HE). The recommendations vary regarding the underlying terminology (e.g. transferability/generalisability), the assessment approaches (e.g. structure), the assessment criteria and the integration in the review process.

Conclusion

Only few HTA organisations address the assessment of transferability in their methodological recommendations for SR-HEs. Transferability considerations are related to different purposes. The assessment concepts and criteria are heterogeneous. Developing standards to consider transferability in SR-HEs is desirable.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01536-6.

Appraising clinical applicability of studies: mapping and synthesis of current frameworks, and proposal of the FrACAS framework and VICORT checklist

BACKGROUND

Not all research findings are translated to clinical practice. Reasons for lack of applicability are varied, and multiple frameworks and criteria exist to appraise the general applicability of epidemiological and clinical research. In this two-part study, we identify, map, and synthesize frameworks and criteria; we develop a framework to assist clinicians to appraise applicability specifically from a clinical perspective.

METHODS

We conducted a literature search in PubMed and Embase to identify frameworks appraising applicability of study results. Conceptual thematic analysis was used to synthesize frameworks and criteria. We carried out a framework development process integrating contemporary debates in epidemiology, findings from the literature search and synthesis, iterative pilot-testing, and brainstorming and consensus discussions to propose a concise framework to appraise clinical applicability.

RESULTS

Of the 4622 references retrieved, we identified 26 unique frameworks featuring 21 criteria. Frameworks and criteria varied by scope and level of aggregation of the evidence appraised, target user, and specific area of applicability (internal validity, clinical applicability, external validity, and system applicability). Our proposed Framework Appraising the Clinical Applicability of Studies (FrACAS) classifies studies in three domains (research, practice informing, and practice changing) by examining six criteria sequentially: Validity, Indication-informativeness, Clinical relevance, Originality, Risk-benefit comprehensiveness, and Transposability (VICORT checklist).

CONCLUSIONS

Existing frameworks to applicability vary by scope, target user, and area of applicability. We introduce FrACAS to specifically assess applicability from a clinical perspective. Our framework can be used as a tool for the design, appraisal, and interpretation of epidemiological and clinical studies.

A Rapid Review of Interventions to Prevent First Pregnancy among Adolescents and Its Applicability to Latin America

STUDY OBJECTIVE

To summarize recent literature on the effectiveness of interventions to prevent adolescent pregnancy and to explore the applicability of these interventions to Latin America (LA). Design, Setting, Participants, Interventions, and Main Outcome Measures: We carried out a rapid review of the literature (2005-2019). Studies were included if: they evaluated interventions targeting adolescents and prevention of pregnancy; they used a randomized controlled design; and pregnancy was measured as an outcome. Applicability of the interventions to LA was assessed using the following information: target population; intervention design and resources; type, skills, and training of providers; system arrangements; and acceptability and social context.

RESULTS

Nine studies were included, 5 described interventions in African countries, 2 in the United Kingdom, and 2 in the United States. Interventions were rated as highly applicable to LA in the context of target population, profile of the providers, and design; however, variations arose when assessing system arrangements and social context. Incentive-based interventions showed significant effects in the prevention of adolescent pregnancy and were rated as highly applicable.

CONCLUSION

This review provides professionals, policymakers, researchers, and educators potential criteria to consider when adapting successful evidence-based interventions to prevent adolescent pregnancy in LA.

Evaluation of the completeness of intervention reporting in Cochrane surgical systematic reviews using the TIDieR-SR checklist: a cross-sectional study

INTRODUCTION

Complete reporting of systematic reviews of interventions is essential to the interpretation of research findings and the reproducibility of research results. The Template for Intervention Description and Replication (TIDieR) checklist-and the version specific to systematic reviews (TIDieR-SR)-was created to provide authors and researchers an evidence-based guide for reporting trial and systematic review interventions. In this study, we apply TIDieR-SR to Cochrane systematic reviews of surgical interventions.

METHODS

We searched the Cochrane Database for relevant systematic reviews. Two investigators applied inclusion/exclusion criteria to all titles/abstracts and full texts. These same investigators extracted all data in duplicate while masked to the other's data. The primary outcome was adherence to TIDieR-SR items.

RESULTS

Two hundred and thirty-eight systematic reviews were included. Overall, included SRs adhered to a median of 6 (IQR 5-7) out of eight TIDieR-SR items. The item with the lowest adherence was item 7 (share intervention materials, 1/238 (0.4%).

DISCUSSION

Our results are encouraging, but the generalisability of our findings is compromised by the inclusion of only Cochrane systematic reviews. Future reporting of intervention materials is likely to improve the application of effective surgical interventions in the clinical practice.

Dynamic meta-analysis: a method of using global evidence for local decision making

Background

Meta-analysis is often used to make generalisations across all available evidence at the global scale. But how can these global generalisations be used for evidence-based decision making at the local scale, if the global evidence is not perceived to be relevant to local decisions? We show how an interactive method of meta-analysis—dynamic meta-analysis—can be used to assess the local relevance of global evidence.

Results

We developed Metadataset (www.metadataset.com) as a proof-of-concept for dynamic meta-analysis. Using Metadataset, we show how evidence can be filtered and weighted, and results can be recalculated, using dynamic methods of subgroup analysis, meta-regression, and recalibration. With an example from agroecology, we show how dynamic meta-analysis could lead to different conclusions for different subsets of the global evidence. Dynamic meta-analysis could also lead to a rebalancing of power and responsibility in evidence synthesis, since evidence users would be able to make decisions that are typically made by systematic reviewers—decisions about which studies to include (e.g. critical appraisal) and how to handle missing or poorly reported data (e.g. sensitivity analysis).

Conclusions

In this study, we show how dynamic meta-analysis can meet an important challenge in evidence-based decision making—the challenge of using global evidence for local decisions. We suggest that dynamic meta-analysis can be used for subject-wide evidence synthesis in several scientific disciplines, including agroecology and conservation biology. Future studies should develop standardised classification systems for the metadata that are used to filter and weight the evidence. Future studies should also develop standardised software packages, so that researchers can efficiently publish dynamic versions of their meta-analyses and keep them up-to-date as living systematic reviews. Metadataset is a proof-of-concept for this type of software, and it is open source. Future studies should improve the user experience, scale the software architecture, agree on standards for data and metadata storage and processing, and develop protocols for responsible evidence use.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12915-021-00974-w.

A review of methods for addressing components of interventions in meta-analysis

Many healthcare interventions are complex, consisting of multiple, possibly interacting, components. Several methodological articles addressing complex interventions in the meta-analytical context have been published. We hereby provide an overview of methods used to evaluate the effects of complex interventions with meta-analytical models. We summarized the methodology, highlighted new developments, and described the benefits, drawbacks, and potential challenges of each identified method. We expect meta-analytical methods focusing on components of several multicomponent interventions to become increasingly popular due to recently developed, easy-to-use, software tools that can be used to conduct the relevant analyses. The different meta-analytical methods are illustrated through two examples comparing psychotherapies for panic disorder.

Missed nursing care: An overview of reviews

Missed nursing care is care that is delayed, partially completed, or not completed at all. The aim of this overview of reviews was to identify the nursing care that is missed, the factors that influence missed nursing care and the outcomes from it. To be included, reviews had to use the systematic review process and focus on hospital care. Databases were searched from inception until August sixth, 2020. One author screened the papers and extracted data on included reviews and a second checked this. Two authors independently assessed the quality of the reviews. Seven reviews were included in this overview. Categories of care missed included: (a) communication and information sharing; (b) self-management, autonomy, and education including care planning, discharge planning and decision; (c) fundamental physical care; and (d) emotional and psychological care including spiritual support. Factors associated with missed care were related to staffing levels and/or labor resources skill mix, material resources not being available, patient acuity and teamwork/communication. Outcomes of missed nursing care included: less/poorer quality of patient care, patient satisfaction, and nurses' job satisfaction, increased patient adverse events, and the organizational outcomes of increasing hospital length of stay and hospital readmission. In-depth qualitative and mixed methods research is needed to better understand how nurses prioritize care and why care is missed. Longitudinal and experimental research is required to better clarify if these relationships between missed care and negative patient outcomes are likely cause and effect.

Assessing context suitability (generalizability, external validity, applicability or transferability) of findings in evidence syntheses in healthcare-An integrative review of methodological guidance

BACKGROUND

Evidence syntheses provide the basis for evidence-based decision making in healthcare. To judge the certainty of findings for the specific decision context evidence syntheses should consider context suitability (ie, generalizability, external validity, applicability or transferability). Our objective was to determine the status quo and to provide a comprehensive overview of existing methodological recommendations of Health Technology Assessment (HTA) and Systematic Review (SR) producing organizations in assessing context suitability of evidence on effectiveness of health care interventions. Additionally, we analyzed similarities and differences between the recommendations.

METHODS

In this Integrative Review we performed a structured search for methods documents from evidence synthesis producing organizations that include recommendations on appraising context suitability in effectiveness assessments. Two reviewers independently selected documents according to predefined eligibility criteria. Data were extracted in standardized and piloted tables by one reviewer and verified by a second reviewer. We performed a thematic analysis to identify and summarize the main themes and categories regarding recommended context suitability assessments.

RESULTS

We included 14 methods documents of 12 organizations in our synthesis. Assessment approaches are very heterogeneous both regarding the general concepts (eg, integration in the evidence synthesis preparation process) and the content of assessments (eg, assessment criteria).

CONCLUSION

Some heterogeneity seems to be justified because of the need to tailor the assessment to different settings and medical areas. However, most differences were inexplicable. More harmonization is desirable and appears possible.

Adherence Measures for Patients with Metastatic Castration-Resistant Prostate Cancer Treated with Abiraterone Acetate plus Prednisone: Results of a Prospective, Cluster-Randomized Trial

Simple Summary

We report the final results of a multicenter, prospective, 2-arm trial in a real world setting for patients with metastatic castration resistant prostate cancer. A number of 675 patients were allocated by center-based cluster-randomization to arm A with adherence enhancing measures or arm B without adherence enhancing measures. Our study reveals a generally high medication adherence in patients with mCRPC with no clear difference between Arm A and Arm B. Our results confirm the benefit of Abiraterone acetate plus Prednisolone in terms of effectiveness and quality of life in a real world setting.

Abstract

Residual androgen production causes tumor progression in metastatic, castration-resistant prostate cancer (mCRPC) patients. Abiraterone acetate (AA), a prodrug of abiraterone, is an oral CYP-17 inhibitor that blocks androgen production. It was hypothesized that adherence-enhancing measures (AEM) might be beneficial for mCRPC patients receiving abiraterone acetate plus prednisone (AA + P). This multicenter, prospective, 2-arm trial allocated mCRPC patients who were progressive after docetaxel-based chemotherapy or asymptomatic/mildly symptomatic after failure of an androgen deprivation therapy to Arm A (with AEM) or Arm B (without AEM) by center-based cluster-randomization. The primary objective was to assess the influence of AEM on discontinuation rates and medication adherence in mCRPC patients treated with AA + P. A total of 360 patients were allocated to Arm A, and 315 patients to Arm B. At month 3, the rate of treatment discontinuation, not due to disease progression or the start of new cancer therapy, was low in both arms (A: 9.0% vs. B: 7.3%, OR = 1.230). Few patients had a medium/low Morisky Medication Adherence Scale (MMAS-4) score (A: 6.4% vs. B: 9.1%, OR = 0.685). The results obtained did not suggest any clear adherence difference between Arm A and Arm B. In patients with mCRPC taking AA + P medication, adherence seemed to be generally high.

Understanding further education as a context for public health intervention: qualitative findings from a study process evaluation

BACKGROUND

Over 1.2 million 16-18 year-olds are enrolled in further education (FE-advanced secondary education) in England. Life course transitions provide opportunities to change, establish or reinforce health behaviours. FE presents an opportunity for public health improvement, yet few interventions target this setting. Using a smoking prevention intervention, we explore how young people were viewed in FE and how this affected intervention acceptability.

METHODS

Eleven student and five staff focus groups were conducted in three intervention institutions (two colleges, one school sixth-form), as part of the process evaluation of a smoking prevention feasibility study. FE managers in intervention and control institutions were also interviewed (n = 5). Data were analysed using thematic analysis.

RESULTS

In both colleges and the sixth-form, students were viewed as emergent adults and treated differently from 'school-children', in practice if not in policy. Colleges permitted smoking in designated areas; in the school sixth-form smoking was unofficially tolerated but concealed from younger students. Using staff to deliver anti-smoking messages reintroduced an unwanted power dynamic which disrupted perceptions of students as young adults.

CONCLUSIONS

FE is an important setting for young people's health. Understanding the culture and context of FE is critical in designing acceptable and effective public health interventions.

The TRANSFER Approach for assessing the transferability of systematic review findings

BACKGROUND

Systematic reviews are a key input to health and social welfare decisions. Studies included in systematic reviews often vary with respect to contextual factors that may impact on how transferable review findings are to the review context. However, many review authors do not consider the transferability of review findings until the end of the review process, for example when assessing confidence in the evidence using GRADE or GRADE-CERQual. This paper describes the TRANSFER Approach, a novel approach for supporting collaboration between review authors and stakeholders from the beginning of the review process to systematically and transparently consider factors that may influence the transferability of systematic review findings.

METHODS

We developed the TRANSFER Approach in three stages: (1) discussions with stakeholders to identify current practices and needs regarding the use of methods to consider transferability, (2) systematic search for and mapping of 25 existing checklists related to transferability, and (3) using the results of stage two to develop a structured conversation format which was applied in three systematic review processes.

RESULTS

None of the identified existing checklists related to transferability provided detailed guidance for review authors on how to assess transferability in systematic reviews, in collaboration with decision makers. The content analysis uncovered seven categories of factors to consider when discussing transferability. We used these to develop a structured conversation guide for discussing potential transferability factors with stakeholders at the beginning of the review process. In response to feedback and trial and error, the TRANSFER Approach has developed, expanding beyond the initial conversation guide, and is now made up of seven stages which are described in this article.

CONCLUSIONS

The TRANSFER Approach supports review authors in collaborating with decision makers to ensure an informed consideration, from the beginning of the review process, of the transferability of the review findings to the review context. Further testing of TRANSFER is needed.

Health Technology Assessment of Public Health Interventions Published 2012 to 2016: An Analysis of Characteristics and Comparison of Methods

Objectives

The aim of this study was to provide an overview of the methodological characteristics and compare the assessment methods applied in health technology assessments (HTAs) of public health interventions (PHIs).

Methods

We defined a PHI as a population-based intervention on health promotion or for primary prevention of chronic or nonchronic diseases. HTAs on PHIs were identified by systematically searching the Web pages of members of international HTA networks. We included only full HTA reports published between 2012 and 2016. Two reviewers extracted data on the methods used to assess effectiveness/safety, as well as on economic, social, cultural, ethical, and legal aspects using a-priori standardized tables.

Results

We included ten HTAs provided by four different organizations. Of these, all reports assessed the effectiveness of the interventions and conducted economic evaluations, seven investigated social/cultural aspects, and four each considered legal and ethical aspects, respectively. Some reports addressed applicability, context/setting, and intervention fidelity issues in different ways. We found that most HTAs adapted their methods to some extent, for example, by including nonrandomized studies, expanding the search strategy, involving stakeholders, or applying a framework to guide the HTA process.

Conclusions

Our analysis provides a comprehensive overview of methods applied in HTAs on public health interventions. We found that a heterogeneous set of approaches is used to deal with the challenges of evaluating complex public health interventions.

Towards greater understanding of implementation during systematic reviews of complex healthcare interventions: the framework for implementation transferability applicability reporting (FITAR)

Background

There have been calls for greater consideration of applicability and transferability in systematic reviews, to improve their usefulness in informing policy and practice. Understanding how evidence is, or is not applicable and transferable to varying local situations and contexts, is a key challenge for systematic review synthesis in healthcare. Assessing applicability and transferability in systematic reviews is reported to be difficult, particularly in reviews of complex interventions. There is a need for exploration of factors perceived to be important by policy-makers, and for further guidance on which items should be reported. In this paper we focus on the process of development of a framework that can be used by systematic reviewers to identify and report data across studies relating to applicability and transferability.

Methods

The framework was developed by scrutinising existing literature on applicability and transferability, examining data during a systematic review of highly complex changes to health service delivery, and was informed by stakeholder engagement. The items of the framework were thus grounded in both data identified during a real review, and stakeholder input. The paper describes examples of data identified using the framework during a review of integrated care interventions, and outlines how it informed analysis and reporting of the review findings.

Results

The Framework for Implementation Transferability Applicability Reporting (FITAR) comprises 44 items which can be used to structure analysis and reporting across studies during systematic reviews of complex interventions. The framework prompts detailed consideration of contextual data during extraction and reporting, within areas of: patient type and populations; type of organisations and systems; financial and commissioning processes; systems leadership elements; features of services; features of the workforce; and finally elements of the interventions/initiatives.

Conclusions

Use of the framework during our review of complex healthcare interventions helped the review team to surface contextual data, which may not be commonly extracted, analysed and reported. Further exploration and evaluation of systems for identifying and reporting these factors during reviews is required.

Electronic supplementary material

The online version of this article (10.1186/s12874-019-0723-y) contains supplementary material, which is available to authorized users.

Pharmacological and non-pharmacological interventions for non-respiratory sleep disturbance in children with neurodisabilities: a systematic review

BACKGROUND

There is uncertainty about the most appropriate ways to manage non-respiratory sleep disturbances in children with neurodisabilities (NDs).

OBJECTIVE

To assess the clinical effectiveness and safety of NHS-relevant pharmacological and non-pharmacological interventions to manage sleep disturbance in children and young people with NDs, who have non-respiratory sleep disturbance.

DATA SOURCES

Sixteen databases, including The Cochrane Central Register of Controlled Trials, EMBASE and MEDLINE, were searched up to February 2017, and grey literature searches and hand-searches were conducted.

REVIEW METHODS

For pharmacological interventions, only randomised controlled trials (RCTs) were included. For non-pharmacological interventions, RCTs, non-randomised controlled studies and before-and-after studies were included. Data were extracted and quality assessed by two researchers. Meta-analysis and narrative synthesis were undertaken. Data on parents' and children's experiences of receiving a sleep disturbance intervention were collated into themes and reported narratively.

RESULTS

Thirty-nine studies were included. Sample sizes ranged from 5 to 244 participants. Thirteen RCTs evaluated oral melatonin. Twenty-six studies (12 RCTs and 14 before-and-after studies) evaluated non-pharmacological interventions, including comprehensive parent-directed tailored (n = 9) and non-tailored (n = 8) interventions, non-comprehensive parent-directed interventions (n = 2) and other non-pharmacological interventions (n = 7). All but one study were reported as having a high or unclear risk of bias, and studies were generally poorly reported. There was a statistically significant increase in diary-reported total sleep time (TST), which was the most commonly reported outcome for melatonin compared with placebo [pooled mean difference 29.6 minutes, 95% confidence interval (CI) 6.9 to 52.4 minutes; p = 0.01]; however, statistical heterogeneity was extremely high (97%). For the single melatonin study that was rated as having a low risk of bias, the mean increase in TST was 13.2 minutes and the lower CI included the possibility of reduced sleep time (95% CI -13.3 to 39.7 minutes). There was mixed evidence about the clinical effectiveness of the non-pharmacological interventions. Sixteen studies included interventions that investigated the feasibility, acceptability and/or parent or clinician views of sleep disturbance interventions. The majority of these studies reported the 'family experience' of non-pharmacological interventions.

LIMITATIONS

Planned subgroup analysis was possible in only a small number of melatonin trials.

CONCLUSIONS

There is some evidence of benefit for melatonin compared with placebo, but the degree of benefit is uncertain. There are various types of non-pharmacological interventions for managing sleep disturbance; however, clinical and methodological heterogeneity, few RCTs, a lack of standardised outcome measures and risk of bias means that it is not possible to draw conclusions with regard to their effectiveness. Future work should include the development of a core outcome, further evaluation of the clinical effectiveness and cost-effectiveness of pharmacological and non-pharmacological interventions and research exploring the prevention of, and methods for identifying, sleep disturbance. Research mapping current practices and exploring families' understanding of sleep disturbance and their experiences of obtaining help may facilitate service provision development.

STUDY REGISTRATION

This study is registered as PROSPERO CRD42016034067.

FUNDING

The National Institute for Health Research Health Technology Assessment programme.

Taking account of context in systematic reviews and guidelines considering a complexity perspective

Systematic review teams and guideline development groups face considerable challenges when considering context within the evidence production process. Many complex interventions are context-dependent and are frequently evaluated within considerable contextual variation and change. This paper considers the extent to which current tools used within systematic reviews and guideline development are suitable in meeting these challenges. The paper briefly reviews strengths and weaknesses of existing approaches to specifying context. Illustrative tools are mapped to corresponding stages of the systematic review process. Collectively, systematic review and guideline production reveals a rich diversity of frameworks and tools for handling context. However, current approaches address only specific elements of context, are derived from primary studies which lack information or have not been tested within systematic reviews. A hypothetical example is used to illustrate how context could be integrated throughout the guideline development process. Guideline developers and evidence synthesis organisations should select an appropriate level of contextual detail for their specific guideline that is parsimonious and yet sensitive to health systems contexts and the values, preferences and needs of their target populations.

Systematic mapping of checklists for assessing transferability

BACKGROUND

Systematic reviews of research evidence have become an expected basis for decisions about practice guidelines and policy decisions in the health and welfare sectors. Review authors define inclusion criteria to help them determine which studies to search for and include in their reviews. However, these studies may still vary in the extent to which they reflect the context of interest in the review question. While most review authors would agree that systematic reviews should be relevant and useful for decision makers, there appears to be few well known, if any, established methods for supporting review authors to assess the transferability of review findings to the context of interest in the review. With this systematic mapping and content analysis, we aim to identify whether there exists checklists to support review authors in considering transferability early in the systematic review process. The secondary aim was to develop a comprehensive list of factors that influence transferability as discussed in existing checklists.

METHODS

We conducted a systematic mapping of checklists and performed a content analysis of the checklist criteria included in the identified checklists. In June 2016, we conducted a systematic search of eight databases to identify checklists to assess transferability of findings from primary or secondary research, without limitations related to publication type, status, language, or date. We also conducted a gray literature search and searched the EQUATOR repository of checklists for any relevant document. We used search terms such as modified versions of the terms "transferability," "applicability," "generalizability," etc. and "checklist," "guideline," "tool," "criteria," etc. We did not include papers that discussed transferability at a theoretical level or checklists to assess the transferability of guidelines to local contexts.

RESULTS

Our search resulted in 11,752 titles which were screened independently by two review authors. The 101 articles which were considered potentially relevant were subsequently read by two authors, independently in full text and assessed for inclusion. We identified 31 relevant checklists. Six of these examined transferability of economic evaluations, and 25 examined transferability of primary or secondary research findings in health (n = 23) or social welfare (n = 2). The content analysis is based on the 25 health and social welfare checklists. We identified seven themes under which we grouped categories of checklist criteria: population, intervention, implementation context (immediate), comparison intervention, outcomes, environmental context, and researcher conduct.

CONCLUSIONS

We identified a variety of checklists intended to support end users (researchers, review authors, practitioners, etc.) to assess transferability or related concepts. While four of these checklists are intended for use in systematic reviews of effectiveness, we found no checklists for qualitative evidence syntheses or for the field of social welfare practice or policy. Furthermore, none of the identified checklists for review authors included guidance to on how to assess transferability, or present assessments in a systematic review. The results of the content analysis can serve as the basis for developing a comprehensive list of factors to be used in an approach to support review authors in systematically and transparently considering transferability from the beginning of the review process.

SYSTEMATIC REVIEWS OF ECONOMIC EVALUATIONS IN HEALTH TECHNOLOGY ASSESSMENT: A REVIEW OF CHARACTERISTICS AND APPLIED METHODS

OBJECTIVES

When making decisions in health care, it is essential to consider economic evidence about an intervention. The objective of this study was to analyze the methods applied for systematic reviews of health economic evaluations (SR-HEs) in HTA and to identify common challenges.

METHODS

We manually searched the Web pages of HTA organizations and included HTA-reports published since 2015. Prerequisites for inclusion were the conduct of an SR-HE in at least one electronic database and the use of the English, German, French, or Spanish language. Methodological features were extracted in standardized tables. We prepared descriptive statistical (e.g., median, range) measures to describe the applied methods. Data were synthesized in a structured narrative way.

RESULTS

Eighty-three reports were included in the analysis. We identified inexplicable heterogeneity, particularly concerning literature search strategy, data extraction, assessment of quality, and applicability. Furthermore, process steps were often missing or reported in a nontransparent way. The use of a standardized data extraction form was indicated in one-third of reports (32 percent). Fifty-four percent of authors systematically appraised included studies. In 10 percent of reports, the applicability of included studies was assessed. Involvement of two reviewers was rarely reported for the study selection (43 percent), data extraction (28 percent), and quality assessment (39 percent).

CONCLUSIONS

The methods applied for SR-HEs in HTA and their reporting quality are very heterogeneous. Efforts toward a detailed, standardized guidance for the preparation of SR-HEs definitely seem necessary. A general harmonization and improvement of the applied methodology would increase the value of SR-HE for decision makers.

Assessing the applicability of public health intervention evaluations from one setting to another: a methodological study of the usability and usefulness of assessment tools and frameworks

BACKGROUND

Public health interventions can be complicated, complex and context dependent, making the assessment of applicability challenging. Nevertheless, for them to be of use beyond the original study setting, they need to be generalisable to other settings and, crucially, research users need to be able to identify to which contexts it may be applicable. There are many tools with set criteria for assessing generalisability/applicability, yet few seem to be widely used and there is no consensus on which should be used, or when. This methodological study aimed to test these tools to assess how easy they were to use and how useful they appeared to be.

METHODS

We identified tools from an existing review and an update of its search. References were screened on pre-specified criteria. Included tools were tested by using them to assess the applicability of a Swedish weight management intervention to the English context. Researcher assessments and reflections on the usability and utility of the tools were gathered using a standard pro-forma.

RESULTS

Eleven tools were included. Their length, content, style and time required to complete varied. No tool was considered ideal for assessing applicability. Their limitations included unrealistic criteria (requiring unavailable information), a focus on implementation to the neglect of transferability (i.e. little focus on potential effectiveness in the new setting), overly broad criteria (associated with low reliability), and a lack of an explicit focus on how interventions worked (i.e. their mechanisms of action).

CONCLUSION

Tools presenting criteria ready to be used may not be the best method for applicability assessments. They are likely to be either too long or incomplete, too focused on differences and fail to address elements that matter for the specific topic of interest. It is time to progress from developing lists of set criteria that are not widely used in the literature, to creating a new approach to applicability assessment. Focusing on mechanisms of action, rather than solely on characteristics, could be a useful approach, and one that remains underutilised in current tools. New approaches to assessing generalisability that evolve away from checklist style assessments need to be developed, tested, reported and discussed.

Understanding new models of integrated care in developed countries: a systematic review

Background

The NHS has been challenged to adopt new integrated models of service delivery that are tailored to local populations. Evidence from the international literature is needed to support the development and implementation of these new models of care.

Objectives

The study aimed to carry out a systematic review of international evidence to enhance understanding of the mechanisms whereby new models of service delivery have an impact on health-care outcomes.

Design

The study combined rigorous and systematic methods for identification of literature, together with innovative methods for synthesis and presentation of findings.

Setting

Any setting.

Participants

Patients receiving a health-care service and/or staff delivering services.

Interventions

Changes to service delivery that increase integration and co-ordination of health and health-related services.

Main outcome measures

Outcomes related to the delivery of services, including the views and perceptions of patients/service users and staff.

Study design

Empirical work of a quantitative or qualitative design.

Data sources

We searched electronic databases (between October 2016 and March 2017) for research published from 2006 onwards in databases including MEDLINE, EMBASE, PsycINFO, Cumulative Index to Nursing and Allied Health Literature, Science Citation Index, Social Science Citation Index and The Cochrane Library. We also searched relevant websites, screened reference lists and citation searched on a previous review.

Review methods

The identified evidence was synthesised in three ways. First, data from included studies were used to develop an evidence-based logic model, and a narrative summary reports the elements of the pathway. Second, we examined the strength of evidence underpinning reported outcomes and impacts using a comparative four-item rating system. Third, we developed an applicability framework to further scrutinise and characterise the evidence.

Results

We included 267 studies in the review. The findings detail the complex pathway from new models to impacts, with evidence regarding elements of new models of integrated care, targets for change, process change, influencing factors, service-level outcomes and system-wide impacts. A number of positive outcomes were reported in the literature, with stronger evidence of perceived increased patient satisfaction and improved quality of care and access to care. There was stronger UK-only evidence of reduced outpatient appointments and waiting times. Evidence was inconsistent regarding other outcomes and system-wide impacts such as levels of activity and costs. There was an indication that new models have particular potential with patients who have complex needs.

Limitations

Defining new models of integrated care is challenging, and there is the potential that our study excluded potentially relevant literature. The review was extensive, with diverse study populations and interventions that precluded the statistical summary of effectiveness.

Conclusions

There is stronger evidence that new models of integrated care may enhance patient satisfaction and perceived quality and increase access; however, the evidence regarding other outcomes is unclear. The study recommends factors to be considered during the implementation of new models.

Future work

Links between elements of new models and outcomes require further study, together with research in a wider variety of populations.

Study registration

This study is registered as PROSPERO CRD37725.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Nutritional supplements for patients being treated for active visceral leishmaniasis

BACKGROUND

Visceral leishmaniasis (VL) is a disease caused by a parasite, which can lead to death if untreated. Poor nutritional status hastens the progression of VL infection, and VL worsens malnutrition status. Malnutrition is one of the poor prognostic factors identified for leishmaniasis. However, the effects of nutritional supplementation in people treated for VL are not known.

OBJECTIVES

To assess the effects of oral nutritional supplements in people being treated with anti-leishmanial drug therapy for VL.

SEARCH METHODS

We searched the Cochrane Infectious Diseases Group (CIDG) Specialized Register, CENTRAL, MEDLINE, Embase, LILACS, and two trial registers up to 12 September 2017. We checked conference proceedings and WHO consultative meeting reports, the reference lists of key documents and existing reviews, and contacted experts and nutritional supplement companies.

SELECTION CRITERIA

Randomized controlled trials (RCTs), quasi-randomized controlled trials (quasi-RCTs), and non-randomized controlled trials (NRCTs) of any oral nutritional supplement, compared to no nutritional intervention, placebo, or dietary advice alone, in people being treated for VL.

DATA COLLECTION AND ANALYSIS

Two review authors independently screened the literature search results for studies that met the inclusion criteria. We had planned for two review authors to independently extract data and assess the risk of bias of the included studies. We planned to follow the Cochrane standard methodological procedures for assessing risk of bias and analysing the data.

MAIN RESULTS

We identified no eligible studies for this review, either completed or ongoing.

AUTHORS' CONCLUSIONS

We found no studies, either completed or ongoing, that assessed the effects of oral nutritional supplements in people with VL who were being treated with anti-leishmanial drug therapy. Thus, we could not draw any conclusions on the impact of these interventions on primary cure of VL, definitive cure of VL, treatment completion, self-reported recovery from illness or resolution of symptoms, weight gain, increased skinfold thickness, other measures of lean or total mass, or growth in children.This absence of evidence should not be interpreted as evidence of no effect for nutritional supplements in people under VL treatment. It means that we did not identify research that fulfilled our review inclusion criteria.The effects of oral nutritional supplements in people with VL who are being treated with anti-leishmanial drug therapy have yet to be determined by rigorous experimental studies, such as cluster-randomized trials, that focus on outcomes relevant for patients.

Looking beyond the forest: Using harvest plots, gap analysis, and expert consultations to assess effectiveness, engage stakeholders, and inform policy

We describe a combination of methods for assessing the effectiveness of complex interventions, especially where substantial heterogeneity with regard to the population, intervention, comparison, outcomes, and study design of interest is expected. We applied these methods in a recent systematic review of the effectiveness of reinforced home-based palliative care (rHBPC) interventions, which included home-based care with an additional and explicit component of lay caregiver support. We first summarized the identified evidence, deemed inappropriate for statistical pooling, graphically by creating harvest plots. Although very useful as a tool for summary and presentation of overall effectiveness, such graphical summary approaches may obscure relevant differences between studies. Thus, we then used a gap analysis and conducted expert consultations to look beyond the aggregate level at how the identified evidence of effectiveness may be explained. The goal of these supplemental methods was to step outside of the conventional systematic review and explore this heterogeneity from a broader perspective, based on the experience of palliative care researchers and practitioners. The gap analysis and expert consultations provided valuable input into possible underlying explanations in the evidence, which could be helpful in the further adaptation and testing of existing rHBPC interventions or the development and evaluation of new ones. We feel that such a combination of methods could prove accessible, understandable, and useful in informing decisions and could thus help increase the relevance of systematic reviews to the decision-making process.

AHRQ series on complex intervention systematic reviews-paper 4: selecting analytic approaches

BACKGROUND

Systematic reviews of complex interventions can vary widely in purpose, data availability and heterogeneity, and stakeholder expectations.

RATIONALE

This article addresses the uncertainty that systematic reviewers face in selecting methods for reviews of complex interventions. Specifically, it lays out parameters for systematic reviewers to consider when selecting analytic approaches that best answer the questions at hand and suggests analytic techniques that may be appropriate in different circumstances.

DISCUSSION

Systematic reviews of complex interventions comprising multiple questions may use multiple analytic approaches. Parameters to consider when choosing analytic methods for complex interventions include nature and timing of the decision (clinical practice guideline, policy, or other); purpose of the review; extent of existing evidence; logistic factors such as the timeline, process, and resources for deciding the scope of the review; and value of information to be obtained from choosing specific systematic review methods. Reviewers may elect to revise their analytic approach based on new or changing considerations during the course of the review but should guard against bias through transparency of reporting.

A CONSULTATION GUIDE FOR ASSESSING THE APPLICABILITY OF HEALTH TECHNOLOGIES:A CASE STUDY

Objectives:The translation of research findings into policy and practice is crucially dependent on the applicability of such findings in a given decision-making context. We explored in a case study whether a generic consultation guide to assess the applicability of a health technology could be rapidly deployed and deliver useful insights.

Methods:A consultation guide based on the context and implementation for complex interventions (CICI) framework was developed and piloted to assess the applicability of reinforced home-based palliative care in three European countries. Individual consultations in England and Germany and a panel discussion in Poland were completed.

Results:Various barriers may hinder successful implementation of reinforced home-based palliative care in the three countries. Whilst the experts across all countries emphasized the lack of funding along with organization and structure as major barriers, information varied by country for many of the other identified barriers and facilitators. Participants in the pilot study provided positive feedback in terms of understanding the topic and purpose of the consultation, and both individual and panel consultations could be easily implemented.

Conclusions:In this case study, the consultation guide presented a pragmatic, ready-to-use tool to assess the applicability of a health technology. As shown here, it can be used in a generic manner without discrete empirical information on the technology in question or, ideally, makes use of specific information collected as part of a HTA. Further studies are needed to validate this guide and apply it to other types of health technologies and more diverse decision-making contexts.

AHRQ series on complex intervention systematic reviews-paper 2: defining complexity, formulating scope, and questions

BACKGROUND

The early stages of a systematic review set the scope and expectations. This can be particularly challenging for complex interventions given their multidimensional and dynamic nature.

RATIONALE

This paper builds on concepts introduced in paper 1 of this series. It describes the methodological, practical, and philosophical challenges and potential approaches for formulating the questions and scope of systematic reviews of complex interventions. Furthermore, it discusses the use of theory to help organize reviews of complex interventions.

DISCUSSION

Many interventions in medicine, public health, education, social services, behavioral health, and community programs are complex, and they may not fit neatly within the established paradigm for reviews of straightforward interventions. This paper provides conceptual and operational guidance for these early stages of scope formulation to assist authors of systematic reviews of complex interventions.

Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR)

BACKGROUND

Health interventions fall along a spectrum from simple to more complex. There is wide interest in methods for reviewing 'complex interventions', but few transparent approaches for assessing intervention complexity in systematic reviews. Such assessments may assist review authors in, for example, systematically describing interventions and developing logic models. This paper describes the development and application of the intervention Complexity Assessment Tool for Systematic Reviews (iCAT_SR), a new tool to assess and categorise levels of intervention complexity in systematic reviews.

METHODS

We developed the iCAT_SR by adapting and extending an existing complexity assessment tool for randomized trials. We undertook this adaptation using a consensus approach in which possible complexity dimensions were circulated for feedback to a panel of methodologists with expertise in complex interventions and systematic reviews. Based on these inputs, we developed a draft version of the tool. We then invited a second round of feedback from the panel and a wider group of systematic reviewers. This informed further refinement of the tool.

RESULTS

The tool comprises ten dimensions: (1) the number of active components in the intervention; (2) the number of behaviours of recipients to which the intervention is directed; (3) the range and number of organizational levels targeted by the intervention; (4) the degree of tailoring intended or flexibility permitted across sites or individuals in applying or implementing the intervention; (5) the level of skill required by those delivering the intervention; (6) the level of skill required by those receiving the intervention; (7) the degree of interaction between intervention components; (8) the degree to which the effects of the intervention are context dependent; (9) the degree to which the effects of the interventions are changed by recipient or provider factors; (10) and the nature of the causal pathway between intervention and outcome. Dimensions 1-6 are considered 'core' dimensions. Dimensions 7-10 are optional and may not be useful for all interventions.

CONCLUSIONS

The iCAT_SR tool facilitates more in-depth, systematic assessment of the complexity of interventions in systematic reviews and can assist in undertaking reviews and interpreting review findings. Further testing of the tool is now needed.

HEALTH TECHNOLOGY ASSESSMENT OF PUBLIC HEALTH INTERVENTIONS: A SYNTHESIS OF METHODOLOGICAL GUIDANCE

Objectives: The evaluation of public health interventions poses some challenges. As a consequence, health technology assessment (HTA) methods for public health interventions (PHI) have to be adapted. This study aimed to summarize the available guidance on methods for HTA of PHI.

Methods: We systematically searched for methodological guidance on HTA of PHIs. Our focus was on research synthesis methods to evaluate effectiveness. Relevant information was synthesized narratively in a standardized way.

Results: Only four guidance documents were identified specifically for HTAs of PHI. The approaches used for HTAs of PHIs are broader and more flexible than those for medical interventions. For this reason, there is a tendency to identify the intervention components and context factors that influence the effectiveness and transferability of an intervention rather than to assess its effectiveness in general. The details in the guidance vary without justification. Unjustified heterogeneity between the different guidance approaches is most pronounced for quality assessment, assessment of applicability, and methods to integrate qualitative and quantitative evidence. Descriptions for the assessment of integrity, heterogeneity, sustainability, context factors, and applicability are often vague.

Conclusions: The heterogeneity in approaches indicates that there is currently no consensus on methods to deal with the challenges of the PHI evaluations. A possible explanation for this may be that the methods are not sufficiently developed, and advantages and disadvantages of a certain method in relation to the research question (e.g., broad/focused) have not yet been sufficiently evaluated.

Parents' and informal caregivers' views and experiences of communication about routine childhood vaccination: a synthesis of qualitative evidence

BACKGROUND

Childhood vaccination is an effective way to prevent serious childhood illnesses, but many children do not receive all the recommended vaccines. There are various reasons for this; some parents lack access because of poor quality health services, long distances or lack of money. Other parents may not trust vaccines or the healthcare workers who provide them, or they may not see the need for vaccination due to a lack of information or misinformation about how vaccinations work and the diseases they can prevent.Communication with parents about childhood vaccinations is one way of addressing these issues. Communication can take place at healthcare facilities, at home or in the community. Communication can be two-way, for example face-to-face discussions between parents and healthcare providers, or one-way, for instance via text messages, posters or radio programmes. Some types of communication enable parents to actively discuss vaccines and their benefits and harms, as well as diseases they can prevent. Other communication types simply give information about vaccination issues or when and where vaccines are available. People involved in vaccine programmes need to understand how parents experience different types of communication about vaccination and how this influences their decision to vaccinate.

OBJECTIVES

The specific objectives of the review were to identify, appraise and synthesise qualitative studies exploring: parents' and informal caregivers' views and experiences regarding communication about childhood vaccinations and the manner in which it is communicated; and the influence that vaccination communication has on parents' and informal caregivers' decisions regarding childhood vaccination.

SEARCH METHODS

We searched MEDLINE (OvidSP), MEDLINE In-process and Other Non-Index Citations (Ovid SP), Embase (Ovid), CINAHL (EbscoHOST), and Anthropology Plus (EbscoHost) databases for eligible studies from inception to 30 August 2016. We developed search strategies for each database, using guidelines developed by the Cochrane Qualitative Research Methods Group for searching for qualitative evidence as well as modified versions of the search developed for three related reviews of effectiveness. There were no date or geographic restrictions for the search.

SELECTION CRITERIA

We included studies that utilised qualitative methods for data collection and analysis; focused on the views and experiences of parents and informal caregivers regarding information about vaccination for children aged up to six years; and were from any setting globally where information about childhood vaccinations was communicated or distributed.

DATA COLLECTION AND ANALYSIS

We used maximum variation purposive sampling for data synthesis, using a three-step sampling frame. We conducted a thematic analysis using a constant comparison strategy for data extraction and synthesis. We assessed our confidence in the findings using the GRADE-CERQual approach. High confidence suggests that it is highly likely that the review finding is a reasonable representation of the phenomenon of interest, while very low confidence indicates that it is not clear whether the review finding is a reasonable representation of it. Using a matrix model, we then integrated our findings with those from other Cochrane reviews that assessed the effects of different communication strategies on parents' knowledge, attitudes and behaviour about childhood vaccination.

MAIN RESULTS

We included 38 studies, mostly from high-income countries, many of which explored mothers' perceptions of vaccine communication. Some focused on the MMR (measles, mumps, rubella) vaccine.In general, parents wanted more information than they were getting (high confidence in the evidence). Lack of information led to worry and regret about vaccination decisions among some parents (moderate confidence).Parents wanted balanced information about vaccination benefits and harms (high confidence), presented clearly and simply (moderate confidence) and tailored to their situation (low confidence in the evidence). Parents wanted vaccination information to be available at a wider variety of locations, including outside health services (low confidence) and in good time before each vaccination appointment (moderate confidence).Parents viewed health workers as an important source of information and had specific expectations of their interactions with them (high confidence). Poor communication and negative relationships with health workers sometimes impacted on vaccination decisions (moderate confidence).Parents generally found it difficult to know which vaccination information source to trust and challenging to find information they felt was unbiased and balanced (high confidence).The amount of information parents wanted and the sources they felt could be trusted appeared to be linked to acceptance of vaccination, with parents who were more hesitant wanting more information (low to moderate confidence).Our synthesis and comparison of the qualitative evidence shows that most of the trial interventions addressed at least one or two key aspects of communication, including the provision of information prior to the vaccination appointment and tailoring information to parents' needs. None of the interventions appeared to respond to negative media stories or address parental perceptions of health worker motives.

AUTHORS' CONCLUSIONS

We have high or moderate confidence in the evidence contributing to several review findings. Further research, especially in rural and low- to middle-income country settings, could strengthen evidence for the findings where we had low or very low confidence. Planners should consider the timing for making vaccination information available to parents, the settings where information is available, the provision of impartial and clear information tailored to parental needs, and parents' perceptions of health workers and the information provided.

Consideration of health inequalities in systematic reviews: a mapping review of guidance

BACKGROUND

Given that we know that interventions shown to be effective in improving the health of a population may actually widen the health inequalities gap while others reduce it, it is imperative that all systematic reviewers consider how the findings of their reviews may impact (reduce or increase) on the health inequality gap. This study reviewed existing guidance on incorporating considerations of health inequalities in systematic reviews in order to examine the extent to which they can help reviewers to incorporate such issues.

METHODS

A mapping review was undertaken to identify guidance documents that purported to inform reviewers on whether and how to incorporate considerations of health inequalities. Searches were undertaken in Medline, CINAHL and The Cochrane Library Methodology Register. Review guidance manuals prepared by international organisations engaged in undertaking systematic reviews, and their associated websites were scanned. Studies were included if they provided an overview or discussed the development and testing of guidance for dealing with the incorporation of considerations of health inequalities in evidence synthesis. Results are summarised in narrative and tabular forms.

RESULTS

Twenty guidance documents published between 2009 and 2016 were included. Guidance has been produced to inform considerations of health inequalities at different stages of the systematic review process. The Campbell and Cochrane Equity Group have been instrumental in developing and promoting such guidance. Definitions of health inequalities and guidance differed across the included studies. All but one guidance document were transparent in their method of production. Formal methods of evaluation were reported for six guidance documents. Most of the guidance was operationalised in the form of examples taken from published systematic reviews. The number of guidance items to operationalise ranges from 3 up to 26 with a considerable overlap noted.

CONCLUSIONS

Adhering to the guidance will require more work for the reviewers. It requires a deeper understanding of how reviewers can operationalise the guidance taking into consideration the barriers and facilitators involved. This has implications not only for understanding the usefulness and burden of the guidance but also for the uptake of guidance and its ultimate goal of improving health inequalities considerations in systematic reviews.

Expanding the evidence base for global recommendations on health systems: strengths and challenges of the OptimizeMNH guidance process

BACKGROUND

In 2012, the World Health Organization (WHO) published recommendations on the use of optimization or "task-shifting" strategies for key, effective maternal and newborn interventions (the OptimizeMNH guidance). When making recommendations about complex health system interventions such as task-shifting, information about the feasibility and acceptability of interventions can be as important as information about their effectiveness. However, these issues are usually not addressed with the same rigour. This paper describes our use of several innovative strategies to broaden the range of evidence used to develop the OptimizeMNH guidance. In this guidance, we systematically included evidence regarding the acceptability and feasibility of relevant task-shifting interventions, primarily using qualitative evidence syntheses and multi-country case study syntheses; we used an approach to assess confidence in findings from qualitative evidence syntheses (the Grading of Recommendations, Assessment, Development and Evaluation-Confidence in Evidence from Reviews of Qualitative Research (GRADE-CERQual) approach); we used a structured evidence-to-decision framework for health systems (the DECIDE framework) to help the guidance panel members move from the different types of evidence to recommendations.

RESULTS

The systematic inclusion of a broader range of evidence, and the use of new guideline development tools, had a number of impacts. Firstly, this broader range of evidence provided relevant information about the feasibility and acceptability of interventions considered in the guidance as well as information about key implementation considerations. However, inclusion of this evidence required more time, resources and skills. Secondly, the GRADE-CERQual approach provided a method for indicating to panel members how much confidence they should place in the findings from the qualitative evidence syntheses and so helped panel members to use this qualitative evidence appropriately. Thirdly, the DECIDE framework gave us a structured format in which we could present a large and complex body of evidence to panel members and end users. The framework also prompted the panel to justify their recommendations, giving end users a record of how these decisions were made.

CONCLUSIONS

By expanding the range of evidence assessed in a guideline process, we increase the amount of time and resources required. Nevertheless, the WHO has assessed the outputs of this process to be valuable and is currently repeating the approach used in OptimizeMNH in other guidance processes.

Knowledge synthesis methods for integrating qualitative and quantitative data: a scoping review reveals poor operationalization of the methodological steps

OBJECTIVES

To describe and compare, through a scoping review, emerging knowledge synthesis methods for integrating qualitative and quantitative evidence in health care, in terms of expertise required, similarities, differences, strengths, limitations, and steps involved in using the methods.

STUDY DESIGN AND SETTING

Electronic databases (e.g., MEDLINE) were searched, and two reviewers independently selected studies and abstracted data for qualitative analysis.

RESULTS

In total, 121 articles reporting seven knowledge synthesis methods (critical interpretive synthesis, integrative review, meta-narrative review, meta-summary, mixed studies review, narrative synthesis, and realist review) were included after screening of 17,962 citations and 1,010 full-text articles. Common similarities among methods related to the entire synthesis process, while common differences related to the research question and eligibility criteria. The most common strength was a comprehensive synthesis providing rich contextual data, whereas the most common weakness was a highly subjective method that was not reproducible. For critical interpretive synthesis, meta-narrative review, meta-summary, and narrative synthesis, guidance was not provided for some steps of the review process.

CONCLUSION

Some of the knowledge synthesis methods provided guidance on all steps, whereas other methods were missing guidance on the synthesis process. Further work is needed to clarify these emerging knowledge synthesis methods.

What's in a name? The challenge of describing interventions in systematic reviews: analysis of a random sample of reviews of non-pharmacological stroke interventions

OBJECTIVE

To assess, in a sample of systematic reviews of non-pharmacological interventions, the completeness of intervention reporting, identify the most frequently missing elements, and assess review authors' use of and beliefs about providing intervention information.

DESIGN

Analysis of a random sample of systematic reviews of non-pharmacological stroke interventions; online survey of review authors.

DATA SOURCES AND STUDY SELECTION

The Cochrane Library and PubMed were searched for potentially eligible systematic reviews and a random sample of these assessed for eligibility until 60 (30 Cochrane, 30 non-Cochrane) eligible reviews were identified.

DATA COLLECTION

In each review, the completeness of the intervention description in each eligible trial (n=568) was assessed by 2 independent raters using the Template for Intervention Description and Replication (TIDieR) checklist. All review authors (n=46) were invited to complete a survey.

RESULTS

Most reviews were missing intervention information for the majority of items. The most incompletely described items were: modifications, fidelity, materials, procedure and tailoring (missing from all interventions in 97%, 90%, 88%, 83% and 83% of reviews, respectively). Items that scored better, but were still incomplete for the majority of reviews, were: 'when and how much' (in 31% of reviews, adequate for all trials; in 57% of reviews, adequate for some trials); intervention mode (in 22% of reviews, adequate for all trials; in 38%, adequate for some trials); and location (in 19% of reviews, adequate for all trials). Of the 33 (71%) authors who responded, 58% reported having further intervention information but not including it, and 70% tried to obtain information.

CONCLUSIONS

Most focus on intervention reporting has been directed at trials. Poor intervention reporting in stroke systematic reviews is prevalent, compounded by poor trial reporting. Without adequate intervention descriptions, the conduct, usability and interpretation of reviews are restricted and therefore, require action by trialists, systematic reviewers, peer reviewers and editors.

Critical EPICOT items were absent in Cochrane human immunodeficiency virus systematic reviews: a bibliometric analysis

OBJECTIVES

To summarize the current gaps in human immunodeficiency virus (HIV) research evidence, describe the adequacy of reporting "implications for research," and map the number of studies that inform reviews with the prevalence of HIV for each country.

STUDY DESIGN AND SETTING

A bibliometric analyses of HIV reviews in the Cochrane Database of Systematic Reviews with content analysis of the "implications for research" section.

RESULTS

We analyzed 103 high-quality reviews published as of March 2014. They included a median of five studies (min 0 and max 44). Almost all the reviews recommended more trials (89.3%). Limitations in trial design, duration, setting, sample size, and choice of participants were also noted. Reporting of EPICOT+ items was as follows: evidence (35.9%), population (57.3%), intervention (71.8%), comparison (20.4%), outcomes (57.3%), time stamp (34.0%), and disease burden (13.6%). The primary studies were conducted in 67 countries. Six of the top 10 countries in which primary studies were conducted had a high HIV prevalence.

CONCLUSION

Knowledge gaps were identified for research in younger participants, over longer periods, using more pragmatic interventions, conducted in resource-limited settings and incorporating economic evaluations. Implications for research are not always reported according to the EPICOT+ format. Not all countries with a high prevalence of HIV are contributing sufficiently to HIV research.

The effect of patient-practitioner communication on pain: a systematic review

BACKGROUND AND OBJECTIVE

Communication between patients and health care practitioners is expected to benefit health outcomes. The objective of this review was to assess the effects of experimentally varied communication on clinical patients' pain.

DATABASES AND DATA TREATMENT

We searched in July 2012, 11 databases supplemented with forward and backward searches for (quasi-) randomized controlled trials in which face-to-face communication was manipulated. We updated in June 2015 using the four most relevant databases (CINAHL, Cochrane Central, Psychinfo, PubMed).

RESULTS

Fifty-one studies covering 5079 patients were included. The interventions were separated into three categories: cognitive care, emotional care, procedural preparation. In all but five studies the outcome concerned acute pain. We found that, in general, communication has a small effect on (acute) pain. The 19 cognitive care studies showed that a positive suggestion may reduce pain, whereas a negative suggestion may increase pain, but effects are small. The 14 emotional care studies showed no evidence of a direct effect on pain, although four studies showed a tendency for emotional care lowering patients' pain. Some of the 23 procedural preparation interventions showed a weak to moderate effect on lowering pain.

CONCLUSIONS

Different types of communication have a significant but small effect on (acute) pain. Positive suggestions and informational preparation seem to lower patients' pain. Communication interventions show a large variety in quality, complexity and methodological rigour; they often used multiple components and it remains unclear what the effective elements of communication are. Future research is warranted to identify the effective components.

Extending the PRISMA statement to equity-focused systematic reviews (PRISMA-E 2012): explanation and elaboration

BACKGROUND

The promotion of health equity, the absence of avoidable and unfair differences in health outcomes, is a global imperative. Systematic reviews are an important source of evidence for health decision-makers, but have been found to lack assessments of the intervention effects on health equity. The Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) is a 27 item checklist intended to improve transparency and reporting of systematic reviews. We developed an equity extension for PRISMA (PRISMA-E 2012) to help systematic reviewers identify, extract, and synthesise evidence on equity in systematic reviews.

METHODS AND FINDINGS

In this explanation and elaboration paper we provide the rationale for each extension item. These items are additions or modifications to the existing PRISMA Statement items, in order to incorporate a focus on equity. An example of good reporting is provided for each item as well as the original PRISMA item.

CONCLUSIONS

This explanation and elaboration document is intended to accompany the PRISMA-E 2012 Statement and the PRISMA Statement to improve understanding of the reporting guideline for users. The PRISMA-E 2012 reporting guideline is intended to improve transparency and completeness of reporting of equity-focused systematic reviews. Improved reporting can lead to better judgement of applicability by policy makers which may result in more appropriate policies and programs and may contribute to reductions in health inequities. To encourage wide dissemination of this article it is accessible on the International Journal for Equity in Health, Journal of Clinical Epidemiology, and Journal of Development Effectiveness web sites.

Conceptual decomposition of complex health care interventions for evidence synthesis: a literature review

RATIONALE, AIMS AND OBJECTIVES

The clarity of pivotal concepts is an important prerequisite for the development, evaluation and exchange of scientific ideas. The term 'complex intervention' is increasingly used in the health care literature, although it often remains unclear what is actually meant by this concept. Therefore, our aim was to analyse the literature regarding definitions of the terms 'complex intervention' and 'components' of such interventions.

METHOD

To identify the methodological publications, systematic and snowballing techniques were combined for the literature search. Relevant units of meaning were extracted from 68 included publications. Afterwards, we deduced categories and related frequencies by inductive and quantitative content analysis techniques.

RESULTS

Several types of complexity were distinguished in the literature. Most authors viewed complex interventions as multicomponent interventions that are characterized by interactions between the components themselves, with the context or as systemic interventions. Components of complex interventions were described in the publications as having the potential to causally influence outcomes, thus being essential for achieving an effect. Other definitions and inconsistencies among the definitions are highlighted and discussed.

CONCLUSIONS

From our synthesis, we derived definitions of the central health care-related concepts 'complex intervention' and 'components' of an intervention. Although we found numerous diverse definitions, they could be reduced to a defined number of core characteristics. These characteristics may facilitate communication regarding complex interventions and enable the deduction of methodological approaches for evidence synthesis.

Gloves, gowns and masks for reducing the transmission of meticillin-resistant Staphylococcus aureus (MRSA) in the hospital setting

BACKGROUND

Meticillin-resistant Staphylococcus aureus (MRSA; also known as methicillin-resistant S aureus) is a common hospital-acquired pathogen that increases morbidity, mortality, and healthcare costs. Its control continues to be an unresolved issue in many hospitals worldwide. The evidence base for the effects of the use of gloves, gowns or masks as control measures for MRSA is unclear.

OBJECTIVES

To assess the effectiveness of wearing gloves, a gown or a mask when contact is anticipated with a hospitalised patient colonised or infected with MRSA, or with the patient's immediate environment.

SEARCH METHODS

We searched the Specialised Registers of three Cochrane Groups (Wounds Group on 5 June 2015; Effective Practice and Organisation of Care (EPOC) Group on 9 July 2013; and Infectious Diseases Group on 5 January 2009); CENTRAL (The Cochrane Library 2015, Issue 6); DARE, HTA, NHS EED, and the Methodology Register (The Cochrane Library 2015, Issue 6); MEDLINE and MEDLINE In-Process & Other Non-Indexed Citations (1946 to June week 1 2015); EMBASE (1974 to 4 June 2015); Web of Science (WOS) Core Collection (from inception to 7 June 2015); CINAHL (1982 to 5 June 2015); British Nursing Index (1985 to 6 July 2010); and ProQuest Dissertations & Theses Database (1639 to 11 June 2015). We also searched three trials registers (on 6 June 2015), references list of articles, and conference proceedings. We finally contacted relevant individuals for additional studies.

SELECTION CRITERIA

Studies assessing the effects on MRSA transmission of the use of gloves, gowns or masks by any person in the hospital setting when contact is anticipated with a hospitalised patient colonised or infected with MRSA, or with the patient's immediate environment. We did not assess adverse effects or economic issues associated with these interventions.We considered any comparator to be eligible. With regard to study design, only randomised controlled trials (clustered or not) and the following non-randomised experimental studies were eligible: quasi-randomised controlled trials (clustered or not), non-randomised controlled trials (clustered or not), controlled before-and-after studies, controlled cohort before-after studies, interrupted time series studies (controlled or not), and repeated measures studies. We did not exclude any study on the basis of language or date of publication.

DATA COLLECTION AND ANALYSIS

Two review authors independently decided on eligibility of the studies. Had any study having been included, two review authors would have extracted data (at least for outcome data) and assessed the risk of bias independently. We would have followed the standard methodological procedures suggested by Cochrane and the Cochrane EPOC Group for assessing risk of bias and analysing the data.

MAIN RESULTS

We identified no eligible studies for this review, either completed or ongoing.

AUTHORS' CONCLUSIONS

We found no studies assessing the effects of wearing gloves, gowns or masks for contact with MRSA hospitalised patients, or with their immediate environment, on the transmission of MRSA to patients, hospital staff, patients' caregivers or visitors. This absence of evidence should not be interpreted as evidence of no effect for these interventions. The effects of gloves, gowns and masks in these circumstances have yet to be determined by rigorous experimental studies, such as cluster-randomised trials involving multiple wards or hospitals, or interrupted time series studies.