Development of the WHO-INTEGRATE evidence-to-decision framework: an overview of systematic reviews of decision criteria for health decision-making

Developing and Evaluating Digital Public Health Interventions Using the Digital Public Health Framework DigiPHrame: A Framework Development Study

BACKGROUND

Digital public health (DiPH) interventions may help us tackle substantial public health challenges and reach historically underserved populations, in addition to presenting valuable opportunities to improve and complement existing services. However, DiPH interventions are often triggered through technological advancements and opportunities rather than public health needs. To develop and evaluate interventions designed to serve public health needs, a comprehensive framework is needed that systematically covers all aspects with relevance for public health. This includes considering the complexity of the technology, the context in which the technology is supposed to operate, its implementation, and its effects on public health, including ethical, legal, and social aspects.

OBJECTIVE

We aimed to develop such a DiPH framework with a comprehensive list of core principles to be considered throughout the development and evaluation process of any DiPH intervention.

METHODS

The resulting digital public health framework (DigiPHrame) was based on a scoping review of existing digital health and public health frameworks. After extracting all assessment criteria from these frameworks, we clustered the criteria. During a series of multidisciplinary meetings with experts from the Leibniz ScienceCampus Digital Public Health, we restructured each domain to represent the complexity of DiPH. In this paper, we used a COVID-19 contact-tracing app as a use case to illustrate how DigiPHrame may be applied to assess DiPH interventions.

RESULTS

The current version of DigiPHrame consists of 182 questions nested under 12 domains. Domain 1 describes the current status of health needs and existing interventions; domains 2 and 3, the DiPH technology under assessment and aspects related to human-computer interaction, respectively; domains 4 and 5, structural and process aspects, respectively; and domains 6-12, contextual conditions and the outcomes of the DiPH intervention from broad perspectives. In the CWA use case, a number of questions relevant during its development but also important for assessors once the CWA was available were highlighted.

CONCLUSIONS

DigiPHrame is a comprehensive framework for the development and assessment of digital technologies designed for public health purposes. It is a living framework and will, therefore, be updated regularly and as new public health needs and technological advancements emerge.

Applying the WHO-INTEGRATE evidence-to-decision framework in the development of WHO guidelines on parenting interventions: step-by-step process and lessons learnt

BACKGROUND

Development of guidelines for public health, health system, and health policy interventions demands complex systems thinking to understand direct and indirect effects of interventions within dynamic systems. The WHO-INTEGRATE framework, an evidence-to-decision framework rooted in the norms and values of the World Health Organization (WHO), provides a structured method to assess complexities in guidelines systematically, such as the balance of an intervention's health benefits and harms and their human rights and socio-cultural acceptability. This paper provides a worked example of the application of the WHO-INTEGRATE framework in developing the WHO guidelines on parenting interventions to prevent child maltreatment, and shares reflective insights regarding the value added, challenges encountered, and lessons learnt.

METHODS

The methodological approach comprised describing the intended step-by-step application of the WHO-INTEGRATE framework and gaining reflective insights from introspective sessions within the core team guiding the development of the WHO guidelines on parenting interventions and a methodological workshop.

RESULTS

The WHO-INTEGRATE framework was used throughout the guideline development process. It facilitated reflective deliberation across a broad range of decision criteria and system-level aspects in the following steps: (1) scoping the guideline and defining stakeholder engagement, (2) prioritising WHO-INTEGRATE sub-criteria and guideline outcomes, (3) using research evidence to inform WHO-INTEGRATE criteria, and (4) developing and presenting recommendations informed by WHO-INTEGRATE criteria. Despite the value added, challenges, such as substantial time investment required, broad scope of prioritised sub-criteria, integration across diverse criteria, and sources of evidence and translation of insights into concise formats, were encountered.

CONCLUSIONS

Application of the WHO-INTEGRATE framework was crucial in the integration of effectiveness evidence with insights into implementation and broader implications of parenting interventions, extending beyond health benefits and harms considerations and fostering a whole-of-society-perspective. The evidence reviews for prioritised WHO-INTEGRATE sub-criteria were instrumental in guiding guideline development group discussions, informing recommendations and clarifying uncertainties. This experience offers important lessons for future guideline panels and guideline methodologists using the WHO-INTEGRATE framework.

Real-World Evidence to Reinforce Clinical Trial Evidence in Health Technology Assessment: A Critical Review of Real-World Evidence Requirements from Seven Countries and Recommendations to Improve Acceptance

BACKGROUND

Real-world evidence (RWE) can reinforce clinical trial evidence in health technology assessment (HTA).

OBJECTIVES

Review HTA bodies' (HTAbs) requirements for RWE, real uses, and acceptance across seven countries (Brazil, Canada, France, Germany, Italy, Spain, and the United Kingdom) and outline recommendations that may improve acceptance of RWE in efficacy/effectiveness assessments and appraisals processes.

METHODS

RWE requirements were summarized based on HTAbs' guidelines. Acceptance by HTAbs was evaluated based on industry experience and case studies.

RESULTS

As of June 2022, RWE methodological guidelines were in place in three of the seven countries. HTAbs typically requested analyses based on local data sources, but the preferred study design and data sources differed. HTAbs had individual submission, assessment, and appraisal processes; some allowed early meetings for the protocol and/or results validation, though few involved external experts or medical societies to provide input to assessment and appraisal. The extent of submission, assessment, and appraisal requirements did not necessarily reflect the degree of acceptance.

CONCLUSION

All the countries reviewed face common challenges regarding the use of RWE. Our proposals address the need to facilitate collaboration and communication with industry and regulatory agencies and the need for specific guidelines describing RWE design and criteria of acceptance throughout the assessment and appraisal processes.

Exploring evidence use and capacity for health services management and planning in Swiss health administrations: A mixed-method interview study

BACKGROUND

Health administrations require evidence, meaning robust information, data, and research, on health services and systems. Little is known about the resources and processes available within administrations to support evidence-informed policymaking. This study assessed Swiss health administrations' capacity for evidence use and investigated civil servants' needs and perspectives regarding the role and use of evidence in health services management and planning.

METHODS

In this mixed-method study, we interviewed civil servants from Swiss German-speaking cantonal health administrations. We quantitatively assessed administrations' organization-level capacity by applying six structured interviews using an existing measurement tool (ORACLe). Individual-level needs and perspectives regarding evidence use and capacity were qualitatively explored with twelve in-depth interviews that were analyzed using the framework method.

FINDINGS

Respondents indicated moderate evidence-use capacity in all administrations. Administrations displayed a similar pattern of high and low capacity in specific capacity areas, generally with considerable variation within administrations. Most administrations indicated high capacity for producing or commissioning evidence and close relationships with research. They showed limited capacity in the documentation of processes and availability of tools, programs, or training opportunities. Administrations place the responsibility for engagement with evidence at the level of individual civil servants rather than at the organizational level. Although administrations highly value evidence-informed policymaking and consider it vital to effective health services management and planning, they face significant constraints in accessing evidence-specific resources and receive little organizational support. Administrations rely on external capacity to compensate for these limitations and engage with evidence pragmatically.

CONCLUSION

Our findings indicate moderate and improvable capacity for evidence use in Swiss health administrations that place limited value on organizational support. Besides strengthening organizational support, leadership buy-in, particular staff needs, and balancing the implementation of specific measures with the provision of more general resources should be considered to unlock the potential of strengthened engagement with evidence.

Priority-setting for hospital funding of high-cost innovative drugs and therapeutics: A qualitative institutional case study

OBJECTIVES

Rising costs of innovative drugs and therapeutics (D&Ts) have led to resource allocation challenges for healthcare institutions. There is limited evidence to guide priority-setting for institutional funding of high-cost D&Ts. This study sought to identify and elaborate on the substantive principles and procedures that should inform institutional funding decisions for high-cost off-formulary D&Ts through a case study of a quaternary care paediatric hospital.

METHODS

Semi-structured, qualitative interviews, both virtual and in-person, were conducted with institutional stakeholders (i.e. staff clinicians, senior leadership, and pharmacists) (n = 23) and two focus groups at The Hospital for Sick Children in Toronto, Canada. Participants involved in, and impacted by, high-cost off-formulary drug funding decisions were recruited through stratified, purposive sampling. Participants were approached for study involvement between July 27, 2020 and June 7, 2022. Data was analysed through reflexive thematic analysis.

RESULTS

Institutional resource allocation for high-cost D&Ts was identified as ethically challenging but critical to sustainable access to novel therapies. Important substantive principles included: 1) clinical evidence of safety and efficacy, 2) economic considerations (direct costs, opportunity costs, value for money), 3) ethical principles (social justice, professional/organizational responsibility), and 4) disease-specific considerations. Multidisciplinary deliberation was identified as an essential procedural component of decision-making. Participants identified tension between innovation and the need for evidence-based decision-making; clinician and institutional responsibilities; and value for money and social justice. Participants emphasized the role of health system-level funding allocation in alleviating the financial and moral burden of decision-making by institutions.

CONCLUSIONS

This study identifies values and processes to aid in the development and implementation of institutional resource allocation frameworks for high-cost innovative D&Ts.

Institutional Priority-Setting for Novel Drugs and Therapeutics: A Qualitative Systematic Review

BACKGROUND

There is a lack of guidance on approaches to formulary management and funding for high-cost drugs and therapeutics by individual healthcare institutions. The objective of this review was to assess institutional approaches to resource allocation for such therapeutics, with a particular focus on paediatric and rare disease populations.

METHODS

A search of Embase and MEDLINE was conducted for studies relevant to decision-making for off-formulary, high-cost drugs and therapeutics. Abstracts were evaluated for inclusion based on the Simple Multiple-Attribute Rating Techniques (SMART) criteria. A framework of 30 topics across 4 categories was used to guide data extraction and was based on findings from the initial abstract review and previous health technology assessment (HTA) publications. Reflexive thematic analysis was conducted using QSR NVivo 12 software.

RESULTS

A total of 168 studies were included for analysis. Only 4 (2%) focused on paediatrics, while 21 (12%) centred on adults and the remainder (85%) did not specify. Thirty-two (19%) studies discussed the importance of high-cost therapeutics and 34 (23%) focused on rare/orphan drugs. Five themes were identified as being relevant to institutional decision-making for high-cost therapeutics: institutional strategy, substantive criteria, procedural considerations, guiding principles and frameworks, and operational activities. Each of these themes encompassed several sub-themes and was complemented by a sixth category specific to paediatrics and rare diseases.

CONCLUSION

The rising cost of novel drugs and therapeutics underscores the need for robust, evidence-based, and ethically defensible decision-making processes for health technology funding, particularly at the hospital level. Our study highlights practices and themes to aid decision-makers in thinking critically about institutional, substantive, procedural, and operational considerations in support of legitimate decisions about institutional funding of high-cost drugs and therapeutics, as well as opportunities and challenges that exist for paediatric and rare disease populations.

The potential impact of novel tuberculosis vaccine introduction on economic growth in low- and middle-income countries: A modeling study

BACKGROUND

Most individuals developing tuberculosis (TB) are working age adults living in low- and middle-income countries (LMICs). The resulting disability and death impact economic productivity and burden health systems. New TB vaccine products may reduce this burden. In this study, we estimated the impact of introducing novel TB vaccines on gross domestic product (GDP) growth in 105 LMICs.

METHODS AND FINDINGS

We adapted an existing macroeconomic model to simulate country-level GDP trends between 2020 and 2080, comparing scenarios for introduction of hypothetical infant and adolescent/adult vaccines to a no-new-vaccine counterfactual. We parameterized each scenario using estimates of TB-related mortality, morbidity, and healthcare spending from linked epidemiological and costing models. We assumed vaccines would be introduced between 2028 and 2047 and estimated incremental changes in GDP within each country from introduction to 2080, in 2020 US dollars. We tested the robustness of results to alternative analytic specifications. Both vaccine scenarios produced greater cumulative GDP in the modeled countries over the study period, equivalent to $1.6 (95% uncertainty interval: $0.8, 3.0) trillion for the adolescent/adult vaccine and $0.2 ($0.1, 0.4) trillion for the infant vaccine. These GDP gains were substantially lagged relative to the time of vaccine introduction, particularly for the infant vaccine. GDP gains resulting from vaccine introduction were concentrated in countries with higher current TB incidence and earlier vaccine introduction. Results were sensitive to secular trends in GDP growth but relatively robust to other analytic assumptions. Uncertain projections of GDP could alter these projections and affect the conclusions drawn by this analysis.

CONCLUSIONS

Under a range of assumptions, introducing novel TB vaccines would increase economic growth in LMICs.

Cost-Effectiveness of Thrombopoietin Mimetics in Patients with Thrombocytopenia: A Systematic Review

OBJECTIVES

Thrombopoietin (TPO) mimetics are a potential alternative to platelet transfusion to minimize blood loss in patients with thrombocytopenia. This systematic review aimed to evaluate the cost-effectiveness of TPO mimetics, compared with not using TPO mimetics, in adult patients with thrombocytopenia.

METHODS

Eight databases and registries were searched for full economic evaluations (EEs) and randomized controlled trials (RCTs). Incremental cost-effectiveness ratios (ICERs) were synthesized as cost per quality-adjusted life year gained (QALY) or as cost per health outcome (e.g. bleeding event avoided). Included studies were critically appraised using the Philips reporting checklist.

RESULTS

Eighteen evaluations from nine different countries were included, evaluating the cost-effectiveness of TPO mimetics compared with no TPO, watch-and-rescue therapy, the standard of care, rituximab, splenectomy or platelet transfusion. ICERs varied from a dominant strategy (i.e. cost-saving and more effective), to an incremental cost per QALY/health outcome of EUR 25,000-50,000, EUR 75,000-750,000 and EUR > 1 million, to a dominated strategy (cost-increasing and less effective). Few evaluations (n = 2, 10%) addressed the four principal types of uncertainty (methodological, structural, heterogeneity and parameter). Parameter uncertainty was most frequently reported (80%), followed by heterogeneity (45%), structural uncertainty (43%) and methodological uncertainty (28%).

CONCLUSIONS

Cost-effectiveness of TPO mimetics in adult patients with thrombocytopenia ranged from a dominant strategy to a significant incremental cost per QALY/health outcome or a strategy that is clinically inferior and has increased costs. Future validation and tackling the uncertainty of these models with country-specific cost data and up-to-date efficacy and safety data are needed to increase the generalizability.

Prioritisation processes for programme implementation and evaluation in public health: A scoping review

Background

Programme evaluation is an essential and systematic activity for improving public health programmes through useful, feasible, ethical, and accurate methods. Finite budgets require prioritisation of which programmes can be funded, first, for implementation, and second, evaluation. While criteria for programme funding have been discussed in the literature, a similar discussion around criteria for which programmes are to be evaluated is limited. We reviewed the criteria and frameworks used for prioritisation in public health more broadly, and those used in the prioritisation of programmes for evaluation. We also report on stakeholder involvement in prioritisation processes, and evidence on the use and utility of the frameworks or sets of criteria identified. Our review aims to inform discussion around which criteria and domains are best suited for the prioritisation of public health programmes for evaluation.

Methods

We reviewed the peer-reviewed literature through OVID MEDLINE (PubMed) on 11 March 2022. We also searched the grey literature through Google and across key websites including World Health Organization (WHO), US Centers for Disease Control and Prevention (CDC), European Centre for Disease Prevention and Control (ECDC), and the International Association of National Public Health Institutes (IANPHI) (14 March 2022). Articles were limited to those published between 2002 and March 2022, in English, French or German.

Results

We extracted over 300 unique criteria from 40 studies included in the analysis. These criteria were categorised into 16 high-level conceptual domains to allow synthesis of the findings. The domains most frequently considered in the studies were “burden of disease” (33 studies), “social considerations” (30 studies) and “health impacts of the intervention” (28 studies). We only identified one paper which proposed criteria for use in the prioritisation of public health programmes for evaluation. Few prioritisation frameworks had evidence of use outside of the setting in which they were developed, and there was limited assessment of their utility. The existing evidence suggested that prioritisation frameworks can be used successfully in budget allocation, and have been reported to make prioritisation more robust, systematic, transparent, and collaborative.

Conclusion

Our findings reflect the complexity of prioritisation in public health. Development of a framework for the prioritisation of programmes to be evaluated would fill an evidence gap, as would formal assessment of its utility. The process itself should be formal and transparent, with the aim of engaging a diverse group of stakeholders including patient/public representatives.

Do Health Technology Assessment organisations consider manufacturers' costs in relation to drug price? A study of reimbursement reports

Introduction

Drug reimbursement decisions are often made based on a price set by the manufacturer. In some cases, this price leads to public and scientific debates about whether its level can be justified in relation to its costs, including those related to research and development (R&D) and manufacturing. Such considerations could enter the decision process in collectively financed health care systems. This paper investigates whether manufacturers’ costs in relation to drug prices, or profit margins, are explicitly mentioned and considered by health technology assessment (HTA) organisations.

Method

An analysis of reimbursement reports for cancer drugs was performed. All relevant Dutch HTA-reports, published between 2017 and 2019, were selected and matched with HTA-reports from three other jurisdictions (England, Canada, Australia). Information was extracted. Additionally, reimbursement reports for three cases of expensive non-oncolytic orphan drugs prominent in pricing debates in the Netherlands were investigated in depth to examine consideration of profit margins.

Results

A total of 66 HTA-reports concerning 15 cancer drugs were included. None of these reports contained information on manufacturer’s costs or profit margins. Some reports contained general considerations of the HTA organisation which related prices to manufacturers’ costs: six contained a statement on the lack of price setting transparency, one mentioned recouping R&D costs as a potential argument to justify a high price. For the case studies, 21 HTA-reports were selected. One contained a cost-based price justification provided by the manufacturer. None of the other reports contained information on manufacturer’s costs or profit margins. Six reports contained a discussion about lack of transparency. Reports from two jurisdictions contained invitations to justify high prices by demonstrating high costs.

Conclusion

Despite the attention given to manufacturers’ costs in relation to price in public debates and in the literature, this issue does not seem to get explicit systematic consideration in the reimbursement reports of expensive drugs.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12962-022-00383-y.

Adverse and other unintended consequences of setting-based interventions to prevent illicit drug use: A systematic review of reviews

This article explores adverse and unintended consequences (AUCs) of setting-based public health interventions to prevent illicit drug use, including the mechanisms leading to these AUCs. Additionally, the reporting of AUCs in systematic reviews was assessed. Therefore, we conducted a systematic review of reviews and searched four big databases were searched. We included systematic reviews concerned with setting-based interventions to prevent illicit drug use. We used AMSTAR 2 to rate the overall confidence of the results presented in the reviews. Data on study characteristics, types and mechanisms of AUCs were extracted. An a priori categorisation of consequences drew on the WHO-INTEGRATE framework, and the categorisation of mechanisms on the Behaviour Change Wheel. For reviews reporting AUCs, the same information was also retrieved from relevant primary studies. Findings were synthesised narratively and in tables. Finally, we included 72 reviews, of which 18 reported on AUCs. From these, 11 primary studies were identified. Most of the reviews and primary studies were conducted in educational settings. The most prevalent AUCs reported in systematic reviews and primary studies were paradoxical health effects (i.e. increase of drug use). Potential mechanisms discussed primarily focussed on the change though social norms and practices. Changes of knowledge and perception were also mentioned. Concluding, the identified reviews and primary studies paid insufficient attention to AUCs of public health interventions to prevent illicit drug use. Where reported, it was mostly as an afterthought and narrowly framed as health related. No mentions of potential broader social consequences were found.

Do Profit Margins of Pharmaceuticals Influence Reimbursement Decisions? A Discrete Choice Experiment Among Dutch Healthcare Decision Makers

OBJECTIVES

This study aimed to investigate whether the profit margins of pharmaceuticals would influence the outcome of reimbursement decisions within the Dutch policy context.

METHODS

We conducted a discrete choice experiment among 58 Dutch decision makers. In 20 choice sets, we asked respondents to indicate which of 2 pharmaceutical treatment options they would select for reimbursement. Options were described using 5 attributes (disease severity, incremental costs per quality-adjusted life-year, health gain, budget impact, and profit margin) with 3 levels each. Additionally, cognitive debriefing questions were presented, and for validation debriefing, interviews were conducted. Choice data were analyzed using mixed logit models, also to calculate marginal effects and choice probabilities.

RESULTS

Results indicated that the specified levels of profit margins significantly influenced choices made. Decision makers were less likely to reimburse a product with a higher profit margin. The relative importance of profit margins was lower than that of the included traditional health technology assessment criteria, but not negligible. When asked directly, 61% of respondents indicated that profit margin should play a role in reimbursement decision making, although concerns about feasibility and the connection to price negotiations were voiced.

CONCLUSIONS

Our results suggest that if available to decision makers the profit margin of pharmaceutical products would influence reimbursement decisions within the Dutch policy context. Higher profit margins would reduce the likelihood of reimbursement. Whether adding profit margin as an additional, explicit criterion to the health technology assessment decision framework would be feasible and desirable is open to further exploration.

Recommendations for the development of Clinical Practice Guidelines

Clinical practice guidelines are statements that include recommendations intended to optimize patient care, are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options, and ensure that the best available clinical knowledge is used to provide effective and quality care. They can reduce inappropriate care and variability in clinical practice and can support the translation of new research knowledge into clinical practice. Recommendations from clinical practice guidelines can support health professionals by facilitating the decision-making process, empowering them to make more informed health care choices, clarifying which interventions should be priorities based on a favorable trade-off, and discouraging the use of those that have proven ineffective, dangerous, or wasteful. This review aims to summarize the key components of high-quality and trustworthy guidelines. Articles were retrieved from various libraries, databases, and search engines using free-text term searches adapted for different databases, and selected according to author discretion. Clinical practice guidelines in geriatrics can have a major impact on prevention, diagnosis, treatment, rehabilitation, health care, and the management of diseases and conditions, but they should only be implemented when they have high-quality, rigorous, and unbiased methodologies that consider older adult priorities and provide valid recommendations.

Safety of Dietary Guanidinoacetic Acid: A Villain of a Good Guy?

Guanidinoacetic acid (GAA) is a natural amino acid derivative that is well-recognized for its central role in the biosynthesis of creatine, an essential compound involved in cellular energy metabolism. GAA (also known as glycocyamine or betacyamine) has been investigated as an energy-boosting dietary supplement in humans for more than 70 years. GAA is suggested to effectively increase low levels of tissue creatine and improve clinical features of cardiometabolic and neurological diseases, with GAA often outcompeting traditional bioenergetics agents in maintaining ATP status during stress. This perhaps happens due to a favorable delivery of GAA through specific membrane transporters (such as SLC6A6 and SLC6A13), previously dismissed as un-targetable carriers by other therapeutics, including creatine. The promising effects of dietary GAA might be countered by side-effects and possible toxicity. Animal studies reported neurotoxic and pro-oxidant effects of GAA accumulation, with exogenous GAA also appearing to increase methylation demand and circulating homocysteine, implying a possible metabolic burden of GAA intervention. This mini-review summarizes GAA toxicity evidence in human nutrition and outlines functional GAA safety through benefit-risk assessment and multi-criteria decision analysis.

Evidence-to-decision frameworks: a review and analysis to inform decision-making for environmental health interventions

BACKGROUND

Evidence-to-decision (EtD) frameworks provide a structured and transparent approach for groups of experts to use when formulating recommendations or making decisions. While extensively used for clinical and public health recommendations, EtD frameworks are not in widespread use in environmental health. This review sought to identify, compare and contrast key EtD frameworks for decisions on interventions used in clinical medicine, public health or environmental health. This information can be used to develop an EtD framework suitable for formulating recommendations for interventions in environmental health.

METHODS

We identified a convenience sample of EtD frameworks used by a range of organizations. We searched Medline for systematic reviews of frameworks. We summarized the decision criteria in the selected frameworks and reviews in a qualitative manner.

FINDINGS

Fourteen organizations provided 18 EtD frameworks; most frameworks focused on clinical medicine or public health interventions; four focused on environmental health and three on economic considerations. Harms of interventions were examined in all frameworks and benefits in all but one. Other criteria included certainty of the body of evidence (15 frameworks), resource considerations (15), feasibility (13), equity (12), values (11), acceptability (11), and human rights (2). There was variation in how specific criteria were defined. The five identified systematic reviews reported a similar spectrum of EtD criteria.

INTERPRETATION

The EtD frameworks examined encompassed similar criteria, with tailoring to specific audience needs. Existing frameworks are a useful starting point for development of one tailored to decision-making in environmental health.

FUNDER

JPB Foundation.

Modular literature review: a novel systematic search and review method to support priority setting in health policy and practice

BACKGROUND

There is an unmet need for review methods to support priority-setting, policy-making and strategic planning when a wide variety of interventions from differing disciplines may have the potential to impact a health outcome of interest. This article describes a Modular Literature Review, a novel systematic search and review method that employs systematic search strategies together with a hierarchy-based appraisal and synthesis of the resulting evidence.

METHODS

We designed the Modular Review to examine the effects of 43 interventions on a health problem of global significance. Using the PICOS (Population, Intervention, Comparison, Outcome, Study design) framework, we developed a single four-module search template in which population, comparison and outcome modules were the same for each search and the intervention module was different for each of the 43 interventions. A series of literature searches were performed in five databases, followed by screening, extraction and analysis of data. "ES documents", source documents for effect size (ES) estimates, were systematically identified based on a hierarchy of evidence. The evidence was categorised according to the likely effect on the outcome and presented in a standardised format with quantitative effect estimates, meta-analyses and narrative reporting. We compared the Modular Review to other review methods in health research for its strengths and limitations.

RESULTS

The Modular Review method was used to review the impact of 46 antenatal interventions on four specified birth outcomes within 12 months. A total of 61,279 records were found; 35,244 were screened by title-abstract. Six thousand two hundred seventy-two full articles were reviewed against the inclusion criteria resulting in 365 eligible articles.

CONCLUSIONS

The Modular Review preserves principles that have traditionally been important to systematic reviews but can address multiple research questions simultaneously. The result is an accessible, reliable answer to the question of "what works?". Thus, it is a well-suited literature review method to support prioritisation, decisions and planning to implement an agenda for health improvement.

Appraising clinical applicability of studies: mapping and synthesis of current frameworks, and proposal of the FrACAS framework and VICORT checklist

BACKGROUND

Not all research findings are translated to clinical practice. Reasons for lack of applicability are varied, and multiple frameworks and criteria exist to appraise the general applicability of epidemiological and clinical research. In this two-part study, we identify, map, and synthesize frameworks and criteria; we develop a framework to assist clinicians to appraise applicability specifically from a clinical perspective.

METHODS

We conducted a literature search in PubMed and Embase to identify frameworks appraising applicability of study results. Conceptual thematic analysis was used to synthesize frameworks and criteria. We carried out a framework development process integrating contemporary debates in epidemiology, findings from the literature search and synthesis, iterative pilot-testing, and brainstorming and consensus discussions to propose a concise framework to appraise clinical applicability.

RESULTS

Of the 4622 references retrieved, we identified 26 unique frameworks featuring 21 criteria. Frameworks and criteria varied by scope and level of aggregation of the evidence appraised, target user, and specific area of applicability (internal validity, clinical applicability, external validity, and system applicability). Our proposed Framework Appraising the Clinical Applicability of Studies (FrACAS) classifies studies in three domains (research, practice informing, and practice changing) by examining six criteria sequentially: Validity, Indication-informativeness, Clinical relevance, Originality, Risk-benefit comprehensiveness, and Transposability (VICORT checklist).

CONCLUSIONS

Existing frameworks to applicability vary by scope, target user, and area of applicability. We introduce FrACAS to specifically assess applicability from a clinical perspective. Our framework can be used as a tool for the design, appraisal, and interpretation of epidemiological and clinical studies.

Human papillomavirus (HPV) vaccination in the transition between adolescence and adulthood

PURPOSE

Young adulthood is characterized by changes in health care decision-making, insurance coverage, and sexual risk. Although the human papillomavirus (HPV) vaccine is now approved for adults up to age 45, and catch-up vaccination is currently recommended up through age 26, vaccination rates remain low in young adults. This study explored perspectives on HPV vaccination among young adults receiving care at the student health center of a large public university.

METHODS

We conducted semi-structured interviews (n = 27) and four focus groups with female and male undergraduate and graduate students (n = 18) and semi-structured interviews with health care providers (n = 6). Interviews and focus groups explored perceived risk of HPV infection, benefits of the HPV vaccine, and motivations for and barriers to HPV vaccination.

RESULTS

Many young adults cited their parents' views and recommendations from medical providers as influential on their decision-making process. Students perceived that cervical cancer prevention was a main benefit of the HPV vaccine and sexual activity was a risk factor for HPV infection. Students often lacked knowledge about the vaccine's benefits for males and expressed some concerns about the safety and side effects of a vaccine perceived as new. Logistical barriers to vaccination included uncertainty over vaccination status and insurance coverage for the vaccine, and concerns about balancing the vaccine schedule with school obligations. Providers' vaccine recommendations were impacted by health system factors, including clinical infrastructure, processes for recommending and documenting vaccination, and office visit priorities. Suggested vaccination promotion strategies included improving the timing and messaging of outreach efforts on campus and bolstering clinical infrastructure.

CONCLUSIONS

Although college may be an opportune time to reach young adults for HPV vaccination, obstacles including navigating parental influence and independent decision-making, lack of awareness of vaccination status, and numerous logistical and system-level barriers may impede vaccination during this time.

Guidelines that use the GRADE approach often fail to provide complete economic information for recommendations: A systematic survey

OBJECTIVE

Little is known about how developers and panel members report cost and cost effectiveness considerations in GRADE guideline Evidence-to-Decision (EtD) frameworks. A systematic survey was conducted to explore approaches and factors contributing to variability in economic information reporting.

STUDY DESIGN AND SETTING

Guideline organization websites were systematically searched to create a convenience sample of guidelines. Reviewers screened published EtD frameworks and generated frequencies of reporting approaches. We used thematic analysis to summarize factors related to variability of economic information reporting.

RESULTS

We included 142 guidelines. The overall rate of reporting economic information was high (91%); however, there was variability across completion of predefined EtD Likert-type judgments (70%), noting information as not identified across EtD framework domains (57%), and providing remarks to justify recommendations (38%). Six themes contributing to variability emerged, related to: intervention, population, payor, provider, healthcare resource use, and economic model building factors. Only 2 guidelines performed a GRADE certainty appraisal of economic outcomes.

CONCLUSION

Completing predefined EtD Likert-type judgments, specifically reporting a literature review approach, study selection criteria and economic model building limitations, as well as linking these to recommendation justification remarks are potential areas for improved use, adoption and adaptation of recommendation, and transparency of GRADE EtD frameworks.

Which Criteria are Considered and How are They Evaluated in Health Technology Assessments? A Review of Methodological Guidelines Used in Western and Asian Countries

OBJECTIVES

This study aimed to provide an exhaustive description of criteria and methodological recommendations for evaluating them in health technology assessment (HTA) in Western and Asian countries.

METHODS

We conducted a system literature review of HTA-related guidelines by searching the websites of HTA agencies and related data sources. The guidelines, reports, or recommendations introducing the HTA evaluation methods, processes, decision-making frameworks, and criteria for priority setting were eligible to be included. The review was limited to guidelines from countries belonging to the European Network for Health Technology Assessment (EUnetHTA) and HTAsiaLink organisations and other countries with well-established available guidelines.

RESULTS

A total of 52 guidelines from 24 countries were identified, including 13 countries from the EUnetHTA organisation, 9 countries from the HTAsiaLink organisation and 2 other countries (Canada and the USA). A strong consensus was observed among the HTA agencies on the core set of criteria including efficacy or effectiveness, cost-effectiveness, safety, and budget impact. More similarities were observed than differences in methodological recommendations for clinical and economic evaluations among the agencies.

CONCLUSIONS

Substantial convergence is seen in the criteria included in the HTA process, as well as the methods to evaluate and quantify them. Further efforts are needed to verify whether the criteria identified from the guidelines are incorporated in real HTA decisions, and how they are assessed and weighted in practice.

'They just came with the medication dispenser'- a qualitative study of elderly service users' involvement and welfare technology in public home care services

BACKGROUND

Public home care for the elderly is a key area in relation to improving health care quality. It is an important political goal to increase elderly people's involvement in their care and in the use of welfare technology. The aim of this study was to explore elderly service users' experience of user involvement in the implementation and everyday use of welfare technology in public home care services.

METHOD

This qualitative study has an explorative and descriptive design. Sixteen interviews of service users were conducted in five different municipalities over a period of six months. The data were analysed using reflexive thematic analysis.

RESULTS

Service users receiving public home care service are not a homogenous group, and the participants had different wishes and needs as regards user involvement and the use of welfare technology. The analysis led to four main themes: 1) diverse preferences as regards user involvement, 2) individual differences as regards information, knowledge and training, 3) feeling safe and getting help, and 4) a wish to stay at home for as long as possible.

CONCLUSION

The results indicated that user involvement was only to a limited extent an integral part of public home care services. Participants had varying insight into and interest in welfare technology, which was a challenge for user involvement. User involvement must be facilitated and implemented in a gentle way, highlighting autonomy and collaboration, and with the focus on respect, reciprocity and dialogue.

Selecting Review Outcomes for Systematic Reviews of Public Health Interventions

For systematic reviews to have an impact on public health, they must report outcomes that are important for decision-making. Systematic reviews of public health interventions, however, have a range of potential end users, and identifying and prioritizing the most important and relevant outcomes represents a considerable challenge.In this commentary, we describe potentially useful approaches that systematic review teams can use to identify review outcomes to best inform public health decision-making. Specifically, we discuss the importance of stakeholder engagement, the use of logic models, consideration of core outcome sets, reviews of the literature on end users' needs and preferences, and the use of decision-making frameworks in the selection and prioritization of outcomes included in reviews.The selection of review outcomes is a critical step in the production of public health reviews that are relevant to those who use them. Utilizing the suggested strategies may help the review teams better achieve this.

Multiple criteria decision analysis for medicine reimbursement in the Lebanese context

Objective

The objective of this exploratory analysis is to reflect and discuss which criteria of multiple criteria decision analysis (MCDA) would be relevant as part of value determination when appraising healthcare interventions in the Lebanese context.

Methods

A workshop was conducted as part of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Lebanon Chapter and included the two frameworks: Evidence and Value: Impact on Decision Making (EVIDEM) and Advance Value Framework. Thirty-seven participants expressed their individual preferences through a qualitative and a quantitative exercise.

Results

In the qualitative analysis of both frameworks, participants unanimously agreed on the relevance of comparative efficacy, safety, and impact of medical costs. In EVIDEM, disease severity and unmet needs were also considered to be important criteria by more than 90 percent of the participants. In the quantitative analysis of both frameworks, disease severity ranked first (a mean normalized weight of .1 in EVIDEM and .27 in Advance Value Framework), followed by the size of the population (.09), the type of therapeutic benefit at the patient level (.09) and population level (.08), and the efficacy (.07) in EVIDEM. In the Advance Value Framework, the combined unmet need/disease severity criteria were followed by direct and meaningful end points (.15), safety (.12), contraindications (.08), and indirect surrogate end points (.07).

Conclusions

The results were concordant with those reported in countries that have conducted similar surveys such as France, Italy, and Spain. The MCDA methodology could be used as a cornerstone to enhance evidence-based discussions among Lebanese stakeholders involved in evaluation and decision-making purposes.

WICID framework version 1.0: criteria and considerations to guide evidence-informed decision-making on non-pharmacological interventions targeting COVID-19

Introduction

Public health decision-making requires the balancing of numerous, often conflicting factors. However, participatory, evidence-informed decision-making processes to identify and weigh these factors are often not possible- especially, in the context of the SARS-CoV-2 pandemic. While evidence-to-decision frameworks are not able or intended to replace stakeholder participation, they can serve as a tool to approach relevancy and comprehensiveness of the criteria considered.

Objective

To develop a decision-making framework adapted to the challenges of decision-making on non-pharmacological interventions to contain the global SARS-CoV-2 pandemic.

Methods

We employed the ‘best fit’ framework synthesis technique and used the WHO-INTEGRATE framework as a starting point. First, we adapted the framework through brainstorming exercises and application to case studies. Next, we conducted a content analysis of comprehensive strategy documents intended to guide policymakers on the phasing out of applied lockdown measures in Germany. Based on factors and criteria identified in this process, we developed the WICID (WHO-INTEGRATE COVID-19) framework version 1.0.

Results

Twelve comprehensive strategy documents were analysed. The revised framework consists of 11+1 criteria, supported by 48 aspects, and embraces a complex systems perspective. The criteria cover implications for the health of individuals and populations due to and beyond COVID-19, infringement on liberties and fundamental human rights, acceptability and equity considerations, societal, environmental and economic implications, as well as implementation, resource and feasibility considerations.

Discussion

The proposed framework will be expanded through a comprehensive document analysis focusing on key stakeholder groups across the society. The WICID framework can be a tool to support comprehensive evidence-informed decision-making processes.

Use of the GRADE approach in health policymaking and evaluation: a scoping review of nutrition and physical activity policies

BACKGROUND

Nutrition and physical activity policies have the potential to influence lifestyle patterns and reduce the burden of non-communicable diseases. In the world of health-related guidelines, GRADE (Grading of Recommendations, Assessment, Development and Evaluation) is the most widely used approach for assessing the certainty of evidence and determining the strength of recommendations. Thus, it is relevant to explore its usefulness also in the process of nutrition and physical activity policymaking and evaluation. The purpose of this scoping review was (i) to generate an exemplary overview of documents using the GRADE approach in the process of nutrition and physical activity policymaking and evaluation, (ii) to find out how the GRADE approach has been applied, and (iii) to explore which facilitators of and barriers to the use of GRADE have been described on the basis of the identified documents. The overarching aim of this work is to work towards improving the process of evidence-informed policymaking in the areas of dietary behavior, physical activity, and sedentary behavior.

METHODS

A scoping review was conducted according to current reporting standards. MEDLINE via Ovid, the Cochrane Library, and Web of Science were systematically searched up until 4 July 2019. Documents describing a body of evidence which was assessed for the development or evaluation of a policy, including documents labeled as "guidelines," or systematic reviews used to inform policymaking were included.

RESULTS

Thirty-six documents were included. Overall, 313 GRADE certainty of evidence ratings were identified in systematic reviews and guidelines; the strength of recommendations/policies was assessed in four documents, and six documents mentioned facilitators or barriers for the use of GRADE. The major reported barrier was the initial low starting level of a body of evidence from non-randomized studies when assessing the certainty of evidence.

CONCLUSION

This scoping review found that the GRADE approach has been used for policy evaluations, in the evaluation of the effectiveness of policy-relevant interventions (policymaking), as well as in the development of guidelines intended to guide policymaking. Several areas for future research were identified to explore the use of GRADE in health policymaking and evaluation.