Multi-criteria development and incorporation into decision tools for health technology adoption

Synthesis of relevant information around non-core domains to support Multi-Criteria Decision Analysis (MCDA) for decision making

Countries fundamentally base macro and micro decision making in the field of health on economic considerations, the budgetary impact of technologies being a major criterion. Nevertheless, the value of the technology of interest and its dimensions are more complex if we seek to take decisions based on the value itself. The use of structured and explicit approaches that require the assessment of multiple criteria that reflect the dimensions of this value may significantly improve the quality of the decision making. Multi-criteria decision analysis (MCDA) is a complementary decision-making tool that is able to systematically incorporate dimensions or domains such as ethical, organisational, legal, environmental and social considerations, as well as costs and benefits of medical interventions, together with the distinct perspectives of the interested parties. The objective of this article is to propose the implementation of analysis of non-core domains, in reports of Health Technology Assessment (HTA) agencies/units. To assess the scientific evidence on MCDA techniques a systematic review was conducted using structured searches in biomedical databases and websites of various HTA organisations. A consensus group was held using the nominal group technique and involving users of healthcare services, providers, managers and academics. Complementary, a survey was sent to HTA agencies to ascertain the degree of implementation of MCDA in their methods. 42 articles reporting the use of non-core criteria for the assessment of health technologies were included in the analysis. From these articles, a total of 216 non-core criteria were retrieved and categorised into domains by the researchers, and of these, 56 were classified as socioeconomic, 59 as organisational, 10 as legal, 8 as environmental and 47 as ethical, while 36 were considered to relate to other domains. The consensus group, based on the 216 non-core criteria obtained from the systematic review, proposed, and defined 26 criteria that participants considered necessary for decision making in healthcare. The consensus group did not consider that any of the domains should be given more weight than others or that any individual criteria should dominate. These approaches can serve as a framework of reference for a well-structured systematic discussion concerning the basis of individual criteria and the evidence supporting them.

Integrating organizational impacts into health technology assessment (HTA): an analysis of the content and use of existing evaluation frameworks

Objective

The French health technology assessment (HTA) agency initiated a research between 2018 and 2019 with the aim of determining whether other HTA organizations (agencies, bodies, institutes, and expert networks) and researchers had developed an evaluation framework of organizational impacts (OIs).

Methods

Three types of investigation were carried out: (i) an analysis of documents published by selected HTA organizations, (ii) a rapid review on the OI issues, (iii) a questionnaire survey to experts of the International Network of Agencies for Health Technology Assessment.

Results

The analyses highlight six key points: (i) there is no explicit conceptual definition of OIs; (ii) OIs are often not included in a specific dimension of the evaluation or in the same dimensions; (iii) three recurring categories emerge from the assessment of OIs: processes, structure, and culture; (iv) despite its limitations, the European Network for Health Technology Assessment framework (Core Model) is the most mature assessment model to date; (v) the question of the scope of OIs to be considered is unresolved (micro-meso-macro); and (vi) the delineation between OI assessment and economic assessment must be addressed.

Conclusions

Although the issue of considering OI in HTA has been raised for many years, it remains largely unresolved. Defining the concept of OI is a prerequisite for taking the next step toward an evaluation framework. As the question of the impact of innovation goes beyond the health sector, extensive research on how to define and take into account these OIs may be relevant.

Evaluation frameworks for digital nursing technologies: analysis, assessment, and guidance. An overview of the literature

Background

The evaluation of digital nursing technologies (DNT) plays a major role in gaining knowledge about certain aspects of a technology such as acceptance, effectiveness, or efficiency. Evaluation frameworks can help to classify the success or failure of a DNT or to further develop the technology. In general, there are many different evaluation frameworks in the literature that provide overviews of a wide variety of aspects, which makes this a highly diverse field and raises the question how to select a suitable framework. The aim of this article is to provide orientation in the field of comprehensive evaluation frameworks that can be applied to the field of DNT and to conduct a detailed analysis and assessment of these frameworks to guide field researchers.

Methods

This overview was conducted using a three-component search process to identify relevant frameworks. These components were (1) a systematized literature search in PubMed; (2) a narrative review and (3) expert consultations. Data relating to the frameworks’ evaluation areas, purpose, perspectives, and success definitions were extracted. Quality criteria were developed in an expert workshop and a strength and weakness assessment was carried out.

Results

Eighteen relevant comprehensive evaluation frameworks for DNT were identified. Nine overarching evaluation areas, seven categories of purposes, five evaluation perspectives and three categories of success definitions could be identified. Eleven quality criteria for the strengths and weaknesses of DNT-related evaluation frameworks were developed and the included frameworks were assessed against them.

Conclusion

Evaluators can use the concise information and quality criteria of this article as a starting point to select and apply appropriate DNT evaluation frameworks for their research projects or to assess the quality of an evaluation framework for DNT, as well as a basis for exploring the questions raised in this article. Future research could address gaps and weaknesses in existing evaluation frameworks, which could improve the quality of future DNT evaluations.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12912-021-00654-8.

Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art

BACKGROUND

Multi-criteria decision analysis (MCDA) concepts, models and tools have been used increasingly in health technology assessment (HTA), with several studies pointing out practical and theoretical issues related to its use. This study provides a critical review of published studies on MCDA in the context of HTA by assessing their methodological quality and summarising methodological challenges.

METHODS

A systematic review was conducted to identify studies discussing, developing or reviewing the use of MCDA in HTA using aggregation approaches. Studies were classified according to publication time and type, country of study, technology type and study type. The PROACTIVE-S approach was constructed and used to analyse methodological quality. Challenges and limitations reported in eligible studies were collected and summarised; this was followed by a critical discussion on research requirements to address the identified challenges.

RESULTS

129 journal articles were eligible for review, 56% of which were published in 2015-2017; 42% focused on pharmaceuticals; 36, 26 and 18% reported model applications, issues regarding MCDA implementation analyses, and proposing frameworks, respectively. Poor compliance with good methodological practice (< 25% complying studies) was found regarding behavioural analyses, discussion of model assumptions and uncertainties, modelling of value functions, and dealing with judgment inconsistencies. The five most reported challenges related to evidence and data synthesis; value system differences and participant selection issues; participant difficulties; methodological complexity and resource balance; and criteria and attributes modelling. A critical discussion on ways to address these challenges ensues.

DISCUSSION

Results highlight the need for advancement in robust methodologies, procedures and tools to improve methodological quality of MCDA in HTA studies. Research pathways include developing new model features, good practice guidelines, technologies to enable participation and behavioural research.

A Framework for Aiding the Translation of Scientific Evidence into Policy: The Experience of a Hospital-Based Technology Assessment Unit

OBJECTIVES

Very few practical frameworks exist to guide the formulation of recommendations at hospital-based health technology assessment (HTA) units. The objectives of our study were: (i) to identify decision criteria specific to the context of hospital-based health technologies and interventions, (ii) to estimate the extent to which the expert community agrees on the importance of the identified criteria, (iii) to incorporate the identified criteria into a decision-aid tool, and (iv) to illustrate the application of a prototype decision-aid tool.

METHODS

Relevant decision criteria were identified using existing frameworks for HTA recommendations, our past experience, a literature search, and feedback from a survey of diverse stakeholders.

RESULTS

Based on the survey results, twenty-three decision criteria were incorporated into the final framework. We defined an approach that eschewed a scoring system, but instead relied on a visual means for arriving at a final recommendation, by juxtaposing the importance rating for each criterion against the results of the health technology assessment. For a technology to be approved, a majority of criteria considered important should also have received favorable findings.

CONCLUSIONS

We created a simple and practical decision-aid tool that incorporates all decision criteria relevant to a hospital-based HTA unit. With its ease of use and accessibility, our tool renders the subjective decision-making process more structured and transparent.

Higher Sustainability and Lower Opportunistic Behaviour in Healthcare: A New Framework for Performing Hospital-Based Health Technology Assessment

Innovative health technology deployment represents the primary challenge within the sustainability of public health systems. On one hand, new technologies may potentially improve access to care and the quality of services. On the other hand, their rapid evolution and broad implications on existing procedures increase the risk to adopt technologies that are not value for money. As a consequence, Health Technology Assessment (HTA) is a critical process at each level of the National Health System. Focusing on the organisational level, this paper explores the current practices of Hospital-Based HTA (HB-HTA) in terms of management, control and behaviours of various actors involved. Among several tasks, decision-makers are appointed at managing the conflict of interest around health technology development, that could pave the way for corruption or other misleading behaviours. Accordingly, the purpose of the study is proposing a new strategic framework, named Health Technology Balanced Assessment (HTBA), to foster hospital-based health technology management aimed to align strategy and actions. The conceptual model is developed on three perspectives (clinical, economic and organisational) to make the actors involved in the assessment (clinicians, health professionals, hospital managers and patients) aware of the impact of new technology on the value chain. Besides supporting the decision-making process, such a tool represents support for the internal control system as a whole. By promoting structured evaluation, it increases transparency and accountability of public health organisations. Moreover, in the long run, the framework proposed will be useful to reach selected United Nations Sustainable Development Goals (UN SDGs) to enhance the quality of healthcare in the future.

'Real-world' health care priority setting using explicit decision criteria: a systematic review of the literature

Background

Health care decision making requires making resource allocation decisions among programs, services, and technologies that all compete for a finite resource pool. Methods of priority setting that use explicitly defined criteria can aid health care decision makers in arriving at funding decisions in a transparent and systematic way. The purpose of this paper is to review the published literature and examine the use of criteria-based methods in ‘real-world’ health care allocation decisions.

Methods

A systematic review of the published literature was conducted to find examples of ‘real-world’ priority setting exercises that used explicit criteria to guide decision-making.

Results

We found thirty-three examples in the peer-reviewed and grey literature, using a variety of methods and criteria. Program effectiveness, equity, affordability, cost-effectiveness, and the number of beneficiaries emerged as the most frequently-used decision criteria. The relative importance of criteria in the ‘real-world’ trials differed from the frequency in preference elicitation exercises. Neither the decision-making method used, nor the relative economic strength of the country in which the exercise took place, appeared to have a strong effect on the type of criteria chosen.

Conclusions

Health care decisions are made based on criteria related both to the health need of the population and the organizational context of the decision. Following issues related to effectiveness and affordability, ethical issues such as equity and accessibility are commonly identified as important criteria in health care resource allocation decisions.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-015-0814-3) contains supplementary material, which is available to authorized users.

Introduction of new technologies and decision making processes: a framework to adapt a Local Health Technology Decision Support Program for other local settings

PURPOSE

Introducing new health technologies, including medical devices, into a local setting in a safe, effective, and transparent manner is a complex process, involving many disciplines and players within an organization. Decision making should be systematic, consistent, and transparent. It should involve translating and integrating scientific evidence, such as health technology assessment (HTA) reports, with context-sensitive evidence to develop recommendations on whether and under what conditions a new technology will be introduced. However, the development of a program to support such decision making can require considerable time and resources. An alternative is to adapt a preexisting program to the new setting.

MATERIALS AND METHODS

We describe a framework for adapting the Local HTA Decision Support Program, originally developed by the Department of Surgery and Surgical Services (Calgary, AB, Canada), for use by other departments. The framework consists of six steps: 1) development of a program review and adaptation manual, 2) education and readiness assessment of interested departments, 3) evaluation of the program by individual departments, 4) joint evaluation via retreats, 5) synthesis of feedback and program revision, and 6) evaluation of the adaptation process.

RESULTS

Nine departments revised the Local HTA Decision Support Program and expressed strong satisfaction with the adaptation process. Key elements for success were identified.

CONCLUSION

Adaptation of a preexisting program may reduce duplication of effort, save resources, raise the health care providers' awareness of HTA, and foster constructive stakeholder engagement, which enhances the legitimacy of evidence-informed recommendations for introducing new health technologies. We encourage others to use this framework for program adaptation and to report their experiences.