Priority setting for health technology assessment. Theoretical considerations and practical approaches. Priority setting Subgroup of the EUR-ASSESS Project

Topic identification, selection, and prioritization for health technology assessment in selected countries: a mixed study design

BACKGROUND

There is limited evidence-informed guidance on TISP processes for countries where health technology assessment (HTA) is in a nascent phase. We aimed to explore the range of topic identification, selection and prioritization (TISP) processes and practices for HTA in selected countries and identify aspects relevant to emerging HTA systems.

METHODS

This mixed design study included a systematic literature review, an electronic survey, and individual interviews. We conducted a systematic literature review with criteria that were developed a priori to identify countries deemed to have a recently formalized HTA system. Based on the literature review, a twenty-three item online survey was shared with the identified countries, we completed follow-up interviews with ten participants who have experience with HTA. We analyzed documents, survey responses and interview transcripts thematically to identify lessons related to TISP processes and practices.

RESULTS

The literature review identified 29 nine candidate countries as having a "potential" recently formalized HTA system. Twenty-one survey responses were analyzed and supplemented with ten individual interviews. We found variation in countries' approaches to TISP - particularly between pharmaceutical and non-pharmaceutical interventions. Results indicate that TISP is heavily driven by policy makers, expert involvement, and to a lesser extent, relevant stakeholders. The use of horizon-scanning and early warning systems is uncommon. Interviewee participants provided further insight to the survey data, reporting that political awareness and an institutional framework were important to support TISP. TISP can be optimized by stronger national regulations and legislative structures, in addition to education and advocacy about HTA among politicians and decision-makers. In some settings regional networks have been useful, particularly in the development of TISP guidelines and methodologies. Additionally, the technical capacity to conduct TISP, and access to relevant local data were factors limiting TISP in national settings. Increased network collaboration and capacity building were reported as future needs.

CONCLUSIONS

This study provides current insights into a topic where there is limited published peer reviewed literature. TISP is an important first step of HTA, and topics should be selected and prioritized based on local need and relevance. The limited capacity for TISP in settings where HTA is emerging may be supported by local and international collaboration to increase capacity and knowledge. To succeed, both TISP and HTA need to be embedded within national health care priority setting and decision-making. More in-depth understanding of where countries are situtated in formalizing the TISP process may help others to overcome factors that facilitate or hinder progress.

[Population Health Intervention Research, Health Technology Assessment, Health Services Research and Intervention Implementation Research : Convergences and Singularities]

In public health, intervention is an object of research and evaluation which, over time, has given rise to numerous approaches. The first part of the article proposes to reposition intervention research in population health and intervention evaluation on a continuum. Although the former has a more cognitive objective and the latter a more pragmatic objective, they are not mutually exclusive. The distinction between these two practices is based on the predominance of the following characteristics: the objectives pursued, the scope of the investigations, the regulatory constraints, the financing obtained, the ethical approaches taken, and the deliverables established. The second part of the article offers a glance different fields and approaches within the continuum between these two poles: Health Technology Assessment, Health Services Research and Implementation Research. While all of them have the study of health interventions at their core, but each has developed through specialisation in one or the other type of intervention, in a particular scope or context, in certain evaluation questions, or in specific approaches. all as gateways to the study of public health intervention, these different approaches are by no means mutually exclusive.

The long process by which HTA became a paradigm: A longitudinal conceptual structure analysis

This study maps the conceptual structure of health technology assessment (HTA) research with the aim of contributing to a better understanding of this research stream. A bibliometric analysis of 1,198 HTA articles retrieved from the Web of Science database was conducted. The analysis of descriptive performance indicators identified the main traits of the scientific debate about the HTA in terms of publications, productive countries, and sources. A co-word analysis was performed by adopting social network analysis tools to map the conceptual structure of the dataset. The results highlight the growing academic interest in the research topic, especially in recent years. The results revealed that HTA is a widely known term and represents the reference paradigm for assessment in healthcare technologies. This paper contributes to the field by providing an examination of the current state of the art of HTA research and identifying possible future research directions. This study differs from existing studies because it involved a conceptual analysis of the scientific literature using macro keywords.

Evidence-Informed Deliberative Processes for Health Benefit Package Design - Part II: A Practical Guide

BACKGROUND

Countries around the world are using health technology assessment (HTA) for health benefit package design. Evidence-informed deliberative processes (EDPs) are a practical and stepwise approach to enhance legitimate health benefit package design based on deliberation between stakeholders to identify, reflect and learn about the meaning and importance of values, informed by evidence on these values. This paper reports on the development of practical guidance on EDPs, while the conceptual framework of EDPs is described in a companion paper.

METHODS

The first guide on EDPs (2019) is further developed based on academic knowledge exchange, surveying 27 HTA bodies and 66 experts around the globe, and the implementation of EDPs in several countries. We present the revised steps of EDPs and how selected HTA bodies (in Australia, Brazil, Canada, France, Germany, Scotland, Thailand and the United Kingdom) organize key issues of legitimacy in their processes. This is based on a review of literature via PubMed and HTA bodies' websites.

RESULTS

HTA bodies around the globe vary considerable in how they address legitimacy (stakeholder involvement ideally through participation with deliberation; evidence-informed evaluation; transparency; and appeal) in their processes. While there is increased attention for improving legitimacy in decision-making processes, we found that the selected HTA bodies are still lacking or just starting to develop activities in this area. We provide recommendations on how HTA bodies can improve on this.

CONCLUSION

The design and implementation of EDPs is in its infancy. We call for a systematic analysis of experiences of a variety of countries, from which general principles on EDPs might subsequently be inferred.

Topic selection process in health technology assessment agencies around the world: a systematic review

Objective

The purpose of this study was to systematically review the process for topic selection by health technology assessment (HTA) agencies around the world to provide the knowledge base for the improvement of topic selection frameworks in HTA agencies.

Methods

A systematic search was conducted in PubMed and EMBASE to identify papers up to February 2019. Gray literature was identified by screening the Web sites of HTA agencies on the nonprofit member list of the International Network of Agencies for Health Technology Assessment (INAHTA). Data were extracted for each HTA agency and synthesized, with issues including general contextual information about each agency and the process of topic selection.

Results

Out of forty-nine nonprofit members of INAHTA, a total of seventeen HTA agencies with a framework for topic selection were identified from twenty-two included papers/documents. Multiple criteria were used for topic selection in all frameworks and agencies undertook multiple steps, which could include the specification of criteria for topic selection, identification of topics, short listing of potential topics, scoping of potential topics, scoring and ranking of potential topics, and deliberation and decision on final topics for HTA. Shortcomings were found in relation to methods of scoring and ranking as well as lack of monitoring and the evaluation of the process.

Conclusions

Our study provides insights into the current practice of topic selection in HTA agencies. Multiple criteria decision analysis methodology appears highly relevant to these processes. A consensus approach for the development of methods of topic selection would be valuable for the HTA community.

Smart health via mHealth? Potentials of mobile health apps for improving prevention and adherence of breast cancer patients

Objective

Today there are several health and medical apps (mHealth) in app stores. Germany is the world's first country that introduced apps paid by the regular health insurance service. Even though breast cancer is the most common cancer in women, mHealth for breast cancer has been largely unexplored.

Methods

A total of 33 apps from two major mobile application marketplaces (Google Play Store/Android; App Store/iOS) have been selected for analysis.

Results

The app analysis shows that there are currently only 10 mHealth apps in German, which are specifically dedicated to breast cancer patients. The features of these apps fall into two categories: improvement of health literacy and indirect intervention. These apps can be used for all phases of the patient journey starting with the diagnosis.

Conclusions

mHealth apps have the potential to support the adherence of breast cancer patients. In order to exploit this future potential, the app quality, as well as the information about the available apps, must be urgently improved. Currently, it is very difficult both for laypersons and for doctors/other therapists to identify high-quality apps. Guidance from independent or governmental institutions would be helpful to further the digitalization in health care.

A collaborative pilot on current awareness alerts for disinvestment and horizon scanning

OBJECTIVE

In 2019, members of the Health Technology Assessment international (HTAi) Interest Group for Disinvestment and Early Awareness (DEA-IG) and the HTAi Interest Group for Information Retrieval (IR-IG) agreed to produce quarterly current awareness alerts for members of the DEA-IG. The purpose was to pilot a predefined strategy for sharing new publications on methods and topical issues in this area.

METHODS

Literature search strategies for PubMed and Google were developed. Retrieved citations were posted on the DEA-IG Web site. Members of the DEA-IG received an email notification when new alerts were available. An informal survey of the DEA-IG members was used to provide feedback after the pilot.

RESULTS

Six alerts were issued during the pilot (June 2019-September 2020) with a total of 170 citations. The bulk of the information were 124 PubMed indexed citations, and of these, 96 were retrieved by the PubMed search strategies. Google searches were not found to be useful, but ongoing horizon scanning work at the Canadian Agency for Drugs and Technologies in Health (CADTH) provided additional information. Based on retrospective sorting, we considered thirty-five PubMed citations to be highly relevant for health technology assessment (HTA). The response rate to the survey was limited (seventeen respondents), but most respondents found the alerts useful for their work.

CONCLUSIONS

The results of this pilot project can be used to revise search strategies and information sources, improve the relevance of the alerts, and plan for expanded dissemination strategies.

[Mecanismos de identificación y priorización de las tecnologías sanitarias a ser evaluadas por las Agencias de ETESA para decisiones de cobertura en Latino America]

OBJETIVO

Ningún sistema de salud cuenta con los recursos necesarios para evaluar todas las tecnologías. Contar con un proceso claro para priorizar qué tecnologías serán evaluadas por las agencias de evaluación de tecnologías sanitarias (ETESA) constituye un principio de buena práctica reconocido a nivel internacional. El objetivo del Foro de Políticas en Latino América (LatamPF) 2020 de Health Technology Assessment International fue explorar cómo puede mejorarse la forma en que las agencias de ETESA de Latino América identifican y priorizan las tecnologías a ser evaluadas.

MÉTODOS

Este manuscrito está basado en un documento base, una encuesta, y en el trabajo deliberativo realizado por los miembros (cuarenta y seis participantes, once países) que participaron del LatamPF, a través de la metodología design-thinking.

RESULTADOS

Los participantes coincidieron en que la falta de mecanismos claros de priorización trae como consecuencia una falta de legitimidad de las decisiones y procesos de ETESA, que son percibidos como poco transparentes y demasiado expuestos a presiones políticas o de grupos de interés. También se identificaron barreras y acciones para mejorar los mecanismos de priorización de ETESA en América Latina. Los criterios identificados como más importantes para ser tenidos en cuenta por las agencias de ETESA de la región al momento de priorizar una tecnología para ser evaluada fueron la carga de enfermedad, el potencial beneficio clínico, la alineación con prioridades de salud nacionales, el potencial impacto en la equidad, ausencia de otras alternativas para los pacientes, y el potencial impacto económico.

CONCLUSIONES

Los participantes del Foro coincidieron en que el establecimiento de procesos transparentes de priorización es un elemento clave para todos los sistemas de salud. Las mejoras en este proceso fortalecerán la ETESA en Latino América y darán mayor legitimidad a sus decisiones.

Identification and selection of health technologies for assessment by agencies in support of reimbursement decisions in Latin America

OBJECTIVE

There is no health system that has the resources to evaluate all technologies. The presence of a clear process to prioritize health technologies for assessment by health technology assessment (HTA) agencies is a good practice principle recognized at the international level. The objective of Health Technology Assessment International's 2020 Latin American Policy Forum (LatamPF) was to explore how to improve the way HTA agencies in Latin America identify and prioritize technologies for assessment.

METHODS

This paper is based on a background document, a survey, and the deliberations of the members of the LatamPF (forty-six participants from eleven countries) using a design thinking methodology.

RESULTS

Participants agreed that a lack of clear prioritization mechanisms results in HTA processes and decisions that are perceived to be of low transparency and overly exposed to political or interest group pressures. The LatamPF identified barriers and recommended actions to improve HTA prioritization mechanisms in Latin America. The criteria identified as the most important to be taken into consideration by HTA agencies in the region when prioritizing a technology for assessment were: the burden of illness, the potential clinical benefit, the alignment with national health priorities, the potential impact on equity, a lack of treatment alternatives for patients, and the potential economic impact.

CONCLUSIONS

Forum participants agreed that the establishment of transparent prioritization processes is a key element for all health systems. Improvements in these processes will strengthen HTA and provide greater legitimacy to decision making.

The new definition of health technology assessment: A milestone in international collaboration

Background

An international joint task group co-led by the International Network of Agencies for Health Technology Assessment (INAHTA) and Health Technology Assessment International (HTAi) has developed a new and internationally accepted definition of HTA.

Methods

The task group, consisting of representatives of leading HTA networks, societies and global organizations, developed guiding principles for the process and followed an established consultation plan with the broader HTA community to develop the definition.

Results

The consensus achieved by the international joint task group brings the collective weight of the participating networks, societies, and organizations behind the new definition.

Conclusion

The new definition of HTA is an historic achievement and it is offered to the current and emerging HTA world as a cornerstone reference for today and into the future.

'The problem is small enough, the problem is big enough': a qualitative study of health technology assessment and public policy on drug funding decisions for children

BACKGROUND

Public policy approaches to funding paediatric medicines in developed public health systems remain understudied. Current approaches to HTA present a variety of conceptual, methodological and practical problems in the context of child health. This study explores the technical and sociopolitical determinants of public funding decisions on paediatric drugs, through the analysis of interviews with stakeholders involved in or impacted by HTA for child health technologies at the provincial and national levels in Canada.

METHODS

We undertook in-depth interviews with a purposive sample (n = 22) of stakeholders involved with or affected by drug funding decisions for children at the provincial (Ontario) and national levels in Canada. Grounded theory methods were employed to guide data collection and analysis. Theory on 'technology-as-policy' and the sociopolitics of health technologies served as sensitizing concepts for inductive data coding and analysis. Emergent themes informed the development of conceptual and practical insights on social values and system dynamics related to child HTA, of relevance to public policymaking on the coverage of health technologies for children in Canada.

RESULTS

Participant reflection on the normative and systems dimensions of drug funding for children formed two broad categories: HTA paradigms and sociopolitical context. Our analysis revealed notable differences of context and substance related to child health technology production, evaluation and use. These differences spanned the major phases of HTA (from assembly to assessment to integration) and the surrounding sociopolitical milieu (from markets to governance to politics). Careful analysis of these differences sets in relief a number of substantive and procedural shortcomings of current HTA paradigms in respect of child health. Our findings suggest a need to rethink how HTA is structured and operationalized for child health technologies.

CONCLUSIONS

Current approaches to health technology assessment are not well calibrated to the realities of child health and illness. Our study presents a nuanced and contextually grounded analysis of concepts instrumental to drug funding decisions for children. The insights generated are directly applicable to the Canadian and Ontario contexts, but also yield fundamental knowledge about HTA for children that are germane to drug policy in other health systems.

Systematic review for the development of a pharmaceutical and medical products prioritization framework

Objective

To identify and analyze the criteria, approaches, and conceptual frameworks, used for national/international priority setting.

Data sources

We performed a search of the main biomedical databases (Medline/PubMed, Embase, Centre for Reviews and Dissemination, and Cochrane), and we reviewed assessment agency websites, among other sources.

Study design

An systematic review of the literature was carried out.

Data collection

Eligibility criteria for inclusion were based on set of predefined criteria. Systematic reviews and/or qualitative studies (interviews, surveys, expert consensus, etc) that aimed to identify prioritization criteria or develop general operational frameworks for the selection of health priorities were included. A critical analysis is made of all the aspects that may be useful for any public body that intends to establish priorities in health.

Principal findings

We found that there are no standardized criteria for priority setting, although common trends have been identified regarding key elements. Eight key domains were identified: 1) need for intervention; 2) health outcomes; 3) type of benefit of the intervention; 4) economic consequences; 5) existing knowledge on the intervention/quality and uncertainties of the regarding evidence; 6) implementation and complexity of the intervention/feasibility; 7) justice and ethics; and 8) overall context.

Conclusions

Our review provides a thorough analysis of the relevant issues and offers key recommendations regarding considerations for developing a national prioritization framework. Findings are envisioned to be useful for different public organizations that are aiming to establish healthcare priorities.

Electronic supplementary material

The online version of this article (10.1186/s40545-019-0181-2) contains supplementary material, which is available to authorized users.

Hospital staff perspectives towards health technology assessment: data from a multidisciplinary survey

BACKGROUND

Technology adoption in hospitals is usually based on cost-effectiveness analysis, feasibility and potential success. Different countries have embraced a range of principles to accomplish an effective comprehensive process of health technology assessment (HTA). The aim of the study was to analyse the viewpoints and relative weight of technology-oriented hospital staff members toward the clinical, social, technological and economic aspects of HTA.

METHODS

Using a structured questionnaire, a survey was conducted among different professionals in an 850-bed hospital.

RESULTS

We revealed a range of viewpoints among hospital staff members according to their personal characteristics and professional standpoints. The clinical aspects of HTA were considered 'highly important' (HI) by most participants, especially the 'lifesaving' parameter. Similarly, the 'lack of effective alternative technology' was ranked HI by a high percentage of participants, independent of their profession. Economic aspects were ranked HI only by half of the participants, while social and technological aspects were ranked HI only by a relatively low percentage. Nurses added 'improving quality of life', 'increasing teamwork efficiency' and 'improving medical standards'. Allied health professionals focused on 'lack of effective alternative technologies' as a main argument for adoption of HTA, alongside increasing efficiency, budget savings and contribution to hospital reputation. Engineers emphasised the requirement of significant investment in infrastructure and increasing efficiency. Administrators ranked patient experience as HI. Interestingly, the high ranking of social aspects correlated with older responders, while junior staff ranked safety significantly higher.

CONCLUSIONS

A multi-perspective multidisciplinary approach would be beneficial for policy-makers at hospitals and even on a national scale in Israel.

Identifying the Need for Good Practices in Health Technology Assessment: Summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA

The systematic use of evidence to inform healthcare decisions, particularly health technology assessment (HTA), has gained increased recognition. HTA has become a standard policy tool for informing decision makers who must manage the entry and use of pharmaceuticals, medical devices, and other technologies (including complex interventions) within health systems, for example, through reimbursement and pricing. Despite increasing attention to HTA activities, there has been no attempt to comprehensively synthesize good practices or emerging good practices to support population-based decision-making in recent years. After the identification of some good practices through the release of the ISPOR Guidelines Index in 2013, the ISPOR HTA Council identified a need to more thoroughly review existing guidance. The purpose of this effort was to create a basis for capacity building, education, and improved consistency in approaches to HTA-informed decision-making. Our findings suggest that although many good practices have been developed in areas of assessment and some other key aspects of defining HTA processes, there are also many areas where good practices are lacking. This includes good practices in defining the organizational aspects of HTA, the use of deliberative processes, and measuring the impact of HTA. The extent to which these good practices are used and applied by HTA bodies is beyond the scope of this report, but may be of interest to future researchers.

Testing a new taxonomic model for the assessment of medical devices: Is it plausible and applicable? Insights from HTA reports and interviews with HTA institutions in Europe

OBJECTIVE

Medical devices (MDs) encompass a broad and heterogeneous range of technologies. While practices vary considerably across countries, MDs often find application in patient care with little or no evaluation of their effectiveness and safety following market approval. A recently proposed taxonomy of MDs considered devices from the viewpoint of Health Technology Assessment (HTA). The aim of the work presented here was to test its plausibility and applicability by considering real-world HTA practices.

METHODS

HTA reports on MDs from European institutions were collected in a systematic manner and the evaluated devices and/or related procedures were matched to a position on the taxonomy. Following this, representatives from 16 European HTA institutions were asked about the usefulness of the taxonomy in semi-structured interviews.

RESULTS

1237 HTA reports (2004-2015) from 33 European institutions were included in the sample. The majority of reports was on technologies from the taxonomic positions initially estimated as having high relevance. Most of the experts interviewed stated that they found the taxonomy useful, particularly regarding its potential to aid in selecting technologies for assessment and to highlight potential methodological particularities per taxonomic position.

CONCLUSIONS

Overall, the distribution of identified reports on the matrix confirmed that the initial estimation of the relevance and necessity of HTA provided in the taxonomic model is plausible. In addition, interviews with representatives of European HTA institutions showed that the taxonomy could be useful.

When health technology assessment is confidential and experts have no power: the case of Hungary

Health technology assessment (HTA) is not simply a mechanistic technical exercise as it takes place within a specific institutional context. Yet, we know little about how this context influences the operation of HTA and its ability to influence policy and practice. We seek to demonstrate the importance of considering institutional context, using a case study of Hungary, a country that has pioneered HTA in Central and Eastern Europe. We conducted 26 in-depth, semi-structured interviews with public- and private-sector stakeholders. We found that while the HTA Department, the Hungarian HTA organisation, fulfilled its formal role envisaged in the legislation, its potential for supporting evidence-based decision-making was not fully realised given the low levels of transparency and stakeholder engagement. Further, the Department's practical influence throughout the reimbursement process was perceived as being constrained by the payer and policy-makers, as well as its own limited organisational capacity. There was also scepticism as to whether the current operational form of the HTA process delivered 'good value for money'. Nevertheless, it still had a positive impact on the development of a broader institutional HTA infrastructure in Hungary. Our findings highlight the importance of considering institutional context in analysing the HTA function within health systems.

USING THE HEALTH TECHNOLOGY ASSESSMENT TOOLBOX TO FACILITATE PROCUREMENT: THE CASE OF SMART PUMPS IN A CANADIAN HOSPITAL

OBJECTIVES

The aim of this study was to present the experience of a Canadian hospital-based health technology assessment (HTA) unit that performed the traditional functions of the HTA process along with many other activities to facilitate the choice of smart pumps.

METHODS

A rapid literature review was initiated, but little evidence was found. Moreover, the evidence provided was too far from our hospital context. To help our decision makers, we offered them a list of various services based on the skills of our HTA unit staff.

RESULTS

To involve our HTA unit in the choice of the new smart pumps led to a strong collaboration between hospital services. After a rapid review on smart pumps, we proceeded to establish the clinical needs, followed by an evaluation of technical features. To ascertain clinical needs, we participated in the establishment of a conformity list for the tender, a failure and mode-effect analysis, an audit on the use of actual smart pumps, and simulation exercises with nurses and doctors to evaluate the ease of use and ergonomics. With regard to technical tests, these were mainly conducted to identify potential dysfunction and to assess the efficiency of the pump. This experience with smart pumps was useful for evidence-based procurement and led to the formulation of a nine-step process to guide future work.

CONCLUSIONS

HTA units and agencies are faced with rapid development of new technologies that may not be supported by sufficient amount of pertinent published evidence. Under these circumstances, approaches other than evidence-based selection might provide useful information. Because these activities may be different from those related to classic HTA, this widens the scope of what can be done in HTA to support decision making.

HEALTH TECHNOLOGY ASSESSMENT: THE SCIENTIFIC CAREER OF A POLICY CONCEPT

Objectives: The aim of this work was to provide a comprehensive overview of the evolution of the health technology assessment (HTA) concept in the scientific literature through a scientometric approach.

Methods: A literature search was conducted, by selecting publications, as well as news from the media, containing “health technology assessment” in their title, abstracts, or keywords. We then undertook a bibliometric and network analysis on the corpus of 2,865 publications thus obtained.

Results: Since a first publication in 1978, interest in HTA remained marginal until a turning point in the late 1980s, when growth of the number of publications took off alongside the creation of the U.K.’s NICE agency. Since then, publications have spread across several journals. The ranking of the organizations that publish such articles does not reflect any hegemonic position. However, HTA-related scientific production is strongly concentrated in Commonwealth and Nordic countries. Despite its transnational aspects, research on HTA has been framed within a small number of scientific networks and by a few opinion leaders.

Conclusions: The “career” of the HTA concept may be seen as a scientific-knowledge based institutionalization of a public policy. To succeed in a country, HTA first needs scientific prerequisites, such as an organized scientific community working on the health sector and health services. Then, it appears that the recognition of this research by decision makers plays a key role in the development of the field.

Hospital-based expert model for health technology procurement planning in hospitals

Although in the last years technology innovation in healthcare brought big improvements in care level and patient quality of life, hospital complexity and management cost became higher. For this reason, necessity of planning for medical equipment procurement within hospitals is getting more and more important in order to sustainable provide appropriate technology for both routine activity and innovative procedures. In order to support hospital decision makers for technology procurement planning, an expert model was designed as reported in the following paper. It combines the most widely used approaches for technology evaluation by taking into consideration Health Technology Assessment (HTA) and Medical Equipment Replacement Model (MERM). The designing phases include a first definition of prioritization algorithms, then the weighting process through experts' interviews and a final step for the model validation that included both statistical testing and comparison with real decisions. In conclusion, the designed model was able to provide a semi-automated tool that through the use of multidisciplinary information is able to prioritize different requests of technology acquisition in hospitals. Validation outcomes improved the model accuracy and created different "user profiles" according to the specific needs of decision makers.

[The introduction of technologies in health care systems: from saying to doing]

INTRODUCTION

Health technology assessment (HTA) is a tool based on scientific evidence useful for making decisions about the introduction of technological advances in healthcare systems. However, there are multiple elements that influence the actual use of the results of HTA. These elements are related both to the characteristics of HTA and to the stakeholders involved -health professionals, politicians, and society. The aim of this article was to describe the main elements that influence the applicability of HTA in practice, as well as the use of HTA in clinical practice and health policy.

METHODS

A literature search was performed in PubMed. The references retrieved, as well as additional bibliography and gray literature dealing with different aspects of HTA practice identified from the references retrieved, were reviewed. The results are presented descriptively, with analysis of the studies grouped according to the various HTA-related topics.

RESULTS

Key elements that could be improved are standardization of the evaluation process, description of the organization's mission, prioritization of the technologies to be assessed, the clarity of the presentation of results, inclusion of values other than scientific evidence, such as social and cost-effectiveness values, and increased international coordination.

CONCLUSIONS

For many people, the rationale for HTA is its explicit role in decision making, despite its problems. The immediate future of HTA would seem to include making the assessment process explicit, and the inclusion of HTA in decision making. This entails prioritization of technologies, setting limits on resource allocation and interrelation with political processes.

Health technology assessment (HTA): a brief introduction of history and the current status in the field of cardiology under the economic crisis

In a time of economic recession health technology assessment is an established aid in decision making in many countries in order to identify cost-containment policy options. Moreover, as the volume, complexity, and cost of new medical technology increases, the need for evaluating benefits, risks and costs becomes increasingly important. In recent years there has been a proliferation of health technology assessment initiatives internationally, aimed in introducing rationality in the decision-making process, informing reimbursement, providing clinical guidance on the use of medical technologies across the world in an evidence-based decision-making environment and in pricing decisions.

A model for HTA priority setting: experience in Lithuania

OBJECTIVES

To promote the further development of HTA, this study aims to design a model for HTA priority setting, which would address national needs for a country with little experience in the field of HTA, and assess its feasibility for the health system.

METHODS

Literature search and review, as well as qualitative research have been used in the design and testing of the model for HTA priority setting. To test the model and the methodology, a three-round Delphi study was conducted in 2011 in the form of an electronic questionnaire, which was distributed to the panel of eleven national experts. The panel was composed of experts representing various fields of the health care sector: policy-makers, health care service professionals and academics, with diverse professional roles.

RESULTS

The designed model consists of four stages: (i) selection of experts for the panel, (ii) indication and selection of health policy topics, (iii) identification of health technologies, and (iv) priority setting. Three rounds of the Delphi study were performed to test the model and reach expert consensus on a list of health technologies for assessment, including pharmaceuticals, vaccines, medical devices, diagnostic methods, public health interventions, organizational systems, etc.

CONCLUSIONS

Based on the Delphi technique as a method for consensus building, the model for HTA priority setting was developed for Lithuania; however, it could also be used for other countries with little experience in the field of HTA.

Developing a prioritized list of innovative technologies: the Spanish experience

OBJECTIVES

Selecting technologies for formal assessment poses a great challenge to health technology assessment agencies. This study aims to contribute to the creation of a reference framework for the identification, filtering, and prioritization of new and emerging technologies which could be demanded in clinical practice within the next 1-2 years.

METHODS

Technologies were identified using a prevalidated systematic Medline strategy. They were classified by medical specialty and then sent to selected professionals belonging to the medical units or areas responsible for their application, until there was a minimum of three participants per health care setting. A self-administered questionnaire was drawn up and health professionals were asked to: (1) assess the degree of innovation of the technologies, and (11) score their foreseeable clinical impact on the basis of predefined prioritization criteria (n = 4). Intra-rater reliability was analyzed using the intraclass correlation coefficient (ICC).

RESULTS

The Medline search yielded 246 potentially relevant technologies. When analyzed by health care area or unit, sixty-eight were deemed to be high-impact innovative technologies (median score >6), with ICCs ranging from 0.03 to 0.83. The final list resulting from the aggregate analysis comprised fifty-one technologies.

CONCLUSIONS

This study constitutes an innovative contribution to horizon scanning, providing a systematic and reproducible basis for the identification and selection of relevant new and emerging technologies based on the views and values of health professionals involved in their use. In our opinion, the current proposal could be helpful and useful to many other organizations worldwide, serving to complement already existing strategies.

Analysis of a Small Group of Stakeholders Regarding Advancing Health Technology Assessment in India

OBJECTIVES

This study aimed to explore and understand the perspectives of a small group of stakeholders involved in health technology assessment (HTA) for evidence-informed decision making on policy in the Indian health system.

METHODS

Semi-structured interviews were conducted in April-June 2013 with policymakers, academicians, industry experts, and community representatives in India to understand their knowledge of, position regarding, and interest in HTA. A semi-structured questionnaire was designed on the basis of a World Health Organization framework for evidence-informed health care policymaking.

RESULTS

Seven key informant interviews were conducted to represent the various stakeholders. Although there is a good understanding of HTA among the national-level policymakers, academicians, civil society representatives, and industry experts, there is lack of knowledge about the subject among policymakers at the lower level. There is a positive perception about producing and using HTA for decision making among all the stakeholders interviewed. Nevertheless, at the national level, institutions prefer to treat the use of HTA evidence with caution because the capacity for adopting evidence-based tools in the health system is very limited.

CONCLUSIONS

This small-size stakeholder analysis suggests a mixed response in implementing HTA in India. There are, however, factors involved in implementing such tools that can be dealt with using various approaches. Finally, there is a positive view on the national level toward pushing the HTA agenda forward to improve the decision-making process in health care.

Development of archetypes for non-ranking classification and comparison of European National Health Technology Assessment systems

INTRODUCTION

European countries are increasingly utilising health technology assessment (HTA) to inform reimbursement decision-making. However, the current European HTA environment is very diverse, and projects are already underway to initiate a more efficient and aligned HTA practice within Europe. This study aims to identify a non-ranking method for classifying the diversity of European HTA agencies process and the organisational architecture of the national regulatory review to reimbursement systems.

METHOD/RESULTS

Using a previously developed mapping methodology, this research created process maps to describe national processes for regulatory review to reimbursement for 33 European jurisdictions. These process maps enabled the creation of 2 HTA taxonomic sets. The confluence of the two taxonomic sets was subsequently cross-referenced to identify 10 HTA archetype groups.

DISCUSSION

HTA is a young, rapidly evolving field and it can be argued that optimal practices for performing HTA are yet to emerge. Therefore, a non-ranking classification approach could objectively characterise and compare the diversity observed in the current European HTA environment.

Decision making by NICE: examining the influences of evidence, process and context

The National Institute for Health and Clinical Excellence (NICE) provides guidance to the National Health Service (NHS) in England and Wales on funding and use of new technologies. This study examined the impact of evidence, process and context factors on NICE decisions in 2004–2009. A data set of NICE decisions pertaining to pharmaceutical technologies was created, including 32 variables extracted from published information. A three-category outcome variable was used, defined as the decision to ‘recommend’, ‘restrict’ or ‘not recommend’ a technology. With multinomial logistic regression, the relative contribution of explanatory variables on NICE decisions was assessed. A total of 65 technology appraisals (118 technologies) were analysed. Of the technologies, 27% were recommended, 58% were restricted and 14% were not recommended by NICE for NHS funding. The multinomial model showed significant associations (p ⩽ 0.10) between NICE outcome and four variables: (i) demonstration of statistical superiority of the primary endpoint in clinical trials by the appraised technology; (ii) the incremental cost-effectiveness ratio (ICER); (iii) the number of pharmaceuticals appraised within the same appraisal; and (iv) the appraisal year. Results confirm the value of a comprehensive and multivariate approach to understanding NICE decision making. New factors affecting NICE decision making were identified, including the effect of clinical superiority, and the effect of process and socio-economic factors.

Setting priorities for comparative effectiveness research: from assessing public health benefits to being open with the public

Medicine's current evidence base is insufficient for many of the decisions made daily by clinicians, patients, purchasers, and policy makers. The Patient-Centered Outcomes Research Institute represents an effort by the US government to address this shortcoming by funding comparative effectiveness research. Prioritizing that research will be a critical first step. To better understand components of an optimal process, we reviewed the literature and conducted interviews regarding the prioritization efforts of leading public and private health organizations in the United States and abroad. From this review, we propose a framework for prioritization, pose and answer key questions, and make recommendations regarding application of that framework. We also recommend that during the priority-setting process, there should be transparent conversations among those who make decisions about the priorities and the public.

A checklist for health research priority setting: nine common themes of good practice

Health research priority setting processes assist researchers and policymakers in effectively targeting research that has the greatest potential public health benefit. Many different approaches to health research prioritization exist, but there is no agreement on what might constitute best practice. Moreover, because of the many different contexts for which priorities can be set, attempting to produce one best practice is in fact not appropriate, as the optimal approach varies per exercise. Therefore, following a literature review and an analysis of health research priority setting exercises that were organized or coordinated by the World Health Organization since 2005, we propose a checklist for health research priority setting that allows for informed choices on different approaches and outlines nine common themes of good practice. It is intended to provide generic assistance for planning health research prioritization processes. The checklist explains what needs to be clarified in order to establish the context for which priorities are set; it reviews available approaches to health research priority setting; it offers discussions on stakeholder participation and information gathering; it sets out options for use of criteria and different methods for deciding upon priorities; and it emphasizes the importance of well-planned implementation, evaluation and transparency.

Priority setting for health technology assessment at CADTH

OBJECTIVES

The aim of this study was to describe a current practical approach of priority setting of health technology assessment (HTA) research that involves multi-criteria decision analysis and a deliberative process.

METHODS

Criteria related to HTA prioritization were identified and grouped through a systematic review and consultation with a selection committee. Criteria were scored through a pair-wise comparison approach. Criteria were pruned based on the average weights obtained from consistent (consistency index < 0.2) responders and consensus. HTA proposals are ranked based on available information and a weighted criteria score. The rank, along with additional contextual information and discussion among committee members, is used to achieve consensus on HTA research priorities.

RESULTS

Six of eleven criteria represented > 75 percent of the weight behind committee member decisions to conduct an HTA. These criteria were disease burden, clinical impact, alternatives, budget impact, economic impact, and available evidence. Since May 2006, committees have considered 102 proposals at sixteen biannual in-person advisory committee meetings. These have selected twenty-nine research priorities for the HTA program.

CONCLUSIONS

The approach works well and was easy to implement. Feedback from committee members has been positive. This approach may assist HTA and other research agencies in better priority setting by informing the selection of the most important and policy-relevant topics in the presence of a wide variety of research proposals. This may in turn lead to efficiently allocating resources available for HTA research.

Prioritisation criteria for the selection of new diagnostic technologies for evaluation

BACKGROUND

Currently there is no framework for those involved in the identification, evaluation and prioritisation of new diagnostic technologies. Therefore we aimed to develop prioritisation criteria for the assessment of new diagnostic technologies, by gaining international consensus on not only which criteria should be used, but also their relative importance.

METHODS

A two-round Delphi process was used to generate consensus amongst an international panel of twenty-six experts on priority criteria for diagnostic health technology assessment. Participants represented a range of health care and related professions, including government, industry, health services and academia.

RESULTS

Based on the responses to the first questionnaire 18 criteria were placed into three categories: high, intermediate and moderate priority. For 16 of the 18 criteria, agreement with the categorisation of the criteria into the high, intermediate and moderate categories was high at > or = 70% (10 had agreement > or = 80%). A further questionnaire and panel discussion reduced the criteria to 16 and two categories; seven were classified as high priority and nine intermediate.

CONCLUSIONS

This study proposes an objective structure of prioritisation criteria to use when assessing new diagnostic technologies, based on an expert consensus process. The value of these criteria is that no one single component should be used as the decisive driver for prioritisation of new diagnostic technologies for adoption in healthcare settings. Future studies should be directed at establishing the value of these prioritisation criteria across a range of healthcare settings.

Setting priorities in clinical and health services research: Properties of an adapted and updated method

Objectives: The objectives of this study is to review the set of criteria of the Institute of Medicine (IOM) for priority-setting in research with addition of new criteria if necessary, and to develop and evaluate the reliability and validity of the final priority score.

Methods: Based on the evaluation of 199 research topics, forty-five experts identified additional criteria for priority-setting, rated their relevance, and ranked and weighted them in a three-round modified Delphi technique. A final priority score was developed and evaluated. Internal consistency, test–retest and inter-rater reliability were assessed. Correlation with experts’ overall qualitative topic ratings were assessed as an approximation to validity.

Results: All seven original IOM criteria were considered relevant and two new criteria were added (“potential for translation into practice”, and “need for knowledge”). Final ranks and relative weights differed from those of the original IOM criteria: “research impact on health outcomes” was considered the most important criterion (4.23), as opposed to “burden of disease” (3.92). Cronbach's alpha (0.75) and test–retest stability (interclass correlation coefficient = 0.66) for the final set of criteria were acceptable. The area under the receiver operating characteristic curve for overall assessment of priority was 0.66.

Conclusions: A reliable instrument for prioritizing topics in clinical and health services research has been developed. Further evaluation of its validity and impact on selecting research topics is required.

Methods, procedures, and contextual characteristics of health technology assessment and health policy decision making: comparison of health technology assessment agencies in Germany, United Kingdom, France, and Sweden

OBJECTIVES

The objectives of this study were (i) to develop a systematic framework for describing and comparing different features of health technology assessment (HTA) agencies, (ii) to identify and describe similarities and differences between the agencies, and (iii) to draw conclusions both for producers and users of HTA in research, policy, and practice.

METHODS

We performed a systematic literature search, added information from HTA agencies, and developed a conceptual framework comprising eight main domains: organization, scope, processes, methods, dissemination, decision, implementation, and impact. We grouped relevant items of these domains in an evidence table and chose five HTA agencies to test our framework: DAHTA@DIMDI, HAS, IQWiG, NICE, and SBU. Item and domain similarity was assessed using the percentage of identical characteristics in pairwise comparisons across agencies. RESULTS were interpreted across agencies by demonstrating similarities and differences.

RESULTS

Based on 306 included documents, we identified 90 characteristics of eight main domains appropriate for our framework. After applying the framework to the five agencies, we were able to show 40 percent similarities in "dissemination," 38 percent in "scope," 35 percent in "organization," 29 percent in "methods," 26 percent in "processes," 23 percent in "impact," 19 percent in "decision," and 17 percent in "implementation."

CONCLUSION

We found considerably more differences than similarities of HTA features across agencies and countries. Our framework and comparison provides insights and clarification into the need for harmonization. Our findings could serve as descriptive database facilitating communication between producers and users.

Commentary on the article “Key principles for the improved conduct of health technology assessments for resource allocation decisions”

The article “Key principles for the improved conduct of health technology assessments for resource allocation decisions” presents fifteen principles for health technology assessment. Many of these cannot be disputed, and application of the principles as stated would undoubtedly improve HTA as it is developing in the world at large. My question is, are these the most important principles? The document does not really try to answer this question.

Health technology assessment on reprocessing single-use catheters for cardiac electrophysiology: results of a three-years study

The study aims to define the technical, ethical, juridical and economic issues involved in the assessment of a reprocessing policy for single-use interventional cardiac devices (SUDs). The feasibility of reprocessing was evaluated for cardiac electrophysiology catheters by comparing the chemical, physical and functional properties of new and reprocessed devices. The issue of hygiene was addressed by developing microbiological tests for the quantification of bioburden, sterility and pyrogenic load. The results of more than 1500 tests, conducted on 531 catheters, suggested a precautionary number of regenerations of five cycles. The ethical aspects were reviewed and the European juridical framework was assessed, revealing a need for harmonization. Applying a specific economic model, potential savings were calculated for a representative cardiology department and estimated at national and European level. Potential savings of 41.2% and 32.9% were calculated for diagnostic and ablation catheters, respectively. Safe and effective reprocessing of SUDs could be pursued if quality control processes and certified procedures are met. A reprocessing policy in EP laboratory could lead to savings of about 27,250 euros per 100,000 population, but the economic benefits are strongly dependent on the maximum number of regenerations and the regeneration rate.

Setting priorities for research: a practical application of 'payback' and expected value of information

BACKGROUND

Setting priorities for research using economic in addition to scientific criteria can ensure that resources are spent efficiently and equitably.

OBJECTIVE

This study applies two priority setting methods 'payback' and expected value of information (EVI) to two research areas (osteoporosis and pressure ulcers) and where appropriate to four clinical trials: the Record Trial, the Vitamin D and Calcium Trial and the Hip Protector Trial (osteoporosis), and the Pressure Trial (wound care).

METHODS

Two decision-analytic models were developed. For 'payback', the PATHS model was used to estimate the expected net benefits of conducting the four clinical trials. An EVI framework was applied to estimate the cost-effectiveness of conducting further research in the two disease areas investigated.

RESULTS

The application of 'payback' suggests that the Record Trial and the Vitamin D and Calcium Trial would be cost-effective. The Hip Protector and the Pressure Ulcer Trial are cost-effective under certain assumptions concerning the likelihood of obtaining positive, negative or inconclusive results. The EVI method suggests that research would be potentially cost-effective in these areas in the populations considered.

CONCLUSION

EVI provides strategic information for setting priorities for research between disease areas and study populations. 'Payback' provides information on the cost-effectiveness of specific research designs. However, further work in this area, particularly concerning the issue of implementation of research, is required.

Priority setting for health technology assessments: a systematic review of current practical approaches

OBJECTIVES

This study sought to identify and compare various practical and current approaches of health technology assessment (HTA) priority setting.

METHODS

A literature search was performed across PubMed, MEDLINE, EMBASE, BIOSIS, and Cochrane. Given an earlier review conducted by European agencies (EUR-ASSESS project), the search was limited to literature indexed from 1996 onward. We also searched Web sites of HTA agencies as well as HTAi and ISTAHC conference abstracts. Agency representatives were contacted for information about their priority-setting processes. Reports on practical approaches selected through these sources were identified independently by two reviewers.

RESULTS

A total of twelve current priority-setting frameworks from eleven agencies were identified. Ten countries were represented: Canada, Denmark, England, Hungary, Israel, Scotland, Spain, Sweden, The Netherlands, and United States. Fifty-nine unique HTA priority-setting criteria were divided into eleven categories (alternatives; budget impact; clinical impact; controversial nature of proposed technology; disease burden; economic impact; ethical, legal, or psychosocial implications; evidence; interest; timeliness of review; variation in rates of use). Differences across HTA agencies were found regarding procedures for categorizing, scoring, and weighing of policy criteria.

CONCLUSIONS

Variability exists in the methods for priority setting of health technology assessment across HTA agencies. Quantitative rating methods and consideration of cost benefit for priority setting were seldom used. These study results will assist HTA agencies that are re-visiting or developing their prioritization methods.

Patient-based health technology assessment: a vision of the future

BACKGROUND

In conjunction with other important movements in contemporary medicine, including evidence-based medicine (EBM), health technology assessment (HTA) has promoted a culture of critical evaluation. Despite this impact, institutional and methodological challenges are associated with HTA. For example, only in recent years has HTA attempted an open dialogue with patients; however, this is normally done by giving them a "seat" at the HTA decision-making table, rather than by more scientific means.

OBJECTIVES

The aim of this study was to develop a working definition of patient-based HTA, to identify the current barriers to adopting a patient-based model, and to formulate a vision of how a patient-based HTA could be used to promote patient empowerment and patient-centered care.

RESULTS

In the ideal setting, a patient-based HTA would promote patient knowledge by providing access to information and promoting an informed dialogue between patients and their healthcare professionals. To implement a patient-based HTA, the focus must turn to the patient's issues and incorporate each patient's unique perspective and preferences. Processes must change to increase patient participation in all levels of HTA and aim to promote empowered patients who can make informed decisions.

CONCLUSIONS

Present-day HTA is broad and has numerous stakeholders, with none so important as the patient. By asking patient-oriented questions in HTA and better involving patients throughout the entire process, we can easily promote patient empowerment, and as such make patients more capable to play a more active role in healthcare decision making.

Do we have the data to take informed decisions in Healthcare Technology Management (HTM) related issues? A Conceptual HTM-IS Framework

This paper considers the data and information needs for a broad range of environments, focusing on information-poor environments that may be well resourced in other respects. A generic framework for the establishment of a comprehensive healthcare technology management information system (HTMIS) is suggested.

Selection of new health technologies for assessment aimed at informing decision making: A survey among horizon scanning systems

Objectives:Uncertainty is pervasive in decision making on new health technologies; therefore, some countries have put systems in place to support decision makers with timely information. An important, but as yet undocumented, determinant of the potential value for decision making of these so-called horizon scanning systems (HSSs) is how the most significant health technologies are selected.

Methods:All thirteen member organizations of EuroScan, a collaborative network for HSSs, were surveyed and interviewed on how they prioritize technologies for assessment.

Results:The majority of HSSs directly serves a customer. Some customers actively request early assessments of new health technologies, thereby diminishing the need for priority setting for the HSSs. All systems express a concern to miss an important technology and/or to select an unimportant technology. Almost all HSSs use explicit selection criteria, but these criteria hardly ever are operationalized. The number of criteria used varies, but costs and health benefit of the technology are always taken into account. The process of reaching a final decision is implicit, undocumented in all but one system, and is based on agreement by consensus.

Conclusions:The process of making the final decision on which technologies to assess can be improved by applying existing criteria more consistently and transparently. Current practice does not safeguard against missing an important technology. This finding is probably most important to act upon for systems with customers that do not actively request assessment of specific technologies.

Priority setting for horizon scanning of new health technologies in Denmark: views of health care stakeholders and health economists

In the context of the establishment of a Danish Horizon Scanning System (HSS) the views of health care stakeholders and health economists were solicited by means of postal surveys on the need for adaptation of a priority setting instrument for health technology assessment (HTA). The aim was to investigate if the instrument needed adaptation for priority setting in the context of a Danish HSS and, if so, how the instrument should be changed. A literature study served to enhance interpretation of the findings of the surveys and to formulate changes in the instrument that synthesize or bridge any differing views between the two groups. The results show that the instrument should apply a health care perspective, and that technologies should be prioritised on the basis of the criteria: marginal benefits, marginal costs, budget impact, impact on access to care, and additional criteria with an impact on health policy, such as the educational needs and organisational changes associated with the new technology. The proposed changes are regarded as an intermediate step in the process of producing a fully adapted instrument that can serve as a formal support for priority setting of new health technologies for assessment in the Danish HSS.

Setting priorities for research

Setting priorities for research should be conducted in order to make the most efficient use of scarce resources. Yet the uptake in practice of such methods by researchers and commissioners of research alike has been slow, in part because the methodologies available to do so have not been widely disseminated. This paper argues that an appropriate priority-setting methodology should meet the objectives of the health system, that is to provide the most health benefits to the population that it serves within the budget constraint and while respecting equity considerations. A condition for these criteria to be met is to construct and operationalise an appropriate definition of the value of research. Five different ways that have been used in practice to value research and set priorities were reviewed. Shortcomings in the ways research is valued make it unlikely that the application of subjective methods, burden of disease methods, and clinical variations and payback methods meet the objectives of the health system. Using the fifth method, value of information, priority-setting can meet the objectives of the health system because it expresses the value of research using the same overall cost-effectiveness framework that is employed for decisions on service provision. However, this method still requires further work to evaluate how research outcomes can then be communicated to clinical practitioners and how practitioners can be encouraged to implement them.

Cross-national comparison of technology assessment processes

Objectives:To compare methods and results among four health technology assessment organizations in different countries.

Methods:All assessment reports published between 1999 and 2001 by VATAP (United States), NICE (United Kingdom), CCOHTA (Canada), and AETS (Spain), were reviewed. Detailed information about the organization, the technology assessed, the methods used, and the recommendations made were collected. A descriptive analysis of the variables, as well as comparisons of means and proportions, was performed.

Results:Sixty-one reports assessing seventy-six technologies were published: nine (11.8 percent) by VATAP, thirty-nine (51.3 percent) by NICE, twenty (26.3 percent) by CCOHTA, and eight (10.5 percent) by AETS. A total of 64.5 percent of the technologies assessed were related to a high prevalence disease in the corresponding country. Most of the assessments addressed treatments (73.7 percent) and were mostly drugs (56.6 percent) and devices (23.7 percent). Most organizations used reviews of effectiveness and economic evaluations (64.5 percent), systematic reviews (21.1 percent), and original economic evaluations (36.7 percent). In 38.1 percent, the technology was recommended; the rest of the cases had no formal recommendations.

Conclusions:Critical issues for future technology assessment efforts are making assessment processes more consistent, transparent, and evidence-based; formalizing the inclusion of economic and ethical considerations; and making more explicit the prioritization process for selecting technologies for assessment and reassessment.