Why do health technology assessment drug reimbursement recommendations differ between countries? A parallel convergent mixed methods study

Using automated text classification to explore uncertainty in NICE appraisals for drugs for rare diseases

OBJECTIVE

This study examined the application, feasibility, and validity of supervised learning models for text classification in appraisals for rare disease treatments (RDTs) in relation to uncertainty, and analyzed differences between appraisals based on the classification results.

METHODS

We analyzed appraisals for RDTs (n = 94) published by the National Institute for Health and Care Excellence (NICE) between January 2011 and May 2023. We used Naïve Bayes, Lasso, and Support Vector Machine models in a binary text classification task (classifying paragraphs as either referencing uncertainty in the evidence base or not). To illustrate the results, we tested hypotheses in relation to the appraisal guidance, advanced therapy medicinal product (ATMP) status, disease area, and age group.

RESULTS

The best performing (Lasso) model achieved 83.6 percent classification accuracy (sensitivity = 74.4 percent, specificity = 92.6 percent). Paragraphs classified as referencing uncertainty were significantly more likely to arise in highly specialized technology (HST) appraisals compared to appraisals from the technology appraisal (TA) guidance (adjusted odds ratio = 1.44, 95 percent CI 1.09, 1.90, p = 0.004). There was no significant association between paragraphs classified as referencing uncertainty and appraisals for ATMPs, non-oncology RDTs, and RDTs indicated for children only or adults and children. These results were robust to the threshold value used for classifying paragraphs but were sensitive to the choice of classification model.

CONCLUSION

Using supervised learning models for text classification in NICE appraisals for RDTs is feasible, but the results of downstream analyses may be sensitive to the choice of classification model.

Appraisal of Novel Oncological Therapies by the Scottish Medicines Consortium and the National Institute for Health and Care Excellence: A Comparative Study of Six Years of Data

Background and aims Pharmacoeconomic assessment of novel oncological therapies is an increasingly important factor in determining patient access to therapies. Organisations such as the National Institute for Health and Care Excellence (NICE) in England and the Scottish Medicines Consortium (SMC) in Scotland assess medications for their cost-effectiveness through health technology assessments (HTA) and provide guidance on whether the public health service should fund a therapy. We assessed six years of data to determine if there were any differences in timescales and decisions between NICE and SMC for new oncological therapies. Methods and results Time (days) from marketing authorisation (MA) to publication of final HTA guidance was calculated for single technology appraisals published by NICE and SMC between January 1, 2017, and December 31, 2022, for oncological therapies. We assessed 161 HTAs by NICE and 148 HTAs by SMC published in the study period. The median time from MA to publication of HTA guidance was 291 days (IQR 222-406) for SMC and 257 days (IQR 167-448) for NICE (p=0.054). For solid organ cancer therapies, NICE was significantly faster in publishing guidance, with a median of 231.5 days (IQR 148-392.25), compared to SMC, which took 273 days (IQR 202-378) (p=0.039). Overall recommendation of technologies was similar between the SMC and NICE (90.5% and 89.4%, respectively), with discordance in a minority of cases (12.6%). Conclusions Recommendation rates for single HTAs are similar between NICE and SMC for oncological therapies with discordance in a minority of cases. The time from MA to publication of HTA guidance was similar overall, but NICE was faster in publishing HTA guidance for solid organ cancer indications. Differences in methodology and process between the two organisations, in particular the presence of the Cancer Drugs Fund in England, may explain this difference in publication times.

Health Technology Assessment-Informed Decision Making by the Federal Joint Committee/Institute for Quality and Efficiency in Health Care in Germany and the National Institute for Health and Care Excellence in England: The Role of Budget Impact

OBJECTIVES

This study aimed to test (official) evaluation criteria including the potential role of budget impact (BI) on health technology assessment (HTA) outcomes published by the Federal Joint Committee (Gemeinsamer Bundesausschuss [GBA]) and the Institute for Quality and Efficiency in Health Care (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen [IQWiG]) in Germany as well as the National Institute for Health and Care Excellence (NICE) in England.

METHODS

Data were extracted from all publicly available GBA decisions and IQWiG assessments as well as NICE single technology appraisals between January 2011 and June 2018, and information with regard to evaluation criteria used by these agencies was collected. Data were analyzed using logistic regression to estimate the effect of the BI on the HTA outcomes while controlling for criteria used by GBA/IQWiG and NICE.

RESULTS

NICE recommendations are largely driven by the incremental cost-effectiveness ratio and, if applicable, by end-of-life criteria (P < .01). While IQWiG assessments are significantly affected by the availability of randomized controlled trials and patient-relevant endpoints (P < .01), GBA appraisals primarily focus on endpoints (P < .01). The BI correlated with NICE single technology appraisals (inverted-U relationship, P < .1) and IQWiG recommendations (increasing linear relationship, P < .05), but not with GBA decisions (P > .1). Nevertheless, given that IQWiG assessments seem to be more rigorous than GBA appraisals regarding the consideration of evidence-based evaluation criteria, decisions by GBA might be negatively associated with the BI.

CONCLUSIONS

Results reveal that GBA/IQWiG and NICE follow their official evaluation criteria consistently. After controlling for all significant variables, the BI seems to have an (independent) effect on HTA outcomes as well.

HTA Barriers for Conditional Approval Drugs

BACKGROUND

Conditional approval pathways facilitate accelerated marketing authorisation based on immature clinical evidence for drugs that address an unmet medical need in a life-threatening or chronically debilitating condition. Lowering evidence requirements for marketing authorisation results in higher clinical uncertainty, which may present challenges for the health technology assessment (HTA) of these products.

OBJECTIVES

The objective of this study is to assess whether conditionally approved drugs face higher probabilities of HTA rejection or delays in HTA approval relative to drugs with standard marketing authorisation.

METHODS

This paper adopts a mixed-methods approach to provide a meta-analysis of HTA outcomes across 80 drug-indication pairs in France, England, Scotland and Canada. Differences in the characteristics (i.e. disease rarity and clinical trial design) of conditionally approved drugs and drugs with standard marketing authorisation and drivers of HTA outcomes are assessed through logistics regressions. Delays in HTA approval are assessed through a survival analysis.

RESULTS

Relative to standard approval drugs, conditionally approved drugs are less likely to include phase III trial designs, less likely to include clinical endpoints and less likely to include an active comparator. Uncertainties in clinical and economic evidence are raised more frequently by HTA agencies for conditionally approved drugs, which have a marginally lower probability of receiving HTA approval relative to drugs with standard approval. Conditionally approved drugs face moderate delays (an average of 6 months) in receiving HTA approval relative to standard approval drugs.

CONCLUSIONS

Overall, conditionally approved drugs likely face increased barriers at the HTA level.

Improving Interpretation of Evidence Relating to Quality of Life in Health Technology Assessments of Rare Disease Treatments

Rare diseases are often severe, debilitating, life-limiting conditions, many of which occur in childhood. These complex conditions have a wide range of clinical manifestations that have a substantial impact on the lives of patients, carers and families and often produce heterogeneous clinical outcomes. Therefore, the evaluation of quality-of-life (QoL) impacts is important. In health technology assessment (HTA), patient-reported outcome measures (PROMs) and/or health state utility values (HSUVs) are used to determine QoL impacts of new treatments, but their use in rare diseases is challenging due to small and heterogeneous populations and limited disease knowledge. This paper describes challenges associated with the use of patient-reported outcomes (PROs)/HSUVs to evaluate QoL in HTA of rare disease treatments (RDTs) and identifies five recommendations to ensure appropriate interpretation of QoL impacts. These were derived from mixed methods research (literature reviews, appraisal document analyses, appraisal committee observations and interviews) examining the use of PROs/HSUVs in HTA of RDTs. They highlight that HTAs of RDTs must (1) understand the QoL impacts of the disease and of treatments; (2) critically assess PRO data, recognising the nuances in development and administration of PROMs/HSUVs, considering what is feasible and what matters most to the patient population; (3) recognise that lack of significant effect on a PRO does not imply no QoL benefit; (4) use different forms of evidence to understand QoL impacts, such as patient input; and (5) provide methodological guidance to capture QoL impacts on patients/carers.

Value assessment of medicinal products by the Italian Medicines Agency (AIFA) and French National Authority for Health (HAS): Similarities and discrepancies

The evaluation of pharmaceutical innovation and therapeutic value is an increasingly complex exercise for which different approaches are adopted at the national level, despite the need for standardisation of processes and harmonisation of public health decisions. The objective of our analysis was to compare the approaches of the AIFA (Agenzia Italiana del Farmaco) and the HAS (Haute Autorité de Santé) in assessing the same medicinal products. In Italy, the 1525/2017 AIFA Deliberation introduces a transparent scheme for the evaluation of innovative status (innovative, conditional, not innovative) based on the therapeutic added value (TAV), therapeutic need, and quality of evidence. In contrast, in France, the HAS makes judgements using the effective clinical benefit (Service Médical Rendu) and improvement of effective clinical benefit (Amélioration du Service Médical Rendu, ASMR). This analysis focused on medicinal products evaluated both by the AIFA and by the HAS from July 2017 to September 2021. Similarities between AIFA and HAS evaluations were investigated in terms of the TAV, recognition of innovativeness, and the ASMR. Both total and partial agreements were considered relevant. Therefore, raw agreement, Cohen's kappa (weighted and unweighted), and Bangdiwala's B-statistic were estimated. A total of 102 medicinal products were included in this study. Out of these, 38 (37.2%) were orphan drugs, while 56 (54.9%) had a clinical indication for the treatment of cancer. The AIFA and HAS reached a higher level of agreement on the innovativeness status compared with the TAV. A moderate total agreement emerged in the recognition of innovativeness (k = 0.463, p-value ≤0.0001), and partial agreement was substantial (equal weight k = 0.547, squared k = 0.638), while a lack of agreement resulted in a comparison of the TAV according to the AIFA and the ASMR recognised by the HAS. Indeed, whereas the AIFA determined the TAV to be important, the HAS considered it to be moderate. In addition, whereas the AIFA identified a bias towards a moderate TAV, the HAS identified a bias towards a minor ASMR. A higher level of agreement was reached, both on the TAV and on innovative status, for less critical medical products (non-cancer-related, or non-orphan, or with a standard European Medicines Agency approval). These results underline the importance of implementing European procedures that are more broadly aligned in terms of value definition criteria.

How Can We Optimize the Value Assessment and Appraisal of Orphan Drugs for Reimbursement Purposes? A Qualitative Interview Study Across European Countries

Introduction: The expansion of orphan drug treatment at increasing prices, together with uncertainties regarding their (cost-)effectiveness raises difficulties for decision-makers to assess these drugs for reimbursement. The present qualitative study aims to gain better insight into current value assessment and appraisal frameworks for orphan drugs, and provides guidance for improvement.

Methods: 22 European experts from 19 different countries were included in a qualitative survey, followed by in-depth semi-structured interviews. These experts were academics, members of reimbursement agencies or health authorities, or members of regulatory or health/social insurance institutions. Adopting a Grounded Theory approach, transcripts were analysed according to the QUAGOL method, supported by the qualitative data analysis software Nvivo.

Results: Although participants indicated several good practices (e.g., the involvement of patients and the presence of structure and consistency), several barriers (e.g., the lack of transparency) lead to questions regarding the efficiency of the overall reimbursement process. In addition, the study identified a number of “contextual” determinants (e.g., bias, perverse effects of the orphan drug legislation, and an inadequate consideration of the opportunity cost), which may undermine the legitimacy of orphan drug reimbursement decisions.

Conclusion: The present study provides guidance for decision-makers to improve the efficiency of orphan drug reimbursement. In particular, decision-makers can generate quick wins by limiting the impact of contextual determinants rather than improving the methods included in the HTA. When implemented into a framework that promotes “Accountability for Reasonableness” (A4R), this allows decision-makers to improve the legitimacy of reimbursement decisions concerning future orphan drugs.

Health technology assessment (HTA) in England, France and Germany: what do matched drug pairs tell us about recommendations by national HTA agencies?

Aims: To explore health technology assessment (HTA) outcomes of matched drug pairs by national agencies in Germany (Gemeinsamer Bundesausschuss, GBA), France (Haute Autorité de Santé, HAS) and England and Wales (NICE). Methods: We considered published GBA decisions, HAS reports and NICE guidance from January 2011 to June 2018. HTAs of matched pairs were compared overall, and for non-cancer and cancer drugs separately. We further analyzed the role of additional attributes related to cancer therapies. Results: Matched pairs show higher concordance for GBA/HAS than for GBA/NICE and HAS/NICE. Overall, NICE evaluated technologies more favorably than GBA and HAS. GBA appraisals of cancer drugs, however, tended to be more positive than cancer-related recommendations by NICE and HAS. Conclusion: The findings indicate substantial variations in HTAs, although cancer-related outcomes seem to diverge less than non-cancer results.

Pharmaceutical companies' views on a health technology assessment (HTA) entity in Saudi Arabia

Saudi Arabia is undergoing a massive healthcare transformation to fulfill its new, national "Vision 2030." To align with this objective, Saudi Arabia is establishing a new, independent and evidence-based health technology assessment (HTA) entity to help it maximize health gains through efficient use of resources. This study was designed to ascertain how pharmaceutical companies perceive the creation of such a national HTA entity in Saudi Arabia; what they think about it and expect from it. To achieve the study's aim, we held a workshop in Riyadh, Saudi Arabia, lasting four and a half hours and hosted by the Saudi Ministry of Health (MOH). We invited 16 market access directors and managers from different multinational pharmaceutical companies to discuss the establishment of a national HTA entity. The findings from the workshop were structured around three axes: vision and remit; HTA method; and implementation and practical considerations. Overall, the pharmaceutical company participants were positive about HTA's value for the Saudi healthcare system and expressed willingness to adapt to meet its future requirements.