Systematic bias in predictions of new drugs' budget impact: analysis of a sample of recent US drug launches

Assessing the Accuracy of Sales Forecasts Submitted by Pharmaceutical Companies Applying for Reimbursement in Austria

Objectives: Reimbursement decisions on new medicines require an assessment of their value. In Austria, when applying for reimbursement of new medicines, pharmaceutical companies are also obliged to submit forecasts of future sales. We systematically examined the accuracy of these pharmaceutical sales forecasts and hence the usefulness of these forecasts for reimbursement evaluations. Methods: We retrospectively analyzed reimbursement applications of 102 new drugs submitted between 2005 and 2014, which were accepted for reimbursement outside of hospitals, and for which actual reimbursed sales were available for at least 3 years. The main outcome variable was the accuracy ratio, defined as the ratio of forecasted sales submitted by pharmaceutical companies when applying for reimbursement to actual sales from reimbursement data. Results: The median accuracy ratio [95% confidence interval] was 1.33 [1.03; 1.74, range 0.15–37.5], corresponding to a median overestimation of actual sales by 33%. Forecasts of actual sales for 55.9% of all examined products either overestimated actual sales by more than 100% or underestimated them by more than 50%. The accuracy of sales forecasts did not show systematic change over the analyzed decade nor was it discernibly influenced by reimbursement status (restricted or unrestricted), the degree of therapeutic benefit, or the therapeutic area of the pharmaceutical product. Sales forecasts of drugs with a higher degree of innovation and those within a dynamic market tended to be slightly more accurate. Conclusions: The majority of sales forecasts provided by applicants for reimbursement evaluations in Austria were highly inaccurate and were on average too optimistic. This is in line with published results for other jurisdictions and highlights the need for caution when using such forecasts for reimbursement procedures.

Budget impact analysis of medicines: estimated values versus real-world evidence and the implications

Objectives: Budget Impact Analyses (BIA) of medicines helps managers in promoting health systems' sustainability when assessing the role and value of new medicines. However, it is not clear whether BIAs typically underestimate or overestimate the impact on real-world budgets. This retroactive analysis seeks to compare estimated values obtained by a BIA and Real-World Evidence (RWE) to guide discussions.Methods: The estimated values were obtained through a BIA concerning the incorporation of adalimumab for the treatment of Rheumatoid Arthritis into the Brazilian Unified Health System (SUS) carried out retroactively and per international guidelines. RWE data was extracted from SUS computerized systems. We subsequently compared the number of treatments, costs, and Incremental Budget Impact (IBI).Results - The total number of treatments was underestimated by 10% (6,243) and the total expenditure was overestimated by 463% (US$ 4.7 billion). In five years, the total difference between the estimated values and real IBI reached US$ 1.1 billion. A current expenditure of US$ 1.0 was observed for every US$ 5.60 of estimated expenditure.Conclusion - The higher estimates from the BIA might cause decision makers to be more cautious with the introduction of a new medicine to reduce the opportunity costs for other interventions.

A novel method for predicting the budget impact of innovative medicines: validation study for oncolytics

BACKGROUND

High budget impact (BI) estimates of new drugs have led to decision-making challenges potentially resulting in restrictions in patient access. However, current BI predictions are rather inaccurate and short term. We therefore developed a new approach for BI prediction. Here, we describe the validation of our BI prediction approach using oncology drugs as a case study.

METHODS

We used Dutch population-level data to estimate BI where BI is defined as list price multiplied by volume. We included drugs in the antineoplastic agents ATC category which the European Medicines Agency (EMA) considered a New Active Substance and received EMA marketing authorization (MA) between 2000 and 2017. A mixed-effects model was used for prediction and included tumor site, orphan, first in class or conditional approval designation as covariates. Data from 2000 to 2012 were the training set. BI was predicted monthly from 0 to 45 months after MA. Cross-validation was performed using a rolling forecasting origin with e^|Ln(observed BI/predicted BI)| as outcome.

RESULTS

The training set and validation set included 25 and 44 products, respectively. Mean error, composed of all validation outcomes, was 2.94 (median 1.57). Errors are higher with less available data and at more future predictions. Highest errors occur without any prior data. From 10 months onward, error remains constant.

CONCLUSIONS

The validation shows that the method can relatively accurately predict BI. For payers or policymakers, this approach can yield a valuable addition to current BI predictions due to its ease of use, independence of indications and ability to update predictions to the most recent data.

Affordability of oncology drugs: accuracy of budget impact estimations

Background: In many countries, Budget Impact (BI) informs reimbursement decisions. Evidence has shown that decision-makers have restricted access based on high BI estimates but studies show that BI estimates are often inaccurate. Objective: To assess the accuracy of BI estimations used for informing access decisions on oncology drugs in the Netherlands. Study Design: Oncology products for which European Medicines Agency Marketing Authorisation was granted between 1-1-2000 and 1-10-2017 were selected. Observed BI data were provided by FarmInform. BI estimates were extracted from the reimbursement dossiers of the Dutch Healthcare Institute. Products without an estimated BI in the reimbursement dossier were excluded. Accuracy is defined as the ratio observed BI/estimated BI. Setting: General community, the Netherlands. Results: Ten products were included in the base case analysis. Mean accuracy was 0.64 and observed BI deviated by more than 40% and 100% from the estimated BI for 4 and 5 products, respectively. For all products together, €141 million BI was estimated and €82 million BI was observed, a €59 million difference. Conclusions: The findings indicate that BI estimates for oncology drugs in the Netherlands are inaccurate. The role and use of BI in reimbursement decisions for these potentially life-saving drugs should therefore be considered carefully, as well as BI estimation methodology.

Estimated Versus Observed Expenditure Associated with Medicines Recommended by the All Wales Medicines Strategy Group

OBJECTIVES

The All Wales Medicines Strategy Group (AWMSG) appraises the clinical and cost effectiveness of new medicines being considered for National Health Service (NHS) prescribing in Wales (UK). The aim of this study was to compare the estimated expenditure on selected medicines submitted by pharmaceutical companies for appraisal with the observed expenditure on these medicines following recommendation.

METHODS

Medicines appraised and recommended for use in NHS Wales by AWMSG between May 2005 and December 2013 were identified for inclusion in the study. Estimates of expenditure were obtained from company submissions to AWMSG. Primary and secondary care dispensing databases were used to obtain observed expenditure. The Wilcoxon matched-pairs signed rank test was used to compare the observed and estimated expenditure in each of the 3 years after introduction of the medicine.

RESULTS

Forty-nine medicines appraised and recommended by AWMSG during the period of interest were included in the study. Median estimated and observed expenditure in each of the 3 years post-recommendation were as follows: year 1 £86,400 and £47,300; year 2 £175,500 and £73,200; year 3 £212,100 and £78,900 (p = 0.03, p = 0.006 and p = 0.001, respectively). The expenditure on 42 of the 49 medicines (82%) was overestimated in at least one of the 3 years post-introduction, with 32 (65%) overestimated in all 3 years.

CONCLUSION

In their applications for health technology appraisal, pharmaceutical companies tended to overestimate the expenditure of the majority of medicines recommended by AWMSG. These findings have implications for the assessment of predicted expenditure as part of the process of medicines appraisal in Wales.

Accuracy of budget impact estimations and impact on patient access: a hepatitis C case study

BACKGROUND

High budget impact (BI) estimates of new drugs limit access to patients due to concerns regarding affordability and displacement effects. The accuracy and methodological quality of BI analyses are often low, potentially mis-informing reimbursement decision making. Using hepatitis C as a case study, we aim to quantify the accuracy of the BI predictions used in Dutch reimbursement decision-making and to characterize the influence of market-dynamics on actual BI.

METHODS

We selected hepatitis C direct-acting antivirals (DAAs) that were introduced in the Netherlands between January 2014 and March 2018. Dutch National Health Care Institute (ZIN) BI estimates were derived from the reimbursement dossiers. Actual Dutch BI data were provided by FarmInform. BI prediction accuracy was assessed by comparing the ZIN BI estimates with the actual BI data.

RESULTS

Actual BI, from 1 Jan 2014 to 1 March 2018, was €248 million whilst the BI estimates ranged from €388-€510 million. The latter figure represents the estimated BI for the reimbursement scenario that was adopted, implying a €275 million overestimation. Absent incorporation of timing of regulatory decisions and inadequate correction for the introduction of new products were main drivers of BI overestimation, as well as uncertainty regarding the patient population size and the impact of the final reimbursement decision.

DISCUSSION

BI in reimbursement dossiers largely overestimated actual BI of hepatitis C DAAs. When BI analysis is performed according to existing guidelines, the resulting more accurate BI estimates may lead to better informed reimbursement decisions.

Challenges with Forecasting Budget Impact: A Case Study of Six ICER Reports

BACKGROUND

Payers frequently rely on budget impact model (BIM) results to help determine drug coverage policy and its effect on their bottom line. It is unclear whether BIMs typically overestimate or underestimate real-world budget impact.

OBJECTIVE

We examined how different modeling assumptions influenced the results of 6 BIMs from the Institute for Clinical and Economic Review (ICER).

STUDY DESIGN

Retrospective analysis of pharmaceutical sales data.

METHODS

From ICER reports issued before 2016, we collected estimates of 3 BIM outputs: aggregate therapy cost (ie, cost to treat the patient population with a particular therapy), therapy uptake, and price. We compared these against real-world estimates that we generated using drug sales data. We considered 2 classes of BIM estimates: those forecasting future uptake of new agents, which assumed "unmanaged uptake," and those describing the contemporaneous market state (ie, estimates of current, managed uptake and budget impact for compounds already on the market).

RESULTS

Differences between ICER's estimates and our own were largest for forecasted studies. Here, ICER's uptake estimates exceeded real-world estimates by factors ranging from 7.4 (sacubitril/valsartan) to 54 (hepatitis C treatments). The "unmanaged uptake" assumption (removed from ICER's approach in 2017) yields large deviations between BIM estimates and real-world consumption. Nevertheless, in some cases, ICER's BIMs that relied on current market estimates also deviated substantially from real-world sales data.

CONCLUSIONS

This study highlights challenges with forecasting budget impact. In particular, assumptions about uptake and data source selection can greatly influence the accuracy of results.