A qualitative study of manufacturers' submissions to the UK NICE single technology appraisal process

Assessment of quality of data submitted for NICE technology appraisals over two decades

BACKGROUND

The National Institute for Health and Care Excellence (NICE) pioneered the Health Technology Assessment (HTA) processes and methodologies. Technology appraisals (TAs) focus on pharmaceutical products and clinical and economic data, which are presented by the product manufacturers to the NICE appraisal committee for decision-making. Uncertainty in data reduces the chance of a positive outcome from the HTA process or requires a higher discount.

OBJECTIVE

To investigate the quality of clinical data (comparator, quality of life (QoL), randomised controlled trials (RCTs) and overall quality of evidence) submitted by the manufacturers to NICE.

DESIGN

This retrospective evaluation analysed active TAs published between 2000 and 2019 (up to TA600).

METHODS

For all TAs, we extracted data from the Assessment Group and Evidence Review Group reports and Final Appraisal Determinations on (1) the quality of submitted RCTs and (2) the overall quality of evidence submitted for decision-making. For single TAs, we also extracted data and its critique on QoL and comparators. Each category was scored for quality and analysed using descriptive statistics.

RESULTS

409 TAs were analysed (multiple technology appraisals (MTA)=104, single technology appraisal (STA)=305). In two-thirds of TAs, the overall quality of evidence was either poor (n=224, 55%) or unacceptable (n=41, 10%). In 39% (n=119) of the STAs, the quality of comparative evidence was considered poor, and in 17% (n=51) unacceptable. In 44% (n=135) of STAs, the quality of QoL data was considered poor, 15% (n=47) unacceptable, 33% (n=102) acceptable and 7% (n=21) as good. Over 20 years of longitudinal analysis did not show improvements in the quality of evidence submitted to NICE.

CONCLUSION

We found that the primary components of clinical evidence influencing NICE's decision-making framework were of poor quality. It is essential to continue to generate robust clinical data for premarket and postmarket introduction of medicines into clinical practice to ensure they deliver benefits to patients.

Integrating Qualitative Techniques in Model Development: A Case Study Using the Framework Approach

BACKGROUND

Despite their potential, there is limited uptake of formal qualitative methods in model development by modellers and health economists. The aim of this case study was to highlight in a real-world context how a qualitative approach has been applied to gain insight into current practice (delineating existing care pathways) for typhoid fever in Ghana, which can then assist in model structure conceptualisation in a model-based cost-effectiveness analysis.

METHODS

The perspectives of a range of healthcare professionals working in different settings and across different practices in the Eastern region of Ghana were captured with a self-administered survey using open-ended questions and analysed using the framework method.

RESULTS

A total of 51 completed questionnaires were retrieved representing a 73% response rate. It was found that two main care pathways for typhoid fever exist in Ghana and there was no consensus on how a new test might be applied to the existing pathways.

CONCLUSION

The two settings in Ghana have different care pathways and any cost-effectiveness analysis should consider the alternative pathways separately. This study demonstrated that framework analysis is a qualitative methodology that is likely to be accessible and feasible across a wide range of health economic settings.

How Qualitative Methods Can be Used to Inform Model Development

Decision-analytic models play a key role in informing healthcare resource allocation decisions. However, there are ongoing concerns with the credibility of models. Modelling methods guidance can encourage good practice within model development, but its value is dependent on its ability to address the areas that modellers find most challenging. Further, it is important that modelling methods and related guidance are continually updated in light of any new approaches that could potentially enhance model credibility. The objective of this article was to highlight the ways in which qualitative methods have been used and recommended to inform decision-analytic model development and enhance modelling practices. With reference to the literature, the article discusses two key ways in which qualitative methods can be, and have been, applied. The first approach involves using qualitative methods to understand and inform general and future processes of model development, and the second, using qualitative techniques to directly inform the development of individual models. The literature suggests that qualitative methods can improve the validity and credibility of modelling processes by providing a means to understand existing modelling approaches that identifies where problems are occurring and further guidance is needed. It can also be applied within model development to facilitate the input of experts to structural development. We recommend that current and future model development would benefit from the greater integration of qualitative methods, specifically by studying 'real' modelling processes, and by developing recommendations around how qualitative methods can be adopted within everyday modelling practice.

Issues Related to the Frequency of Exploratory Analyses by Evidence Review Groups in the NICE Single Technology Appraisal Process

BACKGROUND

Evidence Review Groups (ERGs) critically appraise company submissions as part of the National Institute for Health and Care Excellence (NICE) Single Technology Appraisal (STA) process. As part of their critique of the evidence submitted by companies, the ERGs undertake exploratory analyses to explore uncertainties in the company's model. The aim of this study was to explore pre-defined factors that might influence or predict the extent of ERG exploratory analyses.

OBJECTIVE

The aim of this study was to explore predefined factors that might influence or predict the extent of ERG exploratory analyses.

METHODS

We undertook content analysis of over 400 documents, including ERG reports and related documentation for the 100 most recent STAs (2009-2014) for which guidance has been published. Relevant data were extracted from the documents and narrative synthesis was used to summarise the extracted data. All data were extracted and checked by two researchers.

RESULTS

Forty different companies submitted documents as part of the NICE STA process. The most common disease area covered by the STAs was cancer (44%), and most ERG reports (n = 93) contained at least one exploratory analysis. The incidence and frequency of ERG exploratory analyses does not appear to be related to any developments in the appraisal process, the disease area covered by the STA, or the company's base-case incremental cost-effectiveness ratio (ICER). However, there does appear to be a pattern in the mean number of analyses conducted by particular ERGs, but the reasons for this are unclear and potentially complex.

CONCLUSIONS

No clear patterns were identified regarding the presence or frequency of exploratory analyses, apart from the mean number conducted by individual ERGs. More research is needed to understand this relationship.

The use of exploratory analyses within the National Institute for Health and Care Excellence single technology appraisal process: an evaluation and qualitative analysis

BACKGROUND

As part of the National Institute for Health and Care Excellence (NICE) single technology appraisal (STA) process, independent Evidence Review Groups (ERGs) critically appraise the company submission. During the critical appraisal process the ERG may undertake analyses to explore uncertainties around the company's model and their implications for decision-making. The ERG reports are a central component of the evidence considered by the NICE Technology Appraisal Committees (ACs) in their deliberations.

OBJECTIVE

The aim of this research was to develop an understanding of the number and type of exploratory analyses undertaken by the ERGs within the STA process and to understand how these analyses are used by the NICE ACs in their decision-making.

METHODS

The 100 most recently completed STAs with published guidance were selected for inclusion in the analysis. The documents considered were ERG reports, clarification letters, the first appraisal consultation document and the final appraisal determination. Over 400 documents were assessed in this study. The categories of types of exploratory analyses included fixing errors, fixing violations, addressing matters of judgement and the ERG-preferred base case. A content analysis of documents (documentary analysis) was undertaken to identify and extract relevant data, and narrative synthesis was then used to rationalise and present these data.

RESULTS

The level and type of detail in ERG reports and clarification letters varied considerably. The vast majority (93%) of ERG reports reported one or more exploratory analyses. The most frequently reported type of analysis in these 93 ERG reports related to the category 'matters of judgement', which was reported in 83 (89%) reports. The category 'ERG base-case/preferred analysis' was reported in 45 (48%) reports, the category 'fixing errors' was reported in 33 (35%) reports and the category 'fixing violations' was reported in 17 (18%) reports. The exploratory analyses performed were the result of issues raised by an ERG in its critique of the submitted economic evidence. These analyses had more influence on recommendations earlier in the STA process than later on in the process.

LIMITATIONS

The descriptions of analyses undertaken were often highly specific to a particular STA and could be inconsistent across ERG reports and thus difficult to interpret.

CONCLUSIONS

Evidence Review Groups frequently conduct exploratory analyses to test or improve the economic evaluations submitted by companies as part of the STA process. ERG exploratory analyses often have an influence on the recommendations produced by the ACs.

FUTURE WORK

More in-depth analysis is needed to understand how ERGs make decisions regarding which exploratory analyses should be undertaken. More research is also needed to fully understand which types of exploratory analyses are most useful to ACs in their decision-making.

FUNDING

The National Institute for Health Research Health Technology Assessment programme.

Exploring Uncertainty in Economic Evaluations of Drugs and Medical Devices: Lessons from the First Review of Manufacturers' Submissions to the French National Authority for Health

OBJECTIVES

The objective of this paper was to evaluate how uncertainty has been accounted for in the cost-effectiveness analyses (CEAs) submitted by manufacturers to the French National Authority for Health (HAS) and to identify recurring concerns in these submissions.

METHODS

We used a cross-sectional design to evaluate manufacturers' submissions from the beginning of the evaluation process in October 2013 to the end of May 2015 (n = 28). The sources of uncertainty attached to these CEAs were categorized and assessed. Relevant data were extracted independently by two assessors.

RESULTS

Adherence to the HAS reference case was generally considered to be acceptable. Methodological uncertainty and parameter uncertainty were the sources of uncertainty that were most frequently explored by manufacturers. The quality of reporting of deterministic sensitivity analysis and probabilistic sensitivity analysis varied substantially across submissions, with a frequent lack of justification of the plausible range of parameter point estimates in 12 submissions (43 %). Structural uncertainty was explored much less frequently. Concerns related to omission of either important clinical events or relevant health states or extrapolation of the effects of the technology beyond the time horizon of the clinical trials were identified in 16 submissions (57 %).

CONCLUSIONS

This study presented a characterization of the treatment of uncertainty for the first 28 manufacturers' submissions to the HAS. This work identified important concerns regarding the exploration of sources of uncertainty. The findings may help manufacturers to improve the quality of their submissions and may provide useful insights for extending guidelines on uncertainty analysis in CEAs submitted to the HAS.

Management of locally advanced and metastatic breast cancer: guidelines, infrastructures and low resource settings

Breast cancer incidence is rising and it accounts for over 1.6 million cases per year worldwide. It represents about one-third of female cancers and is a significant health issue in countries at all economic levels. In this article, we review global cancer statistics, breast cancer staging including late-stage presentation in limited resource settings, disparities in the breast cancer outcomes and requirements for optimal management, including infrastructure needs for optimal surgery, radiation treatment and systemic therapy. We will discuss controversies related to drug pricing and availability, process and delays in registration of new drugs as well as resource stratification and resource-stratified guidelines for locally advanced breast cancer and metastatic breast cancer.

New French Coverage with Evidence Development for Innovative Medical Devices: Improvements and Unresolved Issues

We describe here recent modifications to the French Coverage with Evidence Development (CED) scheme for innovative medical devices. CED can be defined as temporary coverage for a novel health product during collection of the additional evidence required to determine whether definitive coverage is possible. The principle refinements to the scheme include a more precise definition of what may be considered an innovative product, the possibility for device manufacturers to request CED either independently or in partnership with hospitals, and the establishment of processing deadlines for health authorities. In the long term, these modifications may increase the number of applications to the CED scheme, which could lead to unsustainable funding for future projects. It will also be necessary to ensure that the study conditions required by national health authorities are suitable for medical devices and that processing deadlines are met for the scheme to be fully operational. Overall, the modifications recently applied to the French CED scheme for innovative medical devices should increase the transparency of the process, and therefore be more appealing to medical device manufacturers.

Assessment of the Quality of the Clinical Evidence in Submissions to the Australian Pharmaceutical Benefits Advisory Committee: Fit for Purpose?

BACKGROUND

Assessments of the comparative clinical (and cost) effectiveness of new medicines are increasingly being used to inform decisions on their reimbursement. Assessments of added clinical benefit are invariably based on evidence generated to support registration.

OBJECTIVE

Our objective was to identify and characterize significant problems relating to the quality of the clinical evidence in submissions to the Australian Pharmaceutical Benefits Advisory Committee (PBAC) seeking subsidy on the Pharmaceutical Benefits Scheme and thus determine whether the evidence presented to the committee was "fit for purpose."

METHODS

We conducted a retrospective analysis of submissions considered by the PBAC between 2005 and 2012 using a published evaluation framework. We developed an additional framework to categorize significant problems in more detail. Significant problems related to the choice of comparator, the unavailability of randomized clinical trial evidence, poor-quality data, a claim of clinical superiority, and a claim of clinical noninferiority.

RESULTS

We identified 261 significant problems in 479 major submissions. There was a significant problem with the sponsor's choice of comparator in 11% of the submissions. The most common significant problem (29%) was the determination of a medicine's comparative performance in the target patient population.

CONCLUSIONS

The supporting clinical evidence is the foundation of a PBAC submission. We found a poor fit for purpose; on average, one in every two major submissions had a significant problem with the supporting evidence. The findings from our study, if confirmed in other jurisdictions, raise important questions regarding what clinical evidence should be generated to support the reimbursement of new medicines.

ASSESSING SEARCHES IN NICE SINGLE TECHNOLOGY APPRAISALS: PRACTICE AND CHECKLIST

Objectives: No guidelines exist in the approach that Evidence Review Groups (ERGs) should take to appraise search methodologies in the manufacturer's submission (MS) in Single Technology Appraisals (STA). As a result, ERGs are left to appraise searches using their own approach. This study investigates the limitations of manufacturers' search methodologies as critiqued by ERGs in published STA reports and to provide a recommended checklist.

Methods: Limitations from search critiques in 83 ERG reports published in the NIHR Web site between 2006 and May 2011 were extracted. The limitations were grouped into themes. Comparisons were made between limitations reported in the clinical effectiveness versus cost-effectiveness searches.

Results: Twelve themes were identified, six relating to the search strategy, source, limits, filters, translation, reporting, and missing studies. The search strategy theme contained the most limitations. Missing studies were frequently found by the ERG group in the clinical effectiveness searches. The omission of searches by manufacturers for unpublished and ongoing trials was frequently reported by the ERG. By contrast, failure of the manufacturer to report strategies was the most common limitation in the cost-effectiveness searches. Themes with the most frequent limitations in both types of searches are search strategy, reporting and source.

Conclusions: It is recommended that a checklist that has reporting, source and search strategy elements be used in the appraisal of manufacturer's searches during the STA process.