When is enough evidence enough? - Using systematic decision analysis and value-of-information analysis to determine the need for further evidence

A Guide to an Iterative Approach to Model-Based Decision Making in Health and Medicine: An Iterative Decision-Making Framework

Decision makers frequently face decisions about optimal resource allocation. A model-based economic evaluation can be used to guide decision makers in their choices by systematically evaluating the magnitude of expected health effects and costs of decision options and by making trade-offs explicit. We provide a guide to an iterative approach to the medical decision-making process by following a coherent framework, and outline the overarching iterative steps of model-based decision making. We systematized the framework by performing three steps. First, we compiled the existing guidelines provided by the ISPOR-SMDM Modeling Good Research Practices Task Force, and the ISPOR Value of Information Task Force. Second, we identified other previous work related to frameworks and guidelines for model-based decision analyses through a literature search in PubMed. Third, we assessed the role of the evidence and iterative process in decision making and formalized key steps in a model-based decision-making framework. We provide guidance on an iterative approach to medical decision making by applying the compiled iterative model-based decision-making framework. The framework formally combines the decision problem conceptualization (Part I), the model conceptualization and development (Part II), and the process of model-based decision analysis (Part III). Following the overarching steps of the framework ensures compliance to the principles of evidence-based medicine and regular updates of the evidence, given that value of information analysis represents an essential component of model-based decision analysis in the framework. Following the provided guide and the steps outlined in the framework can help inform various health care decisions, and therefore it has the potential to improve decision making.

Integrative approaches based on genomic techniques in the functional studies on enhancers

With the development of sequencing technology and the dramatic drop in sequencing cost, the functions of noncoding genes are being characterized in a wide variety of fields (e.g. biomedicine). Enhancers are noncoding DNA elements with vital transcription regulation functions. Tens of thousands of enhancers have been identified in the human genome; however, the location, function, target genes and regulatory mechanisms of most enhancers have not been elucidated thus far. As high-throughput sequencing techniques have leapt forwards, omics approaches have been extensively employed in enhancer research. Multidimensional genomic data integration enables the full exploration of the data and provides novel perspectives for screening, identification and characterization of the function and regulatory mechanisms of unknown enhancers. However, multidimensional genomic data are still difficult to integrate genome wide due to complex varieties, massive amounts, high rarity, etc. To facilitate the appropriate methods for studying enhancers with high efficacy, we delineate the principles, data processing modes and progress of various omics approaches to study enhancers and summarize the applications of traditional machine learning and deep learning in multi-omics integration in the enhancer field. In addition, the challenges encountered during the integration of multiple omics data are addressed. Overall, this review provides a comprehensive foundation for enhancer analysis.

Decision-analytic modeling for early health technology assessment of medical devices - a scoping review

Objective

The goal of this review was to identify decision-analytic modeling studies in early health technology assessments (HTA) of high-risk medical devices, published over the last three years, and to provide a systematic overview of model purposes and characteristics. Additionally, the aim was to describe recent developments in modeling techniques.

Methods

For this scoping review, we performed a systematic literature search in PubMed and Embase including studies published in English or German. The search code consisted of terms describing early health technology assessment and terms for decision-analytic models. In abstract and full-text screening, studies were excluded that were not modeling studies for a high-risk medical device or an in-vitro diagnostic test. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram was used to report on the search and exclusion of studies. For all included studies, study purpose, framework and model characteristics were extracted and reported in systematic evidence tables and a narrative summary.

Results

Out of 206 identified studies, 19 studies were included in the review. Studies were either conducted for hypothetical devices or for existing devices after they were already available on the market. No study extrapolated technical data from early development stages to estimate potential value of devices in development. All studies except one included cost as an outcome. Two studies were budget impact analyses. Most studies aimed at adoption and reimbursement decisions. The majority of studies were on in-vitro diagnostic tests for personalized and targeted medicine. A timed automata model, to our knowledge a model type new to HTA, was tested by one study. It describes the agents in a clinical pathway in separate models and, by allowing for interaction between the models, can reflect complex individual clinical pathways and dynamic system interactions. Not all sources of uncertainty for in-vitro tests were explicitly modeled. Elicitation of expert knowledge and judgement was used for substitution of missing empirical data. Analysis of uncertainty was the most valuable strength of decision-analytic models in early HTA, but no model applied sensitivity analysis to optimize the test positivity cutoff with regard to the benefit-harm balance or cost-effectiveness. Value-of-information analysis was rarely performed. No information was found on the use of causal inference methods for estimation of effect parameters from observational data.

Conclusion

Our review provides an overview of the purposes and model characteristics of nineteen recent early evaluation studies on medical devices. The review shows the growing importance of personalized interventions and confirms previously published recommendations for careful modeling of uncertainties surrounding diagnostic devices and for increased use of value-of-information analysis. Timed automata may be a model type worth exploring further in HTA. In addition, we recommend to extend the application of sensitivity analysis to optimize positivity criteria for in-vitro tests with regard to benefit-harm or cost-effectiveness. We emphasize the importance of causal inference methods when estimating effect parameters from observational data.

Challenges around quantifying uncertainty in a holistic approach to hard-to-heal wound management: Health economic perspective

Treatment of hard-to-heal wounds involves a holistic approach for choosing between available treatment options. However, evidence for informing these choices is sparse, introducing uncertainty into decisions about the optimum treatment pathways that reflect the vast heterogeneity in this patient population. This paper discusses the existing clinical and health economic literature in order to provide insight into sources of uncertainty in the evaluation of the holistic approach to management of the hard-to-heal wounds, and how this uncertainty can be appropriately reflected in research. We identified three key sources of uncertainty in the evaluation of chronic wound treatments, namely heterogeneity in aetiology and patient populations, heterogeneity in treatment pathways, and challenges around capturing all relevant outcomes. Reflecting these complexities requires sophisticated modelling of treatment sequencing and long-term outcomes. The paper discusses how the scope specification, scenario analyses, and sensitivity analyses can be used to fully characterise analytical uncertainty.

On the role of data, statistics and decisions in a pandemic

A pandemic poses particular challenges to decision-making because of the need to continuously adapt decisions to rapidly changing evidence and available data. For example, which countermeasures are appropriate at a particular stage of the pandemic? How can the severity of the pandemic be measured? What is the effect of vaccination in the population and which groups should be vaccinated first? The process of decision-making starts with data collection and modeling and continues to the dissemination of results and the subsequent decisions taken. The goal of this paper is to give an overview of this process and to provide recommendations for the different steps from a statistical perspective. In particular, we discuss a range of modeling techniques including mathematical, statistical and decision-analytic models along with their applications in the COVID-19 context. With this overview, we aim to foster the understanding of the goals of these modeling approaches and the specific data requirements that are essential for the interpretation of results and for successful interdisciplinary collaborations. A special focus is on the role played by data in these different models, and we incorporate into the discussion the importance of statistical literacy and of effective dissemination and communication of findings.

KEY LEARNINGS FROM ICER'S REAL WORLD EVIDENCE REASSESSMENT PILOT

Health technology assessment (HTA) agencies are considering adopting a lifecycle approach to assessments to address uncertainties in the evidence base at launch and to revisit the clinical and economic value of therapies in a dynamic clinical landscape. For reassessments of therapies post launch, HTA agencies are looking to real-world evidence (RWE) to enhance the clinical and economic evidence base, though challenges and concerns in using RWE in decision-making exists. Stakeholders are embarking on demonstration projects to address the challenges and concerns and to further define when and how RWE can be used in HTA decision making. The Institute for Clinical and Economic Review piloted a 24-month observational RWE reassessment. Key learnings from this pilot include identifying the benefits and challenges with using RWE in reassessments and considerations on prioritizing and selecting topics relevant for RWE updates.

Development of a core outcome set for myelodysplastic syndromes - a Delphi study from the EUMDS Registry Group

Treatment options for myelodysplastic syndromes (MDS) vary widely, depending on the natural disease course and patient‐related factors. Comparison of treatment effectiveness is challenging as different endpoints have been included in clinical trials and outcome reporting. Our goal was to develop the first MDS core outcome set (MDS‐COS) defining a minimum set of outcomes that should be reported in future clinical studies. We performed a comprehensive systematic literature review among MDS studies to extract patient‐ and/or clinically relevant outcomes. Clinical experts from the European LeukemiaNet MDS (EUMDS) identified 26 potential MDS core outcomes and participated in a three‐round Delphi survey. After the first survey (56 experts), 15 outcomes met the inclusion criteria and one additional outcome was included. The second round (38 experts) resulted in six included outcomes. In the third round, a final check on plausibility and practicality of the six included outcomes and their definitions was performed. The final MDS‐COS includes: health‐related quality of life, treatment‐related mortality, overall survival, performance status, safety, and haematological improvement. This newly developed MDS‐COS represents the first minimum set of outcomes aiming to enhance comparability across future MDS studies and facilitate a better understanding of treatment effectiveness.

The Value of Decision Analytical Modeling in Surgical Research: An Example of Laparoscopic Versus Open Distal Pancreatectomy

OBJECTIVE

To illustrate how decision modeling may identify relevant uncertainty and can preclude or identify areas of future research in surgery.

SUMMARY BACKGROUND DATA

To optimize use of research resources, a tool is needed that assists in identifying relevant uncertainties and the added value of reducing these uncertainties.

METHODS

The clinical pathway for laparoscopic distal pancreatectomy (LDP) versus open (ODP) for nonmalignant lesions was modeled in a decision tree. Cost-effectiveness based on complications, hospital stay, costs, quality of life, and survival was analyzed. The effect of existing uncertainty on the cost-effectiveness was addressed, as well as the expected value of eliminating uncertainties.

RESULTS

Based on 29 nonrandomized studies (3.701 patients) the model shows that LDP is more cost-effective compared with ODP. Scenarios in which LDP does not outperform ODP for cost-effectiveness seem unrealistic, e.g., a 30-day mortality rate of 1.79 times higher after LDP as compared with ODP, conversion in 62.2%, surgically repair of incisional hernias in 21% after LDP, or an average 2.3 days longer hospital stay after LDP than after ODP. Taking all uncertainty into account, LDP remained more cost-effective. Minimizing these uncertainties did not change the outcome.

CONCLUSIONS

The results show how decision analytical modeling can help to identify relevant uncertainty and guide decisions for future research in surgery. Based on the current available evidence, a randomized clinical trial on complications, hospital stay, costs, quality of life, and survival is highly unlikely to change the conclusion that LDP is more cost-effective than ODP.

Economic burden of stroke: a systematic review on post-stroke care

OBJECTIVES

Stroke is a leading cause for disability and morbidity associated with increased economic burden due to treatment and post-stroke care (PSC). The aim of our study is to provide information on resource consumption for PSC, to identify relevant cost drivers, and to discuss potential information gaps.

METHODS

A systematic literature review on economic studies reporting PSC-associated data was performed in PubMed/MEDLINE, Scopus/Elsevier and Cochrane databases, Google Scholar and gray literature ranging from January 2000 to August 2016. Results for post-stroke interventions (treatment and care) were systematically extracted and summarized in evidence tables reporting study characteristics and economic outcomes. Economic results were converted to 2015 US Dollars, and the total cost of PSC per patient month (PM) was calculated.

RESULTS

We included 42 studies. Overall PSC costs (inpatient/outpatient) were highest in the USA ($4850/PM) and lowest in Australia ($752/PM). Studies assessing only outpatient care reported the highest cost in the United Kingdom ($883/PM), and the lowest in Malaysia ($192/PM). Fifteen different segments of specific services utilization were described, in which rehabilitation and nursing care were identified as the major contributors.

CONCLUSION

The highest PSC costs were observed in the USA, with rehabilitation services being the main cost driver. Due to diversity in reporting, it was not possible to conduct a detailed cost analysis addressing different segments of services. Further approaches should benefit from the advantages of administrative and claims data, focusing on inpatient/outpatient PSC cost and its predictors, assuring appropriate resource allocation.

Optimal Information Collection Policies in a Markov Decision Process Framework

BACKGROUND

The cost-effectiveness and value of additional information about a health technology or program may change over time because of trends affecting patient cohorts and/or the intervention. Delaying information collection even for parameters that do not change over time may be optimal.

METHODS

We present a stochastic dynamic programming approach to simultaneously identify the optimal intervention and information collection policies. We use our framework to evaluate birth cohort hepatitis C virus (HCV) screening. We focus on how the presence of a time-varying parameter (HCV prevalence) affects the optimal information collection policy for a parameter assumed constant across birth cohorts: liver fibrosis stage distribution for screen-detected diagnosis at age 50.

RESULTS

We prove that it may be optimal to delay information collection until a time when the information more immediately affects decision making. For the example of HCV screening, given initial beliefs, the optimal policy (at 2010) was to continue screening and collect information about the distribution of liver fibrosis at screen-detected diagnosis in 12 years, increasing the expected incremental net monetary benefit (INMB) by $169.5 million compared to current guidelines.

CONCLUSIONS

The option to delay information collection until the information is sufficiently likely to influence decisions can increase efficiency. A dynamic programming framework enables an assessment of the marginal value of information and determines the optimal policy, including when and how much information to collect.

Conditional power as an aid in making interim decisions in observational studies

Conditional power combines the findings of a partially completed study with assumptions about the future. The goal is to estimate the probability that the eventual study result will be incompatible with a criterion value, such as acceptable risk or the null hypothesis. Some history and motivation for conditional power calculations are provided, with examples illustrating the application to drug safety studies. This is an expository article suggesting that conditional power, which is well-established in clinical trials research, also has application to observational studies. The utility may be highest in regulatory settings where resources are limited and interim decisions have to be made accurately in the shortest possible time.

The development of funding recommendations for health technologies at the state level: A South Australian case study

OBJECTIVES

State governments often face capped budgets that can restrict expenditure on health technologies and their evaluation, yet many technologies are introduced to practice through state-funded institutions such as hospitals, rather than through national evaluation mechanisms. This research aimed to identify the criteria, evidence, and standards used by South Australian committee members to recommend funding for high-cost health technologies.

METHODS

We undertook 8 semi-structured interviews and 2 meeting observations with members of state-wide committees that have a mandate to consider the safety, effectiveness, and cost-effectiveness of high-cost health technologies.

RESULTS

Safety and effectiveness were fundamental criteria for decision makers, who were also concerned with increasing consistency in care and equitable access to technologies. Committee members often consider evidence that is limited in quantity and quality; however, they perceive evaluations to be rigorous and sufficient for decision making. Precise standards for safety, effective, and cost-effectiveness could not be identified.

CONCLUSIONS

Consideration of new technologies at the state level is grounded in the desire to improve health outcomes and equity of access for patients. High quality evidence is often limited. The impact funding decisions have on population health is unclear due to limited use of cost-effectiveness analysis and unclear cost-effectiveness standards.

Prioritizing HIV comparative effectiveness trials based on value of information: generic versus brand-name ART in the US

BACKGROUND

Value of Information (VOI) analysis examines whether to acquire information before making a decision. We introduced VOI to the HIV audience, using the example of generic antiretroviral therapy (ART) in the US.

METHODS AND FINDINGS

We used a mathematical model and probabilistic sensitivity analysis (PSA) to generate probability distributions of survival (in quality-adjusted life years, QALYs) and cost for three potential first-line ART regimens: three-pill generic, two-pill generic, and single-pill branded. These served as input for a comparison of two hypothetical two-arm trials: three-pill generic versus single-pill branded; and two-pill generic versus single-pill branded. We modeled pre-trial uncertainty by defining probability distributions around key inputs, including 24-week HIV-RNA suppression and subsequent ART failure. We assumed that, without a trial, patients received the single-pill branded strategy. Post-trial, we assumed that patients received the most cost-effective strategy. For both trials, we quantified the probability of changing to a generic-based regimen upon trial completion and the expected VOI in terms of improved health outcomes and costs. Assuming a willingness to pay (WTP) threshold of $100 000/QALY, the three-pill trial led to more treatment changes (84%) than the two-pill trial (78%). Estimated VOI was $48 000 (three-pill trial) and $35 700 (two-pill trial) per future patient initiating ART.

CONCLUSIONS

A three-pill trial of generic ART is more likely to lead to post-trial treatment changes and to provide more value than a two-pill trial if policy decisions are based on cost-effectiveness. Value of Information analysis can identify trials likely to confer the greatest impact and value for HIV care.

Oncotyrol--Center for Personalized Cancer Medicine: Methods and Applications of Health Technology Assessment and Outcomes Research

BACKGROUND

The Oncotyrol - Center for Personalized Cancer Medicine is an international and interdisciplinary alliance combining research and commercial competencies to accelerate the development, evaluation and translation of personalized healthcare strategies in cancer. The philosophy of Oncotyrol is to collaborate with relevant stakeholders and advance knowledge "from bench to bedside to population and back". Oncotyrol is funded through the COMET Excellence Program by the Austrian government via the national Austrian Research Promotion Agency (FFG). This article focuses on the role of health technology assessment (HTA) and outcomes research in personalized cancer medicine in the context of Oncotyrol.

METHODS

Oncotyrol, which currently comprises approximately 20 individual projects, has four research areas: Area 1: Biomarker and Drug Target Identification; Area 2: Assay Development and Drug Screening; Area 3: Innovative Therapies; Area 4: Health Technology Assessment and Bioinformatics. Area 4 translates the results from Areas 1 to 3 to populations and society and reports them back to Area 3 to inform clinical studies and guidelines, and to Areas 1 and 2 to guide further research and development.

RESULTS

In a series of international expert workshops, the Oncotyrol International Expert Task Force for Personalized Cancer Medicine developed the Methodological Framework for Early Health Technology Assessment and Decision Modeling in Cancer and practical guidelines in this field. Further projects included applications in the fields of sequential treatment of patients with chronic myeloid leukemia (CML), benefit-harm and cost-effectiveness evaluation of prostate cancer screening, effectiveness and cost-effectiveness of multiple cervical cancer screening strategies, and benefits and cost-effectiveness of genomic test-based treatment strategies in breast cancer.

CONCLUSION

An interdisciplinary setting as generated in Oncotyrol provides unique opportunities such as systematically coordinating lab and bench research, product development, clinical studies and decision science/HTA and transparent joint planning of research and development with a partnership of researchers, manufacturers and health policy decision makers. However, generating a joint research and legal framework with numerous partners from different sectors can be challenging, particularly in the starting period of such an endeavor. The journey to translational personalized medicine through multidisciplinary collaborations may still be long and difficult, but it is evident that it must be continued to turn vision into reality.

Concepts of 'personalization' in personalized medicine: implications for economic evaluation

CONTEXT

This study assesses if, and how, existing methods for economic evaluation are applicable to the evaluation of personalized medicine (PM) and, if not, where extension to methods may be required.

METHODS

A structured workshop was held with a predefined group of experts (n = 47), and was run using a modified nominal group technique. Workshop findings were recorded using extensive note taking, and summarized using thematic data analysis. The workshop was complemented by structured literature searches.

RESULTS

The key finding emerging from the workshop, using an economic perspective, was that two distinct, but linked, interpretations of the concept of PM exist (personalization by 'physiology' or 'preferences'). These interpretations involve specific challenges for the design and conduct of economic evaluations. Existing evaluative (extra-welfarist) frameworks were generally considered appropriate for evaluating PM. When 'personalization' is viewed as using physiological biomarkers, challenges include representing complex care pathways; representing spillover effects; meeting data requirements such as evidence on heterogeneity; and choosing appropriate time horizons for the value of further research in uncertainty analysis. When viewed as tailoring medicine to patient preferences, further work is needed regarding revealed preferences, e.g. treatment (non)adherence; stated preferences, e.g. risk interpretation and attitude; consideration of heterogeneity in preferences; and the appropriate framework (welfarism vs. extra-welfarism) to incorporate non-health benefits.

CONCLUSIONS

Ideally, economic evaluations should take account of both interpretations of PM and consider physiology and preferences. It is important for decision makers to be cognizant of the issues involved with the economic evaluation of PM to appropriately interpret the evidence and target future research funding.

Decision-analytic modeling studies: An overview for clinicians using multiple myeloma as an example

PURPOSE

The purpose of this study was to provide a clinician-friendly overview of decision-analytic models evaluating different treatment strategies for multiple myeloma (MM).

METHODS

We performed a systematic literature search to identify studies evaluating MM treatment strategies using mathematical decision-analytic models. We included studies that were published as full-text articles in English, and assessed relevant clinical endpoints, and summarized methodological characteristics (e.g., modeling approaches, simulation techniques, health outcomes, perspectives).

RESULTS

Eleven decision-analytic modeling studies met our inclusion criteria. Five different modeling approaches were adopted: decision-tree modeling, Markov state-transition modeling, discrete event simulation, partitioned-survival analysis and area-under-the-curve modeling. Health outcomes included survival, number-needed-to-treat, life expectancy, and quality-adjusted life years. Evaluated treatment strategies included novel agent-based combination therapies, stem cell transplantation and supportive measures.

CONCLUSION

Overall, our review provides a comprehensive summary of modeling studies assessing treatment of MM and highlights decision-analytic modeling as an important tool for health policy decision making.

HTA agencies facing model biases: the case of type 2 diabetes

When evaluating new drugs or treatments eligible for reimbursement, health technology assessment (HTA) agencies are repeatedly faced with cost-effectiveness analyses that evidence lack of adequate data and modeling biases. The case of type 2 diabetes illustrates this difficulty. In spite of its high disease burden, type 2 diabetes is poorly documented through existing cost-effectiveness analyses. We support this statement by an exhaustive literature review that enables us to precisely pinpoint the limitations of models used for the assessment of newly marketed (and expensive) drugs. We find that models are mostly restricted to surrogate endpoints and based on non-inferiority clinical trial data; they also show biases in the choice of comparators and inclusion criteria. Such limitations undermine the scope and applicability of HTA practice guidelines based on cost-effectiveness evidence. Nevertheless, cost-effectiveness models remain an opportunity to better inform decision makers and to reduce the uncertainty surrounding their decisions. HTA agencies are best placed to provide incentives for companies to improve the quality of the cost-effectiveness studies submitted for pricing and reimbursement decisions. One such incentive is to include stages of discussion between the company and the health authority during the evaluation process.