Development and Evaluation of an Approach to Using Value of Information Analyses for Real-Time Prioritization Decisions Within SWOG, a Large Cancer Clinical Trials Cooperative Group

Rapid Assessment of the Need for Evidence: Applying the Principles of Value of Information to Research Prioritisation

We propose a short-cut heuristic approach to rapidly estimate value of information (VOI) using information commonly reported in a research funding application to make a case for the need for further evaluative research. We develop a "Rapid VOI" approach, which focuses on uncertainty in the primary outcome of clinical effectiveness and uses this to explore the health consequences of decision uncertainty. We develop a freely accessible online tool, Rapid Assessment of the Need for Evidence (RANE), to allow for the efficient computation of the value of research. As a case study, the method was applied to a proposal for research on shoulder pain rehabilitation. The analysis was included as part of a successful application for research funding to the UK National Institute for Health and Care Research. Our approach enables research funders and applicants to rapidly estimate the value of proposed research. Rapid VOI relies on information that is readily available and reported in research funding applications. Rapid VOI supports research prioritisation and commissioning decisions where there is insufficient time and resources available to develop and validate complex decision-analytic models. The method provides a practical means for implementing VOI in practice, thus providing a starting point for deliberation and contributing to the transparency and accountability of research prioritisation decisions.

Value of Information for Clinical Trial Design: The Importance of Considering All Relevant Comparators

Value of Information (VOI) analyses calculate the economic value that could be generated by obtaining further information to reduce uncertainty in a health economic decision model. VOI has been suggested as a tool for research prioritisation and trial design as it can highlight economically valuable avenues for future research. Recent methodological advances have made it increasingly feasible to use VOI in practice for research; however, there are critical differences between the VOI approach and the standard methods used to design research studies such as clinical trials. We aimed to highlight key differences between the research design approach based on VOI and standard clinical trial design methods, in particular the importance of considering the full decision context. We present two hypothetical examples to demonstrate that VOI methods are only accurate when (1) all feasible comparators are included in the decision model when designing research, and (2) all comparators are retained in the decision model once the data have been collected and a final treatment recommendation is made. Omitting comparators from either the design or analysis phase of research when using VOI methods can lead to incorrect trial designs and/or treatment recommendations. Overall, we conclude that incorrectly specifying the health economic model by ignoring potential comparators can lead to misleading VOI results and potentially waste scarce research resources.

Estimating the Potential Benefits of Confirmatory Trials for Drugs with Accelerated Approval: A Comprehensive Value of Information Framework

BACKGROUND

The US Food and Drug Administration's Accelerated Approval (AA) policy provides a pathway for patients to access potentially life-saving drugs rapidly. However, the use of surrogate endpoints, single-arm designs, and small sample sizes in preliminary trials that support AAs can lead to uncertainty regarding the clinical benefits of such drugs. This study aims to develop a comprehensive value of information (VOI) framework for assessing the potential benefits of future confirmatory trials, accounting for the various uncertainties inherent in preliminary trials.

METHODS

I formulated an expected value of information from confirmatory trial (EVICT) metric, which evaluates the potential benefits of a confirmatory trial that would reduce those uncertainties by using a clinically meaningful endpoint, a randomized control, and increased sample size. The EVICT metric can quantify the expected benefits of a well-designed confirmatory trial or an inadequately designed one that continues to use surrogate endpoints or single-arm design. The framework was illustrated using a hypothetical AA drug for metastatic breast cancer.

RESULTS

The case study demonstrates that a highly uncertain preliminary trial of an AA drug was associated with a substantial EVICT. A confirmatory trial with an increased sample size for this AA drug, utilizing a clinically meaningful endpoint and randomized control, yielded a population-level EVICT of $12.6 million. Persistently using a surrogate endpoint and single-arm trial design would reduce the EVICT by 60%.

CONCLUSIONS

This framework can provide accurate VOI estimates to guide coverage policies, value-based pricing, and the design of confirmatory trials for AA drugs.

Developing and validating a multi-criteria decision analytic tool to assess the value of cancer clinical trials: evaluating cancer clinical trial value

BACKGROUND

Demonstrating safety and efficacy of new medical treatments requires clinical trials but clinical trials are costly and may not provide value proportionate to their costs. As most health systems have limited resources, it is therefore important to identify the trials with the highest value. Tools exist to assess elements of a clinical trial such as statistical validity but are not wholistic in their valuation of a clinical trial. This study aims to develop a measure of clinical trials value and provide an online tool for clinical trial prioritisation.

METHODS

A search of the academic and grey literature and stakeholder consultation was undertaken to identify a set of criteria to aid clinical trial valuation using multi-criteria decision analysis. Swing weighting and ranking exercises were used to calculate appropriate weights of each of the included criteria and to estimate the partial-value function for each underlying metric. The set of criteria and their respective weights were applied to the results of six different clinical trials to calculate their value.

RESULTS

Seven criteria were identified: 'unmet need', 'size of target population', 'eligible participants can access the trial', 'patient outcomes', 'total trial cost', 'academic impact' and 'use of trial results'. The survey had 80 complete sets of responses (51% response rate). A trial designed to address an 'Unmet Need' was most commonly ranked as the most important with a weight of 24.4%, followed by trials demonstrating improved 'Patient Outcomes' with a weight of 21.2%. The value calculated for each trial allowed for their clear delineation and thus a final value ranking for each of the six trials.

CONCLUSION

We confirmed that the use of the decision tool for valuing clinical trials is feasible and that the results are face valid based on the evaluation of six trials. A proof-of-concept applying this tool to a larger set of trials with an external validation is currently underway.

Constructing Relative Effect Priors for Research Prioritization and Trial Design: A Meta-epidemiological Analysis

BACKGROUND

Bayesian methods have potential for efficient design of randomized clinical trials (RCTs) by incorporating existing evidence. Furthermore, value of information (VOI) methods estimate the value of reducing decision uncertainty, aiding transparent research prioritization. These methods require a prior distribution describing current uncertainty in key parameters, such as relative treatment effect (RTE). However, at the time of designing and commissioning research, there may be no data to base the prior on. The aim of this article is to present methods to construct priors for RTEs based on a collection of previous RCTs.

METHODS

We developed 2 Bayesian hierarchical models that captured variability in RTE between studies within disease area accounting for study characteristics. We illustrate the methods using a data set of 743 published RCTs across 9 disease areas to obtain predictive distributions for RTEs for a range of disease areas. We illustrate how the priors from such an analysis can be used in a VOI analysis for an RCT in bladder cancer and compare the results with those using an uninformative prior.

RESULTS

For most disease areas, the predicted RTE favored new interventions over comparators. The predicted effects and uncertainty differed across the 9 disease areas. VOI analysis showed that the expected value of research is much lower with our empirically derived prior compared with an uninformative prior.

CONCLUSIONS

This study demonstrates a novel approach to generating informative priors that can be used to aid research prioritization and trial design. The methods can also be used to combine RCT evidence with expert opinion. Further work is needed to create a rich database of RCT evidence that can be used to form off-the-shelf priors.

HIGHLIGHTS

Bayesian methods have potential to aid the efficient design of randomized clinical trials (RCTs) by incorporating existing evidence. Value-of-information (VOI) methods can be used to aid research prioritization by calculating the value of current decision uncertainty.These methods require a distribution describing current uncertainty in key parameters, that is, "prior distributions."This article demonstrates a methodology to estimate prior distributions for relative treatment effects (odds and hazard ratios) estimated from a collection of previous RCTs.These results may be combined with expert elicitation to facilitate 1) value-of-information methods to prioritize research or 2) Bayesian methods for research design.

Approaches to prioritising research for clinical trial networks: a scoping review

BACKGROUND

Prioritisation of clinical trials ensures that the research conducted meets the needs of stakeholders, makes the best use of resources and avoids duplication. The aim of this review was to identify and critically appraise approaches to research prioritisation applicable to clinical trials, to inform best practice guidelines for clinical trial networks and funders.

METHODS

A scoping review of English-language published literature and research organisation websites (January 2000 to January 2020) was undertaken to identify primary studies, approaches and criteria for research prioritisation. Data were extracted and tabulated, and a narrative synthesis was employed.

RESULTS

Seventy-eight primary studies and 18 websites were included. The majority of research prioritisation occurred in oncology and neurology disciplines. The main reasons for prioritisation were to address a knowledge gap (51 of 78 studies [65%]) and to define patient-important topics (28 studies, [35%]). In addition, research organisations prioritised in order to support their institution's mission, invest strategically, and identify best return on investment. Fifty-seven of 78 (73%) studies used interpretative prioritisation approaches (including Delphi surveys, James Lind Alliance and consensus workshops); six studies used quantitative approaches (8%) such as prospective payback or value of information (VOI) analyses; and 14 studies used blended approaches (18%) such as nominal group technique and Child Health Nutritional Research Initiative. Main criteria for prioritisation included relevance, appropriateness, significance, feasibility and cost-effectiveness.

CONCLUSION

Current research prioritisation approaches for groups conducting and funding clinical trials are largely interpretative. There is an opportunity to improve the transparency of prioritisation through the inclusion of quantitative approaches.

Implementation Barriers to Value of Information Analysis in Health Technology Decision Making: Results From a Process Evaluation

OBJECTIVES

Value of information (VOI) analysis can support health technology assessment decision making, but it is a long way from being standard use. The objective of this study was to understand barriers to the implementation of VOI analysis and propose actions to overcome these.

METHODS

We performed a process evaluation of VOI analysis use within decision making on tomosynthesis versus digital mammography for use in the Dutch breast cancer population screening. Based on steering committee meeting attendance and regular meetings with analysts, we developed a list of barriers to VOI use, which were analyzed using an established diffusion model. We proposed actions to address these barriers. Barriers and actions were discussed and validated in a workshop with stakeholders representing patients, clinicians, regulators, policy advisors, researchers, and the industry.

RESULTS

Consensus was reached on groups of barriers, which included characteristics of VOI analysis itself, stakeholder's attitudes, analysts' and policy makers' skills and knowledge, system readiness, and implementation in the organization. Observed barriers did not only pertain to VOI analysis itself but also to formulating the objective of the assessment, economic modeling, and broader aspects of uncertainty assessment. Actions to overcome these barriers related to organizational changes, knowledge transfer, cultural change, and tools.

CONCLUSIONS

This in-depth analysis of barriers to implementation of VOI analysis and resulting actions and tools may be useful to health technology assessment organizations that wish to implement VOI analysis in technology assessment and research prioritization. Further research should focus on application and evaluation of the proposed actions in real-world assessment processes.

Practical metrics for establishing the health benefits of research to support research prioritisation

INTRODUCTION

We present practical metrics for estimating the expected health benefits of specific research proposals. These can be used by research funders, researchers and healthcare decision-makers within low-income and middle-income countries to support evidence-based research prioritisation.

METHODS

The methods require three key assessments: (1) the current level of uncertainty around the endpoints the proposed study will measure; (2) how uncertainty impacts on the health benefits and costs of healthcare programmes and (3) the health opportunity costs imposed by programme costs. Research is valuable because it can improve health by informing the choice of which programmes should be implemented. We provide a Microsoft Excel tool to allow readers to generate estimates of the health benefits of research studies based on these three assessments. The tool can be populated using existing studies, existing cost-effectiveness models and expert opinion. Where such evidence is not available, the tool can quantify the value of research under different assumptions. Estimates of the health benefits of research can be considered alongside research costs, and the consequences of delaying implementation until research reports, to determine whether research is worthwhile. We illustrate the method using a case study of research on HIV self-testing programmes in Malawi. This analysis combines data from the literature with outputs from the HIV synthesis model.

RESULTS

For this case study, we found a costing study that could be completed and inform decision making within 1 year offered the highest health benefits (67 000 disability-adjusted life years (DALYs) averted). Research on outcomes improved population health to a lesser extent (12 000 DALYs averted) and only if carried out alongside programme implementation.

CONCLUSION

Our work provides a method for estimating the health benefits of research in a practical and timely fashion. This can be used to support accountable use of research funds.

Value of Information Analytical Methods: Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force

The allocation of healthcare resources among competing priorities requires an assessment of the expected costs and health effects of investing resources in the activities and of the opportunity cost of the expenditure. To date, much effort has been devoted to assessing the expected costs and health effects, but there remains an important need to also reflect the consequences of uncertainty in resource allocation decisions and the value of further research to reduce uncertainty. Decision making with uncertainty may turn out to be suboptimal, resulting in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to better inform resource decisions. This value can be quantified using value of information (VOI) analysis. This report from the ISPOR VOI Task Force describes methods for computing 4 VOI measures: the expected value of perfect information, expected value of partial perfect information (EVPPI), expected value of sample information (EVSI), and expected net benefit of sampling (ENBS). Several methods exist for computing EVPPI and EVSI, and this report provides guidance on selecting the most appropriate method based on the features of the decision problem. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing VOI analyses. The software needed to compute VOI is discussed, and areas for future research are highlighted.

Value of Information Analysis for Research Decisions-An Introduction: Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force

Healthcare resource allocation decisions made under conditions of uncertainty may turn out to be suboptimal. In a resource constrained system in which there is a fixed budget, these suboptimal decisions will result in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to make better resource allocation decisions. This value can be quantified using a value of information (VOI) analysis. This report, from the ISPOR VOI Task Force, introduces VOI analysis, defines key concepts and terminology, and outlines the role of VOI for supporting decision making, including the steps involved in undertaking and interpreting VOI analyses. The report is specifically aimed at those tasked with making decisions about the adoption of healthcare or the funding of healthcare research. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing the results of VOI analyses.

Influence of Modeling Choices on Value of Information Analysis: An Empirical Analysis from a Real-World Experiment

BACKGROUND

Value of information (VOI) analysis often requires modeling to characterize and propagate uncertainty. In collaboration with a cancer clinical trial group, we integrated a VOI approach to assessing trial proposals.

OBJECTIVE

This paper aims to explore the impact of modeling choices on VOI results and to share lessons learned from the experience.

METHODS

After selecting two proposals (A: phase III, breast cancer; B: phase II, pancreatic cancer) for in-depth evaluations, we categorized key modeling choices relevant to trial decision makers (characterizing uncertainty of efficacy, evidence thresholds to change clinical practice, and sample size) and modelers (cycle length, survival distribution, simulation runs, and other choices). Using a $150,000 per quality-adjusted life-year (QALY) threshold, we calculated the patient-level expected value of sample information (EVSI) for each proposal and examined whether each modeling choice led to relative change of more than 10% from the averaged base-case estimate. We separately analyzed the impact of the effective time horizon.

RESULTS

The base-case EVSI was $118,300 for Proposal A and $22,200 for Proposal B per patient. Characterizing uncertainty of efficacy was the most important choice in both proposals (e.g. Proposal A: $118,300 using historical data vs. $348,300 using expert survey), followed by the sample size and the choice of survival distribution. The assumed effective time horizon also had a substantial impact on the population-level EVSI.

CONCLUSIONS

Modeling choices can have a substantial impact on VOI. Therefore, it is important for groups working to incorporate VOI into research prioritization to adhere to best practices, be clear in their reporting and justification for modeling choices, and to work closely with the relevant decision makers, with particular attention to modeling choices.

A Framework to Prioritise Health Research Proposals for Funding: Integrating Value for Money

When making funding decisions, research organisations largely consider the merits (e.g. scientific rigour and feasibility) of submitted research proposals; yet, there is often little or no reference to their value for money. This may be attributed to the challenges of assessing and integrating value of research into existing research prioritisation processes. We propose a framework that considers both the merits of research and its value for money to guide health research funding decisions. A practical framework is developed based on current processes followed by funding organizations for assessing investigator-initiated research proposals, and analytical methods for evaluating the expected value of research. We apply the analytical methods to estimate the expected return on investment of two real-world grant applications. The framework comprises four sequential steps: (1) initial screening of applications for eligibility and completeness; (2) merit assessment of eligible proposals; (3) estimating the expected value of research for the shortlisted proposals that pass the first two steps and ranking of proposals based on return on investment; and (4) selecting research proposals for funding. We demonstrate how the expected value for money can be efficiently estimated using certain information provided in funding applications. The proposed framework integrates value-for-money assessment into the existing research prioritisation processes. Considering value for money to inform research funding decisions is vital to achieve efficient utilisation of research budgets and maximise returns on research investments.

Uncertainty and Coverage With Evidence Development: Does Practice Meet Theory?

OBJECTIVES

In theory, a successful coverage with evidence development (CED) scheme is one that addresses the most important uncertainties in a given assessment. We investigated the following: (1) which uncertainties were present during the initial assessment of 3 Dutch CED cases, (2) how these uncertainties were integrated in the initial assessments, (3) whether CED research plans included the identified uncertainties, and (4) issues with managing uncertainty in CED research and ways forward from these issues.

METHODS

Three CED initial assessment dossiers were analyzed and 16 stakeholders were interviewed. Uncertainties were identified in interviews and dossiers and were categorized in different causes: unavailability, indirectness, and imprecision of evidence. Identified uncertainties could be mentioned, described, and explored. Issues and ways forward to address uncertainty in CED schemes were discussed during the interviews.

RESULTS

Forty-two uncertainties were identified. Thirteen (31%) were caused by unavailability, 17 (40%) by indirectness, and 12 (29%) by imprecision. Thirty-four uncertainties (81%) were only mentioned, 19 (45%) were described, and the impact of 3 (7%) uncertainties on the results was explored in the assessment dossiers. Seventeen uncertainties (40%) were included in the CED research plans. According to stakeholders, research did not address the identified uncertainty, but CED research should be designed to focus on these.

CONCLUSIONS

In practice, uncertainties were neither systematically nor completely identified in the analyzed CED schemes. A framework would help to systematically identify uncertainty, and this process should involve all stakeholders. Value of information analysis, and the uncertainties that are not included in this analysis should inform CED research design.

Directing research funds to the right research projects: a review of criteria used by research organisations in Australia in prioritising health research projects for funding

OBJECTIVES

Healthcare budgets are limited, and therefore, research funds should be wisely allocated to ensure high-quality, useful and cost-effective research. We aimed to critically review the criteria considered by major Australian organisations in prioritising and selecting health research projects for funding.

METHODS

We reviewed all grant schemes listed on the Australian Competitive Grants Register that were health-related, active in 2017 and with publicly available selection criteria on the funders' websites. Data extracted included scheme name, funding organisation, selection criteria and the relative weight assigned to each criterion. Selection criteria were grouped into five representative domains: relevance, appropriateness, significance, feasibility (including team quality) and cost-effectiveness (ie, value for money).

RESULTS

Thirty-six schemes were included from 158 identified. One-half of the schemes were under the National Health and Medical Research Council. The most commonly used criteria were research team quality and capability (94%), research plan clarity (94%), scientific quality (92%) and research impact (92%). Criteria considered less commonly were existing knowledge (22%), fostering collaboration (22%), research environment (19%), value for money (14%), disease burden (8%) and ethical/moral considerations (3%). In terms of representative domains, relevance was considered in 72% of the schemes, appropriateness in 92%, significance in 94%, feasibility in 100% and cost-effectiveness in 17%. The relative weights for the selection criteria varied across schemes with 5%-30% for relevance, 20%-60% for each appropriateness and significance, 20%-75% for feasibility and 15%-33% for cost-effectiveness.

CONCLUSIONS

In selecting research projects for funding, Australian research organisations focus largely on research appropriateness, significance and feasibility; however, value for money is most often overlooked. Research funding decisions should include an assessment of value for money in order to maximise return on research investment.

Calculating the Expected Value of Sample Information Using Efficient Nested Monte Carlo: A Tutorial

OBJECTIVE

The expected value of sample information (EVSI) quantifies the economic benefit of reducing uncertainty in a health economic model by collecting additional information. This has the potential to improve the allocation of research budgets. Despite this, practical EVSI evaluations are limited partly due to the computational cost of estimating this value using the gold-standard nested simulation methods. Recently, however, Heath et al. developed an estimation procedure that reduces the number of simulations required for this gold-standard calculation. Up to this point, this new method has been presented in purely technical terms.

STUDY DESIGN

This study presents the practical application of this new method to aid its implementation. We use a worked example to illustrate the key steps of the EVSI estimation procedure before discussing its optimal implementation using a practical health economic model.

METHODS

The worked example is based on a three-parameter linear health economic model. The more realistic model evaluates the cost-effectiveness of a new chemotherapy treatment, which aims to reduce the number of side effects experienced by patients. We use a Markov model structure to evaluate the health economic profile of experiencing side effects.

RESULTS

This EVSI estimation method offers accurate estimation within a feasible computation time, seconds compared to days, even for more complex model structures. The EVSI estimation is more accurate if a greater number of nested samples are used, even for a fixed computational cost.

CONCLUSIONS

This new method reduces the computational cost of estimating the EVSI by nested simulation.

Optimal Information Collection Policies in a Markov Decision Process Framework

BACKGROUND

The cost-effectiveness and value of additional information about a health technology or program may change over time because of trends affecting patient cohorts and/or the intervention. Delaying information collection even for parameters that do not change over time may be optimal.

METHODS

We present a stochastic dynamic programming approach to simultaneously identify the optimal intervention and information collection policies. We use our framework to evaluate birth cohort hepatitis C virus (HCV) screening. We focus on how the presence of a time-varying parameter (HCV prevalence) affects the optimal information collection policy for a parameter assumed constant across birth cohorts: liver fibrosis stage distribution for screen-detected diagnosis at age 50.

RESULTS

We prove that it may be optimal to delay information collection until a time when the information more immediately affects decision making. For the example of HCV screening, given initial beliefs, the optimal policy (at 2010) was to continue screening and collect information about the distribution of liver fibrosis at screen-detected diagnosis in 12 years, increasing the expected incremental net monetary benefit (INMB) by $169.5 million compared to current guidelines.

CONCLUSIONS

The option to delay information collection until the information is sufficiently likely to influence decisions can increase efficiency. A dynamic programming framework enables an assessment of the marginal value of information and determines the optimal policy, including when and how much information to collect.

Integrating value of research into NCI Clinical Trials Cooperative Group research review and prioritization: A pilot study

Background

The Institute of Medicine has called for approaches to help maximize the return on investments (ROI) in cancer clinical trials. Value of Research (VOR) is a health economics technique that estimates ROI and can inform research prioritization. Our objective was to evaluate the impact of using VOR analyses on the clinical trial proposal review process within the SWOG cancer clinical trials consortium.

Methods

We used a previously developed minimal modeling approach to calculate VOR estimates for 9 phase II/III SWOG proposals between February 2015 and December 2016. Estimates were presented to executive committee (EC) members (N = 12) who determine which studies are sent to the National Cancer Institute for funding consideration. EC members scored proposals from 1 (best) to 5 based on scientific merit and potential impact before and after receiving VOR estimates. EC members were surveyed to assess research priorities, proposal evaluation process satisfaction, and the VOR process.

Results

Value of Research estimates ranged from −$2.1B to $16.46B per proposal. Following review of VOR results, the EC changed their score for eight of nine proposals. Proposal rankings were different in pre‐ vs postscores (P value: 0.03). Respondents had mixed views of the ultimate utility of VOR for their decisions with most supporting (42%) or neutral (41%) to the idea of adding VOR to the evaluation process.

Conclusions

The findings from this pilot study indicate use of VOR analyses may be a useful adjunct to inform proposal reviews within NCI Cooperative Clinical Trials groups.

Are Evidence Standards Different for Genomic- vs. Clinical-Based Precision Medicine? A Quantitative Analysis of Individualized Warfarin Therapy

Evidence requirements for implementation of precision medicine (PM), whether informed by genomic or clinical data, are not well defined. Evidence requirements are driven by uncertainty and its attendant consequences; these aspects can be quantified by a novel technique in health economics: value of information analysis (VOI). We utilized VOI analysis to compare the evidence levels over time for warfarin dosing based on pharmacogenomic vs. amiodarone-warfarin drug-drug interaction information. The primary outcome was the expected value of perfect information (EVPI), which is an estimate of the upper limit of the societal value of conducting future research. Over the past decade, the EVPI for the pharmacogenomic strategy decreased from $1,550 to $140 vs. $1,220 to $280 per patient for the drug-interaction strategy. Evidence levels thus appear to be higher for pharmacogenomic-guided vs. drug-interaction-guided warfarin dosing. Clinical guidelines and reimbursement policies for warfarin PM could be informed by these findings.

Regulator Loss Functions and Hierarchical Modeling for Safety Decision Making

BACKGROUND

Regulators must act to protect the public when evidence indicates safety problems with medical devices. This requires complex tradeoffs among risks and benefits, which conventional safety surveillance methods do not incorporate.

OBJECTIVE

To combine explicit regulator loss functions with statistical evidence on medical device safety signals to improve decision making.

METHODS

In the Hospital Cost and Utilization Project National Inpatient Sample, we select pediatric inpatient admissions and identify adverse medical device events (AMDEs). We fit hierarchical Bayesian models to the annual hospital-level AMDE rates, accounting for patient and hospital characteristics. These models produce expected AMDE rates (a safety target), against which we compare the observed rates in a test year to compute a safety signal. We specify a set of loss functions that quantify the costs and benefits of each action as a function of the safety signal. We integrate the loss functions over the posterior distribution of the safety signal to obtain the posterior (Bayes) risk; the preferred action has the smallest Bayes risk. Using simulation and an analysis of AMDE data, we compare our minimum-risk decisions to a conventional Z score approach for classifying safety signals.

RESULTS

The 2 rules produced different actions for nearly half of hospitals (45%). In the simulation, decisions that minimize Bayes risk outperform Z score-based decisions, even when the loss functions or hierarchical models are misspecified.

LIMITATIONS

Our method is sensitive to the choice of loss functions; eliciting quantitative inputs to the loss functions from regulators is challenging.

CONCLUSIONS

A decision-theoretic approach to acting on safety signals is potentially promising but requires careful specification of loss functions in consultation with subject matter experts.

What is the value of conducting a trial of r-tPA for the treatment of mild stroke patients?

Background The Phase IIIb, Double-Blind, Multicenter Study to Evaluate the Efficacy and Safety of Alteplase in Patients With Mild Stroke: Rapidly Improving Symptoms and Minor Neurologic Deficits (PRISMS) trial will assess r-tPA in ischemic stroke patients who present with mild deficits (i.e. mild stroke). Aims To assess PRISMS's societal value in clarifying the optimal care for patients with mild ischemic stroke. Methods A value of information (VOI) decision model was developed to compare the outcomes of mild stroke patients treated vs. not treated with r-tPA. Model inputs were derived from a subset of Third International Stroke Trial patients, a recent meta-analysis of r-tPA trials, expert opinion, and other published sources. VOI analyses were also used to assess the expected US societal value of the PRISMS trial and the expected value of reducing uncertainty in key trial estimates. Results The expected net societal value of the PRISMS trial was approximately $210 million ($160 m-$260 m), representing a six-fold return on investment. The value of reducing uncertainty in r-tPA efficacy was approximately $150 million ($100 m-$200 m), while reducing uncertainty in r-tPA safety (increased risk for symptomatic intracranial hemorrhage) did not add additional value in comparison. Conclusions Developing a better understanding of the outcomes of r-tPA treatment in patients with mild ischemic stroke will provide tremendous societal value by clarifying current uncertainty around treatment effectiveness. Enrollment in the PRISMS trial for patients presenting with mild ischemic stroke within 0-3 h of symptom onset should be highly encouraged.