Using value of information analysis to inform publicly funded research priorities

An Economic Model for Estimating Trial Costs with an Application to Placebo Surgery Trials

BACKGROUND AND OBJECTIVE

Waste in clinical trials remains rife. We developed an economic model to predict the cost of trials based on input costs, duration, power, number of sites, recruitment eligibility and consenting rates.

METHODS

We parameterised the model for three proxy placebo-controlled surgical trials using data from a systematic review, a bespoke cost survey, and from the literature. We used the model to compare target and actual trial performance for (i) a trial that was completed on time but with more sites, (ii) a trial that completed after a time extension, and (iii) an incomplete trial.

RESULTS

Successful trials more accurately anticipated the true recruitment rate that they achieved and those that overestimated this were most likely to fail. The costs of overestimating recruitment rates were dramatic: all proxy trials had significantly higher costs than planned, with additional funding of at least AUD$600,000 (50% above budget) required for trials that completed after adding more sites or more time, and over AUD$2 million (260% above budget) for incomplete trials.

CONCLUSIONS

This model shows the trade-offs between time and cost, or both, when recruitment is lower than anticipated. Greater consideration is needed to improve trial planning, reviewing, and funding of these trials to avoid costly overruns and incomplete trials.

Optimal cholesterol treatment plans and genetic testing strategies for cardiovascular diseases

Atherosclerotic cardiovascular disease (ASCVD) is among the leading causes of death in the US. Although research has shown that ASCVD has genetic elements, the understanding of how genetic testing influences its prevention and treatment has been limited. To this end, we model the health trajectory of patients stochastically and determine treatment and testing decisions simultaneously. Since the cholesterol level of patients is one controllable risk factor for ASCVD events, we model cholesterol treatment plans as Markov decision processes. We determine whether and when patients should receive a genetic test using value of information analysis. By simulating the health trajectory of over 64 million adult patients, we find that 6.73 million patients undergo genetic testing. The optimal treatment plans informed with clinical and genetic information save 5,487 more quality-adjusted life-years while costing $1.18 billion less than the optimal treatment plans informed with clinical information only. As precision medicine becomes increasingly important, understanding the impact of genetic information becomes essential.

Economic evaluation of a web-based menu planning intervention to improve childcare service adherence with dietary guidelines

Background

Despite the known benefits of healthy eating in childhood, few Australian childcare services provide food that is consistent with dietary guidelines. The effectiveness of a web-based menu planning intervention to increase childcare service provision of healthy foods and decrease provision of discretionary foods in long day-care services in Australia was assessed in a randomised controlled trial. Here we consider the costs, consequences, cost-effectiveness and budget impact of the intervention using data collected within the trial.

Methods

The prospective trial-based economic evaluation involved 54 childcare services across New South Wales (NSW), Australia. Services were randomised to a 12-month intervention or usual care. The intervention involved access to a web-based menu planning and decision support tool and online resources. Effectiveness measures included mean number of food groups, overall menu and individual food group compliance with dietary guidelines, and mean servings of food groups at 12 months. Costs (reported in $AUD, 2017/18) were evaluated from both health sector and societal perspectives. The direct cost to support uptake of the intervention was calculated, as were costs to each childcare service. The incremental cost of the intervention was calculated as the net difference in the cost to undertake menu planning and review plus the direct cost of the intervention. Incremental cost-effectiveness ratios (ICERs) including uncertainty intervals were estimated for differences in costs and effects between intervention and control groups. A relative value index was calculated to determine overall value for money.

Results

Over the 12 months of the trial, we calculated a difference in cost between usual practice and intervention groups of − $482 (95% UI − $859, − $56). While the measured increase in menu and food group compliance within the trial did not reach statistical significance, there were significant improvements in mean servings of fruit and discretionary food, represented in the cost-consequence analysis. The calculated relative value index of 1.1 suggests that the intervention returns acceptable value for money for the outcomes generated.

Conclusion

Compared to usual practice, web-based programmes may offer an efficient and sustainable alternative for childcare services to improve the provision of healthy foods to children in their care.

Trial registration

Australian New Zealand Clinical Trials Registry ACTRN12616000974404

Integrating value of research into NCI Clinical Trials Cooperative Group research review and prioritization: A pilot study

Background

The Institute of Medicine has called for approaches to help maximize the return on investments (ROI) in cancer clinical trials. Value of Research (VOR) is a health economics technique that estimates ROI and can inform research prioritization. Our objective was to evaluate the impact of using VOR analyses on the clinical trial proposal review process within the SWOG cancer clinical trials consortium.

Methods

We used a previously developed minimal modeling approach to calculate VOR estimates for 9 phase II/III SWOG proposals between February 2015 and December 2016. Estimates were presented to executive committee (EC) members (N = 12) who determine which studies are sent to the National Cancer Institute for funding consideration. EC members scored proposals from 1 (best) to 5 based on scientific merit and potential impact before and after receiving VOR estimates. EC members were surveyed to assess research priorities, proposal evaluation process satisfaction, and the VOR process.

Results

Value of Research estimates ranged from −$2.1B to $16.46B per proposal. Following review of VOR results, the EC changed their score for eight of nine proposals. Proposal rankings were different in pre‐ vs postscores (P value: 0.03). Respondents had mixed views of the ultimate utility of VOR for their decisions with most supporting (42%) or neutral (41%) to the idea of adding VOR to the evaluation process.

Conclusions

The findings from this pilot study indicate use of VOR analyses may be a useful adjunct to inform proposal reviews within NCI Cooperative Clinical Trials groups.

Using HTA and guideline development as a tool for research priority setting the NICE way: reducing research waste by identifying the right research to fund

BACKGROUND

The National Institute for Health and Care Excellence (NICE) was established in 1999 and provides national guidance and advice to improve health and social care. Several steps in the research cycle have been identified that can support the reduction of waste that occurs in biomedical research. The first step in the process is ensuring appropriate research priority setting occurs so only the questions that are needed to fill existing gaps in the evidence are funded. This paper summarises the research priority setting processes at NICE.

METHODS

NICE uses its guidance production processes to identify and prioritise research questions through systematic reviews, economic analyses and stakeholder consultations and then highlights those priorities by engagement with the research community. NICE also highlights its methodological areas for research to ensure the appropriate development and growth of the evidence landscape.

RESULTS

NICE has prioritised research questions through its guidance production and methodological work and has successfully had several research products funded through the National Institute for Health Research and Medical Research Council. This paper summarises those activities and results.

CONCLUSIONS

This activity of NICE therefore reduces research waste by ensuring that the research it recommends has been systematically prioritised through evidence reviews and stakeholder input.

Quantitative benefit-harm assessment for setting research priorities: the example of roflumilast for patients with COPD

BACKGROUND

When faced with uncertainties about the effects of medical interventions regulatory agencies, guideline developers, clinicians, and researchers commonly ask for more research, and in particular for more randomized trials. The conduct of additional randomized trials is, however, sometimes not the most efficient way to reduce uncertainty. Instead, approaches such as value of information analysis or other approaches should be used to prioritize research that will most likely reduce uncertainty and inform decisions.

DISCUSSION

In situations where additional research for specific interventions needs to be prioritized, we propose the use of quantitative benefit-harm assessments that illustrate how the benefit-harm balance may change as a consequence of additional research. The example of roflumilast for patients with chronic obstructive pulmonary disease shows that additional research on patient preferences (e.g., how important are exacerbations relative to psychiatric harms?) or outcome risks (e.g., what is the incidence of psychiatric outcomes in patients with chronic obstructive pulmonary disease without treatment?) is sometimes more valuable than additional randomized trials. We propose that quantitative benefit-harm assessments have the potential to explore the impact of additional research and to identify research priorities Our approach may be seen as another type of value of information analysis and as a useful approach to stimulate specific new research that has the potential to change current estimates of the benefit-harm balance and decision making.

The value of heterogeneity for cost-effectiveness subgroup analysis: conceptual framework and application

This article develops a general framework to guide the use of subgroup cost-effectiveness analysis for decision making in a collectively funded health system. In doing so, it addresses 2 key policy questions, namely, the identification and selection of subgroups, while distinguishing 2 sources of potential value associated with heterogeneity. These are 1) the value of revealing the factors associated with heterogeneity in costs and outcomes using existing evidence (static value) and 2) the value of acquiring further subgroup-related evidence to resolve the uncertainty given the current understanding of heterogeneity (dynamic value). Consideration of these 2 sources of value can guide subgroup-specific treatment decisions and inform whether further research should be conducted to resolve uncertainty to explain variability in costs and outcomes. We apply the proposed methods to a cost-effectiveness analysis for the management of patients with acute coronary syndrome. This study presents the expected net benefits under current and perfect information when subgroups are defined based on the use and combination of 6 binary covariates. The results of the case study confirm the theoretical expectations. As more subgroups are considered, the marginal net benefit gains obtained under the current information show diminishing marginal returns, and the expected value of perfect information shows a decreasing trend. We present a suggested algorithm that synthesizes the results to guide policy.

Cost-effectiveness analysis: a proposal of new reporting standards in statistical analysis

Cost-effectiveness analysis (CEA) is a method for evaluating the outcomes and costs of competing strategies designed to improve health, and has been applied to a variety of different scientific fields. Yet there are inherent complexities in cost estimation and CEA from statistical perspectives (e.g., skewness, bidimensionality, and censoring). The incremental cost-effectiveness ratio that represents the additional cost per unit of outcome gained by a new strategy has served as the most widely accepted methodology in the CEA. In this article, we call for expanded perspectives and reporting standards reflecting a more comprehensive analysis that can elucidate different aspects of available data. Specifically, we propose that mean- and median-based incremental cost-effectiveness ratios and average cost-effectiveness ratios be reported together, along with relevant summary and inferential statistics, as complementary measures for informed decision making.

Value of information: interim analysis of a randomized, controlled trial of goal-directed hemodynamic treatment for aged patients

BACKGROUND

A randomized, controlled trial, intended to include 460 patients, is currently studying peroperative goal-directed hemodynamic treatment (GDHT) of aged hip-fracture patients. Interim efficacy analysis performed on the first 100 patients was statistically uncertain; thus, the trial is continuing in accordance with the trial protocol. This raised the present investigation's main question: Is it reasonable to continue to fund the trial to decrease uncertainty? To answer this question, a previously developed probabilistic cost-effectiveness model was used. That model depicts (1) a choice between routine fluid treatment and GDHT, given uncertainty of current evidence and (2) the monetary value of further data collection to decrease uncertainty. This monetary value, that is, the expected value of perfect information (EVPI), could be used to compare future research costs. Thus, the primary aim of the present investigation was to analyze EVPI of an ongoing trial with interim efficacy observed.

METHODS

A previously developed probabilistic decision analytic cost-effectiveness model was employed to compare the routine fluid treatment to GDHT. Results from the interim analysis, published trials, the meta-analysis, and the registry data were used as model inputs. EVPI was predicted using (1) combined uncertainty of model inputs; (2) threshold value of society's willingness to pay for one, quality-adjusted life-year; and (3) estimated number of future patients exposed to choice between GDHT and routine fluid treatment during the expected lifetime of GDHT.

RESULTS

If a decision to use GDHT were based on cost-effectiveness, then the decision would have a substantial degree of uncertainty. Assuming a 5-year lifetime of GDHT in clinical practice, the number of patients who would be subject to future decisions was 30,400. EVPI per patient would be €204 at a €20,000 threshold value of society's willingness to pay for one quality-adjusted life-year. Given a future population of 30,400 individuals, total EVPI would be €6.19 million.

CONCLUSIONS

If future trial costs are below EVPI, further data collection is potentially cost-effective. When applying a cost-effectiveness model, statements such as 'further research is needed' are replaced with 'further research is cost-effective and 'further funding of a trial is justified'.

TRIAL REGISTRATION

ClinicalTrials.gov NCT01141894.

A systematic review and methodological evaluation of published cost-effectiveness analyses of aromatase inhibitors versus tamoxifen in early stage breast cancer

BACKGROUND

A key priority in developing policies for providing affordable cancer care is measuring the value for money of new therapies using cost-effectiveness analyses (CEAs). For CEA to be useful it should focus on relevant outcomes and include thorough investigation of uncertainty. Randomized controlled trials (RCTs) of five years of aromatase inhibitors (AI) versus five years of tamoxifen in the treatment of post-menopausal women with early stage breast cancer, show benefit of AI in terms of disease free survival (DFS) but not overall survival (OS) and indicate higher risk of fracture with AI. Policy-relevant CEA of AI versus tamoxifen should focus on OS and include analysis of uncertainty over key assumptions.

METHODS

We conducted a systematic review of published CEAs comparing an AI to tamoxifen. We searched Ovid MEDLINE, EMBASE, PsychINFO, and the Cochrane Database of Systematic Reviews without language restrictions. We selected CEAs with outcomes expressed as cost per life year or cost per quality adjusted life year (QALY). We assessed quality using the Neumann checklist. Using structured forms two abstractors collected descriptive information, sources of data, baseline assumptions on effectiveness and adverse events, and recorded approaches to assessing parameter uncertainty, methodological uncertainty, and structural uncertainty.

RESULTS

We identified 1,622 citations and 18 studies met inclusion criteria. All CE estimates assumed a survival benefit for aromatase inhibitors. Twelve studies performed sensitivity analysis on the risk of adverse events and 7 assumed no additional mortality risk with any adverse event. Sub-group analysis was limited; 6 studies examined older women, 2 examined women with low recurrence risk, and 1 examined women with multiple comorbidities.

CONCLUSION

Published CEAs comparing AIs to tamoxifen assumed an OS benefit though none has been shown in RCTs, leading to an overestimate of the cost-effectiveness of AIs. Results of these CEA analyses may be suboptimal for guiding policy.

A systematic and critical review of the evolving methods and applications of value of information in academia and practice

OBJECTIVE

This article provides a systematic and critical review of the evolving methods and applications of value of information (VOI) in academia and practice and discusses where future research needs to be directed.

METHODS

Published VOI studies were identified by conducting a computerized search on Scopus and ISI Web of Science from 1980 until December 2011 using pre-specified search terms. Only full-text papers that outlined and discussed VOI methods for medical decision making, and studies that applied VOI and explicitly discussed the results with a view to informing healthcare decision makers, were included. The included papers were divided into methodological and applied papers, based on the aim of the study.

RESULTS

A total of 118 papers were included of which 50 % (n = 59) are methodological. A rapidly accumulating literature base on VOI from 1999 onwards for methodological papers and from 2005 onwards for applied papers is observed. Expected value of sample information (EVSI) is the preferred method of VOI to inform decision making regarding specific future studies, but real-life applications of EVSI remain scarce. Methodological challenges to VOI are numerous and include the high computational demands, dealing with non-linear models and interdependency between parameters, estimations of effective time horizons and patient populations, and structural uncertainties.

CONCLUSION

VOI analysis receives increasing attention in both the methodological and the applied literature bases, but challenges to applying VOI in real-life decision making remain. For many technical and methodological challenges to VOI analytic solutions have been proposed in the literature, including leaner methods for VOI. Further research should also focus on the needs of decision makers regarding VOI.

Systematic review of the impact of urinary tract infections on health-related quality of life

What's known on the subject? and What does the study add? Values for equivalent health states can vary substantially depending on the measure used and method of valuation; this has a direct impact on the results of economic analyses. To date, the majority of existing economic evaluations that include UTI as a health state refer to an analysis in which the Index of Well Being was used to estimate the quality of life experienced by young women with UTIs. Currently, there are no validated methods or filters for systematically searching for the type of generic quality of life data required for decision analytic models. This study is the only systematic review of quality of life in people with UTI in the literature. Twelve studies were identified which report quality of life using a variety of generic methods; the results of these papers were summarized in a way that is useful for a health researcher seeking to populate a decision model, design a clinical study or assess the effect of UTI on quality of life relative to other conditions. One research group provided previously unpublished data from a large cohort study; these scores were mapped to EuroQol 5-Dimension values using published algorithms and probabilistic simulations. The aim of this review was to identify studies that have evaluated the impact of symptomatic urinary tract infection (UTI) and UTI-associated bacteraemia on quality of life, and to summarize these data in a way that is useful for a health researcher seeking to populate a cost-utility model, design a clinical study or assess the effect of UTIs on quality of life relative to other conditions. We conducted a systematic search of the literature using MEDLINE, EMBASE, the NHS Economic Evaluations database, Health Technology Assessment database, Health Economics Evaluations database, Cost-Effectiveness Analysis Registry and EuroQol website. Studies that reported utility values for symptomatic UTI or UTI-associated bacteraemia derived from a generic QoL measurement tool or expert opinion were included. Studies using disease-specific instruments were excluded. Twelve studies were identified that included a generic measure of health-related quality of life for patients with UTIs. These measures included: the short-form (SF)-36 and SF-12 questionnaires; the Health Utilities Index Mark 2; Quality of Well Being; the Index of Well Being, standard gamble; the Health and Activity Limitation Index; and expert opinion. The authors of studies using either of the SF questionnaires were contacted for additional data. One research group provided previously unpublished data from a large cohort study; these scores were mapped to EuroQol 5-Dimension (EQ-5D) values using published algorithms and probabilistic simulations. The present review provides health researchers with several sources from which to select utility values to populate cost-utility models. It also shows that very few studies have measured quality of life in patients with UTI using generic preference-based measures of health and none have evaluated the impact of this health state on quality of life in children. Future studies ought to consider the inclusion of commonly used preference-based measures of health, such as the EQ-5D, in all patient populations experiencing symptomatic UTI or UTI-related complications.

Developing clinical practice guidelines: types of evidence and outcomes; values and economics, synthesis, grading, and presentation and deriving recommendations

Clinical practice guidelines are one of the foundations of efforts to improve healthcare. In 1999, we authored a paper about methods to develop guidelines. Since it was published, the methods of guideline development have progressed both in terms of methods and necessary procedures and the context for guideline development has changed with the emergence of guideline clearinghouses and large scale guideline production organisations (such as the UK National Institute for Health and Clinical Excellence). It therefore seems timely to, in a series of three articles, update and extend our earlier paper. In this second paper, we discuss issues of identifying and synthesizing evidence: deciding what type of evidence and outcomes to include in guidelines; integrating values into a guideline; incorporating economic considerations; synthesis, grading, and presentation of evidence; and moving from evidence to recommendations.

Public health and prevention in Europe: is it cost-effective?

Objectives

In the public debate surrounding public health and prevention, it is sometimes assumed that preventive interventions are by definition cost-effective. This paper aims to explore whether preventive pharmaceutical interventions are more cost-effective than a curative approach to diseases.

Methods

A descriptive study identified European economic evaluations in the Tufts Medical Center Cost-Effectiveness Analysis Registry between 2000 and 2007. Data were extracted on publication year, target population, intervention, patient sample, disease, prevention stage and incremental cost-utility ratio of each economic evaluation. Preventive interventions were defined as measures preventing disease onset. Curative interventions related to measures identifying patients with risk factors or preclinical disease or interventions limiting disability after harm has occurred. Results were expressed in terms of costs (valued in Euros at 2008 prices) per quality-adjusted life year. The association between incremental cost-utility ratios and prevention stage was examined by means of the Mann–Whitney U test.

Key findings

The analysis included 231 studies that reported information about 608 incremental cost-effectiveness ratios. Preventive interventions included interventions that were more effective and less expensive than comparators (41% of incremental cost-effectiveness ratios), and interventions that improved outcomes and increased costs (59%). Both preventive and curative interventions covered the full range of cost-effectiveness results. However, preventive interventions had a significantly lower median ratio of €6255 per quality-adjusted life year and were thus more cost-effective than curative interventions (€12 917 per quality-adjusted life year) (P=0.002).

Conclusions

Although the cost-effectiveness of preventive interventions varies substantially, preventive interventions tended to be more cost-effective than curative interventions.

Statin therapy in rheumatoid arthritis: a cost-effectiveness and value-of-information analysis

HMG-CoA reductase inhibitors (statins) are potentially excellent candidate agents for patients with rheumatoid arthritis (RA). They reduce both cardiovascular risks and RA disease activity. To evaluate the potential long-term effects of statin therapy among patients with RA, and to determine their associated cost effectiveness by incorporating both the cardiovascular and the anti-rheumatic benefits. A Markov decision-analytic model was developed to simulate cardiovascular and RA disease profiles over time. The impact of statin therapy was estimated by adjusting the risk of coronary heart disease (CHD) events and changes in the RA Disease Activity Score (DAS28), based on the results of a randomized trial. The benefits (QALYs) and costs (in year 2005 values) were evaluated from a US payer perspective. A full uncertainty analysis, including a value-of-information (VOI) analysis, was undertaken to evaluate the importance of individual parameters. Using a 10-year time horizon, the additional cost and QALYs of statin therapy were estimated to be USD4690 and 0.44 QALYs, respectively, resulting in an incremental cost-effectiveness ratio (ICER) of USD10 650 per QALY (95% CI 1525, 156 565). The QALY gain associated with statin therapy depended more on the anti-rheumatic effects of statin therapy than on its cardiovascular prevention effect. The VOI analysis found the long-term benefit of statin therapy (i.e. >or=12 months) and the consequent impact on quality of life to be the most uncertain and, therefore, influential parameters. Our analysis indicates that the dual anti-inflammatory/cardiovascular benefits of statins could make this therapy highly cost effective in RA. However, uncertainties remain in the available data, warranting further research on refining the precise RA disease-activity benefits and health-utility changes associated with statin therapy, at least over a 12-month period.

Considerations of net present value in policy making regarding diagnostic and therapeutic technologies

BACKGROUND

The pharmaceutical and medical device industries function in a business environment in which shareholders expect companies to optimize profit within legal and ethical standards. A fundamental tool used to optimize decision making is the net present value calculation, which estimates the current value of cash flows relating to an investment.

METHODS

We examined 3 prototypical research investment decisions that have been the source of public scrutiny to illustrate how policy decisions can be better understood when their impact on societally desirable investments by industry are viewed from the standpoint of their impact on net present value.

RESULTS

In the case of direct, comparative clinical trials, a simple net present value calculation provides insight into why companies eschew such investments. In the case of pediatric clinical trials, the Pediatric Extension Rule changed the net present value calculation from unattractive to potentially very attractive by allowing patent extensions; thus, the dramatic increase in pediatric clinical trials can be explained by the financial return on investment. In the case of products for small markets, the fixed costs of development make this option financially unattractive.

CONCLUSIONS

Policy decisions can be better understood when their impact on societally desirable investments by the pharmaceutical and medical device industries are viewed from the standpoint of their impact on net present value.

The value of implementation and the value of information: combined and uneven development

AIM

In a budget-constrained health care system, the decision to invest in strategies to improve the implementation of cost-effective technologies must be made alongside decisions regarding investment in the technologies themselves and investment in further research. This article presents a single, unified framework that simultaneously addresses the problem of allocating funds between these separate but linked activities.

METHODS

The framework presents a simple 4-state world where both information and implementation can be either at the current level or "perfect.'' Through this framework, it is possible to determine the maximum return to further research and an upper bound on the value of adopting implementation strategies. The framework is illustrated through case studies of health care technologies selected from those previously considered by the UK National Institute for Health and Clinical Excellence (NICE).

RESULTS

Through the case studies, several key factors that influence the expected values of perfect information and perfect implementation are identified. These factors include the maximum acceptable cost-effectiveness ratio, the level of uncertainty surrounding the adoption decision, the expected net benefits associated with the technologies, the current level of implementation, and the size of the eligible population.

CONCLUSIONS

Previous methods for valuing implementation strategies have not distinguished the value of efficacy research and the value of strategies to change the level of implementation. This framework demonstrates that the value of information and the value of implementation can be examined separately but simultaneously in a single framework. This can usefully inform policy decisions about investment in health care services, further research, and adopting implementation strategies that are likely to differ between technologies.

[Public access defibrillation. Limited use by trained first responders and laymen]

As ventricular fibrillation is the most frequent initial heart rhythm causing out-of-hospital sudden cardiac arrest, defibrillation is of essential significance. Automated external defibrillators (AEDs) have been available for some years and as a result defibrillation can be carried out by individuals other than physicians and healthcare providers such as trained first responders and untrained lay rescuers. This so-called public access defibrillation nourished hope of progress in the treatment of sudden cardiac arrest. However, several limitations exist, such as low frequency of sudden cardiac arrest in public, rare use of publicly placed AEDs, low cost effectiveness, legal requirements and insufficient public willingness to help. Due to these restrictions of public access defibrillation other measures are more promising than the attempt at general distribution of AEDs. These measures are primary or secondary prophylaxis of sudden cardiac arrest, general knowledge of adequate activation of emergency medical services, implementation of first responder teams equipped with AEDs and particularly a better education in and application of the well-established principles of cardiopulmonary resuscitation.

The application of special technologies in diagnostic anatomic pathology: is it consistent with the principles of evidence-based medicine?

Proponents of evidence-based medicine (EBM) have emphasized the need to consider the quality of different sources of medical information and have proposed various methods to integrate available "best evidence" into rules, guidelines and other diagnostic, therapeutic and prognostic models. The various factors that can affect the internal validity of studies in anatomic pathology, such as interobserver variability, use of retrospective rather than prospective data and others, are reviewed. The need for testing for the external validity of the results of anatomic pathology studies is introduced, using "test sets" of cases that have not been used to generate the classification or prognostic models. This methodology has been seldom used in anatomic pathology to validate the generalizability of various "entities," usefulness of diagnostic tests under different conditions and other information. Basic concepts of meta-analysis for research synthesis are introduced; these methods have been seldom used in anatomic pathology to integrate information from different studies using quantitative techniques rather than summary tables that merely list the results of various publications. The potential use of decision analysis and value of information analysis for the adoption of new tests is briefly discussed.