The statistical basis of public policy: a paradigm shift is overdue

Quantitative decision making for investment in global health intervention trials: Case study of the NEWBORN study on emollient therapy in preterm infants in Kenya

Background

Partners from an NGO, academia, industry and government applied a tool originating in the private sector – Quantitative Decision Making (QDM) – to rigorously assess whether to invest in testing a global health intervention. The proposed NEWBORN study was designed to assess whether topical emollient therapy with sunflower seed oil in infants with very low birthweight <1500 g in Kenya would result in a significant reduction in neonatal mortality compared to standard of care.

Methods

The QDM process consisted of prior elicitation, modelling of prior distributions, and simulations to assess Probability of Success (PoS) via assurance calculations. Expert opinion was elicited on the probability that emollient therapy with sunflower seed oil will have any measurable benefit on neonatal mortality based on available evidence. The distribution of effect sizes was modelled and trial data simulated using Statistical Analysis System to obtain the overall assurance which represents the PoS for the planned study. A decision-making framework was then applied to characterise the ability of the study to meet pre-selected decision-making endpoints.

Results

There was a 47% chance of a positive outcome (defined as a significant relative reduction in mortality of ≥15%), a 45% chance of a negative outcome (defined as a significant relative reduction in mortality <10%), and an 8% chance of ending in the consider zone (ie, a mortality reduction of 10 to <15%) for infants <1500 g.

Conclusions

QDM is a novel tool from industry which has utility for prioritisation of investments in global health, complementing existing tools [eg, Child Health and Nutrition Research Initiative]. Results from application of QDM to the NEWBORN study suggests that it has a high probability of producing clear results. Findings encourage future formation of public-private partnerships for health.

Methodological issues in economic evaluations of emergency transport systems in low-income and middle-income countries

A recent systematic review identified few papers on the economic evaluation of systems for emergency transport of acutely ill or injured patients. In addition, we found no articles dealing with the methodological challenges posed by such studies in low-income or middle-income countries. We therefore carried out an analysis of issues that are of particular salience to this important topic. This is an intellectual study in which we develop models, identify their limitations, suggest potential extensions to the models and discuss priorities for empirical studies to populate models. First, we develop a general model to calculate changes in survival contingent on the reduced time to treatment that an emergency transport system is designed to achieve. Second, we develop a model to estimate transfer times over an area that will be served by a proposed transfer system. Third, we discuss difficulties in obtaining parameters with which to populate the models. Fourth, we discuss costs, both direct and indirect, of an emergency transfer service. Fifth, we discuss the issue that outcomes other than survival should be considered and that the effects of a service are a weighted sum over all the conditions and severities for which the service caters. Lastly, based on the above work, we identify priorities for research. To our knowledge, this is the first study to identify and frame issues in the health economics of acute transfer systems and to develop models to calculate survival rates from basic parameters, such as time delay/survival relationships, that vary by intervention type and context.

Evaluation of the Use of Cancer Registry Data for Comparative Effectiveness Research

Importance

Researchers often analyze cancer registry data to assess for differences in survival among cancer treatments. However, the retrospective, nonrandomized design of these analyses raises questions about study validity.

Objective

To examine the extent to which comparative effectiveness analyses using observational cancer registry data produce results concordant with those of randomized clinical trials.

Design, Setting, and Participants

In this comparative effectiveness study, a total of 141 randomized clinical trials referenced in the National Comprehensive Cancer Network Clinical Practice Guidelines for 8 common solid tumor types were identified. Data on participants within the National Cancer Database (NCDB) diagnosed between 2004 and 2014, matching the eligibility criteria of the randomized clinical trial, were obtained. The present study was conducted from August 1, 2017, to September 10, 2019. The trials included 85 118 patients, and the corresponding NCDB analyses included 1 344 536 patients. Three Cox proportional hazards regression models were used to determine hazard ratios (HRs) for overall survival, including univariable, multivariable, and propensity score-adjusted models. Multivariable and propensity score analyses controlled for potential confounders, including demographic, comorbidity, clinical, treatment, and tumor-related variables.

Main Outcomes and Measures

The main outcome was concordance between the results of randomized clinical trials and observational cancer registry data. Hazard ratios with an NCDB analysis were considered concordant if the NDCB HR fell within the 95% CI of the randomized clinical trial HR. An NCDB analysis was considered concordant if both the NCDB and clinical trial P values for survival were nonsignificant (P ≥ .05) or if they were both significant (P < .05) with survival favoring the same treatment arm in the NCDB and in the randomized clinical trial.

Results

Analyses using the NCDB-produced HRs for survival were concordant with those of 141 randomized clinical trials in 79 univariable analyses (56%), 98 multivariable analyses (70%), and 90 propensity score models (64%). The NCDB analyses produced P values concordant with randomized clinical trials in 58 univariable analyses (41%), 65 multivariable analyses (46%), and 63 propensity score models (45%). No clinical trial characteristics were associated with concordance between NCDB analyses and randomized clinical trials, including disease site, type of clinical intervention, or severity of cancer.

Conclusions and Relevance

The findings of this study suggest that comparative effectiveness research using cancer registry data often produces survival outcomes discordant with those of randomized clinical trial data. These findings may help provide context for clinicians and policy makers interpreting observational comparative effectiveness research in oncology.

Dual Antiplatelet Therapy Continuation Beyond 1 Year After Drug-Eluting Stents

Background—

The benefits and harms of dual antiplatelet therapy (DAPT) continuation beyond 1 year after drug-eluting stent implantation as compared with 1-year DAPT remain controversial.

Methods and Results—

We searched for randomized trials that compared longer than 1-year DAPT versus 1-year DAPT after drug-eluting stenting. A meta-analysis was performed by using standard frequentist and random-effects Bayesian approaches. Four trials comprising 17 650 participants were included. Compared with 1-year DAPT, extended DAPT did not affect all-cause mortality (odds ratio [OR], 1.11; 95% confidence interval [CI], 0.79–1.5; P =0.53) or cardiovascular mortality (OR, 1.03; 95% CI, 0.72–1.46; P =0.88). Extended DAPT was associated with a reduction in the risk of myocardial infarction (OR, 0.56; 95% CI, 0.43–0.73; P <0.001), nonsignificant reductions of stent thrombosis (OR, 0.46; 95% CI, 0.16–1.27; P =0.13), similar risk of stroke (OR, 0.91; 95% CI, 0.65–1.26; P =0.56), and an increased risk of major bleeding (OR, 1.49; 95% CI, 1.06–2.11; P =0.02). By using Bayesian meta-analysis, we found moderate evidence of a reduction of myocardial infarction (OR, 0.62; 95% credible intervals, 0.39–1.05) and weak evidence of an increase in major bleeding (OR, 1.66; 95% credible intervals, 0.89–3.09) associated with extended DAPT.

Conclusions—

In this meta-analysis, extended DAPT beyond 1 year prevented myocardial infarctions and increased major bleedings, but the strength of evidence for these effects was not strong. DAPT continuation beyond 1 year showed no effects on mortality.

Public Response to Scientific Misconduct: Assessing Changes in Public Sentiment Toward the Stimulus-Triggered Acquisition of Pluripotency (STAP) Cell Case via Twitter

Background

In this age of social media, any news—good or bad—has the potential to spread in unpredictable ways. Changes in public sentiment have the potential to either drive or limit investment in publicly funded activities, such as scientific research. As a result, understanding the ways in which reported cases of scientific misconduct shape public sentiment is becoming increasingly essential—for researchers and institutions, as well as for policy makers and funders. In this study, we thus set out to assess and define the patterns according to which public sentiment may change in response to reported cases of scientific misconduct. This study focuses on the public response to the events involved in a recent case of major scientific misconduct that occurred in 2014 in Japan—stimulus-triggered acquisition of pluripotency (STAP) cell case.

Objectives

The aims of this study were to determine (1) the patterns according to which public sentiment changes in response to scientific misconduct; (2) whether such measures vary significantly, coincident with major timeline events; and (3) whether the changes observed mirror the response patterns reported in the literature with respect to other classes of events, such as entertainment news and disaster reports.

Methods

The recent STAP cell scandal is used as a test case. Changes in the volume and polarity of discussion were assessed using a sampling of case-related Twitter data, published between January 28, 2014 and March 15, 2015. Rapidminer was used for text processing and the popular bag-of-words algorithm, SentiWordNet, was used in Rapidminer to calculate sentiment for each sample Tweet. Relative volume and sentiment was then assessed overall, month-to-month, and with respect to individual entities.

Results

Despite the ostensibly negative subject, average sentiment over the observed period tended to be neutral (−0.04); however, a notable downward trend (y=−0.01 x +0.09; R ²=.45) was observed month-to-month. Notably polarized tweets accounted for less than one-third of sampled discussion: 17.49% (1656/9467) negative and 12.59% positive (1192/9467). Significant polarization was found in only 4 out of the 15 months covered, with significant variation month-to-month (P<.001). Significant increases in polarization tended to coincide with increased discussion volume surrounding major events (P<.001).

Conclusions

These results suggest that public opinion toward scientific research may be subject to the same sensationalist dynamics driving public opinion in other, consumer-oriented topics. The patterns in public response observed here, with respect to the STAP cell case, were found to be consistent with those observed in the literature with respect to other classes of news-worthy events on Twitter. Discussion was found to become strongly polarized only during times of increased public attention, and such increases tended to be driven primarily by negative reporting and reactionary commentary.

An exploratory test for an excess of significant findings

Background The published clinical research literature may be distorted by the pursuit of statistically significant results.

Purpose We aimed to develop a test to explore biases stemming from the pursuit of nominal statistical significance.

Methods The exploratory test evaluates whether there is a relative excess of formally significant findings in the published literature due to any reason (e.g., publication bias, selective analyses and outcome reporting, or fabricated data). The number of expected studies with statistically significant results is estimated and compared against the number of observed significant studies. The main application uses α = 0.05, but a range of α thresholds is also examined. Different values or prior distributions of the effect size are assumed. Given the typically low power (few studies per research question), the test may be best applied across domains of many meta-analyses that share common characteristics (interventions, outcomes, study populations, research environment).

Results We evaluated illustratively eight meta-analyses of clinical trials with >50 studies each and 10 meta-analyses of clinical efficacy for neuroleptic agents in schizophrenia; the 10 meta-analyses were also examined as a composite domain. Different results were obtained against commonly used tests of publication bias. We demonstrated a clear or possible excess of significant studies in 6 of 8 large meta-analyses and in the wide domain of neuroleptic treatments.

Limitations The proposed test is exploratory, may depend on prior assumptions, and should be applied cautiously.

Conclusions An excess of significant findings may be documented in some clinical research fields. Clinical Trials 2007; 4: 245—253; http://ctj.sagepub.com

Likelihood ratio meta-analysis: New motivation and approach for an old method

A 95% confidence interval (CI) in an updated meta-analysis may not have the expected 95% coverage. If a meta-analysis is simply updated with additional data, then the resulting 95% CI will be wrong because it will not have accounted for the fact that the earlier meta-analysis failed or succeeded to exclude the null. This situation can be avoided by using the likelihood ratio (LR) as a measure of evidence that does not depend on type-1 error. We show how an LR-based approach, first advanced by Goodman, can be used in a meta-analysis to pool data from separate studies to quantitatively assess where the total evidence points. The method works by estimating the log-likelihood ratio (LogLR) function from each study. Those functions are then summed to obtain a combined function, which is then used to retrieve the total effect estimate, and a corresponding 'intrinsic' confidence interval. Using as illustrations the CAPRIE trial of clopidogrel versus aspirin in the prevention of ischemic events, and our own meta-analysis of higher potency statins and the risk of acute kidney injury, we show that the LR-based method yields the same point estimate as the traditional analysis, but with an intrinsic confidence interval that is appropriately wider than the traditional 95% CI. The LR-based method can be used to conduct both fixed effect and random effects meta-analyses, it can be applied to old and new meta-analyses alike, and results can be presented in a format that is familiar to a meta-analytic audience.

Secular trends and evaluation of complex interventions: the rising tide phenomenon

Evaluations of service delivery interventions with contemporaneous controls often yield null results, even when the intervention appeared promising in advance. There can be many reasons for null results. In this paper we introduce the concept of a ‘rising tide’ phenomenon being a possible explanation of null results. We note that evaluations of service delivery interventions often occur when awareness of the problems they intend to address is already heightened, and pressure to tackle them is mounting throughout a health system. An evaluation may therefore take place in a setting where the system as a whole is improving – where there is a pronounced temporal trend or a ‘rising tide causing all vessels to rise’. As a consequence, control sites in an intervention study will improve. This reduces the difference between intervention and control sites and predisposes the study to a null result, leading to the conclusion that the intervention has no effect. We discuss how a rising tide may be distinguished from other causes of improvement in both control and intervention groups, and give examples where the rising tide provides a convincing explanation of such a finding. We offer recommendations for interpretation of research findings where improvements in the intervention group are matched by improvements in the control group. Understanding the rising tide phenomenon is important for a more nuanced interpretation of null results arising in the context of system-wide improvement. Recognition that a rising tide may have predisposed to a null result in one health system cautions against generalising the result to another health system where strong secular trends are absent.

Application of credibility ceilings probes the robustness of meta-analyses of biomarkers and cancer risk

OBJECTIVES

Meta-analyses of biomarkers often present spurious significant results and large effects. We applied sensitivity analyses with the use of credibility ceilings to assess whether and how the results of meta-analyses of biomarkers and cancer risk would change.

STUDY DESIGN AND SETTING

We evaluated 98 meta-analyses, 43 (44%) of which had nominally statistically significant results. We assumed that any single study cannot give more than a maximum certainty 100 - c% (c, credibility ceiling) that the effect estimate [odds ratio (OR)] exceeds 1 (null) or 1.2.

RESULTS

Nominal statistical significance was maintained for 21 (21%) meta-analyses, for c = 10% and OR >1, and these proportions changed to 7%, 3%, and 6% with ceilings of 20%, 30%, and 40%, respectively. For ceilings for OR >1.2, the respective proportions were 37%, 21%, 7%, and 3%. Seven meta-analyses on infectious agents retained statistical significance even with a high ceiling of c = 20% for OR >1.00. Meta-analyses without other hints of bias (large between-study heterogeneity, small-study effects, excess significance) were more likely to retain statistical significance than those that had such hints of bias.

CONCLUSION

Credibility ceilings may be helpful in meta-analyses of biomarkers to understand the robustness of the results to different levels of uncertainty.

Bayesian methodology for the design and interpretation of clinical trials in critical care medicine: a primer for clinicians

OBJECTIVES

To review Bayesian methodology and its utility to clinical decision making and research in the critical care field.

DATA SOURCE AND STUDY SELECTION

Clinical, epidemiological, and biostatistical studies on Bayesian methods in PubMed and Embase from their inception to December 2013.

DATA SYNTHESIS

Bayesian methods have been extensively used by a wide range of scientific fields, including astronomy, engineering, chemistry, genetics, physics, geology, paleontology, climatology, cryptography, linguistics, ecology, and computational sciences. The application of medical knowledge in clinical research is analogous to the application of medical knowledge in clinical practice. Bedside physicians have to make most diagnostic and treatment decisions on critically ill patients every day without clear-cut evidence-based medicine (more subjective than objective evidence). Similarly, clinical researchers have to make most decisions about trial design with limited available data. Bayesian methodology allows both subjective and objective aspects of knowledge to be formally measured and transparently incorporated into the design, execution, and interpretation of clinical trials. In addition, various degrees of knowledge and several hypotheses can be tested at the same time in a single clinical trial without the risk of multiplicity. Notably, the Bayesian technology is naturally suited for the interpretation of clinical trial findings for the individualized care of critically ill patients and for the optimization of public health policies.

CONCLUSIONS

We propose that the application of the versatile Bayesian methodology in conjunction with the conventional statistical methods is not only ripe for actual use in critical care clinical research but it is also a necessary step to maximize the performance of clinical trials and its translation to the practice of critical care medicine.

Protocol for evaluation of the cost-effectiveness of ePrescribing systems and candidate prototype for other related health information technologies

BACKGROUND

This protocol concerns the assessment of cost-effectiveness of hospital health information technology (HIT) in four hospitals. Two of these hospitals are acquiring ePrescribing systems incorporating extensive decision support, while the other two will implement systems incorporating more basic clinical algorithms. Implementation of an ePrescribing system will have diffuse effects over myriad clinical processes, so the protocol has to deal with a large amount of information collected at various 'levels' across the system.

METHODS/DESIGN

The method we propose is use of Bayesian ideas as a philosophical guide.Assessment of cost-effectiveness requires a number of parameters in order to measure incremental cost utility or benefit - the effectiveness of the intervention in reducing frequency of preventable adverse events; utilities for these adverse events; costs of HIT systems; and cost consequences of adverse events averted. There is no single end-point that adequately and unproblematically captures the effectiveness of the intervention; we therefore plan to observe changes in error rates and adverse events in four error categories (death, permanent disability, moderate disability, minimal effect). For each category we will elicit and pool subjective probability densities from experts for reductions in adverse events, resulting from deployment of the intervention in a hospital with extensive decision support. The experts will have been briefed with quantitative and qualitative data from the study and external data sources prior to elicitation. Following this, there will be a process of deliberative dialogues so that experts can "re-calibrate" their subjective probability estimates. The consolidated densities assembled from the repeat elicitation exercise will then be used to populate a health economic model, along with salient utilities. The credible limits from these densities can define thresholds for sensitivity analyses.

DISCUSSION

The protocol we present here was designed for evaluation of ePrescribing systems. However, the methodology we propose could be used whenever research cannot provide a direct and unbiased measure of comparative effectiveness.

Cohort study design: an underutilized approach for advancement of evidence-based and patient-centered practice in athletic training

OBJECTIVE

Providing patient-centered care requires consideration of numerous factors when making decisions that will influence a patient's health status.

BACKGROUND

Clinical decisions should be informed by relevant research evidence, but the literature often lacks pertinent information for problems encountered in routine clinical practice. Although a randomized clinical trial provides the best research design to ensure the internal validity of study findings, ethical considerations and the competitive culture of sport often preclude random assignment of patients or participants to a control condition.

CLINICAL ADVANTAGES

A cohort study design and Bayesian approach to data analysis can provide valuable evidence to support clinical decisions. Dichotomous classification of both an outcome and 1 or more predictive factors permits quantification of the likelihood of occurrence of a specified outcome.

CONCLUSIONS

Multifactorial prediction models can reduce uncertainty in clinical decision making and facilitate the individualization of treatment, thereby supporting delivery of clinical services that are both evidence based and patient centered.

Heterosexual risk of HIV transmission per sexual act under combined antiretroviral therapy: systematic review and bayesian modeling

BACKGROUND

Although essential for patient counseling and quality of life of human immunodeficiency virus (HIV)-infected individuals, the risk of HIV transmission during 1 unprotected sex act with an HIV-infected person under combination antiretroviral therapy (cART) remains unknown.

METHODS

We reviewed systematically the literature for studies on HIV transmission among heterosexual HIV-serodiscordant couples, where the infected partner was on cART, with regular virological monitoring, reporting on condom use and sexual activity. We used Bayesian statistics to combine data from selected studies, to investigate the per-act risk of HIV transmission through unprotected sex with an HIV-infected person on cART for >6 months.

RESULTS

At most, 1 HIV transmission, over an estimated 113 480 sex acts, of which 17% were not condom protected, was reported within 1672 HIV-serodiscordant couples where the index partner had been treated for >6 months. Data were insufficient to determine whether the reported transmission occurred before or after 6 months of cART. We estimated the upper-bound per-act risk of HIV transmission at either 8.7 or 13:100 000, depending on whether the transmission occurred before or after 6 months of cART. These estimates applied whether or not index partners were virally suppressed. Estimating an upper-bound risk <1:100 000 would require observing no HIV transmission while collecting >12 times the available amount of data.

CONCLUSIONS

Available data do not support zero risk of HIV transmission under cART. The per-act risk of HIV transmission through unprotected sex with HIV-infected individuals on cART in comprehensive care for >6 months (whether or not virally suppressed) is <13:100 000. Estimating a 10-fold lower upper-bound risk may be unfeasible due to high condom use among HIV-serodiscordant couples in most research studies.

Identifying and reducing potentially wrong immunoassay results even when plausible and "not-unreasonable"

The primary role of the clinical laboratory is to report accurate results for diagnosis of disease and management of illnesses. This goal has, to a large extent been achieved for routine biochemical tests, but not for immunoassays which remained susceptible to interference from endogenous immunoglobulin antibodies, causing false, and clinically misleading results. Clinicians regard all abnormal results including false ones as "pathological" necessitating further investigations, or concluding iniquitous diagnosis. Even more seriously, "false-negative" results may wrongly exclude pathology, thus denying patients' necessary treatment. Analytical error rate in immunoassays is relatively high, ranging from 0.4% to 4.0%. Because analytical interference from endogenous antibodies is confined to individuals' sera, it can be inconspicuous, pernicious, sporadic, and insidious because it cannot be detected by internal or external quality assessment procedures. An approach based on Bayesian reasoning can enhance the robustness of clinical validation in highlighting potentially erroneous immunoassay results. When this rational clinical/statistical approach is followed by analytical affirmative follow-up tests, it can help identifying inaccurate and clinically misleading immunoassay data even when they appear plausible and "not-unreasonable." This chapter is largely based on peer reviewed articles associated with and related to this approach. The first section underlines (without mathematical equations) the dominance and misuse of conventional statistics and the underuse of Bayesian paradigm and shows that laboratorians are intuitively (albeit unwittingly) practicing Bayesians. Secondly, because interference from endogenous antibodies is method's dependent (with numerous formats and different reagents), it is almost impossible to accurately assess its incidence in all differently formulated immunoassays and for each analytes/biomarkers. However, reiterating the basic concepts underpinning interference from endogenous antibodies can highlight why interference will remain analytically pernicious, sporadic, and an inveterate problem. The following section discuses various stratagems to reduce this source of inaccuracy in current immunoassay results including the role of Bayesian reasoning. Finally, the role of three commonly used follow-up affirmative tests and their interpretation in confirming analytical interference is discussed.

Incorporating data from various trial designs into a mixed treatment comparison model

Estimates of relative efficacy between alternative treatments are crucial for decision making in health care. Bayesian mixed treatment comparison models provide a powerful methodology to obtain such estimates when head-to-head evidence is not available or insufficient. In recent years, this methodology has become widely accepted and applied in economic modelling of healthcare interventions. Most evaluations only consider evidence from randomized controlled trials, while information from other trial designs is ignored. In this paper, we propose three alternative methods of combining data from different trial designs in a mixed treatment comparison model. Naive pooling is the simplest approach and does not differentiate between-trial designs. Utilizing observational data as prior information allows adjusting for bias due to trial design. The most flexible technique is a three-level hierarchical model. Such a model allows for bias adjustment while also accounting for heterogeneity between-trial designs. These techniques are illustrated using an application in rheumatoid arthritis.

A Bayesian analysis of bisphosphonate effects and cost-effectiveness in post-menopausal osteoporosis

Objective - Effects of oral bisphosphonates on the rate of vertebral fractures in post-menopausal osteoporotic women have been found in clinical trials. We wanted to compare the effects of two bisphosphonates, alendronate and etidronate, and calculate the price difference that would give the same cost-effectiveness for the two drugs. We also intended to give, by means of Bayesian statistics, probability distributions and point and interval estimates for key parameters.Methods - We used published, double-blind, randomized placebo controlled studies describing the results of occurrence of vertebral fractures at 3-year follow-up in post-menopausal women taking bisphosphonates. Four studies were identified, including altogether 3510 women. The women had either suffered a fracture at entry or had a bone density at least 2.5 SD below the mean value for young women. Two of the studies dealt with alendronate and two with etidronate.Results - According to three of the studies, the number of women out of 100 avoiding vertebral fractures during a 3-year observation period varied from two to seven. The fourth study did not contain the necessary data. The four studies showed that, for the incidence rate, the multiplicative treatment effects were respectively 0.45, 0.74, 0.40 and 0.36, where values less than 1 indicate positive treatment effects. Using data from all four studies, a comparison of the two drugs gave a point estimate of 0.247 with 95% credibility interval (CI): -0.051 to 0.496 for a difference in effect in favour of alendronate measured in terms of risk ratio of fracture and 0.302 (CI: 0.099 to 0.539) measured as incidence rate ratio. Based on two studies, showing about the same prevalence of fractures in the control groups, the difference in the risk difference between the two drugs was 0.028 (CI: -0.039 to 0.079).Conclusions - Bisphosphonates effectively reduce risk of new vertebral fractures, but alendronate is somewhat more effective than etidronate. To obtain equal cost-effectiveness alendronate should be priced 40 - 70% higher than etidronate. Copyright (c) 2000 John Wiley & Sons, Ltd.

Evaluation of a predevelopment service delivery intervention: an application to improve clinical handovers

Background

We developed a method to estimate the expected cost-effectiveness of a service intervention at the design stage and ‘road-tested’ the method on an intervention to improve patient handover of care between hospital and community.

Method

The development of a nine-step evaluation framework:

1. Identification of multiple endpoints and arranging them into manageable groups;

2. Estimation of baseline overall and preventable risk;

3. Bayesian elicitation of expected effectiveness of the planned intervention;

4. Assigning utilities to groups of endpoints;

5. Costing the intervention;

6. Estimating health service costs associated with preventable adverse events;

7. Calculating health benefits;

8. Cost-effectiveness calculation;

9. Sensitivity and headroom analysis.

Results

    Literature review suggested that adverse events follow 19% of patient discharges, and that one-third are preventable by improved handover (ie, 6.3% of all discharges). The intervention to improve handover would reduce the incidence of adverse events by 21% (ie, from 6.3% to 4.7%) according to the elicitation exercise. Potentially preventable adverse events were classified by severity and duration. Utilities were assigned to each category of adverse event. The costs associated with each category of event were obtained from the literature. The unit cost of the intervention was €16.6, which would yield a Quality Adjusted Life Year (QALY) gain per discharge of 0.010. The resulting cost saving was €14.3 per discharge. The intervention is cost-effective at approximately €214 per QALY under the base case, and remains cost-effective while the effectiveness is greater than 1.6%.

Conclusions

We offer a usable framework to assist in ex ante health economic evaluations of health service interventions.

Bayesian cohort and cross-sectional analyses of the PINCER trial: a pharmacist-led intervention to reduce medication errors in primary care

Background

Medication errors are an important source of potentially preventable morbidity and mortality. The PINCER study, a cluster randomised controlled trial, is one of the world’s first experimental studies aiming to reduce the risk of such medication related potential for harm in general practice. Bayesian analyses can improve the clinical interpretability of trial findings.

Methods

Experts were asked to complete a questionnaire to elicit opinions of the likely effectiveness of the intervention for the key outcomes of interest - three important primary care medication errors. These were averaged to generate collective prior distributions, which were then combined with trial data to generate Bayesian posterior distributions. The trial data were analysed in two ways: firstly replicating the trial reported cohort analysis acknowledging pairing of observations, but excluding non-paired observations; and secondly as cross-sectional data, with no exclusions, but without acknowledgement of the pairing. Frequentist and Bayesian analyses were compared.

Findings

Bayesian evaluations suggest that the intervention is able to reduce the likelihood of one of the medication errors by about 50 (estimated to be between 20% and 70%). However, for the other two main outcomes considered, the evidence that the intervention is able to reduce the likelihood of prescription errors is less conclusive.

Conclusions

Clinicians are interested in what trial results mean to them, as opposed to what trial results suggest for future experiments. This analysis suggests that the PINCER intervention is strongly effective in reducing the likelihood of one of the important errors; not necessarily effective in reducing the other errors. Depending on the clinical importance of the respective errors, careful consideration should be given before implementation, and refinement targeted at the other errors may be something to consider.

Immunomodulatory therapies delay disease progression in multiple sclerosis

BACKGROUND

Few studies have analysed long-term effects of immunomodulatory disease modifying drugs (DMDs).

OBJECTIVE

Assessment of the efficacy of DMDs on long-term evolution of multiple sclerosis, using a Bayesian approach to overcome methodological problems related to open-label studies.

METHODS

MS patients from three different Italian multiple sclerosis centres were divided into subgroups according to the presence of treatment in their disease history before the endpoint, which was represented by secondary progression. Patients were stratified on the basis of the risk score BREMS (Bayesian risk estimate for multiple sclerosis), which is able to predict the unfavourable long-term evolution of MS at an early stage.

RESULTS

We analysed data from 1178 patients with a relapsing form of multiple sclerosis at onset and at least 10 years of disease duration, treated (59%) or untreated with DMDs. The risk of secondary progression was significantly lower in patients treated with DMDs, regardless of the initial prognosis predicted by BREMS.

CONCLUSIONS

DMDs significantly reduce the risk of multiple sclerosis progression both in patients with initial high-risk and patients with initial low-risk. These findings reinforce the role of DMDs in modifying the natural course of the disease, suggesting that they have a positive effect not only on the inflammatory but also on the neurodegenerative process. The study also confirms the capability of the BREMS score to predict MS evolution.

P values: use and misuse in medical literature

P values are widely used in the medical literature but many authors, reviewers, and readers are unfamiliar with a valid definition of a P value, let alone how to interpret one correctly. Popular explanations such as "the probability that study results are due to chance" are wrong in a variety of ways and can lead to substantial errors in evaluating the evidence from research studies. Belief that "statistical significance" can alone discriminate between truth and falsehood borders on magical thinking. The article points out how to better interpret P values by avoiding common errors. Statistical analyses and P values are important tools in evidence-based medicine, but have to be used cautiously and with better understanding.

Synthesis of observational studies should consider credibility ceilings

OBJECTIVE

Meta-analyses of observational studies often get spuriously precise results. We aimed to factor this skepticism in meta-analysis calculations.

STUDY DESIGN AND SETTING

We developed a simple sensitivity analysis starting from the assumption that any single observational study cannot give us more than a maximum certainty c% (called credibility ceiling) that an effect is in a particular direction and not in the other. Each study included in meta-analysis is adjusted for different credibility ceilings c and the consistency of the conclusion examined. We applied the method in three meta-analyses of observational studies with nominally statistically significant summary effects (mortality with teaching versus nonteaching health care; risk of non-Hodgkin's lymphoma with hair dyes; mortality with omega-3 fatty acids).

RESULTS

Between-study heterogeneity I(2) estimates dropped from 36%-72% without a ceiling effect to 0% with ceilings of 9%, 4%, and 4% in the three meta-analyses, respectively. Nominal statistical significance was lost with ceilings of 10%, 8%, and 11%, respectively. The likelihood ratios suggested that even with minimal ceiling effects, there was no strong support for the credibility of each of these three associations.

CONCLUSIONS

Consideration of credibility ceilings allows conservative interpretation of observational evidence and can be applied routinely to meta-analyses of observational studies.

A dirty dozen: twelve p-value misconceptions

The P value is a measure of statistical evidence that appears in virtually all medical research papers. Its interpretation is made extraordinarily difficult because it is not part of any formal system of statistical inference. As a result, the P value's inferential meaning is widely and often wildly misconstrued, a fact that has been pointed out in innumerable papers and books appearing since at least the 1940s. This commentary reviews a dozen of these common misinterpretations and explains why each is wrong. It also reviews the possible consequences of these improper understandings or representations of its meaning. Finally, it contrasts the P value with its Bayesian counterpart, the Bayes' factor, which has virtually all of the desirable properties of an evidential measure that the P value lacks, most notably interpretability. The most serious consequence of this array of P-value misconceptions is the false belief that the probability of a conclusion being in error can be calculated from the data in a single experiment without reference to external evidence or the plausibility of the underlying mechanism.

Optimal decision criterion for detecting change in bone mineral density during serial monitoring: a Bayesian approach

Interpretation of change in serial bone densitometry using least significant change (LSC) may not lead to optimal decision making. Using the principles of Bayesian statistics and decision sciences, we developed the Optimal Decision Criterion (ODC) which resulted in 11-12.5% higher rate of correct classification compared with the LSC method.

INTRODUCTION

The interpretation of change in serial bone densitometry emphasizes using least significant change (LSC) to distinguish between true changes and measurement error.

METHODS

Using the principles of Bayesian statistics and decision sciences, we developed the optimal decision criterion (ODC) based on maximizing a 'utility' function that rewards the correct and penalizes the incorrect classification of change. The relationship between LSC and ODC is demonstrated using a clinical sample from the Manitoba Bone Density Program.

RESULTS

Under certain conditions, it can be shown that using LSC at the 95% confidence level implicitly equates the benefit of 39 true positive diagnoses with the harm of one false positive classification of BMD change. ODC resulted in an 11% higher rate of correct classification for lumbar spine BMD change and a 12.5% better performance for classifying total hip BMD change compared with LSC with this method.

CONCLUSIONS

ODC has the same clinical interpretation as LSC but with two major advantages: it can incorporate prior knowledge of the likely values of the true change and it can be fine-tuned based on the relative value placed on the correct and incorrect classifications. Bayesian statistics and decision sciences could potentially increase the yield of a BMD monitoring program.

Bayesian statistical inference enhances the interpretation of contemporary randomized controlled trials

OBJECTIVE

Randomized trials generally use "frequentist" statistics based on P-values and 95% confidence intervals. Frequentist methods have limitations that might be overcome, in part, by Bayesian inference. To illustrate these advantages, we re-analyzed randomized trials published in four general medical journals during 2004.

STUDY DESIGN AND SETTING

We used Medline to identify randomized superiority trials with two parallel arms, individual-level randomization and dichotomous or time-to-event primary outcomes. Studies with P<0.05 in favor of the intervention were deemed "positive"; otherwise, they were "negative." We used several prior distributions and exact conjugate analyses to calculate Bayesian posterior probabilities for clinically relevant effects.

RESULTS

Of 88 included studies, 39 were positive using a frequentist analysis. Although the Bayesian posterior probabilities of any benefit (relative risk or hazard ratio<1) were high in positive studies, these probabilities were lower and variable for larger benefits. The positive studies had only moderate probabilities for exceeding the effects that were assumed for calculating the sample size. By comparison, there were moderate probabilities of any benefit in negative studies.

CONCLUSION

Bayesian and frequentist analyses complement each other when interpreting the results of randomized trials. Future reports of randomized trials should include both.

Why most discovered true associations are inflated

Newly discovered true (non-null) associations often have inflated effects compared with the true effect sizes. I discuss here the main reasons for this inflation. First, theoretical considerations prove that when true discovery is claimed based on crossing a threshold of statistical significance and the discovery study is underpowered, the observed effects are expected to be inflated. This has been demonstrated in various fields ranging from early stopped clinical trials to genome-wide associations. Second, flexible analyses coupled with selective reporting may inflate the published discovered effects. The vibration ratio (the ratio of the largest vs. smallest effect on the same association approached with different analytic choices) can be very large. Third, effects may be inflated at the stage of interpretation due to diverse conflicts of interest. Discovered effects are not always inflated, and under some circumstances may be deflated-for example, in the setting of late discovery of associations in sequentially accumulated overpowered evidence, in some types of misclassification from measurement error, and in conflicts causing reverse biases. Finally, I discuss potential approaches to this problem. These include being cautious about newly discovered effect sizes, considering some rational down-adjustment, using analytical methods that correct for the anticipated inflation, ignoring the magnitude of the effect (if not necessary), conducting large studies in the discovery phase, using strict protocols for analyses, pursuing complete and transparent reporting of all results, placing emphasis on replication, and being fair with interpretation of results.

Effect of formal statistical significance on the credibility of observational associations

The author evaluated the implications of nominal statistical significance for changing the credibility of null versus alternative hypotheses across a large number of observational associations for which formal statistical significance (p < 0.05) was claimed. Calculation of the Bayes factor (B) under different assumptions was performed on 272 observational associations published in 2004-2005 and a data set of 50 meta-analyses on gene-disease associations (752 studies) for which statistically significant associations had been claimed (p < 0.05). Depending on the formulation of the prior, statistically significant results offered less than strong support to the credibility (B > 0.10) for 54-77% of the 272 epidemiologic associations for diverse risk factors and 44-70% of the 50 associations from genetic meta-analyses. Sometimes nominally statistically significant results even decreased the credibility of the probed association in comparison with what was thought before the study was conducted. Five of six meta-analyses with less than substantial support (B > 0.032) lost their nominal statistical significance in a subsequent (more recent) meta-analysis, while this did not occur in any of seven meta-analyses with decisive support (B < 0.01). In these large data sets of observational associations, formal statistical significance alone failed to increase much the credibility of many postulated associations. Bayes factors may be used routinely to interpret "significant" associations.

Outcome of beta-interferon treatment in relapsing-remitting multiple sclerosis: a Bayesian analysis

Observational studies of the effect of beta-interferon (IFNbeta) on accumulation of fixed disability in relapsing remitting multiple sclerosis (RRMS) in clinical practice have been difficult to interpret due to bias. The aim of this study of 175 RRMS patients was to use Bayesian analysis to establish whether IFNbeta attenuates disability relative to a cohort of matched historical control subjects from the Sylvia Lawry Centre for MS Research. A sensitivity analysis was based on a range of prior probability distributions for IFNbeta efficacy derived from a published meta-analysis of randomised controlled trials (RCTs) of IFNbeta, and the data were interpreted both unmodified and using variance inflation and point estimate bias correction; the corrected data interpreted in the light of the most likely prior probability distribution yielded a 95 % posterior credible interval for the odds ratio of accumulation of fixed disability after two years of IFNbeta therapy of 0.52, 0.94. It is concluded that two years of IFNbeta therapy for RRMS reduces accumulation of fixed disability in clinical practice.

Modeling payback from research into the efficacy of left-ventricular assist devices as destination therapy

OBJECTIVES

Ongoing developments in design have improved the outlook for left-ventricular assist device (LVAD) implantation as a therapy in end-stage heart failure. Nevertheless, early cost-effectiveness assessments, based on first-generation devices, have not been encouraging. Against this background, we set out (i) to examine the survival benefit that LVADs would need to generate before they could be deemed cost-effective; (ii) to provide insight into the likelihood that this benefit will be achieved; and (iii) from the perspective of a healthcare provider, to assess the value of discovering the actual size of this benefit by means of a Bayesian value of information analysis.

METHODS

Cost-effectiveness assessments are made from the perspective of the healthcare provider, using current UK norms for the value of a quality-adjusted life-year (QALY). The treatment model is grounded in published analyses of the Randomized Evaluation of Mechanical Assistance for the Treatment of Congestive Heart Failure (REMATCH) trial of first-generation LVADs, translated into a UK cost setting. The prospects for patient survival with second-generation devices is assessed using Bayesian prior distributions, elicited from a group of leading clinicians in the field.

RESULTS

Using established thresholds, cost-effectiveness probabilities under these priors are found to be low (approximately .2 percent) for devices costing as much as 60,000 pounds. Sensitivity of the conclusions to both device cost and QALY valuation is examined.

CONCLUSIONS

In the event that the price of the device in use would reduce to 40,000 pounds, the value of the survival information can readily justify investment in further trials.

The contrast and convergence of Bayesian and frequentist statistical approaches in pharmacoeconomic analysis

The application of Bayesian statistical analyses has been facilitated in recent years by methodological advances and an increasing complexity necessitated within research. Substantial debate has historically accompanied this analytic approach relative to the frequentist method, which is the predominant statistical ideology employed in clinical studies. While the essence of the debate between the two branches of statistics centres on differences in the use of prior information and the definition of probability, the ramifications involve the breadth of research design, analysis and interpretation. The purpose of this paper is to discuss the application of frequentist and Bayesian statistics in the pharmacoeconomic assessment of healthcare technology. A description of both paradigms is offered in the context of potential advantages and disadvantages, and applications within pharmacoeconomics are briefly addressed. Additional considerations are presented to stimulate further development and to direct appropriate applications of each method such that the integrity and robustness of scientific inference be strengthened.

Research and theory for Nursing and Midwifery: Rethinking the Nature of Evidence

BACKGROUND AND RATIONALE

The rise in the principles of evidence-based medicine in the 1990s heralded a re-emerging orthodoxy in research methodologies. The view of the randomised controlled trial (RCT) as a "gold standard" for evaluation of medical interventions has extended recently to evaluation of organisational forms and reforms and of change in complex systems-within health care and in other human services. Relatively little attention has been given to the epistemological assumptions underlying such a hierarchy of research evidence.

AIMS AND METHODS

Case studies from research in maternity care are used in this article to describe problems and limitations encountered in using RCTs to evaluate some recent policy-driven and consumer-oriented developments. These are discussed in relation to theory of knowledge and the epistemological assumptions, or paradigms, underpinning health services research. The aim in this discussion is not to advocate, or to reject, particular approaches to research but to advocate a more open and critical engagement with questions about the nature of evidence.

FINDINGS AND DISCUSSION

Experimental approaches are of considerable value in investigating deterministic and probabilistic cause and effect relationships, and in testing often well-established but unevaluated technologies. However, little attention has been paid to contextual and cultural factors in the effects of interventions, in the culturally constructed nature of research questions themselves, or of the data on which much research is based. More complex, and less linear, approaches to methodology are needed to address these issues. A simple hierarchical approach does not represent the complexity of evidence well and should move toward a more cyclical view of knowledge development.

Introduction to Bayesian methods I: measuring the strength of evidence

Bayesian inference is a formal method to combine evidence external to a study, represented by a prior probability curve, with the evidence generated by the study, represented by a likelihood function. Because Bayes theorem provides a proper way to measure and to combine study evidence, Bayesian methods can be viewed as a calculus of evidence, not just belief. In this introduction, we explore the properties and consequences of using the Bayesian measure of evidence, the Bayes factor (in its simplest form, the likelihood ratio). The Bayes factor compares the relative support given to two hypotheses by the data, in contrast to the P-value, which is calculated with reference only to the null hypothesis. This comparative property of the Bayes factor, combined with the need to explicitly predefine the alternative hypothesis, produces a different assessment of the strength of evidence against the null hypothesis than does the P-value, and it gives Bayesian procedures attractive frequency properties. However, the most important contribution of Bayesian methods is the way in which they affect both who participates in a scientific dialogue, and what is discussed. With the emphasis moved from "error rates" to evidence, content experts have an opportunity for their input to be meaningfully incorporated, making it easier for regulatory decisions to be made correctly.

Bayes’ theorem: A negative example of a RCT on grommets in children with glue ear

Bayesian inference presupposes that practitioners' belief in the effectiveness of medical intervention is the product of prior belief and recent evidence from studies. Although increasingly used, up to now the posterior belief calculated according to the theorem has not been compared with an empirically measured posterior belief. We conducted a RCT, which was preceded by elicitation of prior beliefs among ENT-surgeons, and which was followed by elicitation of posterior beliefs among ENT-surgeons, 1 year after completion of the trial. We compared the posterior beliefs of ENT-surgeons about the effect of grommets in children with glue ears, as predicted by Bayes' theorem with actual measured posterior beliefs. The distribution of the measured posterior beliefs was not in line with the calculated posterior, but almost identical to the distribution of the measured prior beliefs. The results showed that our trial had little or no impact on the beliefs of the ENT-surgeons, i.e. they did not adjust their belief to the extent that was expected according to Bayes' theorem.

Bridging the gap between statistical analysis and decision making in public health research

Statisticians have eagerly taken on the role of presenting statistical summaries of quantitative data. In areas of health, this means providing point and interval estimates for quantities of interest such as diagnostic risks and treatment effects or providing curve estimates for quantities of interest such as survival probabilities across time. Methods for providing such summaries are highly formalized and constantly evolving. While decision making is the incentive for nearly all such efforts, the process that transforms statistical summaries into decisions usually remains informal and ad hoc. Statisticians have not eagerly accepted the role of promoting formalized decision-theoretic techniques. This paper will argue that the gap between statistical synthesis and decision making is an unnatural and undesirable one, because it undermines the impact of quantitative information. An argument for bridging the gap by expanding the role of statisticians will be presented.

Bayes and health care research

Bayes' rule shows how one might rationally change one's beliefs in the light of evidence. It is the foundation of a statistical method called Bayesianism. In health care research, Bayesianism has its advocates but the dominant statistical method is frequentism. There are at least two important philosophical differences between these methods. First, Bayesianism takes a subjectivist view of probability (i.e. that probability scores are statements of subjective belief, not objective fact) whilst frequentism takes an objectivist view. Second, Bayesianism is explicitly inductive (i.e. it shows how we may induce views about the world based on partial data from it) whereas frequentism is at least compatible with non-inductive views of scientific method, particularly the critical realism of Popper. Popper and others detail significant problems with induction. Frequentism's apparent ability to avoid these, plus its ability to give a seemingly more scientific and objective take on probability, lies behind its philosophical appeal to health care researchers. However, there are also significant problems with frequentism, particularly its inability to assign probability scores to single events. Popper thus proposed an alternative objectivist view of probability, called propensity theory, which he allies to a theory of corroboration; but this too has significant problems, in particular, it may not successfully avoid induction. If this is so then Bayesianism might be philosophically the strongest of the statistical approaches. The article sets out a number of its philosophical and methodological attractions. Finally, it outlines a way in which critical realism and Bayesianism might work together.