Guidance on the implementation and reporting of a drug safety Bayesian network meta-analysis

Why are There not More Bayesian Clinical Trials? Ability to Interpret Bayesian and Conventional Statistics Among Medical Researchers

OBJECTIVE AND BACKGROUND

We assessed current understandings in interpretation of Bayesian and traditional statistical results within the clinical researcher (non-statistician) community.

METHODS

Within a 22-question survey, including demographics and experience and comfort levels with Bayesian analyses, we included questions on how to interpret both Bayesian and traditional statistical outputs. We also assessed whether Bayesian or traditional interpretations are considered more useful.

RESULTS

Among the 323 respondent clinicians, 42.4% and 36.5% chose the correct interpretations of the posterior probability and 95% credible interval, respectively. Only 11.5% of respondents interpreted the p-value correctly and 23.5% interpreted the 95% confidence interval correctly.

CONCLUSIONS

Based on these survey results, we conclude that most of these clinicians face uncertainty when attempting to interpret results from both Bayesian and traditional statistical outputs. When presented with accurate interpretations, clinicians generally conclude that Bayesian results are more useful than conventional ones. We believe there is a need for education of clinicians in statistical interpretation in ways that are customized to this audience.

Why are not There More Bayesian Clinical Trials? Perceived Barriers and Educational Preferences Among Medical Researchers Involved in Drug Development

OBJECTIVE AND BACKGROUND

The clinical trials community has been hesitant to adopt Bayesian statistical methods, which are often more flexible and efficient with more naturally interpretable results than frequentist methods. We aimed to identify self-reported barriers to implementing Bayesian methods and preferences for becoming comfortable with them.

METHODS

We developed a 22-question survey submitted to medical researchers (non-statisticians) from industry, academia, and regulatory agencies. Question areas included demographics, experience, comfort levels with Bayesian analyses, perceived barriers to these analyses, and preferences for increasing familiarity with Bayesian methods.

RESULTS

Of the 323 respondents, most were affiliated with pharmaceutical companies (33.4%), clinical research organizations (29.7%), and regulatory agencies (18.6%). The rest represented academia, medical practice, or other. Over 56% of respondents expressed little to no comfort in interpreting Bayesian analyses. "Insufficient knowledge of Bayesian approaches" was ranked the most important perceived barrier to implementing Bayesian methods by a plurality (48%). Of the approaches listed, in-person training was the most preferred for gaining comfort with Bayesian methods.

CONCLUSIONS

Based on these survey results, we recommend that introductory level training on Bayesian statistics be presented in an in-person workshop that could also be broadcast online with live Q&A. Other approaches such as online training or collaborative projects may be better suited for higher-level trainings where instructors may assume a baseline understanding of Bayesian statistics. Increased coverage of Bayesian methods at medical conferences and medical school trainings would help improve comfort and overcome the substantial knowledge barriers medical researchers face when implementing these methods.

Classifying information-sharing methods

BACKGROUND

Sparse relative effectiveness evidence is a frequent problem in Health Technology Assessment (HTA). Where evidence directly pertaining to the decision problem is sparse, it may be feasible to expand the evidence-base to include studies that relate to the decision problem only indirectly: for instance, when there is no evidence on a comparator, evidence on other treatments of the same molecular class could be used; similarly, a decision on children may borrow-strength from evidence on adults. Usually, in HTA, such indirect evidence is either included by ignoring any differences ('lumping') or not included at all ('splitting'). However, a range of more sophisticated methods exists, primarily in the biostatistics literature. The objective of this study is to identify and classify the breadth of the available information-sharing methods.

METHODS

Forwards and backwards citation-mining techniques were used on a set of seminal papers on the topic of information-sharing. Papers were included if they specified (network) meta-analytic methods for combining information from distinct populations, interventions, outcomes or study-designs.

RESULTS

Overall, 89 papers were included. A plethora of evidence synthesis methods have been used for information-sharing. Most papers (n=79) described methods that shared information on relative treatment effects. Amongst these, there was a strong emphasis on methods for information-sharing across multiple outcomes (n=42) and treatments (n=25), with fewer papers focusing on study-designs (n=23) or populations (n=8). We categorise and discuss the methods under four 'core' relationships of information-sharing: functional, exchangeability-based, prior-based and multivariate relationships, and explain the assumptions made within each of these core approaches.

CONCLUSIONS

This study highlights the range of information-sharing methods available. These methods often impose more moderate assumptions than lumping or splitting. Hence, the degree of information-sharing that they impose could potentially be considered more appropriate. Our identification of four 'core' methods of information-sharing allows for an improved understanding of the assumptions underpinning the different methods. Further research is required to understand how the methods differ in terms of the strength of sharing they impose and the implications of this for health care decisions.

Change Point Analysis for Detecting Vaccine Safety Signals

It is important to detect signals of abrupt changes in adverse event reporting in order to notice public safety concerns and take prompt action, especially for vaccines under national immunization programs. In this study, we assessed the applicability of change point analysis (CPA) for signal detection in vaccine safety surveillance. The performances of three CPA methods, namely Bayesian change point analysis, Taylor’s change point analysis (Taylor-CPA), and environmental time series change point detection (EnvCpt), were assessed via simulated data with assumptions for the baseline number of events and degrees of change. The analysis was validated using the Korea Adverse Event Reporting System (KAERS) database. In the simulation study, the Taylor-CPA method exhibited better results for the detection of a change point (accuracy of 96% to 100%, sensitivity of 7% to 100%, specificity of 98% to 100%, positive predictive value of 25% to 85%, negative predictive value of 96% to 100%, and balanced accuracy of 53% to 100%) than the other two CPA methods. When the CPA methods were applied to reports of syncope or dizziness following human papillomavirus (HPV) immunization in the KAERS database, Taylor-CPA and EnvCpt detected a change point (Q2/2013), which was consistent with actual public safety concerns. CPA can be applied as an efficient tool for the early detection of vaccine safety signals.

Bayesian Network Meta-Analysis for Assessing Adverse Effects of Anti-hepatitis B Drugs

BACKGROUND AND OBJECTIVE

Oral nucleoside/nucleotide analogues (NAs) have been advocated for chronic hepatitis B (CHB) treatment with good efficacy. However, less attention has been put on their adverse events. Therefore, a Bayesian network meta-analysis (NMA) was performed to evaluate the relative safety of five NAs (lamivudine, adefovir dipivoxil, entecavir, telbivudine, and tenofovir disoproxil fumarate) in CHB treatment among adults.

METHODS

Eligible randomized clinical trials (RCTs) and prospective cohort studies were systematically and thoroughly searched until May 1, 2019. Poisson-prior-based Bayesian NMA was performed to synthesize both direct and indirect evidence with reporting hazard ratios (HRs) and 95% credible intervals (CrIs) for serious adverse events (SAEs) and hepatic/renal impairments.

RESULTS

Thirty-three RCTs and 11 prospective cohort studies were identified. As to SAEs, no statistically significant difference was found of any comparison among five NAs. In terms of hepatotoxicity, lamivudine was safer than telbivudine (HR 0.45; 95% CrI 0.21, 0.85), and entecavir increased the risk by 102% (entecavir vs lamivudine: HR 2.02; 95% CrI 1.19, 3.27).

CONCLUSIONS

The findings from this large NMA could influence clinical practice, and the methodological framework of this study could provide evidence-based support to analyze sparse safety data in the field.

Comparative efficacy of non-pharmacological interventions on agitation in people with dementia: A systematic review and Bayesian network meta-analysis

BACKGROUND

Agitation in people with dementia is common and distressing and can lead to increased caregiver burden. However, medications often have adverse reactions and limited effectiveness. Thus, non-pharmacological interventions are being increasingly implemented.

OBJECTIVES

To compare and rank the efficacy of different non-pharmacological interventions in the management of agitation in people with dementia.

DESIGN

Bayesian network meta-analysis.

METHODS

A comprehensive electronic literature search was performed in five English databases and three Chinese databases to identify relevant randomized controlled trials (RCTs) that were published up to January 2019. A random-effects model was selected to conduct traditional meta-analysis to directly examine the efficacy of different non-pharmacological interventions. The consistency model was selected to conduct a network meta-analysis to evaluate the relative effects and rank probability of different non-pharmacological interventions.

RESULTS

A total of 65 RCTs were included in this network meta-analysis involving 11 different non-pharmacological interventions. Network meta-analysis showed that massage therapy, animal-assisted intervention, and personally tailored intervention were associated with more substantial reductions in agitation compared with other interventions and controls.

CONCLUSIONS

Our study confirmed the effectiveness of non-pharmacological interventions for ameliorating agitation in people with dementia and recommended several interventions for clinical practice. Healthcare professionals should be encouraged to apply promising non-pharmacological interventions (e.g. massage therapy, animal-assisted intervention and personally tailored intervention) for people with dementia during routine care.

Network meta-analysis: a technique to gather evidence from direct and indirect comparisons

Systematic reviews and pairwise meta-analyses of randomized controlled trials, at the intersection of clinical medicine, epidemiology and statistics, are positioned at the top of evidence-based practice hierarchy. These are important tools to base drugs approval, clinical protocols and guidelines formulation and for decision-making. However, this traditional technique only partially yield information that clinicians, patients and policy-makers need to make informed decisions, since it usually compares only two interventions at the time. In the market, regardless the clinical condition under evaluation, usually many interventions are available and few of them have been studied in head-to-head studies. This scenario precludes conclusions to be drawn from comparisons of all interventions profile (e.g. efficacy and safety). The recent development and introduction of a new technique – usually referred as network meta-analysis, indirect meta-analysis, multiple or mixed treatment comparisons – has allowed the estimation of metrics for all possible comparisons in the same model, simultaneously gathering direct and indirect evidence. Over the last years this statistical tool has matured as technique with models available for all types of raw data, producing different pooled effect measures, using both Frequentist and Bayesian frameworks, with different software packages. However, the conduction, report and interpretation of network meta-analysis still poses multiple challenges that should be carefully considered, especially because this technique inherits all assumptions from pairwise meta-analysis but with increased complexity. Thus, we aim to provide a basic explanation of network meta-analysis conduction, highlighting its risks and benefits for evidence-based practice, including information on statistical methods evolution, assumptions and steps for performing the analysis.

A Bayesian meta-analytic approach for safety signal detection in randomized clinical trials

Background/Aim:

Meta-analyses are frequently performed on adverse event data and are primarily used for improving statistical power to detect safety signals. However, in the evaluation of drug safety for New Drug Applications, simple pooling of adverse event data from multiple clinical trials is still commonly used. We sought to propose a new Bayesian hierarchical meta-analytic approach based on consideration of a hierarchical structure of reported individual adverse event data from multiple randomized clinical trials.

Methods:

To develop our meta-analysis model, we extended an existing three-stage Bayesian hierarchical model by including an additional stage of the clinical trial level in the hierarchical model; this generated a four-stage Bayesian hierarchical model. We applied the proposed Bayesian meta-analysis models to published adverse event data from three premarketing randomized clinical trials of tadalafil and to a simulation study motivated by the case example to evaluate the characteristics of three alternative models.

Results:

Comparison of the results from the Bayesian meta-analysis model with those from Fisher’s exact test after simple pooling showed that 6 out of 10 adverse events were the same within a top 10 ranking of individual adverse events with regard to association with treatment. However, more individual adverse events were detected in the Bayesian meta-analysis model than in Fisher’s exact test under the body system “Musculoskeletal and connective tissue disorders.” Moreover, comparison of the overall trend of estimates between the Bayesian model and the standard approach (odds ratios after simple pooling methods) revealed that the posterior median odds ratios for the Bayesian model for most adverse events shrank toward values for no association. Based on the simulation results, the Bayesian meta-analysis model could balance the false detection rate and power to a better extent than Fisher’s exact test. For example, when the threshold value of the posterior probability for signal detection was set to 0.8, the false detection rate was 41% and power was 88% in the Bayesian meta-analysis model, whereas the false detection rate was 56% and power was 86% in Fisher’s exact test.

Limitations:

Adverse events under the same body system were not necessarily positively related when we used “system organ class” and “preferred term” in the Medical Dictionary for Regulatory Activities as a hierarchical structure of adverse events. For the Bayesian meta-analysis models to be effective, the validity of the hierarchical structure of adverse events and the grouping of adverse events are critical.

Conclusion:

Our proposed meta-analysis models considered trial effects to avoid confounding by trial and borrowed strength from both within and across body systems to obtain reasonable and stable estimates of an effect measure by considering a hierarchical structure of adverse events.

An appraisal of meta-analysis guidelines: how do they relate to safety outcomes?

Although well developed to assess efficacy questions, meta-analyses and, more generally, systematic reviews, have received less attention in application to safety-related questions. As a result, many open questions remain on how best to apply meta-analyses in the safety setting. This appraisal attempts to: (i) summarize the current guidelines for assessing individual studies, systematic reviews, and network meta-analyses; (ii) describe several publications on safety meta-analytic approaches; and (iii) present some of the questions and issues that arise with safety data. A number of gaps in the current quality guidelines are identified along with issues to consider when performing a safety meta-analysis. While some work is ongoing to provide guidance to improve the quality of safety meta-analyses, this review emphasizes the critical need for better reporting and increased transparency regarding safety data in the systematic review guidelines. Copyright © 2016 John Wiley & Sons, Ltd.

Treatments for Lupus Nephritis: A Systematic Review and Network Metaanalysis

OBJECTIVE

To compare benefits and harms of lupus nephritis (LN) induction and maintenance treatments.

METHODS

We performed a systematic review and Bayesian network metaanalyses of randomized controlled trials (RCT) of immunosuppressive drugs or corticosteroids (CS) in LN. OR and 95% credible intervals (CrI) were calculated.

RESULTS

There were 65 RCT that met inclusion and exclusion criteria. Significantly lower risk of endstage renal disease (ESRD; 17 studies) was seen with cyclophosphamide (CYC; OR 0.49, 95% CrI 0.25-0.92) or CYC + azathioprine (AZA; OR 0.18, 95% CrI 0.05-0.57) compared with standard-dose CS, and with high-dose (HD) CYC (OR 0.16, 95% CrI 0.03-0.61) or CYC + AZA (OR 0.10, 95% CrI 0.03-0.34) compared with HD CS. HD CS was associated with higher risk of ESRD compared with CYC (OR 3.59, 95% CrI 1.30-9.86), AZA (OR 2.93, 95% CrI 1.08-8.10), or mycophenolate mofetil (MMF; OR 7.05, 95% CrI 1.66-31.91). Compared with CS, a significantly higher proportion of patients had renal response (14 studies) when treated with CYC (OR 1.98, 95% CrI 1.13-3.52), MMF (OR 2.42, 95% CrI 1.27-4.74), or tacrolimus (TAC; OR 4.20, 95% CrI 1.29-13.68). No differences were noted for the risk of malignancy (15 studies). The risk of herpes zoster (17 studies) was as follows: OR (95% CrI) MMF versus CS 4.38 (1.02-23.87), CYC versus CS 6.64 (1.97-25.71), TAC versus CS 9.11 (1.13-70.99), and CYC + AZA versus CS 8.46 (1.99-43.61).

CONCLUSION

Renal benefits and the risk of herpes zoster were higher for immunosuppressive drugs versus CS. Data on relative and absolute differences are now available, which can be incorporated into patient-physician discussions related to systemic lupus erythematosus medication use.

Statistical Models and Methods for Network Meta-Analysis

Meta-analysis, the methodology for analyzing the results from multiple independent studies, has grown tremendously in popularity over the last four decades. Although most meta-analyses involve a single effect size (summary result, such as a treatment difference) from each study, there are often multiple treatments of interest across the network of studies in the analysis. Multi-treatment (or network) meta-analysis can be used for simultaneously analyzing the results from all the treatments. However, the methodology is considerably more complicated than for the analysis of a single effect size, and there have not been adequate explanations of the approach for agricultural investigations. We review the methods and models for conducting a network meta-analysis based on frequentist statistical principles, and demonstrate the procedures using a published multi-treatment plant pathology data set. A major advantage of network meta-analysis is that correlations of estimated treatment effects are automatically taken into account when an appropriate model is used. Moreover, treatment comparisons may be possible in a network meta-analysis that are not possible in a single study because all treatments of interest may not be included in any given study. We review several models that consider the study effect as either fixed or random, and show how to interpret model-fitting output. We further show how to model the effect of moderator variables (study-level characteristics) on treatment effects, and present one approach to test for the consistency of treatment effects across the network. Online supplemental files give explanations on fitting the network meta-analytical models using SAS.

GetReal in network meta-analysis: a review of the methodology

Pairwise meta-analysis is an established statistical tool for synthesizing evidence from multiple trials, but it is informative only about the relative efficacy of two specific interventions. The usefulness of pairwise meta-analysis is thus limited in real-life medical practice, where many competing interventions may be available for a certain condition and studies informing some of the pairwise comparisons may be lacking. This commonly encountered scenario has led to the development of network meta-analysis (NMA). In the last decade, several applications, methodological developments, and empirical studies in NMA have been published, and the area is thriving as its relevance to public health is increasingly recognized. This article presents a review of the relevant literature on NMA methodology aiming to pinpoint the developments that have appeared in the field. Copyright © 2016 John Wiley & Sons, Ltd.

Predictive biomarkers for treatment selection: statistical considerations

Predictive biomarkers are developed for treatment selection to identify patients who are likely to benefit from a particular therapy. This review describes statistical methods and discusses issues in the development of predictive biomarkers to enhance study efficiency for detection of treatment effect on the selected responder patients in clinical studies. The statistical procedure for treatment selection consists of three components: biomarker identification, subgroup selection and clinical utility assessment. Major statistical issues discussed include biomarker designs, procedures to identify predictive biomarkers, classification models for subgroup selection, subgroup analysis and multiple testing for clinical utility assessment and evaluation.

Detecting outlying trials in network meta-analysis

Network meta-analysis (NMA) expands the scope of a conventional pairwise meta-analysis to simultaneously handle multiple treatment comparisons. However, some trials may appear to deviate markedly from the others and thus be inappropriate to be synthesized in the NMA. In addition, the inclusion of these trials in evidence synthesis may lead to bias in estimation. We call such trials trial-level outliers. To the best of our knowledge, while heterogeneity and inconsistency in NMA have been extensively discussed and well addressed, few previous papers have considered the proper detection and handling of trial-level outliers. In this paper, we propose several Bayesian outlier detection measures, which are then applied to a diabetes data set. Simulation studies comparing our approaches in both arm-based and contrast-based model settings are provided in two supporting appendices.

Integration of PKPD relationships into benefit-risk analysis

AIM

Despite the continuous endeavour to achieve high standards in medical care through effectiveness measures, a quantitative framework for the assessment of the benefit-risk balance of new medicines is lacking prior to regulatory approval. The aim of this short review is to summarise the approaches currently available for benefit-risk assessment. In addition, we propose the use of pharmacokinetic-pharmacodynamic (PKPD) modelling as the pharmacological basis for evidence synthesis and evaluation of novel therapeutic agents.

METHODS

A comprehensive literature search has been performed using MESH terms in PubMed, in which articles describing benefit-risk assessment and modelling and simulation were identified. In parallel, a critical review of multi-criteria decision analysis (MCDA) is presented as a tool for characterising a drug's safety and efficacy profile.

RESULTS

A definition of benefits and risks has been proposed by the European Medicines Agency (EMA), in which qualitative and quantitative elements are included. However, in spite of the value of MCDA as a quantitative method, decisions about benefit-risk balance continue to rely on subjective expert opinion. By contrast, a model-informed approach offers the opportunity for a more comprehensive evaluation of benefit-risk balance before extensive evidence is generated in clinical practice.

CONCLUSIONS

Benefit-risk balance should be an integral part of the risk management plan and as such considered before marketing authorisation. Modelling and simulation can be incorporated into MCDA to support the evidence synthesis as well evidence generation taking into account the underlying correlations between favourable and unfavourable effects. In addition, it represents a valuable tool for the optimization of protocol design in effectiveness trials.

Using indirect comparisons to compare interventions within a Cochrane review: a tool for comparative effectiveness research

AIM

Assessing relative performance among competing interventions is an important part of comparative effectiveness research. Bayesian indirect comparisons add information to existing Cochrane reviews, such as which intervention is likely to perform best. However, heterogeneity variance priors may influence results and, potentially, clinical guidance.

METHODS

We highlight the features of Bayesian indirect comparisons using a case study of a Cochrane review update in asthma care. The probability that one self-management educational intervention outperforms others is estimated. Simulation studies investigate the effect of heterogeneity variance prior distributions.

RESULTS

Results suggest a 55% probability that individual education is best, followed by combination (39%) and group (6%). The intervention with few trials was sensitive to prior distributions.

CONCLUSION

Bayesian indirect comparisons updates of Cochrane reviews are valuable comparative effectiveness research tools.

The quality of reporting methods and results in network meta-analyses: an overview of reviews and suggestions for improvement

INTRODUCTION

Some have suggested the quality of reporting of network meta-analyses (a technique used to synthesize information to compare multiple interventions) is sub-optimal. We sought to review information addressing this claim.

OBJECTIVE

To conduct an overview of existing evaluations of quality of reporting in network meta-analyses and indirect treatment comparisons, and to compile a list of topics which may require detailed reporting guidance to enhance future reporting quality.

METHODS

An electronic search of Medline and the Cochrane Registry of methodologic studies (January 2004-August 2013) was performed by an information specialist. Studies describing findings from quality of reporting assessments were sought. Screening of abstracts and full texts was performed by two team members. Descriptors related to all aspects of reporting a network meta-analysis were summarized.

RESULTS

We included eight reports exploring the quality of reporting of network meta-analyses. From past reviews, authors found several aspects of network meta-analyses were inadequately reported, including primary information about literature searching, study selection, and risk of bias evaluations; statement of the underlying assumptions for network meta-analysis, as well as efforts to verify their validity; details of statistical models used for analyses (including information for both Bayesian and Frequentist approaches); completeness of reporting of findings; and approaches for summarizing probability measures as additional important considerations.

CONCLUSIONS

While few studies were identified, several deficiencies in the current reporting of network meta-analyses were observed. These findings reinforce the need to develop reporting guidance for network meta-analyses. Findings from this review will be used to guide next steps in the development of reporting guidance for network meta-analysis in the format of an extension of the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analysis) Statement.