Bayesian methods for design and analysis of safety trials

From statistical inference to machine learning: A paradigm shift in contemporary cardiovascular pharmacotherapy

AIMS

Heart failure with reduced ejection fraction (HFrEF) poses significant challenges for clinicians and researchers, owing to its multifaceted aetiology and complex treatment regimens. In light of this, artificial intelligence methods offer an innovative approach to identifying relationships within complex clinical datasets. Our study aims to explore the potential for machine learning algorithms to provide deeper insights into datasets of HFrEF patients.

METHODS

To this end, we analysed a cohort of 386 HFrEF patients who had been initiated on sodium-glucose co-transporter-2 inhibitor treatment and had completed a minimum of a 6-month follow-up.

RESULTS

In traditional frequentist statistical analyses, patients receiving the highest doses of beta-blockers (BBs) (chi-square test, P = .036) and those newly initiated on sacubitril-valsartan (chi-square test, P = .023) showed better outcomes. However, none of these pharmacological features stood out as independent predictors of improved outcomes in the Cox proportional hazards model. In contrast, when employing eXtreme Gradient Boosting (XGBoost) algorithms in conjunction with the data using Shapley additive explanations (SHAP), we identified several models with significant predictive power. The XGBoost algorithm inherently accommodates non-linear distribution, multicollinearity and confounding. Within this framework, pharmacological categories like 'newly initiated treatment with sacubitril/valsartan' and 'BB dose escalation' emerged as strong predictors of long-term outcomes.

CONCLUSIONS

In this manuscript, we not only emphasize the strengths of this machine learning approach but also discuss its potential limitations and the risk of identifying statistically significant yet clinically irrelevant predictors.

Bayesianism from a philosophical perspective and its application to medicine

Bayesian philosophy and Bayesian statistics have diverged in recent years, because Bayesian philosophers have become more interested in philosophical problems other than the foundations of statistics and Bayesian statisticians have become less concerned with philosophical foundations. One way in which this divergence manifests itself is through the use of direct inference principles: Bayesian philosophers routinely advocate principles that require calibration of degrees of belief to available non-epistemic probabilities, while Bayesian statisticians rarely invoke such principles. As I explain, however, the standard Bayesian framework cannot coherently employ direct inference principles. Direct inference requires a shift towards a non-standard Bayesian framework, which further increases the gap between Bayesian philosophy and Bayesian statistics. This divergence does not preclude the application of Bayesian philosophical methods to real-world problems. Data consolidation is a key challenge for present-day systems medicine and other systems sciences. I show that data consolidation requires direct inference and that the non-standard Bayesian methods outlined here are well suited to this task.

FeBRILe3: Safety Evaluation of Febrile Infant Guidelines Through Prospective Bayesian Monitoring

OBJECTIVES

Despite evidence supporting earlier discharge of well-appearing febrile infants at low risk of serious bacterial infection (SBI), admissions for ≥48 hours remain common. Prospective safety monitoring may support broader guideline implementation.

METHODS

A sequential Bayesian safety monitoring framework was used to evaluate a new hospital guideline recommending early discharge of low-risk infants. Hospital readmissions within 7 days of discharge were regularly assessed against safety thresholds, derived from historic rates and expert opinion, and specified a priori (8 per 100 infants). Infants aged under 3 months admitted to 2 Western Australian metropolitan hospitals for management of fever without source were enrolled (August 2019-December 2021), to a prespecified maximum 500 enrolments.

RESULTS

Readmission rates remained below the prespecified threshold at all scheduled analyses. Median corrected age was 34 days, and 14% met low-risk criteria (n = 71). SBI was diagnosed in 159 infants (32%), including urinary tract infection (n = 140) and bacteraemia (n = 18). Discharge occurred before 48 hours for 192 infants (38%), including 52% deemed low-risk. At study completion, 1 of 37 low-risk infants discharged before 48 hours had been readmitted (3%), for issues unrelated to SBI diagnosis. In total, 20 readmissions were identified (4 per 100 infants; 95% credible interval 3, 6), with >0.99 posterior probability of being below the prespecified noninferiority threshold, indicating acceptable safety.

CONCLUSIONS

A Bayesian monitoring approach supported safe early discharge for many infants, without increased risk of readmission. This framework may be used to embed safety evaluations within future guideline implementation programs to further reduce low-value care.

Why are There not More Bayesian Clinical Trials? Ability to Interpret Bayesian and Conventional Statistics Among Medical Researchers

OBJECTIVE AND BACKGROUND

We assessed current understandings in interpretation of Bayesian and traditional statistical results within the clinical researcher (non-statistician) community.

METHODS

Within a 22-question survey, including demographics and experience and comfort levels with Bayesian analyses, we included questions on how to interpret both Bayesian and traditional statistical outputs. We also assessed whether Bayesian or traditional interpretations are considered more useful.

RESULTS

Among the 323 respondent clinicians, 42.4% and 36.5% chose the correct interpretations of the posterior probability and 95% credible interval, respectively. Only 11.5% of respondents interpreted the p-value correctly and 23.5% interpreted the 95% confidence interval correctly.

CONCLUSIONS

Based on these survey results, we conclude that most of these clinicians face uncertainty when attempting to interpret results from both Bayesian and traditional statistical outputs. When presented with accurate interpretations, clinicians generally conclude that Bayesian results are more useful than conventional ones. We believe there is a need for education of clinicians in statistical interpretation in ways that are customized to this audience.

BAHAMA: A Bayesian Hierarchical Model for the Detection of MedDRA®-Coded Adverse Events in Randomized Controlled Trials

Introduction

Patients participating in randomized controlled trials (RCTs) are susceptible to a wide range of different adverse events (AE) during the RCT. MedDRA^® is a hierarchical standardization terminology to structure the AEs reported in an RCT. The lowest level in the MedDRA hierarchy is a single medical event, and every higher level is the aggregation of the lower levels.

Method

We propose a multi-stage Bayesian hierarchical Poisson model for estimating MedDRA-coded AE rate ratios (RRs). To deal with rare AEs, we introduce data aggregation at a higher level within the MedDRA structure and based on thresholds on incidence and MedDRA structure.

Results

With simulations, we showed the effects of this data aggregation process and the method's performance. Furthermore, an application to a real example is provided and compared with other methods.

Conclusion

We showed the benefit of using the full MedDRA structure and using aggregated data. The proposed model, as well as the pre-processing, is implemented in an R-package: BAHAMA.

Why are not There More Bayesian Clinical Trials? Perceived Barriers and Educational Preferences Among Medical Researchers Involved in Drug Development

OBJECTIVE AND BACKGROUND

The clinical trials community has been hesitant to adopt Bayesian statistical methods, which are often more flexible and efficient with more naturally interpretable results than frequentist methods. We aimed to identify self-reported barriers to implementing Bayesian methods and preferences for becoming comfortable with them.

METHODS

We developed a 22-question survey submitted to medical researchers (non-statisticians) from industry, academia, and regulatory agencies. Question areas included demographics, experience, comfort levels with Bayesian analyses, perceived barriers to these analyses, and preferences for increasing familiarity with Bayesian methods.

RESULTS

Of the 323 respondents, most were affiliated with pharmaceutical companies (33.4%), clinical research organizations (29.7%), and regulatory agencies (18.6%). The rest represented academia, medical practice, or other. Over 56% of respondents expressed little to no comfort in interpreting Bayesian analyses. "Insufficient knowledge of Bayesian approaches" was ranked the most important perceived barrier to implementing Bayesian methods by a plurality (48%). Of the approaches listed, in-person training was the most preferred for gaining comfort with Bayesian methods.

CONCLUSIONS

Based on these survey results, we recommend that introductory level training on Bayesian statistics be presented in an in-person workshop that could also be broadcast online with live Q&A. Other approaches such as online training or collaborative projects may be better suited for higher-level trainings where instructors may assume a baseline understanding of Bayesian statistics. Increased coverage of Bayesian methods at medical conferences and medical school trainings would help improve comfort and overcome the substantial knowledge barriers medical researchers face when implementing these methods.

Systematic review of statistical methods for safety data in malaria chemoprevention in pregnancy trials

BACKGROUND

Drug safety assessments in clinical trials present unique analytical challenges. Some of these include adjusting for individual follow-up time, repeated measurements of multiple outcomes and missing data among others. Furthermore, pre-specifying appropriate analysis becomes difficult as some safety endpoints are unexpected. Although existing guidelines such as CONSORT encourage thorough reporting of adverse events (AEs) in clinical trials, they provide limited details for safety data analysis. The limited guidelines may influence suboptimal analysis by failing to account for some analysis challenges above. A typical example where such challenges exist are trials of anti-malarial drugs for malaria prevention during pregnancy. Lack of proper standardized evaluation of the safety of antimalarial drugs has limited the ability to draw conclusions about safety. Therefore, a systematic review was conducted to establish the current practice in statistical analysis for preventive antimalarial drug safety in pregnancy.

METHODS

The search included five databases (PubMed, Embase, Scopus, Malaria in Pregnancy Library and Cochrane Central Register of Controlled Trials) to identify original English articles reporting Phase III randomized controlled trials (RCTs) on anti-malarial drugs for malaria prevention in pregnancy published from January 2010 to July 2019.

RESULTS

Eighteen trials were included in this review that collected multiple longitudinal safety outcomes including AEs. Statistical analysis and reporting of the safety outcomes in all the trials used descriptive statistics; proportions/counts (n = 18, 100%) and mean/median (n = 2, 11.1%). Results presentation included tabular (n = 16, 88.9%) and text description (n = 2, 11.1%). Univariate inferential methods were reported in most trials (n = 16, 88.9%); including Chi square/Fisher's exact test (n = 12, 66.7%), t test (n = 2, 11.1%) and Mann-Whitney/Wilcoxon test (n = 1, 5.6%). Multivariable methods, including Poisson and negative binomial were reported in few trials (n = 3, 16.7%). Assessment of a potential link between missing efficacy data and safety outcomes was not reported in any of the trials that reported efficacy missing data (n = 7, 38.9%).

CONCLUSION

The review demonstrated that statistical analysis of safety data in anti-malarial drugs for malarial chemoprevention in pregnancy RCTs is inadequate. The analyses insufficiently account for multiple safety outcomes potential dependence, follow-up time and informative missing data which can compromise anti-malarial drug safety evidence development, based on the available data.

E-Synthesis: A Bayesian Framework for Causal Assessment in Pharmacosurveillance

Background: Evidence suggesting adverse drug reactions often emerges unsystematically and unpredictably in form of anecdotal reports, case series and survey data. Safety trials and observational studies also provide crucial information regarding the (un-)safety of drugs. Hence, integrating multiple types of pharmacovigilance evidence is key to minimising the risks of harm.

Methods: In previous work, we began the development of a Bayesian framework for aggregating multiple types of evidence to assess the probability of a putative causal link between drugs and side effects. This framework arose out of a philosophical analysis of the Bradford Hill Guidelines. In this article, we expand the Bayesian framework and add “evidential modulators,” which bear on the assessment of the reliability of incoming study results. The overall framework for evidence synthesis, “E-Synthesis”, is then applied to a case study.

Results: Theoretically and computationally, E-Synthesis exploits coherence of partly or fully independent evidence converging towards the hypothesis of interest (or of conflicting evidence with respect to it), in order to update its posterior probability. With respect to other frameworks for evidence synthesis, our Bayesian model has the unique feature of grounding its inferential machinery on a consolidated theory of hypothesis confirmation (Bayesian epistemology), and in allowing any data from heterogeneous sources (cell-data, clinical trials, epidemiological studies), and methods (e.g., frequentist hypothesis testing, Bayesian adaptive trials, etc.) to be quantitatively integrated into the same inferential framework.

Conclusions: E-Synthesis is highly flexible concerning the allowed input, while at the same time relying on a consistent computational system, that is philosophically and statistically grounded. Furthermore, by introducing evidential modulators, and thereby breaking up the different dimensions of evidence (strength, relevance, reliability), E-Synthesis allows them to be explicitly tracked in updating causal hypotheses.

New Insights in Computational Methods for Pharmacovigilance: E-Synthesis, a Bayesian Framework for Causal Assessment

Today's surge of big data coming from multiple sources is raising the stakes that pharmacovigilance has to win, making evidence synthesis a more and more robust approach in the field. In this scenario, many scholars believe that new computational methods derived from data mining will effectively enhance the detection of early warning signals for adverse drug reactions, solving the gauntlets that post-marketing surveillance requires. This article highlights the need for a philosophical approach in order to fully realize a pharmacovigilance 2.0 revolution. A state of the art on evidence synthesis is presented, followed by the illustration of E-Synthesis, a Bayesian framework for causal assessment. Computational results regarding dose-response evidence are shown at the end of this article.

Efficient methods for signal detection from correlated adverse events in clinical trials

It is an important and yet challenging task to identify true signals from many adverse events that may be reported during the course of a clinical trial. One unique feature of drug safety data from clinical trials, unlike data from post-marketing spontaneous reporting, is that many types of adverse events are reported by only very few patients leading to rare events. Due to the limited study size, the p-values of testing whether the rate is higher in the treatment group across all types of adverse events are in general not uniformly distributed under the null hypothesis that there is no difference between the treatment group and the placebo group. A consequence is that typically fewer than 100 α percent of the hypotheses are rejected under the null at the nominal significance level of α . The other challenge is multiplicity control. Adverse events from the same body system may be correlated. There may also be correlations between adverse events from different body systems. To tackle these challenging issues, we develop Monte-Carlo-based methods for the signal identification from patient-reported adverse events in clinical trials. The proposed methodologies account for the rare events and arbitrary correlation structures among adverse events within and/or between body systems. Extensive simulation studies demonstrate that the proposed method can accurately control the family-wise error rate and is more powerful than existing methods under many practical situations. Application to two real examples is provided.

A Bayesian meta-analytic approach for safety signal detection in randomized clinical trials

Background/Aim:

Meta-analyses are frequently performed on adverse event data and are primarily used for improving statistical power to detect safety signals. However, in the evaluation of drug safety for New Drug Applications, simple pooling of adverse event data from multiple clinical trials is still commonly used. We sought to propose a new Bayesian hierarchical meta-analytic approach based on consideration of a hierarchical structure of reported individual adverse event data from multiple randomized clinical trials.

Methods:

To develop our meta-analysis model, we extended an existing three-stage Bayesian hierarchical model by including an additional stage of the clinical trial level in the hierarchical model; this generated a four-stage Bayesian hierarchical model. We applied the proposed Bayesian meta-analysis models to published adverse event data from three premarketing randomized clinical trials of tadalafil and to a simulation study motivated by the case example to evaluate the characteristics of three alternative models.

Results:

Comparison of the results from the Bayesian meta-analysis model with those from Fisher’s exact test after simple pooling showed that 6 out of 10 adverse events were the same within a top 10 ranking of individual adverse events with regard to association with treatment. However, more individual adverse events were detected in the Bayesian meta-analysis model than in Fisher’s exact test under the body system “Musculoskeletal and connective tissue disorders.” Moreover, comparison of the overall trend of estimates between the Bayesian model and the standard approach (odds ratios after simple pooling methods) revealed that the posterior median odds ratios for the Bayesian model for most adverse events shrank toward values for no association. Based on the simulation results, the Bayesian meta-analysis model could balance the false detection rate and power to a better extent than Fisher’s exact test. For example, when the threshold value of the posterior probability for signal detection was set to 0.8, the false detection rate was 41% and power was 88% in the Bayesian meta-analysis model, whereas the false detection rate was 56% and power was 86% in Fisher’s exact test.

Limitations:

Adverse events under the same body system were not necessarily positively related when we used “system organ class” and “preferred term” in the Medical Dictionary for Regulatory Activities as a hierarchical structure of adverse events. For the Bayesian meta-analysis models to be effective, the validity of the hierarchical structure of adverse events and the grouping of adverse events are critical.

Conclusion:

Our proposed meta-analysis models considered trial effects to avoid confounding by trial and borrowed strength from both within and across body systems to obtain reasonable and stable estimates of an effect measure by considering a hierarchical structure of adverse events.

Predictive biomarkers for treatment selection: statistical considerations

Predictive biomarkers are developed for treatment selection to identify patients who are likely to benefit from a particular therapy. This review describes statistical methods and discusses issues in the development of predictive biomarkers to enhance study efficiency for detection of treatment effect on the selected responder patients in clinical studies. The statistical procedure for treatment selection consists of three components: biomarker identification, subgroup selection and clinical utility assessment. Major statistical issues discussed include biomarker designs, procedures to identify predictive biomarkers, classification models for subgroup selection, subgroup analysis and multiple testing for clinical utility assessment and evaluation.

Causal assessment of pharmaceutical treatments: why standards of evidence should not be the same for benefits and harms?

It is increasingly acknowledged both among epidemiologists and regulators that the assessment of pharmaceutical harm requires specific methodological approaches that cannot simply duplicate those developed for testing efficacy. However, this intuition lacks sound epistemic bases and delivers ad hoc advice. This paper explains why the same methods of scientific inference do not fare equally well for efficacy and safety assessment by tracing them back to their epistemic foundations. To illustrate this, Cartwright's distinction into clinching and vouching methods is adopted and a series of reasons is provided for preferring the latter to the former: (1) the need to take into account all available knowledge and integrate it with incoming data; (2) the awareness that a latent unknown risk may always change the safety profile of a given drug (precautionary principle); (3) cumulative learning over time; (4) requirement of probabilistic causal assessment to allow decision under uncertainty; (5) impartiality; and (6) limited and local information provided by randomised controlled trials. Subsequently, the clinchers/vouchers distinction is applied to a case study concerning the debated causal association between paracetamol and asthma. This study illustrates the tension between implicit epistemologies adopted in evaluating evidence and causality; furthermore, it also shows that discounting causal evidence may be a result of unacknowledged low priors or lack of valid alternative options. We conclude with a presentation of the changing landscape in pharmacology and the trend towards an increased use of Bayesian tools for assessment of harms.

Identification of drug-induced toxicity biomarkers for treatment determination

Drug-induced organ toxicity (DIOT) that leads to the removal of marketed drugs or termination of candidate drugs has been a leading concern for regulatory agencies and pharmaceutical companies. In safety studies, the genomic assays are conducted after the treatment so that drug-induced adverse effects can occur. Two types of biomarkers are observed: biomarkers of susceptibility and biomarkers of response. This paper presents a statistical model to distinguish two types of biomarkers and procedures to identify susceptible subpopulations. The biomarkers identified are used to develop classification model to identify susceptible subpopulation. Two methods to identify susceptibility biomarkers were evaluated in terms of predictive performance in subpopulation identification, including sensitivity, specificity, and accuracy. Method 1 considered the traditional linear model with a variable-by-treatment interaction term, and Method 2 considered fitting a single predictor variable model using only treatment data. Monte Carlo simulation studies were conducted to evaluate the performance of the two methods and impact of the subpopulation prevalence, probability of DIOT, and sample size on the predictive performance. Method 2 appeared to outperform Method 1, which was due to the lack of power for testing the interaction effect. Important statistical issues and challenges regarding identification of preclinical DIOT biomarkers were discussed. In summary, identification of predictive biomarkers for treatment determination highly depends on the subpopulation prevalence. When the proportion of susceptible subpopulation is 1% or less, a very large sample size is needed to ensure observing sufficient number of DIOT responses for biomarker and/or subpopulation identifications.

Guidance on the implementation and reporting of a drug safety Bayesian network meta-analysis

The Drug Information Association Bayesian Scientific Working Group (BSWG) was formed in 2011 with a vision to ensure that Bayesian methods are well understood and broadly utilized for design and analysis and throughout the medical product development process, and to improve industrial, regulatory, and economic decision making. The group, composed of individuals from academia, industry, and regulatory, has as its mission to facilitate the appropriate use and contribute to the progress of Bayesian methodology. In this paper, the safety sub-team of the BSWG explores the use of Bayesian methods when applied to drug safety meta-analysis and network meta-analysis. Guidance is presented on the conduct and reporting of such analyses. We also discuss different structural model assumptions and provide discussion on prior specification. The work is illustrated through a case study involving a network meta-analysis related to the cardiovascular safety of non-steroidal anti-inflammatory drugs.