Statistical Practices of Safety Monitoring: An Industry Survey

Visualising harms in publications of randomised controlled trials: consensus and recommendations

OBJECTIVE

To improve communication of harm in publications of randomised controlled trials via the development of recommendations for visually presenting harm outcomes.

DESIGN

Consensus study.

SETTING

15 clinical trials units registered with the UK Clinical Research Collaboration, an academic population health department, Roche Products, and The BMJ.

PARTICIPANTS

Experts in clinical trials: 20 academic statisticians, one industry statistician, one academic health economist, one data graphics designer, and two clinicians.

MAIN OUTCOME

measures A methodological review of statistical methods identified visualisations along with those recommended by consensus group members. Consensus on visual recommendations was achieved (at least 60% of the available votes) over a series of three meetings with participants. The participants reviewed and critically appraised candidate visualisations against an agreed framework and voted on whether to endorse each visualisation. Scores marginally below this threshold (50-60%) were revisited for further discussions and votes retaken until consensus was reached.

RESULTS

28 visualisations were considered, of which 10 are recommended for researchers to consider in publications of main research findings. The choice of visualisations to present will depend on outcome type (eg, binary, count, time-to-event, or continuous), and the scenario (eg, summarising multiple emerging events or one event of interest). A decision tree is presented to assist trialists in deciding which visualisations to use. Examples are provided of each endorsed visualisation, along with an example interpretation, potential limitations, and signposting to code for implementation across a range of standard statistical software. Clinician feedback was incorporated into the explanatory information provided in the recommendations to aid understanding and interpretation.

CONCLUSIONS

Visualisations provide a powerful tool to communicate harms in clinical trials, offering an alternative perspective to the traditional frequency tables. Increasing the use of visualisations for harm outcomes in clinical trial manuscripts and reports will provide clearer presentation of information and enable more informative interpretations. The limitations of each visualisation are discussed and examples of where their use would be inappropriate are given. Although the decision tree aids the choice of visualisation, the statistician and clinical trial team must ultimately decide the most appropriate visualisations for their data and objectives. Trialists should continue to examine crude numbers alongside visualisations to fully understand harm profiles.

Comparison of statistical methods for the analysis of recurrent adverse events in the presence of non-proportional hazards and unobserved heterogeneity: a simulation study

Background

In preventive drug trials such as intermittent preventive treatment for malaria prevention during pregnancy (IPTp), where there is repeated treatment administration, recurrence of adverse events (AEs) is expected. Challenges in modelling the risk of the AEs include accounting for time-to-AE and within-patient-correlation, beyond the conventional methods. The correlation comes from two sources; (a) individual patient unobserved heterogeneity (i.e. frailty) and (b) the dependence between AEs characterised by time-dependent treatment effects. Potential AE-dependence can be modelled via time-dependent treatment effects, event-specific baseline and event-specific random effect, while heterogeneity can be modelled via subject-specific random effect. Methods that can improve the estimation of both the unobserved heterogeneity and treatment effects can be useful in understanding the evolution of risk of AEs, especially in preventive trials where time-dependent treatment effect is expected.

Methods

Using both a simulation study and the Chloroquine for Malaria in Pregnancy (NCT01443130) trial data to demonstrate the application of the models, we investigated whether the lognormal shared frailty models with restricted cubic splines and non-proportional hazards (LSF-NPH) assumption can improve estimates for both frailty variance and treatment effect compared to the conventional inverse Gaussian shared frailty model with proportional hazard (ISF-PH), in the presence of time-dependent treatment effects and unobserved patient heterogeneity. We assessed the bias, precision gain and coverage probability of 95% confidence interval of the frailty variance estimates for the models under varying known unobserved heterogeneity, sample sizes and time-dependent effects.

Results

The ISF-PH model provided a better coverage probability of 95% confidence interval, less bias and less precise frailty variance estimates compared to the LSF-NPH models. The LSF-NPH models yielded unbiased hazard ratio estimates at the expense of imprecision and high mean square error compared to the ISF-PH model.

Conclusion

The choice of the shared frailty model for the recurrent AEs analysis should be driven by the study objective. Using the LSF-NPH models is appropriate if unbiased hazard ratio estimation is of primary interest in the presence of time-dependent treatment effects. However, ISF-PH model is appropriate if unbiased frailty variance estimation is of primary interest.

Trial registration

ClinicalTrials.gov; NCT01443130

Improving the analysis of adverse event data in randomised controlled trials

Analysing treatment harm is vital but problematic with the relatively small sample sizes afforded in randomised controlled trials (RCTs). Good analysis practice for efficacy outcomes are well established but there has been minimal progress for the analysis of adverse events (AEs). In this commentary we examine four key issues for AE analysis. Namely, why harm data in RCTs is undervalued, why AE analysis is difficult, what aspects of current analysis practice are unsatisfactory, and the challenges for selection and interpretation of AEs results in publications. We discuss how the value of harm data from RCTs could be better realised by reframing the research question to one for detecting signals of adverse reactions. We align established good statistical practice to current unsatisfactory practice. We encourage use of Bayesian analyses to enable cumulative assessment of harm across trial research phases and discourage selecting AEs to report based on arbitrary rules. We propose comprehendible summaries to be based on core outcome sets, serious and pre-specified events, and events leading to discontinuation. Analysis of AEs in contemporary clinical trials needs attention to progress. In the following we have outlined immediate, mid and longer-term strategies for trialists to adopt to support a change in practice.

Reducing Events of Noncompliance in Neurology Human Subjects Research: the Effect of Human Subjects Research Protection Training and Site Initiation Visits

In an effort to minimize protocol noncompliance in neurological research studies that can potentially compromise patient safety, delay completion of the study, and result in premature termination and added costs, we determined the effect of investigator trainings and site initiation visits (SIVs) on the occurrence of noncompliance events. Results of protocol audits conducted at the National Institute of Neurological Disorders and Stroke from 2003 to 2019 on 97 research protocols were retrospectively analyzed. Based on the depth of auditing and provision of investigator research training, audit data were separated into four arms: 1) Early Period, 2003 to 2012; 2) Middle Period, 2013 to 2016; and Late Period, 2017 to 2019, further divided into 3) Late Period without SIVs; and 4) Late Period with SIVs. Events of noncompliance were classified by the type of protocol deviation, the category, and the cause. In total, 952 events occurred across 1080 participants. Protocols audited during the Middle Period, compared to the Early Period, showed a decrease in the percentage of protocols with at least 1 noncompliance event. Protocols with SIVs had a further decrease in major, minor, procedural, eligibility, and policy events. Additionally, protocols audited during the Early Period had on average 0.46 major deviations per participant, compared to 0.26 events in protocols audited during the Middle Period, and 0.08 events in protocols audited during the Late Period with SIVs. Protocol deviations and noncompliance events in neurological clinical trials can be reduced by targeted investigator trainings and SIVs. These measures have major impacts on the integrity, safety, and effectiveness of human subjects research in neurology.

Aggregate Safety Assessment Planning for the Drug Development Life-Cycle

The Program Safety Analysis Plan (PSAP) was proposed previously as a tool to proactively plan for integrated analyses of product safety data. Building on the PSAP and taking into consideration the evolving regulatory landscape, the Drug Information Association-American Statistical Association (DIA-ASA) Interdisciplinary Safety Evaluation scientific working group herein proposes the Aggregate Safety Assessment Plan (ASAP) process. The ASAP evolves over a product's life-cycle and promotes interdisciplinary, systematic safety planning as well as ongoing data review and characterization of the emerging product safety profile. Objectives include alignment on the safety topics of interest, identification of safety knowledge gaps, planning for aggregate safety evaluation of the clinical trial data and preparing for safety communications. The ASAP seeks to tailor the analyses for a drug development program while standardizing the analyses across studies within the program. The document is intended to be modular and flexible in nature, depending on the program complexity, phase of development and existing sponsor processes. Implementation of the ASAP process will facilitate early safety signal detection, improve characterization of product risks, harmonize safety messaging, and inform program decision-making.

Understanding current practice, identifying barriers and exploring priorities for adverse event analysis in randomised controlled trials: an online, cross-sectional survey of statisticians from academia and industry

OBJECTIVES

To gain a better understanding of current adverse event (AE) analysis practices and the reasons for the lack of use of sophisticated statistical methods for AE data analysis in randomised controlled trials (RCTs), with the aim of identifying priorities and solutions to improve practice.

DESIGN

A cross-sectional, online survey of statisticians working in clinical trials, followed up with a workshop of senior statisticians working across the UK.

PARTICIPANTS

We aimed to recruit into the survey a minimum of one statistician from each of the 51 UK Clinical Research Collaboration registered clinical trial units (CTUs) and industry statisticians from both pharmaceuticals and clinical research organisations.

OUTCOMES

To gain a better understanding of current AE analysis practices, measure awareness of specialist methods for AE analysis and explore priorities, concerns and barriers when analysing AEs.

RESULTS

Thirty-eight (38/51; 75%) CTUs, 5 (5/7; 71%) industry and 21 attendees at the 2019 Promoting Statistical Insights Conference participated in the survey. Of the 64 participants that took part, 46 participants were classified as public sector participants and 18 as industry participants. Participants indicated that they predominantly (80%) rely on subjective comparisons when comparing AEs between treatment groups. Thirty-eight per cent were aware of specialist methods for AE analysis, but only 13% had undertaken such analyses. All participants believed guidance on appropriate AE analysis and 97% thought training specifically for AE analysis is needed. These were both endorsed as solutions by workshop participants.

CONCLUSIONS

This research supports our earlier work that identified suboptimal AE analysis practices in RCTs and confirms the underuse of more sophisticated AE analysis approaches. Improvements are needed, and further research in this area is required to identify appropriate statistical methods. This research provides a unanimous call for the development of guidance, as well as training on suitable methods for AE analysis to support change.