Statistical modeling and prediction of clinical trial recruitment

The time-dependent Poisson-gamma model in practice: Recruitment forecasting in HIV trials

Despite a growing body of literature in the area of recruitment modeling for multicenter studies, in practice, statistical models to predict enrollments are rarely used and when they are, they often rely on unrealistic assumptions. The time-dependent Poisson-Gamma model (tPG) is a recently developed flexible methodology which allows analysts to predict recruitments in an ongoing multicenter trial, and its performance has been validated on data from a cohort study. In this article, we illustrate and further validate the tPG model on recruitment data from randomized controlled trials. Additionally, in the appendix, we provide a practical and easy to follow guide to its implementation via the tPG R package. To validate the model, we show the predictive performance of the proposed methodology in forecasting the recruitment process of two HIV vaccine trials conducted by the HIV Vaccine Trials Network in multiple Sub-Saharan countries.

Freely accessible software for recruitment prediction and recruitment monitoring of clinical trials: A systematic review

Background

The successful completion of clinical trials ultimately depends on realistic recruitment predictions. Statistical methods for recruitment prediction implemented in a free-of-charge open-source software could be routinely used by researchers worldwide to design clinical trials. However, the availability of such software implementations is currently unclear.

Methods

Two independent reviewers conducted a systematic review following PRISMA guidelines. Eligible articles included English publications focused on statistical methods for recruitment prediction and monitoring that referred to software implementations. The list of articles retrieved from well-established data bases was enriched by backtracking of references provided by eligible articles. The current software availability and open-source status were tabulated.

Results

We found 21 eligible articles, 7 of which (33 %) provide freely accessible software. Ultimately, only one article provides a link to an easy-to-comprehend, well-documented, and currently directly applicable free-of-charge open-source software. The lack of availability is mainly caused by blocked access and outdated links.

Conclusions

While several software implementations exist for recruitment prediction, only a small fraction is freely accessible. These results highlight the need for future efforts to achieve free access to well-documented software implementations supporting researchers in routinely using statistical methods to arrive at realistic recruitment predictions in clinical trials.

Enrollment forecast for clinical trials at the portfolio planning phase based on site-level historical data

An accurate forecast of a clinical trial enrollment timeline at the planning phase is of great importance to both corporate strategic planning and trial operational excellence. The naive approach often calculates an average enrollment rate from historical data and generates an inaccurate prediction based on a linear trend with the average rate. Under the traditional framework of a Poisson-Gamma model, site activation delays are often modeled with either fixed initiation time or a simple random distribution while incorporating the user-provided site planning information to achieve good forecast accuracy. However, such user-provided information is not available at the early portfolio planning stage. We present a novel statistical approach based on generalized linear mixed-effects models and the use of non-homogeneous Poisson processes through the Bayesian framework to model the country initiation, site activation, and subject enrollment sequentially in a systematic fashion. We validate the performance of our proposed enrollment modeling framework based on a set of 25 preselected studies from four therapeutic areas. Our modeling framework shows a substantial improvement in prediction accuracy in comparison to the traditional statistical approach. Furthermore, we show that our modeling and simulation approach calibrates the data variability appropriately and gives correct coverage rates for prediction intervals of various nominal levels. Finally, we demonstrate the use of our approach to generate the predicted enrollment curves through time with confidence bands overlaid.

Enrollment Forecast for Clinical Trials at the Planning Phase with Study-Level Historical Data

Given progressive developments and demands on clinical trials, accurate enrollment timeline forecasting is increasingly crucial for both strategic decision-making and trial execution excellence. Naïve approach assumes flat rates on enrollment using average of historical data, while traditional statistical approach applies simple Poisson-Gamma model using time-invariant rates for site activation and subject recruitment. Both of them are lack of non-trivial factors such as time and location. We propose a novel two-segment statistical approach based on Quasi-Poisson regression for subject accrual rate and Poisson process for subject enrollment and site activation. The input study-level data are publicly accessible and it can be integrated with historical study data from user's organization to prospectively predict enrollment timeline. The new framework is neat and accurate compared to preceding works. We validate the performance of our proposed enrollment model and compare the results with other frameworks on 7 curated studies.

Designing a phase-III time-to-event clinical trial using a modified sample size formula and Poisson-Gamma model for subject accrual that accounts for the lag in site initiation using the PERT distribution

A priori estimation of sample size and subject accrual in multi-site, time-to-event clinical trials is often challenging. Such trials are powered based on the number of events needed to detect a clinically significant difference. Sample size based on number of events relates to the expected duration of observation time for each subject. Temporal patterns in site initiation and subject enrollment ultimately affect when subjects can be accrued into the study. Lag times are common as the site start-up process optimizes, resulting in delays that may curtail observational follow-up and therefore undermine power. The proposed method introduces a Program Evaluation and Review Technique (PERT) model into the sample size estimation which accounts for the lag in site start-up. Additionally, a PERT model is introduced into a Poisson-Gamma subject accrual model to predict the quantity of study sites needed. The introduction of the PERT model provides greater flexibility in both a priori power assessment and planning the number of sites, as it specifically allows for the inclusion of anticipated delays in site start-up time. This model results in minimal power loss even when PERT distribution inputs are misspecified compared to the traditional assumption of simultaneous start-up for all sites. Together these updated formulations for sample size and subject accrual models offer an improved method for designing a multi-site time-to-event clinical trial that accounts for a flexible site start-up process.

A time-dependent Poisson-Gamma model for recruitment forecasting in multicenter studies

Forecasting recruitments is a key component of the monitoring phase of multicenter studies. One of the most popular techniques in this field is the Poisson-Gamma recruitment model, a Bayesian technique built on a doubly stochastic Poisson process. This approach is based on the modeling of enrollments as a Poisson process where the recruitment rates are assumed to be constant over time and to follow a common Gamma prior distribution. However, the constant-rate assumption is a restrictive limitation that is rarely appropriate for applications in real studies. In this paper, we illustrate a flexible generalization of this methodology which allows the enrollment rates to vary over time by modeling them through B-splines. We show the suitability of this approach for a wide range of recruitment behaviors in a simulation study and by estimating the recruitment progression of the Canadian Co-infection Cohort.

Modeling time-varying recruitment rates in multicenter clinical trials

Multicenter phase II/III clinical trials are large-scale operations that often include hundreds of recruiting centers in several countries. Therefore, the operational aspects of a trial must be thoroughly planned and closely monitored to ensure better oversight and study conduct. Predicting patient recruitment plays a pivotal role in trial monitoring as it informs how many people are expected to be recruited on a given day. As such, study teams may rely on predictions to assess progress and detect any deviations from the original plan that might put the study and potentially, patients at risk. Recruitment predictions are often based on a Poisson-Gamma model that assumes that centers have a constant recruitment rate throughout the trial. The model has useful features and has extensively been used, yet its main assumption is often unrealistic. A nonhomogeneous Poisson process has been recently proposed that can incorporate time-varying recruitment rates. In this work, we predict patient recruitment using both approaches and assess the impact of said assumption in a real-world setting where studies may not necessarily have constant center-specific recruitment rates. The paper showcases experience from modeling recruitment in trials sponsored by AstraZeneca between 2005 and 2018. In these data, there is evidence of time-varying recruitment rates. The predictive performance of models that account for both constant and time-varying recruitment effects is presented. Following a descriptive analysis of data, we assess model performance and investigate the impact of model misspecification.

A simple and robust model for enrollment projection in clinical trials

Enrollment projection in clinical trials is a topic gaining attention in the statistics literature in recent years. A number of methods have been proposed in this area. Some approaches are sophisticated but complicated to implement. We aim to implement a simple and robust empiric Bayes Poisson Gamma model (PGM) that is suitable for practical use. We assume a constant and site-specific underlying enrollment rate over time, which comes from a common Gamma distribution. Choice of prior parameters is data driven. We tested the proposed PGM in a simulation study as well as a number of oncology trials with various enrollment patterns, which yield satisfactory results. Compared to a flexible nonparametric model (Zhang and Long, 2010), the PGM is associated with a narrower credible interval as a result of parametric assumptions. However, the model prediction may be off when the assumptions are substantially violated.

Real time monitoring and prediction of time to endpoint maturation in clinical trials

In clinical trials, almost all key milestone dates can be defined in terms of time to endpoint maturation (TTEM). The real time monitoring and accurate prediction of TTEM have a significant impact on clinical trial planning and execution and can bring significant value to clinical trial practitioners. TTEM is defined as the time to achieve or observe a certain number or percentage of some endpoint of interest. It is a combination of time to site initiation, time to subject enrollment after site initiation and time to event of interest after subject enrollment. To better predict TTEM during the trial, the future site initiation and subject enrollment have to be taken into account while predicting the number of events. In this article, we propose a novel simulation-based framework combining time to site initiation, time to subject enrollment and time to event in order to predict TTEM. A nonhomogeneous Poisson process with a quadratic time-varying rate function is used to model site initiation and subject enrollment and more advanced time to event models had been explored and integrated on top of them, such as Weibull, piecewise exponential, and model averaging which is equivalent to a Bayesian model selection strategy. To evaluate the predictive performance of the proposed methodology, we conducted extensive simulations and applied the methodology to 14 randomly selected real oncology phase 2 and phase 3 studies in both solid tumor and hematology with a total 31 study-endpoint combinations. The predictive performance of the proposed methodology was then compared with popular and commonly available commercial software, for example, East (Cytel, Cambridge, MA, USA). From both simulation and real data, the proposed methodology can significantly improve the prediction accuracy by up to 54% compared to the commonly available method.

Prediction of clinical trial enrollment rates

Clinical trials represent a critical milestone of translational and clinical sciences. However, poor recruitment to clinical trials has been a long standing problem affecting institutions all over the world. One way to reduce the cost incurred by insufficient enrollment is to minimize initiating trials that are most likely to fall short of their enrollment goal. Hence, the ability to predict which proposed trials will meet enrollment goals prior to the start of the trial is highly beneficial. In the current study, we leveraged a data set extracted from ClinicalTrials.gov that consists of 46,724 U.S. based clinical trials from 1990 to 2020. We constructed 4,636 candidate predictors based on data collected by ClinicalTrials.gov and external sources for enrollment rate prediction using various state-of-the-art machine learning methods. Taking advantage of a nested time series cross-validation design, our models resulted in good predictive performance that is generalizable to future data and stable over time. Moreover, information content analysis revealed the study design related features to be the most informative feature type regarding enrollment. Compared to the performance of models built with all features, the performance of models built with study design related features is only marginally worse (AUC = 0.78 ± 0.03 vs. AUC = 0.76 ± 0.02). The results presented can form the basis for data-driven decision support systems to assess whether proposed clinical trials would likely meet their enrollment goal.

Concept and development of an interactive tool for trial recruitment planning and management

Background

Predicting and monitoring recruitment in large, complex trials is essential to ensure appropriate resource management and budgeting. In a novel partnership between clinical trial investigators of the South African Medical Research Council and industrial engineers from the Stellenbosch University Health Systems Engineering and Innovation Hub, we developed a trial recruitment tool (TRT). The objective of the tool is to serve as a computerised decisions-support system to aid the planning and management phases of the trial recruitment process.

Method

The specific requirements of the TRT were determined in several workshops between the partners. A Poisson process simulation model was formulated and incorporated in the TRT to predict the recruitment duration. The assumptions underlying the model were made in consultation with the trial team at the start of the project and were deemed reasonable. Real-world data extracted from a current cluster trial, Project MIND, based in 24 sites in South Africa was used to verify the simulation model and to develop the monitoring component of the TRT.

Results

The TRT comprises a planning and monitoring component. The planning component generates different trial scenarios for predicted trial recruitment duration based on user inputs, e.g. number of sites, initiation delays. The monitoring component uses and analyses the data retrieved from the trial management information system to generate different levels of information, displayed visually on an interactive, user-friendly dashboard. Users can analyse the results at trial or site level, changing input parameters to see the resultant effect on the duration of trial recruitment.

Conclusion

This TRT is an easy-to-use tool that assists in the management of the trial recruitment process. The TRT has potential to expedite improved management of clinical trials by providing the appropriate information needed for the planning and monitoring of the trial recruitment phase. This TRT extends prior tools describing historic recruitment only to using historic data to predict future recruitment. The broader project demonstrates the value of collaboration between clinicians and engineers to optimise their respective skillsets.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13063-021-05112-z.

Recruitment prediction for multicenter clinical trials based on a hierarchical Poisson-gamma model: Asymptotic analysis and improved intervals

We analyze predictions of future recruitment to a multicenter clinical trial based on a maximum-likelihood fitting of a commonly used hierarchical Poisson-gamma model for recruitments at individual centers. We consider the asymptotic accuracy of quantile predictions in the limit as the number of recruitment centers grows large and find that, in an important sense, the accuracy of the quantiles does not improve as the number of centers increases. When predicting the number of further recruits in an additional time period, the accuracy degrades as the ratio of the additional time to the census time increases, whereas when predicting the amount of additional time to recruit a further n • + patients, the accuracy degrades as the ratio of n • + to the number recruited up to the census period increases. Our analysis suggests an improved quantile predictor. Simulation studies verify that the predicted pattern holds for typical recruitment scenarios in clinical trials and verify the much improved coverage properties of prediction intervals obtained from our quantile predictor. In the process of extending the applicability of our methodology, we show that in terms of the accuracy of all integer moments it is always better to approximate the sum of independent gamma random variables by a single gamma random variable matched on the first two moments than by the moment-matched Gaussian available from the central limit theorem.

Interim recruitment prediction for multi-center clinical trials

We introduce a general framework for monitoring, modeling, and predicting the recruitment to multi-center clinical trials. The work is motivated by overly optimistic and narrow prediction intervals produced by existing time-homogeneous recruitment models for multi-center recruitment. We first present two tests for detection of decay in recruitment rates, together with a power study. We then introduce a model based on the inhomogeneous Poisson process with monotonically decaying intensity, motivated by recruitment trends observed in oncology trials. The general form of the model permits adaptation to any parametric curve-shape. A general method for constructing sensible parameter priors is provided and Bayesian model averaging is used for making predictions which account for the uncertainty in both the parameters and the model. The validity of the method and its robustness to misspecification are tested using simulated datasets. The new methodology is then applied to oncology trial data, where we make interim accrual predictions, comparing them to those obtained by existing methods, and indicate where unexpected changes in the accrual pattern occur.

Modelling the patient accrual with truncated Gaussian mixture distribution for the accurate estimation of sample size in survival trials

In survival trials with fixed trial length, the patient accrual rate has a significant impact on the sample size estimation or equivalently, on the power of trials. A larger sample size is required for the staggered patient entry. During enrollment, the patient accrual rate changes with the recruitment publicity effect, disease incidence and many other factors and fluctuations of the accrual rate occur frequently. However, the existing accrual models are either over-simplified for the constant rate assumption or complicated in calculation for the subdivision of the accrual period. A more flexible accrual model is required to represent the fluctuant patient accrual rate for accurate sample size estimation. In this paper, inspired by the flexibility of the Gaussian mixture distribution in approximating continuous densities, we propose the truncated Gaussian mixture distribution accrual model to represent different variations of accrual rate by different parameter configurations. The sample size calculation formula and the parameter setting of the proposed accrual model are discussed further.

Statistical models to predict recruitment in clinical trials were rarely used by statisticians in UK and European networks

OBJECTIVE

Identify the current practice for recruitment prediction and monitoring within clinical trials.

STUDY DESIGN AND SETTING

Chief investigators (CIs) were surveyed to identify data sources and adjustments made to support recruitment prediction. Statisticians were surveyed to determine methods and adjustments used when predicting and monitoring recruitment. Participants were identified from the National Institute for Health Research recently funded studies, the UK Clinical Research Collaboration registered Clinical Trial Units network or by the European Clinical Research Infrastructure Network.

RESULTS

A total of 51 CIs (UK = 32, ECRIN = 19) and 104 statisticians (UK = 51, ECRIN = 53) were contacted. Response rates varied (CIs UK = 53% ECRIN = 32%; statisticians UK = 98% ECRIN = 36%). Multiple data sources are used to support recruitment rates, most commonly audit data from multiple sites. Variation in individual site recruitment rates are frequently incorporated, but staggered site openings were featured more commonly among UK respondents. Simple prediction methods are preferred to rarely used statistical models. Lack of familiarity with statistical methods are barriers to their use with evidence needed to justify the time required to support their implementation.

CONCLUSION

Simplistic methods will continue as the mainstay of prediction; however, generation of evidence supporting the benefits of complex statistical models should promote their implementations. Multiple data sources to support recruitment prediction are being used, and further work on the quality of these data is needed. Pressure to be optimistic about recruitment rates for the trial to be attractive to funders was felt by a sizable minority.