Comparison of Model Averaging and Model Selection in Dose Finding Trials Analyzed by Nonlinear Mixed Effect Models

Development and comparison of model-integrated evidence approaches for bioequivalence studies with pharmacokinetic end points

By applying nonlinear mixed-effect (NLME) models, model-integrated evidence (MIE) approaches are able to analyze bioequivalence (BE) data with pharmacokinetic end points that have sparse sampling, which is problematic for non-compartmental analysis (NCA). However, MIE approaches may suffer from inflation of type I error due to underestimation of parameter uncertainty and to the assumption of asymptotic normality. In this study, we developed a MIE BE analysis method that is based on a pre-defined model and consists of several steps including model fitting, uncertainty assessment, simulation, and BE determination. The presented MIE approach has several improvements compared with the previously reported model-integrated methods: (1) treatment, sequence, and period effects are only added to absorption parameters (such as relative bioavailability and rate of absorption) instead of all PK parameters; (2) a simulation step is performed to generate confidence intervals of the pharmacokinetic metrics for BE assessment; and (3) in an effort to maintain type I error, two more advanced parameter uncertainty evaluation approaches are explored, a nonparametric (case resampling) bootstrap, and sampling importance resampling (SIR). To evaluate the developed method and compare the uncertainty assessment methods, simulation experiments were performed for BE studies using a two-way crossover design with different amounts of information (sparse to rich designs) and levels of variability. Based on the simulation results, the method using SIR for parameter uncertainty quantification controls type I error at the nominal level of 0.05 (i.e., the significance level set for BE evaluation) even for studies with small sample size and/or sparse sampling. As expected, our MIE approach for BE assessment exhibited higher power than the NCA-based method, especially as the data becomes sparser and/or more variable.

Robust and Adaptive Two-stage Designs in Nonlinear Mixed Effect Models

To get informative studies for nonlinear mixed effect models (NLMEM), design optimization can be performed based on Fisher Information Matrix (FIM) using the D-criterion. Its computation requires knowledge about models and parameters, which are often prior guesses. Thus, adaptive designs composed of several stages may be used. Robust approach can also be used to account for various candidate models. In the estimation step of a given stage, model selection (MS) or model averaging (MA) can be performed. In this work we propose a new two-stage adaptive design strategy, based on the robust expected FIM and MA over several candidate models. The methodology is applied to a clinical trial simulation in ophthalmology to optimize doses and time measurements. A set of dose-response candidate models is defined, and one-stage designs are compared to two-stage 50/50 designs (i.e., each stage performed with half of the available subjects), using either local optimal design or robust design, and performing analysis with one model, MS or MA. Performing a two-stage design with MS at the interim analysis can correct the choice of a wrong model for designing the first stage. Overall, starting from a robust design (1- or 2-stage) is valuable and leads to reasonable bias and precision. The proposed robust adaptive design strategy is a new tool to design longitudinal studies that could be used in different therapeutic areas.

Comparison of Seven Non-Linear Mixed Effect Model-Based Approaches to Test for Treatment Effect

Analyses of longitudinal data with non-linear mixed-effects models (NLMEM) are typically associated with high power, but sometimes at the cost of inflated type I error. Approaches to overcome this problem were published recently, such as model-averaging across drug models (MAD), individual model-averaging (IMA), and combined Likelihood Ratio Test (cLRT). This work aimed to assess seven NLMEM approaches in the same framework: treatment effect assessment in balanced two-armed designs using real natural history data with or without the addition of simulated treatment effect. The approaches are MAD, IMA, cLRT, standard model selection (STDs), structural similarity selection (SSs), randomized cLRT (rcLRT), and model-averaging across placebo and drug models (MAPD). The assessment included type I error, using Alzheimer's Disease Assessment Scale-cognitive (ADAS-cog) scores from 817 untreated patients and power and accuracy in the treatment effect estimates after the addition of simulated treatment effects. The model selection and averaging among a set of pre-selected candidate models were driven by the Akaike information criteria (AIC). The type I error rate was controlled only for IMA and rcLRT; the inflation observed otherwise was explained by the placebo model misspecification and selection bias. Both IMA and rcLRT had reasonable power and accuracy except under a low typical treatment effect.

Optimization of trial duration to predict long-term HbA1c change with therapy: A pharmacometrics simulation-based evaluation

Glycated hemoglobin (HbA1c) is the main biomarker of diabetes drug development. However, because of its delayed turnover, trial duration is rarely shorter than 12 weeks, and being able to predict long-term HbA1c with precision using data from shorter studies would be beneficial. The feasibility of reducing study duration was therefore investigated in this study, assuming a model-based analysis. The aim was to investigate the predictive performance of 24- and 52-week extrapolations using data from up to 4, 6, 8 or 12 weeks, with six previously published pharmacometric models of HbA1c. Predictive performance was assessed through simulation-based dose-response predictions and model averaging (MA) with two hypothetical drugs. Results were consistent across the methods of assessment, with MA supporting the results derived from the model-based framework. The models using mean plasma glucose (MPG) or nonlinear fasting plasma glucose (FPG) effect, driving the HbA1c formation, showed good predictive performance despite a reduced study duration. The models, using the linear effect of FPG to drive the HbA1c formation, were sensitive to the limited amount of data in the shorter studies. The MA with bootstrap demonstrated strongly that a 4-week study duration is insufficient for precise predictions of all models. Our findings suggest that if data are analyzed with a pharmacometric model with MPG or FPG with a nonlinear effect to drive HbA1c formation, a study duration of 8 weeks is sufficient with maintained accuracy and precision of dose-response predictions.

Impact of model misspecification on model-based tests in PK studies with parallel design: real case and simulation studies

This article evaluates the performance of pharmacokinetic (PK) equivalence testing between two formulations of a drug through the Two-One Sided Tests (TOST) by a model-based approach (MB-TOST), as an alternative to the classical non-compartmental approach (NCA-TOST), for a sparse design with a few time points per subject. We focused on the impact of model misspecification and the relevance of model selection for the reference data. We first analysed PK data from phase I studies of gantenerumab, a monoclonal antibody for the treatment of Alzheimer’s disease. Using the original rich sample data, we compared MB-TOST to NCA-TOST for validation. Then, the analysis was repeated on a sparse subset of the original data with MB-TOST. This analysis inspired a simulation study with rich and sparse designs. With rich designs, we compared NCA-TOST and MB-TOST in terms of type I error and study power. With both designs, we explored the impact of misspecifying the model on the performance of MB-TOST and adding a model selection step. Using the observed data, the results of both approaches were in general concordance. MB-TOST results were robust with sparse designs when the underlying PK structural model was correctly specified. Using the simulated data with a rich design, the type I error of NCA-TOST was close to the nominal level. When using the simulated model, the type I error of MB-TOST was controlled on rich and sparse designs, but using a misspecified model led to inflated type I errors. Adding a model selection step on the reference data reduced the inflation. MB-TOST appears as a robust alternative to NCA-TOST, provided that the PK model is correctly specified and the test drug has the same PK structural model as the reference drug.

Improved Confidence in a Confirmatory Stage by Application of Item-Based Pharmacometrics Model: Illustration with a Phase III Active Comparator-Controlled Trial in COPD Patients

PURPOSE

The current study aimed to illustrate how a non-linear mixed effect (NLME) model-based analysis may improve confidence in a Phase III trial through more precise estimates of the drug effect.

METHODS

The FULFIL clinical trial was a Phase III study that compared 24 weeks of once daily inhaled triple therapy with twice daily inhaled dual therapy in patients with chronic obstructive pulmonary disease (COPD). Patient reported outcome data, obtained by using The Evaluating Respiratory Symptoms in COPD (E-RS:COPD) questionnaire, from the FULFIL study were analyzed using an NLME item-based response theory model (IRT). The change from baseline (CFB) in E-RS:COPD total score over 4-week intervals for each treatment arm was obtained using the IRT and compared with published results obtained with a mixed model repeated measures (MMRM) analysis.

RESULTS

The IRT included a graded response model characterizing item parameters and a Weibull function combined with an offset function to describe the COPD symptoms-time course in patients receiving either triple therapy (n = 907) or dual therapy (n = 894). The IRT improved precision of the estimated drug effect compared to MMRM, resulting in a sample size of at least 3.64 times larger for the MMRM analysis to achieve the IRT precision in the CFB estimate.

CONCLUSION

This study shows the advantage of IRT over MMRM with a direct comparison of the same primary endpoint for the two analyses using the same observed clinical trial data, resulting in an increased confidence in Phase III.

SAMBA: A novel method for fast automatic model building in nonlinear mixed-effects models

The success of correctly identifying all the components of a nonlinear mixed-effects model is far from straightforward: it is a question of finding the best structural model, determining the type of relationship between covariates and individual parameters, detecting possible correlations between random effects, or also modeling residual errors. We present the Stochastic Approximation for Model Building Algorithm (SAMBA) procedure and show how this algorithm can be used to speed up this process of model building by identifying at each step how best to improve some of the model components. The principle of this algorithm basically consists in "learning something" about the "best model," even when a "poor model" is used to fit the data. A comparison study of the SAMBA procedure with Stepwise Covariate Modeling (SCM) and COnditional Sampling use for Stepwise Approach (COSSAC) show similar performances on several real data examples but with a much reduced computing time. This algorithm is now implemented in Monolix and in the R package Rsmlx.

Improved Decision-Making Confidence Using Item-Based Pharmacometric Model: Illustration with a Phase II Placebo-Controlled Trial

This study aimed to illustrate how a new methodology to assess clinical trial outcome measures using a longitudinal item response theory–based model (IRM) could serve as an alternative to mixed model repeated measures (MMRM). Data from the EXACT (Exacerbation of chronic pulmonary disease tool) which is used to capture frequency, severity, and duration of exacerbations in COPD were analyzed using an IRM. The IRM included a graded response model characterizing item parameters and functions describing symptom-time course. Total scores were simulated (month 12) using uncertainty in parameter estimates. The 50th (2.5th, 97.5th) percentiles of the resulting simulated differences in average total score (drug minus placebo) represented the estimated drug effect (95%CI), which was compared with published MMRM results. Furthermore, differences in sample size, sensitivity, specificity, and type I and II errors between approaches were explored. Patients received either oral danirixin 75 mg twice daily (n = 45) or placebo (n = 48) on top of standard of care over 52 weeks. A step function best described the COPD symptoms-time course in both trial arms. The IRM improved precision of the estimated drug effect compared to MMRM, resulting in a sample size of 2.5 times larger for the MMRM analysis to achieve the IRM precision. The IRM showed a higher probability of a positive predictive value (34%) than MMRM (22%). An item model–based analysis data gave more precise estimates of drug effect than MMRM analysis for the same endpoint in this one case study.

Conditional Non-parametric Bootstrap for Non-linear Mixed Effect Models

PURPOSE

Non-linear mixed effect models are widely used and increasingly integrated into decision-making processes. Propagating uncertainty is an important element of this process, and while standard errors (SE) on pa- rameters are most often computed using asymptotic approaches, alternative methods such as the bootstrap are also available. In this article, we propose a modified residual parametric bootstrap taking into account the different levels of variability involved in these models.

METHODS

The proposed approach uses samples from the individual conditional distribution, and was implemented in R using the saemix algorithm. We performed a simulation study to assess its performance in different scenarios, comparing it to the asymptotic approximation and to standard bootstraps in terms of coverage, also looking at bias in the parameters and their SE.

RESULTS

Simulations with an Emax model with different designs and sigmoidicity factors showed a similar coverage rate to the parametric bootstrap, while requiring less hypotheses. Bootstrap improved coverage in several scenarios compared to the asymptotic method especially for the variance param-eters. However, all bootstraps were sensitive to estimation bias in the original datasets.

CONCLUSIONS

The conditional bootstrap provided better coverage rate than the traditional residual bootstrap, while preserving the structure of the data generating process.

Assessing Treatment Effects with Pharmacometric Models: A New Method that Addresses Problems with Standard Assessments

Longitudinal pharmacometric models offer many advantages in the analysis of clinical trial data, but potentially inflated type I error and biased drug effect estimates, as a consequence of model misspecifications and multiple testing, are main drawbacks. In this work, we used real data to compare these aspects for a standard approach (STD) and a new one using mixture models, called individual model averaging (IMA). Placebo arm data sets were obtained from three clinical studies assessing ADAS-Cog scores, Likert pain scores, and seizure frequency. By randomly (1:1) assigning patients in the above data sets to “treatment” or “placebo,” we created data sets where any significant drug effect was known to be a false positive. Repeating the process of random assignment and analysis for significant drug effect many times (N = 1000) for each of the 40 to 66 placebo-drug model combinations, statistics of the type I error and drug effect bias were obtained. Across all models and the three data types, the type I error was (5th, 25th, 50th, 75th, 95th percentiles) 4.1, 11.4, 40.6, 100.0, 100.0 for STD, and 1.6, 3.5, 4.3, 5.0, 6.0 for IMA. IMA showed no bias in the drug effect estimates, whereas in STD bias was frequently present. In conclusion, STD is associated with inflated type I error and risk of biased drug effect estimates. IMA demonstrated controlled type I error and no bias.

cLRT-Mod: An efficient methodology for pharmacometric model-based analysis of longitudinal phase II dose finding studies under model uncertainty

Within the challenging context of phase II dose-finding trials, longitudinal analyses may increase drug effect detection power compared to an end-of-treatment analysis. This work proposes cLRT-Mod, a pharmacometric adaptation of the MCP-Mod methodology, which allows the use of nonlinear mixed effect models to first detect a dose-response signal and then identify the doses for the confirmatory phase while accounting for model structure uncertainty. The method was evaluated through extensive clinical trial simulations of a hypothetical phase II dose-finding trial using different scenarios and comparing different methods such as MCP-Mod. The results show an increase in power using cLRT with longitudinal data compared to an EOT multiple contrast tests for scenarios with small sample size and weak drug effect while maintaining pre-specifiability of the models prior to data analysis and the nominal type I error. This work shows how model averaging provides better coverage probability of the drug effect in the prediction step, and avoids under-estimation of the size of the confidence interval. Finally, for illustration purpose cLRT-Mod was applied to the analysis of a real phase II dose-finding trial.

A Model Averaging/Selection Approach Improves the Predictive Performance of Model-Informed Precision Dosing: Vancomycin as a Case Study

Many important drugs exhibit substantial variability in pharmacokinetics and pharmacodynamics leading to a loss of the desired clinical outcomes or significant adverse effects. Forecasting drug exposures using pharmacometric models can improve individual target attainment when compared with conventional therapeutic drug monitoring (TDM). However, selecting the "correct" model for this model-informed precision dosing (MIPD) is challenging. We derived and evaluated a model selection algorithm (MSA) and a model averaging algorithm (MAA), which automates model selection and finds the best model or combination of models for each patient using vancomycin as a case study, and implemented both algorithms in the MIPD software "TDMx." The predictive performance (based on accuracy and precision) of the two algorithms was assessed in (i) a simulation study of six distinct populations and (ii) a clinical dataset of 180 patients undergoing TDM during vancomycin treatment and compared with the performance obtained using a single model. Throughout the six virtual populations the MSA and MAA (imprecision: 9.9-24.2%, inaccuracy: less than ± 8.2%) displayed more accurate predictions than the single models (imprecision: 8.9-51.1%; inaccuracy: up to 28.9%). In the clinical dataset, the predictive performance of the single models applying at least one plasma concentration varied substantially (imprecision: 28-62%, inaccuracy: -16 to 25%), whereas the MSA or MAA utilizing these models simultaneously resulted in unbiased and precise predictions (imprecision: 29% and 30%, inaccuracy: -5% and 0%, respectively). MSA and MAA approaches implemented in TDMx might thereby lower the burden of fit-for-purpose validation of individual models and streamline MIPD.

New Model-Based Bioequivalence Statistical Approaches for Pharmacokinetic Studies with Sparse Sampling

In traditional pharmacokinetic (PK) bioequivalence analysis, two one-sided tests (TOST) are conducted on the area under the concentration-time curve and the maximal concentration derived using a non-compartmental approach. When rich sampling is unfeasible, a model-based (MB) approach, using nonlinear mixed effect models (NLMEM) is possible. However, MB-TOST using asymptotic standard errors (SE) presents increased type I error when asymptotic conditions do not hold. In this work, we propose three alternative calculations of the SE based on (i) an adaptation to NLMEM of the correction proposed by Gallant, (ii) the a posteriori distribution of the treatment coefficient using the Hamiltonian Monte Carlo algorithm, and (iii) parametric random effects and residual errors bootstrap. We evaluate these approaches by simulations, for two-arms parallel and two-period, two-sequence cross-over design with rich (n = 10) and sparse (n = 3) sampling under the null and the alternative hypotheses, with MB-TOST. All new approaches correct for the inflation of MB-TOST type I error in PK studies with sparse designs. The approach based on the a posteriori distribution appears to be the best compromise between controlled type I errors and computing times. MB-TOST using non-asymptotic SE controls type I error rate better than when using asymptotic SE estimates for bioequivalence on PK studies with sparse sampling.

Efficient model-based bioequivalence testing

The classical approach to analyze pharmacokinetic (PK) data in bioequivalence studies aiming to compare two different formulations is to perform noncompartmental analysis (NCA) followed by two one-sided tests (TOST). In this regard, the PK parameters area under the curve (AUC) and $C_{\max}$ are obtained for both treatment groups and their geometric mean ratios are considered. According to current guidelines by the U.S. Food and Drug Administration and the European Medicines Agency, the formulations are declared to be sufficiently similar if the $90\%$ confidence interval for these ratios falls between $0.8$ and $1.25 $. As NCA is not a reliable approach in case of sparse designs, a model-based alternative has already been proposed for the estimation of $\rm AUC$ and $C_{\max}$ using nonlinear mixed effects models. Here we propose another, more powerful test than the TOST and demonstrate its superiority through a simulation study both for NCA and model-based approaches. For products with high variability on PK parameters, this method appears to have closer type I errors to the conventionally accepted significance level of $0.05$, suggesting its potential use in situations where conventional bioequivalence analysis is not applicable.

Model Averaging in Viral Dynamic Models

The paucity of experimental data makes both inference and prediction particularly challenging in viral dynamic models. In the presence of several candidate models, a common strategy is model selection (MS), in which models are fitted to the data but only results obtained with the "best model" are presented. However, this approach ignores model uncertainty, which may lead to inaccurate predictions. When several models provide a good fit to the data, another approach is model averaging (MA) that weights the predictions of each model according to its consistency to the data. Here, we evaluated by simulations in a nonlinear mixed-effect model framework the performances of MS and MA in two realistic cases of acute viral infection, i.e., (1) inference in the presence of poorly identifiable parameters, namely, initial viral inoculum and eclipse phase duration, (2) uncertainty on the mechanisms of action of the immune response. MS was associated in some scenarios with a large rate of false selection. This led to a coverage rate lower than the nominal coverage rate of 0.95 in the majority of cases and below 0.50 in some scenarios. In contrast, MA provided better estimation of parameter uncertainty, with coverage rates ranging from 0.72 to 0.98 and mostly comprised within the nominal coverage rate. Finally, MA provided similar predictions than those obtained with MS. In conclusion, parameter estimates obtained with MS should be taken with caution, especially when several models well describe the data. In this situation, MA has better performances and could be performed to account for model uncertainty.

Optimal designs for frequentist model averaging

We consider the problem of designing experiments for estimating a target parameter in regression analysis when there is uncertainty about the parametric form of the regression function. A new optimality criterion is proposed that chooses the experimental design to minimize the asymptotic mean squared error of the frequentist model averaging estimate. Necessary conditions for the optimal solution of a locally and Bayesian optimal design problem are established. The results are illustrated in several examples, and it is demonstrated that Bayesian optimal designs can yield a reduction of the mean squared error of the model averaging estimator by up to 45%.

Robust designs accounting for model uncertainty in longitudinal studies with binary outcomes

To optimize designs for longitudinal studies analyzed by mixed-effect models with binary outcomes, the Fisher information matrix can be used. Optimal design approaches, however, require a priori knowledge of the model. We aim to propose, for the first time, a robust design approach accounting for model uncertainty in longitudinal trials with two treatment groups, assuming mixed-effect logistic models. To optimize designs given one model, we compute several optimality criteria based on Fisher information matrix evaluated by the new approach based on Monte-Carlo/Hamiltonian Monte-Carlo. We propose to use the DDS-optimality criterion, as it ensures a compromise between the precision of estimation of the parameters, and hence the Wald test power, and the overall precision of parameter estimation. To account for model uncertainty, we assume candidate models with their respective weights. We compute robust design across these models using compound DDS-optimality. Using the Fisher information matrix, we propose to predict the average power over these models. Evaluating this approach by clinical trial simulations, we show that the robust design is efficient across all models, allowing one to achieve good power of test. The proposed design strategy is a new and relevant approach to design longitudinal studies with binary outcomes, accounting for model uncertainty.