When and how should multiple imputation be used for handling missing data in randomised clinical trials - a practical guide with flowcharts

Background

Missing data may seriously compromise inferences from randomised clinical trials, especially if missing data are not handled appropriately. The potential bias due to missing data depends on the mechanism causing the data to be missing, and the analytical methods applied to amend the missingness. Therefore, the analysis of trial data with missing values requires careful planning and attention.

Methods

The authors had several meetings and discussions considering optimal ways of handling missing data to minimise the bias potential. We also searched PubMed (key words: missing data; randomi*; statistical analysis) and reference lists of known studies for papers (theoretical papers; empirical studies; simulation studies; etc.) on how to deal with missing data when analysing randomised clinical trials.

Results

Handling missing data is an important, yet difficult and complex task when analysing results of randomised clinical trials. We consider how to optimise the handling of missing data during the planning stage of a randomised clinical trial and recommend analytical approaches which may prevent bias caused by unavoidable missing data. We consider the strengths and limitations of using of best-worst and worst-best sensitivity analyses, multiple imputation, and full information maximum likelihood. We also present practical flowcharts on how to deal with missing data and an overview of the steps that always need to be considered during the analysis stage of a trial.

Conclusions

We present a practical guide and flowcharts describing when and how multiple imputation should be used to handle missing data in randomised clinical.

Electronic supplementary material

The online version of this article (10.1186/s12874-017-0442-1) contains supplementary material, which is available to authorized users.

The proportion of missing data should not be used to guide decisions on multiple imputation

OBJECTIVES

Researchers are concerned whether multiple imputation (MI) or complete case analysis should be used when a large proportion of data are missing. We aimed to provide guidance for drawing conclusions from data with a large proportion of missingness.

STUDY DESIGN AND SETTING

Via simulations, we investigated how the proportion of missing data, the fraction of missing information (FMI), and availability of auxiliary variables affected MI performance. Outcome data were missing completely at random or missing at random (MAR).

RESULTS

Provided sufficient auxiliary information was available; MI was beneficial in terms of bias and never detrimental in terms of efficiency. Models with similar FMI values, but differing proportions of missing data, also had similar precision for effect estimates. In the absence of bias, the FMI was a better guide to the efficiency gains using MI than the proportion of missing data.

CONCLUSION

We provide evidence that for MAR data, valid MI reduces bias even when the proportion of missingness is large. We advise researchers to use FMI to guide choice of auxiliary variables for efficiency gain in imputation analyses, and that sensitivity analyses including different imputation models may be needed if the number of complete cases is small.

Multiple imputation as a flexible tool for missing data handling in clinical research

The last 20 years has seen an uptick in research on missing data problems, and most software applications now implement one or more sophisticated missing data handling routines (e.g., multiple imputation or maximum likelihood estimation). Despite their superior statistical properties (e.g., less stringent assumptions, greater accuracy and power), the adoption of these modern analytic approaches is not uniform in psychology and related disciplines. Thus, the primary goal of this manuscript is to describe and illustrate the application of multiple imputation. Although maximum likelihood estimation is perhaps the easiest method to use in practice, psychological data sets often feature complexities that are currently difficult to handle appropriately in the likelihood framework (e.g., mixtures of categorical and continuous variables), but relatively simple to treat with imputation. The paper describes a number of practical issues that clinical researchers are likely to encounter when applying multiple imputation, including mixtures of categorical and continuous variables, item-level missing data in questionnaires, significance testing, interaction effects, and multilevel missing data. Analysis examples illustrate imputation with software packages that are freely available on the internet.

Framework for the treatment and reporting of missing data in observational studies: The Treatment And Reporting of Missing data in Observational Studies framework

Missing data are ubiquitous in medical research. Although there is increasing guidance on how to handle missing data, practice is changing slowly and misapprehensions abound, particularly in observational research. Importantly, the lack of transparency around methodological decisions is threatening the validity and reproducibility of modern research. We present a practical framework for handling and reporting the analysis of incomplete data in observational studies, which we illustrate using a case study from the Avon Longitudinal Study of Parents and Children. The framework consists of three steps: 1) Develop an analysis plan specifying the analysis model and how missing data are going to be addressed. An important consideration is whether a complete records' analysis is likely to be valid, whether multiple imputation or an alternative approach is likely to offer benefits and whether a sensitivity analysis regarding the missingness mechanism is required; 2) Examine the data, checking the methods outlined in the analysis plan are appropriate, and conduct the preplanned analysis; and 3) Report the results, including a description of the missing data, details on how the missing data were addressed, and the results from all analyses, interpreted in light of the missing data and the clinical relevance. This framework seeks to support researchers in thinking systematically about missing data and transparently reporting the potential effect on the study results, therefore increasing the confidence in and reproducibility of research findings.

Missing data in a multi-item instrument were best handled by multiple imputation at the item score level

OBJECTIVES

Regardless of the proportion of missing values, complete-case analysis is most frequently applied, although advanced techniques such as multiple imputation (MI) are available. The objective of this study was to explore the performance of simple and more advanced methods for handling missing data in cases when some, many, or all item scores are missing in a multi-item instrument.

STUDY DESIGN AND SETTING

Real-life missing data situations were simulated in a multi-item variable used as a covariate in a linear regression model. Various missing data mechanisms were simulated with an increasing percentage of missing data. Subsequently, several techniques to handle missing data were applied to decide on the most optimal technique for each scenario. Fitted regression coefficients were compared using the bias and coverage as performance parameters.

RESULTS

Mean imputation caused biased estimates in every missing data scenario when data are missing for more than 10% of the subjects. Furthermore, when a large percentage of subjects had missing items (>25%), MI methods applied to the items outperformed methods applied to the total score.

CONCLUSION

We recommend applying MI to the item scores to get the most accurate regression model estimates. Moreover, we advise not to use any form of mean imputation to handle missing data.

A comparison of multiple imputation methods for missing data in longitudinal studies

Background

Multiple imputation (MI) is now widely used to handle missing data in longitudinal studies. Several MI techniques have been proposed to impute incomplete longitudinal covariates, including standard fully conditional specification (FCS-Standard) and joint multivariate normal imputation (JM-MVN), which treat repeated measurements as distinct variables, and various extensions based on generalized linear mixed models. Although these MI approaches have been implemented in various software packages, there has not been a comprehensive evaluation of the relative performance of these methods in the context of longitudinal data.

Method

Using both empirical data and a simulation study based on data from the six waves of the Longitudinal Study of Australian Children (N = 4661), we investigated the performance of a wide range of MI methods available in standard software packages for investigating the association between child body mass index (BMI) and quality of life using both a linear regression and a linear mixed-effects model.

Results

In this paper, we have identified and compared 12 different MI methods for imputing missing data in longitudinal studies. Analysis of simulated data under missing at random (MAR) mechanisms showed that the generally available MI methods provided less biased estimates with better coverage for the linear regression model and around half of these methods performed well for the estimation of regression parameters for a linear mixed model with random intercept. With the observed data, we observed an inverse association between child BMI and quality of life, with available data as well as multiple imputation.

Conclusion

Both FCS-Standard and JM-MVN performed well for the estimation of regression parameters in both analysis models. More complex methods that explicitly reflect the longitudinal structure for these analysis models may only be needed in specific circumstances such as irregularly spaced data.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0615-6) contains supplementary material, which is available to authorized users.

Principled Missing Data Treatments

We review a number of issues regarding missing data treatments for intervention and prevention researchers. Many of the common missing data practices in prevention research are still, unfortunately, ill-advised (e.g., use of listwise and pairwise deletion, insufficient use of auxiliary variables). Our goal is to promote better practice in the handling of missing data. We review the current state of missing data methodology and recent missing data reporting in prevention research. We describe antiquated, ad hoc missing data treatments and discuss their limitations. We discuss two modern, principled missing data treatments: multiple imputation and full information maximum likelihood, and we offer practical tips on how to best employ these methods in prevention research. The principled missing data treatments that we discuss are couched in terms of how they improve causal and statistical inference in the prevention sciences. Our recommendations are firmly grounded in missing data theory and well-validated statistical principles for handling the missing data issues that are ubiquitous in biosocial and prevention research. We augment our broad survey of missing data analysis with references to more exhaustive resources.

Model checking in multiple imputation: an overview and case study

Background

Multiple imputation has become very popular as a general-purpose method for handling missing data. The validity of multiple-imputation-based analyses relies on the use of an appropriate model to impute the missing values. Despite the widespread use of multiple imputation, there are few guidelines available for checking imputation models.

Analysis

In this paper, we provide an overview of currently available methods for checking imputation models. These include graphical checks and numerical summaries, as well as simulation-based methods such as posterior predictive checking. These model checking techniques are illustrated using an analysis affected by missing data from the Longitudinal Study of Australian Children.

Conclusions

As multiple imputation becomes further established as a standard approach for handling missing data, it will become increasingly important that researchers employ appropriate model checking approaches to ensure that reliable results are obtained when using this method.

Electronic supplementary material

The online version of this article (doi:10.1186/s12982-017-0062-6) contains supplementary material, which is available to authorized users.

Outcome-sensitive multiple imputation: a simulation study

Background

Multiple imputation is frequently used to deal with missing data in healthcare research. Although it is known that the outcome should be included in the imputation model when imputing missing covariate values, it is not known whether it should be imputed. Similarly no clear recommendations exist on: the utility of incorporating a secondary outcome, if available, in the imputation model; the level of protection offered when data are missing not-at-random; the implications of the dataset size and missingness levels.

Methods

We used realistic assumptions to generate thousands of datasets across a broad spectrum of contexts: three mechanisms of missingness (completely at random; at random; not at random); varying extents of missingness (20–80% missing data); and different sample sizes (1,000 or 10,000 cases). For each context we quantified the performance of a complete case analysis and seven multiple imputation methods which deleted cases with missing outcome before imputation, after imputation or not at all; included or did not include the outcome in the imputation models; and included or did not include a secondary outcome in the imputation models. Methods were compared on mean absolute error, bias, coverage and power over 1,000 datasets for each scenario.

Results

Overall, there was very little to separate multiple imputation methods which included the outcome in the imputation model. Even when missingness was quite extensive, all multiple imputation approaches performed well. Incorporating a secondary outcome, moderately correlated with the outcome of interest, made very little difference. The dataset size and the extent of missingness affected performance, as expected. Multiple imputation methods protected less well against missingness not at random, but did offer some protection.

Conclusions

As long as the outcome is included in the imputation model, there are very small performance differences between the possible multiple imputation approaches: no outcome imputation, imputation or imputation and deletion. All informative covariates, even with very high levels of missingness, should be included in the multiple imputation model. Multiple imputation offers some protection against a simple missing not at random mechanism.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-016-0281-5) contains supplementary material, which is available to authorized users.

Avoiding bias due to perfect prediction in multiple imputation of incomplete categorical variables

Multiple imputation is a popular way to handle missing data. Automated procedures are widely available in standard software. However, such automated procedures may hide many assumptions and possible difficulties from the view of the data analyst. Imputation procedures such as monotone imputation and imputation by chained equations often involve the fitting of a regression model for a categorical outcome. If perfect prediction occurs in such a model, then automated procedures may give severely biased results. This is a problem in some standard software, but it may be avoided by bootstrap methods, penalised regression methods, or a new augmentation procedure.

The treatment of incomplete data: Reporting, analysis, reproducibility, and replicability

Proper analysis and reporting of incomplete data continues to be a challenging task for practitioners from various research areas. Recently Nguyen, Strazdins, Nicholson and Cooklin (NSNC; 2018) evaluated the impact of complete case analysis and multiple imputation in studies of parental employment and health. Their work joins interdisciplinary efforts to educate and motivate scientists across the research community to use principled statistical methods when analyzing incomplete data. Although we fully support and encourage work in parallel to NSNC's, we also think that further actions should be taken by the research community to improve current practices. In this commentary, we discuss some aspects and misconceptions related to analysis of incomplete data, in particular multiple imputation. In our view, the missing data problem is part of a larger problem of research reproducibility and replicability today. Thus, we believe that improving analysis and reporting of incomplete data will make reproducibility and replicability efforts easier. We also provide a brief checklist of recommendations which could be used by members of the scientific community, including practitioners, journal editors, and reviewers to set higher publication standards.

A comparison of different methods to handle missing data in the context of propensity score analysis

Propensity score analysis is a popular method to control for confounding in observational studies. A challenge in propensity methods is missing values in confounders. Several strategies for handling missing values exist, but guidance in choosing the best method is needed. In this simulation study, we compared four strategies of handling missing covariate values in propensity matching and propensity weighting. These methods include: complete case analysis, missing indicator method, multiple imputation and combining multiple imputation and missing indicator method. Concurrently, we aimed to provide guidance in choosing the optimal strategy. Simulated scenarios varied regarding missing mechanism, presence of effect modification or unmeasured confounding. Additionally, we demonstrated how missingness graphs help clarifying the missing structure. When no effect modification existed, complete case analysis yielded valid causal treatment effects even when data were missing not at random. In some situations, complete case analysis was also able to partially correct for unmeasured confounding. Multiple imputation worked well if the data were missing (completely) at random, and if the imputation model was correctly specified. In the presence of effect modification, more complex imputation models than default options of commonly used statistical software were required. Multiple imputation may fail when data are missing not at random. Here, combining multiple imputation and the missing indicator method reduced the bias as the missing indicator variable can be a proxy for unobserved confounding. The optimal way to handle missing values in covariates of propensity score models depends on the missing data structure and the presence of effect modification. When effect modification is present, default settings of imputation methods may yield biased results even if data are missing at random.

Methods for significance testing of categorical covariates in logistic regression models after multiple imputation: power and applicability analysis

Background

Multiple imputation is a recommended method to handle missing data. For significance testing after multiple imputation, Rubin’s Rules (RR) are easily applied to pool parameter estimates. In a logistic regression model, to consider whether a categorical covariate with more than two levels significantly contributes to the model, different methods are available. For example pooling chi-square tests with multiple degrees of freedom, pooling likelihood ratio test statistics, and pooling based on the covariance matrix of the regression model. These methods are more complex than RR and are not available in all mainstream statistical software packages. In addition, they do not always obtain optimal power levels. We argue that the median of the p-values from the overall significance tests from the analyses on the imputed datasets can be used as an alternative pooling rule for categorical variables. The aim of the current study is to compare different methods to test a categorical variable for significance after multiple imputation on applicability and power.

Methods

In a large simulation study, we demonstrated the control of the type I error and power levels of different pooling methods for categorical variables.

Results

This simulation study showed that for non-significant categorical covariates the type I error is controlled and the statistical power of the median pooling rule was at least equal to current multiple parameter tests. An empirical data example showed similar results.

Conclusions

It can therefore be concluded that using the median of the p-values from the imputed data analyses is an attractive and easy to use alternative method for significance testing of categorical variables.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-017-0404-7) contains supplementary material, which is available to authorized users.

Associations between birth outcomes and maternal PM2.5 exposure in Shanghai: A comparison of three exposure assessment approaches

BACKGROUND

Few studies have estimated effects of maternal PM_2.5 exposure on birth outcomes in China due to the lack of historical air pollution data.

OBJECTIVES

We estimated the associations between maternal PM_2.5 exposure and birth outcomes using gap-filled satellite estimates in Shanghai, China.

METHODS

We obtained birth registration records of 132,783 singleton live births during 2011-2014 in Shanghai. PM_2.5 exposures were assessed from satellite-derived estimates or central-site measurements. Linear and logistic regressions were used to estimate associations with term birth weight and term low birth weight (LBW), respectively. Logistic and discrete-time survival models were used to estimate associations with preterm birth. Effect modification by maternal age and parental education levels was investigated.

RESULTS

A 10 μg/m³ increase in gap-filled satellite-based whole-pregnancy PM_2.5 exposure was associated with a -12.85 g (95% CI: -18.44, -7.27) change in term birth weight, increased risk of preterm birth (OR 1.27, 95% CI: 1.20, 1.36), and increased risk of term LBW (OR 1.22, 95% CI: 1.06, 1.41). Sensitivity analyses during 2013-2014, when ground PM_2.5 measurements were available, showed that the health associations using gap-filled satellite PM_2.5 concentrations were higher than those obtained using satellite PM_2.5 concentrations without accounting for missingness. The health associations using gap-filled satellite PM_2.5 had similar magnitudes to those using central-site measurements, but with narrower confidence intervals.

CONCLUSIONS

The magnitude of associations between maternal PM_2.5 exposure and adverse birth outcomes in Shanghai was higher than previous findings. One reason could be reduced exposure error of the gap-filled high-resolution satellite PM_2.5 estimates.

Appropriate inclusion of interactions was needed to avoid bias in multiple imputation

Objective

Missing data are a pervasive problem, often leading to bias in complete records analysis (CRA). Multiple imputation (MI) via chained equations is one solution, but its use in the presence of interactions is not straightforward.

Study Design and Setting

We simulated data with outcome Y dependent on binary explanatory variables X and Z and their interaction XZ. Six scenarios were simulated (Y continuous and binary, each with no interaction, a weak and a strong interaction), under five missing data mechanisms. We use directed acyclic graphs to identify when CRA and MI would each be unbiased. We evaluate the performance of CRA, MI without interactions, MI including all interactions, and stratified imputation. We also illustrated these methods using a simple example from the National Child Development Study (NCDS).

Results

MI excluding interactions is invalid and resulted in biased estimates and low coverage. When XZ was zero, MI excluding interactions gave unbiased estimates but overcoverage. MI including interactions and stratified MI gave equivalent, valid inference in all cases. In the NCDS example, MI excluding interactions incorrectly concluded there was no evidence for an important interaction.

Conclusions

Epidemiologists carrying out MI should ensure that their imputation model(s) are compatible with their analysis model.

The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data

Multiple imputation can be used as a tool in the process of constructing prediction models in medical and epidemiological studies with missing covariate values. Such models can be used to make predictions for model performance assessment, but the task is made more complicated by the multiple imputation structure. We summarize various predictions constructed from covariates, including multiply imputed covariates, and either the set of imputation-specific prediction model coefficients or the pooled prediction model coefficients. We further describe approaches for using the predictions to assess model performance. We distinguish between ideal model performance and pragmatic model performance, where the former refers to the model's performance in an ideal clinical setting where all individuals have fully observed predictors and the latter refers to the model's performance in a real-world clinical setting where some individuals have missing predictors. The approaches are compared through an extensive simulation study based on the UK700 trial. We determine that measures of ideal model performance can be estimated within imputed datasets and subsequently pooled to give an overall measure of model performance. Alternative methods to evaluate pragmatic model performance are required and we propose constructing predictions either from a second set of covariate imputations which make no use of observed outcomes, or from a set of partial prediction models constructed for each potential observed pattern of covariate. Pragmatic model performance is generally lower than ideal model performance. We focus on model performance within the derivation data, but describe how to extend all the methods to a validation dataset.

Missing data should be handled differently for prediction than for description or causal explanation

Missing data are much studied in epidemiology and statistics. Theoretical development and application of methods for handling missing data have mostly been conducted in the context of prospective research data and with a goal of description or causal explanation. However, it is now common to build predictive models using routinely collected data, where missing patterns may convey important information, and one might take a pragmatic approach to optimizing prediction. Therefore, different methods to handle missing data may be preferred. Furthermore, an underappreciated issue in prediction modeling is that the missing data method used in model development may not match the method used when a model is deployed. This may lead to overoptimistic assessments of model performance. For prediction, particularly with routinely collected data, methods for handling missing data that incorporate information within the missingness pattern should be explored and further developed. Where missing data methods differ between model development and model deployment, the implications of this must be explicitly evaluated. The trade-off between building a prediction model that is causally principled, and building a prediction model that maximizes the use of all available information, should be carefully considered and will depend on the intended use of the model.

Paraspinal muscle cross-sectional area predicts low back disability but not pain intensity

BACKGROUND AND CONTEXT

The lumbar paraspinal muscles, including the erector spinae and multifidus, play an important role in movement and control of the spine. However, our understanding of their contribution to low back pain and disability is unclear. Systematic reviews have reported conflicting evidence for an association between paraspinal muscle size and low back pain, and a paucity of data examining muscle cross-sectional area (CSA) and low back disability.

PURPOSE

To investigate the relationship between paraspinal muscle CSA and both low back pain intensity and disability.

STUDY DESIGN/SETTING

One-year longitudinal cohort study.

PATIENT SAMPLE

Participants were selected from the SpineData Registry (Denmark), which enrolls people with low back pain of 2 to 12 months duration without radiculopathy and a satisfactory response to primary intervention.

OUTCOME MEASURES

Current, typical, and worst pain in the prior 2 weeks were assessed by 11-point numeric rating scales and an average pain score was calculated, and disability was measured using the 23-item Roland-Morris Disability Questionnaire. CSA (cm²) of the lumbar paraspinal muscles was measured at levels L3-L5 from magnetic resonance images.

METHODS

Participants completed the study questionnaires and underwent the lumbar spine magnetic resonance images at baseline and were followed up 12 months later to repeat the questionnaires. Statistical analyses involved multivariable linear regression (cross-sectional analysis) and linear mixed-models (longitudinal analysis) with adjustment for confounders. Multiple imputation was conducted to account for missing data.

RESULTS

A total of 962 participants were included and 588 (65.8%) were followed up at 12-months. Multivariable analysis showed that greater paraspinal muscle CSA was associated with lower levels of disability, after adjusting for confounders (right mean CSA: baseline beta -0.16, 95% CI -0.26 to -0.06, p<.01; longitudinal beta -0.11, 95% CI -0.21 to -0.01, p=.03). This was evident at all levels, except L5 which was marginal at baseline (beta -0.08, 95% CI -0.15 to -0.001, p=.045) and not significant longitudinally (beta -0.05, 95% CI -0.12 to 0.02, p=.18). However, there were no associations between muscle CSA and pain intensity (baseline beta -0.02, 95% CI -0.06 to 0.02, p=.29; longitudinal beta -0.02, 95% CI -0.06 to 0.02, p=.34). Results were similar for both complete case and multiple imputation analyses.

CONCLUSIONS

This study found an inverse relationship between lumbar paraspinal muscle CSA and low back disability, but not pain intensity. While further investigation is needed, these findings suggest that treatment strategies directed at increasing paraspinal muscle size may be effective in reducing low back disability.

Arsenic exposure, diabetes-related genes and diabetes prevalence in a general population from Spain

Inorganic arsenic exposure may be associated with diabetes, but the evidence at low-moderate levels is not sufficient. Polymorphisms in diabetes-related genes have been involved in diabetes risk. We evaluated the association of inorganic arsenic exposure on diabetes in the Hortega Study, a representative sample of a general population from Valladolid, Spain. Total urine arsenic was measured in 1451 adults. Urine arsenic speciation was available in 295 randomly selected participants. To account for the confounding introduced by non-toxic seafood arsenicals, we designed a multiple imputation model to predict the missing arsenobetaine levels. The prevalence of diabetes was 8.3%. The geometric mean of total arsenic was 66.0 μg/g. The adjusted odds ratios (95% confidence interval) for diabetes comparing the highest with the lowest tertile of total arsenic were 1.76 (1.01, 3.09) and 2.14 (1.47, 3.11) before and after arsenobetaine adjustment, respectively. Polymorphisms in several genes including IL8RA, TXN, NR3C2, COX5A and GCLC showed suggestive differential associations of urine total arsenic with diabetes. The findings support the role of arsenic on diabetes and the importance of controlling for seafood arsenicals in populations with high seafood intake. Suggestive arsenic-gene interactions require confirmation in larger studies.

Bootstrap inference for multiple imputation under uncongeniality and misspecification

Multiple imputation has become one of the most popular approaches for handling missing data in statistical analyses. Part of this success is due to Rubin’s simple combination rules. These give frequentist valid inferences when the imputation and analysis procedures are so-called congenial and the embedding model is correctly specified, but otherwise may not. Roughly speaking, congeniality corresponds to whether the imputation and analysis models make different assumptions about the data. In practice, imputation models and analysis procedures are often not congenial, such that tests may not have the correct size, and confidence interval coverage deviates from the advertised level. We examine a number of recent proposals which combine bootstrapping with multiple imputation and determine which are valid under uncongeniality and model misspecification. Imputation followed by bootstrapping generally does not result in valid variance estimates under uncongeniality or misspecification, whereas certain bootstrap followed by imputation methods do. We recommend a particular computationally efficient variant of bootstrapping followed by imputation.

Assessment of predictive performance in incomplete data by combining internal validation and multiple imputation

BACKGROUND

Missing values are a frequent issue in human studies. In many situations, multiple imputation (MI) is an appropriate missing data handling strategy, whereby missing values are imputed multiple times, the analysis is performed in every imputed data set, and the obtained estimates are pooled. If the aim is to estimate (added) predictive performance measures, such as (change in) the area under the receiver-operating characteristic curve (AUC), internal validation strategies become desirable in order to correct for optimism. It is not fully understood how internal validation should be combined with multiple imputation.

METHODS

In a comprehensive simulation study and in a real data set based on blood markers as predictors for mortality, we compare three combination strategies: Val-MI, internal validation followed by MI on the training and test parts separately, MI-Val, MI on the full data set followed by internal validation, and MI(-y)-Val, MI on the full data set omitting the outcome followed by internal validation. Different validation strategies, including bootstrap und cross-validation, different (added) performance measures, and various data characteristics are considered, and the strategies are evaluated with regard to bias and mean squared error of the obtained performance estimates. In addition, we elaborate on the number of resamples and imputations to be used, and adopt a strategy for confidence interval construction to incomplete data.

RESULTS

Internal validation is essential in order to avoid optimism, with the bootstrap 0.632+ estimate representing a reliable method to correct for optimism. While estimates obtained by MI-Val are optimistically biased, those obtained by MI(-y)-Val tend to be pessimistic in the presence of a true underlying effect. Val-MI provides largely unbiased estimates, with a slight pessimistic bias with increasing true effect size, number of covariates and decreasing sample size. In Val-MI, accuracy of the estimate is more strongly improved by increasing the number of bootstrap draws rather than the number of imputations. With a simple integrated approach, valid confidence intervals for performance estimates can be obtained.

CONCLUSIONS

When prognostic models are developed on incomplete data, Val-MI represents a valid strategy to obtain estimates of predictive performance measures.

Multiple imputation in quantile regression

We propose a multiple imputation estimator for parameter estimation in a quantile regression model when some covariates are missing at random. The estimation procedure fully utilizes the entire dataset to achieve increased efficiency, and the resulting coefficient estimators are root-n consistent and asymptotically normal. To protect against possible model misspecification, we further propose a shrinkage estimator, which automatically adjusts for possible bias. The finite sample performance of our estimator is investigated in a simulation study. Finally, we apply our methodology to part of the Eating at American's Table Study data, investigating the association between two measures of dietary intake.

Improving upon the efficiency of complete case analysis when covariates are MNAR

Missing values in covariates of regression models are a pervasive problem in empirical research. Popular approaches for analyzing partially observed datasets include complete case analysis (CCA), multiple imputation (MI), and inverse probability weighting (IPW). In the case of missing covariate values, these methods (as typically implemented) are valid under different missingness assumptions. In particular, CCA is valid under missing not at random (MNAR) mechanisms in which missingness in a covariate depends on the value of that covariate, but is conditionally independent of outcome. In this paper, we argue that in some settings such an assumption is more plausible than the missing at random assumption underpinning most implementations of MI and IPW. When the former assumption holds, although CCA gives consistent estimates, it does not make use of all observed information. We therefore propose an augmented CCA approach which makes the same conditional independence assumption for missingness as CCA, but which improves efficiency through specification of an additional model for the probability of missingness, given the fully observed variables. The new method is evaluated using simulations and illustrated through application to data on reported alcohol consumption and blood pressure from the US National Health and Nutrition Examination Survey, in which data are likely MNAR independent of outcome.

Evaluation of Multiple Imputation with Large Proportions of Missing Data: How Much Is Too Much?

Background:

Multiple Imputation (MI) is known as an effective method for handling missing data in public health research. However, it is not clear that the method will be effective when the data contain a high percentage of missing observations on a variable.

Methods:

Using data from “Predictive Study of Coronary Heart Disease” study, this study examined the effectiveness of multiple imputation in data with 20% missing to 80% missing observations using absolute bias (|bias|) and Root Mean Square Error (RMSE) of MI measured under Missing Completely at Random (MCAR), Missing at Random (MAR), and Not Missing at Random (NMAR) assumptions.

Results:

The |bias| and RMSE of MI was much smaller than of the results of CCA under all missing mechanisms, especially with a high percentage of missing. In addition, the |bias| and RMSE of MI were consistent regardless of increasing imputation numbers from M=10 to M=50. Moreover, when comparing imputation mechanisms, MCMC method had universally smaller |bias| and RMSE than those of Regression method and Predictive Mean Matching method under all missing mechanisms.

Conclusion:

As missing percentages become higher, using MI is recommended, because MI produced less biased estimates under all missing mechanisms. However, when large proportions of data are missing, other things need to be considered such as the number of imputations, imputation mechanisms, and missing data mechanisms for proper imputation.

Is using multiple imputation better than complete case analysis for estimating a prevalence (risk) difference in randomized controlled trials when binary outcome observations are missing?

Background

Missing outcomes can seriously impair the ability to make correct inferences from randomized controlled trials (RCTs). Complete case (CC) analysis is commonly used, but it reduces sample size and is perceived to lead to reduced statistical efficiency of estimates while increasing the potential for bias. As multiple imputation (MI) methods preserve sample size, they are generally viewed as the preferred analytical approach.

We examined this assumption, comparing the performance of CC and MI methods to determine risk difference (RD) estimates in the presence of missing binary outcomes. We conducted simulation studies of 5000 simulated data sets with 50 imputations of RCTs with one primary follow-up endpoint at different underlying levels of RD (3–25 %) and missing outcomes (5–30 %).

Results

For missing at random (MAR) or missing completely at random (MCAR) outcomes, CC method estimates generally remained unbiased and achieved precision similar to or better than MI methods, and high statistical coverage. Missing not at random (MNAR) scenarios yielded invalid inferences with both methods. Effect size estimate bias was reduced in MI methods by always including group membership even if this was unrelated to missingness. Surprisingly, under MAR and MCAR conditions in the assessed scenarios, MI offered no statistical advantage over CC methods.

Conclusion

While MI must inherently accompany CC methods for intention-to-treat analyses, these findings endorse CC methods for per protocol risk difference analyses in these conditions. These findings provide an argument for the use of the CC approach to always complement MI analyses, with the usual caveat that the validity of the mechanism for missingness be thoroughly discussed. More importantly, researchers should strive to collect as much data as possible.