Diagnosing imputation models by applying target analyses to posterior replicates of completed data

Impact of Insurance Status and Region on Angiotensin Receptor-Neprilysin Inhibitor Prescription During Heart Failure Hospitalizations

BACKGROUND

An angiotensin receptor-neprilysin inhibitor (ARNI) is the preferred renin-angiotensin system (RAS) inhibitor for heart failure with reduced ejection fraction (HFrEF). Among eligible patients, insurance status and prescriber concern regarding out-of-pocket costs may constrain early initiation of ARNI and other new therapies.

OBJECTIVES

In this study, the authors sought to evaluate the association of insurance and other social determinants of health with ARNI initiation at discharge from HFrEF hospitalization.

METHODS

The authors analyzed ARNI initiation from January 2017 to June 2020 among patients with HFrEF eligible to receive RAS inhibitor at discharge from hospitals in the Get With The Guidelines-Heart Failure registry. The primary outcome was the proportion of ARNI prescription at discharge among those prescribed RAS inhibitor who were not on ARNI on admission. A logistic regression model was used to determine the association of insurance status, U.S. region, and their interaction, as well as self-reported race, with ARNI initiation at discharge.

RESULTS

From 42,766 admissions, 24,904 were excluded for absolute or relative contraindications to RAS inhibitors. RAS inhibitors were prescribed for 16,817 (94.2%) of remaining discharges, for which ARNI was prescribed in 1,640 (9.8%). Self-reported Black patients were less likely to be initiated on ARNI compared to self-reported White patients (OR: 0.64; 95% CI: 0.50-0.81). Compared to Medicare beneficiaries, patients with third-party insurance, Medicaid, or no insurance were less likely to be initiated on ARNI (OR: 0.47 [95% CI: 0.31-0.72], OR: 0.41 [95% CI: 0.25-0.67], and OR: 0.20 [95% CI: 0.08-0.47], respectively). ARNI therapy varied by hospital region, with lowest utilization in the Mountain region. An interaction was demonstrated between the impact of insurance disparities and hospital region.

CONCLUSIONS

Among patients hospitalized between 2017 and 2020 for HFrEF who were prescribed RAS inhibitor therapy at discharge, insurance status, geographic region, and self-reported race were associated with ARNI initiation.

Graphical and numerical diagnostic tools to assess multiple imputation models by posterior predictive checking

Problem

The congenial of the imputation model is crucial for valid statistical inferences. Hence, it is important to develop methodologies for diagnosing imputation models.

Aim

We propose and evaluate a new diagnostic method based on posterior predictive checking to diagnose the congeniality of fully conditional imputation models. Our method applies to multiple imputation by chained equations, which is widely used in statistical software.

Methods

The proposed method compares the observed data with their replicates generated under the corresponding posterior predictive distributions to diagnose the performance of imputation models. The method applies to various imputation models, including parametric and semi-parametric approaches and continuous and discrete incomplete variables. We studied the validity of the method through simulation and application.

Results

The proposed diagnostic method based on posterior predictive checking demonstrates its validity in assessing the performance of imputation models. The method can diagnose the consistency of imputation models with the substantive model and can be applied to a broad range of research contexts.

Conclusion

The diagnostic method based on posterior predictive checking provides a valuable tool for researchers who use fully conditional specification to handle missing data. By assessing the performance of imputation models, our method can help researchers improve the accuracy and reliability of their analyzes. Furthermore, our method applies to different imputation models. Hence, it is a versatile and valuable tool for researchers identifying plausible imputation models.

Unified estimation for Cox regression model with nonmonotone missing at random covariates

This article investigates a unified estimator for Cox regression model (Cox, 1972) when covariate data are missing at random (Rubin, 1976). It extends the idea of using parametric working models (Zhao and Liu, 2021) to extract the partial information contained in the incomplete observations. The working models are flexible and convenient to deal with nonmonotone missing data patterns. It can also incorporate auxiliary variables into the analysis to reduce estimation bias and improve efficiency. The unified estimator is consistent and more efficient than the (weighted) complete case estimator. Similar to multiple imputation (MI) method (Rubin, 1987 and 1996), the proposed method is also based on standard (weighted) complete data analysis and can be easily implemented in standard software. Simulation studies comparing the unified estimator with the substantive model compatible modification of the fully conditional specification MI (SMC-FCS) estimator (Bartlett et al., 2015) in various settings indicate that the unified estimator is consistent and as efficient as SMC-FCS estimator. Data from a clinical trial in patients with early breast cancer are analyzed for illustration.

Androgens In Men Study (AIMS): protocol for meta-analyses of individual participant data investigating associations of androgens with health outcomes in men

INTRODUCTION

This study aims to clarify the role(s) of endogenous sex hormones to influence health outcomes in men, specifically to define the associations of plasma testosterone with incidence of cardiovascular events, cancer, dementia and mortality risk, and to identify factors predicting testosterone concentrations. Data will be accrued from at least three Australian, two European and four North American population-based cohorts involving approximately 20 000 men.

METHODS AND ANALYSIS

Eligible studies include prospective cohort studies with baseline testosterone concentrations measured using mass spectrometry and 5 years of follow-up data on incident cardiovascular events, mortality, cancer diagnoses or deaths, new-onset dementia or decline in cognitive function recorded. Data for men, who were not taking androgens or drugs suppressing testosterone production, metabolism or action; and had no prior orchidectomy, are eligible. Systematic literature searches were conducted from 14 June 2019 to 31 December 2019, with no date range set for searches. Aggregate level data will be sought where individual participant data (IPD) are not available. One-stage IPD random-effects meta-analyses will be performed, using linear mixed models, generalised linear mixed models and either stratified or frailty-augmented Cox regression models. Heterogeneity in estimates from different studies will be quantified and bias investigated using funnel plots. Effect size estimates will be presented in forest plots and non-negligible heterogeneity and bias investigated using subgroup or meta-regression analyses.

ETHICS AND DISSEMINATION

Ethics approvals obtained for each of the participating cohorts state that participants have consented to have their data collected and used for research purposes. The Androgens In Men Study has been assessed as exempt from ethics review by the Human Ethics office at the University of Western Australia (file reference number RA/4/20/5014). Each of the component studies had obtained ethics approvals; please refer to respective component studies for details. Research findings will be disseminated to the scientific and broader community via the publication of four research articles, with each involving a separate set of IPD meta-analyses (articles will investigate different, distinct outcomes), at scientific conferences and meetings of relevant professional societies. Collaborating cohort studies will disseminate findings to study participants and local communities.

PROSPERO REGISTRATION NUMBER

CRD42019139668.

Limitations in Using Multiple Imputation to Harmonize Individual Participant Data for Meta-Analysis

Individual participant data (IPD) meta-analysis is a meta-analysis in which the individual-level data for each study are obtained and used for synthesis. A common challenge in IPD meta-analysis is when variables of interest are measured differently in different studies. The term harmonization has been coined to describe the procedure of placing variables on the same scale in order to permit pooling of data from a large number of studies. Using data from an IPD meta-analysis of 19 adolescent depression trials, we describe a multiple imputation approach for harmonizing 10 depression measures across the 19 trials by treating those depression measures that were not used in a study as missing data. We then apply diagnostics to address the fit of our imputation model. Even after reducing the scale of our application, we were still unable to produce accurate imputations of the missing values. We describe those features of the data that made it difficult to harmonize the depression measures and provide some guidelines for using multiple imputation for harmonization in IPD meta-analysis.

Model validation and influence diagnostics for regression models with missing covariates

Missing covariate values are prevalent in regression applications. While an array of methods have been developed for estimating parameters in regression models with missing covariate data for a variety of response types, minimal focus has been given to validation of the response model and influence diagnostics. Previous research has mainly focused on estimating residuals for observations with missing covariates using expected values, after which specialized techniques are needed to conduct proper inference. We suggest a multiple imputation strategy that allows for the use of standard methods for residual analyses on the imputed data sets or a stacked data set. We demonstrate the suggested multiple imputation method by analyzing the Sleep in Mammals data in the context of a linear regression model and the New York Social Indicators Status data with a logistic regression model.

Model checking in multiple imputation: an overview and case study

Background

Multiple imputation has become very popular as a general-purpose method for handling missing data. The validity of multiple-imputation-based analyses relies on the use of an appropriate model to impute the missing values. Despite the widespread use of multiple imputation, there are few guidelines available for checking imputation models.

Analysis

In this paper, we provide an overview of currently available methods for checking imputation models. These include graphical checks and numerical summaries, as well as simulation-based methods such as posterior predictive checking. These model checking techniques are illustrated using an analysis affected by missing data from the Longitudinal Study of Australian Children.

Conclusions

As multiple imputation becomes further established as a standard approach for handling missing data, it will become increasingly important that researchers employ appropriate model checking approaches to ensure that reliable results are obtained when using this method.

Electronic supplementary material

The online version of this article (doi:10.1186/s12982-017-0062-6) contains supplementary material, which is available to authorized users.

Is structural stigma's effect on the mortality of sexual minorities robust? A failure to replicate the results of a published study

BACKGROUND

The study of stigma's influence on health has surged in recent years. Hatzenbuehler et al.'s (2014) study of structural stigma's effect on mortality revealed an average of 12 years' shorter life expectancy for sexual minorities who resided in communities thought to exhibit high levels of anti-gay prejudice, using data from the 1988-2002 administrations of the US General Social Survey linked to mortality outcome data in the 2008 National Death Index.

METHODS

In the original study, the key predictor variable (structural stigma) led to results suggesting the profound negative influence of structural stigma on the mortality of sexual minorities. Attempts to replicate the study, in order to explore alternative hypotheses, repeatedly failed to generate the original study's key finding on structural stigma. Efforts to discern the source of the disparity in results revealed complications in the multiple imputation process for missing values of the components of structural stigma. This prompted efforts at replication using 10 different imputation approaches.

RESULTS

Efforts to replicate Hatzenbuehler et al.'s (2014) key finding on structural stigma's notable influence on the premature mortality of sexual minorities, including a more refined imputation strategy than described in the original study, failed. No data imputation approach yielded parameters that supported the original study's conclusions. Alternative hypotheses, which originally motivated the present study, revealed little new information.

CONCLUSION

Ten different approaches to multiple imputation of missing data yielded none in which the effect of structural stigma on the mortality of sexual minorities was statistically significant. Minimally, the original study's structural stigma variable (and hence its key result) is so sensitive to subjective measurement decisions as to be rendered unreliable.

Multiple imputation for harmonizing longitudinal non-commensurate measures in individual participant data meta-analysis

There are many advantages to individual participant data meta-analysis for combining data from multiple studies. These advantages include greater power to detect effects, increased sample heterogeneity, and the ability to perform more sophisticated analyses than meta-analyses that rely on published results. However, a fundamental challenge is that it is unlikely that variables of interest are measured the same way in all of the studies to be combined. We propose that this situation can be viewed as a missing data problem in which some outcomes are entirely missing within some trials and use multiple imputation to fill in missing measurements. We apply our method to five longitudinal adolescent depression trials where four studies used one depression measure and the fifth study used a different depression measure. None of the five studies contained both depression measures. We describe a multiple imputation approach for filling in missing depression measures that makes use of external calibration studies in which both depression measures were used. We discuss some practical issues in developing the imputation model including taking into account treatment group and study. We present diagnostics for checking the fit of the imputation model and investigate whether external information is appropriately incorporated into the imputed values.

Sensitivity analysis in multiple imputation in effectiveness studies of psychotherapy

The importance of preventing and treating incomplete data in effectiveness studies is nowadays emphasized. However, most of the publications focus on randomized clinical trials (RCT). One flexible technique for statistical inference with missing data is multiple imputation (MI). Since methods such as MI rely on the assumption of missing data being at random (MAR), a sensitivity analysis for testing the robustness against departures from this assumption is required. In this paper we present a sensitivity analysis technique based on posterior predictive checking, which takes into consideration the concept of clinical significance used in the evaluation of intra-individual changes. We demonstrate the possibilities this technique can offer with the example of irregular longitudinal data collected with the Outcome Questionnaire-45 (OQ-45) and the Helping Alliance Questionnaire (HAQ) in a sample of 260 outpatients. The sensitivity analysis can be used to (1) quantify the degree of bias introduced by missing not at random data (MNAR) in a worst reasonable case scenario, (2) compare the performance of different analysis methods for dealing with missing data, or (3) detect the influence of possible violations to the model assumptions (e.g., lack of normality). Moreover, our analysis showed that ratings from the patient's and therapist's version of the HAQ could significantly improve the predictive value of the routine outcome monitoring based on the OQ-45. Since analysis dropouts always occur, repeated measurements with the OQ-45 and the HAQ analyzed with MI are useful to improve the accuracy of outcome estimates in quality assurance assessments and non-randomized effectiveness studies in the field of outpatient psychotherapy.

Posterior predictive checking of multiple imputation models

Multiple imputation is gaining popularity as a strategy for handling missing data, but there is a scarcity of tools for checking imputation models, a critical step in model fitting. Posterior predictive checking (PPC) has been recommended as an imputation diagnostic. PPC involves simulating "replicated" data from the posterior predictive distribution of the model under scrutiny. Model fit is assessed by examining whether the analysis from the observed data appears typical of results obtained from the replicates produced by the model. A proposed diagnostic measure is the posterior predictive "p-value", an extreme value of which (i.e., a value close to 0 or 1) suggests a misfit between the model and the data. The aim of this study was to evaluate the performance of the posterior predictive p-value as an imputation diagnostic. Using simulation methods, we deliberately misspecified imputation models to determine whether posterior predictive p-values were effective in identifying these problems. When estimating the regression parameter of interest, we found that more extreme p-values were associated with poorer imputation model performance, although the results highlighted that traditional thresholds for classical p-values do not apply in this context. A shortcoming of the PPC method was its reduced ability to detect misspecified models with increasing amounts of missing data. Despite the limitations of posterior predictive p-values, they appear to have a valuable place in the imputer's toolkit. In addition to automated checking using p-values, we recommend imputers perform graphical checks and examine other summaries of the test quantity distribution.

The rise of multiple imputation: a review of the reporting and implementation of the method in medical research

Background

Missing data are common in medical research, which can lead to a loss in statistical power and potentially biased results if not handled appropriately. Multiple imputation (MI) is a statistical method, widely adopted in practice, for dealing with missing data. Many academic journals now emphasise the importance of reporting information regarding missing data and proposed guidelines for documenting the application of MI have been published. This review evaluated the reporting of missing data, the application of MI including the details provided regarding the imputation model, and the frequency of sensitivity analyses within the MI framework in medical research articles.

Methods

A systematic review of articles published in the Lancet and New England Journal of Medicine between January 2008 and December 2013 in which MI was implemented was carried out.

Results

We identified 103 papers that used MI, with the number of papers increasing from 11 in 2008 to 26 in 2013. Nearly half of the papers specified the proportion of complete cases or the proportion with missing data by each variable. In the majority of the articles (86%) the imputed variables were specified. Of the 38 papers (37%) that stated the method of imputation, 20 used chained equations, 8 used multivariate normal imputation, and 10 used alternative methods. Very few articles (9%) detailed how they handled non-normally distributed variables during imputation. Thirty-nine papers (38%) stated the variables included in the imputation model. Less than half of the papers (46%) reported the number of imputations, and only two papers compared the distribution of imputed and observed data. Sixty-six papers presented the results from MI as a secondary analysis. Only three articles carried out a sensitivity analysis following MI to assess departures from the missing at random assumption, with details of the sensitivity analyses only provided by one article.

Conclusions

This review outlined deficiencies in the documenting of missing data and the details provided about imputation. Furthermore, only a few articles performed sensitivity analyses following MI even though this is strongly recommended in guidelines. Authors are encouraged to follow the available guidelines and provide information on missing data and the imputation process.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-015-0022-1) contains supplementary material, which is available to authorized users.

Diagnosing problems with imputation models using the Kolmogorov-Smirnov test: a simulation study

Background

Multiple imputation (MI) is becoming increasingly popular as a strategy for handling missing data, but there is a scarcity of tools for checking the adequacy of imputation models. The Kolmogorov-Smirnov (KS) test has been identified as a potential diagnostic method for assessing whether the distribution of imputed data deviates substantially from that of the observed data. The aim of this study was to evaluate the performance of the KS test as an imputation diagnostic.

Methods

Using simulation, we examined whether the KS test could reliably identify departures from assumptions made in the imputation model. To do this we examined how the p-values from the KS test behaved when skewed and heavy-tailed data were imputed using a normal imputation model. We varied the amount of missing data, the missing data models and the amount of skewness, and evaluated the performance of KS test in diagnosing issues with the imputation models under these different scenarios.

Results

The KS test was able to flag differences between the observations and imputed values; however, these differences did not always correspond to problems with MI inference for the regression parameter of interest. When there was a strong missing at random dependency, the KS p-values were very small, regardless of whether or not the MI estimates were biased; so that the KS test was not able to discriminate between imputed variables that required further investigation, and those that did not. The p-values were also sensitive to sample size and the proportion of missing data, adding to the challenge of interpreting the results from the KS test.

Conclusions

Given our study results, it is difficult to establish guidelines or recommendations for using the KS test as a diagnostic tool for MI. The investigation of other imputation diagnostics and their incorporation into statistical software are important areas for future research.