Treatment of missing data in follow-up studies of randomised controlled trials: A systematic review of the literature

Trial characteristics and treatment effect estimates in randomized controlled trials of Chinese herbal medicine: A meta-epidemiological study

BACKGROUND

Previously published meta-epidemiological studies focused on Western medicine have identified some trial characteristics that impact the treatment effect of randomized controlled trials (RCTs). Nevertheless, it remains unclear if similar associations exist in RCTs on Chinese herbal medicine (CHM). Further, Chinese medicine-related characteristics have not been explored yet.

OBJECTIVE

To investigate trial characteristics related to treatment effect estimates on CHM RCTs.

SEARCH STRATEGY

This meta-epidemiological study searched 5 databases for systematic reviews on CHM treatment published between January 2011 and July 2021.

INCLUSION CRITERIA

An eligible systematic review should only include RCTs of CHM and conduct at least one meta-analysis.

DATA EXTRACTION AND ANALYSIS

Two reviewers independently conducted data extraction on general characteristics of systematic reviews, meta-analyses and included RCTs. They also assessed the risk of bias of RCTs using the Cochrane risk of bias tool. A two-step approach was used for data analyses. The ratio of odds ratios (ROR) and difference in standardized mean differences (dSMD) with 95% confidence interval (CI) were applied to present the difference in effect estimates for binary and continuous outcomes, respectively.

RESULTS

Ninety-one systematic reviews, comprising 1338 RCTs were identified. For binary outcomes, RCTs incorporated with syndrome differentiation (ROR: 1.23; 95 % CI: [1.07, 1.39]), adopting Chinese medicine formula (ROR: 1.19; 95% CI: [1.03, 1.34]), with low risk of bias on incomplete outcome data (ROR: 1.29; 95% CI: [1.06, 1.52]) and selective outcome reporting (ROR: 1.12; 95% CI: [1.01, 1.24]), as well as a trial size ≥ 100 (ROR: 1.23; 95% CI: [1.04, 1.42]) preferred to show larger effect estimates. As for continuous outcomes, RCTs with Chinese medicine diagnostic criteria (dSMD: 0.23; 95% CI: [0.06, 0.41]), judged as high/unclear risk of bias on allocation concealment (dSMD: -0.70; 95% CI: [-0.99, -0.42]), with low risk of bias on incomplete outcome data (dSMD: 0.30; 95% CI: [0.18, 0.43]), conducted at a single center (dSMD: -0.33; 95% CI: [-0.61, -0.05]), not using intention-to-treat analysis (dSMD: -0.75; 95% CI: [-1.43, -0.07]), and without funding support (dSMD: -0.22; 95% CI: [-0.41, -0.02]) tended to show larger effect estimates.

CONCLUSION

This study provides empirical evidence for the development of a specific critical appraisal tool for risk of bias assessments on CHM RCTs. Please cite this article as: Wang BH, Lin YL, Gao YY, Song JL, Qin L, Li LQ, Liu WQ, Zhong CCW, Jiang MY, Mao C, Yang XB, Chung VCH, Wu IXY. Trial characteristics and treatment effect estimates in randomized controlled trials of Chinese herbal medicine: A meta-epidemiological study. J Integr Med. 2024; 22(3): 223-234.

Accounting for nonmonotone missing data using inverse probability weighting

Inverse probability weighting can be used to correct for missing data. New estimators for the weights in the nonmonotone setting were introduced in 2018. These estimators are the unconstrained maximum likelihood estimator (UMLE) and the constrained Bayesian estimator (CBE), an alternative if UMLE fails to converge. In this work we describe and illustrate these estimators, and examine performance in simulation and in an applied example estimating the effect of anemia on spontaneous preterm birth in the Zambia Preterm Birth Prevention Study. We compare performance with multiple imputation (MI) and focus on the setting of an observational study where inverse probability of treatment weights are used to address confounding. In simulation, weighting was less statistically efficient at the smallest sample size and lowest exposure prevalence examined (n = 1500, 15% respectively) but in other scenarios statistical performance of weighting and MI was similar. Weighting had improved computational efficiency taking, on average, 0.4 and 0.05 times the time for MI in R and SAS, respectively. UMLE was easy to implement in commonly used software and convergence failure occurred just twice in >200 000 simulated cohorts making implementation of CBE unnecessary. In conclusion, weighting is an alternative to MI for nonmonotone missingness, though MI performed as well as or better in terms of bias and statistical efficiency. Weighting's superior computational efficiency may be preferred with large sample sizes or when using resampling algorithms. As validity of weighting and MI rely on correct specification of different models, both approaches could be implemented to check agreement of results.

Handling missing covariate data in clinical studies in haematology

Missing data are frequently encountered across studies in clinical haematology. Failure to handle these missing values in an appropriate manner can complicate the interpretation of a study's findings, as estimates presented may be biased and/or imprecise. In the present work, we first provide an overview of current methods for handling missing covariate data, along with their advantages and disadvantages. Furthermore, a systematic review is presented, exploring both contemporary reporting of missing values in major haematological journals, and the methods used for handling them. A principal finding was that the method of handling missing data was explicitly specified in a minority of articles (in 76 out of 195 articles reporting missing values, 39%). Among these, complete case analysis and the missing indicator method were the most common approaches to dealing with missing values, with more complex methods such as multiple imputation being extremely rare (in 7 out of 195 articles). An example analysis (with associated code) is also provided using hematopoietic stem cell transplantation data, illustrating the different approaches to handling missing values. We conclude with various recommendations regarding the reporting and handling of missing values for future studies in clinical haematology.

Dietary and/or physical activity interventions in women with overweight or obesity prior to fertility treatment: protocol for a systematic review and individual participant data meta-analysis

INTRODUCTION

Dietary and/or physical activity interventions are often recommended for women with overweight or obesity as the first step prior to fertility treatment. However, randomised controlled trials (RCTs) so far have shown inconsistent results. Therefore, we propose this individual participant data meta-analysis (IPDMA) to evaluate the effectiveness and safety of dietary and/or physical activity interventions in women with infertility and overweight or obesity on reproductive, maternal and perinatal outcomes and to explore if there are subgroup(s) of women who benefit from each specific intervention or their combination (treatment-covariate interactions).

METHODS AND ANALYSIS

We will include RCTs with dietary and/or physical activity interventions as core interventions prior to fertility treatment in women with infertility and overweight or obesity. The primary outcome will be live birth. We will search MEDLINE, Embase, Cochrane Central Register of Controlled Trials and trial registries to identify eligible studies. We will approach authors of eligible trials to contribute individual participant data (IPD). We will perform risk of bias assessments according to the Risk of Bias 2 tool and a random-effects IPDMA. We will then explore treatment-covariate interactions for important participant-level characteristics.

ETHICS AND DISSEMINATION

Formal ethical approval for the project (Venus-IPD) was exempted by the medical ethics committee of the University Medical Center Groningen (METc code: 2021/563, date: 17 November 2021). Data transfer agreement will be obtained from each participating institute/hospital. Outcomes will be disseminated internationally through the collaborative group, conference presentations and peer-reviewed publication.

PROSPERO REGISTRATION NUMBER

CRD42021266201.

Sensitivity analyses for data missing at random versus missing not at random using latent growth modelling: a practical guide for randomised controlled trials

Background

Missing data are ubiquitous in randomised controlled trials. Although sensitivity analyses for different missing data mechanisms (missing at random vs. missing not at random) are widely recommended, they are rarely conducted in practice. The aim of the present study was to demonstrate sensitivity analyses for different assumptions regarding the missing data mechanism for randomised controlled trials using latent growth modelling (LGM).

Methods

Data from a randomised controlled brief alcohol intervention trial was used. The sample included 1646 adults (56% female; mean age = 31.0 years) from the general population who had received up to three individualized alcohol feedback letters or assessment-only. Follow-up interviews were conducted after 12 and 36 months via telephone. The main outcome for the analysis was change in alcohol use over time. A three-step LGM approach was used. First, evidence about the process that generated the missing data was accumulated by analysing the extent of missing values in both study conditions, missing data patterns, and baseline variables that predicted participation in the two follow-up assessments using logistic regression. Second, growth models were calculated to analyse intervention effects over time. These models assumed that data were missing at random and applied full-information maximum likelihood estimation. Third, the findings were safeguarded by incorporating model components to account for the possibility that data were missing not at random. For that purpose, Diggle-Kenward selection, Wu-Carroll shared parameter and pattern mixture models were implemented.

Results

Although the true data generating process remained unknown, the evidence was unequivocal: both the intervention and control group reduced their alcohol use over time, but no significant group differences emerged. There was no clear evidence for intervention efficacy, neither in the growth models that assumed the missing data to be at random nor those that assumed the missing data to be not at random.

Conclusion

The illustrated approach allows the assessment of how sensitive conclusions about the efficacy of an intervention are to different assumptions regarding the missing data mechanism. For researchers familiar with LGM, it is a valuable statistical supplement to safeguard their findings against the possibility of nonignorable missingness.

Trial registration

The PRINT trial was prospectively registered at the German Clinical Trials Register (DRKS00014274, date of registration: 12th March 2018).

Prediction Model Performance With Different Imputation Strategies: A Simulation Study Using a North American ICU Registry

OBJECTIVES

To evaluate the performance of pragmatic imputation approaches when estimating model coefficients using datasets with varying degrees of data missingness.

DESIGN

Performance in predicting observed mortality in a registry dataset was evaluated using simulations of two simple logistic regression models with age-specific criteria for abnormal vital signs (mentation, systolic blood pressure, respiratory rate, WBC count, heart rate, and temperature). Starting with a dataset with complete information, increasing degrees of biased missingness of WBC and mentation were introduced, depending on the values of temperature and systolic blood pressure, respectively. Missing data approaches evaluated included analysis of complete cases only, assuming missing data are normal, and multiple imputation by chained equations. Percent bias and root mean square error, in relation to parameter estimates obtained from the original data, were evaluated as performance indicators.

SETTING

Data were obtained from the Virtual Pediatric Systems, LLC, database (Los Angeles, CA), which provides clinical markers and outcomes in prospectively collected records from 117 PICUs in the United States and Canada.

PATIENTS

Children admitted to a participating PICU in 2017, for whom all required data were available.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Simulations demonstrated that multiple imputation by chained equations is an effective strategy and that even a naive implementation of multiple imputation by chained equations significantly outperforms traditional approaches: the root mean square error for model coefficients was lower using multiple imputation by chained equations in 90 of 99 of all simulations (91%) compared with discarding cases with missing data and lower in 97 of 99 (98%) compared with models assuming missing values are in the normal range. Assuming missing data to be abnormal was inferior to all other approaches.

CONCLUSIONS

Analyses of large observational studies are likely to encounter the issue of missing data, which are likely not missing at random. Researchers should always consider multiple imputation by chained equations (or similar imputation approaches) when encountering even only small proportions of missing data in their work.

BDNF levels in adolescent patients with anorexia nervosa increase continuously to supranormal levels 2.5 years after first hospitalization

BACKGROUND

Brain-derived neurotrophic factor (BDNF) influences brain plasticity and feeding behaviour, and it has been linked to anorexia nervosa in numerous studies. Findings in mostly adult patients point to reduced serum BDNF levels in the acute stage of anorexia nervosa and rising levels with weight recovery. However, it is unclear whether this increase leads to normalization or supranormal levels, a difference that is potentially important for the etiology of anorexia nervosa and relapse.

METHODS

We measured serum BDNF at admission (n = 149), discharge (n = 130), 1-year follow-up (n = 116) and 2.5-year follow-up (n = 76) in adolescent female patients with anorexia nervosa hospitalized for the first time, and in healthy controls (n = 79). We analyzed associations with body mass index, eating disorder psychopathology and comorbidities.

RESULTS

Serum BDNF was only nominally lower at admission in patients with anorexia nervosa compared to healthy controls, but it increased continuously and reached supranormal levels at 2.5-year follow-up. BDNF was inversely associated with eating disorder psychopathology at discharge and positively associated with previous weight gain at 1-year follow-up.

LIMITATIONS

We compensated for attrition and batch effects using statistical measures.

CONCLUSION

In this largest longitudinal study to date, we found only nonsignificant reductions in BDNF in the acute stage of anorexia nervosa, possibly because of a shorter illness duration in adolescent patients. Supranormal levels of BDNF at 2.5-year follow-up could represent a pre-existing trait or a consequence of the illness. Because of the anorexigenic effect of BDNF, it might play an important predisposing role for relapse and should be explored further in studies that test causality.

Missing not at random in end of life care studies: multiple imputation and sensitivity analysis on data from the ACTION study

Background

Missing data are common in end-of-life care studies, but there is still relatively little exploration of which is the best method to deal with them, and, in particular, if the missing at random (MAR) assumption is valid or missing not at random (MNAR) mechanisms should be assumed. In this paper we investigated this issue through a sensitivity analysis within the ACTION study, a multicenter cluster randomized controlled trial testing advance care planning in patients with advanced lung or colorectal cancer.

Methods

Multiple imputation procedures under MAR and MNAR assumptions were implemented. Possible violation of the MAR assumption was addressed with reference to variables measuring quality of life and symptoms. The MNAR model assumed that patients with worse health were more likely to have missing questionnaires, making a distinction between single missing items, which were assumed to satisfy the MAR assumption, and missing values due to completely missing questionnaire for which a MNAR mechanism was hypothesized. We explored the sensitivity to possible departures from MAR on gender differences between key indicators and on simple correlations.

Results

Up to 39% of follow-up data were missing. Results under MAR reflected that missingness was related to poorer health status. Correlations between variables, although very small, changed according to the imputation method, as well as the differences in scores by gender, indicating a certain sensitivity of the results to the violation of the MAR assumption.

Conclusions

The findings confirmed the importance of undertaking this kind of analysis in end-of-life care studies.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-020-01180-y.

Potential impact of missing outcome data on treatment effects in systematic reviews: imputation study

OBJECTIVE

To assess the risk of bias associated with missing outcome data in systematic reviews.

DESIGN

Imputation study.

SETTING

Systematic reviews.

POPULATION

100 systematic reviews that included a group level meta-analysis with a statistically significant effect on a patient important dichotomous efficacy outcome.

MAIN OUTCOME MEASURES

Median percentage change in the relative effect estimate when applying each of the following assumption (four commonly discussed but implausible assumptions (best case scenario, none had the event, all had the event, and worst case scenario) and four plausible assumptions for missing data based on the informative missingness odds ratio (IMOR) approach (IMOR 1.5 (least stringent), IMOR 2, IMOR 3, IMOR 5 (most stringent)); percentage of meta-analyses that crossed the threshold of the null effect for each method; and percentage of meta-analyses that qualitatively changed direction of effect for each method. Sensitivity analyses based on the eight different methods of handling missing data were conducted.

RESULTS

100 systematic reviews with 653 randomised controlled trials were included. When applying the implausible but commonly discussed assumptions, the median change in the relative effect estimate varied from 0% to 30.4%. The percentage of meta-analyses crossing the threshold of the null effect varied from 1% (best case scenario) to 60% (worst case scenario), and 26% changed direction with the worst case scenario. When applying the plausible assumptions, the median percentage change in relative effect estimate varied from 1.4% to 7.0%. The percentage of meta-analyses crossing the threshold of the null effect varied from 6% (IMOR 1.5) to 22% (IMOR 5) of meta-analyses, and 2% changed direction with the most stringent (IMOR 5).

CONCLUSION

Even when applying plausible assumptions to the outcomes of participants with definite missing data, the average change in pooled relative effect estimate is substantive, and almost a quarter (22%) of meta-analyses crossed the threshold of the null effect. Systematic review authors should present the potential impact of missing outcome data on their effect estimates and use this to inform their overall GRADE (grading of recommendations assessment, development, and evaluation) ratings of risk of bias and their interpretation of the results.

Meta-Analyses Proved Inconsistent in How Missing Data Were Handled Across Their Included Primary Trials: A Methodological Survey

Background

How systematic review authors address missing data among eligible primary studies remains uncertain.

Objective

To assess whether systematic review authors are consistent in the way they handle missing data, both across trials included in the same meta-analysis, and with their reported methods.

Methods

We first identified 100 eligible systematic reviews that included a statistically significant meta-analysis of a patient-important dichotomous efficacy outcome. Then, we successfully retrieved 638 of the 653 trials included in these systematic reviews’ meta-analyses. From each trial report, we extracted statistical data used in the analysis of the outcome of interest to compare with the data used in the meta-analysis. First, we used these comparisons to classify the “analytical method actually used” for handling missing data by the systematic review authors for each included trial. Second, we assessed whether systematic reviews explicitly reported their analytical method of handling missing data. Third, we calculated the proportion of systematic reviews that were consistent in their “analytical method actually used” across trials included in the same meta-analysis. Fourth, among systematic reviews that were consistent in the “analytical method actually used” across trials and explicitly reported on a method for handling missing data, we assessed whether the “analytical method actually used” and the reported methods were consistent.

Results

We were unable to determine the “analytical method reviews actually used” for handling missing outcome data among 397 trials. Among the remaining 241, systematic review authors most commonly conducted “complete case analysis” (n=128, 53%) or assumed “none of the participants with missing data had the event of interest” (n=58, 24%). Only eight of 100 systematic reviews were consistent in their approach to handling missing data across included trials, but none of these reported methods for handling missing data. Among seven reviews that did explicitly report their analytical method of handling missing data, only one was consistent in their approach across included trials (using complete case analysis), and their approach was inconsistent with their reported methods (assumed all participants with missing data had the event).

Conclusion

The majority of systematic review authors were inconsistent in their approach towards reporting and handling missing outcome data across eligible primary trials, and most did not explicitly report their methods to handle missing data. Systematic review authors should clearly identify missing outcome data among their eligible trials, specify an approach for handling missing data in their analyses, and apply their approach consistently across all primary trials.

Intergenerational transmission of child maltreatment using a multi-informant multi-generation family design

In the current study a three-generational design was used to investigate intergenerational transmission of child maltreatment (ITCM) using multiple sources of information on child maltreatment: mothers, fathers and children. A total of 395 individuals from 63 families reported on maltreatment. Principal Component Analysis (PCA) was used to combine data from mother, father and child about maltreatment that the child had experienced. This established components reflecting the convergent as well as the unique reports of father, mother and child on the occurrence of maltreatment. Next, we tested ITCM using the multi-informant approach and compared the results to those of two more common approaches: ITCM based on one reporter and ITCM based on different reporters from each generation. Results of our multi-informant approach showed that a component reflecting convergence between mother, father, and child reports explained most of the variance in experienced maltreatment. For abuse, intergenerational transmission was consistently found across approaches. In contrast, intergenerational transmission of neglect was only found using the perspective of a single reporter, indicating that transmission of neglect might be driven by reporter effects. In conclusion, the present results suggest that including multiple informants may be necessary to obtain more valid estimates of ITCM.

The application of unsupervised deep learning in predictive models using electronic health records

Background

The main goal of this study is to explore the use of features representing patient-level electronic health record (EHR) data, generated by the unsupervised deep learning algorithm autoencoder, in predictive modeling. Since autoencoder features are unsupervised, this paper focuses on their general lower-dimensional representation of EHR information in a wide variety of predictive tasks.

Methods

We compare the model with autoencoder features to traditional models: logistic model with least absolute shrinkage and selection operator (LASSO) and Random Forest algorithm. In addition, we include a predictive model using a small subset of response-specific variables (Simple Reg) and a model combining these variables with features from autoencoder (Enhanced Reg). We performed the study first on simulated data that mimics real world EHR data and then on actual EHR data from eight Advocate hospitals.

Results

On simulated data with incorrect categories and missing data, the precision for autoencoder is 24.16% when fixing recall at 0.7, which is higher than Random Forest (23.61%) and lower than LASSO (25.32%). The precision is 20.92% in Simple Reg and improves to 24.89% in Enhanced Reg. When using real EHR data to predict the 30-day readmission rate, the precision of autoencoder is 19.04%, which again is higher than Random Forest (18.48%) and lower than LASSO (19.70%). The precisions for Simple Reg and Enhanced Reg are 18.70 and 19.69% respectively. That is, Enhanced Reg can have competitive prediction performance compared to LASSO. In addition, results show that Enhanced Reg usually relies on fewer features under the setting of simulations of this paper.

Conclusions

We conclude that autoencoder can create useful features representing the entire space of EHR data and which are applicable to a wide array of predictive tasks. Together with important response-specific predictors, we can derive efficient and robust predictive models with less labor in data extraction and model training.

Patients and investigators prefer measures of absolute risk in subgroups for pragmatic randomized trials

OBJECTIVES

Pragmatic randomized trials are important tools for shared decision-making, but no guidance exists on patients' preferences for types of causal information. We aimed to assess preferences of patients and investigators toward causal effects in pragmatic randomized trials.

STUDY DESIGN AND SETTING

We (a) held three focus groups with patients (n = 23) in Boston, MA; (b) surveyed (n = 12) and interviewed (n = 5) investigators with experience conducting pragmatic trials; and (c) conducted a systematic literature review of pragmatic trials (n = 63).

RESULTS

Patients were distrustful of new-to-market medications unless substantially more effective than existing choices, preferred stratified absolute risks, and valued adherence-adjusted analyses when they expected to adhere. Investigators wanted both intention-to-treat and per-protocol effects but felt methods for estimating per-protocol effects were lacking. When estimating per-protocol effects, many pragmatic trials used inappropriate methods to adjust for adherence and loss to follow-up.

CONCLUSION

We made four recommendations for pragmatic trials to improve patient centeredness: (1) focus on superiority in effectiveness or safety, rather than noninferiority; (2) involve patients in specifying a priori subgroups; (3) report absolute measures of risk; and (4) complement intention-to-treat effect estimates with valid per-protocol effect estimates.

The treatment of incomplete data: Reporting, analysis, reproducibility, and replicability

Proper analysis and reporting of incomplete data continues to be a challenging task for practitioners from various research areas. Recently Nguyen, Strazdins, Nicholson and Cooklin (NSNC; 2018) evaluated the impact of complete case analysis and multiple imputation in studies of parental employment and health. Their work joins interdisciplinary efforts to educate and motivate scientists across the research community to use principled statistical methods when analyzing incomplete data. Although we fully support and encourage work in parallel to NSNC's, we also think that further actions should be taken by the research community to improve current practices. In this commentary, we discuss some aspects and misconceptions related to analysis of incomplete data, in particular multiple imputation. In our view, the missing data problem is part of a larger problem of research reproducibility and replicability today. Thus, we believe that improving analysis and reporting of incomplete data will make reproducibility and replicability efforts easier. We also provide a brief checklist of recommendations which could be used by members of the scientific community, including practitioners, journal editors, and reviewers to set higher publication standards.