Missing data: Discussion points from the PSI missing data expert group

Reference-based multiple imputation for missing data sensitivity analyses in trial-based cost-effectiveness analysis

Missing data are a common issue in cost-effectiveness analysis (CEA) alongside randomised trials and are often addressed assuming the data are 'missing at random'. However, this assumption is often questionable, and sensitivity analyses are required to assess the implications of departures from missing at random. Reference-based multiple imputation provides an attractive approach for conducting such sensitivity analyses, because missing data assumptions are framed in an intuitive way by making reference to other trial arms. For example, a plausible not at random mechanism in a placebo-controlled trial would be to assume that participants in the experimental arm who dropped out stop taking their treatment and have similar outcomes to those in the placebo arm. Drawing on the increasing use of this approach in other areas, this paper aims to extend and illustrate the reference-based multiple imputation approach in CEA. It introduces the principles of reference-based imputation and proposes an extension to the CEA context. The method is illustrated in the CEA of the CoBalT trial evaluating cognitive behavioural therapy for treatment-resistant depression. Stata code is provided. We find that reference-based multiple imputation provides a relevant and accessible framework for assessing the robustness of CEA conclusions to different missing data assumptions.

The effect of combined functional anaerobic and strength training on treadmill gait kinematics and kinetics in ambulatory young adults with cerebral palsy

BACKGROUND

Leg muscle weakness is a major impairment for individuals with cerebral palsy (CP) and is related to reduced functional capacity. Evidence is limited regarding the translation of strength improvements following conventional resistance training to improved gait outcomes.

RESEARCH QUESTION

Does a combined functional anaerobic and lower limb strength training intervention improve gait kinematics and kinetics in individuals with CP aged 15-30 years? 17 young adults (21 ± 4 years, 9 males, GMFCS I = 11, II = 6) were randomized to 12 weeks, 3 sessions per week, of high intensity functional anaerobic and progressive resistance training of the lower limbs (n = 8), or a waitlist control group (n = 9). Pre- and post-training outcomes included maximum ankle dorsiflexion angle at foot contact and during stance, gait profile score, ankle and hip power generation during late stance, and the ratio of ankle to hip power generation.

RESULTS

There were no between-group differences after the intervention for any kinematic or kinetic gait outcome variable. Within-group analysis revealed an increase in peak ankle power during late stance (0.31 ± 0.28 W·kg^-1, p = 0.043) and ankle to hip power ratio (0.43 ± 0.37, p = 0.034) following training in the intervention group.

SIGNIFICANCE

We have previously reported increased overground walking capacity, agility and sprint power, in the training group compared to the control group at 12-weeks. These changes in overground measures of functional capacity occurred in the absence of changes in treadmill gait kinematics and kinetics reported here.

ANZCTR

12614001217695.

Sensitivity Analysis for Not-at-Random Missing Data in Trial-Based Cost-Effectiveness Analysis: A Tutorial

Cost-effectiveness analyses (CEA) of randomised controlled trials are a key source of information for health care decision makers. Missing data are, however, a common issue that can seriously undermine their validity. A major concern is that the chance of data being missing may be directly linked to the unobserved value itself [missing not at random (MNAR)]. For example, patients with poorer health may be less likely to complete quality-of-life questionnaires. However, the extent to which this occurs cannot be ascertained from the data at hand. Guidelines recommend conducting sensitivity analyses to assess the robustness of conclusions to plausible MNAR assumptions, but this is rarely done in practice, possibly because of a lack of practical guidance. This tutorial aims to address this by presenting an accessible framework and practical guidance for conducting sensitivity analysis for MNAR data in trial-based CEA. We review some of the methods for conducting sensitivity analysis, but focus on one particularly accessible approach, where the data are multiply-imputed and then modified to reflect plausible MNAR scenarios. We illustrate the implementation of this approach on a weight-loss trial, providing the software code. We then explore further issues around its use in practice.

Missing data in trial-based cost-effectiveness analysis: An incomplete journey

Cost-effectiveness analyses (CEA) conducted alongside randomised trials provide key evidence for informing healthcare decision making, but missing data pose substantive challenges. Recently, there have been a number of developments in methods and guidelines addressing missing data in trials. However, it is unclear whether these developments have permeated CEA practice. This paper critically reviews the extent of and methods used to address missing data in recently published trial-based CEA. Issues of the Health Technology Assessment journal from 2013 to 2015 were searched. Fifty-two eligible studies were identified. Missing data were very common; the median proportion of trial participants with complete cost-effectiveness data was 63% (interquartile range: 47%-81%). The most common approach for the primary analysis was to restrict analysis to those with complete data (43%), followed by multiple imputation (30%). Half of the studies conducted some sort of sensitivity analyses, but only 2 (4%) considered possible departures from the missing-at-random assumption. Further improvements are needed to address missing data in cost-effectiveness analyses conducted alongside randomised trials. These should focus on limiting the extent of missing data, choosing an appropriate method for the primary analysis that is valid under contextually plausible assumptions, and conducting sensitivity analyses to departures from the missing-at-random assumption.

The treatment of incomplete data: Reporting, analysis, reproducibility, and replicability

Proper analysis and reporting of incomplete data continues to be a challenging task for practitioners from various research areas. Recently Nguyen, Strazdins, Nicholson and Cooklin (NSNC; 2018) evaluated the impact of complete case analysis and multiple imputation in studies of parental employment and health. Their work joins interdisciplinary efforts to educate and motivate scientists across the research community to use principled statistical methods when analyzing incomplete data. Although we fully support and encourage work in parallel to NSNC's, we also think that further actions should be taken by the research community to improve current practices. In this commentary, we discuss some aspects and misconceptions related to analysis of incomplete data, in particular multiple imputation. In our view, the missing data problem is part of a larger problem of research reproducibility and replicability today. Thus, we believe that improving analysis and reporting of incomplete data will make reproducibility and replicability efforts easier. We also provide a brief checklist of recommendations which could be used by members of the scientific community, including practitioners, journal editors, and reviewers to set higher publication standards.

Changing physical activity behaviour for people with multiple sclerosis: protocol of a randomised controlled feasibility trial (iStep-MS)

INTRODUCTION

Although physical activity may reduce disease burden, fatigue and disability, and improve quality of life among people with multiple sclerosis (MS), many people with MS are physically inactive and spend significant time in sedentary behaviour. Behaviour change interventions may assist people with MS to increase physical activity and reduce sedentary behaviour. However, few studies have investigated their effectiveness using objective measures of physical activity, particularly in the long term. Further, interventions that have proven effective in the short term may not be feasible in clinical practice because of the large amount of support provided. The iStep-MS trial aims to determine the safety, feasibility and acceptability of a behaviour change intervention to increase physical activity and reduce sedentary behaviour among people with MS.

METHODS AND ANALYSIS

Sixty people with MS will be randomised (1:1 ratio) to receive a 12-week intervention or usual care only. The intervention consists of four physical activity consultations with a physiotherapist supported by a handbook and pedometer. Outcomes assessed at baseline, 12 weeks and 9 months are physical activity (ActiGraph wGT3X-BT accelerometer), sedentary behaviour (activPAL3µ), self-reported activity and sitting time, walking capability, fatigue, self-efficacy, participation, quality of life and health service use. The safety of the intervention will be determined by assessing change in pain and fatigue and the incidence of adverse events during the follow-up period. A parallel process evaluation will assess the feasibility and acceptability of the intervention through assessment of fidelity to the programme and semistructured interviews exploring participants' and therapists' experiences of the intervention. The feasibility of conducting an economic evaluation will be determined by collecting data on quality of life and resource use.

ETHICS AND DISSEMINATION

Research ethics committee approval has been granted from Brunel University London. Results of the trial will be submitted for publication in journals and distributed to people with MS and physiotherapists.

TRIAL REGISTRATION NUMBER

ISRCTN15343862 (doi 10.1186/ISRCTN15343862). Protocol version: 1.0; Pre-results.

Self-monitoring of lower leg skin temperature: accuracy of self-reported data and adherence to a cooling protocol for the prevention of venous leg ulcers

BACKGROUND

For intervention studies that require the use of participant self-reports, the quality and accuracy of recorded data and variability in participant adherence rates to the treatment can cause significant outcome bias.

PURPOSE

To assess the quality and accuracy of participant documentation of daily self-monitoring of leg skin temperature, adherence to a graduated cooling treatment protocol to prevent venous leg ulcers, and the potential for bias in treatment effect in a randomized controlled trial that included a population with chronic venous disease.

METHODS

Individuals were randomized to a leg cooling intervention or placebo treatment group to daily self-monitor and record lower leg skin temperature over a 9-month period on monthly paper study logs. Returned study logs for the first 100 completed participants (n=54 cooling intervention, n=46 control) were reviewed for quality and accuracy. Adherence was determined from evaluating the accuracy of participant documentation. To examine potential outcome bias in treatment effect, mean between group and within group comparisons of the before and after treatment differences were conducted using an intention-to-treat (ITT) versus a modified intention-to-treat (mITT) analysis approach with an 85% accuracy cut-off rate. Data were collected in 2011-2014.

RESULTS

Of the expected 900 study logs, 91.8% (826/900) were returned and 8.2% (74/900) were not. Non-mutually exclusive main error types in returned documentation included: 59.2% (489/826) white-outs, cross-off and/or overwrites, 34.9% (288/826) entries omitted, 29.4% (243/826) no performance of daily self-monitoring, 28.7% (237/826) no performance of the treatment intervention per the prescribed protocol regime, 26.8% (221/826) extraneous data, 8.6% (71/826) suspected fabrication, and 7.6% (63/826) questionable validity. Under ITT analysis, 38.4% (346/900) of all returned logs were <85% accurate, 25.0% (225/900) were 85%-99% accurate, and 36.6% (329/900) were 100% accurate. Mean overall participant adherence rates were: 22.0% at <85% accuracy, 53.0% at 85%-99% accuracy, and 25.0% at 100% accuracy. Under the mITT analysis, 54.0% (483/900) of returned logs were deemed adherent with ≥85% accuracy.

CONCLUSION

This study found good rates of adherence. Under ITT analysis, 78.0% of participants were deemed adherent to the study protocol with ≥85% accuracy in documenting daily self-monitoring of skin temperatures in response to a topically applied experimental cooling cuff intervention for the prevention of venous leg ulcers.

Analysis of longitudinal trials with protocol deviation: a framework for relevant, accessible assumptions, and inference via multiple imputation

Protocol deviations, for example, due to early withdrawal and noncompliance, are unavoidable in clinical trials. Such deviations often result in missing data. Additional assumptions are then needed for the analysis, and these cannot be definitively verified from the data at hand. Thus, as recognized by recent regulatory guidelines and reports, clarity about these assumptions and their implications is vital for both the primary analysis and framing relevant sensitivity analysis. This article focuses on clinical trials with longitudinal quantitative outcome data. For the target population, we define two estimands, the de jure estimand, "does the treatment work under the best case scenario," and the de facto estimand, "what would be the effect seen in practice." We then carefully define the concept of a deviation from the protocol relevant to the estimand, or for short a deviation. Each patient's postrandomization data can then be divided into predeviation data and postdeviation data. We set out an accessible framework for contextually appropriate assumptions relevant to de facto and de jure estimands, that is, assumptions about the joint distribution of pre- and postdeviation data relevant to the clinical question at hand. We then show how, under these assumptions, multiple imputation provides a practical approach to estimation and inference. We illustrate with data from a longitudinal clinical trial in patients with chronic asthma.

Sensitivity analysis for a partially missing binary outcome in a two-arm randomized clinical trial

Although recent guidelines for dealing with missing data emphasize the need for sensitivity analyses, and such analyses have a long history in statistics, universal recommendations for conducting and displaying these analyses are scarce. We propose graphical displays that help formalize and visualize the results of sensitivity analyses, building upon the idea of 'tipping-point' analysis for randomized experiments with a binary outcome and a dichotomous treatment. The resulting 'enhanced tipping-point displays' are convenient summaries of conclusions obtained from making different modeling assumptions about missingness mechanisms. The primary goal of the displays is to make formal sensitivity analysesmore comprehensible to practitioners, thereby helping them assess the robustness of the experiment's conclusions to plausible missingness mechanisms. We also present a recent example of these enhanced displays in amedical device clinical trial that helped lead to FDA approval.

Sensitivity analysis of incomplete longitudinal data departing from the missing at random assumption: Methodology and application in a clinical trial with drop-outs

Statistical analyses of longitudinal data with drop-outs based on direct likelihood, and using all the available data, provide unbiased and fully efficient estimates under some assumptions about the drop-out mechanism. Unfortunately, these assumptions can never be tested from the data. Thus, sensitivity analyses should be routinely performed to assess the robustness of inferences to departures from these assumptions. However, each specific scientific context requires different considerations when setting up such an analysis, no standard method exists and this is still an active area of research. We propose a flexible procedure to perform sensitivity analyses when dealing with continuous outcomes, which are described by a linear mixed model in an initial likelihood analysis. The methodology relies on the pattern-mixture model factorisation of the full data likelihood and was validated in a simulation study. The approach was prompted by a randomised clinical trial for sleep-maintenance insomnia treatment. This case study illustrated the practical value of our approach and underlined the need for sensitivity analyses when analysing data with drop-outs: some of the conclusions from the initial analysis were shown to be reliable, while others were found to be fragile and strongly dependent on modelling assumptions. R code for implementation is provided.

Including all individuals is not enough: lessons for intention-to-treat analysis

BACKGROUND

Intention-to-treat (ITT) analysis requires all randomised individuals to be included in the analysis in the groups to which they were randomised. However, there is confusion about how ITT analysis should be performed in the presence of missing outcome data.

PURPOSES

To explain, justify, and illustrate an ITT analysis strategy for randomised trials with incomplete outcome data.

METHODS

We consider several methods of analysis and compare their underlying assumptions, plausibility, and numbers of individuals included. We illustrate the ITT analysis strategy using data from the UK700 trial in the management of severe mental illness.

RESULTS

Depending on the assumptions made about the missing data, some methods of analysis that include all randomised individuals may be less valid than methods that do not include all randomised individuals. Furthermore, some methods of analysis that include all randomised individuals are essentially equivalent to methods that do not include all randomised individuals.

LIMITATIONS

This work assumes that the aim of analysis is to obtain an accurate estimate of the difference in outcome between randomised groups and not to obtain a conservative estimate with bias against the experimental intervention.

CONCLUSIONS

Clinical trials should employ an ITT analysis strategy, comprising a design that attempts to follow up all randomised individuals, a main analysis that is valid under a stated plausible assumption about the missing data, and sensitivity analyses that include all randomised individuals in order to explore the impact of departures from the assumption underlying the main analysis. Following this strategy recognises the extra uncertainty arising from missing outcomes and increases the incentive for researchers to minimise the extent of missing data.