How easily can omission of patients, or selection amongst poorly-reproducible measurements, create artificial correlations? Methods for detection and implications for observational research design in cardiology

Discrepancies in autologous bone marrow stem cell trials and enhancement of ejection fraction (DAMASCENE): weighted regression and meta-analysis

OBJECTIVE

To investigate whether discrepancies in trials of use of bone marrow stem cells in patients with heart disease account for the variation in reported effect size in improvement of left ventricular function.

DESIGN

Identification and counting of factual discrepancies in trial reports, and sample size weighted regression against therapeutic effect size. Meta-analysis of trials that provided sufficient information.

DATA SOURCES

PubMed and Embase from inception to April 2013.

ELIGIBILITY FOR SELECTING STUDIES

Randomised controlled trials evaluating the effect of autologous bone marrow stem cells for heart disease on mean left ventricular ejection fraction.

RESULTS

There were over 600 discrepancies in 133 reports from 49 trials. There was a significant association between the number of discrepancies and the reported increment in EF with bone marrow stem cell therapy (Spearman's r=0.4, P=0.005). Trials with no discrepancies were a small minority (five trials) and showed a mean EF effect size of -0.4%. The 24 trials with 1-10 discrepancies showed a mean effect size of 2.1%. The 12 with 11-20 discrepancies showed a mean effect of size 3.0%. The three with 21-30 discrepancies showed a mean effect size of 5.7%. The high discrepancy group, comprising five trials with over 30 discrepancies each, showed a mean effect size of 7.7%.

CONCLUSIONS

Avoiding discrepancies is difficult but is important because discrepancy count is related to effect size. The mechanism is unknown but should be explored in the design of future trials because in the five trials without discrepancies the effect of bone marrow stem cell therapy on ejection fraction is zero.

Cardiac resynchronization therapy and AV optimization increase myocardial oxygen consumption, but increase cardiac function more than proportionally

BACKGROUND

The mechanoenergetic effects of atrioventricular delay optimization during biventricular pacing ("cardiac resynchronization therapy", CRT) are unknown.

METHODS

Eleven patients with heart failure and left bundle branch block (LBBB) underwent invasive measurements of left ventricular (LV) developed pressure, aortic flow velocity-time-integral (VTI) and myocardial oxygen consumption (MVO2) at 4 pacing states: biventricular pacing (with VV 0 ms) at AVD 40 ms (AV-40), AVD 120 ms (AV-120, a common nominal AV delay), at their pre-identified individualised haemodynamic optimum (AV-Opt); and intrinsic conduction (LBBB).

RESULTS

AV-120, relative to LBBB, increased LV developed pressure by a mean of 11(SEM 2)%, p=0.001, and aortic VTI by 11(SEM 3)%, p=0.002, but also increased MVO2 by 11(SEM 5)%, p=0.04. AV-Opt further increased LV developed pressure by a mean of 2(SEM 1)%, p=0.035 and aortic VTI by 4(SEM 1)%, p=0.017. MVO2 trended further up by 7(SEM 5)%, p=0.22. Mechanoenergetics at AV-40 were no different from LBBB. The 4 states lay on a straight line for Δexternal work (ΔLV developed pressure × Δaortic VTI) against ΔMVO2, with slope 1.80, significantly >1 (p=0.02).

CONCLUSIONS

Biventricular pacing and atrioventricular delay optimization increased external cardiac work done but also myocardial oxygen consumption. Nevertheless, the increase in cardiac work was ~80% greater than the increase in oxygen consumption, signifying an improvement in cardiac mechanoenergetics. Finally, the incremental effect of optimization on external work was approximately one-third beyond that of nominal AV pacing, along the same favourable efficiency trajectory, suggesting that AV delay dominates the biventricular pacing effect - which may therefore not be mainly "resynchronization".

Autologous bone marrow-derived stem cell therapy in heart disease: discrepancies and contradictions

BACKGROUND

Autologous bone marrow stem cell therapy is the greatest advance in the treatment of heart disease for a generation according to pioneering reports. In response to an unanswered letter regarding one of the largest and most promising trials, we attempted to summarise the findings from the most innovative and prolific laboratory.

METHOD AND RESULTS

Amongst 48 reports from the group, there appeared to be 5 actual clinical studies ("families" of reports). Duplicate or overlapping reports were common, with contradictory experimental design, recruitment and results. Readers cannot always tell whether a study is randomised versus not, open-controlled or blinded placebo-controlled, or lacking a control group. There were conflicts in recruitment dates, criteria, sample sizes, million-fold differences in cell counts, sex reclassification, fractional numbers of patients and conflation of competitors' studies with authors' own. Contradictory results were also common. These included arithmetical miscalculations, statistical errors, suppression of significant changes, exaggerated description of own findings, possible silent patient deletions, fractional numbers of coronary arteries, identical results with contradictory sample sizes, contradictory results with identical sample sizes, misrepresented survival graphs and a patient with a negative NYHA class. We tabulate over 200 discrepancies amongst the reports. The 5 family-flagship papers (Strauer 2002, STAR, IACT, ABCD, BALANCE) have had 2665 citations. Of these, 291 citations were to the pivotal STAR or IACT-JACC papers, but 97% of their eligible citing papers did not mention any discrepancies. Five meta-analyses or systematic reviews covered these studies, but none described any discrepancies and all resolved uncertainties by undisclosed methods, in mutually contradictory ways. Meta-analysts disagreed whether some studies were randomised or "accepter-versus-rejecter". Our experience of presenting the discrepancies to journals is that readers may remain unaware of such problems.

CONCLUSIONS

Modern reporting of clinical research can still be imperfect. The scientific literature absorbs such reports largely uncritically. Even meta-analyses seem to resolve contradictions haphazardly. Discrepancies communicated to journals are not guaranteed to reach the scientific community. Journals could consider prioritising systematic reporting of queries even if seemingly minor, and establishing a policy of "habeas data".

Comparison of different invasive hemodynamic methods for AV delay optimization in patients with cardiac resynchronization therapy: implications for clinical trial design and clinical practice

Background

Reproducibility and hemodynamic efficacy of optimization of AV delay (AVD) of cardiac resynchronization therapy (CRT) using invasive LV dp/dt_max are unknown.

Method and results

25 patients underwent AV delay (AVD) optimisation twice, using continuous left ventricular (LV) dp/dt_max, systolic blood pressure (SBP) and pulse pressure (PP). We compared 4 protocols for comparing dp/dt_max between AV delays:

Immediate absolute: mean of 10 s recording of dp/dt_max acquired immediately after programming the tested AVD,

Delayed absolute: mean of 10 s recording acquired 30 s after programming AVD,

Single relative: relative difference between reference AVD and the tested AVD,

Multiple relative: averaged difference, from multiple alternations between reference and tested AVD.

We assessed for dp/dt_max, LVSBP and LVPP, test–retest reproducibility of the optimum.

Optimization using immediate absolute dp/dt_max had poor reproducibility (SDD of replicate optima = 41 ms; R² = 0.45) as did delayed absolute (SDD 39 ms; R² = 0.50). Multiple relative had better reproducibility: SDD 23 ms, R² = 0.76, and (p < 0.01 by F test).

Compared with AAI pacing, the hemodynamic increment from CRT, with the nominal AV delay was LVSBP 2% and LVdp/dt_max 5%, while CRT with pre-determined optimal AVD gave 6% and 9% respectively.

Conclusions

Because of inevitable background fluctuations, optimization by absolute dp/dt_max has poor same-day reproducibility, unsuitable for clinical or research purposes. Reproducibility is improved by comparing to a reference AVD and making multiple consecutive measurements. More than 6 measurements would be required for even more precise optimization — and might be advisable for future study designs. With optimal AVD, instead of nominal, the hemodynamic increment of CRT is approximately doubled.

Evidence-based recommendations for PISA measurements in mitral regurgitation: systematic review, clinical and in-vitro study

Background

Guidelines for quantifying mitral regurgitation (MR) using “proximal isovelocity surface area” (PISA) instruct operators to measure the PISA radius from valve orifice to Doppler flow convergence “hemisphere”. Using clinical data and a physically-constructed MR model we (A) analyse the actually-observed colour Doppler PISA shape and (B) test whether instructions to measure a “hemisphere” are helpful.

Methods and results

In part A, the true shape of PISA shells was investigated using three separate approaches. First, a systematic review of published examples consistently showed non-hemispherical, “urchinoid” shapes. Second, our clinical data confirmed that the Doppler-visualized surface is non-hemispherical. Third, in-vitro experiments showed that round orifices never produce a colour Doppler hemisphere.

In part B, six observers were instructed to measure hemisphere radius r_h and (on a second viewing) urchinoid distance (d_u) in 11 clinical PISA datasets; 6 established experts also measured PISA distance as the gold standard. r_h measurements, generated using the hemisphere instruction significantly underestimated expert values (− 28%, p < 0.0005), meaning r_h² was underestimated by approximately 2-fold. d_u measurements, generated using the non-hemisphere instruction were less biased (+ 7%, p = 0.03).

Finally, frame-to-frame variability in PISA distance was found to have a coefficient of variation (CV) of 25% in patients and 9% in in-vitro data. Beat-to-beat variability had a CV of 15% in patients.

Conclusions

Doppler-visualized PISA shells are not hemispherical: we should avoid advising observers to measure a hemispherical radius because it encourages underestimation of orifice area by approximately two-fold. If precision is needed (e.g. to detect changes reliably) multi-frame averaging is essential.

A systematic approach to designing reliable VV optimization methodology: assessment of internal validity of echocardiographic, electrocardiographic and haemodynamic optimization of cardiac resynchronization therapy

Background

In atrial fibrillation (AF), VV optimization of biventricular pacemakers can be examined in isolation. We used this approach to evaluate internal validity of three VV optimization methods by three criteria.

Methods and results

Twenty patients (16 men, age 75 ± 7) in AF were optimized, at two paced heart rates, by LVOT VTI (flow), non-invasive arterial pressure, and ECG (minimizing QRS duration). Each optimization method was evaluated for: singularity (unique peak of function), reproducibility of optimum, and biological plausibility of the distribution of optima.

The reproducibility (standard deviation of the difference, SDD) of the optimal VV delay was 10 ms for pressure, versus 8 ms (p = ns) for QRS and 34 ms (p < 0.01) for flow.

Singularity of optimum was 85% for pressure, 63% for ECG and 45% for flow (Chi² = 10.9, p < 0.005).

The distribution of pressure optima was biologically plausible, with 80% LV pre-excited (p = 0.007). The distributions of ECG (55% LV pre-excitation) and flow (45% LV pre-excitation) optima were no different to random (p = ns).

The pressure-derived optimal VV delay is unaffected by the paced rate: SDD between slow and fast heart rate is 9 ms, no different from the reproducibility SDD at both heart rates.

Conclusions

Using non-invasive arterial pressure, VV delay optimization by parabolic fitting is achievable with good precision, satisfying all 3 criteria of internal validity. VV optimum is unaffected by heart rate. Neither QRS minimization nor LVOT VTI satisfy all validity criteria, and therefore seem weaker candidate modalities for VV optimization. AF, unlinking interventricular from atrioventricular delay, uniquely exposes resynchronization concepts to experimental scrutiny.