Bias Estimation in Method Comparison Studies

Preanalytical investigations of phlebotomy: methodological aspects, pitfalls and recommendations

Phlebotomy is often addressed as a crucial process in the pre-analytical phase, in which a large part of laboratory errors take place, but to date there is not yet a consolidated methodological paradigm. Seeking literature, we found 36 suitable investigations issued between 1996 and 2016 (April) dealing with the investigation of pre-analytical factors related to phlebotomy. We found that the largest part of studies had a cohort of healthy volunteers (22/36) or outpatients (11/36), with the former group showing a significantly smaller median sample size (N = 20, IQR: 17.5-30 and N = 88, IQR: 54.5-220.5 respectively, P < 0.001). Moreover, the largest part investigated one pre-analytical factor (26/36) and regarded more than one laboratory test (29/36), and authors preferably used paired Student’s t-test (17/36) or Wilcoxon’s test (11/36), but calibration (i.e. sample size calculation for a detectable effect) was addressed only in one manuscript. The Bland-Altman plot was often the preferred method used to estimate bias (12/36), as well as the Passing-Bablok regression for agreement (8/36). However, often papers did assess neither bias (12/36) nor agreement (24/36). Clinical significance of bias was preferably assessed comparing to a database value (16/36), and it resulted uncorrelated with the size of the effect produced by the factor (P = 0.142). However, the median effect size (ES) resulted significantly larger if the associated factor was clinically significant instead of non-significant (ES = 1.140, IQR: 0.815-1.700 and ES = 0.349, IQR: 0.228-0.531 respectively, P < 0.001). On these evidences, we discussed some recommendations for improving methodological consistency, delivering reliable results, as well as ensuring accessibility to practical evidences.

Visual, tactile, and contact force feedback: which one is more important for catheter ablation? Results from an in vitro experimental study

BACKGROUND

During radiofrequency ablation, effective contact is crucial in determining lesions efficacy.

OBJECTIVE

The purpose of this study was to compare operators' ability to assess contact pressure using visual and tactile feedbacks together or alone in an experimental model.

METHODS

In a in vitro experimental setup replicating manual catheter manipulation and recording the applied force, evaluators were asked to identify three levels of force (first, ablation, and maximum contact) as the catheter contacted the tissue model using (1) visual feedback only by fluoroscopy, "blinded" to touch; (2) tactile feedback only, blinded to fluoroscopy; and (3) both tactile and visual feedback together. The latter was regarded as reference. The experiment was repeated using a catheter force sensing technology during robotic navigation.

RESULTS

During manual navigation, tighter association was shown for the visual method than for the tactile method: median difference with reference: first contact -1 (P = .97) vs -2 (P = .90); ablation contact 2 (P = .1) vs -7 (P = .03); maximum contact 2 (P = .06) vs -28 (P = .02). Bland-Altman plot and Deming regression confirmed for the visual method the good agreement with reference and the absence of bias at any level and showed for the tactile higher values and proportional bias that reached statistical significance at ablation and maximum contact. During robotic navigation, agreement was higher for the tactile than for the visual only method.

CONCLUSION

During manual navigation, visual feedback alone is in better agreement with the reference compared to the tactile only approach. During robotic navigation, agreement is looser for the visual only approach. More objective feedback of contact pressure during ablation procedures is desirable.

Comparison of the ATS versus EU Mini Wright peak flow meter in normal volunteers

OBJECTIVES

The American Thoracic Society (ATS) and European Union (EU) have precise and accurate Mini Wright peak flow meters. The purpose of this investigation was to compare both 1) for accuracy using a pneumotachometer, 2) in volunteers to determine whether they are interchangeable, and 3) to spirometrically predicted peak flows.

METHODS

Lab testing: A pneumotachometer was connected in series with each peak flow meter and varying flows pushed through both meters for comparison. Human subjects: Nonsmoking adult volunteers did three standing peak flows. The order of peak flow meter used was random. The best of three efforts was used for analysis. The t-test, concordance correlation coefficient (CCC), Deming regression, and Bland-Altman plot were the analytic strategies used to determine agreement. Peak flow results were compared to spirometrically predicted values.

RESULTS

Fifty-seven volunteers, average age 37 ± 12 years and mean BMI 24.9 ± 2.5 years, were included. The average peak flows were different at 541 ± 114 and 526 ± 112 L/min for the ATS and EU meters, respectively (p < .01). Both peak flow meter values were significantly different than spirometrically predicted values of 483 ± 86 L/min (p < .01). The CCC was 0.98 (0.97-0.99) and regression revealed a slope and y-intercept consistent with 1 and 0, respectively. The Bland-Altman plot revealed no increase in scatter of values over the range of peak flows versus the difference with a mean bias of 15 ± 15 L/min. Laboratory testing revealed that the ATS and EU peak flow meters read 3.0 ± 2.1% above and -2.0 ± 1.5% below the comparison pneumotachometer, respectively. The pneumotachometer comparison was significantly different for both meters at p < .01, paired t-test.

CONCLUSIONS

The ATS peak flow meter reads 2.8% higher than the EU peak flow meter across a range of flows. Both meters have similar accuracy with a different bias compared with a pneumotachometer. Finally, both peak flow meters read slightly and significantly higher than spirometrically derived peak flows. Therefore, the peak flow meters are not interchangeable and both may obtain slightly higher values than those determined using current spirometrically derived prediction equations.

A rapid and reliable method of counting neurons and other cells in brain tissue: a comparison of flow cytometry and manual counting methods

It is of critical importance to understand the numbers and distributions of neurons and non-neurons in the cerebral cortex because cell numbers are reduced with normal aging and by diseases of the CNS. The isotropic fractionator method provides a faster way of estimating numbers of total cells and neurons in whole brains and dissected brain parts. Several comparative studies have illustrated the accuracy and utility of the isotropic fractionator method, yet it is a relatively new methodology, and there is opportunity to adjust procedures to optimize its efficiency and minimize error. In the present study, we use 142 samples from a dissected baboon cortical hemisphere to evaluate if isotropic fractionator counts using a Neubauer counting chamber and fluorescence microscopy could be accurately reproduced using flow cytometry methods. We find greater repeatability in flow cytometry counts, and no evidence of constant or proportional bias when comparing microscopy to flow cytometry counts. We conclude that cell number estimation using a flow cytometer is more efficient and more precise than comparable counts using a Neubauer chamber on a fluorescence microscope. This method for higher throughput, precise estimation of cell numbers has the potential to rapidly advance research in post-mortem human brains and vastly improve our understanding of cortical and subcortical structures in normal, injured, aged, and diseased brains.

Estimation of bias between 2 analytical methods at clinically important ranges

An approach for estimating bias between 2 analytical methods at different clinical ranges is introduced in this article. The approach models replicated data obtained from the reference and the test method in terms of repeatability and trueness bias. The latter can be partitioned into constant and proportional bias. The approach is based on maximum likelihood estimation and can accommodate normal as well as Poisson and binomial distributions that apply to hematology applications and/or other laboratory methods that count particles per unit of volume and/or time. A full spectrum of statistical inference in the form of confidence intervals for each estimate as well as any statistical hypothesis testing is provided. At the same time these estimates can be practically interpreted and related to any clinical important range or decision point. We recommend this approach as an alternative to the National Committee for Clinical Laboratory Standards (NCCLS) EP9-A2 approach in cases where the application of the NCCLS standard is not appropriate.

Whole blood fixation and permeabilization protocol with red blood cell lysis for flow cytometry of intracellular phosphorylated epitopes in leukocyte subpopulations

BACKGROUND

Previous studies of intracellular expression of phospho-epitopes in human leukocytes using flow cytometry have used erythrocyte removal or lysis before fixation. Because many of the phospho-epitopes of interest are part of signaling networks that respond to the environment and turn over rapidly, the interval and manipulations used to eliminate erythrocytes from samples have the potential to introduce artifacts. We report a procedure to fix samples containing red blood cells with formaldehyde and then remove erythrocytes by lysis. Detection of phospho-Thr 202/Tyr 204-p44/42 extracellular-regulated kinase (ERK) after phorbol ester acetate (PMA) stimulation was used as a model to measure phospho-epitopes in leukocyte populations in whole blood.

METHODS

Normal blood samples were activated with PMA followed by formaldehyde fixation and subsequent treatments with detergents and protein denaturants. The effects of each treatment were monitored by light scatter, selected CD expression intensity, and phosphorylated ERK (pERK) expression.

RESULTS

Red cells could be lysed using 0.1% Triton X-100 after brief fixation of whole blood with 2% or 4% formaldehyde. Light scatter improved as a function of formaldehyde concentration and inversely with MeOH concentration. CD3 signal intensity increased when MeOH concentration was reduced. The ratio of pERK immunofluorescence in PMA-stimulated versus nonstimulated (control) samples was highest with high MeOH (90%) and lowest without MeOH treatment. This pattern is consistent with epitope unmasking by alcohol. The pERK epitope could also be unmasked by treatment with high salt, urea, acid, or heat, but none of these produced the level of unmasking of MeOH and each of these was associated with degradation of light scatter and CD3 staining intensity. The final procedure employed 4% formaldehyde, 0.1% Triton X-100, followed by 50% methanol denaturation. Samples prepared in this way demonstrated good preservation of light scatter and surface immunophenotypic patterns, similar to those obtained using a commercial whole blood/red blood cell lysing system (Q-Prep) and an acceptable PMA-stimulated pERK signal (essentially 100% of CD3+ cells that are pERK positive).

CONCLUSIONS

Brief fixation of whole blood in 4% formaldehyde followed by treatment with Triton X-100 results in erythrocyte lysis and leukocyte light scatter and immunophenotypic features equivalent to those of other commercial lysis reagents. Intracellular pERK staining is significantly improved by treatment with methanol, but levels of MeOH above 50% degrade light scatter and CD3 expression. This protocol (formaldehyde/Triton X-100/MeOH) circumvents potential artifactual changes in phospho-epitopes due to removal of erythrocytes or erythrocyte lysis followed by fixation, and results in a pERK signal that resolves positive from negative cell populations.