Common pitfalls in statistical analysis: "P" values, statistical significance and confidence intervals

In the second part of a series on pitfalls in statistical analysis, we look at various ways in which a statistically significant study result can be expressed. We debunk some of the myths regarding the ‘P’ value, explain the importance of ‘confidence intervals’ and clarify the importance of including both values in a paper

Introducing the Concept of the Minimally Important Difference to Determine a Clinically Relevant Change on Patient-Reported Outcome Measures in Patients with Intermittent Claudication

Purpose

The minimally important difference (MID) represents the smallest change in score on patient-reported outcome measures that is relevant to patients. The aim of this study was to introduce the MID for the Vascular Quality of Life Questionnaire (VascuQol) and the walking impairment questionnaire (WIQ) for patients with intermittent claudication (IC).

Methods

In this multicenter study, we recruited 294 patients with IC between July and October 2012. Patients completed the VascuQol, with scores ranging from 1 to 7 (worst to best), and the WIQ, with scores ranging from 0 to 1 (worst to best) at first visit and after 4 months follow-up. In addition, patients answered an anchor-question rating their health status compared to baseline, as being improved, unchanged, or deteriorated. The MID for improvement and deterioration was calculated by an anchor-based approach, and determined with the upper and lower limits of the 95 % confidence interval of the mean change of the group who had not changed according to the anchor-question.

Results

For the MID analyses of the VascuQol and WIQ, 163 and 134 patients were included, respectively. The MID values for the VascuQol (mean baseline score 4.25) were 0.87 for improvement and 0.23 for deterioration. For the WIQ (mean baseline score 0.39), we found MID values of 0.11 and −0.03 for improvement and deterioration, respectively.

Conclusion

In this study, we calculated the MID for the VascuQol and the WIQ. Applying these MID facilitates better interpretation of treatment outcomes and can help to set treatment goals for individual care.

Electronic supplementary material

The online version of this article (doi:10.1007/s00270-015-1060-0) contains supplementary material, which is available to authorized users.

Common pitfalls in statistical analysis: "No evidence of effect" versus "evidence of no effect"

This article is the first in a series exploring common pitfalls in statistical analysis in biomedical research. The power of a clinical trial is the ability to find a difference between treatments, where such a difference exists. At the end of the study, the lack of difference between treatments does not mean that the treatments can be considered equivalent. The distinction between "no evidence of effect" and "evidence of no effect" needs to be understood.

Effects of presenting risk information in different formats to cardiologists. A Latin American survey

OBJECTIVE

Previous publications demonstrated that the presentation of treatment benefits in terms of relative risk reduction (RRR) rather than in terms of absolute risk reduction (ARR) or number of patients to treat (NNT) might favor the perception of outcome effectiveness. The objective was to perform a cognitive evaluation to assess how the manner in which risks and benefits of screening methods and treatments are presented can affect medical care decision-taking in a sample of cardiologists.

METHODS

Four-hundred and six Latin American cardiologists answered a questionnaire reporting the results of clinical trials presented as RRR, ARR or NNT, associated or not to biased graphs.

RESULTS

Cardiologists' decision-taking was different when comparing treatment benefits presented as RRR (62.2%) vs. ARR (40.4%) (p=0.000000), and as RRR vs. NNT (44.4%) (p=0.000000). However, their decision-taking was similar when information was presented as NNT or ARR (p=0.073). The inclusion of biased graphs was misinterpreted as an actual data difference (RRR: 61.6% vs. ARR: 14.0%, p=0.000000).

CONCLUSIONS

This study demonstrated that Latin American cardiologists could misinterpret statistical data when information of clinical trials is presented in terms of RRR. We emphasize the need to enhance cardiologists' training in quantitative techniques, to improve medical care decision-making.

Analysis of Statistical Methods Currently used in Toxicology Journals

Statistical methods are frequently used in toxicology, yet it is not clear whether the methods employed by the studies are used consistently and conducted based on sound statistical grounds. The purpose of this paper is to describe statistical methods used in top toxicology journals. More specifically, we sampled 30 papers published in 2014 from Toxicology and Applied Pharmacology, Archives of Toxicology, and Toxicological Science and described methodologies used to provide descriptive and inferential statistics. One hundred thirteen endpoints were observed in those 30 papers, and most studies had sample size less than 10, with the median and the mode being 6 and 3 & 6, respectively. Mean (105/113, 93%) was dominantly used to measure central tendency, and standard error of the mean (64/113, 57%) and standard deviation (39/113, 34%) were used to measure dispersion, while few studies provide justifications regarding why the methods being selected. Inferential statistics were frequently conducted (93/113, 82%), with one-way ANOVA being most popular (52/93, 56%), yet few studies conducted either normality or equal variance test. These results suggest that more consistent and appropriate use of statistical method is necessary which may enhance the role of toxicology in public health.

Does waist circumference uncorrelated with BMI add valuable information?

BACKGROUND

Estimation of relative contribution of Body Mass Index (BMI) and waist circumference (WC) on health outcomes requires a regression model that includes both obesity metrics. But, multicollinearity could yield biased estimates.

METHODS

To address the multicollinearity issue between BMI and WC, we used the residual model approach. The standard WC (Y-axis) was regressed on the BMI (X-axis) to obtain residual WC. Data from two adult population surveys (Nunavik Inuit and James Bay Cree) were analysed to evaluate relative effect of BMI and WC on four cardiometabolic risk factors: insulin, triglycerides, systolic blood pressure and high-density lipoprotein levels.

RESULTS

In multivariate models, standard WC and BMI were significantly associated with cardiometabolic outcomes. Residual WC was not linked with any outcomes. The BMI effect was weakened by including standard WC in the model, but its effect remained unchanged if residual WC was considered.

CONCLUSIONS

The strong correlation between standard WC and BMI does not allow assessment of their relative contributions to health in the same model without a risk of making erroneous estimations. By contrast with BMI, fat distribution (residual WC) does not add valuable information to a model that already contains overall adiposity (BMI) in Inuit and Cree.

An interactive Bayesian model for prediction of lymph node ratio and survival in pancreatic cancer patients

BACKGROUND

Regional lymph node status has long been used as a dichotomous predictor of clinical outcomes in cancer patients. More recently, interest has turned to the prognostic utility of lymph node ratio (LNR), quantified as the proportion of positive nodes examined. However, statistical tools for the joint modeling of LNR and its effect on cancer survival are lacking.

METHODS

Data were obtained from the NCI SEER cancer registry on 6400 patients diagnosed with pancreatic ductal adenocarcinoma from 2004 to 2010 and who underwent radical oncologic resection. A novel Bayesian statistical approach was developed and applied to model simultaneously patients' true, but unobservable, LNR statuses and overall survival. New web development tools were then employed to create an interactive web application for individualized patient prediction.

RESULTS

Histologic grade and T and M stages were important predictors of LNR status. Significant predictors of survival included age, gender, marital status, grade, histology, T and M stages, tumor size, and radiation therapy. LNR was found to have a highly significant, non-linear effect on survival. Furthermore, predictive performance of the survival model compared favorably to those from studies with more homogeneous patients and individualized predictors.

CONCLUSIONS

We provide a new approach and tool set for the prediction of LNR and survival that are generally applicable to a host of cancer types, including breast, colon, melanoma, and stomach. Our methods are illustrated with the development of a validated model and web applications for the prediction of survival in a large set of pancreatic cancer patients.

Manipulating measurement scales in medical statistical analysis and data mining: A review of methodologies

BACKGROUND

selecting the correct statistical test and data mining method depends highly on the measurement scale of data, type of variables, and purpose of the analysis. Different measurement scales are studied in details and statistical comparison, modeling, and data mining methods are studied based upon using several medical examples. We have presented two ordinal-variables clustering examples, as more challenging variable in analysis, using Wisconsin Breast Cancer Data (WBCD).

ORDINAL-TO-INTERVAL SCALE CONVERSION EXAMPLE

a breast cancer database of nine 10-level ordinal variables for 683 patients was analyzed by two ordinal-scale clustering methods. The performance of the clustering methods was assessed by comparison with the gold standard groups of malignant and benign cases that had been identified by clinical tests.

RESULTS

the sensitivity and accuracy of the two clustering methods were 98% and 96%, respectively. Their specificity was comparable.

CONCLUSION

by using appropriate clustering algorithm based on the measurement scale of the variables in the study, high performance is granted. Moreover, descriptive and inferential statistics in addition to modeling approach must be selected based on the scale of the variables.

Heterogeneity of schizoaffective disorder compared with schizophrenia and bipolar disorder

OBJECTIVE

Low diagnostic reliability, the need to meet criteria of two disorders, and its status as residual diagnosis in clinical practice led us to hypothesize that schizoaffective disorder (SAD) is characterized by considerable heterogeneity, particularly in comparison with schizophrenia (SZ) and bipolar disorder (BD). As this has not been investigated the aim of this study is to test whether heterogeneity is larger in SAD than in SZ and BD.

METHOD

Systematic search for studies simultaneously comparing all three diagnoses regarding demographic, clinical, psychometric (clinical rating scales and IQ tests), and biological parameters; comparison of heterogeneity as measured by standard deviation (SD).

RESULTS

Standard deviation of SAD samples (N = 47) was smaller than in both differential diagnoses. SDs were 7% higher in BD than in SAD (SZ: 2% higher); in studies employing DSM-IIIR/-IV pooled SD was 4% higher in BD (8% lower in SZ). Differences between diagnoses were limited to the comparison of SAD and BD, and became smaller when only psychotic BD was considered.

CONCLUSION

Heterogeneity of SZ and BD is not smaller than that of SAD. SAD seems not to be more diverse than other functional psychoses. Results are preliminary because of the novelty of the approach and to the small number of studies.

Reporting of harms data in RCTs: a systematic review of empirical assessments against the CONSORT harms extension

OBJECTIVE

To determine the standard of reporting of harms-related data, in randomised controlled trials (RCTs) according to the Consolidated Standards of Reporting Trials (CONSORT) statement extension for harms.

DESIGN

Systematic review.

DATA SOURCES

The Cochrane library, Ovid MEDLINE, Scopus and ISI Web of Knowledge were searched for relevant literature.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

We included publications of studies that used the CONSORT harms extension to assess the reporting of harms in RCTs.

RESULTS

We identified 7 studies which included between 10 and 205 RCTs. The clinical areas of the 7 studies were: hypertension (1), urology (1), epilepsy (1), complimentary medicine (2) and two not restricted to a clinical topic. Quality of the 7 studies was assessed by a risk of bias tool and was found to be variable. Adherence to the CONSORT harms criteria reported in the 7 studies was inadequate and variable across the items in the checklist. Adverse events are poorly defined, with 6 studies failing to exceed 50% adherence to the items in the checklist.

CONCLUSIONS

Readers of RCT publications need to be able to balance the trade-offs between benefits and harms of interventions. This systematic review suggests that this is compromised due to poor reporting of harms which is evident across a range of clinical areas. Improvements in quality could be achieved by wider adoption of the CONSORT harms criteria by journals reporting RCTs.

Statistical reporting deficiencies in environmental toxicology

Null hypothesis significance testing is one of the most widely used forms of statistical testing in environmental toxicology. In this short communication, the authors show that the reporting of statistical information when using null hypothesis significance testing is frequently inadequate in environmental toxicology research. The authors demonstrate this by analyzing the statistical information reported for papers employing t tests or analyses of variance in the Environmental Toxicology section of Environmental Toxicology and Chemistry in 2010, which comprised 68% of papers published by this journal in that year. Of these papers, 60% fail to report exact p values, 85% fail to provide degrees of freedom, and 90% fail to report critical effect sizes. Statistical power was reported in only <2% of the published papers. The insufficient provision of statistical information makes interpretation of study results by reviewers and readers difficult. Consistently reporting exact p values with degrees of freedom, considering and explicitly stating biologically relevant critical effect sizes, and reporting statistical power associated with nonsignificant results would be easy to implement and would promote scientific progress in environmental toxicology through increased statistical transparency.

The views of Academic Staff on Biostatistics Education in Health Sciences

OBJECTIVE

The aim of this study was to assess the level of importance, adequacy and need of biostatistics and biostatistics education regarding health sciences in Turkey.

METHODOLOGY

Within the scope of the study a survey (questionnaire) was applied to a total of 237 academicians (academicians holding various titles employed by different departments) officiating at medical, veterinary medicine and health sciences faculties of six universities (Afyon Kocatepe, Gazi, Ankara, Hacettepe, Marmara ve Düzce) in Turkey. With this survey were taken views of academics on the status of the training they had received regarding biostatistics, their need for biostatistics education, the importance of biostatistics education and its level of adequacy, the source and adequacy of current information on biostatistics, the solutions to meet biostatistics needs and the statistical techniques which were required.

RESULTS

According to the results, 27.8% of the participants have not received biostatistics education. It was determined that the importance of biostatistics education was emphasized as "very and exactly important" by 88.19% of the participants. 14.35% of the participants reported that biostatistics education was far from adequate for post graduate. They needed biostatistics knowledge mainly in the analysis of researches data (χ̄=4.01). On the other hand, they met biostatistics knowledge needs mostly by "help from friends".

CONCLUSIONS

The study revealed that most of the academicians found biostatistics education important but claimed it had been inadequate. They emphasized the need for the organization of courses, seminars, etc. regarding biostatistics at sufficient intervals. As a consequence of that, this study reveals the importance of biostatistics and biostatistics education once again in the data analysis process in health sciences.

Peri-implant bone length changes and survival rates of implants penetrating the sinus membrane at the posterior maxilla in patients with limited vertical bone height

Purpose

The aim of this study was to measure the peri-implant bone length surrounding implants that penetrate the sinus membrane at the posterior maxilla and to evaluate the survival rate of these implants.

Methods

Treatment records and orthopantomographs of 39 patients were reviewed and analyzed. The patients had partial edentulism at the posterior maxilla and limited vertical bone height below the maxillary sinus. Implants were inserted into the posterior maxilla, penetrating the sinus membrane. Four months after implant insertion, provisional resin restorations were temporarily cemented to the abutments and used for one month. Then, a final impression was taken at the abutment level, and final cement-retained restorations were delivered with mutually protected occlusion. The complications from the implant surgery were examined, the number of failed implants was counted, and the survival rate was calculated. The peri-implant bone lengths were measured using radiographs. The changes in initial and final peri-implant bone lengths were statistically analyzed.

Results

Nasal bleeding occurred after implant surgery in three patients. No other complications were found. There were no failures of the investigated implants, resulting in a survival rate of 100%. Significantly more bone gain around the implants (estimated difference=-0.6 mm, P=0.025) occurred when the initial residual bone height was less than 5 mm compared to the >5 mm groups. No significant change in peri-implant bone length was detected when the initial residual bone height was 5 mm or larger.

Conclusions

This study suggests that implants penetrating the sinus membrane at the posterior maxilla in patients with limited vertical bone height may be safe and functional.

Statistics: The stethoscope of a thinking urologist

Understanding statistical terminology and the ability to appraise clinical research findings and statistical tests are critical to the practice of evidence-based medicine. Urologists require statistics in their toolbox of skills in order to successfully sift through increasingly complex studies and realize the drawbacks of statistical tests. Currently, the level of evidence in urology literature is low and the majority of research abstracts published for the American Urological Association (AUA) meetings lag behind for full-text publication because of a lack of statistical reporting. Underlying these issues is a distinct deficiency in solid comprehension of statistics in the literature and a discomfort with the application of statistics for clinical decision-making. This review examines the plight of statistics in urology and investigates the reason behind the white-coat aversion to biostatistics. Resources such as evidence-based medicine websites, primers in statistics, and guidelines for statistical reporting exist for quick reference by urologists. Ultimately, educators should take charge of monitoring statistical knowledge among trainees by bolstering competency requirements and creating sustained opportunities for statistics and methodology exposure.

Modeling sequence and function similarity between proteins for protein functional annotation

A common task in biological research is to predict function for proteins by comparing sequences between proteins of known and unknown function. This is often done using pair-wise sequence alignment algorithms (e.g. BLAST). A problem with this approach is the assumption of a simple equivalence between a minimum sequence similarity threshold and the function similarity between proteins. This assumption is based on the binary concept of homology in that proteins are or not homologous. The relationship between sequence and function however is more complex as well as pertinent for predicting protein function, e.g. evaluating BLAST alignments or developing training sets for profile models based on functional rather than homologous groupings. Our motivation for this study was to model sequence and function similarity between proteins to gain insights into the "sequence-function similarity relationship between proteins for predicting function. Using our model we found that function similarity generally increases with sequence similarity but with a high degree of variability. This result has implications for pair-wise approaches in that it appears sequence similarity must be very high to ensure high function similarity. Profile models which enable higher sensitivity are a potential solution. However, multiple sequences alignments (a necessary prerequisite) are a problem in that current algorithms have difficulty aligning sequences with very low sequence similarity, which is common in our data set, or are intractable for high numbers of sequences. Given the importance of predicting protein function and the need for multiple sequence alignments, algorithms for accomplishing this task should be further refined and developed.

Procedures for numerical analysis of circadian rhythms

This article reviews various procedures used in the analysis of circadian rhythms at the populational, organismal, cellular and molecular levels. The procedures range from visual inspection of time plots and actograms to several mathematical methods of time series analysis. Computational steps are described in some detail, and additional bibliographic resources and computer programs are listed.

What does biostatistics mean to us

It is human nature to try to recognize patterns and to make sense of that which we observe. Unfortunately, our intuition is often wrong, and so there is a need to impose some objectivity on the methods by which observations are converted into knowledge. One definition of biostatistics could be precisely this, the rigorous and objective conversion of medical and/or biological observations into knowledge. Both consumers of biostatistical principles and biostatisticians themselves vary in the extent to which they recognize the need to continue the improvement. Some may not recognize the need for (some or all of) the methods that have already been developed; others may accept these as they find them completely sufficient; still others recognize both the value and the shortcomings of these methods, and seek to develop even better methods to ensure that future medical conclusions are less subject to biases than current ones are.