Common statistical and research design problems in manuscripts submitted to high-impact medical journals

Teaching students to R3eason, not merely to solve problem sets: The role of philosophy and visual data communication in accessible data science education

Much guidance on statistical training in STEM fields has been focused largely on the undergraduate cohort, with graduate education often being absent from the equation. Training in quantitative methods and reasoning is critical for graduate students in biomedical and science programs to foster reproducible and responsible research practices. We argue that graduate student education should more center around fundamental reasoning and integration skills rather than mainly on listing 1 statistical test method after the other without conveying the bigger context picture or critical argumentation skills that will enable student to improve research integrity through rigorous practice. Herein, we describe the approach we take in a quantitative reasoning course in the R3 program at the Johns Hopkins Bloomberg School of Public Health, with an error-focused lens, based on visualization and communication competencies. Specifically, we take this perspective stemming from the discussed causes of irreproducibility and apply it specifically to the many aspects of good statistical practice in science, ranging from experimental design to data collection and analysis, and conclusions drawn from the data. We also provide tips and guidelines for the implementation and adaptation of our course material to various graduate biomedical and STEM science programs.

Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

Background: Papers describing the results of a randomised trial should include a baseline table that compares the characteristics of randomised groups. Researchers who fraudulently generate trials often unwittingly create baseline tables that are implausibly similar (under-dispersed) or have large differences between groups (over-dispersed). I aimed to create an automated algorithm to screen for under- and over-dispersion in the baseline tables of randomised trials. Methods: Using a cross-sectional study I examined 2,245 randomised controlled trials published in health and medical journals on PubMed Central. I estimated the probability that a trial's baseline summary statistics were under- or over-dispersed using a Bayesian model that examined the distribution of t-statistics for the between-group differences, and compared this with an expected distribution without dispersion. I used a simulation study to test the ability of the model to find under- or over-dispersion and compared its performance with an existing test of dispersion based on a uniform test of p-values. My model combined categorical and continuous summary statistics, whereas the uniform test used only continuous statistics. Results: The algorithm had a relatively good accuracy for extracting the data from baseline tables, matching well on the size of the tables and sample size. Using t-statistics in the Bayesian model out-performed the uniform test of p-values, which had many false positives for skewed, categorical and rounded data that were not under- or over-dispersed. For trials published on PubMed Central, some tables appeared under- or over-dispersed because they had an atypical presentation or had reporting errors. Some trials flagged as under-dispersed had groups with strikingly similar summary statistics. Conclusions: Automated screening for fraud of all submitted trials is challenging due to the widely varying presentation of baseline tables. The Bayesian model could be useful in targeted checks of suspected trials or authors.

Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

Background: Papers describing the results of a randomised trial should include a baseline table that compares the characteristics of randomised groups. Researchers who fraudulently generate trials often unwittingly create baseline tables that are implausibly similar (under-dispersed) or have large differences between groups (over-dispersed). I aimed to create an automated algorithm to screen for under- and over-dispersion in the baseline tables of randomised trials. Methods: Using a cross-sectional study I examined 2,245 randomised controlled trials published in health and medical journals on PubMed Central. I estimated the probability that a trial's baseline summary statistics were under- or over-dispersed using a Bayesian model that examined the distribution of t-statistics for the between-group differences, and compared this with an expected distribution without dispersion. I used a simulation study to test the ability of the model to find under- or over-dispersion and compared its performance with an existing test of dispersion based on a uniform test of p-values. My model combined categorical and continuous summary statistics, whereas the uniform test used only continuous statistics. Results: The algorithm had a relatively good accuracy for extracting the data from baseline tables, matching well on the size of the tables and sample size. Using t-statistics in the Bayesian model out-performed the uniform test of p-values, which had many false positives for skewed, categorical and rounded data that were not under- or over-dispersed. For trials published on PubMed Central, some tables appeared under- or over-dispersed because they had an atypical presentation or had reporting errors. Some trials flagged as under-dispersed had groups with strikingly similar summary statistics. Conclusions: Automated screening for fraud of all submitted trials is challenging due to the widely varying presentation of baseline tables. The Bayesian model could be useful in targeted checks of suspected trials or authors.

Body image as a multidimensional concept: A systematic review of body image facets in eating disorders and muscle dysmorphia

Body image disturbance is core to the psychopathology of eating disorders (EDs), and related disorders such as muscle dysmorphia (MD). Global measures of body image fail to quantify specific aspects of body image disturbance that characterizes EDs, and may be differentially associated to outcomes. The aim of this systematic review was to provide an overview of specific body image facets and synthesize findings from controlled studies that compared clinical ED/MD and control-comparison groups in body image disturbance. One-hundred sixty-seven studies met inclusion criteria, and reported on comparisons among 30,584 individuals in 28 body image facets, which were more broadly grouped into evaluative, perceptual, cognitive-affective and motivational categories for the purpose of the present review. Effect sizes were calculated as Cohen's d for every comparison between ED and control groups. Body dissatisfaction (evaluative category) was the most prevalent facet assessed across studies (62 %), and differences between clinical and control groups were the largest in this category, especially for bulimia nervosa (d = 1.37). Scarcity of studies with male and MD clinical samples, and use of single-item and non-validated measures, should encourage development of instruments for body image facets pertinent to EDs and MD that can be validly applied across gender.

How to write statistical analysis section in medical research

Reporting of statistical analysis is essential in any clinical and translational research study. However, medical research studies sometimes report statistical analysis that is either inappropriate or insufficient to attest to the accuracy and validity of findings and conclusions. Published works involving inaccurate statistical analyses and insufficient reporting influence the conduct of future scientific studies, including meta-analyses and medical decisions. Although the biostatistical practice has been improved over the years due to the involvement of statistical reviewers and collaborators in research studies, there remain areas of improvement for transparent reporting of the statistical analysis section in a study. Evidence-based biostatistics practice throughout the research is useful for generating reliable data and translating meaningful data to meaningful interpretation and decisions in medical research. Most existing research reporting guidelines do not provide guidance for reporting methods in the statistical analysis section that helps in evaluating the quality of findings and data interpretation. In this report, we highlight the global and critical steps to be reported in the statistical analysis of grants and research articles. We provide clarity and the importance of understanding study objective types, data generation process, effect size use, evidence-based biostatistical methods use, and development of statistical models through several thematic frameworks. We also provide published examples of adherence or non-adherence to methodological standards related to each step in the statistical analysis and their implications. We believe the suggestions provided in this report can have far-reaching implications for education and strengthening the quality of statistical reporting and biostatistical practice in medical research.

Assessment of long-term outcomes post living liver donation highlights the importance of scientific integrity when presenting transplant registry data

Living donor liver transplantation has expanded in recent years, particularly in North America. As experience with this procedure has matured over the last 25 years, centers are increasingly faced with potential living donors who are more medically complex. As donors move through the evaluation process, completing the informed consent process continues to be challenged by a paucity of granular data demonstrating long-term outcomes and overall safety specifically in the otherwise "healthy" living liver donor population. Two recently published studies examined long-term outcomes post-living liver donation using Korean registry data and reported similar results, with excellent overall survival when compared to appropriately matched controls. However, the authors of these studies were presented differently, with one reporting an alarmist view based on one aspect of a suboptimal analysis approach using an inappropriate comparator group. Herein, the North American Living Liver Donor Innovation Group (NALLDIG) consortium discusses these two studies and their potential impact on living liver donation in North America, ultimately highlighting the importance of scientific integrity in data presentation and dissemination when using transplant registry data.

Differential evolution and particle swarm optimization against COVID-19

COVID-19 disease, which highly affected global life in 2020, led to a rapid scientific response. Versatile optimization methods found their application in scientific studies related to COVID-19 pandemic. Differential Evolution (DE) and Particle Swarm Optimization (PSO) are two metaheuristics that for over two decades have been widely researched and used in various fields of science. In this paper a survey of DE and PSO applications for problems related with COVID-19 pandemic that were rapidly published in 2020 is presented from two different points of view: 1. practitioners seeking the appropriate method to solve particular problem, 2. experts in metaheuristics that are interested in methodological details, inter comparisons between different methods, and the ways for improvement. The effectiveness and popularity of DE and PSO is analyzed in the context of other metaheuristics used against COVID-19. It is found that in COVID-19 related studies: 1. DE and PSO are most frequently used for calibration of epidemiological models and image-based classification of patients or symptoms, but applications are versatile, even interconnecting the pandemic and humanities; 2. reporting on DE or PSO methodological details is often scarce, and the choices made are not necessarily appropriate for the particular algorithm or problem; 3. mainly the basic variants of DE and PSO that were proposed in the late XX century are applied, and research performed in recent two decades is rather ignored; 4. the number of citations and the availability of codes in various programming languages seems to be the main factors for choosing metaheuristics that are finally used.

A Review of Statistical Reporting in Dietetics Research (2010-2019): How is a Canadian Journal Doing?

Despite the widespread use of statistical techniques in quantitative research, methodological flaws and inadequate statistical reporting persist. The objective of this study is to evaluate the quality of statistical reporting and procedures in all original, quantitative articles published in the Canadian Journal of Dietetic Practice and Research (CJDPR) from 2010 to 2019 using a checklist created by our research team. In total, 107 articles were independently evaluated by 2 raters. The hypothesis or objective(s) was clearly stated in 97.2% of the studies. Over half (51.4%) of the articles reported the study design and 57.9% adequately described the statistical techniques used. Only 21.2% of the studies that required a prestudy sample size calculation reported one. Of the 281 statistical tests conducted, 88.3% of them were correct. P values >0.05-0.10 were reported as "statistically significant" and/or a "trend" in 11.4% of studies. While this evaluation reveals both strengths and areas for improvement in the quality of statistical reporting in CJDPR, we encourage dietitians to pursue additional statistical training and/or seek the assistance of a statistician. Future research should consider validating this new checklist and using it to evaluate the statistical quality of studies published in other nutrition journals and disciplines.

Meta-research publications in dentistry: a review

The present scoping review has the objective of providing an overview of meta-research in dentistry. A search of the PubMed database was performed for the period 11 October 2014 to 10 October 2019. Study selection and data extraction were performed independently by one author; prior to this, a random sample of 10% of the retrieved titles and abstracts were independently screened by two authors, achieving agreement of >80% on eligibility for initial inclusion, corresponding to good agreement. The following information was extracted from the full text of each article: meta-research area of interest; study design; type of studies evaluated in the meta-research; type of methodology used in assessment of the primary research; conflicts of interest reported; sponsorships reported; dental discipline; journal of publication; country of the first author; number of citations; and impact factor. A total of 7800 documents were initially retrieved. After analysis of the title/abstract and the full text of each article, and a snowballing procedure, 155 meta-research studies were identified and included. The 'methods' and 'reporting' meta-research areas were the most prevalent, with 73 (47%) and 61 (40%) studies, respectively. General dentistry, and orthodontics and dentofacial orthopaedics were the dental specialties with the greatest number/proportion of included studies with 45 (29%) and 28 (18%) studies, respectively. These findings may help to prioritize future meta-research in dentistry, consequently avoiding unnessecary investigations, and increasing the value of oral and dental research.

Understanding the literature: Complexity of statistical methods used in high-impact cardiothoracic surgery research

OBJECTIVE

Biostatistics are frequently used in research published in the domain of cardiothoracic surgery. The objective of this study was to describe the scope of statistical techniques reported in the literature and to highlight implications for editorial review and critical appraisal.

METHODS

Original research articles published between January and April 2017 in the Journal of Thoracic and Cardiovascular Surgery, Annals of Thoracic Surgery, and the European Journal of Cardio-Thoracic Surgery were examined. For each article, the statistical method(s) reported were recorded and categorized by complexity.

RESULTS

We reviewed 293 articles that reported 1068 statistical methods. The mean number of different statistical methods reported per article was 3.6 ± 1.9, with variation by subspecialty and journal. The most common statistical methods were contingency tables (in 59% of articles), t tests (49%), and survival methods (49%). Only 4% of articles used descriptive statistics alone. An introductory level of statistical knowledge was deemed sufficient for understanding 16% of articles, whereas for the remainder a higher level of knowledge would be needed.

CONCLUSIONS

Contemporary cardiothoracic surgery research frequently requires the use of complex statistical methods. This was evident across articles for all cardiothoracic surgical subspecialties as reported in 3 high-impact journals. Routine review of manuscript submissions by biostatisticians is needed to ensure the appropriate use and reporting of advanced statistical methods in cardiothoracic surgery research.

How often do leading biomedical journals use statistical experts to evaluate statistical methods? The results of a survey

Scientific claims in biomedical research are typically derived from statistical analyses. However, misuse or misunderstanding of statistical procedures and results permeate the biomedical literature, affecting the validity of those claims. One approach journals have taken to address this issue is to enlist expert statistical reviewers. How many journals do this, how statistical review is incorporated, and how its value is perceived by editors is of interest. Here we report an expanded version of a survey conducted more than 20 years ago by Goodman and colleagues (1998) with the intention of characterizing contemporary statistical review policies at leading biomedical journals. We received eligible responses from 107 of 364 (28%) journals surveyed, across 57 fields, mostly from editors in chief. 34% (36/107) rarely or never use specialized statistical review, 34% (36/107) used it for 10–50% of their articles and 23% used it for all articles. These numbers have changed little since 1998 in spite of dramatically increased concern about research validity. The vast majority of editors regarded statistical review as having substantial incremental value beyond regular peer review and expressed comparatively little concern about the potential increase in reviewing time, cost, and difficulty identifying suitable statistical reviewers. Improved statistical education of researchers and different ways of employing statistical expertise are needed. Several proposals are discussed.

Time-to-event survival statistics in ophthalmology: Methodological research

IMPORTANCE

Understanding the outcomes of interventions over time is essential for clinical decision making in surgical specialties.

BACKGROUND

Analysis of survival time (or time to event) is complicated when loss to follow up occurs. This article explores transparent data analysis methods where missing ("censored") data are present.

DESIGN

Manual search of the top 20 Ophthalmology journals from a recent year of the established literature (2014).

SAMPLES

A total of 4565 articles were identified, of which 218 reported outcomes of treatment over time in humans.

METHODS

Pertinent details to assist the use of Kaplan-Meier and life table actuarial statistics are explained, and criteria that define whether each has high, acceptable or poor quality are explored. The quality of reporting from the literature sample is analysed.

MAIN OUTCOME MEASURES

Reporting quality of survival curves and life tables from each sampled article is assessed according to the established criteria.

RESULTS

In total, 31.2% of samples (n = 68) presented survival curves, 53.2% (n = 116) presented life tables, 22% (n = 48) presented both, whilst 46.8% (n = 102) presented neither; 2% of survival curves and 13% of life tables were high quality, with quality of life tables significantly better than survival curves (P = .0042). 90.36% (n = 197) of articles reported time to event data which was classified as poor: due to poor analysis of survival curves (n = 50, 43.10%) poor analysis of life tables (n = 45, 66.18%); and complete omission of survival graphics (n = 102, 46.97%).

CONCLUSIONS AND RELEVANCE

Ophthalmology research that follows patient outcomes over time can be analysed with "time-to-event" statistics, and reported with transparency. This analysis showed that important contextural information was omitted from 90% of ophthalmic studies, and this could impact patient decision making.

Incorporating professional recommendations into a graduate-level statistical consulting laboratory: A case study

Introduction:

There has been a recent trend in medical research towards a more collaborative relationship between statisticians and clinical investigators. This has led to an increased focus on the most efficient and effective ways to structure, conduct, and measure the impact of organizations that provide statistical services to clinical investigators. Several recent guidelines and recommendations on the conduct of statistical consulting services(SCSs) have been made in response to this need, focusing on larger SCSs consisting primarily of faculty and staff statisticians. However, the application of these recommendations to consulting services primarily staffed by graduate students, which have the dual role of providing a professional service and training, remains unclear.

Methods:

Guidelines and recommendations, primarily from the Clinical and Translational Science (CTSA) consortium, were applied to a SCS staffed primarily by graduate students in an academic health center. A description of the organizational structure and outcomes after 3 years of operation is presented.

Results:

The guidelines recommended by the CTSA consortium and other groups were successfully incorporated into the graduate consulting laboratory. At almost one new project request per week, the consulting laboratory demonstrated a large bandwidth and had an excellent feedback from investigators.

Conclusions:

Guidelines developed for larger statistical consulting organizations are able to be applied in student-led consultation organizations. Outcomes and recommendations from 3.5 years of operation are used to describe the successes and challenges we have encountered.

Learning gaps among statistical competencies for clinical and translational science learners

INTRODUCTION

Statistical literacy is essential in clinical and translational science (CTS). Statistical competencies have been published to guide coursework design and selection for graduate students in CTS. Here, we describe common elements of graduate curricula for CTS and identify gaps in the statistical competencies.

METHODS

We surveyed statistics educators using e-mail solicitation sent through four professional organizations. Respondents rated the degree to which 24 educational statistical competencies were included in required and elective coursework in doctoral-level and master's-level programs for CTS learners. We report competency results from institutions with Clinical and Translational Science Awards (CTSAs), reflecting institutions that have invested in CTS training.

RESULTS

There were 24 CTSA-funded respondents representing 13 doctoral-level programs and 23 master's-level programs. For doctoral-level programs, competencies covered extensively in required coursework for all doctoral-level programs were basic principles of probability and hypothesis testing, understanding the implications of selecting appropriate statistical methods, and computing appropriate descriptive statistics. The only competency extensively covered in required coursework for all master's-level programs was understanding the implications of selecting appropriate statistical methods. The least covered competencies included understanding the purpose of meta-analysis and the uses of early stopping rules in clinical trials. Competencies considered to be less fundamental and more specialized tended to be covered less frequently in graduate courses.

CONCLUSION

While graduate courses in CTS tend to cover many statistical fundamentals, learning gaps exist, particularly for more specialized competencies. Educational material to fill these gaps is necessary for learners pursuing these activities.

The Statistical Curriculum Within Randomized Controlled Trials in Critical Illness

OBJECTIVES

Incomplete biostatistical knowledge among clinicians is widely described. This study aimed to categorize and summarize the statistical methodology within recent critical care randomized controlled trials.

DESIGN

Descriptive analysis, with comparison of findings to previous work.

SETTING

Ten high-impact clinical journals publishing trials in critical illness.

SUBJECTS

Randomized controlled trials published between 2011 and 2015 inclusive.

INTERVENTIONS

Data extraction from published reports.

MEASUREMENTS AND MAIN RESULTS

The frequency and overall proportion of each statistical method encountered, grouped according to those used to generate each trial's primary outcome and separately according to underlying statistical methodology. Subsequent analysis compared these proportions with previously published reports. A total of 580 statistical tests or methods were identified within 116 original randomized controlled trials published between 2011 and 2015. Overall, the chi-square test was the most commonly encountered (70/116; 60%), followed by the Cox proportional hazards model (63/116; 54%) and logistic regression (53/116; 46%). When classified according to underlying statistical assumptions, the most common types of analyses were tests of 2 × 2 contingency tables and nonparametric tests of rank order. A greater proportion of more complex methodology was observed compared with trial reports from previous work.

CONCLUSIONS

Physicians assessing recent randomized controlled trials in critical illness encounter results derived from a substantial and potentially expanding range of biostatistical methods. In-depth training in the assumptions and limitations of these current and emerging biostatistical methods may not be practically achievable for most clinicians, making accessible specialist biostatistical support an asset to evidence-based clinical practice.

The Principles of Biomedical Scientific Writing: Materials and Methods

The materials and methods (M&M) section is the heart of a scientific paper and is subject to initial screening of the editor to decide whether the manuscript should be sent for external review. If the M&M section of a scientific paper be considered as a recipe, its ingredients would be who, what, when, where, how, and why. M&M should effectively respond to the study question/hypothesis using the following basic elements including materials, study design, study population/subjects or animals, methods of measurements/assessments, and statistical analysis. A well-organized M&M permits other scientists to evaluate the study findings and repeat the experiments. Although there are several disciplinary differences in the M&M, similar dos and don'ts may be considered to organize a well-written M&M. Briefly, authors need to provide clear-cut, adequate, and detailed information in the M&M section. In this review, the structure, the principles, and the most common recommendations for writing the M&M section are provided, both in general and study-specific; these could help authors effectively prepare the M&M section of a scientific biomedical manuscript.

The Importance of Statistical Competencies for Medical Research Learners

It is very important for medical professionals and medical researchers to be literate in statistics. However, we have found that the degree of literacy that is required should not be identical for every statistical competency or even for every learner. We first begin by describing why the development, teaching, and assessment of statistical competencies for medical professionals and medical researchers are critical tasks. We next review our three substantial efforts at developing a comprehensive list of statistical competencies that can be used as a guide for what medical research learners should know about statistics, for curricular development, and for assessment of statistical education. We then summarize the origin and the inclusion of the statistical competency items. We follow this with a description of potential uses and applications of the statistical competencies to improve targeted learning for medical research learners. Finally, we discuss implications of the statistical competencies for undergraduate statistics education.

Methodological update in Medicina Intensiva

Research in the critically ill is complex by the heterogeneity of patients, the difficulties to achieve representative sample sizes and the number of variables simultaneously involved. However, the quantity and quality of records is high as well as the relevance of the variables used, such as survival. The methodological tools have evolved to offering new perspectives and analysis models that allow extracting relevant information from the data that accompanies the critically ill patient. The need for training in methodology and interpretation of results is an important challenge for the intensivists who wish to be updated on the research developments and clinical advances in Intensive Medicine.

Top ten errors of statistical analysis in observational studies for cancer research

Observational studies using registry data make it possible to compile quality information and can surpass clinical trials in some contexts. However, data heterogeneity, analytical complexity, and the diversity of aspects to be taken into account when interpreting results makes it easy for mistakes to be made and calls for mastery of statistical methodology. Some questionable research practices that include poor analytical data management are responsible for the low reproducibility of some results; yet, there is a paucity of information in the literature regarding specific statistical pitfalls of cancer studies. In addition to proposing how to avoid or solve them, this article seeks to expose ten common problematic situations in the analysis of cancer registries: convenience, dichotomization, stratification, regression to the mean, impact of sample size, competing risks, immortal time and survivor bias, management of missing values, and data dredging.

Guidelines for standardizing and increasing the transparency in the reporting of biomedical research

There is a lack of awareness about the guidelines for standardized and transparent reporting of biomedical research, among the medical professionals. This paper aims to familiarize the clinical researchers and practitioners regarding the issues related to transparency and the evolving guidelines for standardizing them, in the reporting of biomedical research. A narrative review method is adopted here, primarily based on the EQUATOR and SAMPL guidelines for reporting studies and statistical analyses. As study methods and statistical approaches support each other, their reporting practices as per the standardized guidelines have been dealt here in a congruous manner.

A simulation based method for assessing the statistical significance of logistic regression models after common variable selection procedures

Classification models can demonstrate apparent prediction accuracy even when there is no underlying relationship between the predictors and the response. Variable selection procedures can lead to false positive variable selections and overestimation of true model performance. A simulation study was conducted using logistic regression with forward stepwise, best subsets, and LASSO variable selection methods with varying total sample sizes (20, 50, 100, 200) and numbers of random noise predictor variables (3, 5, 10, 15, 20, 50). Using our critical values can help reduce needless follow-up on variables having no true association with the outcome.

Normal tissue complication probability (NTCP) modelling using spatial dose metrics and machine learning methods for severe acute oral mucositis resulting from head and neck radiotherapy

BACKGROUND AND PURPOSE

Severe acute mucositis commonly results from head and neck (chemo)radiotherapy. A predictive model of mucositis could guide clinical decision-making and inform treatment planning. We aimed to generate such a model using spatial dose metrics and machine learning.

MATERIALS AND METHODS

Predictive models of severe acute mucositis were generated using radiotherapy dose (dose-volume and spatial dose metrics) and clinical data. Penalised logistic regression, support vector classification and random forest classification (RFC) models were generated and compared. Internal validation was performed (with 100-iteration cross-validation), using multiple metrics, including area under the receiver operating characteristic curve (AUC) and calibration slope, to assess performance. Associations between covariates and severe mucositis were explored using the models.

RESULTS

The dose-volume-based models (standard) performed equally to those incorporating spatial information. Discrimination was similar between models, but the RFCstandard had the best calibration. The mean AUC and calibration slope for this model were 0.71 (s.d.=0.09) and 3.9 (s.d.=2.2), respectively. The volumes of oral cavity receiving intermediate and high doses were associated with severe mucositis.

CONCLUSIONS

The RFCstandard model performance is modest-to-good, but should be improved, and requires external validation. Reducing the volumes of oral cavity receiving intermediate and high doses may reduce mucositis incidence.

Does bad inference drive out good?

The (mis)use of statistics in practice is widely debated, and a field where the debate is particularly active is medicine. Many scholars emphasize that a large proportion of published medical research contains statistical errors. It has been noted that top class journals like Nature Medicine and The New England Journal of Medicine publish a considerable proportion of papers that contain statistical errors and poorly document the application of statistical methods. This paper joins the debate on the (mis)use of statistics in the medical literature. Even though the validation process of a statistical result may be quite elusive, a careful assessment of underlying assumptions is central in medicine as well as in other fields where a statistical method is applied. Unfortunately, a careful assessment of underlying assumptions is missing in many papers, including those published in top class journals. In this paper, it is shown that nonparametric methods are good alternatives to parametric methods when the assumptions for the latter ones are not satisfied. A key point to solve the problem of the misuse of statistics in the medical literature is that all journals have their own statisticians to review the statistical method/analysis section in each submitted paper.

Multiple imputation of multiple multi-item scales when a full imputation model is infeasible

Background

Missing data in a large scale survey presents major challenges. We focus on performing multiple imputation by chained equations when data contain multiple incomplete multi-item scales. Recent authors have proposed imputing such data at the level of the individual item, but this can lead to infeasibly large imputation models.

Methods

We use data gathered from a large multinational survey, where analysis uses separate logistic regression models in each of nine country-specific data sets. In these data, applying multiple imputation by chained equations to the individual scale items is computationally infeasible. We propose an adaptation of multiple imputation by chained equations which imputes the individual scale items but reduces the number of variables in the imputation models by replacing most scale items with scale summary scores. We evaluate the feasibility of the proposed approach and compare it with a complete case analysis. We perform a simulation study to compare the proposed method with alternative approaches: we do this in a simplified setting to allow comparison with the full imputation model.

Results

For the case study, the proposed approach reduces the size of the prediction models from 134 predictors to a maximum of 72 and makes multiple imputation by chained equations computationally feasible. Distributions of imputed data are seen to be consistent with observed data. Results from the regression analysis with multiple imputation are similar to, but more precise than, results for complete case analysis; for the same regression models a 39 % reduction in the standard error is observed. The simulation shows that our proposed method can perform comparably against the alternatives.

Conclusions

By substantially reducing imputation model sizes, our adaptation makes multiple imputation feasible for large scale survey data with multiple multi-item scales. For the data considered, analysis of the multiply imputed data shows greater power and efficiency than complete case analysis. The adaptation of multiple imputation makes better use of available data and can yield substantively different results from simpler techniques.

Electronic supplementary material

The online version of this article (doi:10.1186/s13104-016-1853-5) contains supplementary material, which is available to authorized users.

Statistical methods and errors in family medicine articles between 2010 and 2014-Suez Canal University, Egypt: A cross-sectional study

BACKGROUND

With limited statistical knowledge of most physicians it is not uncommon to find statistical errors in research articles.

OBJECTIVES

To determine the statistical methods and to assess the statistical errors in family medicine (FM) research articles that were published between 2010 and 2014.

METHODS

This was a cross-sectional study. All 66 FM research articles that were published over 5 years by FM authors with affiliation to Suez Canal University were screened by the researcher between May and August 2015. Types and frequencies of statistical methods were reviewed in all 66 FM articles. All 60 articles with identified inferential statistics were examined for statistical errors and deficiencies. A comprehensive 58-item checklist based on statistical guidelines was used to evaluate the statistical quality of FM articles.

RESULTS

Inferential methods were recorded in 62/66 (93.9%) of FM articles. Advanced analyses were used in 29/66 (43.9%). Contingency tables 38/66 (57.6%), regression (logistic, linear) 26/66 (39.4%), and t-test 17/66 (25.8%) were the most commonly used inferential tests. Within 60 FM articles with identified inferential statistics, no prior sample size 19/60 (31.7%), application of wrong statistical tests 17/60 (28.3%), incomplete documentation of statistics 59/60 (98.3%), reporting P value without test statistics 32/60 (53.3%), no reporting confidence interval with effect size measures 12/60 (20.0%), use of mean (standard deviation) to describe ordinal/nonnormal data 8/60 (13.3%), and errors related to interpretation were mainly for conclusions without support by the study data 5/60 (8.3%).

CONCLUSION

Inferential statistics were used in the majority of FM articles. Data analysis and reporting statistics are areas for improvement in FM research articles.

The Value of the P Value

Recently, the discussion on the implications of irreproducibility in the sciences has been brought into the spotlight. This topic has been discussed for years in the literature. A multitude of reasons have been attributed to this issue; one commonly labeled culprit is the overuse of the p value as a determinant of significance by the scientific community. Both scientists and statisticians have questioned the use of null hypothesis testing as the basis of scientific analysis. This survey of the current issues at hand in irreproducibility in research emphasizes potential causes of the issue, impacts that this can have for drug development and efforts been taken to increase transparency of findings in research.

Assessing statistical competencies in clinical and translational science education: one size does not fit all

INTRODUCTION

Statistics is an essential training component for a career in clinical and translational science (CTS). Given the increasing complexity of statistics, learners may have difficulty selecting appropriate courses. Our question was: what depth of statistical knowledge do different CTS learners require?

METHODS

For three types of CTS learners (principal investigator, co-investigator, informed reader of the literature), each with different backgrounds in research (no previous research experience, reader of the research literature, previous research experience), 18 experts in biostatistics, epidemiology, and research design proposed levels for 21 statistical competencies.

RESULTS

Statistical competencies were categorized as fundamental, intermediate, or specialized. CTS learners who intend to become independent principal investigators require more specialized training, while those intending to become informed consumers of the medical literature require more fundamental education. For most competencies, less training was proposed for those with more research background.

DISCUSSION

When selecting statistical coursework, the learner's research background and career goal should guide the decision. Some statistical competencies are considered to be more important than others. Baseline knowledge assessments may help learners identify appropriate coursework.

CONCLUSION

Rather than one size fits all, tailoring education to baseline knowledge, learner background, and future goals increases learning potential while minimizing classroom time.

Statistical methodology in oral and dental research: pitfalls and recommendations

OBJECTIVES

This study describes the pitfalls for commonly used statistical techniques in dental research and gives some recommendations for avoiding them. It also explores the potential of some of the newer statistical techniques for dental research.

METHODS

Each of the commonly used techniques e.g. descriptive statistics, correlation and regression, hypothesis tests (parametric and non-parametric) and survival analysis are explored with examples and recommendations for their use are provided. Common sources of error including those of study design, insufficient information, ignoring the impact of clustering and underuse of confidence intervals are outlined. The potential of statistical techniques such as multivariate survival models, generalized estimating equations and multilevel models are also explored.

CONCLUSIONS

Reviews of published dental research repeatedly identify statistical errors in the design, analysis and conclusions of the study. Educating researchers on common pitfalls and giving recommendations for avoiding them may help researchers to eliminate statistical errors. Developments in statistical methodology should be routinely monitored to ensure the most appropriate statistical methods are used in dental research.

To test or not to test: Preliminary assessment of normality when comparing two independent samples

Background

Student’s two-sample t test is generally used for comparing the means of two independent samples, for example, two treatment arms. Under the null hypothesis, the t test assumes that the two samples arise from the same normally distributed population with unknown variance. Adequate control of the Type I error requires that the normality assumption holds, which is often examined by means of a preliminary Shapiro-Wilk test. The following two-stage procedure is widely accepted: If the preliminary test for normality is not significant, the t test is used; if the preliminary test rejects the null hypothesis of normality, a nonparametric test is applied in the main analysis.

Methods

Equally sized samples were drawn from exponential, uniform, and normal distributions. The two-sample t test was conducted if either both samples (Strategy I) or the collapsed set of residuals from both samples (Strategy II) had passed the preliminary Shapiro-Wilk test for normality; otherwise, Mann-Whitney’s U test was conducted. By simulation, we separately estimated the conditional Type I error probabilities for the parametric and nonparametric part of the two-stage procedure. Finally, we assessed the overall Type I error rate and the power of the two-stage procedure as a whole.

Results

Preliminary testing for normality seriously altered the conditional Type I error rates of the subsequent main analysis for both parametric and nonparametric tests. We discuss possible explanations for the observed results, the most important one being the selection mechanism due to the preliminary test. Interestingly, the overall Type I error rate and power of the entire two-stage procedure remained within acceptable limits.

Conclusion

The two-stage procedure might be considered incorrect from a formal perspective; nevertheless, in the investigated examples, this procedure seemed to satisfactorily maintain the nominal significance level and had acceptable power properties.

Evidence-based practice in nephrology: critical appraisal of nephrology clinical research: were the correct statistical tests used?

Nephrologists rely on valid clinical studies to inform their health care decisions. Knowledge of simple statistical principles equips the prudent nephrologist with the skills that allow him or her to critically evaluate clinical studies and to determine the validity of the results. Important in this process is knowing when certain statistical tests are used appropriately and if their application in interpreting research data will most likely lead to the most robust or valid conclusions. The research team bears the responsibility for determining the statistical analysis during the design phase of the study and subsequently for carrying out the appropriate analysis. This will ensure that bias is minimized and "valid" results are reported. We have summarized the important caveats and components in correctly choosing a statistical test with a series of tables. With this format, we wish to provide a tool for the nephrologist/researcher that he or she can use when required to decide if an appropriate statistical analysis plan was implemented for any particular study. We have included in these tables the types of statistical tests that might be used best for analysis of different types of comparisons on small and on larger patient samples.