Poor Quality of Reporting Confounding Bias in Observational Intervention Studies: A Systematic Review

Poor statistical reporting: do we have a reason for concern? A narrative review and recommendations

PURPOSE OF REVIEW

The aim of the review conducted was to present recent articles indicating the need to implement statistical recommendations in the daily work of biomedical journals.

RECENT FINDINGS

The most recent literature shows an unchanged percentage of journals using specialized statistical review over 20 years. The problems of finding statistical reviewers, the impractical way in which biostatistics is taught and the nonimplementation of published statistical recommendations contribute to the fact that a small percentage of accepted manuscripts contain correctly performed analysis. The statistical recommendations published for authors and editorial board members in recent years contain important advice, but more emphasis should be placed on their practical and rigorous implementation. If this is not the case, we will additionally continue to experience low reproducibility of the research.

SUMMARY

There is a low level of statistical reporting these days. Recommendations related to the statistical review of submitted manuscripts should be followed more rigorously.

Genetically Predicted Vascular Proteins and Risk of Intracranial Aneurysms: A Mendelian Randomization Study

The relationship between vascular proteins (VPs) and intracranial aneurysms (IAs) has not been fully elucidated. We used Mendelian randomization (MR) analysis to explore the effect of VPs on IAs. Dataset of aneurysmal subarachnoid hemorrhage (aSAH) [5140 cases and 71,934 controls] and unruptured intracranial aneurysm (uIA) [2070 cases and 71,934 controls] were obtained from individuals of European ancestry. Univariate MR was used to explore the associations between 90 VPs and IAs. Then, we performed multivariate MR (MVMR) to further investigate the identified VP-to-IA estimates. Two-sample MR showed that TNFSF14 was inversely associated with aSAH (odds ratio [OR] = 0.831, 95% CI: 0.713-0.969, p = 0.018). IL-16 (OR = 1.218, 95% CI: 1.032-1.438, p = 0.020) and AgRP (OR = 1.394, 95% CI: 1.048-1.855, p = 0.023) were positively associated with aSAH. HBEGF (OR = 0.642, 95% CI: 0.461-0.894, p = 0.009), MCP-1 (OR = 1.537, 95% CI: 1.007-2.344, p = 0.046), and CX3CL1 (OR = 0.762, 95% CI: 0.581-0.999, 0.049 < p < 0.050) were associated with uIA risk. The MVMR showed that the TNFSF14-to-aSAH estimate remained statistically significant after adjustment for past tobacco smoking, alcohol consumption, systolic blood pressure and body mass index. Our study indicated that low serum TNFSF14 levels might be a potential risk factor for IA rupture. Five VPs (HBEGF, MCP-1, IL-6, CX3CL1, and AgRP) are associated with the risk of IAs (both uIA and aSAH).

Implementation and Evaluation of a Breast Cancer Disease Model Using Real-World Claims Data in Germany from 2010 to 2020

Breast cancer is the leading cause of cancer-related mortality among women in Germany and worldwide. This retrospective claims data analysis utilizing data from AOK Baden-Wuerttemberg, a major statutory German health insurance provider, aimed to construct and assess a real-world data breast cancer disease model. The study included 27,869 female breast cancer patients and 55,738 age-matched controls, analyzing data from 2010 to 2020. Three distinct breast cancer stages were analyzed: Stage A (early breast cancer without lymph node involvement), Stage B (early breast cancer with lymph node involvement), and Stage C (primary distant metastatic breast cancer). Tumor subtypes were estimated based on the prescription of antihormonal or HER2-targeted therapy. The study established that 77.9% of patients had HR+ breast cancer and 9.8% HER2+; HR+/HER2- was the most common subtype (70.9%). Overall survival (OS) analysis demonstrated significantly lower survival rates for stages B and C than for controls, with 5-year OS rates ranging from 79.3% for stage B to 35.4% for stage C. OS rates were further stratified by tumor subtype and stage, revealing varying prognoses. Distant recurrence-free survival (DRFS) analysis showed higher recurrence rates in stage B than in stage A, with HR-/HER2- displaying the worst DRFS. This study, the first to model breast cancer subtypes, stages, and outcomes using German claims data, provides valuable insights into real-world breast cancer epidemiology and demonstrates that this breast cancer disease model has the potential to be representative of treatment outcomes.

Educational strategies for oral assistance for the disabled and their effects on dentistry students' formation: A systematic review

AIMS

This systematic review aimed to analyze the influence of educational strategies for oral health care for Persons with disabilities (PwDs) on the technical, ethical, and social training of undergraduate dental students.

METHODS AND RESULTS

An extensive search was performed in the following databases: Cochrane, Embase, PubMed, Scopus, Web of Science, LILACS/BBO and OpenGrey database. Observational, nonrandomized, and quantitative comparisons before and after some type of educational strategy in undergraduate dentistry was included. The risk of bias was assessed using the risk of bias in nonrandomized intervention studies (ROBINS-I) tool. The certainty of evidence was assessed using the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) approach. Nine were included and suggested that a significant improvement was observed in the technical training of undergraduate dental students although there was no consensus on the acquisition of ethical/social competences after the application of educational strategies. The included studies presented a moderate to serious risk of bias, and the certainty of evidence for both, technical and ethical/social evaluative constructs, as very low.

CONCLUSION

Although studies show that awareness regarding PwD significantly increases technical skills in dental students, the evidence is poor due to the high risk of bias and inconsistency observed in both the technical and ethical/social constructs; further studies on this topic are therefore essential. Prospero: (CRD42021288463).

A comparison of full model specification and backward elimination of potential confounders when estimating marginal and conditional causal effects on binary outcomes from observational data

A common view in epidemiology is that automated confounder selection methods, such as backward elimination, should be avoided as they can lead to biased effect estimates and underestimation of their variance. Nevertheless, backward elimination remains regularly applied. We investigated if and under which conditions causal effect estimation in observational studies can improve by using backward elimination on a prespecified set of potential confounders. An expression was derived that quantifies how variable omission relates to bias and variance of effect estimators. Additionally, 3960 scenarios were defined and investigated by simulations comparing bias and mean squared error (MSE) of the conditional log odds ratio, log(cOR), and the marginal log risk ratio, log(mRR), between full models including all prespecified covariates and backward elimination of these covariates. Applying backward elimination resulted in a mean bias of 0.03 for log(cOR) and 0.02 for log(mRR), compared to 0.56 and 0.52 for log(cOR) and log(mRR), respectively, for a model without any covariate adjustment, and no bias for the full model. In less than 3% of the scenarios considered, the MSE of the log(cOR) or log(mRR) was slightly lower (max 3%) when backward elimination was used compared to the full model. When an initial set of potential confounders can be specified based on background knowledge, there is minimal added value of backward elimination. We advise not to use it and otherwise to provide ample arguments supporting its use.

Misspecification of confounder-exposure and confounder-outcome associations leads to bias in effect estimates

BACKGROUND

Confounding is a common issue in epidemiological research. Commonly used confounder-adjustment methods include multivariable regression analysis and propensity score methods. Although it is common practice to assess the linearity assumption for the exposure-outcome effect, most researchers do not assess linearity of the relationship between the confounder and the exposure and between the confounder and the outcome before adjusting for the confounder in the analysis. Failing to take the true non-linear functional form of the confounder-exposure and confounder-outcome associations into account may result in an under- or overestimation of the true exposure effect. Therefore, this paper aims to demonstrate the importance of assessing the linearity assumption for confounder-exposure and confounder-outcome associations and the importance of correctly specifying these associations when the linearity assumption is violated.

METHODS

A Monte Carlo simulation study was used to assess and compare the performance of confounder-adjustment methods when the functional form of the confounder-exposure and confounder-outcome associations were misspecified (i.e., linearity was wrongly assumed) and correctly specified (i.e., linearity was rightly assumed) under multiple sample sizes. An empirical data example was used to illustrate that the misspecification of confounder-exposure and confounder-outcome associations leads to bias.

RESULTS

The simulation study illustrated that the exposure effect estimate will be biased when for propensity score (PS) methods the confounder-exposure association is misspecified. For methods in which the outcome is regressed on the confounder or the PS, the exposure effect estimate will be biased if the confounder-outcome association is misspecified. In the empirical data example, correct specification of the confounder-exposure and confounder-outcome associations resulted in smaller exposure effect estimates.

CONCLUSION

When attempting to remove bias by adjusting for confounding, misspecification of the confounder-exposure and confounder-outcome associations might actually introduce bias. It is therefore important that researchers not only assess the linearity of the exposure-outcome effect, but also of the confounder-exposure or confounder-outcome associations depending on the confounder-adjustment method used.

Quality of observational studies of clinical interventions: a meta-epidemiological review

BACKGROUND

This meta-epidemiological study aimed to assess methodological quality of a sample of contemporary non-randomised clinical studies of clinical interventions.

METHODS

This was a cross-sectional study of observational studies published between January 1, 2012 and December 31, 2018. Studies were identified in PubMed using search terms 'association', 'observational,' 'non-randomised' 'comparative effectiveness' within titles or abstracts. Each study was appraised against 35 quality criteria by two authors independently, with each criterion rated fully, partially or not satisfied. These quality criteria were grouped into 6 categories: justification for observational design (n = 2); minimisation of bias in study design and data collection (n = 11); use of appropriate methods to create comparable groups (n = 6); appropriate adjustment of observed effects (n = 5); validation of observed effects (n = 9); and authors interpretations (n = 2).

RESULTS

Of 50 unique studies, 49 (98%) were published in two US general medical journals. No study fully satisfied all applicable criteria; the mean (+/-SD) proportion of applicable criteria fully satisfied across all studies was 72% (+/- 10%). The categories of quality criteria demonstrating the lowest proportions of fully satisfied criteria were measures used to adjust observed effects (criteria 20, 23, 24) and validate observed effects (criteria 25, 27, 33). Criteria associated with ≤50% of full satisfaction across studies, where applicable, comprised: imputation methods to account for missing data (50%); justification for not performing an RCT (42%); interaction analyses in identifying independent prognostic factors potentially influencing intervention effects (42%); use of statistical correction to minimise type 1 error in multiple outcome analyses (33%); clinically significant effect sizes (30%); residual bias analyses for unmeasured or unknown confounders (14%); and falsification tests for residual confounding (8%). The proportions of fully satisfied criteria did not change over time.

CONCLUSIONS

Recently published observational studies fail to fully satisfy more than one in four quality criteria. Criteria that were not or only partially satisfied were identified which serve as remediable targets for researchers and journal editors.

The urocortin peptides: biological relevance and laboratory aspects of UCN3 and its receptor

The urocortins are polypeptides belonging to the corticotropin-releasing hormone family, known to modulate stress responses in mammals. Stress, whether induced physically or psychologically, is an underlying cause or consequence of numerous clinical syndromes. Identifying biological markers associated with the homeostatic regulation of stress could provide a clinical laboratory approach for the management of stress-related disorders. The neuropeptide, urocortin 3 (UCN3), and the corticotropin-releasing hormone receptor 2 (CRHR2) constitute a regulatory axis known to mediate stress homeostasis. Dysregulation of this peptide/receptor axis is believed to play a role in several clinical conditions including post-traumatic stress, sleep apnea, cardiovascular disease, and other health problems related to stress. Understanding the physiology and measurement of the UCN3/CRHR2 axis is important for establishing a viable clinical laboratory diagnostic. In this article, we focus on evidence supporting the role of UCN3 and its receptor in stress-related clinical syndromes. We also provide insight into the measurements of UCN3 in blood and urine. These potential biomarkers provide new opportunities for clinical research and applications of laboratory medicine diagnostics in stress management.

Design and methodological characteristics of studies using observational routinely collected health data for investigating the link between cancer and neurodegenerative diseases: protocol for a meta-research study

INTRODUCTION

Health services generate large amounts of routine health data (eg, administrative databases, disease registries and electronic health records), which have important secondary uses for research. Increases in the availability and the ability to access and analyse large amounts of data represent a major opportunity for conducting studies on the possible relationships between complex diseases. The objective of this study will be to evaluate the design, methods and reporting of studies conducted using observational routinely collected health data for investigating the link between cancer and neurodegenerative diseases.

METHODS AND ANALYSIS

This is the protocol for a meta-research study. We registered the study protocol within the Open Science Framework: https://osf.io/h2qjg. We will evaluate observational studies (eg, cohort and case-control) conducted using routinely collected health data for investigating the associations between cancer and neurodegenerative diseases (such as Alzheimer's disease, amyotrophic lateral sclerosis/motor neuron disease, Huntington's disease, multiple sclerosis and Parkinson's disease). The following electronic databases will be searched (from their inception onwards): MEDLINE, Embase and Web of Science Core Collection. Screening and selection of articles will be conducted by at least two researchers. Potential discrepancies will be resolved via discussion. Design, methods and reporting characteristics in each article will be extracted using a standardised data extraction form. Information on general, methodological and transparency items will be reported. We will summarise our findings with tables and graphs (eg, bar charts, forest plots).

ETHICS AND DISSEMINATION

Due to the nature of the proposed study, no ethical approval will be required. We plan to publish the full study in an open access peer-reviewed journal and disseminate the findings at scientific conferences and via social media. All data will be deposited in a cross-disciplinary public repository.

The confounder matrix: A tool to assess confounding bias in systematic reviews of observational studies of etiology

Systematic reviews and meta-analyses are essential for drawing conclusions regarding etiologic associations between exposures or interventions and health outcomes. Observational studies comprise a substantive source of the evidence base. One major threat to their validity is residual confounding, which may occur when component studies adjust for different sets of confounders, fail to control for important confounders, or have classification errors resulting in only partial control of measured confounders. We present the confounder matrix-an approach for defining and summarizing adequate confounding control in systematic reviews of observational studies and incorporating this assessment into meta-analyses. First, an expert group reaches consensus regarding the core confounders that should be controlled and the best available method for their measurement. Second, a matrix graphically depicts how each component study accounted for each confounder. Third, the assessment of control adequacy informs quantitative synthesis. We illustrate the approach with studies of the association between short interpregnancy intervals and preterm birth. Our findings suggest that uncontrolled confounding, notably by reproductive history and sociodemographics, resulted in exaggerated estimates. Moreover, no studies adequately controlled for all core confounders, so we suspect residual confounding is present, even among studies with better control. The confounder matrix serves as an extension of previously published methodological guidance for observational research synthesis, enabling transparent reporting of confounding control and directly informing meta-analysis so that conclusions are drawn from the best available evidence. Widespread application could raise awareness about gaps across a body of work and allow for more valid inference with respect to confounder control.

High prevalence of spin was found in pharmacovigilance studies using disproportionality analyses to detect safety signals: a meta-epidemiological study

OBJECTIVE

To systematically review and appraise misinterpretation of pharmacovigilance disproportionality analysis results in published studies.

STUDY DESIGN AND SETTING

We randomly selected 100 studies that performed disproportionality analyses and indexed in Medline identified during a systematic literature search. Titles, abstracts and main texts (results, discussion and conclusion) were evaluated for spin independently by two reviewers. Spin in pharmacovigilance studies was classified according to three main categories: inappropriate interpretation, inappropriate extrapolations and misleading reporting.

RESULTS

Of the 100 studies evaluated, we found that 63%, 56% and 51% had at least one type of spin in their abstract, main text or conclusion respectively, and 40% used causal language to interpret their results in the abstract or conclusion. Spin in titles and results were exclusively represented by inappropriate interpretations of findings (12% and 21% respectively), with terms such as "risk of" or "risks associated with" or results erroneously presented as regular Odds Ratios. Spin in discussion sections mostly concerned inappropriate interpretations (38%)and misleading reporting (12%). Misleading reporting, notably failing to acknowledge the limitations of disproportionality analyses, was the most frequent type of spin in abstracts (55%) and conclusion sections (37%).

CONCLUSION

We found that spin is frequent in publications of pharmacovigilance disproportionality analyses, notably in abstracts. This consisted notably in an over-interpretation of the results suggesting a proven causative link between a drug use and the risk of an event.

'Real-world' observational studies in arrhythmia research: data sources, methodology, and interpretation. A position document from European Heart Rhythm Association (EHRA), endorsed by Heart Rhythm Society (HRS), Asia-Pacific HRS (APHRS), and Latin America HRS (LAHRS)

The field of observational studies or "real world studies" is in rapid development with many new techniques introduced and increased understanding of traditional methods. For this reason the current paper provides an overview of current methods with focus on new techniques. Some highlights can be emphasized: We provide an overview of sources of data for observational studies. There is an overview of sources of bias and confounding. Next There is an overview of causal inference techniques that are increasingly used. The most commonly used techniques for statistical modelling are reviewed with focus on the important distinction of risk versus prediction. The final section provides examples of common problems with reporting observational data.

Consideration of confounding in epidemiologic studies assessing alcohol consumption on the risk of breast cancer: A brief report

BACKGROUND

Individual observational studies have suggested null, weak, linear, and J-shaped associations between alcohol consumption and breast cancer risk. However, observational studies are susceptible to confounders, which can obscure the true impact of an exposure on an outcome. Given the uncertainty regarding the association between alcohol consumption and breast cancer, and the challenges of identifying, measuring, and accounting for all potential confounders, we assessed whether and how authors of observational studies evaluating the impact of alcohol consumption on the risk of breast cancer considered bias when interpreting their main study findings.

METHODS

We identified all observational studies included in a recent alcohol-breast cancer meta-analysis. The Abstract and/or Discussion sections were reviewed to determine whether authors considered confounding.

RESULTS

Among 101 eligible studies, 73 (72.3%) mentioned confounding explicitly in the Abstract and Discussion sections. There were 33 (32.7%) studies that included statements regarding specific confounders that were not adjusted for in the analyses and 60 (59.4%) studies without any statements about the impact of residual confounding on their main findings. Although none of the studies outlined that their main findings were "likely" to be affected by residual confounding, 25 (24.8%) mentioned a "possible" impact and 16 (15.8%) claimed an "unlikely" impact. Only one (1.0%) article explicitly stated that caution was needed when interpreting their findings due to confounding.

CONCLUSION

These results highlight the need for more adequate consideration of the potential impact of residual confounding in observational studies evaluating the impact of alcohol consumption on the risk of breast cancer.

Evaluation of confounding in epidemiologic studies assessing alcohol consumption on the risk of ischemic heart disease

Background

Among different investigators studying the same exposures and outcomes, there may be a lack of consensus about potential confounders that should be considered as matching, adjustment, or stratification variables in observational studies. Concerns have been raised that confounding factors may affect the results obtained for the alcohol-ischemic heart disease relationship, as well as their consistency and reproducibility across different studies. Therefore, we assessed how confounders are defined, operationalized, and discussed across individual studies evaluating the impact of alcohol on ischemic heart disease risk.

Methods

For observational studies included in a recent alcohol-ischemic heart disease meta-analysis, we identified all variables adjusted, matched, or stratified for in the largest reported multivariate model (i.e. potential confounders). We recorded how the variables were measured and grouped them into higher-level confounder domains. Abstracts and Discussion sections were then assessed to determine whether authors considered confounding when interpreting their study findings.

Results

85 of 87 (97.7%) studies reported multivariate analyses for an alcohol-ischemic heart disease relationship. The most common higher-level confounder domains included were smoking (79, 92.9%), age (74, 87.1%), and BMI, height, and/or weight (57, 67.1%). However, no two models adjusted, matched, or stratified for the same higher-level confounder domains. Most (74/87, 85.1%) articles mentioned or alluded to “confounding” in their Abstract or Discussion sections, but only one stated that their main findings were likely to be affected by residual confounding. There were five (5/87, 5.7%) authors that explicitly asked for caution when interpreting results.

Conclusion

There is large variation in the confounders considered across observational studies evaluating the impact of alcohol on ischemic heart disease risk and almost all studies spuriously ignore or eventually dismiss confounding in their conclusions. Given that study results and interpretations may be affected by the mix of potential confounders included within multivariate models, efforts are necessary to standardize approaches for selecting and accounting for confounders in observational studies.

Methods for the Selection of Covariates in Nutritional Epidemiology Studies: A Meta-Epidemiological Review

BACKGROUND

Observational studies provide important information about the effects of exposures that cannot be easily studied in clinical trials, such as nutritional exposures, but are subject to confounding. Investigators adjust for confounders by entering them as covariates in analytic models.

OBJECTIVE

The aim of this study was to evaluate the reporting and credibility of methods for selection of covariates in nutritional epidemiology studies.

METHODS

We sampled 150 nutritional epidemiology studies published in 2007/2008 and 2017/2018 from the top 5 high-impact nutrition and medical journals and extracted information on methods for selection of covariates.

RESULTS

Most studies did not report selecting covariates a priori (94.0%) or criteria for selection of covariates (63.3%). There was general inconsistency in choice of covariates, even among studies investigating similar questions. One-third of studies did not acknowledge potential for residual confounding in their discussion.

CONCLUSION

Studies often do not report methods for selection of covariates, follow available guidance for selection of covariates, nor discuss potential for residual confounding.

Reporting quality in systematic reviews of in vitro studies: a systematic review

Background: Systematic reviews (SRs) and/or meta-analyses of in vitro research have an important role in establishing the foundation for clinical studies. In this study, we aimed to evaluate the reporting quality of SRs of in vitro studies using the PRISMA checklist.Method: Four databases were searched including PubMed, Virtual Health Library (VHL), Web of Science (ISI) and Scopus. The search was limited from 2006 to 2016 to include all SRs and/or meta-analyses (MAs) of pure in vitro studies. The evaluation of reporting quality was done using the PRISMA checklist.Results: Out of 7702 search results, 65 SRs were included and evaluated with the PRISMA checklist. Overall, the mean overall quality score of reported items of the PRISMA checklist was 68%. We have noticed an increasing pattern in the numbers of published SRs of in vitro studies over the last 10 years. In contrast, the reporting quality was not significantly improved over the same period (p = .363). There was a positive but not significant correlation between the overall quality score and the journal impact factor of the included studies.Conclusions: The adherence of SRs of in vitro studies to the PRISMA guidelines was poor. Therefore, we believe that using reporting guidelines and journals paying attention to this fact will improve the quality of SRs of in vitro studies.

Simple decision-tree tool to facilitate author identification of reporting guidelines during submission: a before-after study

Background

There is evidence that direct journal endorsement of reporting guidelines can lead to important improvements in the quality and reliability of the published research. However, over the last 20 years, there has been a proliferation of reporting guidelines for different study designs, making it impractical for a journal to explicitly endorse them all. The objective of this study was to investigate whether a decision tree tool made available during the submission process facilitates author identification of the relevant reporting guideline.

Methods

This was a prospective 14-week before–after study across four speciality medical research journals. During the submission process, authors were prompted to follow the relevant reporting guideline from the EQUATOR Network and asked to confirm that they followed the guideline (‘before’). After 7 weeks, this prompt was updated to include a direct link to the decision-tree tool and an additional prompt for those authors who stated that ‘no guidelines were applicable’ (‘after’). For each article submitted, the authors’ response, what guideline they followed (if any) and what reporting guideline they should have followed (including none relevant) were recorded.

Results

Overall, 590 manuscripts were included in this analysis—300 in the before cohort and 290 in the after. There were relevant reporting guidelines for 75% of manuscripts in each group; STROBE was the most commonly applicable reporting guideline, relevant for 35% (n = 106) and 37% (n = 106) of manuscripts, respectively. Use of the tool was associated with an 8.4% improvement in the number of authors correctly identifying the relevant reporting guideline for their study (p < 0.0001), a 14% reduction in the number of authors incorrectly stating that there were no relevant reporting guidelines (p < 0.0001), and a 1.7% reduction in authors choosing a guideline (p = 0.10). However, the ‘after’ cohort also saw a significant increase in the number of authors stating that there were relevant reporting guidelines for their study, but not specifying which (34 vs 29%; p = 0.04).

Conclusion

This study suggests that use of a decision-tree tool during submission of a manuscript is associated with improved author identification of the relevant reporting guidelines for their study type; however, the majority of authors still failed to correctly identify the relevant guidelines.

Electronic supplementary material

The online version of this article (10.1186/s41073-017-0044-9) contains supplementary material, which is available to authorized users.

Interpretation of epidemiologic studies very often lacked adequate consideration of confounding

BACKGROUND AND OBJECTIVE

Confounding bias is a most pervasive threat to validity of observational epidemiologic research. We assessed whether authors of observational epidemiologic studies consider confounding bias when interpreting the findings.

STUDY DESIGN AND SETTING

We randomly selected 120 cohort or case-control studies published in 2011 and 2012 by the general medical, epidemiologic, and specialty journals with the highest impact factors. We used Web of Science to assess citation metrics through January 2017.

RESULTS

Sixty-eight studies (56.7%, 95% confidence interval: 47.8-65.5%) mentioned "confounding" in the Abstract or Discussion sections, another 20 (16.7%; 10.0-23.3%) alluded to it, and there was no mention or allusion at all in 32 studies (26.7%; 18.8-34.6%). Authors often acknowledged that for specific confounders, there was no adjustment (34 studies; 28.3%) or deem it possible or likely that confounding affected their main findings (29 studies; 24.2%). However, only two studies (1.7%; 0-4.0%) specifically used the words "caution" or "cautious" for the interpretation because of confounding-related reasons and eventually only four studies (3.3%; 0.1-6.5%) had limitations related to confounding or any other bias in their Conclusions. Studies mentioning that the findings were possibly or likely affected by confounding were more frequently cited than studies with a statement that findings were unlikely affected (median 6.3 vs. 4.0 citations per year, P = 0.04).

CONCLUSIONS

Many observational studies lack satisfactory discussion of confounding bias. Even when confounding bias is mentioned, authors are typically confident that it is rather irrelevant to their findings and they rarely call for cautious interpretation. More careful acknowledgment of possible impact of confounding is not associated with lower citation impact.

Methods and processes of developing the strengthening the reporting of observational studies in epidemiology - veterinary (STROBE-Vet) statement

BACKGROUND

The reporting of observational studies in veterinary research presents many challenges that often are not adequately addressed in published reporting guidelines.

OBJECTIVE

To develop an extension of the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) statement that addresses unique reporting requirements for observational studies in veterinary medicine related to health, production, welfare, and food safety.

DESIGN

A consensus meeting of experts was organized to develop an extension of the STROBE statement to address observational studies in veterinary medicine with respect to animal health, animal production, animal welfare, and food safety outcomes.

SETTING

Consensus meeting May 11-13, 2014 in Mississauga, Ontario, Canada.

PARTICIPANTS

Seventeen experts from North America, Europe, and Australia attended the meeting. The experts were epidemiologists and biostatisticians, many of whom hold or have held editorial positions with relevant journals.

METHODS

Prior to the meeting, 19 experts completed a survey about whether they felt any of the 22 items of the STROBE statement should be modified and if items should be added to address unique issues related to observational studies in animal species with health, production, welfare, or food safety outcomes. At the meeting, the participants were provided with the survey responses and relevant literature concerning the reporting of veterinary observational studies. During the meeting, each STROBE item was discussed to determine whether or not re-wording was recommended, and whether additions were warranted. Anonymous voting was used to determine whether there was consensus for each item change or addition.

RESULTS

The consensus was that six items needed no modifications or additions. Modifications or additions were made to the STROBE items numbered: 1 (title and abstract), 3 (objectives), 5 (setting), 6 (participants), 7 (variables), 8 (data sources/measurement), 9 (bias), 10 (study size), 12 (statistical methods), 13 (participants), 14 (descriptive data), 15 (outcome data), 16 (main results), 17 (other analyses), 19 (limitations), and 22 (funding).

LIMITATION

Published literature was not always available to support modification to, or inclusion of, an item.

CONCLUSION

The methods and processes used in the development of this statement were similar to those used for other extensions of the STROBE statement. The use of this extension to the STROBE statement should improve the reporting of observational studies in veterinary research related to animal health, production, welfare, or food safety outcomes by recognizing the unique features of observational studies involving food-producing and companion animals, products of animal origin, aquaculture, and wildlife.

Methods and Processes of Developing the Strengthening the Reporting of Observational Studies in Epidemiology-Veterinary (STROBE-Vet) Statement

Reporting of observational studies in veterinary research presents challenges that often are not addressed in published reporting guidelines. Our objective was to develop an extension of the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) statement that addresses unique reporting requirements for observational studies in veterinary medicine related to health, production, welfare, and food safety. We conducted a consensus meeting with 17 experts in Mississauga, Canada. Experts completed a premeeting survey about whether items in the STROBE statement should be modified or added to address unique issues related to observational studies in animal species with health, production, welfare, or food safety outcomes. During the meeting, each STROBE item was discussed to determine whether or not rewording was recommended, and whether additions were warranted. Anonymous voting was used to determine consensus. Six items required no modifications or additions. Modifications or additions were made to the STROBE items 1 (title and abstract), 3 (objectives), 5 (setting), 6 (participants), 7 (variables), 8 (data sources and measurement), 9 (bias), 10 (study size), 12 (statistical methods), 13 (participants), 14 (descriptive data), 15 (outcome data), 16 (main results), 17 (other analyses), 19 (limitations), and 22 (funding). The methods and processes used were similar to those used for other extensions of the STROBE statement. The use of this STROBE statement extension should improve reporting of observational studies in veterinary research by recognizing unique features of observational studies involving food-producing and companion animals, products of animal origin, aquaculture, and wildlife.

Methods and Processes of Developing the Strengthening the Reporting of Observational Studies in Epidemiology - Veterinary (STROBE-Vet) Statement

Background

Reporting of observational studies in veterinary research presents challenges that often are not addressed in published reporting guidelines.

Objective

To develop an extension of the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) statement that addresses unique reporting requirements for observational studies in veterinary medicine related to health, production, welfare, and food safety.

Design

Consensus meeting of experts.

Setting

Mississauga, Canada.

Participants

Seventeen experts from North America, Europe, and Australia.

Methods

Experts completed a pre‐meeting survey about whether items in the STROBE statement should be modified or added to address unique issues related to observational studies in animal species with health, production, welfare, or food safety outcomes. During the meeting, each STROBE item was discussed to determine whether or not rewording was recommended and whether additions were warranted. Anonymous voting was used to determine consensus.

Results

Six items required no modifications or additions. Modifications or additions were made to the STROBE items 1 (title and abstract), 3 (objectives), 5 (setting), 6 (participants), 7 (variables), 8 (data sources/measurement), 9 (bias), 10 (study size), 12 (statistical methods), 13 (participants), 14 (descriptive data), 15 (outcome data), 16 (main results), 17 (other analyses), 19 (limitations), and 22 (funding).

Conclusion

The methods and processes used were similar to those used for other extensions of the STROBE statement. The use of this STROBE statement extension should improve reporting of observational studies in veterinary research by recognizing unique features of observational studies involving food‐producing and companion animals, products of animal origin, aquaculture, and wildlife.

The reporting of studies using routinely collected health data was often insufficient

OBJECTIVES

To assess reporting quality of studies using routinely collected health data (RCD) to inform the REporting of studies Conducted using Observational Routinely collected health Data (RECORD) guideline development.

STUDY DESIGN AND SETTING

PubMed search for observational studies using RCD on any epidemiologic or clinical topic. Sample of studies published in 2012. Evaluation of five items based on the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) guideline and eight newly developed items for RCD studies.

RESULTS

Of 124 included studies, 39 (31.5%) clearly described its design in title or abstract. Complete information to frame a focused research question, that is, on the population, intervention/exposure, and outcome, was provided for 51 studies (41.1%). In 44 studies where definitions of codes or classification algorithms would be necessary to operationalize such a research question, only nine (20.5%) reported all items adequately. In 81 studies describing multivariable analyses, 54 (66.7%) reported all variables used for modeling and 34 (42.0%) reported basic details required for replication. Database linkage was reported adequately in 12 of 41 studies (29.3%). Statements about data sharing/availability were rare (5/124; 4%).

CONCLUSION

Most RCD studies are insufficiently reported. Specific reporting guidelines and more awareness and education on their use are urgently needed.

Analysis of recruitment in a pragmatic observational study on C-reactive protein point-of-care testing in primary care

INTRODUCTION

Failure to recruit all eligible study patients can lead to biased results. Little is known on selective patient recruitment in studies on implementation of diagnostic devices.

OBJECTIVES

The aim of this observational study was to measure recruitment of patients in an implementation study in primary care on use of point-of-care (POC) C-reactive protein (CRP) and to evaluate recruitment bias and its impact on the study endpoint.

METHODS

In a cross-sectional observational study on POC CRP implementation and related antibiotics prescribing, we compared included patients with all eligible patients to assess the representativeness of the included subjects. Eligible patients were adults presenting with acute cough in primary care between March and September 2012. The frequency of POC CRP testing and the proportion of prescribed antibiotics were compared between recruited and non-recruited patients. As measure of bias, odds ratios (ORs) with accompanying 95% confidence intervals (CIs) for the association between CRP level (<20 mg/l or not) and antibiotic prescribing were computed.

RESULTS

Of all 1473 eligible patients 348 (24%) were recruited. In recruited patients, POC CRP tests were conducted and antibiotics prescribed more frequently as compared to non-recruited patients (81% versus 6% and 44% versus 29%, respectively). The ORs were 18.2 (95%CI: 9.6-34.3), 30.5 (95%CI: 13.2-70.3) and 3.8 (95%CI: 0.9-14.8) respectively in all eligible patients, the recruited and the non-recruited patients.

CONCLUSION

Selective recruitment resulted in an overestimation of POC CRP test use and antibiotic prescribing.

Quality Reporting of Multivariable Regression Models in Observational Studies: Review of a Representative Sample of Articles Published in Biomedical Journals

Controlling for confounders is a crucial step in analytical observational studies, and multivariable models are widely used as statistical adjustment techniques. However, the validation of the assumptions of the multivariable regression models (MRMs) should be made clear in scientific reporting. The objective of this study is to review the quality of statistical reporting of the most commonly used MRMs (logistic, linear, and Cox regression) that were applied in analytical observational studies published between 2003 and 2014 by journals indexed in MEDLINE.Review of a representative sample of articles indexed in MEDLINE (n = 428) with observational design and use of MRMs (logistic, linear, and Cox regression). We assessed the quality of reporting about: model assumptions and goodness-of-fit, interactions, sensitivity analysis, crude and adjusted effect estimate, and specification of more than 1 adjusted model.The tests of underlying assumptions or goodness-of-fit of the MRMs used were described in 26.2% (95% CI: 22.0-30.3) of the articles and 18.5% (95% CI: 14.8-22.1) reported the interaction analysis. Reporting of all items assessed was higher in articles published in journals with a higher impact factor.A low percentage of articles indexed in MEDLINE that used multivariable techniques provided information demonstrating rigorous application of the model selected as an adjustment method. Given the importance of these methods to the final results and conclusions of observational studies, greater rigor is required in reporting the use of MRMs in the scientific literature.

Epidemiology, quality and reporting characteristics of meta-analyses of observational studies published in Chinese journals

OBJECTIVE

The aim of this study was to examine the epidemiological and reporting characteristics as well as the methodological quality of meta-analyses (MAs) of observational studies published in Chinese journals.

METHODS

5 Chinese databases were searched for MAs of observational studies published from January 1978 to May 2014. Data were extracted into Excel spreadsheets, and Meta-analysis of Observational Studies in Epidemiology (MOOSE) and Assessment of Multiple Systematic Reviews (AMSTAR) checklists were used to assess reporting characteristics and methodological quality, respectively.

RESULTS

A total of 607 MAs were included. Only 52.2% of the MAs assessed the quality of the included primary studies, and the retrieval information was not comprehensive in more than half (85.8%) of the MAs. In addition, 50 (8.2%) MAs did not search any Chinese databases, while 126 (20.8%) studies did not search any English databases. Approximately 41.2% of the MAs did not describe the statistical methods in sufficient details, and most (95.5%) MAs did not report on conflicts of interest. However, compared with the before publication of the MOOSE Checklist, the quality of reporting improved significantly for 20 subitems after publication of the MOOSE Checklist, and 7 items of the included MAs demonstrated significant improvement after publication of the AMSTAR Checklist (p<0.05).

CONCLUSIONS

Although many MAs of observational studies have been published in Chinese journals, the reporting quality is questionable. Thus, there is an urgent need to increase the use of reporting guidelines and methodological tools in China; we recommend that Chinese journals adopt the MOOSE and AMSTAR criteria.

Poor quality in the reporting and use of statistical methods in public health - the case of unemployment and health

BACKGROUND

It has previously been reported that many research articles fail to fulfill important criteria for statistical analyses, but, to date, these reports have not focused on public health problems. The aim of this study was to investigate the quality of reporting and use of statistical methods in articles analyzing the effect of unemployment on health.

METHODS

Forty-one articles were identified and evaluated in terms of how they addressed 12 specified criteria.

RESULTS

For most of these criteria, the majority of articles were inadequate. These criteria were conformity with a linear gradient (100 % of the articles), validation of the statistical model (100 %), collinearity of independent variables (97 %), fitting procedure (93 %), goodness of fit test (78 %), selection of variables (68 % for the candidate model; 88 % for the final model), and interactions between independent variables (66 %). Fewer, but still alarmingly many articles, failed to fulfill the criteria coefficients presented in statistical models (48 %), coding of variables (34 %) and discussion of methodological concerns (24 %). There was a lack of explicit reporting of statistical significance/confidence intervals; 34 % of the articles only presented p-values as being above or below the significance level, and 42 % did not present confidence intervals. Events per variable was the only criterion met at an undoubtedly acceptable level (2.5 %).

CONCLUSIONS

There were critical methodological shortcomings in the reviewed studies. It is difficult to obtain unbiased estimates, but there clearly needs to be some improvement in the quality of documentation on the use and performance of statistical methods. A suggestion here is that journals not only demand that articles fulfill the criteria within the STROBE statement, but that they include additional criteria to decrease the risk of incorrect conclusions being drawn.

The quality of control groups in nonrandomized studies published in the Journal of Hand Surgery

PURPOSE

To evaluate control group selection in nonrandomized studies published in the Journal of Hand Surgery American (JHS).

METHODS

We reviewed all papers published in JHS in 2013 to identify studies that used nonrandomized control groups. Data collected included type of study design and control group characteristics. We then appraised studies to determine whether authors discussed confounding and selection bias and how they controlled for confounding.

RESULTS

Thirty-seven nonrandomized studies were published in JHS in 2013. The source of control was either the same institution as the study group, a different institution, a database, or not provided in the manuscript. Twenty-nine (78%) studies statistically compared key characteristics between control and study group. Confounding was controlled with matching, exclusion criteria, or regression analysis. Twenty-two (59%) papers explicitly discussed the threat of confounding and 18 (49%) identified sources of selection bias.

CONCLUSIONS

In our review of nonrandomized studies published in JHS, papers had well-defined controls that were similar to the study group, allowing for reasonable comparisons. However, we identified substantial confounding and bias that were not addressed as explicit limitations, which might lead the reader to overestimate the scientific validity of the data.

CLINICAL RELEVANCE

Incorporating a brief discussion of control group selection in scientific manuscripts should help readers interpret the study more appropriately. Authors, reviewers, and editors should strive to address this component of clinical importance.

Reducing waste from incomplete or unusable reports of biomedical research

Research publication can both communicate and miscommunicate. Unless research is adequately reported, the time and resources invested in the conduct of research is wasted. Reporting guidelines such as CONSORT, STARD, PRISMA, and ARRIVE aim to improve the quality of research reports, but all are much less adopted and adhered to than they should be. Adequate reports of research should clearly describe which questions were addressed and why, what was done, what was shown, and what the findings mean. However, substantial failures occur in each of these elements. For example, studies of published trial reports showed that the poor description of interventions meant that 40-89% were non-replicable; comparisons of protocols with publications showed that most studies had at least one primary outcome changed, introduced, or omitted; and investigators of new trials rarely set their findings in the context of a systematic review, and cited a very small and biased selection of previous relevant trials. Although best documented in reports of controlled trials, inadequate reporting occurs in all types of studies-animal and other preclinical studies, diagnostic studies, epidemiological studies, clinical prediction research, surveys, and qualitative studies. In this report, and in the Series more generally, we point to a waste at all stages in medical research. Although a more nuanced understanding of the complex systems involved in the conduct, writing, and publication of research is desirable, some immediate action can be taken to improve the reporting of research. Evidence for some recommendations is clear: change the current system of research rewards and regulations to encourage better and more complete reporting, and fund the development and maintenance of infrastructure to support better reporting, linkage, and archiving of all elements of research. However, the high amount of waste also warrants future investment in the monitoring of and research into reporting of research, and active implementation of the findings to ensure that research reports better address the needs of the range of research users.

Evaluating methodological assumptions in comparative effectiveness research: overcoming pitfalls

The scope of comparative effectiveness research (CER) is wide and therefore requires the application of complex statistical tools and nonstandard procedures. The commonly used methods presuppose the realization of important, and often untestable, assumptions pertaining to the underlying distribution, study heterogeneity and targeted population. Accordingly, the value of the results obtained based on such tools is in large part dependent on the validity of the underlying assumptions relating to the operating characteristics of the procedures. In this article, we elucidate some of the pitfalls that may arise with use of the most commonly used techniques, including those that are applied in network meta-analysis, observational data analysis and patient-reported outcome evaluation. In addition, reference is made to the impact of data quality and database heterogeneity on the performance of commonly used CER tools and the need for standards in order to inform researchers engaged in CER.

Issues of reporting in observational studies in veterinary medicine

Observational studies are common in veterinary medicine; the results may be used to inform decision-making, future research, or as inputs to systematic reviews or risk assessment. To be of use, the results must be published, all of the outcomes that were assessed must be included in the publication, and the research (methods and results) must be reported in sufficient detail that the reader can evaluate the internal and external validity. In human healthcare, concerns about the completeness of reporting - and evidence that poor reporting is associated with study results - have led to the creation of reporting guidelines; these include the STROBE statement for observational studies. There is evidence from a limited body of research that there also are reporting inadequacies in veterinary observational studies. There are differences between human and veterinary observational studies that might be relevant to recommendations for reporting. Such differences include: the use of observational studies in animal populations for simultaneously estimating disease frequency and risk-factor identification; the distinction between the animal owners who consent to participate and the animals that are the study subjects; and the complexity of organizational levels inherent in animal research (in particular, for studies in livestock species). In veterinary medicine, it is common to have clustering within outcomes (due to animal grouping) and clustering of predictor variables. We argue that there is a compelling need for the scientific community involved in veterinary observational studies to use the STROBE statement, use an amended version of STROBE, or to develop and use reporting guidelines that are specific to veterinary medicine to improve reporting of these studies.

The teacher, the physician and the person: exploring causal connections between teaching performance and role model types using directed acyclic graphs

Background

In fledgling areas of research, evidence supporting causal assumptions is often scarce due to the small number of empirical studies conducted. In many studies it remains unclear what impact explicit and implicit causal assumptions have on the research findings; only the primary assumptions of the researchers are often presented. This is particularly true for research on the effect of faculty’s teaching performance on their role modeling. Therefore, there is a need for robust frameworks and methods for transparent formal presentation of the underlying causal assumptions used in assessing the causal effects of teaching performance on role modeling. This study explores the effects of different (plausible) causal assumptions on research outcomes.

Methods

This study revisits a previously published study about the influence of faculty’s teaching performance on their role modeling (as teacher-supervisor, physician and person). We drew eight directed acyclic graphs (DAGs) to visually represent different plausible causal relationships between the variables under study. These DAGs were subsequently translated into corresponding statistical models, and regression analyses were performed to estimate the associations between teaching performance and role modeling.

Results

The different causal models were compatible with major differences in the magnitude of the relationship between faculty’s teaching performance and their role modeling. Odds ratios for the associations between teaching performance and the three role model types ranged from 31.1 to 73.6 for the teacher-supervisor role, from 3.7 to 15.5 for the physician role, and from 2.8 to 13.8 for the person role.

Conclusions

Different sets of assumptions about causal relationships in role modeling research can be visually depicted using DAGs, which are then used to guide both statistical analysis and interpretation of results. Since study conclusions can be sensitive to different causal assumptions, results should be interpreted in the light of causal assumptions made in each study.

Is weight-bearing asymmetry associated with postural instability after stroke? A systematic review

Introduction. Improvement of postural stability is an important goal during poststroke rehabilitation. Since weight-bearing asymmetry (WBA) towards the nonparetic leg is common, training of weight-bearing symmetry has been a major focus in post-stroke balance rehabilitation. It is assumed that restoration of a more symmetrical weight distribution is associated with improved postural stability. Objective. To determine to what extent WBA is associated with postural instability in people after stroke. Methods. Electronic databases were searched (Cochrane, MEDLINE, EMBASE, and CINAHL) until March 2012. Main Eligibility Criteria. (1) Participants were people after stroke. (2) The association between WBA and postural stability was reported. Quality of reporting was assessed with the STROBE checklist and a related tool for reporting of confounding. Results. Nine observational studies met all criteria. Greater spontaneous WBA was associated with higher center of pressure (COP) velocity and with poorer synchronization of COP trajectories between the legs (two and one studies, resp.). Evidence for associations between WBA and performance on clinical balance tests or falls was weak. Conclusion. Greater WBA after stroke was associated with increased postural sway, but the current literature does not provide evidence for a causal relationship. Further studies should investigate whether reducing WBA would improve postural stability.

Assessment of effectiveness of percutaneous adhesiolysis in managing chronic low back pain secondary to lumbar central spinal canal stenosis

BACKGROUND

Chronic persistent low back and lower extremity pain secondary to central spinal stenosis is common and disabling. Lumbar surgical interventions with decompression or fusion are most commonly performed to manage severe spinal stenosis. However, epidural injections are also frequently performed in managing central spinal stenosis. After failure of epidural steroid injections, the next sequential step is percutaneous adhesiolysis and hypertonic saline neurolysis with a targeted delivery. The literature on the effectiveness of percutaneous adhesiolysis in managing central spinal stenosis after failure of epidural injections has not been widely studied.

STUDY DESIGN

A prospective evaluation.

SETTING

An interventional pain management practice, a specialty referral center, a private practice setting in the United States.

OBJECTIVE

To evaluate the effectiveness of percutaneous epidural adhesiolysis in patients with chronic low back and lower extremity pain with lumbar central spinal stenosis.

METHODS

Seventy patients were recruited. The initial phase of the study was randomized, double-blind with a comparison of percutaneous adhesiolysis with caudal epidural injections. The 25 patients from the adhesiolysis group continued with follow-up, along with 45 additional patients, leading to a total of 70 patients. All patients received percutaneous adhesiolysis and appropriate placement of the Racz catheter, followed by an injection of 5 mL of 2% preservative-free lidocaine with subsequent monitoring in the recovery room. In the recovery room, each patient also received 6 mL of 10% hypertonic sodium chloride solution, and 6 mg of non-particulate betamethasone, followed by an injection of 1 mL of sodium chloride solution and removal of the catheter.

OUTCOMES ASSESSMENT

Multiple outcome measures were utilized including the Numeric Rating Scale (NRS), the Oswestry Disability Index 2.0 (ODI), employment status, and opioid intake with assessment at 3, 6, and 12, 18 and 24 months post treatment. The primary outcome measure was 50% or more improvement in pain scores and ODI scores.

RESULTS

Overall, a primary outcome or significant pain relief and functional status improvement of 50% or more was seen in 71% of patients at the end of 2 years. The overall number of procedures over a period of 2 years were 5.7 ± 2.73.

LIMITATIONS

The lack of a control group and a prospective design.

CONCLUSIONS

Significant relief and functional status improvement as seen in 71% of the 70 patients with percutaneous adhesiolysis utilizing local anesthetic steroids and hypertonic sodium chloride solution may be an effective management strategy in patients with chronic function limiting low back and lower extremity pain with central spinal stenosis after failure of conservatie management and fluoroscopically directed epidural injections.

Confounding and effect modification in studies of diet and childhood asthma and allergies

OBJECTIVE

To propose a comprehensive set of confounders and effect modifiers that should be considered in epidemiologic investigations.

METHODS

Two reviewers independently critiqued studies included in a recent systematic review and extracted data on the confounders and effect modifiers that were considered and the approaches used to justify inclusion.

RESULTS

Of the 62 studies reviewed, 20 were cohort, 16 case-control, 25 cross-sectional studies, and one ecologic study. All cohort, cross-sectional, and ecologic studies had some adjustment for confounding or consideration of effect modification, but this was only the case for 7/16 (44%) case-control studies. Of the 53 studies that considered confounding or effect modification, 39/53 (74%) gave no justification for the inclusion of the variables considered. Studies that justified the inclusion of the variables did so based on empirical evidence (n = 10), conceptual justification (n = 7), or a combination of the two (n = 3). Confounding was handled mainly by using regression modeling, but some case-control studies utilized matching and anova. Ten studies handled effect modification by stratification, eight tested for interaction, and five used both strategies.

CONCLUSIONS

We have found substantial shortcomings in the handling of confounding and effect modification in studies of diet and development of childhood asthma/allergies. Selection of variables should be based on conceptual considerations and empirical evidence. Using this approach, we have proposed a comprehensive set of confounders and effect modifiers that need to be considered in future studies.

Improving the effectiveness of clinical medicine: the need for better science

Effective clinical practice is predicated on valid and relevant clinical science - a commodity in increasingly short supply. The pre-eminent place of clinical research has become tainted by methodological shortcomings, commercial influences and neglect of the needs of patients and clinicians. Researchers need to be more proactive in evaluating clinical interventions in terms of patient-important benefit, wide applicability and comparative effectiveness, and in adopting study designs and reporting standards that ensure accurate and transparent research outputs. Funders of research need to be more supportive of applied clinical research that rigorously evaluates effectiveness of new treatments and synthesis existing knowledge into clinically useful systematic reviews. Several strategies for improving the state of the science are possible but their implementation requires collective action of all those undertaking and reporting clinical research.

The use of low-calorie sweeteners by adults: impact on weight management

The application of low-calorie sweeteners (LCS) in foods and beverages has increased over the past 35 y. At the same time, many characteristics of the American diet have changed, including variations in fat and carbohydrate content and composition, increased nutrient additions, and new dietary patterns due to changing lifestyles and attitudes toward food and the changing cost of food. During this same time period, the prevalence of overweight and obesity has increased from ~30 to 70% of adults in the United States. Clearly, these trends lead to a variety of hypotheses and efforts to explain the role of LCS in this association. The aim of this review is to gain clarity on the role of LCS in weight management and their impact on diet quality. In addition, because the majority of studies aimed at identifying associations between LCS and these outcomes are based on observational data, the pitfalls in designing and evaluating data from observational studies are also discussed. We conclude that there is no evidence that LCS can be claimed to be a cause of higher body weights in adults. Similarly, evidence supporting a role for LCS in weight management is lacking. Due to the confounders in most observational studies, randomized controlled trials are needed to advance understanding.

Selection of confounding variables should not be based on observed associations with exposure

In observational studies, selection of confounding variables for adjustment is often based on observed baseline incomparability. The aim of this study was to evaluate this selection strategy. We used clinical data on the effects of inhaled long-acting beta-agonist (LABA) use on the risk of mortality among patients with obstructive pulmonary disease to illustrate the impact of selection of confounding variables for adjustment based on baseline comparisons. Among 2,394 asthma and COPD patients included in the analyses, the LABA ever-users were considerably older than never-users, but cardiovascular co-morbidity was equally prevalent (19.9% vs. 19.9%). Adjustment for cardiovascular co-morbidity status did not affect the crude risk ratio (RR) for mortality: crude RR 1.19 (95% CI 0.93–1.51) versus RR 1.19 (95% CI 0.94–1.50) after adjustment for cardiovascular co-morbidity. However, after adjustment for age (RR 0.95, 95% CI 0.76–1.19), additional adjustment for cardiovascular co-morbidity status did affect the association between LABA use and mortality (RR 1.01, 95% CI 0.80–1.26). Confounding variables should not be discarded based on balanced distributions among exposure groups, because residual confounding due to the omission of confounding variables from the adjustment model can be relevant.

Method and reporting quality in health professions education research: a systematic review

CONTEXT

Studies evaluating reporting quality in health professions education (HPE) research have demonstrated deficiencies, but none have used comprehensive reporting standards. Additionally, the relationship between study methods and effect size (ES) in HPE research is unknown.

OBJECTIVES

This review aimed to evaluate, in a sample of experimental studies of Internet-based instruction, the quality of reporting, the relationship between reporting and methodological quality, and associations between ES and study methods.

METHODS

We conducted a systematic search of databases including MEDLINE, Scopus, CINAHL, EMBASE and ERIC, for articles published during 1990-2008. Studies (in any language) quantifying the effect of Internet-based instruction in HPE compared with no intervention or other instruction were included. Working independently and in duplicate, we coded reporting quality using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement, and coded study methods using a modified Newcastle-Ottawa Scale (m-NOS), the Medical Education Research Study Quality Instrument (MERSQI), and the Best Evidence in Medical Education (BEME) global scale.

RESULTS

For reporting quality, articles scored a mean±standard deviation (SD) of 51±25% of STROBE elements for the Introduction, 58±20% for the Methods, 50±18% for the Results and 41±26% for the Discussion sections. We found positive associations (all p<0.0001) between reporting quality and MERSQI (ρ=0.64), m-NOS (ρ=0.57) and BEME (ρ=0.58) scores. We explored associations between study methods and knowledge ES by subtracting each study's ES from the pooled ES for studies using that method and comparing these differences between subgroups. Effect sizes in single-group pretest/post-test studies differed from the pooled estimate more than ESs in two-group studies (p=0.013). No difference was found between other study methods (yes/no: representative sample, comparison group from same community, randomised, allocation concealed, participants blinded, assessor blinded, objective assessment, high follow-up).

CONCLUSIONS

Information is missing from all sections of reports of HPE experiments. Single-group pre-/post-test studies may overestimate ES compared with two-group designs. Other methodological variations did not bias study results in this sample.