Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review

Reporting of noninferiority and equivalence randomized trials for major prostaglandins: a systematic survey of the ophthalmology literature

BACKGROUND

Standards for reporting clinical trials have improved the transparency of patient-important research. The Consolidated Standards of Reporting Trials (CONSORT) published an extension to address noninferiority and equivalence trials. We aimed to determine the reporting quality of prostaglandin noninferiority and equivalence trials in the treatment of glaucoma.

METHODS

We searched, independently and in duplicate, 6 electronic databases for eligible trials evaluating prostaglandins. We abstracted data on reporting of methodological criteria, including reporting of per-protocol [PP] and intention-to-treat [ITT] analysis, sample size estimation with margins, type of statistical analysis conducted, efficacy summaries, and use of hyperemia measures.

RESULTS

Trials involving the four major prostaglandin groups (latanoprost, travoprost, bimatoprost, unoprostone) were analyzed. We included 36 noninferiority and 11 equivalence trials. Seventeen out of the included 47 trials (36%, 95% Confidence Intervals [CI]: 24-51) were crossover designs. Only 3 studies (6%, 95% CI: 2-17) reported a presented results of both ITT and PP populations. Twelve studies (26%, 95% CI: 15-39) presented only ITT results but mentioned that PP population had similar results. Thirteen trials (28%, 95% CI: 17-42) presented only PP results with no mention of ITT population results while 17 studies (36%, 95% CI: 24-51) presented only ITT results with no mention of PP population results. Thirty-four (72%, 95% CI: 58-83) of studies adequately described their margin of noninferiority/equivalence. Sequence generation was reported in 22/47 trials (47%, 95% CI: 33-61). Allocation concealment was reported in only 10/47 (21%, 95% CI: 12-35) of the trials. Thirty-five studies (74%, 95% CI: 60-85) employed masking of at least two groups, 4/47 (9%, 95% CI: 3-20) masked only patients and 8/47 (17%, 95% CI: 9-30) were open label studies. Eight (17%, 95% CI: 9-30) of the 47 trials employed a combined test of noninferiority and superiority. We also found 6 differing methods of evaluating hyperemia.

CONCLUSION

The quality of reporting noninferiority/equivalency trials in the field of glaucoma is markedly heterogeneous. The adoption of the extended CONSORT statement by journals will potentially improve the transparency of this field.

Application of systematic review methodology to the field of nutrition

Systematic reviews represent a rigorous and transparent approach to synthesizing scientific evidence that minimizes bias. They evolved within the medical community to support development of clinical and public health practice guidelines, set research agendas, and formulate scientific consensus statements. The use of systematic reviews for nutrition-related topics is more recent. Systematic reviews provide independently conducted comprehensive and objective assessments of available information addressing precise questions. This approach to summarizing available data is a useful tool for identifying the state of science including knowledge gaps and associated research needs, supporting development of science-based recommendations and guidelines, and serving as the foundation for updates as new data emerge. Our objective is to describe the steps for performing systematic reviews and highlight areas unique to the discipline of nutrition that are important to consider in data assessment. The steps involved in generating systematic reviews include identifying staffing and planning for outside expert input, forming a research team, developing an analytic framework, developing and refining research questions, defining eligibility criteria, identifying search terms, screening abstracts according to eligibility criteria, retrieving articles for evaluation, constructing evidence and summary tables, assessing methodological quality and applicability, and synthesizing results including performing meta-analysis, if appropriate. Unique and at times challenging, nutrition-related considerations include baseline nutrient exposure, nutrient status, bioequivalence of bioactive compounds, bioavailability, multiple and interrelated biological functions, undefined nature of some interventions, and uncertainties in intake assessment. Systematic reviews are a valuable and independent component of decision-making processes by groups responsible for developing science-based recommendations and policies.

Publication guidelines for quality improvement studies in health care: evolution of the SQUIRE project

In 2005 we published draft guidelines for reporting studies of quality improvement interventions as the initial step in a consensus process for development of a more definitive version. The current article contains the revised version, which we refer to as SQUIRE (Standards for QUality Improvement Reporting Excellence). We describe the consensus process, which included informal feedback, formal written commentaries, input from publication guideline developers, review of the literature on the epistemology of improvement and on methods for evaluating complex social programs, and a meeting of stakeholders for critical review of the guidelines' content and wording, followed by commentary on sequential versions from an expert consultant group. Finally, we examine major differences between SQUIRE and the initial draft, and consider limitations of and unresolved questions about SQUIRE; we also describe ancillary supporting documents and alternative versions under development, and plans for dissemination, testing, and further development of SQUIRE.

The impact of the CONSORT statement on reporting of randomized clinical trials in psychiatry

To determine whether the CONSORT recommendations influenced the quality of reporting of randomized controlled trials (RCTs) in the field of psychiatry, we evaluated the quality of clinical trial reports before and after the introduction of CONSORT statement. We selected seven high impact journals and retrieved the randomized, clinical trials in the field of psychiatry during the period of 1992-1996 (pre-CONSORT) and 2002-2007 (post-CONSORT). Among the total 5201 articles screened, 736 were identified and entered in our database. After critical review of the publications, 442 articles met the inclusion and exclusion criteria. The CONSORT Index (sum of 22 items of the checklist) during the post-CONSORT period was significantly higher than that during the pre-CONSORT period. However, over 40% of post-CONSORT studies did not adhere to CONSORT statement for reporting the process of randomization, and details of the process for obtaining informed consent were still insufficient. Furthermore, adherence to the CONSORT guidelines of reporting how blinding was accomplished and evaluated actually decreased after publication of the CONSORT statement. Although the overall quality of reporting on psychiatric RCTs generally improved after publication of the CONSORT statement, reporting the details of randomization, blinding, and obtaining informed consent remain insufficient.

A systematic review of trends in the methodological quality of randomized controlled trials in various research fields

OBJECTIVES

We sought to evaluate the trends in the methodological quality of randomized controlled trials in various medical fields.

STUDY DESIGN AND SETTING

Relevant studies were retrieved by the PubMed and the ISI Web of science databases.

RESULTS

Thirty-five out of 457 retrieved studies met the inclusion criteria. Twenty-one out of 35 selected studies reported significant improvement in at least one methodological quality factor. Overall quality scores were increased in 13 out of 26 studies providing relevant data. The most commonly separately examined key quality factors were allocation concealment and blinding in 13 out of 21 studies that reported relevant data. Allocation concealment was the quality characteristic most commonly reported as significantly improving during the reviewed period (in five out of eight studies reporting relevant comparative data).

CONCLUSION

Certain aspects of methodological quality have improved significantly over time, but others remain stagnant. Further efforts to improve study design, conduct, and reporting of randomized controlled trials are warranted.

Improving the reporting of pragmatic trials: an extension of the CONSORT statement

BACKGROUND

The CONSORT statement is intended to improve reporting of randomised controlled trials and focuses on minimising the risk of bias (internal validity). The applicability of a trial's results (generalisability or external validity) is also important, particularly for pragmatic trials. A pragmatic trial (a term first used in 1967 by Schwartz and Lellouch) can be broadly defined as a randomised controlled trial whose purpose is to inform decisions about practice. This extension of the CONSORT statement is intended to improve the reporting of such trials and focuses on applicability. Methods At two, two-day meetings held in Toronto in 2005 and 2008, we reviewed the CONSORT statement and its extensions, the literature on pragmatic trials and applicability, and our experiences in conducting pragmatic trials. Recommendations We recommend extending eight CONSORT checklist items for reporting of pragmatic trials: the background, participants, interventions, outcomes, sample size, blinding, participant flow, and generalisability of the findings. These extensions are presented, along with illustrative examples of reporting, and an explanation of each extension. Adherence to these reporting criteria will make it easier for decision makers to judge how applicable the results of randomised controlled trials are to their own conditions. Empirical studies are needed to ascertain the usefulness and comprehensiveness of these CONSORT checklist item extensions. In the meantime we recommend that those who support, conduct, and report pragmatic trials should use this extension of the CONSORT statement to facilitate the use of trial results in decisions about health care.

Technical editing of research reports in biomedical journals

BACKGROUND

Most journals try to improve their articles by technical editing processes such as proof-reading, editing to conform to 'house styles', grammatical conventions and checking accuracy of cited references. Despite the considerable resources devoted to technical editing, we do not know whether it improves the accessibility of biomedical research findings or the utility of articles. This is an update of a Cochrane methodology review first published in 2003.

OBJECTIVES

To assess the effects of technical editing on research reports in peer-reviewed biomedical journals, and to assess the level of accuracy of references to these reports.

SEARCH STRATEGY

We searched The Cochrane Library Issue 2, 2007; MEDLINE (last searched July 2006); EMBASE (last searched June 2007) and checked relevant articles for further references. We also searched the Internet and contacted researchers and experts in the field.

SELECTION CRITERIA

Prospective or retrospective comparative studies of technical editing processes applied to original research articles in biomedical journals, as well as studies of reference accuracy.

DATA COLLECTION AND ANALYSIS

Two review authors independently assessed each study against the selection criteria and assessed the methodological quality of each study. One review author extracted the data, and the second review author repeated this.

MAIN RESULTS

We located 32 studies addressing technical editing and 66 surveys of reference accuracy. Only three of the studies were randomised controlled trials. A 'package' of largely unspecified editorial processes applied between acceptance and publication was associated with improved readability in two studies and improved reporting quality in another two studies, while another study showed mixed results after stricter editorial policies were introduced. More intensive editorial processes were associated with fewer errors in abstracts and references. Providing instructions to authors was associated with improved reporting of ethics requirements in one study and fewer errors in references in two studies, but no difference was seen in the quality of abstracts in one randomised controlled trial. Structuring generally improved the quality of abstracts, but increased their length. The reference accuracy studies showed a median citation error rate of 38% and a median quotation error rate of 20%.

AUTHORS' CONCLUSIONS

Surprisingly few studies have evaluated the effects of technical editing rigorously. However there is some evidence that the 'package' of technical editing used by biomedical journals does improve papers. A substantial number of references in biomedical articles are cited or quoted inaccurately.

Publication guidelines for quality improvement in health care: evolution of the SQUIRE project

In 2005, draft guidelines were published for reporting studies of quality improvement interventions as the initial step in a consensus process for development of a more definitive version. This article contains the full revised version of the guidelines, which the authors refer to as SQUIRE (Standards for QUality Improvement Reporting Excellence). This paper also describes the consensus process, which included informal feedback from authors, editors and peer reviewers who used the guidelines; formal written commentaries; input from a group of publication guideline developers; ongoing review of the literature on the epistemology of improvement and methods for evaluating complex social programmes; a two-day meeting of stakeholders for critical discussion and debate of the guidelines' content and wording; and commentary on sequential versions of the guidelines from an expert consultant group. Finally, the authors consider the major differences between SQUIRE and the initial draft guidelines; limitations of and unresolved questions about SQUIRE; ancillary supporting documents and alternative versions that are under development; and plans for dissemination, testing and further development of SQUIRE.

Quality of reporting of randomized controlled trials in general endocrinology literature

CONTEXT

The reporting quality of randomized controlled trials (RCTs) is poor in general medicine and several areas of specialization but unknown in endocrinology.

OBJECTIVE

Our aim was to assess the reporting quality of RCTs in general endocrinology. A secondary objective was to identify predictors for better reporting quality.

DESIGN AND SETTING

We systematically reviewed RCTs published in three general endocrinology journals between January 2005 and December 2006.

PARTICIPANTS

We included parallel-design RCTs that addressed a question of treatment or prevention. Article selection and data abstraction were conducted by two reviewers independently, and disagreements were resolved by consensus.

MAIN OUTCOMES

There were two main outcomes: 1) a 15-point overall reporting quality score (OQS) based on the Consolidated Standards for Reporting Trials (CONSORT); and 2) a 3-point key score, based on allocation concealment, blinding, and use of intention-to-treat analysis.

RESULTS

Eighty nine RCTs were included. The median OQS was 10 (interquartile range = 2). Allocation concealment, blinding, and analysis by intention to treat were reported in 10, 20, and 16 of the 89 RCTs, respectively. A multivariable regression analysis showed that complete industrial funding [incidence rate ratio (IRR) = 1.014; 95% confidence interval (CI), 1.010-1.018], journal of publication (IRR = 1.068; 95% CI, 1.007-1.132), and sample size (IRR = 1.048; 95% CI, 1.026-1.070) were significantly associated with a slightly better OQS.

CONCLUSIONS

The quality of RCT reporting in general endocrine literature is suboptimal. We discuss our results, highlight the areas where improvements are needed, and provide some recommendations.

Enhancing access to reports of randomized trials published world-wide--the contribution of EMBASE records to the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library

Background

Randomized trials are essential in assessing the effects of healthcare interventions and are a key component in systematic reviews of effectiveness. Searching for reports of randomized trials in databases is problematic due to the absence of appropriate indexing terms until the 1990s and inconsistent application of these indexing terms thereafter.

Objectives

The objectives of this study are to devise a search strategy for identifying reports of randomized trials in EMBASE which are not already indexed as trials in MEDLINE and to make these reports easily accessible by including them in the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library, with the permission of Elsevier, the publishers of EMBASE.

Methods

A highly sensitive search strategy was designed for EMBASE based on free-text and thesaurus terms which occurred frequently in the titles, abstracts, EMTREE terms (or some combination of these) of reports of trials indexed in EMBASE. This search strategy was run against EMBASE from 1980 to 2005 (1974 to 2005 for four of the terms) and records retrieved by the search, which were not already indexed as randomized trials in MEDLINE, were downloaded from EMBASE, printed and read. An analysis of the language of publication was conducted for the reports of trials published in 2005 (the most recent year completed at the time of this study).

Results

Twenty-two search terms were used (including nine which were later rejected due to poor cumulative precision). More than a third of a million records were downloaded and scanned and approximately 80,000 reports of trials were identified which were not already indexed as randomized trials in MEDLINE. These are now easily identifiable in CENTRAL, in The Cochrane Library. Cumulative sensitivity ranged from 0.1% to 60% and cumulative precision ranged from 8% to 61%. The truncated term 'random$' identified 60% of the total number of reports of trials but only 35% of the more than 130,000 records retrieved by this term were reports of trials. The language analysis for the sample year 2005 indicated that of the 18,427 reports indexed as randomized trials in MEDLINE, 959 (5%) were in languages other than English. The EMBASE search identified an additional 658 reports in languages other than English, of which the highest number were in Chinese (320).

Conclusion

The results of the search to date have greatly increased access to reports of trials in EMBASE, especially in some languages other than English. The search strategy used was subjectively derived from a small 'gold standard' set of test records and was not validated in an independent test set. We intend to design an objectively-derived validated search strategy using logistic regression based on the frequency of occurrence of terms in the approximately 80,000 reports of randomized trials identified compared with the frequency of these terms across the entire EMBASE database.

[The Strengthening the Reporting of Observational Studies in Epidemiology [STROBE] statement: guidelines for reporting observational studies]

Much biomedical research is observational. The reporting of such research is often inadequate, which hampers the assessment of its strengths and weaknesses and of a study's generalisability. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) initiative developed recommendations on what should be included in an accurate and complete report of an observational study. We defined the scope of the recommendations to cover three main study designs: cohort, case-control, and cross-sectional studies. We convened a 2-day workshop in September, 2004, with methodologists, researchers, and journal editors to draft a che-cklist of items. This list was subsequently revised during several meetings of the coordinating group and in e-mail discussions with the larger group of STROBE contributors, taking into account empirical evidence and methodological considerations. The workshop and the subsequent iterative process of consultation and revision resulted in a checklist of 22 items (the STROBE statement) that relate to the title, abstract, introduction, methods, results, and discussion sections of articles. 18 items are common to all three study designs and four are specific for cohort, case-control, or cross-sectional studies. A detailed explanation and elaboration document is published separately and is freely available on the websites of PLoS Medicine, Annals of Internal Medicine, and Epidemiology. We hope that the STROBE statement will contribute to improving the quality of reporting of observational studies.

Good laboratory practice: preventing introduction of bias at the bench

BACKGROUND AND PURPOSE

As a research community, we have failed to demonstrate that drugs which show substantial efficacy in animal models of cerebral ischemia can also improve outcome in human stroke. Summary of Review- Accumulating evidence suggests this may be due, at least in part, to problems in the design, conduct and reporting of animal experiments which create a systematic bias resulting in the overstatement of neuroprotective efficacy.

CONCLUSIONS

Here, we set out a series of measures to reduce bias in the design, conduct and reporting of animal experiments modeling human stroke.

Summary of evidence-based review on vitamin D efficacy and safety in relation to bone health

The objective of this evidence review was to synthesize the literature on the effectiveness and safety of nutritional and ultraviolet radiation sources of vitamin D with respect to bone health outcomes at all stages of life. The goals were to identify knowledge gaps for the research community and to highlight areas that required further research. We completed an extensive literature search of multiple databases and a multilevel selection process with synthesis of results from 167 included studies. We included a variety of outcomes (eg, falls, bone mineral density, fractures, and adverse events). This report provides an overview of the methods and a summary of the key findings. In addition, we discuss areas where the evidence is inconclusive, as well as methodologic issues that we encountered. We found inconsistent evidence of an association between serum 25-hydroxyvitamin D [25(OH)D] concentration and bone mineral content in infants and fair evidence of an association with bone mineral content or density in older children and older adults. The evidence of an association between serum 25(OH)D concentration and some clinical outcomes (fractures, performance measures) in postmenopausal women and older men was inconsistent, and the evidence of an association with falls was fair. We found good evidence of a positive effect of consuming vitamin D-fortified foods on 25(OH)D concentrations. The evidence for a benefit of vitamin D on falls and fractures varied. We found fair evidence that adults tolerated vitamin D at doses above current dietary reference intake levels, but we had no data on the association between long-term harms and higher doses of vitamin D.

What is missing from descriptions of treatment in trials and reviews?

Replicating non-pharmacological treatments in practice depends on how well they have been described in research studies

Guidelines for reporting health research: the EQUATOR network's survey of guideline authors

Iveta Simera and colleagues survey the authors of 37 guidelines on reporting health research.

Helping editors, peer reviewers and authors improve the clarity, completeness and transparency of reporting health research

Inadequate reporting is problematic for several reasons. If authors do not provide sufficient details concerning the conduct of their study, readers are left with an incomplete picture of what was done. As such, they are not able to judge the merits of the results and interpret them. The EQUATOR Network is a new initiative aimed at improving the clarity and transparency of reporting health research.

Chinese authors do need CONSORT: reporting quality assessment for five leading Chinese medical journals

OBJECTIVE

Only a few Chinese medical journals have recommended CONSORT in their "Instruction for authors or Guide for authors". This study aims to evaluate the reporting quality of randomized controlled trials (RCTs) published in the five leading Chinese medical journals indexed by MEDLINE.

METHODS

We identified RCTs published from 2004 to January 2007 in five leading Chinese medical journals by searching three important Chinese databases systematically, namely CNKI (China National Knowledge Infrastructure/Chinese Academic Journals full text Database), VIP (a full text database of China) and CBM disc (China Biomedicine Database Disc) and assessed the quality of each RCT by using the Consolidated Standards for Reporting of Trials (CONSORT) and the 5-point Jadad scale.

RESULTS

One hundred and forty two RCTs were included. Based on the items in the revised CONSORT statement, 130 (91.55%) of the 142 RCTs mentioned "randomization" in the title or abstract, but only 38 (26.76%) RCTs described the method to generate the random sequence; only 6 RCTs had adequate allocation concealment; 24 (17.61%) RCTs mentioned "masking", but only 7 described the process of masking. Three out of 40 items were reported clearly in all included trials, while five items were not mentioned at all. The quality of RCTs was low as assessed by the Jadad scale and 22 RCTs were high-quality research (>/=3 points).

CONCLUSIONS

The reporting quality of RCTs published in the five leading Chinese medical journals is low. Chinese journals should adopt the CONSORT statement to improve the reporting quality of Chinese randomized controlled trials.

Endorsement of the CONSORT Statement by high impact factor medical journals: a survey of journal editors and journal 'Instructions to Authors'

BACKGROUND

The CONSORT Statement provides recommendations for reporting randomized controlled trials. We assessed the extent to which leading medical journals that publish reports of randomized trials incorporate the CONSORT recommendations into their journal and editorial processes.

METHODS

This article reports on two observational studies. Study 1: We examined the online version of 'Instructions to Authors' for 165 high impact factor medical journals and extracted all text mentioning the CONSORT Statement or CONSORT extension papers. Any mention of the International Committee of Medical Journal Editors (ICMJE) or clinical trial registration were also sought and extracted. Study 2: We surveyed the editor-in-chief, or editorial office, for each of the 165 journals about their journal's endorsement of CONSORT recommendations and its incorporation into their editorial and peer-review processes.

RESULTS

Study 1: Thirty-eight percent (62/165) of journals mentioned the CONSORT Statement in their online 'Instructions to Authors'; of these 37% (23/62) stated this was a requirement, 63% (39/62) were less clear in their recommendations. Very few journals mentioned the CONSORT extension papers. Journals that referred to CONSORT were more likely to refer to ICMJE guidelines (RR 2.16; 95% CI 1.51 to 3.08) and clinical trial registration (RR 3.67; 95% CI 2.36 to 5.71) than those journals which did not.Study 2: Thirty-nine percent (64/165) of journals responded to the on-line survey, the majority were journal editors. Eighty-eight percent (50/57) of journals recommended authors comply with the CONSORT Statement; 62% (35/56) said they would require this. Forty-one percent (22/53) reported incorporating CONSORT into their peer-review process and 47% (25/53) into their editorial process. Eighty-one percent (47/58) reported including CONSORT in their 'Instructions to Authors' although there was some inconsistency when cross checking information on the journal's website. Sixty-nine percent (31/45) of journals recommended authors comply with the CONSORT extension for cluster trials, 60% (27/45) for harms and 42% (19/45) for non-inferiority and equivalence trials. Few journals mentioned these extensions in their 'Instructions to Authors'.

CONCLUSION

Journals should be more explicit in their recommendations and expectations of authors regarding the CONSORT Statement and related CONSORT extensions papers.

The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies

Much of biomedical research is observational. The reporting of such research is often inadequate, which hampers the assessment of its strengths and weaknesses and of a study's generalizability. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Initiative developed recommendations on what should be included in an accurate and complete report of an observational study. We defined the scope of the recommendations to cover three main study designs: cohort, case-control, and cross-sectional studies. We convened a 2-day workshop in September 2004, with methodologists, researchers, and journal editors to draft a checklist of items. This list was subsequently revised during several meetings of the coordinating group and in e-mail discussions with the larger group of STROBE contributors, taking into account empirical evidence and methodological considerations. The workshop and the subsequent iterative process of consultation and revision resulted in a checklist of 22 items (the STROBE Statement) that relate to the title, abstract, introduction, methods, results, and discussion sections of articles. Eighteen items are common to all three study designs and four are specific for cohort, case-control, or cross-sectional studies. A detailed Explanation and Elaboration document is published separately and is freely available on the web sites of PLoS Medicine, Annals of Internal Medicine, and Epidemiology. We hope that the STROBE Statement will contribute to improving the quality of reporting of observational studies.

The making of STROBE

This commentary gives a personal view of drafting the "Strengthening the Reporting of Observational Studies in Epidemiology" (STROBE) statement, by one of its authors. My initial wariness about guidelines for observational research was overcome by focusing on clarity of reporting, rather than on how research should be done. Areas of tension that arose when drafting STROBE include the problem of finding common ground among researchers with different research backgrounds, questions of the intended audience (professional epidemiologists or statisticians vs. all researchers who use epidemiologic study designs), and the fine line between encouraging clarity of reporting vs. prescribing how to do research. STROBE is not an instrument to evaluate the quality of research: research can be reported clearly or not, irrespective of its intrinsic quality. However, the ultimate benefit of STROBE might be that more comprehensive reporting allows for better discussions about published observational research, which may lead to better decisions about what new analyses or new studies are needed to solve a problem.

The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies

Much biomedical research is observational. The reporting of such research is often inadequate, which hampers the assessment of its strengths and weaknesses and of a study's generalizability. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Initiative developed recommendations on what should be included in an accurate and complete report of an observational study. We defined the scope of the recommendations to cover three main study designs: cohort, case-control and cross-sectional studies. We convened a 2-day workshop in September 2004, with methodologists, researchers, and journal editors to draft a checklist of items. This list was subsequently revised during several meetings of the coordinating group and in e-mail discussions with the larger group of STROBE contributors, taking into account empirical evidence and methodological considerations. The workshop and the subsequent iterative process of consultation and revision resulted in a checklist of 22 items (the STROBE Statement) that relate to the title, abstract, introduction, methods, results, and discussion sections of articles. 18 items are common to all three study designs and four are specific for cohort, case-control, or cross-sectional studies. A detailed "Explanation and Elaboration" document is published separately and is freely available on the web sites of PLoS Medicine, Annals of Internal Medicine, and Epidemiology. We hope that the STROBE Statement will contribute to improving the quality of reporting of observational studies.

Systematic reviews and meta-analyses in coloproctology: interpretation and potential pitfalls

A systematic review (SR) is the unbiased appraisal of systematically identified relevant studies. Implicit in its definition is a robust and scientifically valid process, and when performed as such, SR is an important clinical research tool and influence in health policy decision-making. This educational paper outlines that, from the original prototype based on randomized trials, there are now many other types of SRs including those based on: nonrandomized comparative studies, observational studies, prognostic studies, and studies of diagnostic and screening tools. While each of these has a similar 'anatomy' or format, at an individual class level, there are principles specific to each SR type. Several examples from the coloproctology literature are used as case-studies to illustrate potential pitfalls, and upon re-analysis, often reverse or attenuate the conclusions stated in the original publication. These examples serve to emphasize the need for health professionals to understand the process of SR and meta-analysis so that we all arrive at appropriate interpretations to the benefit of our patients.

Towards guidelines for evaluation of measures: an introduction with application to spinal cord injury

BACKGROUND

Both clinical practice and research in spinal cord injury (SCI) continue to struggle with issues of the quality and utility of outcome measures employed. Despite widespread deference to dicta on "reliability and validity," systematic means of grading the level of evidence for measures are lacking.

OBJECTIVES

This paper explains the methods and principles for use in systematic reviews of measures in SCI. It explains how extant measurement standards and principles can be elaborated for extant labels on various types of reliability and validity to define a more judicious method of grading level of evidence. We aim to initiate a process of discussion that will lead to improved systematic review of the measurement quality as a basis for long-term improvements in outcomes measures and their application.

METHODS

This paper is a conceptual review, based on established measurement standards and principles and the incorporation of recent advances in measurement methodology. The scheme of grading of measurement quality is illustrated by examples of measures of health, function, activity/participation, and quality of life after SCI.

RESULTS AND CONCLUSIONS

It is possible to grade the quality of outcome measure in terms of level of evidence, provided the nature of the construct being measured is defined as well as its main use. Definite means of grading the level of evidence for measurement will help to identify priorities for measure development and facilitate more appropriate uses of measures.

Do health technology assessments comply with QUOROM diagram guidance? An empirical study

BACKGROUND

The Quality of Reporting of Meta-analyses (QUOROM) statement provides guidance for improving the quality of reporting of systematic reviews and meta-analyses. To make the process of study selection transparent it recommends "a flow diagram providing information about the number of RCTs identified, included, and excluded and the reasons for excluding them". We undertook an empirical study to identify the extent of compliance in the UK Health Technology Assessment (HTA) programme.

METHODS

We searched Medline to retrieve all systematic reviews of therapeutic interventions in the HTA monograph series published from 2001 to 2005. Two researchers recorded whether each study contained a meta-analysis of controlled trials, whether a QUOROM flow diagram was presented and, if so, whether it expressed the relationship between the number of citations and the number of studies. We used Cohen's kappa to test inter-rater reliability.

RESULTS

87 systematic reviews were retrieved. There was good and excellent inter-rater reliability for, respectively, whether a review contained a meta-analysis and whether each diagram contained a citation-to-study relationship. 49% of systematic reviews used a study selection flow diagram. When only systematic reviews containing a meta-analysis were analysed, compliance was only 32%. Only 20 studies (23% of all systematic reviews; 43% of those having a study selection diagram) had a diagram which expressed the relationship between citations and studies.

CONCLUSION

Compliance with the recommendations of the QUOROM statement is not universal in systematic reviews or meta-analyses. Flow diagrams make the conduct of study selection transparent only if the relationship between citations and studies is clearly expressed. Reviewers should understand what they are counting: citations, papers, studies and trials are fundamentally different concepts which should not be confused in a diagram.

The minimum information about a proteomics experiment (MIAPE)

Both the generation and the analysis of proteomics data are now widespread, and high-throughput approaches are commonplace. Protocols continue to increase in complexity as methods and technologies evolve and diversify. To encourage the standardized collection, integration, storage and dissemination of proteomics data, the Human Proteome Organization's Proteomics Standards Initiative develops guidance modules for reporting the use of techniques such as gel electrophoresis and mass spectrometry. This paper describes the processes and principles underpinning the development of these modules; discusses the ramifications for various interest groups such as experimentalists, funders, publishers and the private sector; addresses the issue of overlap with other reporting guidelines; and highlights the criticality of appropriate tools and resources in enabling 'MIAPE-compliant' reporting.

The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies

Much biomedical research is observational. The reporting of such research is often inadequate, which hampers the assessment of its strengths and weaknesses and of a study's generalizability. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Initiative developed recommendations on what should be included in an accurate and complete report of an observational study. We defined the scope of the recommendations to cover three main study designs: cohort, case-control and cross-sectional studies. We convened a two-day workshop, in September 2004, with methodologists, researchers and journal editors to draft a checklist of items. This list was subsequently revised during several meetings of the coordinating group and in e-mail discussions with the larger group of STROBE contributors, taking into account empirical evidence and methodological considerations. The workshop and the subsequent iterative process of consultation and revision resulted in a checklist of 22 items (the STROBE Statement) that relate to the title, abstract, introduction, methods, results and discussion sections of articles. Eighteen items are common to all three study designs and four are specific for cohort, case-control, or cross-sectional studies. A detailed Explanation and Elaboration document is published separately and is freely available on the web sites of PLoS Medicine, Annals of Internal Medicine and Epidemiology. We hope that the STROBE Statement will contribute to improving the quality of reporting of observational studies.

Effectiveness of multidisciplinary rehabilitation services in postacute care: state-of-the-science. A review

OBJECTIVES

To summarize the efficacy of postacute rehabilitation and to outline future research strategies for increasing knowledge of its effectiveness.

DATA SOURCES

English-language systematic reviews that examined multidisciplinary therapy-based rehabilitation services for adults, published in the last 25 years and available through Cochrane, Medline, or CINAHL databases. We excluded multidisciplinary biopsychosocial rehabilitation programs and mental health services.

STUDY SELECTION

Using the search term rehabilitation, 167 records were identified in the Cochrane database, 1163 meta-analyses and reviews were identified in Medline, and 226 in CINAHL. The Medline and CINAHL search was further refined with 3 additional search terms: therapy, multidisciplinary, and interdisciplinary. In summary, we used 12 reviews to summarize the efficacy of multidisciplinary, therapy-based postacute rehabilitation; the 12 covered only 5 populations.

DATA EXTRACTION

Two reviewers extracted information about study populations, sample sizes, study designs, the settings and timing of rehabilitation, interventions, and findings.

DATA SYNTHESIS

Based on systematic reviews, the evidence for efficacy of postacute rehabilitation services across the continuum was strongest for stroke. There was also strong evidence supporting multidisciplinary inpatient rehabilitation for patients with rheumatoid arthritis, moderate to severe acquired brain injury, including traumatic etiologies, and for older adults. Heterogeneity limited our ability to conclude a benefit or a lack of a benefit for rehabilitation in other postacute settings for the other conditions in which systematic reviews had been completed. The efficacy of multidisciplinary rehabilitation services has not been systematically reviewed for many of the diagnostic conditions treated in rehabilitation. We did not complete a summary of findings from individual studies.

CONCLUSIONS

Given the limitations and paucity of systematic reviews, information from carefully designed nonrandomized studies could be used to complement randomized controlled trials in the study of the effectiveness of postacute rehabilitation. Consequently, a stronger evidence base would become available with which to inform policy decisions, guide the use of services, and improve patient access and outcomes.

Methodological quality of randomised controlled trials comparing short-term results of laparoscopic and conventional colorectal resection

BACKGROUND

Randomised, controlled trials (RCT) and systematic reviews of RCT with meta-analysis are considered to be of highest methodological quality and therefore are given the highest level of evidence (Ia/b). Although, "low-quality" RCT may be downgraded to level of evidence IIb, the methodological quality of each individual RCT is not respected in detail in this classification of the level of evidence.

MATERIALS AND METHODS

Within a systematic Cochrane Review of RCT on short-term benefits of laparoscopic or conventional colorectal resections, the methodological quality of all included RCT was evaluated. All RCT were assessed by the Evans and Pollock questionnaire (E and P increasing quality from 0-100) and the Jadad score (increasing quality from 0-5).

RESULTS

Publications from 28 RCT printed from 1996 to 2005 were included in the analysis. Methodological quality of RCT was only moderate [E & P 55 (32-84); Jadad 2 (1-5)]. There was a significant correlation between the E & P and the Jadad score (r = 0.788; p < 0.001). Methodological quality of RCT slightly increased with increasing number of patients included (r = 0.494; p = 0.009) and year of publication (r = 0.427; p = 0.03). Meta-analysis of all RCT yielded clinically relevant differences for overall and local morbidity when compared to meta-analysis of "high-quality" (E & P > 70) RCT only.

CONCLUSION

The methodological quality of reports of RCT comparing laparoscopic and open colorectal resection varies considerably. In a systematic review, methodological quality of RCT should be assessed because meta-analysis of "high-quality" RCT may yield different results than meta-analysis of all RCT.

The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies

Much biomedical research is observational. The reporting of such research is often inadequate, which hampers the assessment of its strengths and weaknesses and of a study's generalisability. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) initiative developed recommendations on what should be included in an accurate and complete report of an observational study. We defined the scope of the recommendations to cover three main study designs: cohort, case-control, and cross-sectional studies. We convened a 2-day workshop in September, 2004, with methodologists, researchers, and journal editors to draft a checklist of items. This list was subsequently revised during several meetings of the coordinating group and in e-mail discussions with the larger group of STROBE contributors, taking into account empirical evidence and methodological considerations. The workshop and the subsequent iterative process of consultation and revision resulted in a checklist of 22 items (the STROBE statement) that relate to the title, abstract, introduction, methods, results, and discussion sections of articles.18 items are common to all three study designs and four are specific for cohort, case-control, or cross-sectional studies.A detailed explanation and elaboration document is published separately and is freely available on the websites of PLoS Medicine, Annals of Internal Medicine, and Epidemiology. We hope that the STROBE statement will contribute to improving the quality of reporting of observational studies

Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies

Poor reporting of research hampers assessment and makes it less useful. An international group of methodologists, researchers, and journal editors sets out guidelines to improve reports of observational studies

The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies

Much biomedical research is observational. The reporting of such research is often inadequate, which hampers the assessment of its strengths and weaknesses and of a study's generalisability. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Initiative developed recommendations on what should be included in an accurate and complete report of an observational study. We defined the scope of the recommendations to cover three main study designs: cohort, case-control, and cross-sectional studies. We convened a 2-day workshop in September 2004, with methodologists, researchers, and journal editors to draft a checklist of items. This list was subsequently revised during several meetings of the coordinating group and in e-mail discussions with the larger group of STROBE contributors, taking into account empirical evidence and methodological considerations. The workshop and the subsequent iterative process of consultation and revision resulted in a checklist of 22 items (the STROBE Statement) that relate to the title, abstract, introduction, methods, results, and discussion sections of articles. 18 items are common to all three study designs and four are specific for cohort, case-control, or cross-sectional studies. A detailed Explanation and Elaboration document is published separately and is freely available on the Web sites of PLoS Medicine, Annals of Internal Medicine, and Epidemiology. We hope that the STROBE Statement will contribute to improving the quality of reporting of observational studies.

Meta-analysis of gene–environment interactions in developmental psychopathology

As studies of measured gene–environment interactions (G × E) in developmental psychopathology gain momentum, methods for systematically and quantitatively summarizing effects across multiple studies are urgently needed. Meta-analyses of G × E findings are critical for evaluating the overall statistical and theoretical significance of any given G × E based on cumulative and systematically combined knowledge. Although meta-analytic methods for the combination of study findings based on single effect measures such as odds ratios and mean differences are well established, equivalent methods for the meta-analysis of studies investigating interactions are not well developed. This article describes one simple approach to the meta-analysis of G × E effects using, as a contemporaneous example, the interaction of the monoamine oxidase A (MAOA) gene and the impact of childhood maltreatment on risk for developing antisocial behavior.

The quality of reporting of randomized controlled trials of traditional Chinese medicine: a survey of 13 randomly selected journals from mainland China

BACKGROUND

The number of randomized controlled trials (RCTs) of traditional Chinese medicine (TCM) is increasing. However, there have been few systematic assessments of the quality of reporting of these trials.

OBJECTIVE

This study was undertaken to evaluate the quality of reporting of RCTs in TCM journals published in mainland China from 1999 to 2004.

METHODS

Thirteen TCM journals were randomly selected by stratified sampling of the approximately 100 TCM journals published in mainland China. All issues of the selected journals published from 1999 to 2004 were hand-searched according to guidelines from the Cochrane Centre. All reviewers underwent training in the evaluation of RCTs at the Chinese Centre of Evidence-based Medicine. A comprehensive quality assessment of each RCT was completed using a modified version of the Consolidated Standards of Reporting Trials (CONSORT) checklist (total of 30 items) and the Jadad scale. Disagreements were resolved by consensus.

RESULTS

Seven thousand four hundred twenty-two RCTs were identified. The proportion of published RCTs relative to all types of published clinical trials increased significantly over the period studied, from 18.6% in 1999 to 35.9% in 2004 (P < 0.001). The mean (SD) Jadad score was 1.03 (0.61) overall. One RCT had a Jadad score of 5 points; 14 had a score of 4 points; and 102 had a score of 3 points. The mean (SD) Jadad score was 0.85 (0.53) in 1999 (746 RCTs) and 1.20 (0.62) in 2004 (1634 RCTs). Across all trials, 39.4% of the items on the modified CONSORT checklist were reported, which was equivalent to 11.82 (5.78) of the 30 items. Some important methodologic components of RCTs were incompletely reported, such as sample-size calculation (reported in 1.1% of RCTs), randomization sequence (7.9%), allocation concealment (0.3 %), implementation of the random-allocation sequence (0%), and analysis of intention to treat (0%).

CONCLUSION

The findings of this study indicate that the quality of reporting of RCTs of TCM has improved, but remains poor.

The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies

Much biomedical research is observational. The reporting of such research is often inadequate, which hampers the assessment of its strengths and weaknesses and of a study's generalisability. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Initiative developed recommendations on what should be included in an accurate and complete report of an observational study. We defined the scope of the recommendations to cover three main study designs: cohort, case-control and cross-sectional studies. We convened a 2-day workshop in September 2004, with methodologists, researchers, and journal editors to draft a checklist of items. This list was subsequently revised during several meetings of the coordinating group and in e-mail discussions with the larger group of STROBE contributors, taking into account empirical evidence and methodological considerations. The workshop and the subsequent iterative process of consultation and revision resulted in a checklist of 22 items (the STROBE Statement) that relate to the title, abstract, introduction, methods, results, and discussion sections of articles. 18 items are common to all three study designs and four are specific for cohort, case-control, or cross-sectional studies. A detailed Explanation and Elaboration document is published separately and is freely available on the websites of PLoS Medicine, Annals of Internal Medicine and Epidemiology. We hope that the STROBE Statement will contribute to improving the quality of reporting of observational studies.

Quality of reporting of experimental studies in medical education: a systematic review

OBJECTIVE

Determine the prevalence of essential elements of reporting in experimental studies in medical education.

DESIGN

Systematic review.

DATA SOURCES

Articles published in 2003 and 2004 in Academic Medicine, Advances in Health Sciences Education, American Journal of Surgery, Journal of General Internal Medicine, Medical Education, and Teaching and Learning in Medicine.

REVIEW METHODS

Articles describing education experiments, including evaluation studies with experimental designs, were identified (n = 185) by reviewing titles and abstracts. A random sample (n = 110) was selected for full review. The full text of each article was evaluated for the presence of guideline-based features of quality reporting: a critical literature review, conceptual framework, statement of study intent (e.g. aim, research question, or hypothesis), statement of study design, definition of main intervention and comparison intervention or control group, and consideration of human subject rights.

RESULTS

Of the 105 articles suitable for review, 47 (45%) contained a critical literature review and 58 (55%) presented a conceptual framework. A statement of study intent was present in 80 articles (76%), among which the independent and dependent variables were operationally defined in 38 (47%) and 26 articles (32%), respectively. A total of 17 articles (16%) contained an explicit study design statement. Among the 48 studies with a comparison group, 35 (73%) clearly defined the comparison intervention or control group. Institutional review board approval or participant consent was reported in 44 articles (42%).

CONCLUSIONS

The quality of reporting of experimental studies in medical education was generally poor. Criteria are proposed as a starting point for establishing reporting standards for medical education research.