Publication bias in clinical research

In a retrospective survey, 487 research projects approved by the Central Oxford Research Ethics Committee between 1984 and 1987, were studied for evidence of publication bias. As of May, 1990, 285 of the studies had been analysed by the investigators, and 52% of these had been published. Studies with statistically significant results were more likely to be published than those finding no difference between the study groups (adjusted odds ratio [OR] 2.32; 95% confidence interval [Cl] 1.25-4.28). Studies with significant results were also more likely to lead to a greater number of publications and presentations and to be published in journals with a high citation impact factor. An increased likelihood of publication was also associated with a high rating by the investigator of the importance of the study results, and with increasing sample size. The tendency towards publication bias was greater with observational and laboratory-based experimental studies (OR = 3.79; 95% Cl = 1.47-9.76) than with randomised clinical trials (OR = 0.84; 95% Cl = 0.34-2.09). We have confirmed the presence of publication bias in a cohort of clinical research studies. These findings suggest that conclusions based only on a review of published data should be interpreted cautiously, especially for observational studies. Improved strategies are needed to identify the results of unpublished as well as published studies.

Guidelines for reporting experiments involving animals: the ARRIVE guidelines

British Journal of Pharmacology (BJP) is pleased to publish a new set of guidelines for reporting research involving animals, simultaneously with several other journals; the 'ARRIVE' guidelines (Animals in Research: Reporting In Vivo Experiments). This editorial summarizes the background to the guidelines, gives our view of their significance, considers aspects of specific relevance to pharmacology, re-states BJP's guidelines for authors on animal experiments and indicates our commitment to carrying on discussion of this important topic. We also invite feedback via the British Pharmacological Society website.

No surgical innovation without evaluation: the IDEAL recommendations

Surgery and other invasive therapies are complex interventions, the assessment of which is challenged by factors that depend on operator, team, and setting, such as learning curves, quality variations, and perception of equipoise. We propose recommendations for the assessment of surgery based on a five-stage description of the surgical development process. We also encourage the widespread use of prospective databases and registries. Reports of new techniques should be registered as a professional duty, anonymously if necessary when outcomes are adverse. Case series studies should be replaced by prospective development studies for early technical modifications and by prospective research databases for later pre-trial evaluation. Protocols for these studies should be registered publicly. Statistical process control techniques can be useful in both early and late assessment. Randomised trials should be used whenever possible to investigate efficacy, but adequate pre-trial data are essential to allow power calculations, clarify the definition and indications of the intervention, and develop quality measures. Difficulties in doing randomised clinical trials should be addressed by measures to evaluate learning curves and alleviate equipoise problems. Alternative prospective designs, such as interrupted time series studies, should be used when randomised trials are not feasible. Established procedures should be monitored with prospective databases to analyse outcome variations and to identify late and rare events. Achievement of improved design, conduct, and reporting of surgical research will need concerted action by editors, funders of health care and research, regulatory bodies, and professional societies.

Implementing guidelines on reporting research using animals (ARRIVE etc.): new requirements for publication in BJP

The ARRIVE guidelines have been implemented in BJP for 4 years with the aim of increasing transparency in reporting experiments involving animals. BJP has assessed our success in implementing them and concluded that we could do better. This editorial discusses the issues and explains how we are changing our requirements for authors to report their findings in experiments involving animals. This is one of a series of editorials discussing updates to the BJP Instructions to Authors.

Experimental design and analysis and their reporting: new guidance for publication in BJP

Linked Editorials

This Editorial is part of a series. To view the other Editorials in this series, visit: http://onlinelibrary.wiley.com/doi/10.1111/bph.12956/abstract; http://onlinelibrary.wiley.com/doi/10.1111/bph.12954/abstract; http://onlinelibrary.wiley.com/doi/10.1111/bph.12955/abstract and http://onlinelibrary.wiley.com/doi/10.1111/bph.13112/abstract

A psychometric toolbox for testing validity and reliability

PURPOSE

To review the concepts of reliability and validity, provide examples of how the concepts have been used in nursing research, provide guidance for improving the psychometric soundness of instruments, and report suggestions from editors of nursing journals for incorporating psychometric data into manuscripts.

METHODS

CINAHL, MEDLINE, and PsycINFO databases were searched using key words: validity, reliability, and psychometrics. Nursing research articles were eligible for inclusion if they were published in the last 5 years, quantitative methods were used, and statistical evidence of psychometric properties were reported. Reports of strong psychometric properties of instruments were identified as well as those with little supporting evidence of psychometric soundness.

FINDINGS

Reports frequently indicated content validity but sometimes the studies had fewer than five experts for review. Criterion validity was rarely reported and errors in the measurement of the criterion were identified. Construct validity remains underreported. Most reports indicated internal consistency reliability (alpha) but few reports included reliability testing for stability. When retest reliability was asserted, time intervals and correlations were frequently not included.

CONCLUSIONS

Planning for psychometric testing through design and reducing nonrandom error in measurement will add to the reliability and validity of instruments and increase the strength of study findings. Underreporting of validity might occur because of small sample size, poor design, or lack of resources. Lack of information on psychometric properties and misapplication of psychometric testing is common in the literature.

What is a pilot or feasibility study? A review of current practice and editorial policy

BACKGROUND

In 2004, a review of pilot studies published in seven major medical journals during 2000-01 recommended that the statistical analysis of such studies should be either mainly descriptive or focus on sample size estimation, while results from hypothesis testing must be interpreted with caution. We revisited these journals to see whether the subsequent recommendations have changed the practice of reporting pilot studies. We also conducted a survey to identify the methodological components in registered research studies which are described as 'pilot' or 'feasibility' studies. We extended this survey to grant-awarding bodies and editors of medical journals to discover their policies regarding the function and reporting of pilot studies.

METHODS

Papers from 2007-08 in seven medical journals were screened to retrieve published pilot studies. Reports of registered and completed studies on the UK Clinical Research Network (UKCRN) Portfolio database were retrieved and scrutinized. Guidance on the conduct and reporting of pilot studies was retrieved from the websites of three grant giving bodies and seven journal editors were canvassed.

RESULTS

54 pilot or feasibility studies published in 2007-8 were found, of which 26 (48%) were pilot studies of interventions and the remainder feasibility studies. The majority incorporated hypothesis-testing (81%), a control arm (69%) and a randomization procedure (62%). Most (81%) pointed towards the need for further research. Only 8 out of 90 pilot studies identified by the earlier review led to subsequent main studies. Twelve studies which were interventional pilot/feasibility studies and which included testing of some component of the research process were identified through the UKCRN Portfolio database. There was no clear distinction in use of the terms 'pilot' and 'feasibility'. Five journal editors replied to our entreaty. In general they were loathe to publish studies described as 'pilot'.

CONCLUSION

Pilot studies are still poorly reported, with inappropriate emphasis on hypothesis-testing. Authors should be aware of the different requirements of pilot studies, feasibility studies and main studies and report them appropriately. Authors should be explicit as to the purpose of a pilot study. The definitions of feasibility and pilot studies vary and we make proposals here to clarify terminology.

Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review

OBJECTIVE

To determine whether the adoption of the CONSORT checklist is associated with improvement in the quality of reporting of randomised controlled trials (RCTs).

DATA SOURCES

MEDLINE, EMBASE, Cochrane CENTRAL, and reference lists of included studies and of experts were searched to identify eligible studies published between 1996 and 2005.

STUDY SELECTION

Studies were eligible if they (a) compared CONSORT-adopting and non-adopting journals after the publication of CONSORT, (b) compared CONSORT adopters before and after publication of CONSORT, or (c) a combination of (a) and (b). Outcomes examined included reports for any of the 22 items on the CONSORT checklist or overall trial quality.

DATA SYNTHESIS

1128 studies were retrieved, of which 248 were considered possibly relevant. Eight studies were included in the review. CONSORT adopters had significantly better reporting of the method of sequence generation (risk ratio [RR], 1.67; 95% CI, 1.19-2.33), allocation concealment (RR, 1.66; 95% CI, 1.37-2.00) and overall number of CONSORT items than non-adopters (standardised mean difference, 0.83; 95% CI, 0.46-1.19). CONSORT adoption had less effect on reporting of participant flow (RR, 1.14; 95% CI, 0.89-1.46) and blinding of participants (RR, 1.09; 95% CI, 0.84-1.43) or data analysts (RR, 5.44; 95% CI, 0.73-36.87). In studies examining CONSORT-adopting journals before and after the publication of CONSORT, description of the method of sequence generation (RR, 2.78; 95% CI, 1.78-4.33), participant flow (RR, 8.06; 95% CI, 4.10-15.83), and total CONSORT items (standardised mean difference, 3.67 items; 95% CI, 2.09-5.25) were improved after adoption of CONSORT by the journal.

CONCLUSIONS

Journal adoption of CONSORT is associated with improved reporting of RCTs.

Publication bias: evidence of delayed publication in a cohort study of clinical research projects

OBJECTIVES

To determine the extent to which publication is influenced by study outcome.

DESIGN

A cohort of studies submitted to a hospital ethics committee over 10 years were examined retrospectively by reviewing the protocols and by questionnaire. The primary method of analysis was Cox's proportional hazards model.

SETTING

University hospital, Sydney, Australia.

STUDIES

748 eligible studies submitted to Royal Prince Alfred Hospital Ethics Committee between 1979 and 1988.

MAIN OUTCOME MEASURES

Time to publication.

RESULTS

Response to the questionnaire was received for 520 (70%) of the eligible studies. Of the 218 studies analysed with tests of significance, those with positive results (P < 0.05) were much more likely to be published than those with negative results (P > or = 0.10) (hazard ratio 2.32 (95% confidence interval 1.47 to 3.66), P = 0.0003), with a significantly shorter time to publication (median 4.8 v 8.0 years). This finding was even stronger for the group of 130 clinical trials (hazard ratio 3.13 (1.76 to 5.58). P = 0.0001), with median times to publication of 4.7 and 8.0 years respectively. These results were not materially changed after adjusting for other significant predictors of publication. Studies with indefinite conclusions (0.05 < or = P < 0.10) tended to have an even lower publication rate and longer time to publication than studies with negative results (hazard ratio 0.39 (0.13 to 1.12), P = 0.08). For the 103 studies in which outcome was rated qualitatively, there was no clear cut evidence of publication bias, although the number of studies in this group was not large.

CONCLUSIONS

This study confirms the evidence of publication bias found in other studies and identifies delay in publication as an additional important factor. The study results support the need for prospective registration of trials to avoid publication bias and also support restricting the selection of trials to those started before a common date in undertaking systematic reviews.

Reproducible research in computational science

Computational science has led to exciting new developments, but the nature of the work has exposed limitations in our ability to evaluate published findings. Reproducibility has the potential to serve as a minimum standard for judging scientific claims when full independent replication of a study is not possible.

Evaluations of the uptake and impact of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement and extensions: a scoping review

BACKGROUND

The PRISMA Statement is a reporting guideline designed to improve transparency of systematic reviews (SRs) and meta-analyses. Seven extensions to the PRISMA Statement have been published to address the reporting of different types or aspects of SRs, and another eight are in development. We performed a scoping review to map the research that has been conducted to evaluate the uptake and impact of the PRISMA Statement and extensions. We also synthesised studies evaluating how well SRs published after the PRISMA Statement was disseminated adhere to its recommendations.

METHODS

We searched for meta-research studies indexed in MEDLINE® from inception to 31 July 2017, which investigated some component of the PRISMA Statement or extensions (e.g. SR adherence to PRISMA, journal endorsement of PRISMA). One author screened all records and classified the types of evidence available in the studies. We pooled data on SR adherence to individual PRISMA items across all SRs in the included studies and across SRs published after 2009 (the year PRISMA was disseminated).

RESULTS

We included 100 meta-research studies. The most common type of evidence available was data on SR adherence to the PRISMA Statement, which has been evaluated in 57 studies that have assessed 6487 SRs. The pooled results of these studies suggest that reporting of many items in the PRISMA Statement is suboptimal, even in the 2382 SRs published after 2009 (where nine items were adhered to by fewer than 67% of SRs). Few meta-research studies have evaluated the adherence of SRs to the PRISMA extensions or strategies to increase adherence to the PRISMA Statement and extensions.

CONCLUSIONS

Many studies have evaluated how well SRs adhere to the PRISMA Statement, and the pooled result of these suggest that reporting of many items is suboptimal. An update of the PRISMA Statement, along with a toolkit of strategies to help journals endorse and implement the updated guideline, may improve the transparency of SRs.

Prevalence of articles with honorary authors and ghost authors in peer-reviewed medical journals

CONTEXT

Authorship in biomedical publications establishes accountability, responsibility, and credit. Misappropriation of authorship undermines the integrity of the authorship system, but accurate data on its prevalence are limited.

OBJECTIVES

To determine the prevalence of articles with honorary authors (named authors who have not met authorship criteria) and ghost authors (individuals not named as authors but who contributed substantially to the work) in peer-reviewed medical journals and to identify journal characteristics and article types associated with such authorship misappropriation.

DESIGN

Mailed, self-administered, confidential survey.

PARTICIPANTS

A total of 809 corresponding authors (1179 surveyed, 69% response rate) of articles published in 1996 in 3 peer-reviewed, large-circulation general medical journals (Annals of Internal Medicine, JAMA, and The New England Journal of Medicine) and 3 peer-reviewed, smaller-circulation journals that publish supplements (American Journal of Cardiology, American Journal of Medicine, and American Journal of Obstetrics and Gynecology).

MAIN OUTCOME MEASURES

Prevalence of articles with honorary authors and ghost authors, as reported by corresponding authors.

RESULTS

Of the 809 articles, 492 were original research reports, 240 were reviews and articles not reporting original data, and 77 were editorials. A total of 156 articles (1 9%) had evidence of honorary authors (range, 11%-25% among journals); 93 articles (11%) had evidence of ghost authors (range, 7%-16% among journals); and 13 articles (2%) had evidence of both. The prevalence of articles with honorary authors was greater among review articles than research articles (odds ratio [OR], 1.8; 95% confidence interval [CI], 1.2-2.6) but did not differ significantly between large-circulation and smaller-circulation journals (OR, 1.4; 95% CI, 0.96-2.03). Compared with similar-type articles in large-circulation journals, articles with ghost authors in smaller-circulation journals were more likely to be reviews (OR, 4.2; 95% CI, 1.5-13.5) and less likely to be research articles (OR, 0.49; 95% CI, 0.27-0.88).

CONCLUSION

A substantial proportion of articles in peer-reviewed medical journals demonstrate evidence of honorary authors or ghost authors.

Guidelines for medical and health information sites on the internet: principles governing AMA web sites. American Medical Association

Access to medical information via the Internet has the potential to speed the transformation of the patient-physician relationship from that of physician authority ministering advice and treatment to that of shared decision making between patient and physician. However, barriers impeding this transformation include wide variations in quality of content on the Web, potential for commercial interests to influence online content, and uncertain preservation of personal privacy. To address these issues, the American Medical Association (AMA) has developed principles to guide development and posting of Web site content, govern acquisition and posting of online advertising and sponsorship, ensure site visitors' and patients' rights to privacy and confidentiality, and provide effective and secure means of e-commerce. While these guidelines were developed for the AMA Web sites and visitors to these sites, they also may be useful to other providers and users of medical information on the Web. These principles have been developed with the understanding that they will require frequent revision to keep pace with evolving technology and practices on the Internet. The AMA encourages review and feedback from readers, Web site visitors, policymakers, and all others interested in providing reliable quality information via the Web.

Conflict of interest in the debate over calcium-channel antagonists

BACKGROUND

Physicians' financial relationships with the pharmaceutical industry are controversial because such relationships may pose a conflict of interest. It is unknown to what extent industry support of medical education and research influences the opinions and behavior of clinicians and researchers. The recent debate over the safety of calcium-channel antagonists provided an opportunity to examine the effect of financial conflicts of interest.

METHODS

We searched the English-language medical literature published from March 1995 through September 1996 for articles examining the controversy about the safety of calcium-channel antagonists. Articles were reviewed and classified as being supportive, neutral, or critical with respect to the use of calcium-channel antagonists. The authors of the articles were asked about their financial relationships with both manufacturers of calcium-channel antagonists and manufacturers of competing products (i.e., beta-blockers, angiotensin-converting-enzyme inhibitors, diuretics, and nitrates). We examined the authors' published positions on the safety of calcium-channel antagonists according to their financial relationships with pharmaceutical companies.

RESULTS

Authors who supported the use of calcium-channel antagonists were significantly more likely than neutral or critical authors to have financial relationships with manufacturers of calcium-channel antagonists (96 percent, vs. 60 percent and 37 percent, respectively; P<0.001). Supportive authors were also more likely than neutral or critical authors to have financial relationships with any pharmaceutical manufacturer, irrespective of the product (100 percent, vs. 67 percent and 43 percent, respectively; P< 0.001).

CONCLUSIONS

Our results demonstrate a strong association between authors' published positions on the safety of calcium-channel antagonists and their financial relationships with pharmaceutical manufacturers. The medical profession needs to develop a more effective policy on conflict of interest. We support complete disclosure of relationships with pharmaceutical manufacturers for clinicians and researchers who write articles examining pharmaceutical products.