Reducing waste from incomplete or unusable reports of biomedical research

Can animal data translate to innovations necessary for a new era of patient-centred and individualised healthcare? Bias in preclinical animal research

Background

The public and healthcare workers have a high expectation of animal research which they perceive as necessary to predict the safety and efficacy of drugs before testing in clinical trials. However, the expectation is not always realised and there is evidence that the research often fails to stand up to scientific scrutiny and its 'predictive value' is either weak or absent.

Discussion

Problems with the use of animals as models of humans arise from a variety of biases and systemic failures including: 1) bias and poor practice in research methodology and data analysis; 2) lack of transparency in scientific assessment and regulation of the research; 3) long-term denial of weaknesses in cross-species translation; 4) profit-driven motives overriding patient interests; 5) lack of accountability of expenditure on animal research; 6) reductionist-materialism in science which tends to dictate scientific inquiry and control the direction of funding in biomedical research.

Summary

Bias in animal research needs to be addressed before medical research and healthcare decision-making can be more evidence-based. Research funding may be misdirected on studying 'disease mechanisms' in animals that cannot be replicated outside tightly controlled laboratory conditions, and without sufficient critical evaluation animal research may divert attention away from avenues of research that hold promise for human health. The potential for harm to patients and trial volunteers from reliance on biased animal data¹ requires measures to improve its conduct, regulation and analysis. This article draws attention to a few of the many forms of bias in animal research that have come to light in the last decade and offers a strategy incorporating ten recommendations stated at the end of each section on bias. The proposals need development through open debate and subsequent rigorous implementation so that reviewers may determine the value of animal research to human health. The 10Rs + are protected by a Creative Commons Attribution 3.0 Unported License and therefore may be 'shared, remixed or built on, even commercially, so long as attributed by giving appropriate credit with a link to the license, and indicate if changes were made.’

Electronic supplementary material

The online version of this article (doi:10.1186/s12910-015-0043-7) contains supplementary material, which is available to authorized users.

Are licensed canine parvovirus (CPV2 and CPV2b) vaccines able to elicit protection against CPV2c subtype in puppies?: A systematic review of controlled clinical trials

Severe gastroenteritis caused by canine parvovirus type 2 (CPV2) is a serious life-threatening disease in puppies less than 4-months of age. The emergence of new variants has provoked some concern about the cross-protection elicited by licensed canine parvovirus modified-live type 2 (CPV2) and type 2b (CPV2b) vaccines against the most recent subtype CPV2c. A systematic review was carried out to assess the efficacy of commercial vaccines. We conducted a literature search of Pub Med/MEDLINE from January 1990 to May 2014. This was supplemented by hand-searching of related citations and searches in Google/Google Scholar. Controlled clinical trials in which vaccinated puppies were challenged with CPV2c virus were evaluated. Reporting of outcome measures and results for vaccine efficacy were critically appraised through a variety of clinical signs, serological tests, virus shedding and the ability to overcome maternally derived antibodies (MDA) titres. Six controlled clinical trials were included in the review. In most cases, the results of the selected studies reported benefits in terms of clinical signs, serological tests and virus shedding. However, MDA interference was not considered or evaluated in 5 of the selected trials. No accurate definitions of baseline healthy status and/or clinical outcomes were provided. Methods of randomization, allocation concealment and blinding were usually poorly reported. As a result of the limited number of included studies matching the inclusion criteria, the small sample sizes, short follow-up and the methodological limitations observed, it was not possible to reach a final conclusion regarding the cross-protection of licensed CPV2 and CPV2b vaccines against the subtype 2c in puppies. Further and specifically designed trials are required in order to elucidate whether cross-protection is acquired from licensed CPV vaccines.

Occurrence and determinants of selective reporting of clinical drug trials: design of an inception cohort study

INTRODUCTION

Responsible conduct of research implies that results of clinical trials should be completely and adequately reported. This article describes the design of a cohort study that aims to investigate the occurrence and the determinants of selective reporting in an inception cohort of all clinical drug trials that were reviewed by the Dutch Institutional Review Boards (IRBs) in 2007. It also describes the characteristics of the study cohort.

METHODS AND ANALYSIS

In 2007, Dutch IRBs reviewed 622 clinical drug trials. For each trial, we assessed the stages of progress. We discriminated five intermediate stages and five definite stages. Intermediate stages of progress are: approved by an IRB; started inclusion; completed as planned; terminated early; published as article. The definite stages of progress are: rejected by an IRB; never started inclusion; not published as article; completely reported; selectively reported. We will use univariate and multivariate Cox regression models to identify trial characteristics associated with non-publication. We will identify seven trial-specific discrepancy items, including the objectives, inclusion and exclusion criteria, end points, sample size, additional analyses, type of population analysis and sponsor acknowledgement. The percentage of trials with discrepancies between the protocol and the publication will be scored. We will investigate the association between trial characteristics and the occurrence of discrepancies.

ETHICS AND DISSEMINATION

No IRB-approval is required for this study. Access to confidential research protocols was provided by the Central Committee on Research Involving Human Subjects. We plan to finish data collection in June 2015, and expect to complete data cleaning, analysis and manuscript preparation within the next 3 months. Hence, a first draft of an article containing the results is expected before the end of October 2015.

Laboratory animal research published in plastic surgery journals in 2014 has extensive waste: A systematic review

Laboratory animal research must be designed in a manner that minimizes bias if it is to yield valid and reproducible results. In 2009, a survey that examined 271 animal studies found that 87% did not use randomization and 86% did not use blinding. This has been called "research waste" because it wasted time and resources. This systematic review measured the quantity of research waste in plastic surgery journals in 2014.

METHOD

The PRISMA-P protocol was used. SCOPUS and PubMed searches were done for all animal studies published in 2014 in Aesthetic Plast Surg, Aesthet Surg J, Ann Plast Surg, JPRAS, J Plast Surg Hand Surg and Plast Reconstr Surg. These were supplemented by manual searches of the 2014 issues not indexed. Articles were analyzed for descriptions of randomization, randomization methodology, allocation concealment, and blinding of the primary outcome assessment. Corresponding authors who mentioned randomization without elaborating were emailed for details.

RESULTS

112 of 154 articles met the inclusion criteria. Only 24/112 (21.4%) had blinding of the primary outcome measure, 28/110 (25.5%) of articles that required randomization mentioned it. While 12/28 articles clearly described randomizing the intervention, only 4/28 described the method of randomization, and 2/28 mentioned allocation concealment. Only two authors responded and described the randomization methodology.

CONCLUSION

The quality of plastic surgery laboratory animal research published in 2014 was poor. Use of the National Centre for the Replacement Refinement & Reduction of Animals in Research's "Animal Research: Reporting In Vivo Experiments" (ARRIVE) Guidelines by authors, and enforcement of them by editors and reviewers could improve research quality and reduce waste.

Using ClinicalTrials.gov to supplement information in ophthalmology conference abstracts about trial outcomes: a comparison study

Background

Including results from unpublished randomized controlled trials (RCTs) in a systematic review may ameliorate the effect of publication bias in systematic review results. Unpublished RCTs are sometimes described in abstracts presented at conferences, included in trials registers, or both. Trial results may not be available in a trials register and abstracts describing RCT results often lack study design information. Complementary information from a trials register record may be sufficient to allow reliable inclusion of an unpublished RCT only available as an abstract in a systematic review.

Methods

We identified 496 abstracts describing RCTs presented at the 2007 to 2009 Association for Research in Vision and Ophthalmology (ARVO) meetings; 154 RCTs were registered in ClinicalTrials.gov. Two persons extracted verbatim primary and non-primary outcomes reported in the abstract and ClinicalTrials.gov record. We compared each abstract outcome with all ClinicalTrials.gov outcomes and coded matches as complete, partial, or no match.

Results

We identified 800 outcomes in 152 abstracts (95 primary [51 abstracts] and 705 [141 abstracts] non-primary outcomes). No outcomes were reported in 2 abstracts. Of 95 primary outcomes, 17 (18%) agreed completely, 53 (56%) partially, and 25 (26%) had no match with a ClinicalTrials.gov primary or non-primary outcome. Among 705 non-primary outcomes, 56 (8%) agreed completely, 205 (29%) agreed partially, and 444 (63%) had no match with a ClinicalTrials.gov primary or non-primary outcome. Among the 258 outcomes partially agreeing, we found additional information on the time when the outcome was measured more often in ClinicalTrials.gov than in the abstract (141/258 (55%) versus 55/258 (21%)). We found no association between the presence of non-matching “new” outcomes and year of registration, time to registry update, industry sponsorship, or multi-center status.

Conclusion

Conference abstracts may be a valuable source of information about results for outcomes of unpublished RCTs that have been registered in ClinicalTrials.gov. Complementary additional descriptive information may be present for outcomes reported in both sources. However, ARVO abstract authors also present outcomes not reported in ClinicalTrials.gov and these may represent analyses not originally planned.

Quality of methods reporting in animal models of colitis

BACKGROUND

Current understanding of the onset of inflammatory bowel diseases relies heavily on data derived from animal models of colitis. However, the omission of information concerning the method used makes the interpretation of studies difficult or impossible. We assessed the current quality of methods reporting in 4 animal models of colitis that are used to inform clinical research into inflammatory bowel disease: dextran sulfate sodium, interleukin-10, CD45RB T cell transfer, and 2,4,6-trinitrobenzene sulfonic acid (TNBS).

METHODS

We performed a systematic review based on PRISMA guidelines, using a PubMed search (2000-2014) to obtain publications that used a microarray to describe gene expression in colitic tissue. Methods reporting quality was scored against a checklist of essential and desirable criteria.

RESULTS

Fifty-eight articles were identified and included in this review (29 dextran sulfate sodium, 15 interleukin-10, 5 T cell transfer, and 16 TNBS; some articles use more than 1 colitis model). A mean of 81.7% (SD = ±7.038) of criteria were reported across all models. Only 1 of the 58 articles reported all essential criteria on our checklist. Animal age, gender, housing conditions, and mortality/morbidity were all poorly reported.

CONCLUSIONS

Failure to include all essential criteria is a cause for concern; this failure can have large impact on the quality and replicability of published colitis experiments. We recommend adoption of our checklist as a requirement for publication to improve the quality, comparability, and standardization of colitis studies and will make interpretation and translation of data to human disease more reliable.

Outcome discrepancies and selective reporting: impacting the leading journals?

Background

Selective outcome reporting of either interesting or positive research findings is problematic, running the risk of poorly-informed treatment decisions. We aimed to assess the extent of outcome and other discrepancies and possible selective reporting between registry entries and published reports among leading medical journals.

Methods

Randomized controlled trials published over a 6-month period from July to December 31^st, 2013, were identified in five high impact medical journals: The Lancet, British Medical Journal, New England Journal of Medicine, Annals of Internal Medicine and Journal of American Medical Association were obtained. Discrepancies between published studies and registry entries were identified and related to factors including registration timing, source of funding and presence of statistically significant results.

Results

Over the 6-month period, 137 RCTs were found. Of these, 18% (n = 25) had discrepancies related to primary outcomes with the primary outcome changed in 15% (n = 20). Moreover, differences relating to non-primary outcomes were found in 64% (n = 87) with both omission of pre-specified non-primary outcomes (39%) and introduction of new non-primary outcomes (44%) common. No relationship between primary or non-primary outcome change and registration timing (prospective or retrospective; P = 0.11), source of funding (P = 0.92) and presence of statistically significant results (P = 0.92) was found.

Conclusions

Discrepancies between registry entries and published articles for primary and non-primary outcomes were common among trials published in leading medical journals. Novel approaches are required to address this problem.

Core outcome sets and trial registries

Some reasons for registering trials might be considered as self-serving, such as satisfying the requirements of a journal in which the researchers wish to publish their eventual findings or publicising the trial to boost recruitment. Registry entries also help others, including systematic reviewers, to know about ongoing or unpublished studies and contribute to reducing research waste by making it clear what studies are ongoing. Other sources of research waste include inconsistency in outcome measurement across trials in the same area, missing data on important outcomes from some trials, and selective reporting of outcomes. One way to reduce this waste is through the use of core outcome sets: standardised sets of outcomes for research in specific areas of health and social care. These do not restrict the outcomes that will be measured, but provide the minimum to include if a trial is to be of the most use to potential users. We propose that trial registries, such as ISRCTN, encourage researchers to note their use of a core outcome set in their entry. This will help people searching for trials and those worried about selective reporting in closed trials. Trial registries can facilitate these efforts to make new trials as useful as possible and reduce waste. The outcomes section in the entry could prompt the researcher to consider using a core outcome set and facilitate the specification of that core outcome set and its component outcomes through linking to the original core outcome set. In doing this, registries will contribute to the global effort to ensure that trials answer important uncertainties, can be brought together in systematic reviews, and better serve their ultimate aim of improving health and well-being through improving health and social care.

Can findings from randomized controlled trials of social skills training in autism spectrum disorder be generalized? The neglected dimension of external validity

Systematic reviews have traditionally focused on internal validity, while external validity often has been overlooked. In this study, we systematically reviewed determinants of external validity in the accumulated randomized controlled trials of social skills group interventions for children and adolescents with autism spectrum disorder. We extracted data clustered into six overarching themes: source population, included population, context, treatment provider, treatment intervention, and outcome. A total of 15 eligible randomized controlled trials were identified. The eligible population was typically limited to high-functioning school-aged children with autism spectrum disorder, and the included population was predominantly male and Caucasian. Scant information about the recruitment of participants was provided, and details about treatment providers and settings were sparse. It was not evident from the trials to what extent acquired social skills were enacted in everyday life and maintained over time. We conclude that the generalizability of the accumulated evidence is unclear and that the determinants of external validity are often inadequately reported. At this point, more effectiveness-oriented randomized controlled trials of equally high internal and external validity are needed. More attention to the determinants of external validity is warranted when this new generation of randomized controlled trials are planned and reported. We provide a tentative checklist for this purpose.

RIPOSTE: a framework for improving the design and analysis of laboratory-based research

Lack of reproducibility is an ongoing problem in some areas of the biomedical sciences. Poor experimental design and a failure to engage with experienced statisticians at key stages in the design and analysis of experiments are two factors that contribute to this problem. The RIPOSTE (Reducing IrreProducibility in labOratory STudiEs) framework has been developed to support early and regular discussions between scientists and statisticians in order to improve the design, conduct and analysis of laboratory studies and, therefore, to reduce irreproducibility. This framework is intended for use during the early stages of a research project, when specific questions or hypotheses are proposed. The essential points within the framework are explained and illustrated using three examples (a medical equipment test, a macrophage study and a gene expression study). Sound study design minimises the possibility of bias being introduced into experiments and leads to higher quality research with more reproducible results.

Evidence-informed recommendations to reduce dissemination bias in clinical research: conclusions from the OPEN (Overcome failure to Publish nEgative fiNdings) project based on an international consensus meeting

BACKGROUND

Dissemination bias in clinical research severely impedes informed decision-making not only for healthcare professionals and patients, but also for funders, research ethics committees, regulatory bodies and other stakeholder groups that make health-related decisions. Decisions based on incomplete and biased evidence cannot only harm people, but may also have huge financial implications by wasting resources on ineffective or harmful diagnostic and therapeutic measures, and unnecessary research. Owing to involvement of multiple stakeholders, it remains easy for any single group to assign responsibility for resolving the problem to others.

OBJECTIVE

To develop evidence-informed general and targeted recommendations addressing the various stakeholders involved in knowledge generation and dissemination to help overcome the problem of dissemination bias on the basis of previously collated evidence.

METHODS

Based on findings from systematic reviews, document analyses and surveys, we developed general and targeted draft recommendations. During a 2-day workshop in summer 2013, these draft recommendations were discussed with external experts and key stakeholders, and refined following a rigorous and transparent methodological approach.

RESULTS

Four general, overarching recommendations applicable to all or most stakeholder groups were formulated, addressing (1) awareness raising, (2) implementation of targeted recommendations, (3) trial registration and results posting, and (4) systematic approaches to evidence synthesis. These general recommendations are complemented and specified by 47 targeted recommendations tailored towards funding agencies, pharmaceutical and device companies, research institutions, researchers (systematic reviewers and trialists), research ethics committees, trial registries, journal editors and publishers, regulatory agencies, benefit (health technology) assessment institutions and legislators.

CONCLUSIONS

Despite various recent examples of dissemination bias and several initiatives to reduce it, the problem of dissemination bias has not been resolved. Tailored recommendations based on a comprehensive approach will hopefully help increase transparency in biomedical research by overcoming the failure to disseminate negative findings.

Recent meta-analyses neglect previous systematic reviews and meta-analyses about the same topic: a systematic examination

BACKGROUND

As the number of systematic reviews is growing rapidly, we systematically investigate whether meta-analyses published in leading medical journals present an outline of available evidence by referring to previous meta-analyses and systematic reviews.

METHODS

We searched PubMed for recent meta-analyses of pharmacological treatments published in high impact factor journals. Previous systematic reviews and meta-analyses were identified with electronic searches of keywords and by searching reference sections. We analyzed the number of meta-analyses and systematic reviews that were cited, described and discussed in each recent meta-analysis. Moreover, we investigated publication characteristics that potentially influence the referencing practices.

RESULTS

We identified 52 recent meta-analyses and 242 previous meta-analyses on the same topics. Of these, 66% of identified previous meta-analyses were cited, 36% described, and only 20% discussed by recent meta-analyses. The probability of citing a previous meta-analysis was positively associated with its publication in a journal with a higher impact factor (odds ratio, 1.49; 95% confidence interval, 1.06 to 2.10) and more recent publication year (odds ratio, 1.19; 95% confidence interval 1.03 to 1.37). Additionally, the probability of a previous study being described by the recent meta-analysis was inversely associated with the concordance of results (odds ratio, 0.38; 95% confidence interval, 0.17 to 0.88), and the probability of being discussed was increased for previous studies that employed meta-analytic methods (odds ratio, 32.36; 95% confidence interval, 2.00 to 522.85).

CONCLUSIONS

Meta-analyses on pharmacological treatments do not consistently refer to and discuss findings of previous meta-analyses on the same topic. Such neglect can lead to research waste and be confusing for readers. Journals should make the discussion of related meta-analyses mandatory.

Avoidable waste of research related to inadequate methods in clinical trials

OBJECTIVE

To assess the waste of research related to inadequate methods in trials included in Cochrane reviews and to examine to what extent this waste could be avoided. A secondary objective was to perform a simulation study to re-estimate this avoidable waste if all trials were adequately reported.

DESIGN

Methodological review and simulation study.

DATA SOURCES

Trials included in the meta-analysis of the primary outcome of Cochrane reviews published between April 2012 and March 2013.

DATA EXTRACTION AND SYNTHESIS

We collected the risk of bias assessment made by the review authors for each trial. For a random sample of 200 trials with at least one domain at high risk of bias, we re-assessed risk of bias and identified all related methodological problems. For each problem, possible adjustments were proposed that were then validated by an expert panel also evaluating their feasibility (easy or not) and cost. Avoidable waste was defined as trials with at least one domain at high risk of bias for which easy adjustments with no or minor cost could change all domains to low risk. In the simulation study, after extrapolating our re-assessment of risk of bias to all trials, we considered each domain rated as unclear risk of bias as missing data and used multiple imputations to determine whether they were at high or low risk.

RESULTS

Of 1286 trials from 205 meta-analyses, 556 (43%) had at least one domain at high risk of bias. Among the sample of 200 of these trials, 142 were confirmed as high risk; in these, we identified 25 types of methodological problem. Adjustments were possible in 136 trials (96%). Easy adjustments with no or minor cost could be applied in 71 trials (50%), resulting in 17 trials (12%) changing to low risk for all domains. So the avoidable waste represented 12% (95% CI 7% to 18%) of trials with at least one domain at high risk. After correcting for incomplete reporting, avoidable waste due to inadequate methods was estimated at 42% (95% CI 36% to 49%).

CONCLUSIONS

An important burden of wasted research is related to inadequate methods. This waste could be partly avoided by simple and inexpensive adjustments.

Use of guidelines to improve the quality and transparency of reporting oral health research

OBJECTIVE

The use of reporting guideline is directed at enhancing the completeness and transparency of biomedical publications. The aims of this paper are to present some of the key initiatives and guidelines providing indications and directions on the use of specific tools in oral health research.

METHODS

The EQUATOR Network and five established guidelines (CONSORT, STROBE, PRISMA, CARE and SPIRIT) are introduced.

RESULTS

Five guidelines are presented covering reporting of case reports, non-randomized studies, randomized controlled trials and systematic reviews. The importance of adherence to these guidelines by oral health researchers is emphasized.

CONCLUSIONS

Endorsement and robust implementation of reporting guidelines will translate into improved and more complete reporting in health research. Moreover, by ingraining the use of guidelines, it may be possible to indirectly improve the methodological quality of clinical studies. Active implementation strategies to encourage adherence to these guidelines among researchers, reviewers, editors and publishers may be an important facet in the advancement of knowledge in dentistry.

CLINICAL SIGNIFICANCE

Inadequate reporting of research can lead to wasted research resources and risks publication of inaccurate or misleading findings with implications on healthcare decisions. Familiarity and diligent compliance with methodological and reporting guidelines are therefore essential to maximize the yield from dental research.

Reporting on interventions: issues and guidelines for rehabilitation researchers

Observers commonly note the poor reporting of research, including rehabilitation research. The Consolidated Standards of Reporting Trials (CONSORT) checklist (supplemented by the CONSORT extension for nonpharmacologic interventions) has been published for improving the reporting of intervention research. However, the items on these checklists are considered to be inadequate to guide authors as to which information to include when reporting on the intervention(s) studied, and the Template for Intervention Description and Replication, Journal Article Reporting Standards, and the checklist of the Western Journal of Nursing Research are recommended to rehabilitation researchers. The Rehabilitation Treatment Taxonomy framework is recommended as a conceptual scheme to assist authors in thinking through the linkages between intervention ingredients, targets of treatment, and the mechanisms of action linking these 2 areas. Recommendations are made for prospective authors and journal editors who desire to see improved reporting of rehabilitation interventions.

Literature survey of high-impact journals revealed reporting weaknesses in abstracts of diagnostic accuracy studies

OBJECTIVES

Informative journal abstracts are crucial for the identification and initial appraisal of studies. We aimed to evaluate the informativeness of abstracts of diagnostic accuracy studies.

STUDY DESIGN AND SETTING

PubMed was searched for reports of studies that had evaluated the diagnostic accuracy of a test against a clinical reference standard, published in 12 high-impact journals in 2012. Two reviewers independently evaluated the information contained in included abstracts using 21 items deemed important based on published guidance for adequate reporting and study quality assessment.

RESULTS

We included 103 abstracts. Crucial information on study population, setting, patient sampling, and blinding as well as confidence intervals around accuracy estimates were reported in <50% of the abstracts. The mean number of reported items per abstract was 10.1 of 21 (standard deviation 2.2). The mean number of reported items was significantly lower for multiple-gate (case-control type) studies, in reports in specialty journals, and for studies with smaller sample sizes and lower abstract word counts. No significant differences were found between studies evaluating different types of tests.

CONCLUSION

Many abstracts of diagnostic accuracy study reports in high-impact journals are insufficiently informative. Developing guidelines for such abstracts could help the transparency and completeness of reporting.

Guidelines for translational research in heart failure

Heart failure (HF) remains a major cause of death and hospitalization worldwide. Despite medical advances, the prognosis of HF remains poor and new therapeutic approaches are urgently needed. The development of new therapies for HF is hindered by inappropriate or incomplete preclinical studies. In these guidelines, we present a number of recommendations to enhance similarity between HF animal models and the human condition in order to reduce the chances of failure in subsequent clinical trials. We propose different approaches to address safety as well as efficacy of new therapeutic products. We also propose that good practice rules are followed from the outset so that the chances of eventual approval by regulatory agencies increase. We hope that these guidelines will help improve the translation of results from animal models to humans and thereby contribute to more successful clinical trials and development of new therapies for HF.

Better duplicate detection for systematic reviewers: evaluation of Systematic Review Assistant-Deduplication Module

BACKGROUND

A major problem arising from searching across bibliographic databases is the retrieval of duplicate citations. Removing such duplicates is an essential task to ensure systematic reviewers do not waste time screening the same citation multiple times. Although reference management software use algorithms to remove duplicate records, this is only partially successful and necessitates removing the remaining duplicates manually. This time-consuming task leads to wasted resources. We sought to evaluate the effectiveness of a newly developed deduplication program against EndNote.

METHODS

A literature search of 1,988 citations was manually inspected and duplicate citations identified and coded to create a benchmark dataset. The Systematic Review Assistant-Deduplication Module (SRA-DM) was iteratively developed and tested using the benchmark dataset and compared with EndNote's default one step auto-deduplication process matching on ('author', 'year', 'title'). The accuracy of deduplication was reported by calculating the sensitivity and specificity. Further validation tests, with three additional benchmarked literature searches comprising a total of 4,563 citations were performed to determine the reliability of the SRA-DM algorithm.

RESULTS

The sensitivity (84%) and specificity (100%) of the SRA-DM was superior to EndNote (sensitivity 51%, specificity 99.83%). Validation testing on three additional biomedical literature searches demonstrated that SRA-DM consistently achieved higher sensitivity than EndNote (90% vs 63%), (84% vs 73%) and (84% vs 64%). Furthermore, the specificity of SRA-DM was 100%, whereas the specificity of EndNote was imperfect (average 99.75%) with some unique records wrongly assigned as duplicates. Overall, there was a 42.86% increase in the number of duplicates records detected with SRA-DM compared with EndNote auto-deduplication.

CONCLUSIONS

The Systematic Review Assistant-Deduplication Module offers users a reliable program to remove duplicate records with greater sensitivity and specificity than EndNote. This application will save researchers and information specialists time and avoid research waste. The deduplication program is freely available online.

Reproducibility in Science

Medical and scientific advances are predicated on new knowledge that is robust and reliable and that serves as a solid foundation on which further advances can be built. In biomedical research, we are in the midst of a revolution with the generation of new data and scientific publications at a previously unprecedented rate. However, unfortunately, there is compelling evidence that the majority of these discoveries will not stand the test of time. To a large extent, this reproducibility crisis in basic and preclinical research may be as a result of failure to adhere to good scientific practice and the desperation to publish or perish. This is a multifaceted, multistakeholder problem. No single party is solely responsible, and no single solution will suffice. Here we review the reproducibility problems in basic and preclinical biomedical research, highlight some of the complexities, and discuss potential solutions that may help improve research quality and reproducibility.

Beyond limitations: Improving how we handle uncertainty in health professions education research

The researchers' assumptions invariably influence research outcomes. This is true for both qualitative and quantitative studies. Assumptions or choices regarding underlying theories, causal relations, study setting and population, sampling strategies, participant non-response, data collection, data analysis, and researchers' perceptions and interpretations of results are among factors that can induce uncertainty in research findings. Researchers tend to treat these factors as potential study limitations, but how they may impact research findings is rarely explicated and, therefore, mostly unknown. In this article, we approach uncertainty as unavoidable in research and argue that communicating about uncertainty can inform researchers, policy makers and practitioners about the validity and applicability of the study findings for their interests and contexts. We illustrate approaches to address, interpret, and explicate uncertainty in medical education research in both qualitative and quantitative paradigms. Across research paradigms, we call on researchers to consider the uncertainty in their research findings, employ appropriate methods to explore its extent and effects in their work, and communicate it explicitly in their research papers. This will help to advance our understanding of the nature and implications of the emerging knowledge in our field.

Improving Concordance in Environmental Epidemiology: A Three-Part Proposal

In observational research, evidence is usually derived from multiple studies, and any single result is rarely considered sufficient for public health decision making. Despite more than five decades of research and thousands of studies published, the ability to draw robust conclusions regarding the presence or absence of causal links between specific environmental exposures and human health remains limited. To develop policies that are protective of public health and can withstand scrutiny, agencies need to rely on investigations of satisfactory quality that follow sufficiently concordant protocols in terms of exposure assessment, outcome ascertainment, data analysis, and reporting of results. Absent such concordance, the ability of environmental epidemiology studies to inform decision making is greatly diminished. Systems and tools are proposed here to improve concordance among environmental epidemiology studies. Specifically, working systems in place in other fields of research are critically examined and used as guidelines to develop analogous policies and procedures for environmental epidemiology. A three-part path forward toward more concordant, transparent, and readily accessible environmental epidemiology evidence that parallels ongoing efforts in medical research is proposed. The three parts address methods for improving quality and accessibility of systematic reviews, access to information on ongoing and completed studies, and principles for reporting. The goals are to increase the value of epidemiological research in public health decision making and to stimulate discussions around solutions proposed herein.

[Potentials for research and innovations in allied health professions in Germany]

In order to address the increasing complexity and continuously changing needs and demands in the German healthcare system, there is a need to strengthen knowledge translation, evidence-based practice and the conduct of clinical trials in the field of allied health professions. An interdisciplinary working group representing the fields of nursing, midwifery, physiotherapy, occupational therapy and speech therapy developed a guide and a concept for seminars to provide potential analyses for research and innovations in the allied health professions in Germany. These potential analyses compare the current state of health care delivery for specific health problems and the corpus of evidence for the effectiveness of related interventions. Thus innovations can be identified which might improve client-centred healthcare in Germany. The introductory paper briefly reports the activities and results of the working group, describes the international context of transferring research into practice and outlines possibilities for the future development of coordinated research strategies in Germany. The following papers consist of five potential analyses: (1) Advanced Practice Nursing (APN) in long-term care; (2) giving birth in an upright position; (3) treadmill training for patients with Parkinson's disease; (4) training of everyday activities after stroke; and (5) communication training for patients with aphasia.

Stem cells in animal asthma models: a systematic review

BACKGROUND AIMS

Asthma control frequently falls short of the goals set in international guidelines. Treatment options for patients with poorly controlled asthma despite inhaled corticosteroids and long-acting β-agonists are limited, and new therapeutic options are needed. Stem cell therapy is promising for a variety of disorders but there has been no human clinical trial of stem cell therapy for asthma. We aimed to systematically review the literature regarding the potential benefits of stem cell therapy in animal models of asthma to determine whether a human trial is warranted.

METHODS

The MEDLINE and Embase databases were searched for original studies of stem cell therapy in animal asthma models.

RESULTS

Nineteen studies were selected. They were found to be heterogeneous in their design. Mesenchymal stromal cells were used before sensitization with an allergen, before challenge with the allergen and after challenge, most frequently with ovalbumin, and mainly in BALB/c mice. Stem cell therapy resulted in a reduction of bronchoalveolar lavage fluid inflammation and eosinophilia as well as Th2 cytokines such as interleukin-4 and interleukin-5. Improvement in histopathology such as peribronchial and perivascular inflammation, epithelial thickness, goblet cell hyperplasia and smooth muscle layer thickening was universal. Several studies showed a reduction in airway hyper-responsiveness.

CONCLUSIONS

Stem cell therapy decreases eosinophilic and Th2 inflammation and is effective in several phases of the allergic response in animal asthma models. Further study is warranted, up to human clinical trials.

A stronger post-publication culture is needed for better science

Hilda Bastian considers post-publication commenting and the cultural changes that are needed to better capture this intellectual effort. Please see later in the article for the Editors' Summary.

The reproducibility of biomedical research: Sleepers awake!

There is increasing concern about the reliability of biomedical research, with recent articles suggesting that up to 85% of research funding is wasted. This article argues that an important reason for this is the inappropriate use of molecular techniques, particularly in the field of RNA biomarkers, coupled with a tendency to exaggerate the importance of research findings.

Internet-based intervention for smoking cessation (StopAdvisor) in people with low and high socioeconomic status: a randomised controlled trial

BACKGROUND

Internet-based interventions for smoking cessation could help millions of people stop smoking at very low unit costs; however, long-term biochemically verified evidence is scarce and such interventions might be less effective for smokers with low socioeconomic status than for those with high status because of lower online literacy to engage with websites. We aimed to assess a new interactive internet-based intervention (StopAdvisor) for smoking cessation that was designed with particular attention directed to people with low socioeconomic status.

METHODS

We did this online randomised controlled trial between Dec 6, 2011, and Oct 11, 2013, in the UK. Participants aged 18 years and older who smoked every day were randomly assigned (1:1) to receive treatment with StopAdvisor or an information-only website. Randomisation was automated with an unseen random number function embedded in the website to establish which treatment was revealed after the online baseline assessment. Recruitment continued until the required sample size had been achieved from both high and low socioeconomic status subpopulations. Participants, and researchers who obtained data and did laboratory analyses, were masked to treatment allocation. The primary outcome was 6 month sustained, biochemically verified abstinence. The main secondary outcome was 6 month, 7 day biochemically verified point prevalence. Analysis was by intention to treat. Homogeneity of intervention effect across the socioeconomic subsamples was first assessed to establish whether overall or separate subsample analyses were appropriate. The study is registered as an International Standard Randomised Controlled Trial, number ISRCTN99820519.

FINDINGS

We randomly assigned 4613 participants to the StopAdvisor group (n=2321) or the control group (n=2292); 2142 participants were of low socioeconomic status and 2471 participants were of high status. The overall rate of smoking cessation was similar between participants in the StopAdvisor and control groups for the primary (237 [10%] vs 220 [10%] participants; relative risk [RR] 1·06, 95% CI 0·89-1·27; p=0·49) and the secondary (358 [15%] vs 332 [15%] participants; 1·06, 0·93-1·22; p=0·37) outcomes; however, the intervention effect differed across socioeconomic status subsamples (1·44, 0·99-2·09; p=0·0562 and 1·37, 1·02-1·84; p=0·0360, respectively). StopAdvisor helped participants with low socioeconomic status stop smoking compared with the information-only website (primary outcome: 90 [8%] of 1088 vs 64 [6%] of 1054 participants; RR 1·36, 95% CI 1·00-1·86; p=0·0499; secondary outcome: 136 [13%] vs 100 [10%] participants; 1·32, 1·03-1·68, p=0·0267), but did not improve cessation rates in those with high socioeconomic status (147 [12%] of 1233 vs 156 [13%] of 1238 participants; 0·95, 0·77-1·17; p=0·61 and 222 [18%] vs 232 [19%] participants; 0·96, 0·81-1·13, p=0·64, respectively).

INTERPRETATION

StopAdvisor was more effective than an information-only website in smokers of low, but not high, socioeconomic status. StopAdvisor could be implemented easily and made freely available, which would probably improve the success rates of smokers with low socioeconomic status who are seeking online support.

FUNDING

National Prevention Research Initiative.

Impact of the transparent reporting of evaluations with nonrandomized designs reporting guideline: ten years on

OBJECTIVES

We assessed how the Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) reporting guideline was used by authors and journal editors in journals' instructions to authors. We also evaluated its impact on reporting completeness and study quality.

METHODS

We extracted data from publications that cited TREND on how TREND was used in those reports; we also extracted information on journals' instructions to authors. We then undertook a case-control study of relevant publications to evaluate the impact of using TREND.

RESULTS

Between 2004 and 2013, TREND was cited 412 times, but it was only evidently applied to study reports 47 times. TREND was specifically mentioned 14 times in the sample of 61 instructions to authors. Some evidence suggested that use of TREND was associated with more comprehensive reporting and higher study quality ratings.

CONCLUSIONS

TREND appeared to be underutilized by authors and journal editors despite its potential application and benefits. We found evidence that suggested that using TREND could contribute to more transparent and complete study reports. Even when authors reported using TREND, reporting completeness was still suboptimal.

[Measurement and assessment of study quality and reporting quality]

Clinical trials are undertaken to evaluate the effectiveness and safety of clinical interventions or to investigate other unclarified (e.g. diagnostic) research questions. But not every clinical study is optimally planned, conducted and analysed and is therefore exposed to risk of systematic errors which can distort (bias) the study results. Clinical studies should be conducted in accordance with specific quality criteria in order to reduce these risks. To estimate the methodological quality of studies instruments such as the Cochrane Risk of Bias tool were developed. But it can only be appraised what is reported. Often important information on the conduct and analysis of a study is missing in research articles, and in most cases detailed study protocols are not freely available. Reporting guidelines available for different study types can help improve the transparency and completeness of reports of clinical studies and thus the reporting quality.

Reproducibility of results in preclinical studies: a perspective from the bone field

The biomedical research enterprise-and the public support for it-is predicated on the belief that discoveries and the conclusions drawn from them can be trusted to build a body of knowledge which will be used to improve human health. As in all other areas of scientific inquiry, knowledge and understanding grow by layering new discoveries upon earlier ones. The process self-corrects and distills knowledge by discarding false ideas and unsubstantiated claims. Although self-correction is inexorable in the long-term, in recent years biomedical scientists and the public alike have become alarmed and deeply troubled by the fact that many published results cannot be reproduced. The chorus of concern reached a high pitch with a recent commentary from the NIH Director, Francis S. Collins, and Principal Deputy Director, Lawrence A. Tabak, and their announcement of specific plans to enhance reproducibility of preclinical research that relies on animal models. In this invited perspective, we highlight the magnitude of the problem across biomedical fields and address the relevance of these concerns to the field of bone and mineral metabolism. We also suggest how our specialty journals, our scientific organizations, and our community of bone and mineral researchers can help to overcome this troubling trend.

How to make more published research true

In a 2005 paper that has been accessed more than a million times, John Ioannidis explained why most published research findings were false. Here he revisits the topic, this time to address how to improve matters. Please see later in the article for the Editors' Summary.

Impact of sending email reminders of the legal requirement for posting results on ClinicalTrials.gov: cohort embedded pragmatic randomized controlled trial

OBJECTIVE

To evaluate the impact of sending an email to responsible parties of completed trials that do not comply with the Food and Drug Administration Amendments Act 801 legislation, to remind them of the legal requirement to post results.

DESIGN

Cohort embedded pragmatic randomized controlled trial.

SETTING

Trials registered on ClinicalTrials.gov.

PARTICIPANTS

190 out of 379 trials randomly selected by computer generated randomization list to receive the intervention (personalized emails structured as a survey and sent by one of us to responsible parties of the trials, indirectly reminding them of the legal requirement and potential penalties for non-compliance).

MAIN OUTCOME MEASURES

The primary outcome was the proportion of results posted on ClinicalTrials.gov at three months. The secondary outcome was the proportion posted at six months. In a second step, two assessors blinded to the intervention group collected the date of the first results being received on ClinicalTrials.gov. A post hoc sensitivity analysis excluding trials wrongly included was performed.

RESULTS

Among 379 trials included, 190 were randomized to receive the email intervention. The rate of posting of results did not differ at three months between trials with or without the intervention: 36/190 (19%) v 24/189 (13%), respectively (relative risk 1.5, 95% confidence interval 0.9 to 2.4, P=0.096) but did at six months: 46/190 (24%) v 27/189 (14%), 1.7, 1.1 to 2.6, P=0.014. In the sensitivity analysis, which excluded 48/379 trials (13%), 26/190 (14%) and 22/189 (12%), respectively, results were significant at three months (relative risk 5.1, 1.1 to 22.9, P=0.02) and at six months (4.1, 1.3 to 10.6, P=0.001).

CONCLUSIONS

Sending email reminders about the FDA's legal requirement to post results at ClinicalTrials.gov improved significantly the posting rate at six months but not at three months.Trial registration ClinicalTrials.gov NCT01658254.

How good is "evidence" from clinical studies of drug effects and why might such evidence fail in the prediction of the clinical utility of drugs?

Promising evidence from clinical studies of drug effects does not always translate to improvements in patient outcomes. In this review, we discuss why early evidence is often ill suited to the task of predicting the clinical utility of drugs. The current gap between initially described drug effects and their subsequent clinical utility results from deficits in the design, conduct, analysis, reporting, and synthesis of clinical studies-often creating conditions that generate favorable, but ultimately incorrect, conclusions regarding drug effects. There are potential solutions that could improve the relevance of clinical evidence in predicting the real-world effectiveness of drugs. What is needed is a new emphasis on clinical utility, with nonconflicted entities playing a greater role in the generation, synthesis, and interpretation of clinical evidence. Clinical studies should adopt strong design features, reflect clinical practice, and evaluate outcomes and comparisons that are meaningful to patients. Transformative changes to the research agenda may generate more meaningful and accurate evidence on drug effects to guide clinical decision making.