Citation indexes do not reflect methodological quality in lung cancer randomised trials

The reporting quality of N-of-1 trials and protocols still needs improvement

OBJECTIVE

To evaluate the reporting quality of single-patient (N-of-1) trials and protocols based on the CONSORT Extension for N-of-1 trials (CENT) statement and the standard protocol items: recommendations for interventional trials (SPIRIT) extension and elaboration for N-of-1 trials (SPENT) checklist to examine the factors that influenced reporting quality.

METHODS

Four electronic databases were searched to identify N-of-1 trials and protocols from 2015 to 2020. Quality was assessed by two reviewers. We calculated the overall scores based on binary responses in which "Yes" was scored as 1 (if the item was fully reported), and "No" was scored as 0 (if the item was not clearly reported or not definitely stated).

RESULTS

A total of 78 publications (55 N-of-1 trials and 23 protocols) were identified. The mean reporting score (SD) of the N-of-1 trials and protocols were 29.24 (0.89) and 29.61 (1.83), respectively. For the items related to outcomes, sample size, allocation concealment protocol, and informed consent materials, the reporting quality was low. Our results showed that the year of publication (t = -0.793, p = 0.872 for the trials and t = 1.352, p = 0.623 for the protocols) and the impact factor of the journal (t = 1.416, p = 0.619 for the trials and t = 0.359, p = 0.667 for the protocols) were not factors associated with better reporting quality.

CONCLUSION

With the publication of the CENT 2015 statement and the SPENT 2019 checklist, authors should adhere to the relevant reporting guidelines and improve the reporting quality of N-of-1 trials and protocols.

Determinants of societal and academic recognition: Evidence from randomised controlled trials

Given the increasing importance of recognition in academia and the vital role of randomised controlled trials (RCTs) in medical research and clinical decisions, this study verifies how RCTs’ academic and societal impacts are affected by visibility factors, subjects and methodological validity. This study concentrated on a sample of 446 RCTs indexed in Scopus and evaluated by Cochrane reviewers in terms of their methodological validity. The altmetrics, bibliometric and bibliographical information were extracted from Altmetric.com and Scopus, and the contributing countries’ development ranks were obtained from the United Nations Development report. The linear regression analyses revealed that citations and altmetrics depend on some subjects. They are also affected by publication year and journals’ previous reputation. Citations are also affected by keyword counts and reference counts. Keyword counts and contributing countries’ developmental rank also predict the tweet counts. While none of the methodological validity dimensions were found to predict citations, ‘Incomplete Outcome Data’ and ‘Random Sequence Generation’ significantly, though slightly, affect Mendeley Readership and tweets, respectively. By confirming the dependence of RCTs’ recognition on some methodological validity features and attention-inducing characteristics, the study provides further evidence on the interaction of quality and visibility dynamisms in the recognition network and the complementary role of societal mentions for academic citation.

Are methodological quality and completeness of reporting associated with citation-based measures of publication impact? A secondary analysis of a systematic review of dementia biomarker studies

OBJECTIVE

To determine whether methodological and reporting quality are associated with surrogate measures of publication impact in the field of dementia biomarker studies.

METHODS

We assessed dementia biomarker studies included in a previous systematic review in terms of methodological and reporting quality using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) and Standards for Reporting of Diagnostic Accuracy (STARD), respectively. We extracted additional study and journal-related data from each publication to account for factors shown to be associated with impact in previous research. We explored associations between potential determinants and measures of publication impact in univariable and stepwise multivariable linear regression analyses.

OUTCOME MEASURES

We aimed to collect data on four measures of publication impact: two traditional measures-average number of citations per year and 5-year impact factor of the publishing journal and two alternative measures-the Altmetric Attention Score and counts of electronic downloads.

RESULTS

The systematic review included 142 studies. Due to limited data, Altmetric Attention Scores and electronic downloads were excluded from the analysis, leaving traditional metrics as the only analysed outcome measures. We found no relationship between QUADAS and traditional metrics. Citation rates were independently associated with 5-year journal impact factor (β=0.42; p<0.001), journal subject area (β=0.39; p<0.001), number of years since publication (β=-0.29; p<0.001) and STARD (β=0.13; p<0.05). Independent determinants of 5-year journal impact factor were citation rates (β=0.45; p<0.001), statement on conflict of interest (β=0.22; p<0.01) and baseline sample size (β=0.15; p<0.05).

CONCLUSIONS

Citation rates and 5-year journal impact factor appear to measure different dimensions of impact. Citation rates were weakly associated with completeness of reporting, while neither traditional metric was related to methodological rigour. Our results suggest that high publication usage and journal outlet is not a guarantee of quality and readers should critically appraise all papers regardless of presumed impact.

The Journal of the Faculty of Medicine implements a transition process to publish articles in English

La Revista de la Facultad de Medicina, con una historia de publicaciones que inició en 1932, ha experimentado interesantes cambios en la última década, logrando una mayor visibilidad e impacto en el ámbito nacional e internacional. Así mismo, ha logrado mantener la periodicidad de su publicación e ingresar a índices bibliográficos citacionales importantes. En la actualidad, se reconoce que las revistas biomédicas publicadas en inglés adquieren con frecuencia un mayor factor de impacto y se posicionan en mejores puestos en los rankings de publicaciones científicas en español, brindando así prestigio a las instituciones que las soportan (1). Dado este panorama, la revista tiene como objetivo a mediano plazo realizar sus publicaciones solo en inglés, lo que conlleva a una serie de cambios estructurales en la gestión editorial y en el soporte académico administrativo suficiente para hacerlo posible, de tal manera que se mantenga el número de artículos científicos publicados de excelente calidad, pero con una mayor visibilidad internacional y, por ende, posibilidad de citación de sus resultados. 

Factors associated with online media attention to research: a cohort study of articles evaluating cancer treatments

Background

New metrics have been developed to assess the impact of research and provide an indication of online media attention and data dissemination. We aimed to describe online media attention of articles evaluating cancer treatments and identify the factors associated with high online media attention.

Methods

We systematically searched MEDLINE via PubMed on March 1, 2015 for articles published during the first 6 months of 2014 in oncology and medical journals with a diverse range of impact factors, from 3.9 to 54.4, and selected a sample of articles evaluating a cancer treatment regardless of study design. Altmetric Explorer was used to identify online media attention of selected articles. The primary outcome was media attention an article received online as measured by Altmetric score (i.e., number of mentions in online news outlets, science blogs and social media). Regression analysis was performed to investigate the factors associated with high media attention, and regression coefficients represent the logarithm of ratio of mean (RoM) values of Altmetric score per unit change in the covariate.

Results

Among 792 articles, 218 (27.5%) received no online media attention (Altmetric score = 0). The median [Q1–Q3] Altmetric score was 2.0 [0.0–8.0], range 0.0–428.0. On multivariate analysis, factors associated with high Altmetric score were presence of a press release (RoM = 10.14, 95%CI [4.91–20.96]), open access to the article (RoM = 1.48, 95%CI [1.02–2.16]), and journal impact factor (RoM = 1.10, 95%CI [1.07–1.12]. As compared with observational studies, systematic reviews were not associated with high Altmetric score (RoM = 1.46, 95%CI [0.74–2.86]; P = 0.27), nor were RCTs (RoM = 0.65, 95%CI [0.41–1.02]; P = 0.059) and phase I/II non-RCTs (RoM = 0.58, 95%CI [0.33–1.05]; P = 0.07). The articles with abstract conclusions favouring study treatments were not associated with high Altmetric score (RoM = 0.97, 95%CI [0.60–1.58]; P = 0.91).

Conclusions

Most important factors associated with high online media attention were the presence of a press release and the journal impact factor. There was no evidence that study design with high level of evidence and type of abstract conclusion were associated with high online media attention.

Electronic supplementary material

The online version of this article (doi:10.1186/s41073-017-0033-z) contains supplementary material, which is available to authorized users.

Control and eradication of tuberculosis in cattle: a systematic review of economic evidence

Bovine tuberculosis (TB) is a disease of zoonotic importance for which control and eradication programmes have been carried out in many countries for decades. While the impact of these programmes on public health is still uncertain, the impact on trade is significant because of movement restrictions for animals, costs of testing and culling. The objective of this systematic review was to provide a contribution to the general debate over costs against benefits for the control and eradication of bovine TB in cattle. The search strategy was performed on four electronic databases following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. The selection process, data abstraction and quality appraisal were carried out according to the Cochrane Collaboration guidelines. The search identified 66 articles out of which eight fulfilled the inclusion criteria. The evidence gathered in this review by combining the conclusions of the most methodologically sound articles supports the idea that, when multiple cost and benefit components are taken into account, efforts to control or eradicate bovine TB may be effective in reducing disease prevalence, economically viable and worth doing.

Lung injury and lung cancer caused by cigarette smoke-induced oxidative stress: Molecular mechanisms and therapeutic opportunities involving the ceramide-generating machinery and epidermal growth factor receptor

Chronic obstructive pulmonary disease (COPD) and lung cancer are frequently caused by tobacco smoking. However, these diseases present opposite phenotypes involving redox signaling at the cellular level. While COPD is characterized by excessive airway epithelial cell death and lung injury, lung cancer is caused by uncontrolled epithelial cell proliferation. Notably, epidemiological studies have demonstrated that lung cancer incidence is significantly higher in patients who have preexisting emphysema/lung injury. However, the molecular link and common cell signaling events underlying lung injury diseases and lung cancer are poorly understood. This review focuses on studies of molecular mechanism(s) underlying smoking-related lung injury (COPD) and lung cancer. Specifically, the role of the ceramide-generating machinery during cigarette smoke-induced oxidative stress leading to both apoptosis and proliferation of lung epithelial cells is emphasized. Over recent years, it has been established that ceramide is a sphingolipid playing a major role in lung epithelia structure/function leading to lung injury in chronic pulmonary diseases. However, new and unexpected findings draw attention to its potential role in lung development, cell proliferation, and tumorigenesis. To address this dichotomy in detail, evidence is presented regarding several protein targets, including Src, p38 mitogen-activated protein kinase, and neutral sphingomyelinase 2, the major sphingomyelinase that controls ceramide generation during oxidative stress. Furthermore, their roles are presented not only in apoptosis and lung injury but also in enhancing cell proliferation, lung cancer development, and resistance to epidermal growth factor receptor-targeted therapy for treating lung cancer.

Assessing the scientific research productivity of a leading toxicology journal: A case study of Human & Experimental Toxicology from 2003 to 2012

BACKGROUND

Bibliometric studies are increasingly being used for research assessments. Bibliometric indicators involve the application of statistical methods to scientific publications to obtain the bibliographics for each journal. The main objective of this study was to conduct a bibliometric evaluation of Human & Experimental Toxicology retrieved from the Scopus database.

METHODS

This study obtained data from Scopus published from 1 January 2003 till 31 December 2012. The keywords entered in Scopus to accomplish the objective of this study were 'Human', 'Experimental' and 'Toxicology' as 'Source Title'. Research productivity was evaluated based on a methodology developed and used in other bibliometric studies by analysing (a) total and trends in Human & Experimental Toxicology contributions in research between 2003 and 2012; (b) Human & Experimental Toxicology authorship patterns and productivity; (c) collaboration patterns; and (d) the citations received by the publications.

RESULTS

There were 1229 research articles published in Human & Experimental Toxicology. Of the articles included, 947 (77.1%) were original articles and 104 (8.5%) were review articles. The Hirsch-index of the retrieved documents was 35. The largest number of publications in Human & Experimental Toxicology was from the United States (19.6%), followed by India (12.8%) and Turkey (10.9%). The total number of citations was 9119, with a median (interquartile range) of 3 (1-9) in 6797 documents. The highest median (interquartile range) number of citations was 8 (2.7-12.7) for France, followed by 7.5 (2-22.5) for Iran and 6 (3-13.5) for the United Kingdom. The country most often citing articles that were published in Human & Experimental Toxicology was the United States, which made citations in 1508 documents, followed by India with citations in 792 documents.

CONCLUSION

The documents in Human & Experimental Toxicology focus principally on original data, with very few review articles. Review articles tend to have higher citation rates than original articles, and hence, the editors and authors of Human & Experimental Toxicology might usefully promote the submission of reviews in the future to improve the impact of the journal.

Best Evidence in Emergency Medicine (BEEM) rater scores correlate with publications' future citations

BACKGROUND

The "BEEM" (best evidence in emergency medicine) rater scale was created for emergency physicians (EPs) to evaluate the physician-derived clinical relevance score of recently published, emergency medicine (EM)-related studies. BEEM therefore is designed to help make EPs aware of studies most likely to confirm or change current clinical practice.

OBJECTIVES

The objective was to validate the BEEM rater score as a predictor of literature citation, using a bibliometric construct of clinical relevance to EM based on author-, document-, and journal-level measures (first and last author h-indices, number of authors including corporate and group authors, citations from date of publication to 2011, and journal impact factor scores) and study characteristics (design, category, and sample size).

METHODS

Each month from 2007 through 2012, approximately 200 EPs from around the world voluntarily reviewed the titles and conclusions of recently published EM-related studies identified by BEEM faculty via the McMaster Health Information Research Unit. Using the BEEM rater scale, a reliable seven-item instrument that evaluates the clinical relevance of studies, raters independently assigned BEEM scores to approximately 10 to 20 articles each month. Two investigators independently abstracted the bibliometric indices for these articles. A citation rate for each article was calculated by dividing the Thomson Reuters Web of Science (WoS) total citation count by the number of years in publication. BEEM rater scores were correlated with the citation rate using Spearman's rho. The performance of the BEEM rater score was assessed for each article using negative binomial regression with composite citation count as the criterion standard, while controlling for other independent bibliometric variables in three models.

RESULTS

The BEEM raters evaluated 605 articles with a mean (±SD) BEEM score of 3.84 (±0.7) and a median BEEM score of 3.85 (interquartile range = 3.38 to 4.30). Articles were primarily therapeutic (59%) and diagnostic (27%), with various designs, including 37% systematic reviews, 32% randomized controlled trials (RCTs), and 30% observational designs. The citation rate and BEEM rater score correlated positively (0.144), while the BEEM rater score and the Journal Citation Report (JCR) impact factor score were minimally correlated (0.053). In the first model, the BEEM rater score significantly predicted WoS citation rate (p < 0.0001) with an odds ratio (OR) of 1.24 (95% confidence interval [CI] = 1.106 to 1.402). In subsequent models adjusting for the JCR impact factor score, the h-indices of the first and last authors, number of authors, and study design, the BEEM rater score was not significant (p = 0.08).

CONCLUSIONS

To the best of our knowledge, the BEEM rater score is the only known measure of clinical relevance. It has a high interrater reliability and face validity and correlates with future citations. Future research should assess this instrument against alternative constructs of clinical relevance.

Citation of previous meta-analyses on the same topic: a clue to perpetuation of incorrect methods?

CLINICAL RELEVANCE

Systematic reviews and meta-analyses serve as a basis for decision-making and clinical practice guidelines and should be carried out using appropriate methodology to avoid incorrect inferences.

TOPIC

We describe the characteristics, statistical methods used for meta-analyses, and citation patterns of all 21 glaucoma systematic reviews we identified pertaining to the effectiveness of prostaglandin analog eye drops in treating primary open-angle glaucoma, published between December 2000 and February 2012.

METHODS

We abstracted data, assessed whether appropriate statistical methods were applied in meta-analyses, and examined citation patterns of included reviews.

RESULTS

We identified two forms of problematic statistical analyses in 9 of the 21 systematic reviews examined. Except in 1 case, none of the 9 reviews that used incorrect statistical methods cited a previously published review that used appropriate methods. Reviews that used incorrect methods were cited 2.6 times more often than reviews that used appropriate statistical methods. We speculate that by emulating the statistical methodology of previous systematic reviews, systematic review authors may have perpetuated incorrect approaches to meta-analysis.

CONCLUSIONS

The use of incorrect statistical methods, perhaps through emulating methods described in previous research, calls conclusions of systematic reviews into question and may lead to inappropriate patient care. We urge systematic review authors and journal editors to seek the advice of experienced statisticians before undertaking or accepting for publication a systematic review and meta-analysis.

FINANCIAL DISCLOSURE(S)

The author(s) have no proprietary or commercial interest in any materials discussed in this article.

Systematic evaluation of the methodology of randomized controlled trials of anticoagulation in patients with cancer

Background

Randomized controlled trials (RCTs) that are inappropriately designed or executed may provide biased findings and mislead clinical practice. In view of recent interest in the treatment and prevention of thrombotic complications in cancer patients we evaluated the characteristics, risk of bias and their time trends in RCTs of anticoagulation in patients with cancer.

Methods

We conducted a comprehensive search, including a search of four electronic databases (MEDLINE, EMBASE, ISI the Web of Science, and CENTRAL) up to February 2010. We included RCTs in which the intervention and/or comparison consisted of: vitamin K antagonists, unfractionated heparin (UFH), low molecular weight heparin (LMWH), direct thrombin inhibitors or fondaparinux. We performed descriptive analyses and assessed the association between the variables of interest and the year of publication.

Results

We included 67 RCTs with 24,071 participants. In twenty one trials (31%) DVT diagnosis was triggered by clinical suspicion; the remaining trials either screened for DVT or were unclear about their approach. 41 (61%), 22 (33%), and 11 (16%) trials respectively reported on major bleeding, minor bleeding, and thrombocytopenia. The percentages of trials satisfying risk of bias criteria were: adequate sequence generation (85%), adequate allocation concealment (61%), participants’ blinding (39%), data collectors’ blinding (44%), providers’ blinding (41%), outcome assessors’ blinding (75%), data analysts’ blinding (15%), intention to treat analysis (57%), no selective outcome reporting (12%), no stopping early for benefit (97%). The mean follow-up rate was 96%. Adequate allocation concealment and the reporting of intention to treat analysis were the only two quality criteria that improved over time.

Conclusions

Many RCTs of anticoagulation in patients with cancer appear to use insufficiently rigorous outcome assessment methods and to have deficiencies in key methodological features. It is not clear whether this reflects a problem in the design, conduct or the reporting of these trials, or both. Future trials should avoid the shortcomings described in this article.

Reference accuracy in the general surgery literature

BACKGROUND

Reference inaccuracy in scientific articles brings the scientific validity of the research into question and may create difficulty when accessing the cited background data. The objective of this study was to examine the reference accuracy in the general surgery literature and its correlation with the journal impact factor.

METHODS

Five general surgery journals were chosen with varying impact factors. From the year 2007, one issue was randomly chosen from each journal, and from each issue 180 citations were randomly chosen for review. Three investigators evaluated the chosen references for primary, citational, and quotational errors. The impact factor of each journal was compared to the percentage of errors detected.

RESULTS

The total number of errors per journal ranged from 31.3 to 39.3%, with a total of 35.4% of all citations reviewed containing some type of error. The most common error type detected was incorrect citation of the primary source supporting a statement, the incidence of which ranged from 13.8 to 25.2%, depending on the journal, and accounting for 53.6% of the total errors found. Citational errors, which included incorrect author names, pagination, dates, and issue and volume numbers, ranged from 1.8 to 18.1% and accounted for 20.4% of the total errors detected. Qualitative errors, which occurred when the author misquoted another author's written assertions or conclusions, ranged from 7.4 to 16.0% and accounted for 34.7% of the total errors detected. Quantitative errors (misquoted numerical data) ranged from 3.1 to 8.6% and accounted for 17.9% of the total errors detected. No association between impact factor and error rate was demonstrated.

CONCLUSIONS

Reference inaccuracy is common in the general surgery literature. The impact factor has no clear association with the error rate, demonstrating that journal quality does not necessarily correlate with reference quality. Further investigation into potential methods for improving reference accuracy in the general surgery literature is warranted.

What can we learn from Chinese randomized controlled trials? A systematic review and meta-analysis of Chinese venlafaxine studies

This systematic review evaluated Chinese trials examining the efficacy of venlafaxine in the treatment of depression. Chinese databases CNKI and VIP and western databases were searched for blinded randomized controlled trial publications comparing venlafaxine to other antidepressants or placebo (in English or Chinese). Trials had to establish diagnosis of depression according to the Chinese Classification of Mental Disorders, Diagnostic and Statistical Manual of Mental Disorders, or International Classification of Diseases. Studies were excluded if more than 20% of participants had a primary diagnosis of dysthymia or if more than 15% had a primary diagnosis of bipolar disorder. Effect sizes were calculated as Hedges' g for rating scale scores and Mantel-Haenszel risk ratios (MH RR) for response and remission data. Effect sizes were combined in a fixed-effects model. A total of 25 studies were included. Nine trials compared venlafaxine to selective serotonin reuptake inhibitor; placebo-controlled trials were lacking. Quality was at best modest, and all trials were underpowered. There were more responders (MH RR, 1.08; 95% confidence interval [CI], 1.02-1.15) and remitters (MH RR, 1.12; 95% CI, 1.02-1.24) in venlafaxine groups compared with those in tricyclic antidepressant group. Hamilton Depression Rating Scale end point scores in the venlafaxine groups were lower (Hedges' g = 0.16; 95% CI, 0.04-0.27), and venlafaxine was better tolerated than tricyclic antidepressant (Hedges' g = 0.56; 95% CI, 0.37-0.74). There were no significant differences between venlafaxine and selective serotonin reuptake inhibitor on any of these parameters. Analyses of publication bias were inconclusive. Chinese researchers have published a number of randomized controlled trials comparing venlafaxine to active comparators, but study quality was found to be low. To make optimal use of their research potential Chinese, researchers will have to improve trial reporting and the peer-review process.

Influence of trial registration on reporting quality of randomized trials: study from highest ranked journals

OBJECTIVE

To evaluate the reporting quality of key methodological items of randomized control trials (RCTs) in 55 of the highest ranked journals.

STUDY DESIGN AND SETTING

A list of the highest top ranked journals was identified, and a search for detecting RCTs in those journals was made. Two hundred sixty four journals were screened and 55 of them were identified having at least one RCT. Three RCTs were randomly selected a priori from each journal; 148 RCTs were finally included. RCTs were assessed by two reviewers using the Consolidated Standards of Reporting Trials (CONSORT) statement.

RESULTS

Only 11 (8%) RCTs had all items adequately reported. In addition, 36% of RCTs reported that the study was registered in any trial registry. We found a significant difference in the quality of reporting for baseline characteristics, recruitment, participant's flow, and randomization implementation between those studies having reported the registration of their RCT in a trial registry and those that have not. Adherence to key methodological items of the CONSORT statement was as follows: sample size determination (60%), sequence generation (49%), allocation concealment (40%), and blinding (25%).

CONCLUSIONS

Reporting of varied CONSORT items remains suboptimal. Registration in a trial registry was associated with improved reporting. Further efforts to enhance RCT registration could contribute to this improvement.

Assessment of the Methodologic Quality of Medical and Surgical Clinical Trials in Patients with Arthroplasty

Objective.

To assess the methodological quality of randomized controlled trials (RCT) of medical and surgical therapy in patients with arthroplasty.

Methods.

We conducted a Medline database search for all arthroplasty RCT from 1997 and 2006. The quality of the methods of all eligible RCT was assessed by a trained abstractor. We used a checklist of trial quality characteristics, and the overall trial quality was assessed by 3 scales: Jadad (range 0–5), Delphi list (range 0–9), and numeric rating scale (NRS; range 1–10), based on User’s Guides to the Medical Literature.

Results.

A total of 196 articles were included in the analysis; most included hip (n = 81) or knee (n = 80) or both hip/knee arthroplasty (n = 19); 66 (34%) assessed pharmacological treatments, 117 (60%) nonpharmacological treatments, and 13 (7%) both. Mean (SEM) overall quality scores of arthroplasty RCT were low: Jadad score 2.36 (1.4), Delphi list 5.33 (1.6), and NRS score 4.30 (2.6). Multivariable analyses revealed that nonpharmacological intervention RCT had lower odds (odds ratio 0.28–0.39; p = 0.008–0.033) and those with no funding had lower odds (OR 0.28–0.50; p = 0.014–0.119) of being in the highest quartiles of the 3 overall quality scores. In contrast, multicenter RCT had 1.8–4.7 times higher odds of being in highest tertiles of quality scores (p = 0.017–0.185).

Conclusion.

Methodological deficiencies in reporting of hip/knee arthroplasty RCT offer an opportunity for improvement. Type of intervention, number of trial centers, and presence of funding were independently associated with overall trial quality. In future, multicenter RCT (rather than single-center) and modeling protocols of single-center RCT similar in rigor to multicenter RCT may improve the quality of arthroplasty RCT.

The geography of clinical cancer research: analysis of abstracts presented at the American Society of Clinical Oncology Annual Meetings

BACKGROUND

The American Society of Clinical Oncology Annual Meeting is the largest forum for presentation of clinical research in oncology. We quantified the contribution of countries and assessed correlates of their presence at such meetings.

METHODS

After stratifying abstracts according to category of presentation (oral, poster, and 'publication only'), we took a random sample of 10% of the studies presented at years 2001-2003 and 2006-2008. We assigned abstract nationality using the affiliation of authors. For multinational studies, we developed an algorithm to assign nationality.

RESULTS

Of the 22 045 eligible abstracts, 2206 were analyzed and represented 71 countries: 905 (41%) abstracts were from a single institution, 969 (44%) were multicenter, uninational studies, and 332 (15%) were multinational studies. United States nationality was assigned to 49% of all abstracts and the next 14 countries with a higher number of studies accounted for 41%. There was a statistically significant temporal trend in the proportion of multinational studies. Also, multinational studies and abstracts with United States nationality were more frequently presented in oral and poster fashion and had more frequent involvement of the pharmaceutical industry.

CONCLUSION

This study provides a geographic overview of clinical cancer research and indicates that multinational collaboration is increasing.

Looking for landmarks: the role of expert review and bibliometric analysis in evaluating scientific publication outputs

OBJECTIVE

To compare expert assessment with bibliometric indicators as tools to assess the quality and importance of scientific research papers.

METHODS AND MATERIALS

Shortly after their publication in 2005, the quality and importance of a cohort of nearly 700 Wellcome Trust (WT) associated research papers were assessed by expert reviewers; each paper was reviewed by two WT expert reviewers. After 3 years, we compared this initial assessment with other measures of paper impact.

RESULTS

Shortly after publication, 62 (9%) of the 687 research papers were determined to describe at least a 'major addition to knowledge' -6 were thought to be 'landmark' papers. At an aggregate level, after 3 years, there was a strong positive association between expert assessment and impact as measured by number of citations and F1000 rating. However, there were some important exceptions indicating that bibliometric measures may not be sufficient in isolation as measures of research quality and importance, and especially not for assessing single papers or small groups of research publications.

CONCLUSION

When attempting to assess the quality and importance of research papers, we found that sole reliance on bibliometric indicators would have led us to miss papers containing important results as judged by expert review. In particular, some papers that were highly rated by experts were not highly cited during the first three years after publication. Tools that link expert peer reviews of research paper quality and importance to more quantitative indicators, such as citation analysis would be valuable additions to the field of research assessment and evaluation.

CONSORT adoption and quality of reporting of randomized controlled trials: a systematic analysis in two dermatology journals

BACKGROUND

CONSORT (Consolidated Standards for Reporting Trials) guidelines were constructed to ensure optimal reporting quality of randomized controlled trials (RCTs).

OBJECTIVES

To determine the effect of the adoption of CONSORT on the reporting quality of RCTs, we performed a systematic evaluation of RCTs published in two dermatology journals pre- and post-CONSORT adoption.

METHODS

The journals selected for the study were the Journal of the American Academy of Dermatology and the British Journal of Dermatology. We selected RCTs published in 1997 and 2006 using both Medline and hand searching. The following critical CONSORT criteria were recorded: sample size, type of disease studied, type of control, single-centre or multicentre study, type of funding, blinding, methods and type of randomization, definition of a primary endpoint, justification for sample size selection and power calculation, population for analysis, and adequacy of group comparison. A multivariable analysis was conducted to determine factors associated with optimal reporting quality.

RESULTS

In total, 98 studies were included. Improvement in reporting quality was evident for the specification of the randomization method (20% in 1997 vs. 45% in 2006, P < 0.01) and for the justification of sample size (22% in 1997 vs. 43% in 2006, P = 0.027). The percentage of studies with optimal reporting quality increased from 11% in 1997 to 28% in 2006 (P = 0.03). Factors significantly associated with a good methodological quality were pharmaceutical industry funding and publication in 2006 vs. 1997.

CONCLUSIONS

There is a need to improve the reporting quality of RCTs published in dermatology journals.

A quarter century of the analysis of verbal behavior: an analysis of impact

The Analysis of Verbal Behavior (TAVB) has been published since 1982, and during this time, interest in verbal behavior research appears to have increased substantially within behavior analysis. The purpose of the present analysis was to assess the influence of TAVB on the field by (a) counting citations of TAVB articles in the Journal of the Experimental Analysis of Behavior (JEAB) and the Journal of Applied Behavior Analysis (JABA) from 1983 through 2007, (b) examining which other journals cite TAVB, and (c) calculating impact-factor estimates for 2003 through 2007. Citations of TAVB articles began to appear in JEAB and JABA in the late 1980s to early 1990s, and by the end of 2007, almost a third of all articles published in TAVB had been cited in either JEAB or JABA. Other journals that cite TAVB include The Behavior Analyst and The Psychological Record. The estimated impact factor ranged from 0.267 to 0.600. Strategies for increasing the impact of TAVB are discussed.

Assessing the quality of a scientific journal: the case of Revista Brasileira de Psiquiatria

Scientific production in the field of Psychiatry is growing exponentially, as reflected by the number of scientific journals now available. Current strategies to objectively estimate the quality of a specific publication are mostly based on indexation and citation criteria. The concept of impact factor, a measure of the frequency with which an article has been cited in a particular period, is key to understanding the way journals are evaluated. The recent indexation of Revista Brasileira de Psiquiatria in the two major medical databases has increased the journal's visibility, as indicated by the rising number of citations received over the last years. As researchers' scientific productivity measures are largely associated with citation indices of journals in which manuscripts are published, it is essential that Revista Brasileira de Psiquiatria continues to attract high quality, innovative articles, with the ultimate goal of providing state-of-the-art continuing medical education for mental health professionals.

Quality of scientific articles

The paper discusses the difficulties in judging the quality of scientific manuscripts and describes some common pitfalls that should be avoided when preparing a paper for submission to a peer-reviewed journal. Peer review is an imperfect system, with less than optimal reliability and uncertain validity. However, as it is likely that it will remain as the principal process of screening papers for publication, authors should avoid some common mistakes when preparing a report based on empirical findings of human research. Among these are: excessively long abstracts, extensive use of abbreviations, failure to report results of parsimonious data analyses, and misinterpretation of statistical associations identified in observational studies as causal. Another common problem in many manuscripts is their excessive length, which makes them more difficult to be evaluated or read by the intended readers, if published. The evaluation of papers after their publication with a view towards their inclusion in a systematic review is also discussed. The limitations of the impact factor as a criterion to judge the quality of a paper are reviewed.

The "impact factor" revisited

The number of scientific journals has become so large that individuals, institutions and institutional libraries cannot completely store their physical content. In order to prioritize the choice of quality information sources, librarians and scientists are in need of reliable decision aids. The "impact factor" (IF) is the most commonly used assessment aid for deciding which journals should receive a scholarly submission or attention from research readership. It is also an often misunderstood tool. This narrative review explains how the IF is calculated, how bias is introduced into the calculation, which questions the IF can or cannot answer, and how different professional groups can benefit from IF use.