Role of editorial and peer review processes in publication bias: analysis of drug trials submitted to eight medical journals

Gender, Racial, and Academic Authorship Diversity in the Otolaryngology Clinical Trial Literature

OBJECTIVE

To investigate the state and trajectory of gender, racial, and academic authorship diversity in the otolaryngology clinical trial literature over the past 2 decades.

STUDY DESIGN

Bibliometric analysis.

SETTING

Otolaryngology clinical trial literature.

METHODS

Clinical trials published in the 9 major otolaryngology journals between 2000 and 2020 were included. The gender, race, and academic seniority of the first, senior, and corresponding authors were recorded for each trial. Multivariable regression models assessed the temporal trajectory of authorship diversity over time and the disparity in citations across author characteristics. Models adjusted for relevant confounders pertaining to publication environment and study design.

RESULTS

Among 2117 trials, first, senior, and corresponding authors have been predominantly White (60%-64%), male (76%-80%), and attending physicians (63%-69%). Trials led by Black (<1%) and Hispanic (<5%) authors were severely underrepresented. Over time, the representation of female (adj. β 0.8%, 95% CI [0.5%, 1.1%] per year), Asian (1.0% [0.7%, 1.3%] per year), and MD resident (0.4% [0.1%, 0.7%] per year) first authorship increased, but representation of female (0.2% [-0.1%, 0.5%] per year), Black (0% [-0.03%, 0.02%] per year), Hispanic (-0.2% [-0.33%, -0.02%] per year) senior authorship remained persistently low. Asian-led trials were cited significantly less compared to White-led trials even after adjusting for study design and publication year (aIRR 0.82 [0.73, 0.92]).

CONCLUSIONS

Despite promising signs of improving authorship diversity over time, persistent underrepresentation of female, Black, Hispanic senior authorship underscore the need for additional efforts to diversify the otolaryngology clinical science workforce.

Risk of bias and problematic trials: characterising the research integrity of trials submitted to Anaesthesia

BACKGROUND

There is some evidence for systematic biases and failures of research integrity in the anaesthesia literature. However, the features of problematic trials and effect of editorial selection on these issues have not been well quantified.

METHODS

We analysed 209 randomised controlled trials submitted to Anaesthesia between 8 March 2019 and 31 March 2020. We evaluated the submitted manuscript, registry data and the results of investigations into the integrity of the trial undertaken at the time of submission. Trials were labelled 'concerning' if failures of research integrity were found, and 'problematic' if identified issues would have warranted retraction if they had been found after publication. We investigated how 'problematic' trials were detected, the distribution of p values and the risk of outcome reporting bias and p-hacking. We also investigated whether there were any factors that differed in problematic trials.

RESULTS

We found that false data was the most common reason for a trial to be labelled as 'concerning', which occurred in 51/62 (82%) cases. We also found that while 195/209 (93%) trials were preregistered, we found adequate registration for only 166/209 (79%) primary outcomes, 100/209 (48%) secondary outcomes and 11/209 (5%) analysis plans. We also found evidence for a step decrease in the frequency of p values > 0.05 compared with p values < 0.05. 'Problematic' trials were all single-centre and appeared to have fewer authors (incident risk ratio (95%CI) 0.8 (0.7-0.9)), but could not otherwise be distinguished reliably from other trials.

CONCLUSIONS

Identification of 'problematic' trials is frequently dependent on individual patient data, which is often unavailable after publication. Additionally, there is evidence of a risk of outcome reporting bias and p-hacking in submitted trials. Implementation of alternative research and editorial practices could reduce the risk of bias and make identification of problematic trials easier.

Time to publication for results of clinical trials

BACKGROUND

Researchers conducting trials have a responsibility to publish the results of their work in a peer-reviewed journal, and failure to do so may introduce bias that affects the accuracy of available evidence. Moreover, failure to publish results constitutes research waste.

OBJECTIVES

To systematically review research reports that followed clinical trials from their inception and their investigated publication rates and time to publication. We also aimed to assess whether certain factors influenced publication and time to publication.

SEARCH METHODS

We identified studies by searching MEDLINE, Embase, Epistemonikos, the Cochrane Methodology Register (CMR) and the database of the US Agency for Healthcare Research and Quality (AHRQ), from inception to 23 August 2023. We also checked reference lists of relevant studies and contacted experts in the field for any additional studies.

SELECTION CRITERIA

Studies were eligible if they tracked the publication of a cohort of clinical trials and contained analyses of any aspect of the publication rate or time to publication of these trials.

DATA COLLECTION AND ANALYSIS

Two review authors performed data extraction independently. We extracted data on the prevalence of publication and the time from the trial start date or completion date to publication. We also extracted data from the clinical trials included in the research reports, including country of the study's first author, area of health care, means by which the publication status of these trials were sought and the risk of bias in the trials.

MAIN RESULTS

A total of 204 research reports tracking 165,135 trials met the inclusion criteria. Just over half (53%) of these trials were published in full. The median time to publication was approximately 4.8 years from the enrolment of the first trial participant and 2.1 years from the trial completion date. Trials with positive results (i.e. statistically significant results favouring the experimental arm) were more likely to be published than those with negative or null results (OR 2.69, 95% CI 2.02 to 3.60; 19 studies), and they were published in a shorter time (adjusted HR 1.92, 95% CI 1.51 to 2.45; 4 studies). On average, trials with positive results took 2 years to publish, whereas trials with negative or null results took 2.6 years. Large trials were more likely to be published than smaller ones (adjusted OR 1.92, 95% CI 1.33 to 2.77; 11 studies), and they were published in a shorter time (adjusted HR 1.41, 95% CI 1.18 to 1.68; 7 studies). Multicentre trials were more likely to be published than single-centre trials (adjusted OR 1.20, 95% CI 1.03 to 1.40; 2 studies). We found no difference between multicentre and single-centre trials in time to publication. Trials funded by non-industry sources (e.g.governments or universities) were more likely to be published than trials funded by industry (e.g. pharmaceutical companies or for-profit organisations) (adjusted OR 2.13, 95% CI 1.82 to 2.49; 14 studies); they were also published in a shorter time (adjusted HR 1.46, 95% CI 1.15 to 1.86; 7 studies).

AUTHORS' CONCLUSIONS

Our updated review shows that trial publication is poor, with only half of all trials that are conducted being published. Factors that may make publication more likely and lead to faster publication are positive results, large sample size and being funded by non-industry sources. Differences in publication rates result in publication bias and time-lag bias that may influence findings and therefore ultimately affect treatment decisions. Systematic review authors should consider the possibility of time-lag bias when conducting a systematic review, especially when updating their review.

FUNDING

This Cochrane review had no dedicated funding.

REGISTRATION

This review combines and updates two earlier Cochrane reviews. The two protocols and previous versions of the two updated reviews are available via 10.1002/14651858.MR000006 and 10.1002/14651858.MR000006.pub3 and 10.1002/14651858.MR000011 and 10.1002/14651858.MR000011.pub2.

Sample Size Considerations for Fine-Tuning Large Language Models for Named Entity Recognition Tasks: Methodological Study

BACKGROUND

Large language models (LLMs) have the potential to support promising new applications in health informatics. However, practical data on sample size considerations for fine-tuning LLMs to perform specific tasks in biomedical and health policy contexts are lacking.

OBJECTIVE

This study aims to evaluate sample size and sample selection techniques for fine-tuning LLMs to support improved named entity recognition (NER) for a custom data set of conflicts of interest disclosure statements.

METHODS

A random sample of 200 disclosure statements was prepared for annotation. All "PERSON" and "ORG" entities were identified by each of the 2 raters, and once appropriate agreement was established, the annotators independently annotated an additional 290 disclosure statements. From the 490 annotated documents, 2500 stratified random samples in different size ranges were drawn. The 2500 training set subsamples were used to fine-tune a selection of language models across 2 model architectures (Bidirectional Encoder Representations from Transformers [BERT] and Generative Pre-trained Transformer [GPT]) for improved NER, and multiple regression was used to assess the relationship between sample size (sentences), entity density (entities per sentence [EPS]), and trained model performance (F1-score). Additionally, single-predictor threshold regression models were used to evaluate the possibility of diminishing marginal returns from increased sample size or entity density.

RESULTS

Fine-tuned models ranged in topline NER performance from F1-score=0.79 to F1-score=0.96 across architectures. Two-predictor multiple linear regression models were statistically significant with multiple R2 ranging from 0.6057 to 0.7896 (all P<.001). EPS and the number of sentences were significant predictors of F1-scores in all cases ( P<.001), except for the GPT-2_large model, where EPS was not a significant predictor (P=.184). Model thresholds indicate points of diminishing marginal return from increased training data set sample size measured by the number of sentences, with point estimates ranging from 439 sentences for RoBERTa_large to 527 sentences for GPT-2_large. Likewise, the threshold regression models indicate a diminishing marginal return for EPS with point estimates between 1.36 and 1.38.

CONCLUSIONS

Relatively modest sample sizes can be used to fine-tune LLMs for NER tasks applied to biomedical text, and training data entity density should representatively approximate entity density in production data. Training data quality and a model architecture's intended use (text generation vs text processing or classification) may be as, or more, important as training data volume and model parameter size.

Positive Outcome Bias in the Influential Otolaryngology Clinical Trial Literature

OBJECTIVE

To assess for the presence of positive outcome bias in the otolaryngology clinical trial literature. Specifically, we investigate the prevalence of clinical trials with positive findings (CTP) and clinical trials with negative findings (CTN), as well as their quality of evidence and subsequent impact.

STUDY DESIGN

Retrospective analysis.

SETTING

Clinical Trials in the Influential Otolaryngology Literature.

METHODS

We reviewed all clinical trials published in 4 major otolaryngology journals between 2000 and 2020. We constructed several multivariable regression models to investigate the relationship of finding direction with randomization status and citation count. Subsequently, we incorporated an interaction term between year and the primary covariate of each model to assess the temporal trajectory of these relationships. All models accounted for sample size, journal, subspecialty, and the affiliated program prestige.

RESULTS

Of the 1367 trials analyzed, 1143 (84%) were CTPs, a rate that persisted throughout the study period (aOR 1.00, 95% CI 0.98-1.03). CTPs were significantly less likely to be randomized compared to CTNs (aOR 0.25, 0.17-0.37), a relationship that persisted over time (aOR 1.05, 0.99-1.03). CTPs received significantly more citations compared to CTNs (aIRR 1.41, 1.25-1.60), a disparity that also persisted over time (aIRR 0.99, 0.97-1.01).

CONCLUSION

The otolaryngology clinical trial literature has been heavily dominated by positive findings. CTPs were more frequently cited and published even with a lower level of evidence compared to CTNs. This bias may influence the objectivity of evidence used to guide clinical practice and warrants attention when reviewing findings and changing practices.

Prospective longitudinal associations between adverse childhood experiences and adult mental health outcomes: a protocol for a systematic review and meta-analysis

BACKGROUND

Research cites a strong, dose-response relationship between adverse childhood experiences (ACEs) and poor adult mental health outcomes including anxiety, depression, post-traumatic stress disorder (PTSD), self-harm, suicidality, and psychotic-like experiences.

AIM

To systematically investigate the existence and strength of association between ACEs and adult mental health outcomes in prospective longitudinal studies. The review will focus on the outcomes: anxiety, depression, PTSD, self-harm, suicidal ideation, and psychotic-like experiences.

METHODS

Twelve electronic databases will be searched: Embase, PsycINFO, MEDLINE, and Global Health through the OVID interface. ProQuest will be used to search Public Affairs Information Service (PAIS), Dissertations and Theses, Sociology Database (including Sociological Abstracts and Social Services Abstracts), PTSDpubs (formerly The Published International Literature on Traumatic Stress (PILOTS) Database) and Applied Social Sciences Index and Abstracts (ASSIA). CINAHL, World Health Organisation (WHO) Global Index Medicus, and WHO Violence Info will also be searched. Eligible studies will be double screened, assessed, and their data will be extracted. Any disagreement throughout these processes will be settled by a third reviewer. If enough studies meet the criteria and the methodological quality of each study is sufficient, a meta-analysis will be conducted.

ANALYSIS

A narrative synthesis of included studies and the associations between ACEs and adult mental health will be completed. If the number of studies included per mental health outcome is two or more, a multi-level meta-analysis will be completed using odds ratio effect sizes as outcomes.

DISCUSSION

This review will contribute to the existing body of literature supporting the long-term effects of ACEs on adult mental health. This review adds to previous reviews that have either synthesised cross-sectional associations between ACEs and mental health outcomes, synthesised longitudinal studies exploring the effect of ACEs on different physical and mental health outcomes or synthesised longitudinal studies exploring the effect of ACEs on the same mental health outcomes using different methods. This review aims to identify methodological weaknesses and knowledge gaps in current literature that can be addressed in future primary studies.

SYSTEMATIC REVIEW REGISTRATION

This protocol has been registered in PROSPERO (CRD42021297882).

Nudging accurate scientific communication

The recent replicability crisis in social and biomedical sciences has highlighted the need for improvement in the honest transmission of scientific content. We present the results of two studies investigating whether nudges and soft social incentives enhance participants' readiness to transmit high-quality scientific news. In two online randomized experiments (Total N = 2425), participants had to imagine that they were science journalists who had to select scientific studies to report in their next article. They had to choose between studies reporting opposite results (for instance, confirming versus not confirming the effect of a treatment) and varying in traditional signs of research credibility (large versus small sample sizes, randomized versus non-randomized designs). In order to steer participants' choices towards or against the trustworthy transmission of science, we used several soft framing nudges and social incentives. Overall, we find that, although participants show a strong preference for studies using high-sample sizes and randomized design, they are biased towards positive results, and express a preference for results in line with previous intuitions (evincing confirmation bias). Our soft framing nudges and social incentives did not help to counteract these biases. On the contrary, the social incentives against honest transmission of scientific content mildly exacerbated the expression of these biases.

Time to Publication and Time-Lag Publication Bias for Randomized Trials on Connective Tissue Diseases

OBJECTIVE

To assess the time from completion to publication of randomized controlled trials (RCTs) on connective tissue diseases (CTDs), investigate the factors associated with, and explore the influence of significance of study results on time to publication (time-lag publication bias).

METHODS

We included interventional, phase 2/3, 3, or 4 RCTs on CTDs registered in Clinicaltrials.gov from 2000 to 2016, whose results had been published in a peer-review journal less than 5 years after their completion. Main trial features, including the significance of primary outcome results, were collected. Time to publication was the time from study completion to the earliest publication date. Multivariable linear regression was used to identify factors associated with time to publication.

RESULTS

We included 62 studies, mostly phase 3 (61%) trials on pharmacologic treatments (94%); we recruited patients with systemic lupus (55%) or systemic sclerosis (23%) and planned to enroll a median of 131 (IQR [interquartile range]: 61-288) patients. Twenty-two (35%) reported at least a statistically significant primary outcome. Median time to publication was 28 months (IQR: 17-36). In a multivariable analysis, time to publication progressively improved over time (faster publication in recent years, with the average time to publication decreasing by 1.3 [95% CI: 0.3-2.3] months per year) and was not influenced by the significance of primary outcome results, funder, impact factor of the journal, number of recruiting countries, and comparator.

CONCLUSION

A high proportion of CTDs-RCTs is published beyond 2 years from completion. We did not find evidence of time-lag publication bias, and time to publication improved over time.

A Content Analysis of Self-Reported Financial Relationships in Biomedical Research

INTRODUCTION

Financial conflicts of interest (fCOI) present well documented risks to the integrity of biomedical research. However, few studies differentiate among fCOI types in their analyses, and those that do tend to use preexisting taxonomies for fCOI identification. Research on fCOI would benefit from an empirically-derived taxonomy of self-reported fCOI and data on fCOI type and payor prevalence.

METHODS

We conducted a content analysis of 6,165 individual self-reported relationships from COI statements distributed across 378 articles indexed with PubMed. Two coders used an iterative coding process to identify and classify individual fCOI types and payors. Inter-rater reliability was κ = 0.935 for fCOI type and κ = 0.884 for payor identification.

RESULTS

Our analysis identified 21 fCOI types, 9 of which occurred at prevalences greater than 1%. These included research funding (24.8%), speaking fees (20.8%), consulting fees (18.8%), advisory relationships (11%), industry employment (7.6%), unspecified fees (4.8%), travel fees (3.2%), stock holdings (3.1%), and patent ownership (1%). Reported fCOI were held with 1,077 unique payors, 22 of which were present in more than 1% of financial relationships. The ten most common payors included Pfizer (4%), Novartis (3.9%), MSD (3.8%), Bristol Myers Squibb (3.2%), AstraZeneca (3.1%), GSK (3%), Boehringer Ingelheim (2.9%), Roche (2.8%), Eli LIlly (2.5%), and AbbVie (2.4%).

CONCLUSIONS

These results provide novel multi-domain prevalence data on self-reported fCOI and payors in biomedical research. As such, they have the potential to catalyze future research that can assess the differential effects of various types of fCOI. Specifically, the data suggest that comparative analyses of the effects of different fCOI types are needed and that special attention should be paid to the diversity of payor types for research relationships.

Characteristics and publication discrepancies of foot & ankle surgery-related clinical trials

BACKGROUND

Trial registration can provide more reliable study designs, however, the final publications are not exempt from bias. This study aimed to show the characteristics of foot & ankle surgery-related clinical trials registered in ClinicalTrials.gov (CTG) and analyze possible risk factors that can lead to discrepancies between the original trial registry and published results.

METHODS

Foot & ankle surgery-related randomized controlled studies registered in CTG with an anticipated completion date until 2018 were extracted. 157 trials were found eligible for final analysis and corresponding publications were investigated through literature databases. Discrepancies between trial registries and published results were analyzed.

RESULTS

67.5% of trials were registered whether during the trial period or after completion. 46.5% were funded by industry. 54.8% of trials were reported to be completed. The results of 66 (42%) trials were published in a journal. There was a sample size discrepancy in 57.6%, follow-up time discrepancy in 28.8%, primary outcome discrepancy in 45.5%, and inclusion/exclusion criteria discrepancy in 89.4% of registries. Sample size, inclusion/exclusion criteria discrepancies were significantly more in trials funded by industry (p = 0.024, 0.001, 0.037, respectively). Industry-funded studies reported positive results more frequently (p = 0.011).

CONCLUSION

Only 42% of all registered foot & ankle surgery-related studies had the chance to be published. The majority of studies presented discrepancies in sample size, outcome assessment method, follow-up time, or inclusion/exclusion criteria. Industry-funded studies showed a higher rate of discrepancies and were more inclined towards reporting positive results indicating a risk of bias in assessments.

LEVEL OF CLINICAL EVIDENCE

Level 3.

On the Utility of ToxCast-Based Predictive Models to Evaluate Potential Metabolic Disruption by Environmental Chemicals

BACKGROUND

Research suggests environmental contaminants can impact metabolic health; however, high costs prohibit in vivo screening of putative metabolic disruptors. High-throughput screening programs, such as ToxCast, hold promise to reduce testing gaps and prioritize higher-order (in vivo) testing.

OBJECTIVES

We sought to a) examine the concordance of in vitro testing in 3T3-L1 cells to a targeted literature review for 38 semivolatile environmental chemicals, and b) assess the predictive utility of various expert models using ToxCast data against the set of 38 reference chemicals.

METHODS

Using a set of 38 chemicals with previously published results in 3T3-L1 cells, we performed a metabolism-targeted literature review to determine consensus activity determinations. To assess ToxCast predictive utility, we used two published ToxPi models: a) the 8-Slice model published by Janesick et al. (2016) and b) the 5-Slice model published by Auerbach et al. (2016). We examined the performance of the two models against the Janesick in vitro results and our own 38-chemical reference set. We further evaluated the predictive performance of various modifications to these models using cytotoxicity filtering approaches and validated our best-performing model with new chemical testing in 3T3-L1 cells.

RESULTS

The literature review revealed relevant publications for 30 out of the 38 chemicals (the remaining 8 chemicals were only examined in our previous 3T3-L1 testing). We observed a balanced accuracy (average of sensitivity and specificity) of 0.86 comparing our previous in vitro results to the literature-derived calls. ToxPi models provided balanced accuracies ranging from 0.55 to 0.88, depending on the model specifications and reference set. Validation chemical testing correctly predicted 29 of 30 chemicals as per 3T3-L1 testing, suggesting good adipogenic prediction performance for our best adapted model.

DISCUSSION

Using the most recent ToxCast data and an updated ToxPi model, we found ToxCast performed similarly to that of our own 3T3-L1 testing in predicting consensus calls. Furthermore, we provide the full ranked list of largely untested chemicals with ToxPi scores that predict adipogenic activity and that require further investigation. https://doi.org/10.1289/EHP6779.

Registry versus publication: discrepancy of primary outcomes and possible outcome reporting bias in child and adolescent mental health

Outcome reporting bias is one of the fundamental forms of publication bias. It implies publishing only outcomes that have positive results. The aim of this observational study was to explore primary outcome discrepancies between registry of clinical trials and their corresponding publications, since these can indicate outcome reporting bias in child mental health. Data were extracted from completed interventional clinical trials from ClinicalTrial.gov registry and its Archive site. Trials were registered under "Behaviours and Mental Disorders" category, and conducted on underage participants (0-17 years). Their primary outcomes were compared to those published in publication which had a corresponding NCT number stated in the text. Sixteen percent of trials did not have the minimum information on primary outcome stated in the registry-neither the measure used nor the measurement time points; 38.9% of trials had the minimum information stated to describe primary outcome, while only 3.3% of trials had all the necessary elements stated in the registry. Most of the publication in our sample had positive results (66.4%). Half of the trials registered before completion had non-matching primary outcomes in the registry and publication; 85.4% of trials with non-matching outcomes indicated possible outcome reporting bias for some of the primary outcome. Middle-sized trials and industry-funded trials were related with higher quality of primary outcome registration. Industry funding was related with positive findings in publication. Non-industry funding proved to be the only significant predictor of discrepancy between registered and published primary outcomes, and possible outcome reporting bias. Journal impact factor was not related with any of the outcome measures. The main limitation of the study is that it primarily offers an insight into discrepancy of registered and published outcomes. The methodology does not imply an access to results of unpublished outcomes - therefore, it was not possible to determine the presence of the bias with sufficient certainty in large number of trials. Further research should be done with improved methodology and additional data.

Association of Accuracy, Conclusions, and Reporting Completeness With Acceptance by Radiology Conferences and Journals

BACKGROUND

Preferential publication of studies with positive findings can lead to overestimation of diagnostic test accuracy (i.e. publication bias). Understanding the contribution of the editorial process to publication bias could inform interventions to optimize the evidence guiding clinical decisions.

PURPOSE/HYPOTHESIS

To evaluate whether accuracy estimates, abstract conclusion positivity, and completeness of abstract reporting are associated with acceptance to radiology conferences and journals.

STUDY TYPE

Meta-research.

POPULATION

Abstracts submitted to radiology conferences (European Society of Gastrointestinal and Abdominal Radiology (ESGAR) and International Society for Magnetic Resonance in Medicine (ISMRM)) from 2008 to 2018 and manuscripts submitted to radiology journals (Radiology, Journal of Magnetic Resonance Imaging [JMRI]) from 2017 to 2018. Primary clinical studies evaluating sensitivity and specificity of a diagnostic imaging test in humans with available editorial decisions were included.

ASSESSMENT

Primary variables (Youden's index [YI > 0.8 vs. <0.8], abstract conclusion positivity [positive vs. neutral/negative], number of reported items on the Standards for Reporting of Diagnostic Accuracy Studies [STARD] for Abstract guideline) and confounding variables (prospective vs. retrospective/unreported, sample size, study duration, interobserver agreement assessment, subspecialty, modality) were extracted.

STATISTICAL TESTS

Multivariable logistic regression to obtain adjusted odds ratio (OR) as a measure of the association between the primary variables and acceptance by radiology conferences and journals; 95% confidence intervals (CIs) and P-values were obtained; the threshold for statistical significance was P < 0.05.

RESULTS

A total of 1000 conference abstracts (500 ESGAR and 500 ISMRM) and 1000 journal manuscripts (505 Radiology and 495 JMRI) were included. Conference abstract acceptance was not significantly associated with YI (adjusted OR = 0.97 for YI > 0.8; CI = 0.70-1.35), conclusion positivity (OR = 1.21 for positive conclusions; CI = 0.75-1.90) or STARD for Abstracts adherence (OR = 0.96 per unit increase in reported items; CI = 0.82-1.18). Manuscripts with positive abstract conclusions were less likely to be accepted by radiology journals (OR = 0.45; CI = 0.24-0.86), while YI (OR = 0.85; CI = 0.56-1.29) and STARD for Abstracts adherence (OR = 1.06; CI = 0.87-1.30) showed no significant association. Positive conclusions were present in 86.7% of submitted conference abstracts and 90.2% of journal manuscripts.

DATA CONCLUSION

Diagnostic test accuracy studies with positive findings were not preferentially accepted by the evaluated radiology conferences or journals.

EVIDENCE LEVEL

3 TECHNICAL EFFICACY: Stage 2.

Publication Rates of Heart Failure Clinical Trials Remain Low

BACKGROUND

Under-reporting of clinical trial results inhibits dissemination of knowledge, limits understanding of therapeutic interventions, and may ultimately harm patients.

OBJECTIVES

This study examined the rates and predictors of heart failure clinical trial publication and how they have changed over time.

METHODS

This study assessed cross-sectional analysis of all heart failure clinical trials registered on ClinicalTrials.gov with at least 2 years follow-up after trial completion. The content area was chosen for the robust clinical trial activity in the field. The primary outcome was manuscript publication with multivariable proportional hazards adjustment to identify associations with publication.

RESULTS

Of the 1,429 included studies, 806 (56%) were published as manuscripts, 623 were unpublished, and 97 (7%) reported results without manuscript publication. Of the total, 1,243 were completed after 2007, when the mean 1-year publication rate for interventional trials rose from 12.7% to 19.6% (p = 0.049), which was possibly associated with changes in government regulation. However, there was no further sustained improvement over time, and there was no multivariable association between later completion dates and reporting or publication of results. Funding from the National Institutes of Health and use of clinical (death, hospitalization, myocardial infarction, changes in functional classification) rather than nonclinical primary endpoints were associated with earlier publication. Whether the results were consistent with the primary study hypothesis was not associated with likelihood of publication.

CONCLUSIONS

The rates of heart failure clinical trial publication or reporting of results remain unacceptably low. Additional efforts by all stakeholders, including investigators, sponsors, regulators, societies, editors, and journals are needed to improve data dissemination.

Assessment of Consistency Between Peer-Reviewed Publications and Clinical Trial Registries

Importance

Clinical trial registries are intended to increase clinical research transparency by nonselectively identifying and documenting clinical trial designs and outcomes. Inconsistencies in reported data undermine the utility of such registries and have previously been noted in general medical literature.

Objective

To assess whether inconsistencies in reported data exist between ophthalmic literature and clinical trial registries.

Design, Setting, and Participants

In this retrospective, cross-sectional study, interventional clinical trials published from January 1, 2014, to December 31, 2014, in the American Journal of Ophthalmology, JAMA Ophthalmology, and Ophthalmology were reviewed. Observational, retrospective, uncontrolled, and post hoc reports were excluded, yielding a sample size of 106 articles. Data collection was performed from January through September 2016. Data review and adjudication continued through January 2017.

Main Outcomes and Measures

If possible, articles were matched to registry entries listed in the ClinicalTrials.gov database or in 1 of 16 international registries indexed by the World Health Organization International Clinical Trials Registry Platform version 3.2 search engine. Each article-registry pair was assessed for inconsistencies in design, results, and funding (each of which was further divided into subcategories) by 2 reviewers and adjudicated by a third.

Results

Of 106 trials that met the study criteria, matching registry entries were found for 68 (64.2%), whereas no matching registry entries were found for 38 (35.8%). Inconsistencies were identified in study design, study results, and funding sources, including specific interventions in 8 (11.8%), primary outcome measure (POM) designs in 32 (47.1%), and POM results in 48 (70.6%). In addition, numerous data pieces were unreported, including analysis methods in 52 (76.5%) and POM results in 38 (55.9%).

Conclusions and Relevance

Clinical trial registries were underused in this sample of ophthalmology clinical trials. For studies with registry data, inconsistency rates between published and registered data were similar to those previously reported for general medical literature. In most cases, inconsistencies involved missing data, but explicit discrepancies in methods and/or data were also found. Transparency and credibility of published trials may be improved by closer attention to their registration and reporting.

Delivery of care for functional gastrointestinal disorders: A systematic review

BACKGROUND

A diverse range of treatments are available for the treatment of functional gastrointestinal disorders (FGIDs). Individual treatments, including drug therapies, behavioral therapy ("biofeedback"), psychological therapies, and dietary therapies, have been well validated in controlled, randomized trials and real-life case series. However, few studies have evaluated models of delivery of care for the whole population of referred patients with an FGID. This review evaluates models of specialist outpatient care for the management of FGIDs.

METHODS

A systematic review was performed of full-text articles published until October 2018 in Pubmed/Medline and Embase. Studies were included if they evaluated a model of outpatient care in a specialist setting for the treatment of adult patients with an FGID and included patient-reported outcomes comprising symptoms, quality of life, or psychological well-being.

RESULTS

Few studies have evaluated the delivery of care for the whole population of referred patients with an FGID, and there was one randomized comparison of different models of care. Two studies that evaluated the outcome of gastroenterologist-only clinics suggested poor long-term results. Two non-comparative case series reported the outcome of multidisciplinary care, including gastroenterologists and psychological therapists, suggesting improved patient quality of life and psychological well-being.

CONCLUSIONS

Despite the high prevalence and cost of treating FGIDs, and the availability of effective treatments, there are few data and limited randomized comparisons reporting the outcome of different types of specialist care. The few data available suggest that multidisciplinary care is superior to gastroenterologist-only care, but this needs to be validated in prospective comparative studies.

Publication bias in diagnostic imaging: conference abstracts with positive conclusions are more likely to be published

OBJECTIVE

To evaluate whether imaging diagnostic test accuracy conference abstracts with positive conclusions or titles are more likely to reach full-text publication than those with negative (or neutral) conclusions or titles.

METHODS

Diagnostic accuracy research abstracts were included if they were presented at the 2011 or 2012 Radiological Society of North America conference. Full-text publication status at 5 years post conference abstract submission was determined. Conclusion and title positivity of conference abstracts were extracted, as well as potential confounding factors. The associations of conclusion and title positivity with publication status at 5 years post conference abstract submission were assessed using a multivariable logistic regression model. Conditional odds ratios were calculated to express the strength of associations, adjusting for the confounders.

RESULTS

In total, 282/400 (71%) of included conference abstracts reached full-text publication. A total of 246 out of 337 (74%) conference abstracts with positive conclusions resulted in full-text publications, compared with 26/48 (54%) with neutral conclusions and 5/15 (33%) with negative conclusions. In multivariable logistic regression, conclusion positivity was significantly associated with full-text publication (odds ratio 3.6; 95% CI 1.9-6.7 for conference abstracts with positive conclusions, compared with those with non-positive conclusions); this did not apply to title positivity (odds ratio 1.2; 95% CI 0.47-3.0).

CONCLUSION

Imaging conference abstracts with positive conclusions were more likely to be published as full-text articles. Title positivity was not associated with publication. This preferential publication pattern may lead to an overrepresentation of positive studies in the literature. An overrepresentation of positive studies may contribute to inflated estimates of test accuracy and has the potential to adversely influence patient care.

KEY POINTS

• Imaging diagnostic test accuracy conference abstracts with positive conclusions were more likely to be reported as full-text articles than those with non-positive conclusions. • The majority (75%) of imaging diagnostic test accuracy conference abstracts with positive conclusions were published, compared with only 53% and 33% with neutral and negative conclusions, respectively. • Conclusion positivity remained associated with the full-text publication of conference abstracts when controlling for multiple potential confounding variables.

What happens to intimate partner violence studies registered on clinicaltrials.gov? A systematic review of a clinical trials registry

Background

There is an increasing number of interventions aimed at reducing the incidence and improving the identification and management of intimate partner violence (IPV), which are being tested in randomized clinical trials. Publication bias, improper reporting, and selective reporting in clinical trials have led to widespread adoption of pre-registration of clinical trials. Non-publication of study results leads to inefficiency, ethical issues, and scientific issues with the IPV literature. When study results and methodology are not made available through publication or other public means, the results cannot be used to their full potential. The objective of this study was to determine the publication rates of IPV trials registered in a large clinical trial registry.

Methods

We conducted a systematic review of all IPV-related clinicaltrials.gov records and determined whether the studies that had been completed for ≥ 18 months have been published in a peer-reviewed journal or in the clinicaltrials.gov registry. Two authors extensively searched the literature and contacted study investigators to locate full-text publications for each included study.

Results

Of 83 completed IPV-related trials registered on clinicaltrials.gov, 64 (77.1%, 95% CI: 66.6–85.6) were subsequently published in full-text form. Of the 19 unpublished studies, authors confirmed that there was no publication for 11 studies; we were unable to contact the investigator or locate a publication for the remaining eight studies. Only four studies (all published) posted their results on clinicaltrials.gov upon completion.

Conclusion

Approximately one in four IPV trials are not published 18 months after completion, indicating that clinicians, researchers, and other evidence users should consider whether publication bias might affect their interpretation of the IPV literature. Further research is warranted to understand reasons for non-publication of IPV research and methods to improve publication rates.

Common Topics of Publication and Levels of Evidence in the Current Hand Surgery Literature

Background Scientific publications are the primary vehicle for the distribution of scientific findings, but there has been limited research on literature topic surveillance. We sought to identify and characterize the most commonly published topic domains in the hand surgery literature. Methods We performed a 6-month hypothesis testing phase to identify the most frequently published topics in three hand surgery journals: Hand, The Journal of Hand Surgery (American), and The Journal of Hand Surgery (European). We reviewed all of the published articles in these journals from June 2010 to May 2015 to identify and characterize publications related to the three most common topic domains. Results A total of 2,146 articles were published during the 5-year study period. The three most frequent topics domains included distal radius (DR) (11% of all articles), flexor tendon (FT) (9%), and carpal tunnel (CT) (7.5%). These subjects accounted for a total of 584 articles (27% of all publications) and 3,014 published pages during the study period. FT, CT, and DR publications were cited on average 2.3 times per year (2.5, 2.4, and 2.0, respectively). Conclusion A small subset of topic domains makes up a significant proportion of scientific publications in hand surgery.

Changes in the Use of Comprehensive Geriatric Assessment in Clinical Trials for Older Patients with Cancer over Time

BACKGROUND

The objective of this study was to describe the implementation of comprehensive geriatric assessment (CGA) in clinical trials dedicated to older patients before and after the creation of the International Society of Geriatric Oncology in the early 2000s.

SUBJECTS, MATERIALS, AND METHODS

All phase I, II, and III trials dedicated to the treatment of cancer among older patients published between 2001 and 2004 and between 2011 and 2014 were reviewed. We considered that a CGA was performed when the authors indicated an intention to do so in the Methods section of the article. We collected each geriatric domain assessed using a validated tool even in the absence of a clear CGA, including nutritional, functional, cognitive, and psychological status, comorbidity, comedication, overmedication, social status and support, and geriatric syndromes.

RESULTS

A total of 260 clinical trials dedicated to older patients were identified over the two time periods: 27 phase I, 193 phase II, and 40 phase III trials. CGA was used in 9% and 8% of phase II and III trials, respectively; it was never used in phase I trials. Performance status was reported in 67%, 79%, and 75% of phase I, II, and III trials, respectively. Functional assessment was reported in 4%, 11%, and 13% of phase I, II, and III trials, respectively. Between the two time periods, use of CGA increased from 1% to 11% (p = .0051) and assessment of functional status increased from 3% to 14% (p = .0094).

CONCLUSION

The use of CGA in trials dedicated to older patients increased significantly but remained insufficient.

IMPLICATIONS FOR PRACTICE

This article identifies the areas in which research efforts should be focused in order to offer physicians well-addressed clinical trials with results that can be extrapolated to daily practice.

Research integrity and the law that never was

Evidence-based medicine is arguably among the most important innovations of the modern era, but publication bias and inadequate research transparency are serious issues affecting the very foundation of evidence-based practice. Despite this truth, these crucial issues have gone largely unaddressed or inadequately addressed for a distressingly long period of time. Regulatory efforts have thus far proven insufficient in eliminating these issues. Fortunately, the last 5 years in particular have seen developments that one hopes will contribute to the eradication of these issues and a future where we can look back on these issues as a sordid story of our past. However, much like the purported fixes of the past, time will be the final arbiter of the efficacy of remedial measures currently underway. This article chronicles the history of these issues, failed attempts to fix these issues, and what can be and is being done with the hope of bringing about true resolution.

Early phase and adaptive design clinical trials in rheumatoid arthritis: a systematic review of early phase trials

Objective

Adaptive designs can enable highly sophisticated and efficient early phase trials, but the clinical inference from these trials is surrounded by complexity, and currently there is a paucity but steadily increasing amount of use of these designs in all fields of medicine. We aim to review early phase trials in RA to discover those that have used adaptive designs and benchmark trial characteristics.

Methods

From an OVID search for journal articles reporting the results of early phase trials in rheumatology, 35 studies were found, with 9 subsequently excluded; 11 were added from manual searches and 19 from searching the references. Study characteristics were extracted from the 56 papers (describing 62 trials), including the number of arms, number of patients, the primary outcome and when it was measured.

Result

One early phase trial using an adaptive design was found. The benchmark early phase trial in RA is a phase II double-blinded randomized trial, with four arms (one control and three intervention), each with 34 patients, and ACR20 measured at 16 weeks as the primary outcome.

Conclusion

The one adaptive design reviewed here, and a simulation study found in the search, both indicate that adaptive designs can be applied to early phase trials in RA. We have described the benchmark, which the efficiency of early phase trials using an adaptive design needs to exceed. These efficient designs could drive down numbers required, time for data collection and thus cost. Changes have been suggested, but more needs to be done.

Dealing with the positive publication bias: Why you should really publish your negative results

Studies with positive results are greatly more represented in literature than studies with negative results, producing so-called publication bias. This review aims to discuss occurring problems around negative results and to emphasize the importance of reporting negative results. Underreporting of negative results introduces bias into meta-analysis, which consequently misinforms researchers, doctors and policymakers. More resources are potentially wasted on already disputed research that remains unpublished and therefore unavailable to the scientific community. Ethical obligations need to be considered when reporting results of studies on human subjects as people have exposed themselves to risk with the assurance that the study is performed to benefit others. Some studies disprove the common conception that journal editors preferably publish positive findings, which are considered as more citable. Therefore, all stakeholders, but especially researchers, need to be conscious of disseminating negative and positive findings alike.

Bias in cervical total disc replacement trials

PURPOSE OF REVIEW

Cervical disc replacement (CDR) has emerged as a motion-preserving alternative to anterior cervical discectomy and fusion in selected cases. Despite favorable literature, CDR is not universally accepted because of concerns regarding bias in the existing literature. The purpose of this review is to identify the possible biases in the disc replacement literature.

RECENT FINDINGS

Recent studies that compare CDR and ACDF have demonstrated equivalent or superior outcomes, lower rates of secondary surgery, and equivalent safety at medium- and long-term follow-up. In our review, we identified four types of bias that may affect the CDR literature: publication bias, external validity, confounding bias, and financial conflicts of interest. Bias, whether intentional or unintentional, can impact the interpretation and outcome of CDR studies. Recognition of this issue is critical when utilizing the existing literature to determine the efficacy of CDR and designing future studies.

Industry sponsorship and research outcome

BACKGROUND

Clinical research affecting how doctors practice medicine is increasingly sponsored by companies that make drugs and medical devices. Previous systematic reviews have found that pharmaceutical-industry sponsored studies are more often favorable to the sponsor's product compared with studies with other sources of sponsorship. A similar association between sponsorship and outcomes have been found for device studies, but the body of evidence is not as strong as for sponsorship of drug studies. This review is an update of a previous Cochrane review and includes empirical studies on the association between sponsorship and research outcome.

OBJECTIVES

To investigate whether industry sponsored drug and device studies have more favorable outcomes and differ in risk of bias, compared with studies having other sources of sponsorship.

SEARCH METHODS

In this update we searched MEDLINE (2010 to February 2015), Embase (2010 to February 2015), the Cochrane Methodology Register (2015, Issue 2) and Web of Science (June 2015). In addition, we searched reference lists of included papers, previous systematic reviews and author files.

SELECTION CRITERIA

Cross-sectional studies, cohort studies, systematic reviews and meta-analyses that quantitatively compared primary research studies of drugs or medical devices sponsored by industry with studies with other sources of sponsorship. We had no language restrictions.

DATA COLLECTION AND ANALYSIS

Two assessors screened abstracts and identified and included relevant papers. Two assessors extracted data, and we contacted authors of included papers for additional unpublished data. Outcomes included favorable results, favorable conclusions, effect size, risk of bias and whether the conclusions agreed with the study results. Two assessors assessed risk of bias of included papers. We calculated pooled risk ratios (RR) for dichotomous data (with 95% confidence intervals (CIs)).

MAIN RESULTS

Twenty-seven new papers were included in this update and in total the review contains 75 included papers. Industry sponsored studies more often had favorable efficacy results, RR: 1.27 (95% CI: 1.17 to 1.37) (25 papers) (moderate quality evidence), similar harms results RR: 1.37 (95% CI: 0.64 to 2.93) (four papers) (very low quality evidence) and more often favorable conclusions RR: 1.34 (95% CI: 1.19 to 1.51) (29 papers) (low quality evidence) compared with non-industry sponsored studies. Nineteen papers reported on sponsorship and efficacy effect size, but could not be pooled due to differences in their reporting of data and the results were heterogeneous. We did not find a difference between drug and device studies in the association between sponsorship and conclusions (test for interaction, P = 0.98) (four papers). Comparing industry and non-industry sponsored studies, we did not find a difference in risk of bias from sequence generation, allocation concealment, follow-up and selective outcome reporting. However, industry sponsored studies more often had low risk of bias from blinding, RR: 1.25 (95% CI: 1.05 to 1.50) (13 papers), compared with non-industry sponsored studies. In industry sponsored studies, there was less agreement between the results and the conclusions than in non-industry sponsored studies, RR: 0.83 (95% CI: 0.70 to 0.98) (six papers).

AUTHORS' CONCLUSIONS

Sponsorship of drug and device studies by the manufacturing company leads to more favorable efficacy results and conclusions than sponsorship by other sources. Our analyses suggest the existence of an industry bias that cannot be explained by standard 'Risk of bias' assessments.

The ethics of reporting all the results of clinical trials

INTRODUCTION OR BACKGROUND

The terms 'publication bias' and 'reporting bias' describe aspects of a phenomenon by which data from trials are not publicized, and so remain inaccessible. This may generate a false impression about the world; but those facts may have important implications for clinical decisions. Thus, the bias may leave patients worse off than they might be.

SOURCES OF DATA

Published journal articles.

AREAS OF AGREEMENT

There is general agreement that the phenomenon happens, and that to the extent that it happens, it is undesirable for moral rather than simply epistemic reasons.

GROWING POINTS

There is a growing demand across the board for data to be better publicized.

AREAS TIMELY FOR DEVELOPING RESEARCH

There is room for further work on how protocols requiring that data be publicized might be enforced; should it be statutory, or non-statutory? Who should decide what should be made public? There is also room for work on what it is necessary to share, and on whether and how IP law should be reformed.

Variables affecting evaluation and publication of oncology case reports: a systematic analysis

BACKGROUND

Studies on factors affecting editorial decisions of scientific journals are scarce. In this study, we focused on case reports submitted to oncology journals and analyzed whether their nature or other relevant variables affected the chances of their acceptance.

METHODS

We analyzed case reports submitted to 2 oncology journals: Tumori Journal and The International Journal of Biological Markers, and split them into 3 predefined groups: those (a) describing rare or unusual presentation of diseases, (b) describing the side effects of an intervention or (c) describing the success of a novel intervention. Publication status was retrospectively retrieved from the submission system, and acceptance rates were calculated taking into account other variables including geographic location of corresponding author.

RESULTS

A total of 326 case reports were suitable for analysis. The acceptance rate was 35.4% for group (a), 27.9% for group (b), 19.6% for group (c) (p = 0.01). After correcting for other variables, the odds ratio (OR) of being accepted for group (c) was 0.58 (95% CI, 0.33-1.00) compared with the other groups combined. There was a highly significant difference of acceptance rates between manuscripts with authors coming from developed vs. developing countries that remained significant (OR = 5.94; 95% CI, 3.05-10.09) after correcting for multiple variables.

CONCLUSIONS

The nature of a case report in oncology may affect acceptance rate, with case reports describing successful approaches or side effects of treatment being accepted with a higher frequency then case reports describing a rare clinical or diagnostic scenario. Also, works coming from developed countries are accepted significantly more frequently than case reports coming from developing countries.

No difference found in time to publication by statistical significance of trial results: a methodological review

Objective

Time-lag from study completion to publication is a potential source of publication bias in randomised controlled trials. This study sought to update the evidence base by identifying the effect of the statistical significance of research findings on time to publication of trial results.

Design

Literature searches were carried out in four general medical journals from June 2013 to June 2014 inclusive (BMJ, JAMA, the Lancet and the New England Journal of Medicine).

Setting

Methodological review of four general medical journals.

Participants

Original research articles presenting the primary analyses from phase 2, 3 and 4 parallel-group randomised controlled trials were included.

Main outcome measures

Time from trial completion to publication.

Results

The median time from trial completion to publication was 431 days (n = 208, interquartile range 278–618). A multivariable adjusted Cox model found no statistically significant difference in time to publication for trials reporting positive or negative results (hazard ratio: 0.86, 95% CI 0.64 to 1.16, p = 0.32).

Conclusion

In contrast to previous studies, this review did not demonstrate the presence of time-lag bias in time to publication. This may be a result of these articles being published in four high-impact general medical journals that may be more inclined to publish rapidly, whatever the findings. Further research is needed to explore the presence of time-lag bias in lower quality studies and lower impact journals.

Publications presenting negative impacts of pharmacists

OBJECTIVES

The main objective was to evaluate the rate of publications with at least one indicator of the negative impact of clinical pharmaceutics activity.

METHODS

This is a descriptive and retrospective literature review. A literature search was conducted using Pubmed. Articles published between 2009-2014 that described the role and impacts of pharmacists were included. We calculated the rate of publication containing at least one negative indicator. We collected the indicators with negative results.

RESULTS

A total of 203 articles were included. Nine articles (4%) that had at least one indicator of negative impact were identified. A total of 66% (6/9) were conducted in the United States. The study designs of the articles included were a meta-analysis (n=1), a systematic review (n=1), randomized studies (n=2), pre-post studies (n=3), a cohort study (n=1) and a survey (n=1). Nine indicators of negative impact were identified.

CONCLUSION

There were nine publications with at least one negative indicator of the impact of clinical pharmacy activity. While there are a large number of studies about the positive impact of clinical pharmacy activities; the publication of negative results should be encouraged.

Editorial behaviors in peer review

Editors play a critical role in the peer review system. How do editorial behaviors affect the performance of peer review? No quantitative model to date allows us to measure the influence of editorial behaviors on different peer review stages such as, manuscript distribution and final decision making. Here, we propose an agent-based model in which the process of peer review is guided mainly by the social interactions among three kinds of agents representing authors, editors and reviewers respectively. We apply this model to analyze a number of editorial behaviors such as decision strategy, number of reviewers and editorial bias on peer review. We find out that peer review outcomes are significantly sensitive to different editorial behaviors. With a small fraction (10 %) of biased editors, the quality of accepted papers declines 11 %, which indicates that effects of editorial biased behavior is worse than that of biased reviewers (7 %). While several peer review models exist, this is the first account for the study of editorial behaviors that is validated on the basis of simulation analysis.

Publication outcome of abstracts presented at the AACC annual meeting

BACKGROUND

Failure to publish study results causes duplication of effort and is a significant source of waste. It also can lead to distortions in the evidence base that can lead to misallocation of resources and medical harm. Failure to publish is commonly studied by comparing the conversion rate of meeting abstracts or publication rate of registered trials and has not been studied in clinical chemistry. The objective of this study was to determine the abstract conversion rate in clinical chemistry.

METHODS

For the set of abstracts published from the 2011 annual meeting of the American Association for Clinical Chemistry, we determined which converted to full publications and which had not. We used 3 methods to match publications to abstracts: 1) a survey sent to corresponding authors of abstracts, 2) a web scrape of Google Scholar, and PubMed, and 3) a manual search using Scopus. Publication rates were compared by topic, country of corresponding author, institution type, and award recognition.

RESULTS

Matching publications were found for 38% (95% CI: 34-42%) of the abstracts. The acceptance rate for submitted manuscripts was 34% (95% CI: 28-43%) among those who responded to the survey. Publication rates varied by topic (range 13% to 59%); rates from academic institutions were higher than commercial institutions (42% vs 16%, p<0.001). The publication rate of abstracts recognized "with distinction" was significantly greater than the publication rate of non-winners (68% vs 37%, p=0.001).

CONCLUSION

A significant proportion of abstracts presented at the AACC national meeting are not followed by full publication.

The Validity of Conscientiousness Is Overestimated in the Prediction of Job Performance

Introduction

Sensitivity analyses refer to investigations of the degree to which the results of a meta-analysis remain stable when conditions of the data or the analysis change. To the extent that results remain stable, one can refer to them as robust. Sensitivity analyses are rarely conducted in the organizational science literature. Despite conscientiousness being a valued predictor in employment selection, sensitivity analyses have not been conducted with respect to meta-analytic estimates of the correlation (i.e., validity) between conscientiousness and job performance.

Methods

To address this deficiency, we reanalyzed the largest collection of conscientiousness validity data in the personnel selection literature and conducted a variety of sensitivity analyses.

Results

Publication bias analyses demonstrated that the validity of conscientiousness is moderately overestimated (by around 30%; a correlation difference of about .06). The misestimation of the validity appears to be due primarily to suppression of small effects sizes in the journal literature. These inflated validity estimates result in an overestimate of the dollar utility of personnel selection by millions of dollars and should be of considerable concern for organizations.

Conclusion

The fields of management and applied psychology seldom conduct sensitivity analyses. Through the use of sensitivity analyses, this paper documents that the existing literature overestimates the validity of conscientiousness in the prediction of job performance. Our data show that effect sizes from journal articles are largely responsible for this overestimation.

Peer review comments on drug trials submitted to medical journals differ depending on sponsorship, results and acceptance: a retrospective cohort study

OBJECTIVE

During peer review, submitted manuscripts are scrutinised by independent experts to assist journal editors in their decision-making and to help improve the quality of articles. In this retrospective cohort study, peer review comments for drug trials submitted to medical journals were analysed to investigate whether there is a relation between the content of these comments and sponsorship, direction of results and decision about acceptance.

DESIGN/SETTING

Descriptive content analysis of reviewer comments made on manuscripts on drug trials submitted to eight medical journals (January 2010-April 2012). For each manuscript, the number of reviewers, decision about acceptance, sponsorship and direction of results were extracted. Reviewer comments were classified using a predefined checklist.

RESULTS

Reviewer reports for 246 manuscripts were assessed. Industry-sponsored trials were more likely to receive comments about lack of novelty (8.9%) than industry-supported (2.5%) and non-industry trials (6.1%, overall p=0.038). Non-industry trials more often received comments about poor experimental design (69.7%) than industry-supported (58.8%) and industry-sponsored trials (52.9%, overall p=0.019). Non-industry trials were also more likely to receive comments regarding inappropriate statistical analyses (28.4%) than industry-supported (23.5%) and industry-sponsored trials (15.1%, overall p=0.006). Manuscripts with negative results were more likely to receive comments about inappropriate conclusions (29.3%) than those with positive results (18.9%, p=0.010). Rejected manuscripts had more often received comments on the research question not being clinically relevant (7.8%) than accepted manuscripts (1.6%, p=0.002), and also on lack of novelty (8.3% vs 2.6%, p=0.008) and poor experimental design (68.6% vs 50.5%, p<0.001).

CONCLUSIONS

Reviewers identified fewer shortcomings regarding design and statistical analyses in industry-related trials, but commented more often on a lack of novelty in industry-sponsored trials. Negative trial results did not significantly influence the nature of comments other than appropriateness of the conclusion. Manuscript acceptance was primarily related to the research question and methodological robustness of studies.

Questioning the dichotomy between vegetative state and minimally conscious state: a review of the statistical evidence

Given the enormous consequences that the diagnosis of vegetative state (VS) vs. minimally conscious state (MCS) may have for the treatment of patients with disorders of consciousness, it is particularly important to empirically legitimate the distinction between these two discrete levels of consciousness. Therefore, the aim of this contribution is to review all the articles reporting statistical evidence concerning the performance of patients in VS vs. patients in MCS, on behavioral or neurophysiological measures. Twenty-three articles matched these inclusion criteria, and comprised behavioral, electroencephalographic (EEG), positron emission tomography (PET) and magnetic resonance imaging (MRI) measures. The analysis of these articles yielded 47 different statistical findings. More than half of these findings (n = 24) did not reveal any statistically significant difference between VS and MCS. Overall, there was no combination of variables that allowed reliably discriminating between VS and MCS. This pattern of results casts doubt on the empirical validity of the distinction between VS and MCS.