CONSORT-TM: Text classification models for assessing the completeness of randomized controlled trial publications

Objective

To develop text classification models for determining whether the checklist items in the CONSORT reporting guidelines are reported in randomized controlled trial publications.

Materials and Methods

Using a corpus annotated at the sentence level with 37 fine-grained CONSORT items, we trained several sentence classification models (PubMedBERT fine-tuning, BioGPT fine-tuning, and in-context learning with GPT-4) and compared their performance. To address the problem of small training dataset, we used several data augmentation methods (EDA, UMLS-EDA, text generation and rephrasing with GPT-4) and assessed their impact on the fine-tuned PubMedBERT model. We also fine-tuned PubMedBERT models limited to checklist items associated with specific sections (e.g., Methods) to evaluate whether such models could improve performance compared to the single full model. We performed 5-fold cross-validation and report precision, recall, F1 score, and area under curve (AUC).

Results

Fine-tuned PubMedBERT model that takes as input the sentence and the surrounding sentence representations and uses section headers yielded the best overall performance (0.71 micro-F1, 0.64 macro-F1). Data augmentation had limited positive effect, UMLS-EDA yielding slightly better results than data augmentation using GPT-4. BioGPT fine-tuning and GPT-4 in-context learning exhibited suboptimal results. Methods-specific model yielded higher performance for methodology items, other section-specific models did not have significant impact.

Conclusion

Most CONSORT checklist items can be recognized reasonably well with the fine-tuned PubMedBERT model but there is room for improvement. Improved models can underpin the journal editorial workflows and CONSORT adherence checks and can help authors in improving the reporting quality and completeness of their manuscripts.

Establishing Institutional Scores With the Rigor and Transparency Index: Large-scale Analysis of Scientific Reporting Quality

BACKGROUND

Improving rigor and transparency measures should lead to improvements in reproducibility across the scientific literature; however, the assessment of measures of transparency tends to be very difficult if performed manually.

OBJECTIVE

This study addresses the enhancement of the Rigor and Transparency Index (RTI, version 2.0), which attempts to automatically assess the rigor and transparency of journals, institutions, and countries using manuscripts scored on criteria found in reproducibility guidelines (eg, Materials Design, Analysis, and Reporting checklist criteria).

METHODS

The RTI tracks 27 entity types using natural language processing techniques such as Bidirectional Long Short-term Memory Conditional Random Field-based models and regular expressions; this allowed us to assess over 2 million papers accessed through PubMed Central.

RESULTS

Between 1997 and 2020 (where data were readily available in our data set), rigor and transparency measures showed general improvement (RTI 2.29 to 4.13), suggesting that authors are taking the need for improved reporting seriously. The top-scoring journals in 2020 were the Journal of Neurochemistry (6.23), British Journal of Pharmacology (6.07), and Nature Neuroscience (5.93). We extracted the institution and country of origin from the author affiliations to expand our analysis beyond journals. Among institutions publishing >1000 papers in 2020 (in the PubMed Central open access set), Capital Medical University (4.75), Yonsei University (4.58), and University of Copenhagen (4.53) were the top performers in terms of RTI. In country-level performance, we found that Ethiopia and Norway consistently topped the RTI charts of countries with 100 or more papers per year. In addition, we tested our assumption that the RTI may serve as a reliable proxy for scientific replicability (ie, a high RTI represents papers containing sufficient information for replication efforts). Using work by the Reproducibility Project: Cancer Biology, we determined that replication papers (RTI 7.61, SD 0.78) scored significantly higher (P<.001) than the original papers (RTI 3.39, SD 1.12), which according to the project required additional information from authors to begin replication efforts.

CONCLUSIONS

These results align with our view that RTI may serve as a reliable proxy for scientific replicability. Unfortunately, RTI measures for journals, institutions, and countries fall short of the replicated paper average. If we consider the RTI of these replication studies as a target for future manuscripts, more work will be needed to ensure that the average manuscript contains sufficient information for replication attempts.

Increased transparency was required when reporting imputation of primary outcome data in clinical trials

OBJECTIVE

To explore the transparency of reporting primary outcome data within randomised controlled trials (RCTs) in the presence of missing data.

STUDY DESIGN

/ Setting: A cohort examination of RCTs published in the four major medical journals (NEJM, JAMA, BMJ, Lancet) in 2013 and the first quarter of 2018. Data was extracted on reporting quality, the number of randomised participants and the number of participants included within the primary outcome analysis with observed or imputed data.

RESULTS

91/159 (57%) of the studies analysed from 2013 and 19/46 (41%) from 2018 included imputed data within the primary outcome analysis. Of these, only 13/91 (14%) studies from 2013 and 1/19 (5%) studies from 2018 explicitly reported the number of imputed values in the CONSORT diagram. Results tables included levels of imputed data in 12/91 (13%) studies in 2013 and 4/19 (21%) in 2018. Consequently, identification of imputed data was a time-consuming task requiring extensive cross-referencing of all manuscript elements.

CONCLUSION

Imputed primary outcome data is poorly reported. Participant flow diagrams frequently reported participant status which does not necessarily correlate to availability of data. We recommended that the number of imputed values are explicitly reported within CONSORT flow diagrams to increase transparency.

Critical Appraisal of Qualitative Studies of Muslim Females' Perceptions of Physical Activity Barriers and Facilitators

Muslim women’s perceptions of cultural, religious, and secular determinants of physical activity have been studied for many years, with information typically acquired through focus groups or interviews. Multiple reviews synthesizing the research have been published, however, individual studies have not been scrutinized for their quality/rigor. Therefore, I critically appraised the quality of the body of qualitative research studies that utilized focus groups to identify Muslim women’s perceptions of physical activity barriers and facilitators. I utilized 26 items from the Consolidated Criteria for Reporting Qualitative Research (COREQ) to assess the quality of 56 papers published between 1987 and 2016. Using crosstabulations, I also examined associations between paper quality (low vs. high) and binary categorical variables for impact factor, maximum paper length allowed, publication year, and database the paper was indexed. Overall, papers averaged only 10.5 of 26 COREQ reporting criteria and only two out of 26 items were reported in more than 75% of the papers. Paper quality was not associated with impact factor and length. High quality papers were more likely published more recently (i.e., 2011 or later) and in journals indexed in the PubMed database compared to low quality papers. There is contention among qualitative researchers about standardizing reporting criteria, and while the trend in quality appears to be improving, journal reviewers and editors ought to hold authors to greater accountability in reporting.

Barriers and Facilitators to Model Replication Within Health Economics

BACKGROUND

Model replication is important because it enables researchers to check research integrity and transparency and, potentially, to inform the model conceptualization process when developing a new or updated model.

OBJECTIVE

The aim of this study was to evaluate the replicability of published decision analytic models and to identify the barriers and facilitators to replication.

METHODS

Replication attempts of 5 published economic modeling studies were made. The replications were conducted using only publicly available information within the manuscripts and supplementary materials. The replicator attempted to reproduce the key results detailed in the paper, for example, the total cost, total outcomes, and if applicable, incremental cost-effectiveness ratio reported. Although a replication attempt was not explicitly defined as a success or failure, the replicated results were compared for percentage difference to the original results.

RESULTS

In conducting the replication attempts, common barriers and facilitators emerged. For most case studies, the replicator needed to make additional assumptions when recreating the model. This was often exacerbated by conflicting information being presented in the text and the tables. Across the case studies, the variation between original and replicated results ranged from -4.54% to 108.00% for costs and -3.81% to 0.40% for outcomes.

CONCLUSION

This study demonstrates that although models may appear to be comprehensively reported, it is often not enough to facilitate a precise replication. Further work is needed to understand how to improve model transparency and in turn increase the chances of replication, thus ensuring future usability.

How transparent is behavioral intervention research on pathological gambling and other gambling-related disorders? A systematic literature review

AIMS

To review the transparency of reports of behavioral interventions for pathological gambling and other gambling-related disorders.

METHODS

We used the Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) Statement to develop the 59-question adapted TREND questionnaire (ATQ). Each ATQ question corresponds to a transparency guideline and asks how clearly a study reports its objectives, research design, analytical methods and conclusions. A subset of 23 ATQ questions is considered particularly important. We searched PubMed, PsychINFO and Web of Science to identify experimental evaluations published between 2000 and 2011 aiming to reduce problem gambling behaviors or decrease problems caused by gambling. Twenty-six English-language reports met the inclusion criteria and were reviewed by three abstractors using the ATQ.

RESULTS

The average report adhered to 38.4 (65.1%) of the 59 ATQ transparency guidelines. Each of the 59 ATQ questions received positive responses from an average of 16.9 (63.8%) of the reports. The subset of 23 particularly relevant questions received an average of 15.3 (66.5%) positive responses. Thirty-two of 59 (54%) ATQ questions were answered positively by 75% or more of the study reports, while 12 (20.3%) received positive responses by 25% or fewer. Publication year did not affect these findings.

CONCLUSIONS

Gambling intervention reports need to improve their transparency by adhering to currently neglected and particularly relevant guidelines. Among them are recommendations for comparing study participants who are lost to follow-up and those who are retained, comparing study participants with the target population, describing methods used to minimize potential bias due to group assignment, and reporting adverse events or unintended effects.