How well are reporting guidelines and trial registration used by dermatology journals to limit bias? A meta-epidemiological study

The endorsement of general and artificial intelligence reporting guidelines in radiological journals: a meta-research study

BACKGROUND

Complete reporting is essential for clinical research. However, the endorsement of reporting guidelines in radiological journals is still unclear. Further, as a field extensively utilizing artificial intelligence (AI), the adoption of both general and AI reporting guidelines would be necessary for enhancing quality and transparency of radiological research. This study aims to investigate the endorsement of general reporting guidelines and those for AI applications in medical imaging in radiological journals, and explore associated journal characteristic variables.

METHODS

This meta-research study screened journals from the Radiology, Nuclear Medicine & Medical Imaging category, Science Citation Index Expanded of the 2022 Journal Citation Reports, and excluded journals not publishing original research, in non-English languages, and instructions for authors unavailable. The endorsement of fifteen general reporting guidelines and ten AI reporting guidelines was rated using a five-level tool: "active strong", "active weak", "passive moderate", "passive weak", and "none". The association between endorsement and journal characteristic variables was evaluated by logistic regression analysis.

RESULTS

We included 117 journals. The top-five endorsed reporting guidelines were CONSORT (Consolidated Standards of Reporting Trials, 58.1%, 68/117), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, 54.7%, 64/117), STROBE (STrengthening the Reporting of Observational Studies in Epidemiology, 51.3%, 60/117), STARD (Standards for Reporting of Diagnostic Accuracy, 50.4%, 59/117), and ARRIVE (Animal Research Reporting of In Vivo Experiments, 35.9%, 42/117). The most implemented AI reporting guideline was CLAIM (Checklist for Artificial Intelligence in Medical Imaging, 1.7%, 2/117), while other nine AI reporting guidelines were not mentioned. The Journal Impact Factor quartile and publisher were associated with endorsement of reporting guidelines in radiological journals.

CONCLUSIONS

The general reporting guideline endorsement was suboptimal in radiological journals. The implementation of reporting guidelines for AI applications in medical imaging was extremely low. Their adoption should be strengthened to facilitate quality and transparency of radiological study reporting.

Prospective registration of randomized clinical trials for total shoulder arthroplasty is low: a systematic review

BACKGROUND

Prospective trial registration has become an important means of improving the transparency and reproducibility of randomized controlled trials (RCTs) and is recommended by the Journal of Shoulder and Elbow Surgery (JSES) per the Consolidated Standards of Reporting Trials (CONSORT) guidelines. Herein, we performed a cross-sectional evaluation of RCTs published in JSES from 2010 to present to determine the prevalence of trial registration and consistency of outcome reporting.

METHODS

The electronic database PubMed was searched to identify all RCTs on total shoulder arthroplasty (TSA) published in JSES from 2010 to 2022 using the search terms "randomized controlled trial" AND "shoulder" AND "arthroplasty OR replacement." RCTs were considered to be registered if they provided a registration number. For articles that were registered, authors also extracted the registry name, registration date, date of first enrollment, date of last enrollment, and if the primary outcomes reported in the registry were either (1) omitted, (2) newly introduced in the publication, (3) reported as a secondary outcome or vice versa, or (4) varied in timing of assessment compared to the publication. "Early" RCTs were considered those published from 2010 to 2016, whereas "later" RCTs were from 2017 to 2022.

RESULTS

Fifty-eight RCTs met inclusion criteria. There were 16 early RCTs and 42 later RCTs. Twenty-three of the 58 (39.7%) studies were registered, with 9 of 22 with an available registry (40.9%) of those being enrolled prior to patient enrollment. Nineteen of the registered studies (82.6%) provided the name of the registry and a registration number. The proportion of later RCTs that were registered was not significantly different from the early RCTs (45.2% vs. 25.0%, P = .232). Seven RCTs (31.8%) had at least 1 inconsistency compared with the registry. The most common discrepancy was the timing of the assessment (ie, follow-up period) reported in the publication vs. the registry.

DISCUSSION

Although JSES recommends prospective trial registration, less than half of shoulder arthroplasty RCTs are registered and more than 30% registered trials have at least 1 inconsistency with their registry record. More rigorous review of trial registration and accuracy is necessary to limit bias in published shoulder arthroplasty RCTs.

Harms Reporting Is Inadequate in Systematic Reviews Regarding Hip Arthroscopy

Purpose

To investigate the quality of harms reporting in systematic reviews (SRs) regarding hip arthroscopy in the current literature.

Methods

In May 2022, an extensive search of 4 major databases was performed identifying SRs regarding hip arthroscopy: MEDLINE (PubMed and Ovid), EMBASE, Epistemonikos, and Cochrane Database of Systematic Reviews. A cross-sectional analysis was conducted, in which investigators performed screening and data extraction of the included studies in a masked, duplicate fashion. AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews-2) was used to assess the methodologic quality and bias of the included studies. The corrected covered area was calculated for SR dyads.

Results

A total of 82 SRs were included in our study for data extraction. Of these SRs, 37 reported under 50% of the harms criteria (37 of 82, 45.1%) and 9 did not report harms at all (9 of 82, 10.9%). A significant relation was found between completeness of harms reporting and overall AMSTAR appraisal (P = .0261), as well as whether a harm was listed as a primary or secondary outcome (P = .0001). Eight SR dyads had corrected covered areas of 50% or greater and were compared for shared harms reported.

Conclusions

In this study, we found inadequate harms reporting in most SRs concerning hip arthroscopy.

Clinical Relevance

With the magnitude of hip arthroscopic procedures being performed, adequate reporting of harms-related information in the research surrounding this treatment is essential in assessing the efficacy of the treatment. This study provides data in relation to harms reporting in SRs regarding hip arthroscopy.

Harms-related data are poorly reported among randomized controlled trials underpinning the American Academy of Orthopaedic Surgeons clinical practice guideline recommendations for rotator cuff injuries

BACKGROUND

Results produced from randomized controlled trials (RCTs) help guide clinical decision making and health policy. Therefore, it is essential that RCT outcomes- including harms (eg, adverse events)-are adequately reported such that clinicians, patients, and policy makers are equipped with all necessary information to complete risk-benefit assessment of the RCT's intervention. Here, we evaluated the quality of reporting of harms (eg, adverse events) in RCTs cited as supporting evidence for recommendations in the American Academy of Orthopaedic Surgeons (AAOS) Management of Rotator Cuff Injuries clinical practice guidelines (CPGs) using the Consolidated Standards of Reporting Trials (CONSORT) Extension for Harms Checklist.

METHODS

To quantify adherence to CONSORT Extension for Harms items, each RCT was screened for pertinent information satisfying each checklist item. Screening of CPG reference sections for RCTs underpinning CPG recommendations, as well as data extraction from each of the included RCTs, was performed in a blind and duplicate manner. Descriptive statistics-including frequencies, percentages, and 95% confidence intervals-were used to summarize overall percent adherence to checklist items. A linear regression model assessed the relationship of CONSORT Harms reporting over time.

RESULTS

Ninety-nine RCTs were included in our final sample. Fifty-seven RCTs (of 99; 57.6%) were conducted at a single center. Common funding sources included private (nonindustry) (17/99; 17.2%), private (industry) (8/99; 8.1%), and public (7/99; 7.1%) sources. Sample size for each trial most often consisted of <50 participants (29/99; 29.3%) or 51-100 participants (50/99; 50.5%). The average number of CONSORT Extension for Harms items adequately reported across all included RCTs was 5.7 (of 18; 31.7%). None of the included trials reported all 18 items. Twenty-six RCTs (of 99; 26.3%) adequately reported ≥50% of eligible checklist items. Fifty-nine RCTs (of 99; 59.6%) adequately reported ≤33% of eligible checklist items. Items with ≥50% adherence included item 2, item 7a, and item 8a. Items with ≤20% adherence included item 3b, item 4d, and item 5. Results from our linear regression demonstrated a slight, yet nonsignificant, improvement in adherence to the Harms Extension over time (R² = 0.009; P = .407).

CONCLUSIONS

Our results illustrate the poor state of harms reporting within RCTs cited as supporting evidence for the AAOS Management of Rotator Cuff Injuries CPG. Efforts to address these gaps in reporting are warranted, as complete knowledge of potential harms is critical to patients, clinicians, and health policy makers when determining best practice decisions in orthopedic surgery.

Evaluating publication bias for clinical trials supporting new dermatologic drug approvals from 2003 to 2018

The degree of publication bias and impact of the Food and Drug Administration Amendments Act (FDAAA) of 2007, which aimed to improve clinical trial transparency, has yet to be examined for recent dermatologic drugs. The objective of our study was to estimate the degree of publication bias for clinical trials supporting FDA approval of new dermatologic drugs. This retrospective cohort study examined all phase II and III efficacy trials supporting approval of new dermatologic drugs from 2003 to 2018. FDA drug approval documents were reviewed for supportive clinical trial information, and publications were matched using PubMed and Google Scholar searches. Ratios of relative risks (RRR) comparing positive versus non-positive trials before and after FDAAA enactment served to estimate publication bias. We found that the likelihood of publishing positive versus non-positive drug trials in dermatology was unchanged before and after FDAAA enactment (RRR 0.87, 95% CI 0.37-2.08), as was the likelihood of publishing without misleading interpretation (RRR 1.51, 95% CI 0.22-10.50). There was no measurable publication bias for efficacy trials supporting new drug approvals in dermatology over the past 15 years. Fewer pre-FDAAA trials (n = 21) compared to post-FDAAA trials (n = 106) met inclusion criteria. Though not analyzed in this study, safety and secondary efficacy results are other potential sources for publication bias.

Poor compliance of clinical trial registration among trials included in systematic reviews: a cohort study

OBJECTIVES

The objective of the study was to examine whether clinical trials that have been included in systematic reviews have been registered in clinical trial registers and, when they have, whether results of the trials were included in the clinical trial register.

STUDY DESIGN AND SETTING

This study used a sample of 100 systematic reviews published by the Cochrane Musculoskeletal, Oral, Skin and Sensory Network between 2014 and 2019.

RESULTS

We identified 2,000 trials (369,778 participants) from a sample of 100 systematic reviews. The median year of trial publication was 2007. Of 1,177 trials published in 2005 or later, a clinical trial registration record was identified for 368 (31%). Of these registered trials, 135 (37%) were registered prospectively and results were posted for 114 (31%); most registered trials evaluated pharmaceutical interventions (62%). Of trials published in the last 10 years, the proportion of registered trials increased to 38% (261 of 682).

CONCLUSION

Although some improvement in clinical trial registration has been observed in recent years, the proportion of registered clinical trials included in recently published systematic reviews remains less than desirable. Prospective clinical trial registration provides an essential role in assessing the risk of bias and judging the quality of evidence in systematic reviews of intervention safety and effectiveness.

Transparent, Reproducible, and Open Science Practices of Published Literature in Dermatology Journals: Cross-Sectional Analysis

Background

Reproducible research is a foundational component for scientific advancements, yet little is known regarding the extent of reproducible research within the dermatology literature.

Objective

This study aimed to determine the quality and transparency of the literature in dermatology journals by evaluating for the presence of 8 indicators of reproducible and transparent research practices.

Methods

By implementing a cross-sectional study design, we conducted an advanced search of publications in dermatology journals from the National Library of Medicine catalog. Our search included articles published between January 1, 2014, and December 31, 2018. After generating a list of eligible dermatology publications, we then searched for full text PDF versions by using Open Access Button, Google Scholar, and PubMed. Publications were analyzed for 8 indicators of reproducibility and transparency—availability of materials, data, analysis scripts, protocol, preregistration, conflict of interest statement, funding statement, and open access—using a pilot-tested Google Form.

Results

After exclusion, 127 studies with empirical data were included in our analysis. Certain indicators were more poorly reported than others. We found that most publications (113, 88.9%) did not provide unmodified, raw data used to make computations, 124 (97.6%) failed to make the complete protocol available, and 126 (99.2%) did not include step-by-step analysis scripts.

Conclusions

Our sample of studies published in dermatology journals do not appear to include sufficient detail to be accurately and successfully reproduced in their entirety. Solutions to increase the quality, reproducibility, and transparency of dermatology research are warranted. More robust reporting of key methodological details, open data sharing, and stricter standards journals impose on authors regarding disclosure of study materials might help to better the climate of reproducible research in dermatology.

Endorsement of reporting guidelines and study registration by endocrine and internal medicine journals: meta-epidemiological study

OBJECTIVES

To improve the trustworthiness of evidence, studies should be prospectively registered and research reports should adhere to existing standards. We aimed to systematically assess the degree to which endocrinology and internal medicine journals endorse study registration and reporting standards for randomised controlled trials (RCTs), systematic reviews (SRs) and observational studies (ObS). Additionally, we evaluated characteristics that predict endorsement of reporting or registration mechanism by these journals.

DESIGN

Meta-epidemiological study.

SETTING

Journals included in the 'Endocrinology and Metabolism' and 'General and Internal Medicine' 2017 Journal Citation Reports.

PARTICIPANTS

Journals with an impact factor of ≥1.0, focused on clinical medicine, and those who publish RCTs, SRs and ObS were included.

PRIMARY OUTCOMES

Requirement of adherence to reporting guideline and study registration as determined from the journals' author instructions.

RESULTS

Of the 170 (82 endocrinology and 88 internal medicine) eligible journals, endorsing of reporting standards was the highest for RCTs, with 35 (43%) of endocrine journals and 55 (63%) of internal medicine journals followed by SRs, with 21 (26%) and 48 (55%), respectively, and lastly, by ObS with 41 (50%) of endocrine journals and 21 (24%) of internal medicine journals. In 78 (46%) journals RCTs were required to be registered and published in adherence to the Consolidated Standards of Reporting Trials statement. Only 11 (6%) journals required registration of SRs. Internal medicine journals were more likely to endorse reporting guidelines than endocrine journals except for Strengthening the Reporting of Observational Studies in Epidemiology. No other journal characteristic proved to be an independent predictor of reporting standard endorsement for RCTs besides trial registration.

CONCLUSION

Our results highlight that study registration requirement and reporting guideline endorsement are suboptimal in internal medicine and endocrine journals. This malpractice may be further enhanced since endorsement does not imply enforcement, impairing the practice of evidence-based medicine.