Time to consider sharing data extracted from trials included in systematic reviews

Data-Sharing Across Otolaryngology: Comparing Journal Policies and Their Adherence to the FAIR Principles

OBJECTIVE

Data-sharing plays an essential role in advancing scientific understanding. Here, we aim to identify the commonalities and differences in data-sharing policies endorsed by otolaryngology journals and to assess their adherence to the FAIR (findable, accessible, interoperable, reusable) principles.

METHODS

Data-sharing policies were searched for among 111 otolaryngology journals, as listed by Scimago Journal & Country Rank. Policy extraction of the top biomedical journals as ranked by Google Scholar metrics were used as a comparison. The FAIR principles for scientific data management and stewardship were used for the extraction framework. This occurred in a blind, masked, and independent fashion.

RESULTS

Of the 111 ranked otolaryngology journals, 100 met inclusion criteria. Of those 100 journals, 79 provided data-sharing policies. There was a clear lack of standardization across policies, along with specific gaps in accessibility and reusability which need to be addressed. Seventy-two policies (of 79; 91%) designated that metadata should have globally unique and persistent identifiers. Seventy-one (of 79; 90%) policies specified that metadata should clearly include the identifier of the data they describe. Fifty-six policies (of 79; 71%) outlined that metadata should be richly described with a plurality of accurate and relevant attributes.

CONCLUSION

Otolaryngology journals have varying data-sharing policies, and adherence to the FAIR principles appears to be moderate. This calls for increased data transparency, allowing for results to be reproduced, confirmed, and debated.

Effectiveness of Exercise Interventions for Preventing Neck Pain: A Systematic Review With Meta-analysis of Randomized Controlled Trials

OBJECTIVE: To update the evidence on the effectiveness of exercise interventions to prevent episodes of neck pain. DESIGN: Systematic review with meta-analysis. LITERATURE SEARCH: MEDLINE, Embase, CENTRAL, CINAHL, SPORTDiscus, PEDro, and trial registries from inception to December 2, 2022. Forward and backward citation searches. STUDY SELECTION CRITERIA: Randomized controlled trials (RCTs) that enrolled adults without neck pain at baseline and compared exercise interventions to no intervention, placebo/sham, attention control, or minimal intervention. Military populations and astronauts were excluded. DATA SYNTHESIS: Random-effects meta-analysis. Risk of bias was assessed using the Cochrane RoB 2 tool. The certainty of evidence was judged according to the GRADE approach. RESULTS: Of 4703 records screened, 5 trials (1722 participants at baseline) were included and eligible for meta-analysis. Most (80%) participants were office workers. Risk of bias was rated as some concerns for 2 trials and high for 3 trials. There was moderate-certainty evidence that exercise interventions probably reduce the risk of a new episode of neck pain (OR, 0.49; 95% confidence interval: 0.31, 0.76) compared to no or minimal intervention in the short-term (≤12 months). The results were not robust to sensitivity analyses for missing outcome data. CONCLUSION: There was moderate-certainty evidence supporting exercise interventions for reducing the risk for an episode of neck pain in the next 12 months. The clinical significance of the effect is unclear. J Orthop Sports Phys Ther 2023;53(10):1-16. Epub: 8 September 2023. doi:10.2519/jospt.2023.12063.

Clinical trial data-sharing policies among journals, funding agencies, foundations, and other professional organizations: a scoping review

BACKGROUND AND OBJECTIVES

To identify the similarities and differences in data-sharing policies for clinical trial data that are endorsed by biomedical journals, funding agencies, and other professional organizations. Additionally, to determine the beliefs, and opinions regarding data-sharing policies for clinical trials discussed in articles published in biomedical journals.

METHODS

Two searches were conducted, a bibliographic search for published articles that present beliefs, opinions, similarities, and differences regarding policies governing the sharing of clinical trial data. The second search analyzed the gray literature (non-peer-reviewed publications) to identify important data-sharing policies in selected biomedical journals, foundations, funding agencies, and other professional organizations.

RESULTS

A total of 471 articles were included after database search and screening, with 45 from the bibliographic search and 426 from the gray literature search. A total of 424 data-sharing policies were included. Fourteen of the 45 published articles from the bibliographic search (31.1%) discussed only advantages specific to data-sharing policies, 27 (27/45; 60%) discussed both advantages and disadvantages, and 4 (4/45; 8.9%) discussed only disadvantages specific. A total of 216 journals (of 270; 80%) specified a data-sharing policy provided by the journal itself. One hundred industry data-sharing policies were included, and 32 (32%) referenced a data-sharing policy on their website. One hundred and thirty-six (42%) organizations (of 327) specified a data-sharing policy.

CONCLUSION

We found many similarities listed as advantages to data-sharing and fewer disadvantages were discussed within the literature. Additionally, we found a wide variety of commonalities and differences-such as the lack of standardization between policies, and inadequately addressed details regarding the accessibility of research data-that exist in data-sharing policies endorsed by biomedical journals, funding agencies, and other professional organizations. Our study may not include information on all data sharing policies and our data is limited to the entities' descriptions of each policy.

The benefits of data sharing and ensuring open sources of systematic review data

AIMS

The benefits of increasing public access to data from clinical trials are widely accepted. Such benefits extend to the sharing of data from high-quality systematic reviews, given the time and cost involved with undertaking reviews. We describe the application of open sources of review data, outline potential challenges and highlight efforts made to address these challenges, with the intent of encouraging publishers, funders and authors to consider sharing review data more broadly.

RESULTS

We describe the application of systematic review data in: (i) advancing understanding of clinical trials and systematic review methods, (ii) repurposing of data to answer public health policy and practice relevant questions, (iii) identification of research gaps and (iv) accelerating the conduct of rapid reviews to inform decision making. While access, logistical, motivational and legal challenges exist, there has been progress made by systematic review, academic and funding agencies to incentivise data sharing and create infrastructure to support greater access to systematic review data.

CONCLUSION

There is opportunity to maximize the benefits of research investment in undertaking systematic reviews by ensuring open sources of systematic review data. Efforts to create such systems should draw on learnings and principles outlined for sharing clinical trial data.

Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: cross sectional meta-research study

OBJECTIVES

To examine changes in completeness of reporting and frequency of sharing data, analytical code, and other review materials in systematic reviews over time; and factors associated with these changes.

DESIGN

Cross sectional meta-research study.

POPULATION

Random sample of 300 systematic reviews with meta-analysis of aggregate data on the effects of a health, social, behavioural, or educational intervention. Reviews were indexed in PubMed, Science Citation Index, Social Sciences Citation Index, Scopus, and Education Collection in November 2020.

MAIN OUTCOME MEASURES

The extent of complete reporting and the frequency of sharing review materials in the systematic reviews indexed in 2020 were compared with 110 systematic reviews indexed in February 2014. Associations between completeness of reporting and various factors (eg, self-reported use of reporting guidelines, journal policies on data sharing) were examined by calculating risk ratios and 95% confidence intervals.

RESULTS

Several items were reported suboptimally among 300 systematic reviews from 2020, such as a registration record for the review (n=113; 38%), a full search strategy for at least one database (n=214; 71%), methods used to assess risk of bias (n=185; 62%), methods used to prepare data for meta-analysis (n=101; 34%), and source of funding for the review (n=215; 72%). Only a few items not already reported at a high frequency in 2014 were reported more frequently in 2020. No evidence indicated that reviews using a reporting guideline were more completely reported than reviews not using a guideline. Reviews published in 2020 in journals that mandated either data sharing or inclusion of data availability statements were more likely to share their review materials (eg, data, code files) than reviews in journals without such mandates (16/87 (18%) v 4/213 (2%)).

CONCLUSION

Incomplete reporting of several recommended items for systematic reviews persists, even in reviews that claim to have followed a reporting guideline. Journal policies on data sharing might encourage sharing of review materials.

Many researchers were not compliant with their published data sharing statement: a mixed-methods study

OBJECTIVES

The objective of the study was to analyze researchers' compliance with their data availability statement (DAS) from manuscripts published in open-access journals with the mandatory DAS.

STUDY DESIGN AND SETTING

We analyzed all articles from 333 open-access journals published during January 2019 by BioMed Central. We categorized types of the DAS. We surveyed corresponding authors who wrote in the DAS that they would share the data. Consent to participate in the study was sought for all included manuscripts. After accessing raw data sets, we checked whether data were available in a way that enabled reanalysis.

RESULTS

Of 3556 analyzed articles, 3416 contained the DAS. The most frequent DAS category (42%) indicated that the data sets are available on reasonable request. Among 1792 manuscripts in which the DAS indicated that authors are willing to share their data, 1669 (93%) authors either did not respond or declined to share their data with us. Among 254 (14%) of 1792 authors who responded to our query for data sharing, only 123 (6.8%) provided the requested data.

CONCLUSION

Even when authors indicate in their manuscript that they will share data upon request, the compliance rate is the same as for authors who do not provide the DAS, suggesting that the DAS may not be sufficient to ensure data sharing.

Perceptions and Opinions Towards Data-Sharing: A Survey of Addiction Journal Editorial Board Members

Background

We surveyed addiction journal editorial board members to better understand their opinions towards data-sharing.

Methods

Survey items consisted of Likert-type (e.g., one to five scale), multiple-choice, and free-response questions. Journal websites were searched for names and email addresses. Emails were distributed using SurveyMonkey. Descriptive statistics were used to characterize the responses.

Results

We received 178 responses (of 1039; 17.1%). Of these, 174 individuals agreed to participate in our study (97.8%). Most respondents did not know whether their journal had a data-sharing policy. Board members "somewhat agree" that addiction journals should recommend but not require data-sharing for submitted manuscripts [M=4.09 (SD=0.06); 95% CI: 3.97-4.22]. Items with the highest perceived benefit ratings were "secondary data use (e.g., meta-analysis)" [M=3.44 (SD=0.06); 95% CI: 3.31-3.56] and "increased transparency" [M=3.29 (SD=0.07); 95% CI: 3.14-3.43]. Items perceived to be the greatest barrier to data-sharing included "lack of metadata standards" [M=3.21 (SD=0.08); 95% CI: 3.06-3.36], "no incentive" [M=3.43 (SD=0.07); 95% CI: 3.30-3.57], "inadequate resources" [M=3.53 (SD=0.05); 95% CI: 3.42-3.63], and "protection of privacy"[M=3.22 (SD=0.07); 95% CI: 3.07-3.36].

Conclusion

Our results suggest addiction journal editorial board members believe data-sharing has a level of importance within the research community. However, most board members are unaware of their journals' data-sharing policies, and most data-sharing should be recommended but not required. Future efforts aimed at better understanding common reservations and benefits towards data-sharing, as well as avenues to optimize data-sharing while minimizing potential risks, are warranted.

Methodological standards for conducting and reporting meta-analyses: Ensuring the replicability of meta-analyses of pharmacist-led medication review

BACKGROUND

Meta-analyses of clinical pharmacy services are frequently criticized for restricted data transparency and reproducibility.

OBJECTIVES

To describe the methodological characteristics of meta-analyses of pharmacist-led medication reviews, to identify the elements that limit their replicability and robustness, and to propose recommendations for an appropriate conduction and reporting.

METHODS

A meta-research study was conducted. Systematic searches of the PubMed, Scopus, and Web of Science databases were performed to identify meta-analyses of pharmacist services. Meta-analyses assessing the effect of pharmacist-led medication reviews were selected for data extraction, analysis and replication. Two replication exercises were performed for the two most common outcomes: (i) considering the data provided by authors to construct the meta-analysis and (ii) considering the raw data available in the primary studies included. Prediction intervals (PI), fragility index (FI), and number needed to treat (NNT) were also calculated for each replicated meta-analysis.

RESULTS

Nine studies reporting meta-analyses about pharmacist-led medication review were found comprising 30 different outcomes. Eleven meta-analyses, including six for hospital admission and five for mortality, were replicated. In five meta-analyses, the pooled effect sizes of the replicated meta-analyses differed from the original ones. Only four meta-analyses mentioned the statistical method used. Other meta-analytic parameters (e.g., q-value, tau2) were omitted in all studies. In nine meta-analyses, the data from primary studies had been incorrectly extracted for at least one variable. The PI demonstrated that the uncertainty intervals of the effect sizes were always underestimated by the authors. NNTs showed wide intervals, ranging from benefit to harm, in almost all meta-analyses. Nine recommendations to facilitate the replication of a meta-analysis were proposed: providing all original data needed to build the analysis; informing about the imputed data or data obtained from different sources; performing sensitivity analyses for imputed or unpublished data; inform about all the statistical methods used; providing all statistical results; and reporting the PI, FI and NNT.

CONCLUSION

Errors in data extraction and poor reporting of meta-analytic parameters are common in the pharmacy literature. We proposed nine recommendations to enhance data reproducibility and interpretability. Journal editors and peer reviewers should ensure that authors strictly comply with minimum standards for conduction and reporting of meta-analyses.

The Systematic Review Data Repository (SRDR): descriptive characteristics of publicly available data and opportunities for research

BACKGROUND

Conducting systematic reviews ("reviews") requires a great deal of effort and resources. Making data extracted during reviews available publicly could offer many benefits, including reducing unnecessary duplication of effort, standardizing data, supporting analyses to address secondary research questions, and facilitating methodologic research. Funded by the US Agency for Healthcare Research and Quality (AHRQ), the Systematic Review Data Repository (SRDR) is a free, web-based, open-source, data management and archival platform for reviews. Our specific objectives in this paper are to describe (1) the current extent of usage of SRDR and (2) the characteristics of all projects with publicly available data on the SRDR website.

METHODS

We examined all projects with data made publicly available through SRDR as of November 12, 2019. We extracted information about the characteristics of these projects. Two investigators extracted and verified the data.

RESULTS

SRDR has had 2552 individual user accounts belonging to users from 80 countries. Since SRDR's launch in 2012, data have been made available publicly for 152 of the 735 projects in SRDR (21%), at a rate of 24.5 projects per year, on average. Most projects are in clinical fields (144/152 projects; 95%); most have evaluated interventions (therapeutic or preventive) (109/152; 72%). The most frequent health areas addressed are mental and behavioral disorders (31/152; 20%) and diseases of the eye and ocular adnexa (23/152; 15%). Two-thirds of the projects (104/152; 67%) were funded by AHRQ, and one-sixth (23/152; 15%) are Cochrane reviews. The 152 projects each address a median of 3 research questions (IQR 1-5) and include a median of 70 studies (IQR 20-130).

CONCLUSIONS

Until we arrive at a future in which the systematic review and broader research communities are comfortable with the accuracy of automated data extraction, re-use of data extracted by humans has the potential to help reduce redundancy and costs. The 152 projects with publicly available data through SRDR, and the more than 15,000 studies therein, are freely available to researchers and the general public who might be working on similar reviews or updates of reviews or who want access to the data for decision-making, meta-research, or other purposes.

Methodological evaluation of reviews that support recommendations from three consensus workshops in periodontology

OBJECTIVE

This study aimed to evaluate comprehensiveness and reproducibility of reviews that support consensus guidelines in periodontology.

METHODS

We included the reviews that support consensus guidelines from three workshops in periodontology, which were overseen by likely the two most important organisations in the field: the European Federation of Periodontology and the American Academy of Periodontology. We independently evaluated the comprehensiveness of literature searches by determining whether authors had searched reference lists, journals, registries and grey literature and whether the searches were limited to only one or a few languages. We evaluated whether review authors reported the eligibility criteria, the search strategies, and the list of included/excluded articles. We tested whether the search and selection of articles in one major database was reproducible.

RESULTS

Twenty-nine reviews were evaluated. Two (7%) reviews reported grey literature searches, and more than two-thirds of the reviews did not report hand-searching. Almost half of the reviews did not report whether there was language restriction for the literature searches. Two-thirds of the reviews reported the use of keywords only (without Boolean operators). One-fourth of the reviews reported the presence of a list of excluded articles after the full-text assessment. None of the reviews reported a detailed list of excluded articles after screening of titles/abstracts. None of the reviews reported enough information to allow reproduction of the findings of the PubMed search.

CONCLUSIONS

There is room to improve the reporting of the methodologies that are used in reviews that support periodontology consensus guidelines, although heterogeneity in reporting was found across all the reviews.

Reproducibility and replicability of systematic reviews

Irreproducibility of research causes a major concern in academia. This concern affects all study designs regardless of scientific fields. Without testing the reproducibility and replicability it is almost impossible to repeat the research and to gain the same or similar results. In addition, irreproducibility limits the translation of research findings into practice where the same results are expected. To find the solutions, the Interacademy Partnership for Health gathered academics from established networks of science, medicine and engineering around a table to introduce seven strategies that can enhance the reproducibility: pre-registration, open methods, open data, collaboration, automation, reporting guidelines, and post-publication reviews. The current editorial discusses the generalisability and practicality of these strategies to systematic reviews and claims that systematic reviews have even a greater potential than other research designs to lead the movement toward the reproducibility of research. Moreover, I discuss the potential of reproducibility, on the other hand, to upgrade the systematic review from review to research. Furthermore, there are references to the successful and ongoing practices from collaborative efforts around the world to encourage the systematic reviewers, the journal editors and publishers, the organizations linked to evidence synthesis, and the funders and policy makers to facilitate this movement and to gain the public trust in research.

Reproducible research practices are underused in systematic reviews of biomedical interventions

OBJECTIVES

To evaluate how often reproducible research practices, which allow others to recreate the findings of studies, given the original data, are used in systematic reviews (SRs) of biomedical research.

STUDY DESIGN AND SETTING

We evaluated a random sample of SRs indexed in MEDLINE during February 2014, which focused on a therapeutic intervention and reported at least one meta-analysis. Data on reproducible research practices in each SR were extracted using a 26-item form by one author, with a 20% random sample extracted in duplicate. We explored whether the use of reproducible research practices was associated with an SR being a Cochrane review, as well as with the reported use of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.

RESULTS

We evaluated 110 SRs of therapeutic interventions, 78 (71%) of which were non-Cochrane SRs. Across the SRs, there were 2,139 meta-analytic effects (including subgroup meta-analytic effects and sensitivity analyses), 1,551 (73%) of which were reported in sufficient detail to recreate them. Systematic reviewers reported the data needed to recreate all meta-analytic effects in 72 (65%) SRs only. This percentage was higher in Cochrane than in non-Cochrane SRs (30/32 [94%] vs. 42/78 [54%]; risk ratio 1.74, 95% confidence interval 1.39-2.18). Systematic reviewers who reported imputing, algebraically manipulating, or obtaining some data from the study author/sponsor infrequently stated which specific data were handled in this way. Only 33 (30%) SRs mentioned access to data sets and statistical code used to perform analyses.

CONCLUSION

Reproducible research practices are underused in SRs of biomedical interventions. Adoption of such practices facilitates identification of errors and allows the SR data to be reanalyzed.

Increasing value and reducing waste in data extraction for systematic reviews: tracking data in data extraction forms

Data extraction is one of the most time-consuming tasks in performing a systematic review. Extraction is often onto some sort of form. Sharing completed forms can be used to check quality and accuracy of extraction or for re-cycling data to other researchers for updating. However, validating each piece of extracted data is time-consuming and linking to source problematic.In this methodology paper, we summarize three methods for reporting the location of data in original full-text reports, comparing their advantages and disadvantages.