Publishers' Responsibilities in Promoting Data Quality and Reproducibility

Design and validation of a conceptual model regarding impact of open science on healthcare research processes

INTRODUCTION

The development and use of digital tools in various stages of research highlight the importance of novel open science methods for an integrated and accessible research system. The objective of this study was to design and validate a conceptual model of open science on healthcare research processes.

METHODS

This research was conducted in three phases using a mixed-methods approach. The first phase employed a qualitative method, namely purposive sampling and semi-structured interview guides to collect data from healthcare researchers and managers. Influential factors of open science on research processes were extracted for refining the components and developing the proposed model; the second phase utilized a panel of experts and collective agreement through purposive sampling. The final phase involved purposive sampling and Delphi technique to validate the components of the proposed model according to researchers' perspectives.

FINDINGS

From the thematic analysis of 20 interview on the study topic, 385 codes, 38 sub-themes, and 14 main themes were extracted for the initial proposed model. These components were reviewed by expert panel members, resulting in 31 sub-themes, 13 main themes, and 4 approved themes. Ultimately, the agreed-upon model was assessed in four layers for validation by the expert panel, and all the components achieved a score of > 75% in two Delphi rounds. The validated model was presented based on the infrastructure and culture layers, as well as supervision, assessment, publication, and sharing.

CONCLUSION

To effectively implement these methods in the research process, it is essential to create cultural and infrastructural backgrounds and predefined requirements for preventing potential abuses and privacy concerns in the healthcare system. Applying these principles will lead to greater access to outputs, increasing the credibility of research results and the utilization of collective intelligence in solving healthcare system issues.

Evaluation of Transparency and Openness Guidelines in Physical Therapist Journals

OBJECTIVE

The goals of this study were to evaluate the extent that physical therapist journals support open science research practices by adhering to the Transparency and Openness Promotion (TOP) guidelines and to assess the relationship between journal scores and their respective journal impact factor (JIF).

METHODS

Scimago, mapping studies, the National Library of Medicine, and journal author guidelines were searched to identify physical therapist journals for inclusion. Journals were graded on 10 standards (29 available total points) related to transparency with data, code, research materials, study design and analysis, preregistration of studies and statistical analyses, replication, and open science badges. The relationship between journal transparency and openness scores and their JIF was determined.

RESULTS

Thirty-five journals' author guidelines were assigned transparency and openness factor scores. The median score (interquartile range) across journals was 3.00 out of 29 (3.00) points (for all journals the scores ranged from 0 to 8). The 2 standards with the highest degree of implementation were design and analysis transparency (reporting guidelines) and study preregistration. No journals reported on code transparency, materials transparency, replication, and open science badges. TOP factor scores were a significant predictor of JIF scores.

CONCLUSION

There is low implementation of the TOP standards by physical therapist journals. TOP factor scores demonstrated predictive abilities for JIF scores. Policies from journals must improve to make open science practices the standard in research. Journals are in an influential position to guide practices that can improve the rigor of publication which, ultimately, enhances the evidence-based information used by physical therapists.

IMPACT

Transparent, open, and reproducible research will move the profession forward by improving the quality of research and increasing the confidence in results for implementation in clinical care.

Trends in the sample size, statistics, and contributions to the BrainMap database of activation likelihood estimation meta-analyses: An empirical study of 10-year data

The literature of neuroimaging meta-analysis has been thriving for over a decade. A majority of them were coordinate-based meta-analyses, particularly the activation likelihood estimation (ALE) approach. A meta-evaluation of these meta-analyses was performed to qualitatively evaluate their design and reporting standards. The publications listed from the BrainMap website were screened. Six hundred and three ALE papers published during 2010-2019 were included and analysed. For reporting standards, most of the ALE papers reported their total number of Papers involved and mentioned the inclusion/exclusion criteria on Paper selection. However, most papers did not describe how data redundancy was avoided when multiple related Experiments were reported within one paper. The most prevalent repeated-measures correction methods were voxel-level FDR (54.4%) and cluster-level FWE (33.8%), with the latter quickly replacing the former since 2016. For study characteristics, sample size in terms of number of Papers included per ALE paper and number of Experiments per analysis seemed to be stable over the decade. One-fifth of the surveyed ALE papers failed to meet the recommendation of having >17 Experiments per analysis. For data sharing, most of them did not provide input and output data. In conclusion, the field has matured well in terms of rising dominance of cluster-level FWE correction, and slightly improved reporting on elimination of data redundancy and providing input data. The provision of Data and Code availability statements and flow chart of literature screening process, as well as data submission to BrainMap, should be more encouraged.

EBRAINS Live Papers - Interactive Resource Sheets for Computational Studies in Neuroscience

We present here an online platform for sharing resources underlying publications in neuroscience. It enables authors to easily upload and distribute digital resources, such as data, code, and notebooks, in a structured and systematic way. Interactivity is a prominent feature of the Live Papers, with features to download, visualise or simulate data, models and results presented in the corresponding publications. The resources are hosted on reliable data storage servers to ensure long term availability and easy accessibility. All data are managed via the EBRAINS Knowledge Graph, thereby helping maintain data provenance, and enabling tight integration with tools and services offered under the EBRAINS ecosystem.

A survey of researchers' code sharing and code reuse practices, and assessment of interactive notebook prototypes

This research aimed to understand the needs and habits of researchers in relation to code sharing and reuse; gather feedback on prototype code notebooks created by NeuroLibre; and help determine strategies that publishers could use to increase code sharing. We surveyed 188 researchers in computational biology. Respondents were asked about how often and why they look at code, which methods of accessing code they find useful and why, what aspects of code sharing are important to them, and how satisfied they are with their ability to complete these tasks. Respondents were asked to look at a prototype code notebook and give feedback on its features. Respondents were also asked how much time they spent preparing code and if they would be willing to increase this to use a code sharing tool, such as a notebook. As a reader of research articles the most common reason (70%) for looking at code was to gain a better understanding of the article. The most commonly encountered method for code sharing-linking articles to a code repository-was also the most useful method of accessing code from the reader's perspective. As authors, the respondents were largely satisfied with their ability to carry out tasks related to code sharing. The most important of these tasks were ensuring that the code was running in the correct environment, and sharing code with good documentation. The average researcher, according to our results, is unwilling to incur additional costs (in time, effort or expenditure) that are currently needed to use code sharing tools alongside a publication. We infer this means we need different models for funding and producing interactive or executable research outputs if they are to reach a large number of researchers. For the purpose of increasing the amount of code shared by authors, PLOS Computational Biology is, as a result, focusing on policy rather than tools.

The role of metadata in reproducible computational research

Reproducible computational research (RCR) is the keystone of the scientific method for in silico analyses, packaging the transformation of raw data to published results. In addition to its role in research integrity, improving the reproducibility of scientific studies can accelerate evaluation and reuse. This potential and wide support for the FAIR principles have motivated interest in metadata standards supporting reproducibility. Metadata provide context and provenance to raw data and methods and are essential to both discovery and validation. Despite this shared connection with scientific data, few studies have explicitly described how metadata enable reproducible computational research. This review employs a functional content analysis to identify metadata standards that support reproducibility across an analytic stack consisting of input data, tools, notebooks, pipelines, and publications. Our review provides background context, explores gaps, and discovers component trends of embeddedness and methodology weight from which we derive recommendations for future work.

Dataset Reuse: Toward Translating Principles to Practice

The web provides access to millions of datasets that can have additional impact when used beyond their original context. We have little empirical insight into what makes a dataset more reusable than others and which of the existing guidelines and frameworks, if any, make a difference. In this paper, we explore potential reuse features through a literature review and present a case study on datasets on GitHub, a popular open platform for sharing code and data. We describe a corpus of more than 1.4 million data files, from over 65,000 repositories. Using GitHub's engagement metrics as proxies for dataset reuse, we relate them to reuse features from the literature and devise an initial model, using deep neural networks, to predict a dataset's reusability. This demonstrates the practical gap between principles and actionable insights that allow data publishers and tools designers to implement functionalities that provably facilitate reuse.

Publishing computational research - a review of infrastructures for reproducible and transparent scholarly communication

BACKGROUND

The trend toward open science increases the pressure on authors to provide access to the source code and data they used to compute the results reported in their scientific papers. Since sharing materials reproducibly is challenging, several projects have developed solutions to support the release of executable analyses alongside articles.

METHODS

We reviewed 11 applications that can assist researchers in adhering to reproducibility principles. The applications were found through a literature search and interactions with the reproducible research community. An application was included in our analysis if it (i) was actively maintained at the time the data for this paper was collected, (ii) supports the publication of executable code and data, (iii) is connected to the scholarly publication process. By investigating the software documentation and published articles, we compared the applications across 19 criteria, such as deployment options and features that support authors in creating and readers in studying executable papers.

RESULTS

From the 11 applications, eight allow publishers to self-host the system for free, whereas three provide paid services. Authors can submit an executable analysis using Jupyter Notebooks or R Markdown documents (10 applications support these formats). All approaches provide features to assist readers in studying the materials, e.g., one-click reproducible results or tools for manipulating the analysis parameters. Six applications allow for modifying materials after publication.

CONCLUSIONS

The applications support authors to publish reproducible research predominantly with literate programming. Concerning readers, most applications provide user interfaces to inspect and manipulate the computational analysis. The next step is to investigate the gaps identified in this review, such as the costs publishers have to expect when hosting an application, the consideration of sensitive data, and impacts on the review process.

Identifying Data Sharing and Reuse with Scholix: Potentials and Limitations

The Scholexplorer API, based on the Scholix (Scholarly Link eXchange) framework, aims to identify links between articles and supporting data. This quantitative case study demonstrates that the API vastly expanded the number of datasets previously known to be affiliated with University of Bath outputs, allowing improved monitoring of compliance with funder mandates by identifying peer-reviewed articles linked to at least one unique dataset. Availability of author names for research outputs increased from 2.4% to 89.2%, which enabled identification of ten articles reusing non-Bath-affiliated datasets published in external repositories in the first phase, giving valuable evidence of data reuse and impact for data producers. Of these, only three were formally cited in the references. Further enhancement of the Scholix schema and enrichment of Scholexplorer metadata using controlled vocabularies would be beneficial. The adoption of standardized data citations by journals will be critical to creating links in a more systematic manner.

Quality of Open Research Data: Values, Convergences and Governance

This paper focuses on the characteristics of research data quality, and aims to cover the most important issues related to it, giving particular attention to its attributes and to data governance. The corporate word’s considerable interest in the quality of data is obvious in several thoughts and issues reported in business-related publications, even if there are apparent differences between values and approaches to data in corporate and in academic (research) environments. The paper also takes into consideration that addressing data quality would be unimaginable without considering big data.