1
|
Wilkinson MD, Sansone SA, Méndez E, David R, Dennis R, Hecker D, Kleemola M, Lacagnina C, Nikiforova A, Castro LJ. Community-driven governance of FAIRness assessment: an open issue, an open discussion. Open Res Eur 2023; 2:146. [PMID: 38298923 PMCID: PMC10828551 DOI: 10.12688/openreseurope.15364.2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 09/04/2023] [Indexed: 02/02/2024]
Abstract
Although FAIR Research Data Principles are targeted at and implemented by different communities, research disciplines, and research stakeholders (data stewards, curators, etc.), there is no conclusive way to determine the level of FAIRness intended or required to make research artefacts (including, but not limited to, research data) Findable, Accessible, Interoperable, and Reusable. The FAIR Principles cover all types of digital objects, metadata, and infrastructures. However, they focus their narrative on data features that support their reusability. FAIR defines principles, not standards, and therefore they do not propose a mechanism to achieve the behaviours they describe in an attempt to be technology/implementation neutral. Various FAIR assessment metrics and tools have been designed to measure FAIRness. Unfortunately, the same digital objects assessed by different tools often exhibit widely different outcomes because of these independent interpretations of FAIR. This results in confusion among the publishers, the funders, and the users of digital research objects. Moreover, in the absence of a standard and transparent definition of what constitutes FAIR behaviours, there is a temptation to define existing approaches as being FAIR-compliant rather than having FAIR define the expected behaviours. This whitepaper identifies three high-level stakeholder categories -FAIR decision and policymakers, FAIR custodians, and FAIR practitioners - and provides examples outlining specific stakeholders' (hypothetical but anticipated) needs. It also examines possible models for governance based on the existing peer efforts, standardisation bodies, and other ways to acknowledge specifications and potential benefits. This whitepaper can serve as a starting point to foster an open discussion around FAIRness governance and the mechanism(s) that could be used to implement it, to be trusted, broadly representative, appropriately scoped, and sustainable. We invite engagement in this conversation in an open Google Group fair-assessment-governance@googlegroups.com.
Collapse
Affiliation(s)
- Mark D. Wilkinson
- EOSC Task Force on FAIR Metrics and Data Quality, EOSC, Brussels, Belgium
- Departamento de Biotecnología-Biología Vegetal, Escuela Técnica Superior de Ingeniería Agronómica, Alimentaria y de Biosistemas, Centro de Biotecnología y Genómica de Plantas. Universidad Politécnica de Madrid (UPM) - Instituto Nacional de Investigación y Tecnología Agraria y Alimentaria-CSIC (INIA-CSIC), Madrid, Spain
| | - Susanna-Assunta Sansone
- EOSC Task Force on FAIR Metrics and Data Quality, EOSC, Brussels, Belgium
- Department of Engineering Science, Oxford e-Research Centre, The University of Oxford, Oxford, UK
| | - Eva Méndez
- Library and Information Science Department, Universidad Carlos III de Madrid, Madrid, Spain
| | - Romain David
- EOSC Task Force on FAIR Metrics and Data Quality, EOSC, Brussels, Belgium
- European Research Infrastructure on Highly Pathogenic Agents (ERINHA AISBL), Brussels, Belgium
| | - Richard Dennis
- EOSC Task Force on FAIR Metrics and Data Quality, EOSC, Brussels, Belgium
- Novo Nordisk Foundation Center for Stem Cell Medicine – reNEW, University of Copenhagen, Copenhagen, Denmark
| | - David Hecker
- EOSC Task Force on FAIR Metrics and Data Quality, EOSC, Brussels, Belgium
- Research Data Management, German Aerospace Center (DLR), Cologne, Germany
| | - Mari Kleemola
- EOSC Task Force on FAIR Metrics and Data Quality, EOSC, Brussels, Belgium
- Finnish Social Science Data Archive and CESSDA ERIC, Tampere University, Tampere, Finland
| | - Carlo Lacagnina
- EOSC Task Force on FAIR Metrics and Data Quality, EOSC, Brussels, Belgium
- Barcelona Supercomputing Center, Barcelona, Spain
| | - Anastasija Nikiforova
- EOSC Task Force on FAIR Metrics and Data Quality, EOSC, Brussels, Belgium
- Institute of Computer Science, The University of Tartu, Tartu, Estonia
| | - Leyla Jael Castro
- Semantic Technologies team, ZB MED Information Centre for Life Sciences, Cologne, Germany
| |
Collapse
|
2
|
Basereh M, Caputo A, Brennan R. Automatic transparency evaluation for open knowledge extraction systems. J Biomed Semantics 2023; 14:12. [PMID: 37653549 PMCID: PMC10468861 DOI: 10.1186/s13326-023-00293-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2022] [Accepted: 07/30/2023] [Indexed: 09/02/2023] Open
Abstract
BACKGROUND This paper proposes Cyrus, a new transparency evaluation framework, for Open Knowledge Extraction (OKE) systems. Cyrus is based on the state-of-the-art transparency models and linked data quality assessment dimensions. It brings together a comprehensive view of transparency dimensions for OKE systems. The Cyrus framework is used to evaluate the transparency of three linked datasets, which are built from the same corpus by three state-of-the-art OKE systems. The evaluation is automatically performed using a combination of three state-of-the-art FAIRness (Findability, Accessibility, Interoperability, Reusability) assessment tools and a linked data quality evaluation framework, called Luzzu. This evaluation includes six Cyrus data transparency dimensions for which existing assessment tools could be identified. OKE systems extract structured knowledge from unstructured or semi-structured text in the form of linked data. These systems are fundamental components of advanced knowledge services. However, due to the lack of a transparency framework for OKE, most OKE systems are not transparent. This means that their processes and outcomes are not understandable and interpretable. A comprehensive framework sheds light on different aspects of transparency, allows comparison between the transparency of different systems by supporting the development of transparency scores, gives insight into the transparency weaknesses of the system, and ways to improve them. Automatic transparency evaluation helps with scalability and facilitates transparency assessment. The transparency problem has been identified as critical by the European Union Trustworthy Artificial Intelligence (AI) guidelines. In this paper, Cyrus provides the first comprehensive view of transparency dimensions for OKE systems by merging the perspectives of the FAccT (Fairness, Accountability, and Transparency), FAIR, and linked data quality research communities. RESULTS In Cyrus, data transparency includes ten dimensions which are grouped in two categories. In this paper, six of these dimensions, i.e., provenance, interpretability, understandability, licensing, availability, interlinking have been evaluated automatically for three state-of-the-art OKE systems, using the state-of-the-art metrics and tools. Covid-on-the-Web is identified to have the highest mean transparency. CONCLUSIONS This is the first research to study the transparency of OKE systems that provides a comprehensive set of transparency dimensions spanning ethics, trustworthy AI, and data quality approaches to transparency. It also demonstrates how to perform automated transparency evaluation that combines existing FAIRness and linked data quality assessment tools for the first time. We show that state-of-the-art OKE systems vary in the transparency of the linked data generated and that these differences can be automatically quantified leading to potential applications in trustworthy AI, compliance, data protection, data governance, and future OKE system design and testing.
Collapse
Affiliation(s)
- Maryam Basereh
- School of Computing, Dublin City University, Dublin, Ireland.
| | - Annalina Caputo
- School of Computing, Dublin City University, Dublin, Ireland
| | - Rob Brennan
- ADAPT Centre, School of Computer Science, University College Dublin, Dublin, Ireland
| |
Collapse
|