Search

As part of the EOSC Task Force on FAIR Metrics and Data Quality, the FAIR Metrics subgroup works to examine the uptake and application of metrics of FAIRness and the use and utility of FAIRness evaluations. A range of FAIR assessment tools is designed to measure compliance with established FAIR Metrics by measuring one or more Digital Objects (DO, including datasets and repositories). Unfortunately, the same DO assessment by different tools often exhibits widely different results because of independent interpretations of the Metrics, metadata publishing paradigms, and even the intent of FAIR itself. In response to this status quo, the FAIR Metrics subgroup (represented by the authors of this report) brought together developers of several FAIR evaluation tools and enabling services (listed in the Acknowledgements) for a series of hands-on hackathon events to identify a common approach to metadata provision that could be implemented by all data publishers such as databases, repositories, and data catalogue managers. This led to identifying a process for (meta)data publishing based on well-established Web standards already in everyday use within data publishing communities, albeit not uniformly. A specification document describing the approach and a series of “Apples-to-Apples” (A2A1) benchmarks to evaluate compliance with this (meta)data publishing approach were created during a series of hackathon events. The authors of FAIR evaluation tools also began writing the code to ensure their independent tools would behave identically when encountering these A2A benchmark environments, thus helping to ensure that data publishers following this paradigm will be evaluated in a harmonized manner by all assessment tools; additional considerations for assessment tool harmonization are discussed later. This report explains the rationale for these workshop and hackathon events, outlines the outcomes and work done, describes the current status, and then discusses desirable next steps. The authors propose that the EOSC Association, the EOSC Task Force Long-term Data Preservation, and other groups and projects in Europe and worldwide consider this approach to deliver accurate means to all stakeholders that assist in achieving FAIRness of research data.

The European Open Science Cloud (EOSC) Association leverages thirteen Task Forces (TFs), grouped into five Advisory Boards, to help steer the implementation of EOSC. This document is released by the Data Quality subgroup of the “FAIR Metrics and Data Quality” TF. Data quality is critical in ensuring the credibility, legitimacy, and actionability of resources within EOSC. Indeed, certification and conformity mechanisms must be established to assure researchers that the infrastructures they deposit and access data conform to clear rules and criteria. If researchers feel a loss of control and visibility or have concerns about how professionally their data will be managed, additional barriers to data sharing will emerge. Informed by the results of a systematic literature review and community consultation utilising surveys, presentations, and case studies, this TF identified key concepts and formulated recommendations. Let us start with a quick view of the critical concepts.Following the definition given by ISO 8000, with “data quality” we mean the degree to which a set of inherent characteristics of data fulfils requirements. Aligning actual data characteristics with the desired requirements implies that quality depends on context (both dataset application and lifecycle) and stakeholders, central elements in setting the requirements. Requirements must have a clear target aspect (e.g., privacy) and a level to reach (e.g., GDPR standard) and can be distinguished into functional (targeting the question “are you producing the right thing for its application?") and non-functional (targeting the question "are you producing it right?”). The goal of data quality management is to ensure that: (i) valuable information to understand the dataset is available, (ii) the dataset is reliable and ready to be used according to non-functional requirements (fit-for-use), and (iii) the dataset meets functional requirements (fit-for-purpose) when the purpose is known; if the purpose is unknown, data quality management ensures that the information necessary for users to make self-assessments of fitness-for-purpose is available. Note that in a FAIR ecosystem, where datasets could be reused far from the original purpose, information about the intended new purpose and associated functional requirements is limited.This document explains basic concepts to build a solid basis for a mutual understanding of data quality in a multidisciplinary environment. These range from the difference between quality control, assurance, and management to categories of quality dimensions, as well as typical approaches and workflows to curate and disseminate dataset quality information, minimum requirements, indicators, certification, and vocabulary. These concepts are explored considering the importance of evaluating resources carefully when deciding the sophistication of the quality assessments. Human resources, technology capabilities, and capacity-building plans constrain the design of sustainable solutions.We identified several benefits (or risks) of having good (or poor) quality and which are the stakeholders impacted. Despite these benefits, barriers and concerns prevent the provision of quality-assessed datasets, we identified these issues in detail.Distilling the knowledge accumulated in this Task Force, we extracted cross-domain commonalities, lessons learned, and challenges. The resulting main recommendations are:Data quality assessment needs standards to check data against; unfortunately, not all communities have agreed on standards, so EOSC should assist and push each community to agree on community standards to guarantee the FAIR exchange of research data. Although we extracted a few examples highlighting this gap, the current situation requires a more detailed and systematic evaluation in each community. Establishing a quality management function can help in this direction because the process can identify which standard already in use by some initiatives can be enforced as a general requirement for that community. We recommend that EOSC considers taking the opportunity to encourage communities to reach a consensus in using their standards.Data in EOSC need to be served with enough information for the user to understand how to read and correctly interpret the dataset, what restrictions are in place to use it, and what processes participate in its production. EOSC should ensure that the dataset is structured and documented in a way that can be (re)used and understood. Quality assessments in EOSC should not be concerned with checking the soundness of the data content. Aspects like uncertainty are also important to properly (re)use a dataset. Still, these aspects must be evaluated outside the EOSC ecosystem, which only checks that evidence about data content assessments is available. Following stakeholders’ expectations, we recommend that EOSC is equipped with essential data quality management, i.e., it should perform tasks like controlling the availability of basic metadata and documentation and performing basic metadata compliance checks. The EOSC quality management should not change data but point to deficiencies that the data provider or producer can address.Errors found by the curators or users need to be rectified by the data producer/provider. If not possible, errors need to be documented. Improving data quality as close to the source (i.e., producer or provider) as possible is highly recommended. Quality assessments conducted in EOSC should be shown first to the data provider to give a chance to improve the data and then to the users.User engagement is necessary to understand the user requirements (needs, expectations, etc.); it may or may not be part of a quality management function. Determining and evaluating stakeholder needs is not a one-time requirement but a continuous and collaborative part of the service delivery process.It is recommended to develop a proof-of-concept quality function performing basic quality assessments tailored to the EOSC needs (e.g., data reliability and usability). These assessments can also support rewarding research teams most committed to providing FAIR datasets. The proof-of-concept function cannot be a theoretical conceptualization of what is preferable in terms of quality. Still, it must be constrained by the reality of dealing with an enormous amount of data within a reasonable time and workforce.Data quality is a concern for all stakeholders, detailed further in this document. The quality assessments must be a multi-actor process between the data provider, EOSC, and users, potentially extended to other actors in the long run. The resulting content of quality assessments should be captured in structured, human- and machine-readable, and standard-based formats. Dataset information must be easily comparable across similar products, which calls for providing homogeneous quality information.A number of requirements valid for all datasets in EOSC (and beyond) and specific aspects of a maturity matrix gauging the maturity of a community when dealing with quality have been defined. Further refinement will be necessary for the future, and specific standards to follow will need to be identified.

Although FAIR Research Data Principles are targeted at and implemented by different communities, research disciplines, and research stakeholders (data stewards, curators, etc.), there is no conclusive way to determine the level of FAIRness intended or required to make research artefacts (including, but not limited to, research data) Findable, Accessible, Interoperable, and Reusable.The FAIR Principles cover all types of digital objects, metadata, and infrastructures. However, they focus their narrative on data features that support their reusability. FAIR defines principles, not standards, and therefore they do not propose a mechanism to achieve the behaviours they describe in an attempt to be technology/implementation neutral.FAIR is evolving in some expected and some unexpected ways. FAIR “Reusability” sub-principle R1.3 states that "(meta)data should meet domain-relevant community standards," which predicts a proliferation of FAIR interpretations by individual communities as they select their preferred approach to FAIRness. Similarly, as expected, there is an active movement around the adaptation of the FAIR Principles to digital objects other than data (e.g., software and workflows), again with individual communities interpreting what FAIRness means in these expanded contexts. However, there have also been attempts to expand the FAIR Principles themselves in recent years, including features of digital objects beyond reusability, including popularity (reuse/citation), reproducibility, reliability, data quality, etc. All of this is occurring with no overall coordination or planning.A range of FAIR assessment metrics and tools have been designed that measure FAIRness. Unfortunately, the same digital objects assessed by different tools often exhibit widely different outcomes because of these independent interpretations of FAIR. This results in confusion among the publishers, the funders, and the users of digital research objects. Moreover, in the absence of a standard and transparent definition of what constitutes FAIR behaviours, there is a temptation to define existing approaches as being FAIR-compliant rather than having FAIR define the expectedbehaviours. While it is anticipated that communities will define domain-specific FAIR metrics and tests, it is desirable to avoid "gaming the system" and have broadly agreed-upon approaches to FAIRness that do not favour a specific implementation of technology.These observations suggest a growing need to align the different interpretations of the FAIR Principles. However, this whitepaper does not suggest that the FAIR Principles themselves require governance. Indeed, the document argues that the Principles should remain untouched. Specialised communities should extend/edit those Principles to adapt and make them more relevant to their community and their specific research outcome intended to be FAIR.This whitepaper identifies three high-level stakeholder categories -FAIR decision and policymakers, FAIR custodians, and FAIR practitioners - and provides examples outlining specific stakeholders' (hypothetical but anticipated) needs. It also examines possible models for governance based on the existing peer efforts, standardisation bodies, and other ways to acknowledge specifications and potential benefits.This whitepaper can serve as a starting point to foster an open discussion around FAIRness governance and the mechanism(s) that could be used to implement it, to be trusted, broadly representative, appropriately scoped, and sustainable.