Search

This report details work to systematically scope evidence of the academic, societal and economic impacts of Open Science. It is guided by the PRISMA Extension for Scoping Reviews (PRISMA-ScR) methodological framework, and was limited to works in English since 2000 found in academic databases (Web of Science, Scopus) of peer-reviewed literature. This deliverable reports findings from the first stage of this work. Future work will extend this via snowball citation searching and web search for grey literature and will be published as three pre-prints. Through systematic screening and assessment of over 30,000 initial records, we identified 479 relevant studies (311, 155 and 13 related to academic, societal and economic impact, respectively). Our findings show that evidence of impact is concentrated around Open Access (primarily academic impact) and Citizen Science (primarily societal impact), with little evidence of impact for other Open Science aspects, and hardly any evidence of economic impact. Across types of impact, we found: Academic impact: Open Access, especially impact as measured via citations, is most heavily studied. Evidence suggests an Open Access citation advantage; exclusion of authors from less resourced regions and institutions due to APCs; and that “predatory publishing” threatens the quality of the research literature. Open/FAIR Data are associated with data reuse and a citation advantage for associated papers, but their role in fostering (computational) reproducibility seems less significant than expected. Open Code and Software produce efficiency gains in software development, and may also increase citations of associated papers. Citizen Science increases efficiency and scope of data collection, but data quality is sometimes of issue. Open peer review shows neutral to positive effects on review quality. Societal impact: The majority of studies relevant to societal impact concern Citizen Science, across a variety of types including educational, engagement and empowerment benefits for participants and their communities, and the creation of data for use in governmental monitoring and administering of environments and natural resources. Beyond CS, evidence is more limited. Some literature demonstrates societal impacts of OA, including public engagement with scientific literature, use in policy-making, and health-related outcomes. Beyond this, our search revealed little evidence. Especially relevant is the limited evidence (at this stage in our study) regarding the policy impact of OS (a recurrent claim in OS advocacy) and the societal impact of Open/FAIR Data. Economic impact: Evidence here was scarce, with only 13 papers identified as relevant. Evidence was most prevalent from the biomedical and health domains. Some evidence gives positive indications of the potential of OA and Open/FAIR data to power economic activity but this is still largely without rigorous quantification. The report closes by reflecting on evidence gaps, including potential causes and solutions. * This project deliverable is currently pending approval by the EC.

This document contains the Data Management Plan (DMP) of PathOS on M6 of the project. It is a living document that will formally be updated on M18. It has been created using the ARGOS service (https://argos.openaire.eu) an online tool for creating, managing and sharing DMPs and linking them with the research artifacts they correspond to. In the DMP, we describe the input (re-used) and output (created) datasets that will be used in the technical work of PathOS, following the principles of Open (whenever possible) and FAIR as outlined in the Grant Agreement.* *The numbering of the questions skips in order to allow the updating of the DMP in the future (i.e. filling out questions for which the information is currently missing) via the Argos service. * This project deliverable is currently pending approval by the EC.

Diversity, equality and fairness in funding is of great concern to many funders. We here analysed gender differences in the funding rates across several funders and funding programmes. We found no strong evidence of any gender differences. Moreover, we would like to emphasise that observed gender differences in funding cannot be interpreted as evidence of gender bias, understood as a direct causal effect, in the evaluation of grant applications. Observing no gender differences makes it perhaps less plausible that some gender bias is at play, but this cannot be ruled out. In particular, earlier gender biases (e.g. in hiring or promotion) may affect outcomes such that the gender bias in funding is masked. There is some variability in gender differences in funding programmes. Some funding programmes changed during the time of analysis, which we compared to the evaluation criteria being used by the different programmes. Possibly, these criteria may increase the funding rate of women: having some diversity policy; not having an interview stage; evaluating leadership potential instead of established leadership; not considering historical funding track records; not asking for letters of recommendations. However, all differences are very small and very uncertain, and the effect of (not) having these criteria on gender differences is very uncertain. There are still clear gender differences in the science system, with women being less represented among researchers in most fields. Research funding plays a large role in academic careers and outcomes, and funders should analyse the outcomes of their funding process to monitor their effects. Our results show that research funding does not show gender differences and therefore does not seem to contribute to the overall gender difference in the science system. Efforts to improve the representation of women in science may therefore perhaps be better focused elsewhere in the science system.

In this study we provide a first account of the methodological workflow for a large-scale clustering of the Twitter communities of attention around scientific publications as captured in the open database Crossref Event Data. To the best of our knowledge this is the largest algorithmic clustering of Twitter users and scientific publications performed to date. The availability of this type of clustering opens new analytical possibilities in the study of the Twitter dissemination of scientific publications. For example, making possible the study of the diversity of the communities in which publications have been tweeted enabling the differentiation of publications tweeted in smaller or larger communities, or the identification of those communities that tweet more superficially or automatically. From a technical point of view, the use of big data tools (Google BigQuery) was implemented given the large size of data involved in the clustering. Moreover, the use of the relative weight allowed for the determination of well connected communities without much skewness in its sizes. The sheer size and availability of open data opens the way for several kinds of analysis that demand careful use of file formats and computation resources, usually based on big data tools such as data warehouses and running multiprocessor code. Future research will necessarily focus on two additional developments: 1) refining the clustering to include those less connected communities in a meaningful manner, making them also more balanced, and 2) implementing a labelling of the different clusters obtained. For the first, additional clustering (e.g. clustering of clusters) and reclustering of smaller clusters will be very likely the approach to go. For the second, we aim at finding potentially meaningful information by collecting metadata from papers (e.g. journals, titles, topics) and Twitter users (e.g. profile descriptions, geolocations, URLs). That information, combined with language processing techniques will potentially allow the labelling of the clusters in order to better characterise the communities and their dynamics in disseminating scientific publications on Twitter.

Diversity and inclusion is an important topic in society, and is being studied in various fields, including studies of science itself. Many studies on diversity and inclusion in science analyse biases in various processes of research, such as prestige bias in peer review or gender bias in citations. However, bias as a concept is not always clearly defined in this context. We propose to define bias in this context as an unjustified direct causal effect. The proposed definition of bias is closely related to the notion of procedural fairness. In this context, disparity is another frequently used term, which we also define in terms of causal mechanisms. The proposed definition of disparity is closely related to the notion of distributive fairness. These causal foundations allow us to reason about bias, disparities, and various concepts of fairness and their implications, which relate to recent discussions in artificial intelligence. We discuss the implications of the proposed definitions for studying biases in the research system, and briefly reflect on potential effects of interventions to counter biases. We believe the proposed definitions contribute to a better appreciation of the causal intricacies and an improved understanding of policy implications of studies on bias., Presented at VU Amsterdam at 19 May 2022.

This presentation provides an overview of RoRi's Funder Data Platform, for data-sharing and analysis by research funders, and of the early findings of the CRITERIA study--the first project to use the platform.

The academic funding system has been under criticism for many years. While there are some on-going experiments trying to improve the system, there isn’t any clear conclusion or guideline. On the other hand, modeling is a cost-effective way to facilitate the discussion. In this paper, we present a construction of analytical and simulation models to study the relation between the amount of funding and the efficiency of the funding system. In many of the settings, we show that there could exist some optimal amounts of funding to maximize the efficiency. The results are useful to reflect on the funding system in the real world.

Articles in high-impact journals are, on average, more frequently cited. But are they cited more often because those articles are somehow more “citable”? Or are they cited more often simply because they are published in a high-impact journal? Although some evidence suggests the latter the causal relationship is not clear. We here compare citations of preprints to citations of the published version to uncover the causal mechanism. We build on an earlier model of citation dynamics to infer the causal effect of journals on citations. We find that high-impact journals select articles that tend to attract more citations. At the same time, we find that high-impact journals augment the citation rate of published articles. Our results yield a deeper understanding of the role of journals in the research system. The use of journal metrics in research evaluation has been increasingly criticized in recent years and article-level citations are sometimes suggested as an alternative. Our results show that removing impact factors from evaluation does not negate the influence of journals. This insight has important implications for changing practices of research evaluation., Presented at Complexity Science Hub Vienna, 5 February 2021

Presentation given about igraph at CZI’s Essential Open Source Software for Science Meeting on December 9, 2020.