Search

This document presents the state of work in the WP4 of the CS3MESH4EOSC project on integrating user-level applications and usage scenarios within the ScienceMesh federation. The CS3MESH4EOSC consortium is building a federated mesh of sync & share systems (EFSS) and applications focused on scientific and research use-cases, referred to as ScienceMesh. The Interoperability Platform (IOP) worked out in WP3 provides the technical basis for the federation, ensuring compatibility across sync & share products, enabling data flow and user information exchange as well as implementing communication for application workflows. The partner-provided deployments of sync & share systems at federated sites provide the testing and pre-production environments for the federation. Finally, management procedures and tools developed by WP2 constitute the operational basis for the federation. Within WP4, a set of user-level applications is being developed with the aim of being embedded into this federated mesh of sync & share systems. This work is organized in four application-related design, development and testing tasks (T4.1-T4.4) as well as two auxiliary tasks (T4.5-T4.6). This deliverable provides the summary of the work within WP4 related to the integration of the flagship applications identified for the ScienceMesh. It provides an overview of the state of the demonstrators prepared within the first 18 months of the Project. It also summarizes the status of work in the auxiliary tasks dedicated to ensuring that requirements and other information flow among WP4 and WP3, WP4 and WP2, while analysing the future sustainability of those Project results.

The Science Mesh is designed to be a highly distributed platform with a very lightweight and almost fully decentralized infrastructure. It consists of several Independent Sites running their own Enterprise File Sync and Share systems. Each of the Sites is expected to be sustainable by itself financially and each of them has their own policies and procedures related to user management, data handling, operations, and security. Finally, each of the Sites has their own SLDs or SLAs with their clients and/or the users for which they run the service. These sites, instead of operating as a disjoint collection of service islands, become nodes in a mesh of interconnected storage, applications and users. Interoperability between EFSS services is guaranteed by the Open Cloud Mesh standard1 (OCM). To this “technical” coupling of nodes through the OCM standard, Science Mesh will add agreed-upon policies and procedures with respect to operations and security, together with a governance structure. The goal is to create a coherent Infrastructure that can guarantee a service level having an adequate quality. Operational procedures to achieve this are described in this document and the documents it references. Science Mesh’s small technical footprint will be matched by an equally lean administrative structure. It is our aim to rely as much as possible on policies, procedures, and agreements that are already locally available, and augment them with what is necessary for the Science Mesh to operate. This will also help with the platform’s future sustainability and funding of activities, by keeping overhead costs as low as possible. In this document several acronyms are used. These are explained in the Science Mesh Glossary.

Ron Trompert from SURF, did a presentation entitled "Science Mesh – Making a federated infrastructure of Sync-and-Share services sustainable" during the Sustaining EOSC session at the EOSC Symposium 2021

This document accompanies the Work Package deliverable D2.1, which is a demo available as a video (screen recording) at https://www.youtube.com/watch?v=I0PlqiUBT0I. The demonstration itself would be of little information value without proper explanation. Moreover, the demo format does not provide opportunities to explain design decisions that led to the approach taken to build this federated EFSS infrastructure. Besides the demo description, this document sums up the first project year in Work Package 2, reporting on its progress during this time period. Note that this document summarises the work done in the first year and some topics in this document are still subject to discussion and have thus not been finalised yet. As of the inception of the project proposal, which was written between December 2018 and January 2019, it was expected that deliverable D2.1 would be about a demonstration of implemented federated identity management. We then held the view that federated identity management was crucial for the project and the sharing of data and applications over different Enterprise File Sync and Share (EFSS) systems could not exist without it. There has been extensive insight ever since that this may not be the best approach to tackle the problem. The Interoperability Platform (IOP) that is currently under development and has been deployed at majority of partner sites, plays a significant role in federating the infrastructure. Amongst other things, we have considered the option that the IOP would implement an invitation workflow, which would be of key importance to solving this issue in an elegant way. This workflow would result in an exchange of tokens which would allow users to share data and applications over different EFSS installations at different sites where they only need to know each other's email addresses or similar contacts. While still agreeing that federated identity and group management systems can be used for this end, we have realized that they are by no means a necessity in order to create the mesh functionality that the CS3MESH4EOSC project aims to deliver. In Section 2 we describe the demo itself. The description should be self-contained for readers and demo viewers who are just interested in the status of things. More in-depth information may be found in the subsequent sections, especially reasoning behind the design decisions. User, group and application workflows are described in Section 3. Building the Science Mesh federative infrastructure with AAI, monitoring, accounting, security, mesh governance and requirements related to the ingress of new sites are described in Section 4. In Section 5 we list the sites which have currently deployed the interoperability platform. Section 6 deals with entering EOSC and the requirements that need to be fulfilled. Finally, Section 7 contains a brief discussion about future work.

This document describes the development and definition of a set of vendor-neutral protocols and APIs necessary to build the full Science Mesh service, including federated sharing of data, access to application functionality and infrastructure integration. Our approach is based on: maturing and extending protocols already in use by CS3 community (e.g. OCM, CS3APIs) with the aim to advance these protocols from TRL6 to TRL9; adopting other existing standard protocols wherever appropriate (e.g. WOPI, WebDAV,...). This document is divided in several sections. Core APIs identifies the core APIs and protocols for the Science Mesh service: OCM and CS3APIs. Use Cases further explores the usage of these interfaces in the four reference application workflows developed in the context of Work Package 4: Data Science Environments, Open Data Systems, Collaborative Document Editing and On-Demand Data Transfers. EFSS integration with IOP section describes the architecture connecting EFSS with the Interoperability Platform (IOP). IOP provides the reference implementation for the Core and Use-Case APIs. Integration of Peer-to-Peer architecture with IOP is described based on the Cubbit implementation. ScienceMesh Infrastructure APIs are presented for monitoring endpoints.

Book of Abstracts: Cloud Services for Synchronization and Sharing 2020

Book of Abstracts: Cloud Services for Synchronization and Sharing 2019

Book of Abstracts: Cloud Services for Synchronization and Sharing 2018

Book of Abstracts: Cloud Services for Synchronization and Sharing 30 January - 1 February 2017, SURFsara Amsterdam, The Netherlands It has become progressively clearer that services capable to synchronise, share and interact with growing data volumes could become the keystone of new ways to do research. Early prototypes of innovative non-traditional working storage exist: systems having at their centre synchronisation and sharing capabilities are natural candidates to host demanding data processing in the future. Large-scale synchronization and sharing capabilities are already challenging existing IT infrastructures in all areas, from security models to integration with compute resources. This situation calls for renewing computing infrastructures and the way research is done locally and across collaborating institutes. We will meet at the 3rd Cloud Services for Synchronisation and Sharing (CS3) Conference to review the state-of-the-art and share experiences about cloud storage services. The community which gathered around the previous CS3 events includes users, service providers and developers of technological solutions. As in the past, the conference will maximise the opportunity to liaise with key actors of the community. In the previous CS3 conferences we witnessed the grow in complexity of Cloud Storage installations along the maturing of related technologies. Key actors in the Cloud Storage met at CS3 bridging between the evolution of technologies (and the need to standardise and interoperate) with new ideas from user communities exposing their experience and needs. Three facts became clear during last conference at the ETH in Zurich: Existence of a vital community convinced on the potential of Cloud Storage technologies Demonstration of Cloud Storage as new platform for scientific and technical research Need of a Cloud Storage federation to enable collaborative research across different institutes and infrastructures An hallmark of CS3 has always been to invite companies and development teams to share their experience and plans: they are invited to submit abstracts about their strategy and their technologies to be openly discussed. We have seen that this is important in complementing the experience and plans from groups providing Cloud services and users comparing and proposing new ways to use these services. Topics of interest: Innovative user applications and workflows with cloud storage services Protocols and semantics for file sharing and synchronization Consistency and scalability of file synchronization services Backend storage technologies Federated access to cloud storage Programme Committee Guido Aben (AARNet) Massimo Lamanna (CERN) Luca Mascetti (CERN) Jakub T. Moscicki (CERN) Tilo Steiger (ETH) Ron Trompert (SURFsara) Editor of the Book of Abstracts Belinda Chan (CERN)