Search

The O3as service is a tool designed to support the assessment of atmospheric ozone levels and trends. It was developed as one of the thematic services of the EOSC-Synergy project. It allows for the analysis of large datasets from chemistry-climate models and presents the information in a user-friendly format for a broad range of users, including scientists, pupils, and interested citizens. The service utilizes a unified approach to process the data, employs CF conventions for homogenization, and generates figures that can be published or downloaded as csv files. It was developed as part of the EOSC-Synergy project, and it runs on a cloud-based, containerized architecture orchestrated by Kubernetes and HPC resources, and uses the Large Scale Data Facility (LSDF) at the KIT for data storage. The service is developed with best software practices, including quality assurance, continuous integration and delivery, and compliance with the FAIR principles. This presentation will focus in particular on the architecture and functionality of the O3as service, with an example demonstration of its usage.

The European Open Science Cloud-Synergy (EOSC-Synergy) project delivers services that serve to expand the use of EOSC. One of these services, O3as, is being developed for scientists using chemistry-climate models to determine time series and eventually ozone trends for potential use in the quadrennial Global Assessment of Ozone Depletion, which will be published in 2022. A unified approach from a service like ours, which analyses results from a large number of different climate models, helps to harmonise the calculation of ozone trends efficiently and consistently. With O3as, publication-quality figures can be reproduced quickly and in a coherent way. This is done via a web application where users configure their queries to perform simple analyses. These queries are passed to the O3as service via an O3as REST API call. There, the O3as service processes the query and accesses the reduced dataset. To create a reduced dataset, regular tasks are executed on a high performance computer (HPC) to copy the primary data and perform data preparation (e.g. data reduction, standardisation and parameter unification). O3as uses EGI check-in (OIDC) to identify users and grant access to certain functionalities of the service, udocker (a tool to run Docker containers in multi-user space without root privileges) to perform data reduction in the HPC environment, and the Universitat Politècnica de València (UPV) Infrastructure Manager to provision service resources (Kubernetes).

The document outlines the first report on maintenance and integration of federation, access enabling and collaboration services, one of the key components of the EOSC Federating Core, a fundamental asset that EOSC-hub provides to EOSC. It provides a technical description of enhancements for the EOSC-Hub services in Work Package 5 (WP5), results of integration activities and collaboration work with other initiatives made during the first year of the project. The report identifies the integration gaps and elaborates the plans for WP5 services.

In this paper we propose a distributed architecture to provide machine learning practitioners with a set of tools and cloud services that cover the whole machine learning development cycle: ranging from the models creation, training, validation and testing to the models serving as a service, sharing and publication. In such respect, the DEEP-Hybrid-DataCloud framework allows transparent access to existing e-Infrastructures, effectively exploiting distributed resources for the most compute-intensive tasks coming from the machine learning development cycle. Moreover, it provides scientists with a set of Cloud-oriented services to make their models publicly available, by adopting a serverless architecture and a DevOps approach, allowing an easy share, publish and deploy of the developed models., This work was supported by the project DEEP-Hybrid-DataCloud ‘‘Designing and Enabling E-infrastructures for intensive Processing in a Hybrid DataCloud’’ that has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant 777435.

Market-oriented resource allocation in cloud computing is driven by increasingly stringent needs for flexibility, fine-grained allocation, and more critically, revenue maximization. Double combinatorial auctions aptly address these demands, but their \(\mathcal {NP}\)-hardness has hindered them from being widely adopted. Heuristic algorithms, with their input-dependent performance and solution quality, have failed to offer a robust alternative. We posit that a unifying approach for evaluating all existing algorithms, under the umbrella of a consistent problem formulation and a variety of common test cases, can propel combinatorial auctions towards real-world usage.

Journal of grid computing 16(3), 381 - 408 (2018). doi:10.1007/s10723-018-9453-3, This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications., Published by Springer Nature, Dordrecht

Over the past several years, rapid growth of data has affected many fields of science. This has often resulted in the need for overhauling or exchanging the tools and approaches in the disciplines’ data life cycles. However, this allows the application of new data analysis methods and facilitates improved data sharing. The project Large-Scale Data Management and Analysis (LSDMA) of the German Helmholtz Association has been addressing both specific and generic requirements in its data life cycle successfully since 2012. Its data scientists work together with researchers from the fields such as climatology, energy and neuroscience to improve the community-specific data life cycles, in several cases even all stages of the data life cycle, i.e. from data acquisition to data archival. LSDMA scientists also study methods and tools that are of importance to many communities, e.g. data repositories and authentication and authorization infrastructure.

When preparing the Data Management Plan for larger scientific endeavors, PIs have to balance between the most appropriate qualities of storage space along the line of the planned data life-cycle, its price and the available funding. Storage properties can be the media type, implicitly determining access latency and durability of stored data, the number and locality of replicas, as well as available access protocols or authentication mechanisms. Negotiations between the scientific community and the responsible infrastructures generally happen upfront, where the amount of storage space, media types, like: disk, tape and SSD and the foreseeable data life-cycles are negotiated. With the introduction of cloud management platforms, both in computing and storage, resources can be brokered to achieve the best price per unit of a given quality. However, in order to allow the platform orchestrator to programmatically negotiate the most appropriate resources, a standard vocabulary for different properties of resources and a commonly agreed protocol to communicate those, has to be available. In order to agree on a basic vocabulary for storage space properties, the storage infrastructure group in INDIGO-DataCloud together with INDIGO-associated and external scientific groups, created a working group under the umbrella of the Research Data Alliance (RDA). As communication protocol, to query and negotiate storage qualities, the Cloud Data Management Interface (CDMI) has been selected. Necessary extensions to CDMI are defined in regular meetings between INDIGO and the Storage Network Industry Association (SNIA). Furthermore, INDIGO is contributing to the SNIA CDMI reference implementation as the basis for interfacing the various storage systems in INDIGO to the agreed protocol and to provide an official Open-Source skeleton for systems not being maintained by INDIGO partners.

The CrossGrid project is developing new grid middleware components, tools and applications with a special focus on parallel and interactive computing. In order to support the development effort and provide a test infrastructure, an international grid testbed has been deployed across 9 countries. Through the deployment of the testbed and its supporting services, CrossGrid is also contributing to another important project objective, the expansion of the grid coverage in Europe. This paper describes the status of the CrossGrid testbed.

Many identity federations are designed to be used with web browsers. This paper proposes an approach for integrating non web-based applications with web-based identity federations using Kerberos protocol. We evaluate this approach by making NFS server available for users of SAML-based identity federation of Baden-Württemberg state of Germany. We make use of LDAP-Facade software for federating non web-based services. We have modified the web-interface component of LDAP-Facade to enable the registration with kerberized services. Our approach can be used without modifications on the client side. Copyright © 2015 SCITEPRESS - Science and Technology Publications. All rights reserved.

Big Data applications in science are producing huge amounts of data, which require advanced processing, handling, and analysis capabilities. For the organization of large scale data sets it is essential to annotate these with metadata, index them, and make them easily findable. In this paper we investigate two scientific use cases from biology and photon science, which entail complex situations in regard to data volume, data rates and analysis requirements. The LSDMA project provides an ideal context for this research, combining both innovative R&D on the processing, handling, and analysis level and a wide range of research communities in need of scalable solutions. To facilitate the advancement of data life cycles we present preferred metadata management strategies. In biology the Open Microscopy Environment (OME) and in photon science NeXus/ICAT are presented. We show that these are well suited for the respective data life cycles. To facilitate searching across communities we discuss solutions involving the Open Archive Initiative - Protocol for Metadata Harvesting (OAI-PMH) and Apache Lucene/Solr.

Data play a central role in most fields of science. In recent years, the amount of data from experiment, observation, and simulation has increased rapidly and data complexity has grown. Also, communities and shared storage have become geographically more distributed. Therefore, methods and techniques applied to scientific data need to be revised and partially be replaced, while keeping the community-specific needs in focus. The German Helmholtz Association project "Large Scale Data Management and Analysis" (LSDMA) aims to maximize the efficiency of data life cycles in different research areas, ranging from high energy physics to systems biology. In its five Data Life Cycle Labs (DLCLs), data experts closely collaborate with the communities in joint research and development to optimize the respective data life cycle. In addition, the Data Services Integration Team (DSIT) provides data analysis tools and services which are common to several DLCLs. This paper describes the various activities within LSDMA and focuses on the work performed in the DLCLs.

Grid infrastructures provide large computing resources for grand challenge applications. The Interactive European Grid project developed mechanisms for the interactive and MPI-parallel usage of these resources. This chapter presents in detail an interactive approach that allows API style communication with grid resources and execution of parallel applications using MPI. The RPC API of GridSolve and the middleware gLite are integrated by the software Giggle for resource allocation. Furthermore, we describe the adaption of Giggle and GridSolve to allow the remote execution of MPI applications on RPC basis. Performance measurements are carried out to benchmark the solution. We can show an overhead of only few seconds in contrast to a job submitted via gLite, which can take minutes. We find that our approach is mainly limited by the data transfer rates to the different computing sites.

At the Karlsruhe Institute of Technology (KIT) a three-dimensional ultrasound computer tomography (3D USCT) system for early breast cancer diagnosis is being developed. This method promises reproducible volume images of the female breast in 3D. Initial measurements and a simulation based optimization method, which took several physical properties into account, led to a new aperture setup. Yet this simulation is computational too demanding to systematically evaluate the different 'virtual' apertures which can be achieved by rotation and lifting of the system. In optics a Fourier based approach is available to simulate imaging systems as linear systems. For the two apertures used in our project and one hypothetical linear array aperture this concept was evaluated and compared to a reference simulation. An acceptable conformity between the new approach and the reference simulation could be shown. With this approach a fast evaluation of optimal 'virtual' apertures for specific measurement objects and imaging constraints can be carried out within an acceptable time constraint.

gLite is one of the largest distributed computing infrastructures in operation. It provides access to hundreds of different clusters - all installed and maintained in different ways. This paper analyses the difficulties which users typically experience when moving from their own workstation via clusters or supercomputers to the grid. Based on that analysis, this paper presents tools, which helps to overcome this gap and introduces an advanced command line interface to the grid.

The int.eu.grid project aims to provide a production quality grid computing infrastructure for e-Science supporting parallel and interactive applications. The infrastructure capacity is presently about 750 cores distributed over 12 sites in 7 countries, and it implies a much larger management and coordination effort to guarantee high availability and fast start-up mechanisms of (batch and parallel) interactive applications. Int.eu.grid is also strongly focused on providing a user-friendly access to the grid through a powerful GUI allowing a transparent access to grid resources and the possibility to interact in real time with running applications. In this chapter we present the management policies, infrastructure services, and operation mechanisms that enable interactivity over the grid bringing a new added value to grid environments.

At Forschungszentrum Karlsruhe an ultrasound computer tomograph for breast cancer imaging in 3D is under development. The aim of this project is the support of early diagnosis of breast cancer. The process of reconstructing the 20 GB of measurement data into a visual 3D volume is very compute intensive. The reconstruction algorithms are written in Matlab. As a calculation platform we are using grid technologies based on gLite, which provides a powerful computing infrastructure.

Modern science is most often driven by data. Improvements in state-of-the-art technologies and methods in many scientific disciplines lead not only to increasing data rates, but also to the need to improve or even completely overhaul their data life cycle management. Communities usually face two kinds of challenges: generic ones like federated authorization and authentication infrastructures and data preservation, and ones that are specific to their community and their respective data life cycle. In practice, the specific requirements often hinder the use of generic tools and methods. The German Helmholtz Association project "Large-Scale Data Management and Analysis" (LSDMA) addresses both challenges: its five Data Life Cycle Labs (DLCLs) closely collaborate with communities in joint research and development to optimize the communities data life cycle management, while its Data Services Integration Team (DSIT) provides generic data tools and services. We present most recent developments and results from the DLCLs covering communities ranging from heavy ion physics and photon science to high-throughput microscopy, and from DSIT.

We built a cluster of ARM-based Cubieboards2 which has a SATA interface to connect a harddrive. This cluster was set up as a storage system using Ceph and as a compute cluster for high energy physics analyses. To study the performance in these applications, we ran two benchmarks on this cluster. We also checked the energy efficiency of the cluster using the preseted benchmarks. Performance and energy efficency of our cluster were compared with a network-attached storage (NAS), and with a desktop PC.

The development and deployment of middleware and applications in a grid environment spread over many institutions is a complex challenge. The management of the development process is of key importance for the successful completion of the software components. This paper presents the measures taken and the experience gained within the EU project CrossGrid. After briefly presenting the infrastructure, the development procedures and the supporting tools are described in detail. The CrossGrid “common directory” approach allows an effective, centrally managed configuration of distributed resources. The experience of the CrossGrid integration team is discussed.

After analysis and publication, there is no need to keep experimental data online on spinning disks. For reliability and costs inactive data is moved to tape and put into a data archive. The data archive must provide reliable access for at least ten years following a recommendation of the German Science Foundation (DFG), but many scientific communities wish to keep data available much longer. Data archival is on the one hand purely a bit preservation activity in order to ensure the bits read are the same as those written years before. On the other hand enough information must be archived to be able to use and interpret the content of the data. The latter is depending on many also community specific factors and remains an areas of much debate among archival specialists. The paper describes the current practice of archival and bit preservation in use for different science communities at KIT for which a combination of organizational services and technical tools are required. The special monitoring to detect tape related errors, the software infrastructure in use as well as the service certification are discussed. Plans and developments at KIT also in the context of the Large Scale Data Management and Analysis (LSDMA) project are presented. The technical advantages of the T10 SCSI Stream Commands (SSC-4) and the Linear Tape File System (LTFS) will have a profound impact on future long term archival of large data sets.

The International Testbed of the CrossGrid Project has been in operation for the last three years, including 16 sites in 9 countries across Europe. The main achievements in installation and operation are described, and also the substantial experience gained on providing support to application and middleware developers in the project. Results are presented showing the availability of a realistic Grid framework to execute distributed interactive and parallel jobs.

This specification defines how SP-IdP Proxies can provide hints about services towards Discovery Services to improve the user experience of the authentication process.

The Working Group Identity and Access Management (IAM) aims to consolidate processes, policies and architectures in order to facilitate the management of digital identities and federated access to resources within and across the NFDI consortia. The working group therefore follows the three main objectives: Conceptional work including the evaluation and creation of blueprints, especially the AARC BPA, to establish an AAI for the "one NFDI" as well as concepts for the integration of existing tools and resources in this infrastructure Connecting NFDI AAI and the NFDI consortia with relevant initiatives on state, national and international levels. Consulting to support communities and resource providers to connect to the “one NFDI AAI”. Over the course of the next two years, the working group will follow milestones to (1) provide short term AAI guidelines that allow service providers within the NFDI to connect their services to existing AAI infrastructures, (2) Build a baseline scheme for an identity space to exchange informational attributes about identities, (3) establish role and group management services and finally (4) provide an IAM architecture for the "one NFDI".