Search

In this paper we propose a distributed architecture to provide machine learning practitioners with a set of tools and cloud services that cover the whole machine learning development cycle: ranging from the models creation, training, validation and testing to the models serving as a service, sharing and publication. In such respect, the DEEP-Hybrid-DataCloud framework allows transparent access to existing e-Infrastructures, effectively exploiting distributed resources for the most compute-intensive tasks coming from the machine learning development cycle. Moreover, it provides scientists with a set of Cloud-oriented services to make their models publicly available, by adopting a serverless architecture and a DevOps approach, allowing an easy share, publish and deploy of the developed models., This work was supported by the project DEEP-Hybrid-DataCloud ‘‘Designing and Enabling E-infrastructures for intensive Processing in a Hybrid DataCloud’’ that has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant 777435.

Market-oriented resource allocation in cloud computing is driven by increasingly stringent needs for flexibility, fine-grained allocation, and more critically, revenue maximization. Double combinatorial auctions aptly address these demands, but their \(\mathcal {NP}\)-hardness has hindered them from being widely adopted. Heuristic algorithms, with their input-dependent performance and solution quality, have failed to offer a robust alternative. We posit that a unifying approach for evaluating all existing algorithms, under the umbrella of a consistent problem formulation and a variety of common test cases, can propel combinatorial auctions towards real-world usage.

Demands for flexibility, efficiency and fine-grained control for the allocation of cloud resources have steered the research in this field towards market-inspired approaches. Combinatorial auctions can fulfill these demands, but their inherent \(\mathcal {NP}\)-hardness makes them impractical if an optimal solution is desired in a reasonable time. Various heuristic algorithms that yield good allocations fast have been proposed, but their performance and solution quality are highly dependent on the input. In this paper, we investigate which features of a problem instance are predictive of algorithm performance and quality, and propose an algorithm selection method that uses machine learning to find the best heuristic for each given input. We introduce a new cost model for the trade-off between execution time and solution quality, which enables quantitative algorithm comparison. Using feature-based classification to train the algorithm selection model, we can show that our approach outperforms the single best algorithm, as well as a random algorithm selection.

Journal of grid computing 16(3), 381 - 408 (2018). doi:10.1007/s10723-018-9453-3, This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications., Published by Springer Nature, Dordrecht

Over the past several years, rapid growth of data has affected many fields of science. This has often resulted in the need for overhauling or exchanging the tools and approaches in the disciplines’ data life cycles. However, this allows the application of new data analysis methods and facilitates improved data sharing. The project Large-Scale Data Management and Analysis (LSDMA) of the German Helmholtz Association has been addressing both specific and generic requirements in its data life cycle successfully since 2012. Its data scientists work together with researchers from the fields such as climatology, energy and neuroscience to improve the community-specific data life cycles, in several cases even all stages of the data life cycle, i.e. from data acquisition to data archival. LSDMA scientists also study methods and tools that are of importance to many communities, e.g. data repositories and authentication and authorization infrastructure.

The cloud landscape is becoming increasingly diverse, with more businesses, E-science moving their services to the cloud, more resource providers entering the market to meet these growing demands. Resource allocation is a central component of this landscape. Commonly used fixed-price allocation is inefficient due to its inflexible nature. Novel approaches, using market mechanisms such as auctions, can be employed for more efficient allocation, pricing. However, combinatorial auctions are inherently NP-hard, centralised. Scaling the resource allocation problem is therefore a real challenge. In this paper, we investigate the scalability, efficiency of resource allocation in large-scale, dynamic environments by using a decentralised, market-driven approach. We design, implement a generic multi-agent simulator for trading resources. We then extensively evaluate a greedy approach for solving two-sided combinatorial auctions. We show how this approach scales when distributing the allocation task over multiple agents, propose different methods to improve scalability. Moreover, we analyse the scalability of different use cases, present a use case where the centralised algorithm performs poorly. Finally, we show that a deep understanding of centralised algorithms for resource allocation allows us to mix-and-match different algorithms to increase efficiency in the decentralised approach.

Low power hardware—such as ARM CPUs—combined with novel storage concepts—such as Ceph—promise scalable storage solutions at lower energy consumptions than today’s standard solutions like network attached storage (NAS) systems. We have set up an ARM cluster, built of Cubieboards, which uses Ceph for storage management. We compare its performance as well as its energy consumption to typical NAS storages. The energy considerations include networking equipment, storage, as well as computing equipment. Ceph is a novel storage system which provides unified access to distributed storage units. It is a performant and fault tolerant solution for object and block storage, using commodity hardware. This allows a very flexible choice of hardware to support wide ranges of use-cases. The goal of the study is to outline paths for energy efficient storage systems.

The CrossGrid project is developing new grid middleware components, tools and applications with a special focus on parallel and interactive computing. In order to support the development effort and provide a test infrastructure, an international grid testbed has been deployed across 9 countries. Through the deployment of the testbed and its supporting services, CrossGrid is also contributing to another important project objective, the expansion of the grid coverage in Europe. This paper describes the status of the CrossGrid testbed.

Big Data applications in science are producing huge amounts of data, which require advanced processing, handling, and analysis capabilities. For the organization of large scale data sets it is essential to annotate these with metadata, index them, and make them easily findable. In this paper we investigate two scientific use cases from biology and photon science, which entail complex situations in regard to data volume, data rates and analysis requirements. The LSDMA project provides an ideal context for this research, combining both innovative R&D on the processing, handling, and analysis level and a wide range of research communities in need of scalable solutions. To facilitate the advancement of data life cycles we present preferred metadata management strategies. In biology the Open Microscopy Environment (OME) and in photon science NeXus/ICAT are presented. We show that these are well suited for the respective data life cycles. To facilitate searching across communities we discuss solutions involving the Open Archive Initiative - Protocol for Metadata Harvesting (OAI-PMH) and Apache Lucene/Solr.

Data play a central role in most fields of science. In recent years, the amount of data from experiment, observation, and simulation has increased rapidly and data complexity has grown. Also, communities and shared storage have become geographically more distributed. Therefore, methods and techniques applied to scientific data need to be revised and partially be replaced, while keeping the community-specific needs in focus. The German Helmholtz Association project "Large Scale Data Management and Analysis" (LSDMA) aims to maximize the efficiency of data life cycles in different research areas, ranging from high energy physics to systems biology. In its five Data Life Cycle Labs (DLCLs), data experts closely collaborate with the communities in joint research and development to optimize the respective data life cycle. In addition, the Data Services Integration Team (DSIT) provides data analysis tools and services which are common to several DLCLs. This paper describes the various activities within LSDMA and focuses on the work performed in the DLCLs.

As the data needs in every field continue to grow, storage systems have to grow and therefore need to adapt to the increasing demands of performance, reliability and fault tolerance. This also increases their complexity and costs. Improving the performance and scalability of storage systems while maintaining low costs is thus crucial. The evaluated open source storage system Ceph promises to reliably store data distributed across many nodes. Ceph is targeted at commodity hardware. This study investigates how Ceph performs in different setups and compares this with the theoretical maximum performance of the hardware. We used a bottom-up approach to benchmark Ceph at different architectural levels. We varied the amount of storage nodes and clients to test the scalability of the system. Our experiments revealed that Ceph delivers the promised scalability, and uncovered several points with improvement potential. We observed a significant increase of the write throughput by moving the Ceph journal to a faster location (in memory). Moreover, while the system scaled with the increasing number of clients operating the cluster, we noticed a slight performance degradation after the saturation point. We tested two optimisation strategies — increasing the available RAM or the object size — and noted a write throughput increase of up to 9% and 27%, respectively. Our findings improve the understanding of Ceph and should benefit future users through the presented strategies for tackling various performance limitations.

Every distributed computing infrastructure requires authentication and authorisation infrastructures (AAI) to manage access to resources and content. Several of such so called AAI systems are in use within different groups of users. In the Large Scale Data Management and Analysis project we aim to support and bring together many user communities. We therefore need to harmonise the currently used AAI systems. The approach described is to translate between different authentication systems. We furthermore try to maintain the same trust level wherever possible, and to harmonise authorisation across the involved systems.

Grid infrastructures provide large computing resources for grand challenge applications. The Interactive European Grid project developed mechanisms for the interactive and MPI-parallel usage of these resources. This chapter presents in detail an interactive approach that allows API style communication with grid resources and execution of parallel applications using MPI. The RPC API of GridSolve and the middleware gLite are integrated by the software Giggle for resource allocation. Furthermore, we describe the adaption of Giggle and GridSolve to allow the remote execution of MPI applications on RPC basis. Performance measurements are carried out to benchmark the solution. We can show an overhead of only few seconds in contrast to a job submitted via gLite, which can take minutes. We find that our approach is mainly limited by the data transfer rates to the different computing sites.

At the Karlsruhe Institute of Technology (KIT) a three-dimensional ultrasound computer tomography (3D USCT) system for early breast cancer diagnosis is being developed. This method promises reproducible volume images of the female breast in 3D. Initial measurements and a simulation based optimization method, which took several physical properties into account, led to a new aperture setup. Yet this simulation is computational too demanding to systematically evaluate the different 'virtual' apertures which can be achieved by rotation and lifting of the system. In optics a Fourier based approach is available to simulate imaging systems as linear systems. For the two apertures used in our project and one hypothetical linear array aperture this concept was evaluated and compared to a reference simulation. An acceptable conformity between the new approach and the reference simulation could be shown. With this approach a fast evaluation of optimal 'virtual' apertures for specific measurement objects and imaging constraints can be carried out within an acceptable time constraint.

To cope with the growing requirements of data intensive scientific experiments, models and simulations the Large Scale Data Facility(LSDF) at KIT aims to support many scientific disciplines. The LSDFis a distributed storage facility at Exabyte scale providing storage, archives, data bases and meta data repositories. Open interfaces and APIs support a variety of access methods to the highly available services for high throughput data applications. Tools for an easy and transparent access allow scientists to use the LSDF without bothering with the internal structures and technologies. In close cooperation with the scientific communities the LSDF provides assistance to efficiently organize data and metadata structures, and develops and deploys community specific software on the directly connected computing infrastructure.

gLite is one of the largest distributed computing infrastructures in operation. It provides access to hundreds of different clusters - all installed and maintained in different ways. This paper analyses the difficulties which users typically experience when moving from their own workstation via clusters or supercomputers to the grid. Based on that analysis, this paper presents tools, which helps to overcome this gap and introduces an advanced command line interface to the grid.

The int.eu.grid project aims to provide a production quality grid computing infrastructure for e-Science supporting parallel and interactive applications. The infrastructure capacity is presently about 750 cores distributed over 12 sites in 7 countries, and it implies a much larger management and coordination effort to guarantee high availability and fast start-up mechanisms of (batch and parallel) interactive applications. Int.eu.grid is also strongly focused on providing a user-friendly access to the grid through a powerful GUI allowing a transparent access to grid resources and the possibility to interact in real time with running applications. In this chapter we present the management policies, infrastructure services, and operation mechanisms that enable interactivity over the grid bringing a new added value to grid environments.

At Forschungszentrum Karlsruhe an ultrasound computer tomograph for breast cancer imaging in 3D is under development. The aim of this project is the support of early diagnosis of breast cancer. The process of reconstructing the 20 GB of measurement data into a visual 3D volume is very compute intensive. The reconstruction algorithms are written in Matlab. As a calculation platform we are using grid technologies based on gLite, which provides a powerful computing infrastructure.

We built a cluster of ARM-based Cubieboards2 which has a SATA interface to connect a harddrive. This cluster was set up as a storage system using Ceph and as a compute cluster for high energy physics analyses. To study the performance in these applications, we ran two benchmarks on this cluster. We also checked the energy efficiency of the cluster using the preseted benchmarks. Performance and energy efficency of our cluster were compared with a network-attached storage (NAS), and with a desktop PC.

The development and deployment of middleware and applications in a grid environment spread over many institutions is a complex challenge. The management of the development process is of key importance for the successful completion of the software components. This paper presents the measures taken and the experience gained within the EU project CrossGrid. After briefly presenting the infrastructure, the development procedures and the supporting tools are described in detail. The CrossGrid “common directory” approach allows an effective, centrally managed configuration of distributed resources. The experience of the CrossGrid integration team is discussed.

After analysis and publication, there is no need to keep experimental data online on spinning disks. For reliability and costs inactive data is moved to tape and put into a data archive. The data archive must provide reliable access for at least ten years following a recommendation of the German Science Foundation (DFG), but many scientific communities wish to keep data available much longer. Data archival is on the one hand purely a bit preservation activity in order to ensure the bits read are the same as those written years before. On the other hand enough information must be archived to be able to use and interpret the content of the data. The latter is depending on many also community specific factors and remains an areas of much debate among archival specialists. The paper describes the current practice of archival and bit preservation in use for different science communities at KIT for which a combination of organizational services and technical tools are required. The special monitoring to detect tape related errors, the software infrastructure in use as well as the service certification are discussed. Plans and developments at KIT also in the context of the Large Scale Data Management and Analysis (LSDMA) project are presented. The technical advantages of the T10 SCSI Stream Commands (SSC-4) and the Linear Tape File System (LTFS) will have a profound impact on future long term archival of large data sets.

The International Testbed of the CrossGrid Project has been in operation for the last three years, including 16 sites in 9 countries across Europe. The main achievements in installation and operation are described, and also the substantial experience gained on providing support to application and middleware developers in the project. Results are presented showing the availability of a realistic Grid framework to execute distributed interactive and parallel jobs.