Search

In this paper we propose a distributed architecture to provide machine learning practitioners with a set of tools and cloud services that cover the whole machine learning development cycle: ranging from the models creation, training, validation and testing to the models serving as a service, sharing and publication. In such respect, the DEEP-Hybrid-DataCloud framework allows transparent access to existing e-Infrastructures, effectively exploiting distributed resources for the most compute-intensive tasks coming from the machine learning development cycle. Moreover, it provides scientists with a set of Cloud-oriented services to make their models publicly available, by adopting a serverless architecture and a DevOps approach, allowing an easy share, publish and deploy of the developed models., This work was supported by the project DEEP-Hybrid-DataCloud ‘‘Designing and Enabling E-infrastructures for intensive Processing in a Hybrid DataCloud’’ that has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant 777435.

Market-oriented resource allocation in cloud computing is driven by increasingly stringent needs for flexibility, fine-grained allocation, and more critically, revenue maximization. Double combinatorial auctions aptly address these demands, but their \(\mathcal {NP}\)-hardness has hindered them from being widely adopted. Heuristic algorithms, with their input-dependent performance and solution quality, have failed to offer a robust alternative. We posit that a unifying approach for evaluating all existing algorithms, under the umbrella of a consistent problem formulation and a variety of common test cases, can propel combinatorial auctions towards real-world usage.

Demands for flexibility, efficiency and fine-grained control for the allocation of cloud resources have steered the research in this field towards market-inspired approaches. Combinatorial auctions can fulfill these demands, but their inherent \(\mathcal {NP}\)-hardness makes them impractical if an optimal solution is desired in a reasonable time. Various heuristic algorithms that yield good allocations fast have been proposed, but their performance and solution quality are highly dependent on the input. In this paper, we investigate which features of a problem instance are predictive of algorithm performance and quality, and propose an algorithm selection method that uses machine learning to find the best heuristic for each given input. We introduce a new cost model for the trade-off between execution time and solution quality, which enables quantitative algorithm comparison. Using feature-based classification to train the algorithm selection model, we can show that our approach outperforms the single best algorithm, as well as a random algorithm selection.

Journal of grid computing 16(3), 381 - 408 (2018). doi:10.1007/s10723-018-9453-3, This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications., Published by Springer Nature, Dordrecht

Over the past several years, rapid growth of data has affected many fields of science. This has often resulted in the need for overhauling or exchanging the tools and approaches in the disciplines’ data life cycles. However, this allows the application of new data analysis methods and facilitates improved data sharing. The project Large-Scale Data Management and Analysis (LSDMA) of the German Helmholtz Association has been addressing both specific and generic requirements in its data life cycle successfully since 2012. Its data scientists work together with researchers from the fields such as climatology, energy and neuroscience to improve the community-specific data life cycles, in several cases even all stages of the data life cycle, i.e. from data acquisition to data archival. LSDMA scientists also study methods and tools that are of importance to many communities, e.g. data repositories and authentication and authorization infrastructure.

The cloud landscape is becoming increasingly diverse, with more businesses, E-science moving their services to the cloud, more resource providers entering the market to meet these growing demands. Resource allocation is a central component of this landscape. Commonly used fixed-price allocation is inefficient due to its inflexible nature. Novel approaches, using market mechanisms such as auctions, can be employed for more efficient allocation, pricing. However, combinatorial auctions are inherently NP-hard, centralised. Scaling the resource allocation problem is therefore a real challenge. In this paper, we investigate the scalability, efficiency of resource allocation in large-scale, dynamic environments by using a decentralised, market-driven approach. We design, implement a generic multi-agent simulator for trading resources. We then extensively evaluate a greedy approach for solving two-sided combinatorial auctions. We show how this approach scales when distributing the allocation task over multiple agents, propose different methods to improve scalability. Moreover, we analyse the scalability of different use cases, present a use case where the centralised algorithm performs poorly. Finally, we show that a deep understanding of centralised algorithms for resource allocation allows us to mix-and-match different algorithms to increase efficiency in the decentralised approach.

Low power hardware—such as ARM CPUs—combined with novel storage concepts—such as Ceph—promise scalable storage solutions at lower energy consumptions than today’s standard solutions like network attached storage (NAS) systems. We have set up an ARM cluster, built of Cubieboards, which uses Ceph for storage management. We compare its performance as well as its energy consumption to typical NAS storages. The energy considerations include networking equipment, storage, as well as computing equipment. Ceph is a novel storage system which provides unified access to distributed storage units. It is a performant and fault tolerant solution for object and block storage, using commodity hardware. This allows a very flexible choice of hardware to support wide ranges of use-cases. The goal of the study is to outline paths for energy efficient storage systems.

Data play a central role in most fields of science. In recent years, the amount of data from experiment, observation, and simulation has increased rapidly and data complexity has grown. Also, communities and shared storage have become geographically more distributed. Therefore, methods and techniques applied to scientific data need to be revised and partially be replaced, while keeping the community-specific needs in focus. The German Helmholtz Association project "Large Scale Data Management and Analysis" (LSDMA) aims to maximize the efficiency of data life cycles in different research areas, ranging from high energy physics to systems biology. In its five Data Life Cycle Labs (DLCLs), data experts closely collaborate with the communities in joint research and development to optimize the respective data life cycle. In addition, the Data Services Integration Team (DSIT) provides data analysis tools and services which are common to several DLCLs. This paper describes the various activities within LSDMA and focuses on the work performed in the DLCLs.

As the data needs in every field continue to grow, storage systems have to grow and therefore need to adapt to the increasing demands of performance, reliability and fault tolerance. This also increases their complexity and costs. Improving the performance and scalability of storage systems while maintaining low costs is thus crucial. The evaluated open source storage system Ceph promises to reliably store data distributed across many nodes. Ceph is targeted at commodity hardware. This study investigates how Ceph performs in different setups and compares this with the theoretical maximum performance of the hardware. We used a bottom-up approach to benchmark Ceph at different architectural levels. We varied the amount of storage nodes and clients to test the scalability of the system. Our experiments revealed that Ceph delivers the promised scalability, and uncovered several points with improvement potential. We observed a significant increase of the write throughput by moving the Ceph journal to a faster location (in memory). Moreover, while the system scaled with the increasing number of clients operating the cluster, we noticed a slight performance degradation after the saturation point. We tested two optimisation strategies — increasing the available RAM or the object size — and noted a write throughput increase of up to 9% and 27%, respectively. Our findings improve the understanding of Ceph and should benefit future users through the presented strategies for tackling various performance limitations.

At Forschungszentrum Karlsruhe an ultrasound computer tomograph for breast cancer imaging in 3D is under development. The aim of this project is the support of early diagnosis of breast cancer. The process of reconstructing the 20 GB of measurement data into a visual 3D volume is very compute intensive. The reconstruction algorithms are written in Matlab. As a calculation platform we are using grid technologies based on gLite, which provides a powerful computing infrastructure.

We built a cluster of ARM-based Cubieboards2 which has a SATA interface to connect a harddrive. This cluster was set up as a storage system using Ceph and as a compute cluster for high energy physics analyses. To study the performance in these applications, we ran two benchmarks on this cluster. We also checked the energy efficiency of the cluster using the preseted benchmarks. Performance and energy efficency of our cluster were compared with a network-attached storage (NAS), and with a desktop PC.

This specification defines how SP-IdP Proxies can provide hints about services towards Discovery Services to improve the user experience of the authentication process.