Search

Cloud bursting is an application deployment model wherein additional computing resources are provisioned from public clouds in cases where local resources are not sufficient, e.g. during peak demand periods. We propose and experimentally evaluate a cloud-bursting solution for scientific workflows. Our solution is portable thanks to using Kubernetes for deployment of the workflow management system and computing clusters in multiple clouds. We also introduce transparent data access by employing a virtual distributed file system across the clouds, allowing jobs to use a POSIX file system interface, while hiding data transfer between clouds. To balance load distribution and minimize the communication volume between clouds, we leverage graph partitioning, while ensuring that the algorithm distributes the load equally at each parallel execution stage of a workflow. The solution is experimentally evaluated using the HyperFlow workflow management system integrated with the Onedata data management platform, deployed in our on-premise cloud in Cyfronet AGH and in the Google Cloud.

Click-Through Rate estimation is a crucial prediction task in Real-Time Bidding environments prevalent in display advertising. The estimation provides information on how to trade user visits in various systems. Logistic Regression is a popular choice as the model for this task. Due to the amount, dimensionality and sparsity of data, it is challenging to train and evaluate the model. One of the techniques to reduce the training and evaluation cost is dimensionality reduction. In this work, we present Aggregate Encoding, a technique for dimensionality reduction using data aggregates. Our approach is to build aggregate-based estimators and use them as an ensemble of models weighted by logistic regression. The novelty of our work is the separation of feature values according to the value frequency, to better utilise regularization. For our experiments, we use the iPinYou data set, but this approach is universal and can be applied to other problems requiring dimensionality reduction of sparse categorical data.

We present a scientific workflow data management solution that combines global data access with a block-level optimization of data transfer, wherein only the data blocks that are used by a remote job are transferred over the network, significantly reducing data movement for specific common data access patterns. We propose the implementation of the solution based on the HyperFlow workflow management system and the Onedata data management platform. Preliminary results confirm the advantages of the proposed solution.

Many problems, like recommendation services, sensor networks, anti-crime protection, sophisticated AI services, need online data processing coming from the environment in the form of data streams consisting of events. The novelty of the approach in the field of stream processing lies in a synergistic effort toward optimization of such systems and additionally needed client components working as a whole. Building a message passing system for gathering information from mission-critical systems can be beneficial, but it is required to pay close attention to the impact it has on these systems. In this paper, we present the Apache Kafka optimization process for usage Kafka as a messaging system in latency sensitive systems. We propose a set of performance tests that can be used to measure Kafka impact on the system and performance test results of KafkaProducer Java API. KafkaProducer has almost no impact on system overall latency and it has a severe impact on resource consumption in terms of CPU. Optimising Kafka for stream processing in latency sensitive systems we reduce KafkaProducer negative impact by 75%. The tests are performed on an isolated production system.

It is now several years since scientists in Poland can use the resources of the distributed computing infrastructure – PLGrid. It is a flexible, large-scale e-infrastructure, which offers a homogeneous, easy to use access to organizationally distributed, heterogeneous hardware and software resources. It is built in accordance with good organizational and engineering practices, taking advantage of international experience in this field. Since the scientists need assistance and close collaboration with service providers, the e-infrastructure is relied on users’ requirements and needs coming from different scientific disciplines, being equipped with specific environments, solutions and services, suitable for various disciplines. All these tools help to lowering the barriers that hinder researchers to use the infrastructure.

In this paper a method for execution programming of data-intensive applications is presented. The method is based on storage Quality of Service (SQoS) provisioning. SQoS provisioning uses the semantic based storage monitoring based on a storage resources model and a storage performance management. Test results show the gain for the execution time when using the QStorMan toolkit which implements the presented method. Taking into account the SQoS provisioning opportunity on the one hand, and the increasingly growing user demands on the other hand, we believe that the execution programming of data-intensive applications can bring a new quality into the application execution.

In this paper the problem of managing ontological descriptions for a dynamically-changing Grid environment is addressed. The focus of the research is on unification of semantic descriptions for Grid services and resources through ontologies. The Grid Organizational Memory (GOM) has been designed and implemented to enable storing and accessing Grid metadata, kept in the form of ontologies. In GOM, ontology storage and management are designed to support the natural evolution process both in the knowledge structure and knowledge management services. An important element of the GOM framework is the ontology separation schema, specifying the internal vertical and horizontal dependencies according to the Virtual Organization thematic domains, efficiency of knowledge retrieval and ontology development support. The separation schema is applied in the knowledge base distribution model on the Grid. An ontology alignment-based approach is proposed to minimize user commitment on the Grid ontological metadata, to support ontology usage and development. In particular the ontology similarity based approach is presented as support for ontology updates, e.g. extension with new facts from external ontologies and environments, as well as a more efficient and complete, less implementation-bounded querying process. This paper presents research on semantic description of the Grid environment within the scope of the K-Wf Grid project that addresses knowledge-based semiautomatic workflow construction for applications on the Grid.

Using multiple Clouds as a single environment to conduct simulation-based virtual experiments at a large-scale is a challenging problem. This paper describes how this can be achieved with the Scalarm platform in the context of data farming. In particular, a use case with a private Cloud combined with public, commercial Clouds is studied. We discuss the current architecture and implementation of Scalarm in terms of supporting different infrastructures, and propose how it can be extended in order to attain a unification of different Clouds usage. We discuss different aspects of the Cloud usage unification including: scheduling virtual machines, authentication, and virtual machine state monitoring. An experimental evaluation of the presented solution is conducted with a genetic algorithm solving the well-known Travel Salesman Problem. The evaluation uses three different resource configurations: using only public Cloud, using only private Cloud, and using both public and private Clouds.

With the increasing amount of data the research community is facing problems with methods of effectively accessing, storing, and processing data in large scale and geographically distributed environments. This paper addresses major data management issues, in particular use cases and scenarios (on the basis of Polish research community organized around the PLGrid PLUS Project) and discusses architectures of data storage management systems available in both PL-Grid and other similar federated environments. On that basis, a concept of a new meta storage system, named VeilFS, is presented. The proposed system unifies file access methods for geographically distributed large scale systems and hides complexity of data access and management in such environments. However, it should be emphasized that the main purpose of this article is identification and discussion about users’ requirements and existing solutions. The VeilFS system will be described in detail in the future.

In recent years, distributed environments such as grids and clouds have evolved quickly and become widely used for both business and scientific purposes. Grid environments are used for solving increasingly complex problems in order to provide more accurate and up-to-date results. However, evolution of modern grid middlewares does not follow current trends in their utilization, which often leads to problems concerning provisioning of resources in grid environments. Many users of grid systems stumble on performance issues during execution of their applications. A special kind of grid applications which are dependent on effective provisioning of storage resources constitute data-intensive grid applications, i.e. applications which operate on large datasets. This paper addresses the issue of effective provisioning of storage resources for data-intensive grid applications based on the best-effort strategy. In order to cater applications demand on storage resources in heterogeneous and dynamic grid environments, we propose an approach relying on a combination of a cluster file system technology, a dedicated storage resources monitoring service and a management layer. The paper describes the way that this combination of technologies solves the issue of effective storage resources provisioning in grid environments by presentation of the FiVO/QStorMan toolkit, which constitutes an implementation of the proposed approach. In order to prove that the proposed approach actually reduces data-intensive applications execution time, extensive evaluation of the framework is presented for motivating scenarios which overlap most kinds of data-intensive applications.

Monitoring of Service Level Agreements is a crucial phase of SLA management. In the most challenging case, monitoring of SLA fulfillment is required in (near) real-time and needs to combine performance data regarding multiple distributed services and resources. Currently existing Grid monitoring and information services do not provide adequate on-line monitoring capabilities to fulfill this case. We present an application of Complex Event Processing principles and technologies for on-line SLA monitoring in the Grid. The capabilities of the presented SLA monitoring framework include (1) on-demand definition of SLA metrics using a high-level query language; (2) real-time calculation of the defined SLA metrics; (3) advanced query capabilities which allow for defining high-level complex metrics derived from basic metrics. SLA monitoring of data-intensive grid jobs serves as a case study to demonstrate the capabilities of the approach.

In this chapter, we intend to introduce a novel approach to data management within the Grid environment based on user-defined storage policies. This approach aims at enabling Grid application developers to specify requirements concerning storage elements which will be exploited by the application during runtime. Most of the existing Grid middleware focus on unifying access to available storage elements, e.g. by applying various virtualization techniques. While this is suitable for many Grid applications, there is a category of applications, namely the data-intensive one which often has more specific needs. The chapter outlines research and development work carried out in the PL-GRID and OntoStor projects to solve this issue within the PL-GRID infrastructure.

Current trends in modern scientific and business IT infrastructures pose certain requirements in the middleware layer in order to maximize the automation of all life-cycle phases of such infrastructures including inception, deployment, execution, and dissolution. In case this infrastructure is composed of resources of different organizations, for instance in form of a Virtual Organization, the management of these resources is especially needed for achieving new quality in business. In this paper we deal with a specific aspect of the IT infrastructure management related to autonomous enforcement of Service Level Agreement between organizations sharing their resources within a Virtual Organization. The presented framework utilizes semantic technologies in order to virtualize the heterogeneity of underlying middleware components and to allow integration of services between these organizations.

In typical data grids large amounts of replicated data are stored all over the globe in different storage systems with access latencies ranging from seconds to hours. The task of a replica management system is not only to keep track of the replicas but also to select those replicas that can be accessed by an application program with a minimal response or transfer time. Most wide-area replication research focuses on network-based replica selection. However, our past experience with data grids has shown that often hierarchical storage systems are the main bottleneck rather than network links. This is due to the fact that access latencies of hierarchical storage systems can be of the order of seconds up to hours in case the data resides on a tape that is not mounted yet. We give an overview of our replica management framework called Reptor and a storage system cost estimator that is used. Furthermore, we give details on access estimation of file replicas that reside on hierarchical storage systems. The results show that the access estimates provide a good basis for a replica management system to perform efficient replica selection.

In the paper the architecture for accessing data in the grid environment, developed for EU CrossGrid Project, is presented. The local optimization problems for efficient data access and estimation of data access time are discussed mainly, while the overall architecture is proposed on the base of collaboration with European Data Grid.

The paper describes a new language and supporting tools for creation and display of multimedia presentations, specially for the Internet usage. The language implements the message passing paradigm to set up communication between elements of presentations. Model of the language and its flexibility are discussed.

In this paper we compare efficiency of two proposed storage management systems with the efficiency obtained for the commercial solution. The developed prototypes make use of index-based retrieving of file fragments, therefore they can be useful for multimedia databases.

Nowadays, service providers offer a lot of IT services in the public or private cloud. The client can buy various kinds of services like SaaS, PaaS, etc. Recently there was introduced Backup as a Service (BaaS) as a variety of SaaS. At the moment there are available several different BaaSes for archiving the data in the cloud, but they provide only a basic level of service quality. In the paper we propose a model which ensures QoS for BaaS and some methods for management of storage resources aimed at achieving the required SLA. This model introduces a set of parameters responsible for SLA level which can be offered on the basic or higher level of quality. The storage systems (typically HSM), which are distributed between several Data Centres, are built based on disk arrays, VTLs, and tape libraries. The RSMM model does not assume bandwidth reservation or control, but is rather focused on the management of storage resources.

The aim of the presented PARMED Project is to provide medical stuff with a large database of image and video data, distributed over several medical centers in Europe. This database is queried with user-friendly and intelligent interfaces. They are also responsible for processing the incoming video streams for the later use, i.e. for diagnostics or teleeducation. Other aspects of the project is to develop services for multimedia data management and archiving with quality of service requirements and the modeling of the video and image data.

This paper describes a programming toolkit developed in the PL-Grid project, named QStorMan, which supports storage QoS provisioning for data-intensive applications in distributed environments. QStorMan exploits knowledge-oriented methods for matching storage resources to non-functional requirements, which are defined for a data-intensive application. In order to support various usage scenarios, QStorMan provides two interfaces, such as programming libraries or a web portal. The interfaces allow to define the requirements either directly in an application source code or by using an intuitive graphical interface. The first way provides finer granularity, e.g., each portion of data processed by an application can define a different set of requirements. The second method is aimed at legacy applications support, which source code can not be modified. The toolkit has been evaluated using synthetic benchmarks and the production infrastructure of PL-Grid, in particular its storage infrastructure, which utilizes the Lustre file system.

To satisfy requirements of data globalization and high performance access in particular, we introduce the originally created onedata system which virtualizes storage systems provided by storage resource providers distributed globally. onedata introduces new data organization concepts together with providers’ cooperation procedures that involve use of Global Registry as a mediator. The most significant features include metadata synchronization and on-demand file transfer.

National Data Storage is a distributed data storage system intended to provide high quality backup, archiving and data access services. These services guarantee high level of data protection as well as high performance of data storing and retrieval by using replication techniques. In this paper some conceptual and implementation details on creating a Prediction and Load Balancing Subsystem for replica management are presented. Preliminary real system test results are also shown.

As modern SOA and Grid infrastructures are being moved from academic and research environments to more challenging business and commercial applications, such issue as control of resource sharing become of crucial importance. In order to manage and share resources within distributed environments the idea of Virtual Organizations (VO) emerged, which enables sharing only subsets of resources among partners of such a VO within potentially larger settings. This paper describes the Framework for Intelligent Virtual Organizations (FiVO), focusing on its functionality of enforcing security (Authentication and Authorization) in dynamically deployed Virtual Organizations. The paper presents the overall architecture of the framework along with different security settings which FiVO can support within one Virtual Organization.

This paper presents the application of agent technology for managing employee’s experience within an organisation. The proposed approach is discussed on the basis of the software platform developed within the scope of the EU IST Pellucid project. The system is intended to provide universal assistance for organisationally mobile employees. Its functionality is realized by a set of agents. The focus of this paper is on reuse of organisational experience, which is transmitted by active hints. The experience is structured using ontologies.

In this paper we present a research which is aimed at developinga tertiary storage system architecture for efficient index-based retrievingo f video sequences. During the research we built such a system by creating dedicated tertiary storage management software called VTSS. The system is intended to be a part of a multimedia database system. We also present preliminary experimental results from a prototype implementation.

The Multimedia Storage and Retrieval System described in this paper is aimed at storing and retrieving medical multimedia data such as images and videos. There are three significant problems that automatically need to be addressed: providing enough capacity to fit Terabytes of MPEG files, efficient extraction of video fragments from MPEG files and quality of service issues. We are working on two different approaches to build MMSRS. The first one, described in this article, utilizes commercial HSM software for managing the tertiary storage hardware. The second approach that we have started to investigate is an attempt to build a specialised storage management system from scratch.

The Polish Grid Initiative commenced in 2009 as part of the PL-Grid project funded under the framework of the Innovative Economy Operational Programme [1]. The main purpose of this Project is to provide the Polish scientific community with an IT basic platform making use of Grid computer clusters, enabling e-science research in various fields. The Project is establishing a country-wide computing platform, which supports scientific research through integration of experimental data and results of advanced computer simulations carried out by geographically-dispersed teams. The solutions applied in setting up this e-infrastructure will ensure integration with other, similar platforms around the world. In the paper some basic facts concerning the Project history are given, PL-Grid goals are described and several examples of innovative grid services and software as well as support procedures developed to-date are presented.

Storage systems play a key role in any modern data center influencing the overall performance and availability of applications and services, especially, if they deal with large volumes of data. Clients of storage services may have expectations concerning the QoS, which may be formally expressed in a contract called SLA. SLA monitoring is essential part of the automated SLA management process. In this paper we present a method of applying ontologies for SLA monitoring of storage related services.

As the scientific data become larger and larger, a method is needed to store and retrieve it in time- and cost-efficient way. NCSA HDF is a data/programming model widely used for storing and sharing scientific data. We present an approach to store large volumes of structured scientific data in HDF format on tertiary storage.

Applying the principles of open research data is an important factor accelerating the production, analysis of scientific results and worldwide collaboration. However, still very little data is being shared. The aim of this article is analysis of existing data access solutions in order to identify reasons for such situation. After analysis of existing solutions and data access stakeholders needs, the authors propose own vision of data access model evolution.