Search

This Thesis is about a distributed system application which is able to manage large shared data objects in distributed storage systems (DSS) while increasing the number of concurrent accesses, maintaining high levels of consistency assurance, and assuring smooth operation. So, the backend of the program manages the files using the COBFS framework where the files are fragmented to blocks. The fragments belong to the same object, each file is a linked list of objects that can be covered by blocks. The framework supports typical file operations, such as create, read, write/update, delete or change permission access. The users can concurrently access the same files at the same time by modifying or reading different blocks of the object. The front-end part which is a user interface is what this thesis focuses on. This user interface was created with Laravel, a popular PHP web application framework. Laravel is one of the best options for web developers looking to construct high-quality web applications since it offers reliable and effective tools for creating contemporary, dynamic user interfaces. The application starts with a login page where different users have different accesses (for example administrative access where the user has access to all files). When the user logs in, there are actions like rename, read and download a file while he/she can “drag and drop” or create a file in the directory of the application to have access on it. Also, the users can change password and username. Another, part of the interface is the Database. The database consists of three tables, the files table which has information about files, the users table which has information about each user and the permissions that has all the accesses of users on files. The database is running on MySQL. The interface is designed to be efficient and simple so that it helps users to understand how it works and operates. Lastly, the front-end application is connected to the servers that implement the COBFS framework. The files are displayed to the user through a dashboard of the interface. Overall, it is an application that aims to simplicity, efficiency, operability and concurrency.

Scaling up large-scale scientific applications on supercomputing facilities is largely dependent on the ability to scale up efficiently data storage and retrieval. However, there is an ever-widening gap between I/O and computing performance. To address this gap, an increasingly popular approach consists in introducing new intermediate storage tiers (node-local storage, burst-buffers, ...) between the compute nodes and the traditional global shared parallel file-system. Unfortunately, without advanced techniques to allocate and size these resources, they remain underutilized. In this paper, we investigate how heterogeneous storage resources can be allocated on an HPC platform, just like compute resources. To this purpose, we introduce StorAlloc, a simulator used as a testbed for assessing storage-aware job scheduling algorithms and evaluating various storage infrastructures. We illustrate its usefullness by showing through a large series of experiments how this tool can be used to size a burst-buffer partition on a top-tier supercomputer by using the job history of a production year.

As current trends start to move storage and computation to the edge in order to provide support for latency constrained applications, new edge storage systems must emerge that optimise latency and reduce the cost of communication, so that they can provide users with the best possible experience. With this goal in mind, several new storage systems have surfaced that make the most of sophisticated replication techniques and weak consistency models, in particular the causal+ consistency model. A way to build such a system is to use a causal broadcast algorithm to propagate write operations between replicas in an order that is compatible with the causal order. However, existing systems rely on mechanisms that have limitations: they either leverage on static tree topologies, not adapting to scenarios where replicas join or leave the system, or they use metadata that grows linearly with the number of replicas, being therefore incapable of scaling to the hundreds or thousands of (smaller) replicas and, as such, not being suitable for supporting the operation of edge data stores. In this work, we propose a new edge-enabled replication scheme that ensures causal delivery of operations in all replicas and, when used together with CRDTs, guarantees causal+ consistency. Our solution is a decentralised causal broadcast algorithm, ECO SYNC Tree, that makes use of a dynamic tree topology, capable of quickly adapting to nodes joining and leaving the system, to offer causal delivery while using negligible metadata to encode causal dependencies. We present an experimental evaluation of ECO SYNC Tree that shows that our solution captures “the best of both worlds” when it comes to the trade-off between broadcast latency and communication cost in stable environments, and in environments subject to events such as large groups of nodes joining or leaving the system. Moreover, when compared with state-of-the-art broadcast protocols, ECO SYNC Tree proved to be the one that is best suited for edge-based deployment, by exhibiting better performance and scalability in scenarios with high churn rates. Conforme as tendências atuais começam a mover o armazenamento e a computação para a periferia da rede de a maneira suportar aplicações com restrições de latência, devem surgir novos sistemas de armazenamento para este paradigma que otimizem a latência e reduzam o custo de comunicação de forma a proporcionar aos utilizadores a melhor experiência possível. Com esse objetivo em mente, surgiram vários novos sistemas de armazenamento que utilizam técnicas sofisticadas de replicação e modelos de coerência fraca, em particular o modelo de coerência causal+. Uma maneira de construir um destes sistemas é usar um algoritmo de difusão causal para propagar operações de escrita entre réplicas numa ordem que seja compatível com a ordem causal. No entanto, os sistemas existentes usam mecanismos que têm limitações: topologias em árvore estáticas que não se adaptam a cenários em que réplicas se juntam ou saem do sistema, ou metadados que crescem linearmente com o número de réplicas e impedem o sistema de escalar para centenas ou milhares de réplicas, não sendo adequados para suportar sistemas de armazenamento na periferia. Neste trabalho, propomos um novo esquema de replicação adaptado à periferia que entrega operações por ordem causal a todas as réplicas e, quando usado em conjunto com CRDTs, garante coerência causal+. A nossa solução é um algoritmo de difusão causal descentralizado, ECO SYNC Tree, baseado numa topologia em árvore dinâmica capaz de se adaptar rapidamente à entrada e saída de nós do sistema, e que usa metadados de tamanho desprezível para codificar dependências causais. Apresentamos uma avaliação experimental que mostra que a nossa solução captura “o melhor dos dois mundos” no trade-off entre latência de difusão e custo de comunicação, tanto em ambientes estáveis, como em ambientes sujeitos a eventos em que grandes gru- pos de nós entram ou saem do sistema. Além disso, quando comparado com protocolos do estado da arte, ECO SYNC Tree provou ser o mais adequado para ambientes na periferia, exibindo melhor desempenho e escalabilidade em cenários com elevadas taxas de churn.

Integrating cyber, physical, and social spaces together, cyber-physical-social systems (CPSS) bring more conveniences to humans. For practical applications and user convenience, it is essential that the Big Data produced in CPSS be stored in the distributed storage systems of CPSS. In this paper, we study the fault tolerance scheme for distributed storage systems of CPSS, and propose a framework that can recover multiple failed nodes simultaneously. Considering the reliability of storage nodes in distributed storage systems, the research on locally repairable codes has mostly focused on repairing failed nodes within each repair group. However, when entire repair groups have failed, existing locally repairable codes cannot repair more than one failed group. In this paper, local codes with cooperative repair that can recover more than one failed group are proposed. Specifically, the proposed local codes are constructed based on minimum storage regenerating (MSR) codes, and have an interleaving structure among the local codes, so that the parity symbols of any local code can be generated from the MSR codes in its two adjacent local codes. Taking advantage of this property, more than one failed local group can be repaired cooperatively by their adjacent local groups with lower repair locality. Furthermore, the key parameters of local codes with cooperative repair are derived. Theoretical analysis and simulation results show that, compared with previous codes with local regeneration, our codes have higher bandwidth overhead when repairing failed nodes, but advantages in storage overhead and repair locality either for repair of a single failed node or one failed local group. Moreover for a single failed local group, local codes with cooperative repair achieve almost the same tradeoff curve of storage overhead and bandwidth overhead as MSR-local codes and minimum bandwidth regenerating local (MBR-local) codes.

Locally recoverable codes (LRCs) play a significant role in distributed and cloud storage systems. The key ingredient for constructing such optimal LRCs is to characterize the parity-check matrix for LRCs. In this letter based on the parity-check matrix for generalized Reed-Solomon codes we mainly present new constructions of optimal (r, δ)-locally recoverable codes with unbounded lengths in terms of the properties of the Vandermonde matrices, of which the parameters contain the known ones.

A locally repairable code (LRC) is a $[n, k, d]$ linear code with length $n$ , dimension $k$ , minimum distance $d$ and locality $r$ , which means that every code symbol can be repaired by at most $r$ other symbols. LRCs have become an important candidate in distributed storage systems due to their relatively low I/O cost. An LRC is said to be optimal if its minimum distance meets one of the Singleton-like bounds. This paper considers the optimal constructions of LRCs with locality $r=1$ and $r=k-1$ , which involves three types: $r$ -local LRCs, $(r,\delta)$ -LRCs and LRCs with $t$ -availability. Specifically, we first prove that the existence of an optimal LRC with locality $r=1$ is equivalent to that of an MDS code with certain parameters. Thus we can completely characterize the three types of optimal LRCs with $r=1$ based on some known constructions of MDS codes. Near MDS codes is a special class of sub-optimal linear code whose minimum distance $d=n-k$ and the $i$ -th generalized Hamming weight achieves the generalized Singleton bound for $2\leq i \leq k$ . For $r=k-1$ , we have established the connections between optimal $r$ -local LRCs/ LRCs with $t$ -availability and near MDS codes. Such connections can help to construct optimal LRCs with $r=k-1$ from some known classes of near MDS codes.

Due to their tight isolation, low overhead, and efficient packaging of the execution environment, Docker containers have become a prominent solution for deploying modern applications. Consequently, a large amount of Docker images are created and this massive image dataset presents challenges to the registry and container storage infrastructure and so far has remained a largely unexplored area. Hence, there is a need of docker image characterization that can help optimize and improve the storage systems for containerized applications. Moreover, existing deduplication techniques significantly degrade the performance of registries, which will slow down the container startup time. Therefore, there is growing demand for high storage efficiency and high-performance registry storage systems. Last but not least, different storage systems can be integrated with containers as backend storage systems and provide persistent storage for containerized applications. So, it is important to analyze the performance of different backend storage systems and storage drivers and draw out the implications for container storage system design. These above observations and challenges motivate my dissertation. In this dissertation, we aim to improve the flexibility, performance, and efficiency of the storage systems for containerized applications. To this end, we focus on the following three important aspects: Docker images, Docker registry storage system, and Docker container storage drivers with their backend storage systems. Specifically, this dissertation adopts three steps: (1) analyzing the Docker image dataset; (2) deriving the design implications; (3) designing a new storage framework for Docker registries and propose different optimizations for container storage systems. In the first part of this dissertation (Chapter 3), we analyze over 167TB of uncompressed Docker Hub images, characterize them using multiple metrics and evaluate the potential of le level deduplication in Docker Hub. In the second part of this dissertation (Chapter 4), we conduct a comprehensive performance analysis of container storage systems based on the key insights from our image characterizations, and derive several design implications. In the third part of this dissertation (Chapter 5), we propose DupHunter, a new Docker registry architecture, which not only natively deduplicates layers for space savings but also reduces layer restore overhead. DupHunter supports several configurable deduplication modes, which provide different levels of storage efficiency, durability, and performance, to support a range of uses. In the fourth part of this dissertation (Chapter 6), we explore an innovative holistic approach, Chameleon, that employs data redundancy techniques such as replication and erasure-coding, coupled with endurance-aware write offloading, to mitigate wear level imbalance in distributed SSD-based storage systems. This high-performance fash cluster can be used for registries to speedup performance. Doctor of Philosophy The amount of Docker images stored in Docker registries is increasing rapidly and present challenges for the underlying storage infrastructures. Before we do any optimizations for the storage system, we should first analyze this big Docker image dataset. To this end, in this dissertation we perform the first large-scale characterization and redundancy analysis of the images and layers stored in the Docker Hub registry. Based on the findings, this dissertation presents a series of practical and efficient techniques, algorithms, optimizations to achieve high performance and flexibility, and space-efficient storage system for containerized applications. The experimental evaluation demonstrates the effectiveness of our optimizations and techniques to make storage systems flexible and space-efficacy.

With the increasing need for effective storage management due to ever-growing content-generation over the Internet, Distributed Storage Systems (DSS) has arisen as a valuable tool. Although DSS has considerably improved in the past years, it still leverages legacy techniques in its networking. To cope with the demanding requirements, Software Defined Networking (SDN) has revolutionized the way we manage networks and can significantly help in improving DSS network management. In this paper, we propose an SDN-based network control method that is capable of handling DSS network management and improving its performance. This paper presents the design, implementation, and evaluation using an emulated environment of a typical Data Center Network (DCN) deployment. The experiment results show that by applying the proposed method, DSS can increase the performance and service resilience compared to existing solutions.

The Cloud, with its pay-as-you-go model, gives the possibility of elastic resource management; users can claim and release resources as needed. This elasticity leads to financial and energetical cost reductions, and helps applications to cope with varying workloads.Distributed cloud and HPC applications processing large amounts of data are often co-located with a distributed storage system in order to ensure fast data accesses. Although many works have been proposed to dynamically rescale the processing part of such systems to match their workload, the storage is never considered as malleable (able to be dynamically rescaled) since moving massive amounts of data around is assumed to be too slow in practice. However, in recent years hardware and storage techniques have evolved and this assumption needs to be revisited.In this thesis, we present a study of the rescaling operations in distributed storage systems approached from different angles. We start by modeling the minimal duration of rescaling operations to estimate their potential speed. Then, we develop a benchmark to measure the viability of distributed storage system malleability on a given platform. Last, we implement a rescaling manager for distributed storage systems that decides and organizes the data transfers required during a rescaling operation.; Le Cloud, avec son modèle économique, offre la possibilité d’un gestion élastique des ressources; les utilisateurs peuvent louer des ressources selon leurs besoins. Cette élasticité permet de réduire les coûts énergétiques et financiers, et aide les applications à s’adapter aux charges de travail variables.Les applications manipulant de grandes quantités de données exécutées dans le Cloud ou sur des supercalculateurs sont souvent colocalisées avec un système de stockage distribué pour garantir un accès rapide aux données. Bien que de nombreux travaux aient été proposés pour redimensionner dynamiquement les capacités de calcul pour s’ajuster à la charge de travail, le stockage n’est pas considéré comme malléable (capable d’être redimensionné dynamiquement) puisque les transferts de grandes quantités de données nécessaires sont considérés trop lents. Cependant, le matériel et les techniques de stockage ont évolué et cette hypothèse doit être réévaluée.Dans cette thèse, nous présentons une étude sous différents angles des opérations de redimensionnement des systèmes de stockage distribués.Nous commençons par modéliser la durée minimale de ces opérations pour évaluer leur vitesse potentielle. Puis, nous développons un benchmark conçu pour mesurer la viabilité de la malléabilité d’un système de stockage sur une plateforme donnée. Finalement, nous implémentons un gestionnaire d’opérations de redimensionnement pour systèmes de stockage distribués qui décide et organise les transferts de données requis par ces opérations.

Distributed storage systems (DSS) have a potential of storing big amounts of data. Specialized DSS codes were defined with the main idea to enhance the reliability, availability, maintenance and data security in DSS systems. In this paper, we establish a new way of communication, named Distributed Communication Channel (DCC), that uses the coding mechanism that is similar to the one applied in the DSS. DCC represents a communicationchannel between two or more parties, that exchange information through multiple physical channels. It adapts the DSS concept using the benefits established with respect to the communication reliability and security against intruders and attacks. Moreover, with this concept we offer an increased security using the recent developments used in the Blockchain technology. The blockchain is used as a tool to securely bypass the transformation matrix or the exchange of encryption keys, between the sender and the receiver in the communication channel. The whole system uses the so-called spatial spreading mechanism to transfer the data through independent channels. Therefore, it is more reliable and secure, than the basic systems using the DSS algorithm.

Efficient resource utilization is a major concern for large-scale computer platforms. One method used to lower energy consumption and operational cost is to reduce the amount of idle resources. This can be achieved by using malleability, namely, the possibility for resource managers to dynamically increase or decrease the amount of resources of jobs while they are running.Decommissioning (removing from the cluster) the idle nodes as soon as possible allows the resource man- ager to quickly reallocate the nodes to other jobs. Challenges appear when such nodes host part of a dis- tributed storage system. Indeed, distributed storage systems need to transfer large amounts of data during decommission in order to ensure data availability and a constant level of fault tolerance.In this paper, we explore the possibility of relaxing the level of fault tolerance during the decommission in order to reduce the amount of data transfers needed before nodes are released, and thus return nodes to the resource manager faster. We quantify theoretically how much time and resources are saved by such a a fast decommission strategy compared with a standard decommission. We establish lower bounds for the duration of the different phases of a fast decommission. We use the lower bounds to estimate when the fast decommission would be useful to reduce the usage of core-hours.We implement a prototype of fast decommission mechanism. Using the prototype, we validate the lower bounds on the duration of the operation and confirm the findings about the core-hour usage.; Dans ce rapport, nous modélisons la durée de la décommission de noeuds d’un système de stockage distribué basé sur la réplication lorsque la contrainte de tolérance aux pannes est relachée. Une borne inférieure pour chaque phase de l’opération est obtenue. Une implémentation du mécanisme dans Pufferbench permet de valider les résultats théoriques.Cette méthode de décommission est par nature peu sure, et nous montrons que des gains ne sont obtenus que dans le cas d’un système limité par la bande passante du réseau.

Efficient resource utilization is a major concern for large-scale computer platforms. One method used to lower energy consumption and operational cost is to reduce the amount of idle resources. This can be achieved by using malleability, namely, the possibility for resource managers to dynamically increase or decrease the amount of resources of jobs while they are running.Decommissioning (removing from the cluster) the idle nodes as soon as possible allows the resource man- ager to quickly reallocate the nodes to other jobs. Challenges appear when such nodes host part of a dis- tributed storage system. Indeed, distributed storage systems need to transfer large amounts of data during decommission in order to ensure data availability and a constant level of fault tolerance.In this paper, we explore the possibility of relaxing the level of fault tolerance during the decommission in order to reduce the amount of data transfers needed before nodes are released, and thus return nodes to the resource manager faster. We quantify theoretically how much time and resources are saved by such a a fast decommission strategy compared with a standard decommission. We establish lower bounds for the duration of the different phases of a fast decommission. We use the lower bounds to estimate when the fast decommission would be useful to reduce the usage of core-hours.We implement a prototype of fast decommission mechanism. Using the prototype, we validate the lower bounds on the duration of the operation and confirm the findings about the core-hour usage.; Dans ce rapport, nous modélisons la durée de la décommission de noeuds d’un système de stockage distribué basé sur la réplication lorsque la contrainte de tolérance aux pannes est relachée. Une borne inférieure pour chaque phase de l’opération est obtenue. Une implémentation du mécanisme dans Pufferbench permet de valider les résultats théoriques.Cette méthode de décommission est par nature peu sure, et nous montrons que des gains ne sont obtenus que dans le cas d’un système limité par la bande passante du réseau.

International audience; Most distributed storage systems provide limited abilities for querying data by attributes other than their primary keys. Supporting efficient search on secondary attributes is challenging as applications pose varying requirements to query processing systems, and no single system design can be suitable for all needs. In this paper, we show how to overcome these challenges in order to extend distributed data stores to support queries on secondary attributes. We propose a modular architecture that is flexible and allows query processing systems to make trade-offs according to different use case requirements. We describe adap-tive mechanisms that make use of this flexibility to enable query processing systems to dynamically adjust to query and write operation workloads.

We are living in the age of information where our lives are shaped by information and communication technologies. As a consequence, there is an explosion in the amount of generated data. Distribute storage systems (DSSs) are a storage technology wherein large amounts of data are stored on a network of inexpensive storage nodes in a distributed fashion, in an efficient and inexpensive way. This thesis explores three aspects of DSSs: efficient storage, security, and privacy. We start with the description of DSSs as they are the underlying theme for the majority of the thesis. For such systems, we propose a code construction that performs efficient repair of failed systematic nodes with low repair complexity. We construct such codes by concatenating two different classes of codes. The first code serves to provide the ability to tolerate node failures, while the second allows for efficient and low complexity repair. The proposed codes achieve better repair bandwidth compared to maximum distance separable (MDS) codes, codes constructed using piggybacks, and local reconstruction/Pyramid codes, while they have a better repair complexity compared to MDS, Zigzag, Pyramid codes, and codes constructed using piggybacks. Next, we consider the notion of information-theoretic security in DSSs. To this end, we assume an eavesdropper model that has access to the data stored on a subset of nodes and a subset of the data exchanged during the repair of nodes. We present a secure coding scheme that is secure against this adversary. Furthermore, the proposed scheme ensures an efficient node repair. The security is achieved by appending the data with random symbols, and encoding the resulting sequence using a concatenated coding scheme consisting of a Gabidulin code and a repairable fountain code. We then move toward the problem of private information retrieval (PIR) in DSSs. PIR is the problem of retrieving data from the nodes without revealing the identity of the requested data (or files) to the nodes. A scheme that achieves PIR is referred to as a PIR protocol, and its efficiency is determined by its PIR rate, i.e., the ratio of downloaded data and the requested file size. For the DSS scenario, we present PIR protocols for two cases: first, where some nodes act as spies, which do not collaborate, with a goal of determining what the user is requesting. We refer to this scenario as the noncolluding case, and, second, where these nodes collaborate together in order to determine the request of a user, which we refer to as the colluding case. For the noncolluding case, we show that it is possible to achieve MDS-PIR capacity, i.e., the maximum PIR rate achieved by any protocol when the DSS stores the data using an MDS code, even when the codes used by the DSS are non-MDS. We go on to give a necessary and a sufficient condition based on the structure of the storage code for our PIR protocol to achieve this capacity. We refer to such codes as MDS-PIR capacity-achieving codes. Furthermore, we show that the rates achieved by these codes using the proposed protocol are indeed equal to the maximum PIR rate achieved by these codes under any protocol. Subsequently, we show that cyclic codes, Reed-Muller (RM) codes, and a class of distance-optimal local reconstruction codes are MDS-PIR capacity-achieving codes. For the colluding case, we present a PIR protocol for DSSs that store data using arbitrary linear codes. Following this, as in the noncolluding case, we give a necessary and a sufficient condition based on the structure of the underlying codes that allow the proposed protocol to achieve an upper bound on the PIR rate and present codes that allow this. In particular, we show that if the underlying codes are RM codes, then the protocol achieves the maximum PIR rate. The concepts learned while finding solutions to the privacy issues for DSSs are then applied to distributed caching in wireless networks, which is one of the fundamental techniques that will be implemented in future 5G wireless networks. In the considered distributed caching scenario, the files are encoded using MDS codes and then cached on the small-cell base stations (SBSs), with the goal to reduce the backhaul usage. We introduce the notion of PIR in this setting and present a protocol that achieves PIR. One essential difference between DSSs and distributed caching is that in distributed caching the files to be cached are typically cached using codes with different rates that depend on the file popularity distribution. We assume that the SBSs can collude and as such the protocol is a generalization of the protocol considered for the colluding case in DSSs. As a surprising result, we show that to minimize backhaul usage, uniform content placement, i.e., the files are cached using a single code, is optimal.

The report presents an analysis of Big Data storage solutions in different directions. The purpose of this paper is to introduce the technology of Big Data storage, prospects of storage technologies, for example, the software DIRAC. The DIRAC is a software framework for distributed computing. The report considers popular storage technologies and lists their limitations. The main problems are the storage of large data, the lack of quality in the processing, scalability, the lack of rapid availability, the lack of implementation of intelligent data retrieval. Experimental computing tasks demand a wide range of requirements in terms of CPU usage, data access or memory consumption and unstable profile of resource use for a certain period. The DIRAC Data Management System (DMS), together with the DIRAC Storage Management System (SMS) provides the necessary functionality to execute and control all the activities related with data.

Heteregenous distributed storage systems consists of nodes which have different CPU powers and communication bandwiths while erasure codes that are commonly used on them significantly vary in their use of these resources. Clustering the nodes of a distributed storage system based on the resource costs and assigning appropriate erasure codes to each cluster can improve the performance of file repair operations. A simulaton study shows that this approach can reduce file repair costs somewhere between 6 percent to 44 percent by dividing nodes in three clusters and using HSRC, EMBR and HMBR erasure codes on these clusters. © 2017 IEEE.

In wireless distributed storage systems, storage nodes are connected by wireless channels, which are broadcast in nature. This paper exploits this unique feature to design an efficient repair mechanism, called broadcast repair, for wireless distributed storage systems in the presence of multiple-node failures. Due to the broadcast nature of wireless transmission, we advocate a new measure on repair performance called repair-transmission bandwidth. In contrast to repair bandwidth, which measures the average number of packets downloaded by a newcomer to replace a failed node, repair-transmission bandwidth measures the average number of packets transmitted by helper nodes per failed node. A fundamental study on the storage capacity of wireless distributed storage systems with broadcast repair is conducted by modeling the storage system as a multicast network and analyzing the minimum cut of the corresponding information flow graph. The fundamental tradeoff between storage efficiency and repair-transmission bandwidth is also obtained for functional repair. The performance of broadcast repair is compared both analytically and numerically with that of cooperative repair, the basic repair method for wired distributed storage systems with multiple-node failures. While cooperative repair is based on the idea of allowing newcomers to exchange packets, broadcast repair is based on the idea of allowing a helper to broadcast packets to all newcomers simultaneously. We show that broadcast repair outperforms cooperative repair, offering a better tradeoff between storage efficiency and repair-transmission bandwidth., Comment: 28 pages, 7 figures

Distributed storage systems store some redundant data to keep the degree of availability of the stored data constant and also to increase the system's resistance against failures. This type of systems usually use pure replication or methods based on RAID systems as redundancy schemes. In this paper, we study the communication cost of a distributed data storage system using Maximum Distance Separable (MDS) erasure codes. Our focus is reduction of the cost of one-tomany communication used in data reconstruction/repair initialization and update operations. We propose the use of two different communication approaches on the area of distributed storage systems for the above operations; Steiner tree approach and multi-shortest path approach. We also analyse these two communication approaches empirically and theoretically. Our theoretical results indicate that Steiner tree approach has lower message usage, whereas, multi-shortest path approach has lower time usage for data reconstruction/repair initialization operations. On the other hand, Steiner tree approach has better message and time metrics for the data update process. Furthermore, our experimental results support these theoretical results. Thus, users can choose between the two approaches depending on their needs and priorities. © 2017 Slovak Academy of Sciences. All rights reserved.

Distributed storage systems store some redundant data to keep the degree of availability of the stored data constant and also to increase the system's resistance against failures. This type of systems usually use pure replication or methods based on RAID systems as redundancy schemes. In this paper, we study the communication cost of a distributed data storage system using Maximum Distance Separable (MDS) erasure codes. Our focus is reduction of the cost of one-to-many communication used in data reconstruction/repair initialization and update operations. We propose the use of two different communication approaches on the area of distributed storage systems for the above operations; Steiner tree approach and multi-shortest path approach. We also analyse these two communication approaches empirically and theoretically. Our theoretical results indicate that Steiner tree approach has lower message usage, whereas, multi-shortest path approach has lower time usage for data reconstruction/repair initialization operations. On the other hand, Steiner tree approach has better message and time metrics for the data update process. Furthermore, our experimental results support these theoretical results. Thus, users can choose between the two approaches depending on their needs and priorities.

The advent of the information age has bestowed upon us three challenges related to the way we deal with data. Firstly, there is an unprecedented demand for transmitting data at high rates. Secondly, the massive amounts of data being collected from various sources needs to be stored across time. Thirdly, there is a need to process the data collected and perform computations on it in order to extract meaningful information out of it. The interconnected nature of modern systems designed to perform these tasks has unraveled new difficulties when it comes to ensuring their resilience against sources of performance degradation. In the context of network communication and distributed data storage, system-level noise and adversarial errors have to be combated with efficient error correction schemes. In the case of distributed computation, the heterogeneous nature of computing clusters can potentially diminish the speedups promised by parallel algorithms, calling for schemes that mitigate the effect of slow machines and communication delay. This thesis addresses the problem of designing efficient fault tolerance schemes for the three scenarios just described. In the network communication setting, a family of multiple-source multicast networks that employ linear network coding is considered for which capacity-achieving distributed error-correcting codes, based on classical algebraic constructions, are designed. The codes require no coordination between the source nodes and are end to end: except for the source nodes and the destination node, the operation of the network remains unchanged. In the context of data storage, balanced error-correcting codes are constructed so that the encoding effort required is balanced out across the storage nodes. In particular, it is shown that for a fixed row weight, any cyclic Reed-Solomon code possesses a generator matrix in which the number of nonzeros is the same across the columns. In the balanced and sparsest case, where each row of the generator matrix is a minimum distance codeword, the maximal encoding time over the storage nodes is minimized, a property that is appealing in write-intensive settings. Analogous constructions are presented for a locally recoverable code construction due to Tamo and Barg. Lastly, the problem of mitigating stragglers in a distributed computation setup is addressed, where a function of some dataset is computed in parallel. Using Reed-Solomon coding techniques, a scheme is proposed that allows for the recovery of the function under consideration from the minimum number of machines possible. The only assumption made on the function is that it is additively separable, which renders the scheme useful in distributed gradient descent implementations. Furthermore, a theoretical model for the run time of the scheme is presented. When the return time of the machines is modeled probabilistically, the model can be used to optimally pick the scheme's parameters so that the expected computation time is minimized. The recovery is performed using an algorithm that runs in quadratic time and linear space, a notable improvement compared to state-of-the-art schemes. The unifying theme of the three scenarios is the construction of error-correcting codes whose encoding functions adhere to certain constraints. It is shown that in many cases, these constraints can be satisfied by classical constructions. As a result, the schemes presented are deterministic, operate over small finite fields and can be decoded using efficient algorithms.

We present a general framework for private information retrieval (PIR) from arbitrary coded databases that allows one to adjust the rate of the scheme to the suspected number of colluding servers. If the storage code is a generalized Reed--Solomon code of length $n$ and dimension $k$, we design PIR schemes that achieve a PIR rate of $\frac{n-(k+t-1)}{n}$ while protecting against any $t$ colluding servers, for any $1\leq t\leq n-k$. This interpolates between the previously studied cases of $t=1$ and $k=1$ and achieves PIR capacity in both of these cases asymptotically as the number of files in the database grows.

This paper presents a flexible irregular model for heterogeneous cloud storage systems and investigates how the cost of repairing failed nodes can be minimized. The fractional repetition code, originally designed for minimizing repair bandwidth for homogeneous storage systems, is generalized to the irregular fractional repetition code, which is adaptable to heterogeneous environments. The code structure and the associated storage allocation can be obtained by solving an integer linear programming problem. For moderate sized networks, a heuristic algorithm is proposed and shown to be near-optimal by computer simulations., Comment: 12 pages, 10 figures. to appear in IEEE Journal on Selected Areas in Communications 2014

An increasing number of web applications run totally or partially in the client machines - from collaborative editing tools to multi-user games. Avoiding to continuously contact the server allows to reduce latency between clients and to minimize the load on the centralized component. Novel implementation techniques, such as WebRTC, allows to address network problems that previously prevented these systems to be deployed in practice. Legion is a newly developed framework that exploits these mechanisms, allowing client web applications to replicate data from servers, and synchronize these replicas directly among them. This work aims to extend the current Legion framework with the integration of an additional legacy storage system that can be used to support web applications, Antidote. We study the best way to support Legion’s data model into Antidote, we design a synchronization mechanism, and finally we measure the performance cost of such an integration.

Three transmission schemes over a new type of multiple-access channel (MAC) model with inter-source communication links are proposed and investigated in this paper. This new channel model is well motivated by, e.g., wireless distributed storage networks, where communication to repair a lost node takes place from helper nodes to a repairing node over a wireless channel. Since in many wireless networks nodes can come and go in an arbitrary manner, there must be an inherent capability of inter-node communication between every pair of nodes. Assuming that communication is possible between every pair of helper nodes, the newly proposed schemes are based on various smart time-sharing and relaying strategies. In other words, certain helper nodes will be regarded as relays, thereby converting the conventional uncooperative multiple-access channel to a multiple-access relay channel (MARC). The diversity-multiplexing gain tradeoff (DMT) of the system together with efficient sphere-decodability and low structural complexity in terms of the number of antennas required at each end is used as the main design objectives. While the optimal DMT for the new channel model is fully open, it is shown that the proposed schemes outperform the DMT of the simple time-sharing protocol and, in some cases, even the optimal uncooperative MAC DMT. While using a wireless distributed storage network as a motivating example throughout the paper, the MAC transmission techniques proposed here are completely general and as such applicable to any MAC communication with inter-source communication links.

Regenerating codes for distributed storage have attracted much research interest in the past decade. Such codes trade the bandwidth needed to repair a failed node with the overall amount of data stored in the network. Minimum storage regenerating (MSR) codes are an important class of optimal regenerating codes that minimize (first) the amount of data stored per node and (then) the repair bandwidth. Specifically, an $[n,k,d]$-$(\alpha)$ MSR code $\mathbb{C}$ over $\mathbb{F}_q$ is defined as follows. Using such a code $\mathbb{C}$, a file $\cal{F}$ consisting of $\alpha k$ symbols over $\mathbb{F}_q$ can be distributed among $n$ nodes, each storing $\alpha$ symbols, in such a way that: The file $\cal{F}$ can be recovered by downloading the content of any $k$ of the $n$ nodes; and the content of any failed node can be reconstructed by accessing any $d$ of the remaining $n-1$ nodes and downloading $\alpha/(d-k+1)$ symbols from each of these nodes. Unfortunately, explicit constructions of $[n,k,d]$ MSR codes are known only for certain special cases: either low rate, namely $k/n0.5$ and low repair connectivity $k< d, Comment: 9 pages, 3 figures, a short version of this paper has been submitted to the 2016 IEEE International Symposium on Information Theory

Key-homomorphic pseudo random functions (KH-PRF) have many practical applications including proxy re-encryption, distributed credential protection systems and updatable encryption.We present a key-homomorphic pseudo random function that is homomorphic with respect to a significant part of the secret key and analyse its security. Previous constructions rely on the learning with errors problem which adds some small error to the homomorphic operations due to the noisy outputs. Our construction, based on elliptic curves, removes the need of adding this noise at the cost of adding a few bits to the secret key for which homomorphism does not follow. The main advantage of our construction is that homomorphism can be applied several times without incurring into errors. In particular, we show how our KH-PRF can be used to provide key updatable encryption to distributed storage networks. Also, by relaxing the security assumptions, our PRF can be modified to be homomorphic with respect to the entire key. Refereed/Peer-reviewed

In this paper, we propose a new class of regular fractional repetition (FR) codes constructed from perfect difference families and quasi-perfect difference families to store big data in distributed storage systems. The main advantage of the proposed construction method is that it supports a wide range of code parameter values compared to existing ones, which is an important feature to be adopted in practical systems. When using one instance of the proposed codes for a given parameter set, we show that the amount of stored data is very close to that of an existing state-of-the-art optimal FR code.

Over the last decade there has been a proliferation of distributed storage systems. These systems handle various types of data (e.g., text, photos, documents, files) in a variety of settings (e.g., enterprise, cloud, mobile). Typically, each system’s design is optimized towards a specific workload and setting. Yet, when designing new applications, developers often face the onerous task of choosing one of these existing solutions to store their workload’s data. This choice can be difficult for two reasons: (1) existing systems may not offer all of the features required by the new application and the development effort to extend an existing system’s implementation can be prohibitive, and (2) choosing a system design which cost-effectively meets a workload’s performance goal(s) is non-trivial and an incorrect choice may significantly inflate the cost to achieve the desired performance.In this dissertation I design, build, and evaluate two distributed storage systems which, via their flexible implementations, offer improved ease of use and reduced development overhead when implementing data-centric applications.First, I present Simba, a cloud-based data synchronization service which reduces development complexity of mobile apps. It improves upon existing systems which offer disjoint table- or object-only syncing with eventual consistency. Simba is built around a novel data abstraction that unifies a tabular and object data model, and allows apps to choose from a set of distributed consistency schemes. Mobile apps written to this abstraction can effortlessly sync data with the cloud and other mobile devices while benefiting from end-to-end data consistency.Second, I present Lego, a modular emulator of distributed object stores. Lego is a tool which aids in the selection of the system design that can most cost-effectively satisfy a workload’s performance goals. It enables developers to construct, deploy, and evaluate the performance of candidate object store designs in the target setting. Lego’s modular architecture enables rapid prototyping, allowing for any object store design to be instantiated as a collection of modules. Since different object store designs often share similarities, this feature enables reuse of code across system implementations, thus significantly reducing development effort.

Active distributed storages need to assure both consistency and dynamic data support, in addition to availability, confidentiality and resiliency. Further, since storage durability suffers in untrusted and unreliable environments, it becomes crucial to (a) select the most reliable set of servers to assure data retrievability and (b) dynamically identify errant servers and restore the data to ensure data recoverability. We address the issues of concurrency, consistency, dynamic data support, data share repair, and trust management in providing persistent storage and access. The paper focuses primarily on erasure coded distributed storages (storages employing erasure coding for data dispersal). Integration of Quorum based approach using Notification propagation, with a reliability model based on server trust-reputation forms the comprehensive design proposed. Treating servers and their data shares equally at data reconstructions during data retrievals is rather inadequate in untrusted environments. The design provides a suitable platform for use of Soft Decision Decoding to overcome this inadequacy. The design has been validated by the simulation, study, and analysis carried out for Reed Solomon coded storage with varying levels of resiliency and concurrency. The proposed design can be suitably adapted in typical distributed information storages catering to global networked audience in public, untrusted, and unreliable operating environments.

Bu tezde, verinin hata toleranslı bir şekilde depolanmasını sağlamak için kullanılan kaynaklardan başta bant genişliği ve toplam işlem süresi olmak üzere çeşitli kaynakların maliyetinin düşürülmesini sağlayacak çözümler önerilmiştir. Bu tez kapsamındaki çalışmalar beş bölüme ayrılmıştır. Bunlardan ilki MDS silinti kodlama kullanan dağıtık depolama sistemlerinde düğüm tamirinin ve verinin geri çatılmasının başlatımı ve veri güncellemesi işlemlerinde kullanılan süre ve bant genişliği maliyetini düşürmeyi hedefleyen topoloji farkındalıklı bir çalışmadır. İkinci çalışmada ise yeni bir melez kodlama şeması olan Homomorfik Minimum Bant Genişliği Tamir (HMBR) kodları geliştirilmiştir. HMBR kodlama şeması düğüm tamiri ve verinin geri çatılması işlemlerinde sırasıyla bant genişliği ve işlem süresi maliyetlerini iyileştiren iki farklı yöntem sağlamaktadır. Ayrıca önerilen kodlama şeması düğüm tamiri için bütünlük kontrolü mekanizmasına sahiptir. Üçüncü çalışmada da yeni bir melez kodlama şeması olan Homomorfik Minimum Depolama Tamir (HMSR) kodları geliştirilmiştir. HMSR kodları düğüm başına minimum depolama maliyeti gerektirirken bir yandan da sırasıyla bant genişliği ve işlem süresi maliyetlerini azaltan iki farklı düğüm tamiri yöntemi sağlamaktadır. Dördüncü çalışmada ise farklı düğümlerin farklı kodları kullanabildiği kümeleme tabanlı dağıtık bir depolama sistemi tasarlanmıştır. Son çalışmada ise silinti kodlama kullanan dağıtık depolama sistemlerinde veri geri çatma işlemi gerçekleştirilirken ağda bir tıkanma oluştuğunda, TCP soketlerinin yönetilmesi yoluyla bu işlemin toplam süresini düşüren bir algoritma önerilmiştir., In this thesis, solutions for reducing the cost of the sources -mainly bandwidth and total processing time- used to store data in fault tolerant way are proposed. The work in this thesis is divided into five parts. The first of these is a topology aware solution proposed for reducing time and bandwidth used for data update and the initiation of node repair and data reconstruction processes in distributed storage systems using MDS erasure codes. The second one is a new hybrid coding scheme: Homomorphic Minimum Bandwidth Repairing (HMBR) codes. HMBR codes provide two different node repair methods as well as two different data reconstruction methods for reducing bandwidth usage and the processing time. In addition, this new coding scheme presents an integrity checking mechanism for repaired node's content. In the third part of our work, another hybrid coding scheme called Homomorphic Minimum Storage Repairing Codes (HMSR) which minimizes storage cost on the nodes is designed. Moreover, these codes present two different node repair methods reducing either bandwith usage or elapsed time in node repair. In the fourth part of the thesis, the cluster based storage system is proposed in which different nodes can use different coding schemes. As the last part of our work, an algorithm is proposed for reducing the time elapsed in data reconstruction process of distributed storage systems using erasure codes in case of network congestion.

Distributed storage systems and associated storage codes can efficiently store a large amount of data while ensuring that data is retrievable in case of node failure. The study of such systems, particularly the design of storage codes over finite fields, assumes that the physical channel through which the nodes communicate is error-free. This is not always the case, for example, in a wireless storage system. We study the probability that a subpacket is repaired incorrectly during node repair in a distributed storage system, in which the nodes communicate over an AWGN or Rayleigh fading channels. The asymptotic probability (as SNR increases) that a node is repaired incorrectly is shown to be completely determined by the repair locality of the DSS and the symbol error rate of the wireless channel. Lastly, we propose some design criteria for physical layer coding in this scenario, and use it to compute optimally rotated QAM constellations for use in wireless distributed storage systems., To appear in ISITA 2014

Cloud service providers, social networks and data-management companies are witnessing a tremendous increase in the amount of data they receive every day. All this data creates new opportunities to expand human knowledge in fields like healthcare and human behavior and improve offered services like search, recommendation, and many others. It is not by accident that many academics but also public media refer to our era as the “Big Data” era. But these huge opportunities come with the requirement for better data management systems that, on one hand, can safely accommodate this huge and constantly increasing volume of data and, on the other, serve them in a timely and useful manner so that applications can benefit from processing them. This document focuses on the above two challenges that come with “Big Data”. In more detail, we study (i) backup storage systems as a means to safeguard data against a number of factors that may render them unavailable and (ii) data placement strategies on geographically distributed storage systems, with the goal to reduce the user perceived latencies and the network and storage resources are efficiently utilized. Throughout our study, data are placed in the centre of our design choices as we try to leverage content properties for both placement and efficient storage., Les fournisseurs de services de cloud computing, les réseaux sociaux et les entreprises de gestion des données ont assisté à une augmentation considérable du volume de données qu'ils reçoivent chaque jour. Toutes ces données créent des nouvelles opportunités pour étendre la connaissance humaine dans des domaines comme la santé, l'urbanisme et le comportement humain et permettent d'améliorer les services offerts comme la recherche, la recommandation, et bien d'autres. Ce n'est pas par accident que plusieurs universitaires mais aussi les médias publics se référent à notre époque comme l'époque “Big Data”. Mais ces énormes opportunités ne peuvent être exploitées que grâce à de meilleurs systèmes de gestion de données. D'une part, ces derniers doivent accueillir en toute sécurité ce volume énorme de données et, d'autre part, être capable de les restituer rapidement afin que les applications puissent bénéficier de leur traite- ment. Ce document se concentre sur ces deux défis relatifs aux “Big Data”. Dans notre étude, nous nous concentrons sur le stockage de sauvegarde (i) comme un moyen de protéger les données contre un certain nombre de facteurs qui peuvent les rendre indisponibles et (ii) sur le placement des données sur des systèmes de stockage répartis géographiquement, afin que les temps de latence perçue par l'utilisateur soient minimisés tout en utilisant les ressources de stockage et du réseau efficacement. Tout au long de notre étude, les données sont placées au centre de nos choix de conception dont nous essayons de tirer parti des propriétés de contenu à la fois pour le placement et le stockage efficace.

International audience; Distributed storage systems rely heavily on redundancy to ensure data availability as well as durability. In networked systems subject to intermittent node unavailability, the level of redundancy introduced in the system should be minimized and maintained upon failures. Repairs are well- known to be extremely bandwidth-consuming and it has been shown that, without care, they may significantly congest the system. In this paper, we propose an approach to redundancy management accounting for nodes heterogeneity with respect to availability. We show that by using the availability history of nodes, the performance of two important faces of distributed storage (replica placement and repair) can be significantly improved. Replica placement is achieved based on complementary nodes with respect to nodes availability, improving the overall data availability. Repairs can be scheduled thanks to an adaptive per-node timeout according to node availability, so as to decrease the number of repairs while reaching comparable availability. We propose practical heuristics for those two issues. We evaluate our approach through extensive simulations based on real and well-known availability traces. Results clearly show the benefits of our approach with regards to the critical trade-off between data availability, load-balancing and bandwidth consumption.

P2P network architectures have gained popularity as applications for sharing files between users. A P2P network provides a scalable, robust, and economical storage architecture. These features have led to the extended use of P2P network applications, ranging from file sharing to data sharing for video and telecommunication domains. The shift in storage system being used from high cost, reliable servers to usercentered storage devices led to reliability and availability problems for the P2P network. Peers are machines of users that can go offline at any time. The data stored on the machines are not available during the offline time. Data replication is a common approach for handling data unavailability, which is where multiple copies of files are placed on different peers in the network. In data replication, peers transfer complete/partial data to other nodes. Therefore, data replication provides higher data availability in case of churn. I present data replication algorithms in this thesis to improve the availability of data in the network. With an increase in availability and overhead, the basic challenges faced during the development of data replication algorithm are: (i) How many replicas for a data object should be created? (ii) On which peer(s) should the replicated data objects be stored? (iii) Which files should be replicated? Initial work in data replication considered the static replication of data based on the overall availability of nodes in the network. These approaches overestimated the number of replicas, which lead to high maintenance costs. Dynamic approaches for estimating replica numbers were developed to handle this issue. From the analysis of the current approaches, I found that the proposed mechanisms for dynamic approaches to replication did not provide a balanced replication of data. Data were only replicated to highly available nodes, which were overloaded with data. The second issue was the inability to adapt to the changing behaviour of peers. In this thesis, I present an approach that selects a node set comprised of both highly available and lowly available nodes, in order to provide load balancing in the network. I provide a feedback-based approach where previous behaviours are incorporated in the next behavioural analysis. Compared to the existing approaches to replica calculation, this approach is able to determine the appropriate number of replicas and placement locations with the changing dynamics of the system. The replication system relies on node behaviour prediction algorithms using Monte Carlo simulation and Time series analysis. Each node performs an analysis on the historical traces of its online and offline times in the network. Each node shares the availability log with the replication initiator node, and the prediction of future behaviour is made based on the logs received. The data-owning peer uses this information to run the replica placement algorithm to select nodes that are present for a particular duration, supporting the presence of each others in the network. Partial data replication is supported by the system by applying Zipf distribution to calculate the most popular files. I performed the evaluation using my replication approach and dynamic replica placement algorithms, based on the following parameters: replica count, reliability of data, average availability of nodes in the replica set, and failure analysis for querying data. The replica count analysis shows that the number of replicas required were almost half compared to the previous dynamic approaches. The reliability analysis shows that overall reliability of the data was better in this approach compared to the other dynamic replica placement algorithms. My replication algorithm produced replica sets with a lower average availability compared to the replica set of the other approaches, but the reliability analysis suggests that my approach distributes data more evenly between nodes, resulting in better overall data availability. The availability of data in the network was higher than other approaches. The failure analysis for request failures for data shows that my replication algorithm has a better node selection mechanism compared to other approaches, with better data availability.

In a heterogenous networking environment, a new practical distributed storage model is defined by introducing the concepts of repair topology and retrieval sets. How to repair a failed storage node so as to minimize the system repair cost is investigated. It is shown that the repair cost minimization problem can be decomposed into a combinatorial problem and an integer linear programming problem. Moreover, a heuristic algorithm to find suboptimal repair topologies is given. Refereed/Peer-reviewed

Version vectors (VV) are used pervasively to track dependencies between replica versions in multi-version distributed storage systems. In these systems, VV tend to have a dual functionality: identify a version and encode causal dependencies. In this paper, we show that by maintaining the identifier of the version separate from the causal past, it is possible to verify causality in constant time (instead of O(n) for VV) and to precisely track causality with information with size bounded by the degree of replication, and not by the number of concurrent writers., (undefined)

Nowadays increasing multimedia applications e.g., video and voice over IP, social networks and emails poses higher demands for sever storages and bandwidth in the networks. There is a concern that existing resource may not able to support higher demands and reliability. Network coding was introduced to improve distributed storage system. This thesis proposes the way to improve distributed storage system such as increase a chance to recover data in case there is a fail storage node or link fail in a network. In this thesis, we study the concept of network coding in distributed storage systems. We start our description from easy code which is replication coding then follow with higher complex code such as erasure coding. After that we implement these concepts in our test bed and measure performance by the probability of success in download and repair criteria. Moreover we compare success probability for reconstruction of original data between minimum storage regenerating (MSR) and minimum bandwidth regenerating (MBR) method. We also increase field size to increase probability of success. Finally, link failure was added in the test bed for measure reliability in a network. The results are analyzed and it shows that using maximum distance separable and increasing field size can improve the performance of a network. Moreover it also improves reliability of network in case there is a link failure in the repair process.

This thesis investigates the application of autonomic management to a distributed storage system. Effects on performance and resource consumption were measured in experiments, which were carried out in a local area test-bed. The experiments were conducted with components of one specific distributed storage system, but seek to be applicable to a wide range of such systems, in particular those exposed to varying conditions. The perceived characteristics of distributed storage systems depend on their configuration parameters and on various dynamic conditions. For a given set of conditions, one specific configuration may be better than another with respect to measures such as resource consumption and performance. Here, configuration parameter values were set dynamically and the results compared with a static configuration. It was hypothesised that under non-changing conditions this would allow the system to converge on a configuration that was more suitable than any that could be set a priori. Furthermore, the system could react to a change in conditions by adopting a more appropriate configuration. Autonomic management was applied to the peer-to-peer (P2P) and data retrieval components of ASA, a distributed storage system. The effects were measured experimentally for various workload and churn patterns. The management policies and mechanisms were implemented using a generic autonomic management framework developed during this work. The motivation for both groups of experiments was to test management policies with the objective to avoid unsatisfactory situations with respect to resource consumption and performance. Such unsatisfactory situations occur when either the P2P layer or the data retrieval mechanism is configured statically. In a statically configured P2P system two unsatisfactory situations can be identified. The first arises when the frequency with which P2P node states are verified is low and membership churn is high. The P2P node state becomes inaccurate due to a high membership churn, leading to errors during the routing process and a reduction in performance. In this situation it is desirable to increase the frequency to increase P2P state accuracy. The converse situation arises when the frequency is high and churn is low. In this situation network resources are used unnecessarily, which may also reduce performance, making it desirable to decrease the frequency. In ASA’s data retrieval mechanism similar unsatisfactory situations can be identified with respect to the degree of concurrency (DOC). The DOC controls the eagerness with which multiple redundant replicas are retrieved. An unsatisfactory situation arises when the DOC is low and there is a large variation in the times taken to retrieve replicas. In this situation it is desirable to increase the DOC, because by retrieving more replicas in parallel a result can be returned to the user sooner. The converse situation arises when the DOC is high, there is little variation in retrieval time and there is a network bottleneck close to the requesting client. In this situation it is desirable to decrease the DOC, since the low variation removes any benefit in parallel retrieval, and the bottleneck means that decreasing parallelism reduces both bandwidth consumption and elapsed time for the user. The experimental evaluations of autonomic management show promising results, and suggest several future research topics. These include optimisations of the managed mechanisms, alternative management policies, different evaluation methods, and the application of developed management mechanisms to other facets of a distributed storage system. The findings of this thesis could be exploited in building other distributed storage systems that focus on harnessing storage on user workstations, since these are particularly likely to be exposed to varying, unpredictable conditions.

Increasing the shares of Renewable Energy Sources (RES) and Distributed Energy Resources (DER) is one of the most important levers in many countries to cope with the environmental, political, and economic challenges of future energy supply. The underlying research question of this thesis is whether Distributed Storage Systems (DSS) at the end consumer level can economically foster the integration of intermittent and non-dispatchable resources by providing demand-side flexibility.

Remote Data Checking (RDC) is a technique by which clients can establish that data outsourced at untrusted servers remains intact over time. RDC is useful as a prevention tool, allowing clients to periodically check if data has been damaged, and as a repair tool whenever damage has been detected. Initially proposed in the context of a single server, RDC was later extended to verify data integrity in distributed storage systems that rely on replication and on erasure coding to store data redundantly at multiple servers. Recently, a technique was proposed to add redundancy based on network coding, which offers interesting tradeoffs because of its remarkably low communication overhead to repair corrupt servers.Unlike previous work on RDC which focused on minimizing the costs of the prevention phase, we take a holistic look and initiate the investigation of RDC schemes for distributed systems that rely on network coding to minimize the combined costs of both the prevention and repair phases. We propose RDC-NC, a novel secure and efficient RDC scheme for network coding-based distributed storage systems. RDC-NC mitigates new attacks that stem from the underlying principle of network coding. The scheme is able to preserve in an adversarial setting the minimal communication overhead of the repair component achieved by network coding in a benign setting. We implement our scheme and experimentally show that it is computationally inexpensive for both clients and servers.

The scientific applications of the fourth paradigm of science deal with large amounts of data stored in various storage devices or systems. Distributed storage systems are often chosen for storing these data. Some of the requirements posed to those storage systems may concern Quality of Service (QoS) aspects formally expressed in a Service Level Agreement. The QoS management in distributed storage systems is a challenging task given the possible storage device heterogeneity, the dynami- cally changing data access patterns, the client's concurrency and storage resource sharing. The problem becomes even more complicated when distributed computing environments with virtualized and shared resources like Clouds are considered. In this paper we present our research concerning the methods of storage performance management with respect of QoS in dis- tributed environments which has been done within the National Data Storage 2 (NDS2) project. A new model of storage QoS management is proposed and its implementation within NDS2 is described.

Most distributed storage systems provide limited abilities for querying data by attributes other than their primary keys. Supporting efficient search on secondary attributes is challenging as applications pose varying requirements to query processing systems, and no single system design can be suitable for all needs. In this paper, we show how to overcome these challenges in order to extend distributed data stores to support queries on secondary attributes. We propose a modular architecture that is flexible and allows query processing systems to make trade-offs according to different use case requirements. We describe adap-tive mechanisms that make use of this flexibility to enable query processing systems to dynamically adjust to query and write operation workloads., 5th Workshop on Principles and Practice of Consistency for Distributed Data, Apr 2018, Porto, Portugal. 5th Workshop on Principles and Practice of Consistency for Distributed Data April 23--26, 2018, Porto, Portugal, 2018

A simple task of storing a database or transferring it to a different point via a communication channel turns far more complex as the size of the database grows large. Limited bandwidth available for transmission plays a central role in this predicament. In two broad contexts, Content Distribution Networks (CDN) and Distributed Storage Systems (DSS), the adverse effect of the growing size of the database on the transmission bandwidth can be mitigated by exploiting additional storage units. Characterizing the optimal tradeoff between the transmission bandwidth and the storage size is the central quest to numerous works in the recent literature, including this thesis. In a DSS, individual servers fail routinely and must be replicated by downloading data from the remaining servers, a task referred to as the repair process. To render this process of repairing failed servers more straightforward and efficient, various forms of redundancy can be introduced in the system. One of the benchmarks by which the reliability of a DSS is measured is availability, which refers to the number of disjoint sets of servers that can help to repair any failed server. We study the interaction of this parameter with the amount of traffic generated during the repair process (the repair bandwidth) and the storage size. In particular, we propose a novel DSS architecture which can achieve much smaller repair bandwidth for the same availability, compared to the state of the art. In the context of CDNs, the network can be highly congested during certain hours of the day and almost idle at other times. This variability of traffic can be reduced by utilizing local storage units that prefetch the data while the network is idle. This approach is referred to as caching. In this thesis we analyze a CDN that has access to independent data from various content providers. We characterize the best caching strategy in terms of the aggregate peak traffic under the constraint that coding across contents from different libraries is prohibited. Furthermore we prove that under certain set of conditions this restriction is without loss of optimality.