Search

With the increasing need for effective storage management due to ever-growing content-generation over the Internet, Distributed Storage Systems (DSS) has arisen as a valuable tool. Although DSS has considerably improved in the past years, it still leverages legacy techniques in its networking. To cope with the demanding requirements, Software Defined Networking (SDN) has revolutionized the way we manage networks and can significantly help in improving DSS network management. In this paper, we propose an SDN-based network control method that is capable of handling DSS network management and improving its performance. This paper presents the design, implementation, and evaluation using an emulated environment of a typical Data Center Network (DCN) deployment. The experiment results show that by applying the proposed method, DSS can increase the performance and service resilience compared to existing solutions.

The Cloud, with its pay-as-you-go model, gives the possibility of elastic resource management; users can claim and release resources as needed. This elasticity leads to financial and energetical cost reductions, and helps applications to cope with varying workloads.Distributed cloud and HPC applications processing large amounts of data are often co-located with a distributed storage system in order to ensure fast data accesses. Although many works have been proposed to dynamically rescale the processing part of such systems to match their workload, the storage is never considered as malleable (able to be dynamically rescaled) since moving massive amounts of data around is assumed to be too slow in practice. However, in recent years hardware and storage techniques have evolved and this assumption needs to be revisited.In this thesis, we present a study of the rescaling operations in distributed storage systems approached from different angles. We start by modeling the minimal duration of rescaling operations to estimate their potential speed. Then, we develop a benchmark to measure the viability of distributed storage system malleability on a given platform. Last, we implement a rescaling manager for distributed storage systems that decides and organizes the data transfers required during a rescaling operation.; Le Cloud, avec son modèle économique, offre la possibilité d’un gestion élastique des ressources; les utilisateurs peuvent louer des ressources selon leurs besoins. Cette élasticité permet de réduire les coûts énergétiques et financiers, et aide les applications à s’adapter aux charges de travail variables.Les applications manipulant de grandes quantités de données exécutées dans le Cloud ou sur des supercalculateurs sont souvent colocalisées avec un système de stockage distribué pour garantir un accès rapide aux données. Bien que de nombreux travaux aient été proposés pour redimensionner dynamiquement les capacités de calcul pour s’ajuster à la charge de travail, le stockage n’est pas considéré comme malléable (capable d’être redimensionné dynamiquement) puisque les transferts de grandes quantités de données nécessaires sont considérés trop lents. Cependant, le matériel et les techniques de stockage ont évolué et cette hypothèse doit être réévaluée.Dans cette thèse, nous présentons une étude sous différents angles des opérations de redimensionnement des systèmes de stockage distribués.Nous commençons par modéliser la durée minimale de ces opérations pour évaluer leur vitesse potentielle. Puis, nous développons un benchmark conçu pour mesurer la viabilité de la malléabilité d’un système de stockage sur une plateforme donnée. Finalement, nous implémentons un gestionnaire d’opérations de redimensionnement pour systèmes de stockage distribués qui décide et organise les transferts de données requis par ces opérations.

Distributed storage systems (DSS) have a potential of storing big amounts of data. Specialized DSS codes were defined with the main idea to enhance the reliability, availability, maintenance and data security in DSS systems. In this paper, we establish a new way of communication, named Distributed Communication Channel (DCC), that uses the coding mechanism that is similar to the one applied in the DSS. DCC represents a communicationchannel between two or more parties, that exchange information through multiple physical channels. It adapts the DSS concept using the benefits established with respect to the communication reliability and security against intruders and attacks. Moreover, with this concept we offer an increased security using the recent developments used in the Blockchain technology. The blockchain is used as a tool to securely bypass the transformation matrix or the exchange of encryption keys, between the sender and the receiver in the communication channel. The whole system uses the so-called spatial spreading mechanism to transfer the data through independent channels. Therefore, it is more reliable and secure, than the basic systems using the DSS algorithm.

Efficient resource utilization is a major concern for large-scale computer platforms. One method used to lower energy consumption and operational cost is to reduce the amount of idle resources. This can be achieved by using malleability, namely, the possibility for resource managers to dynamically increase or decrease the amount of resources of jobs while they are running.Decommissioning (removing from the cluster) the idle nodes as soon as possible allows the resource man- ager to quickly reallocate the nodes to other jobs. Challenges appear when such nodes host part of a dis- tributed storage system. Indeed, distributed storage systems need to transfer large amounts of data during decommission in order to ensure data availability and a constant level of fault tolerance.In this paper, we explore the possibility of relaxing the level of fault tolerance during the decommission in order to reduce the amount of data transfers needed before nodes are released, and thus return nodes to the resource manager faster. We quantify theoretically how much time and resources are saved by such a a fast decommission strategy compared with a standard decommission. We establish lower bounds for the duration of the different phases of a fast decommission. We use the lower bounds to estimate when the fast decommission would be useful to reduce the usage of core-hours.We implement a prototype of fast decommission mechanism. Using the prototype, we validate the lower bounds on the duration of the operation and confirm the findings about the core-hour usage.; Dans ce rapport, nous modélisons la durée de la décommission de noeuds d’un système de stockage distribué basé sur la réplication lorsque la contrainte de tolérance aux pannes est relachée. Une borne inférieure pour chaque phase de l’opération est obtenue. Une implémentation du mécanisme dans Pufferbench permet de valider les résultats théoriques.Cette méthode de décommission est par nature peu sure, et nous montrons que des gains ne sont obtenus que dans le cas d’un système limité par la bande passante du réseau.

International audience; Most distributed storage systems provide limited abilities for querying data by attributes other than their primary keys. Supporting efficient search on secondary attributes is challenging as applications pose varying requirements to query processing systems, and no single system design can be suitable for all needs. In this paper, we show how to overcome these challenges in order to extend distributed data stores to support queries on secondary attributes. We propose a modular architecture that is flexible and allows query processing systems to make trade-offs according to different use case requirements. We describe adap-tive mechanisms that make use of this flexibility to enable query processing systems to dynamically adjust to query and write operation workloads.

We are living in the age of information where our lives are shaped by information and communication technologies. As a consequence, there is an explosion in the amount of generated data. Distribute storage systems (DSSs) are a storage technology wherein large amounts of data are stored on a network of inexpensive storage nodes in a distributed fashion, in an efficient and inexpensive way. This thesis explores three aspects of DSSs: efficient storage, security, and privacy. We start with the description of DSSs as they are the underlying theme for the majority of the thesis. For such systems, we propose a code construction that performs efficient repair of failed systematic nodes with low repair complexity. We construct such codes by concatenating two different classes of codes. The first code serves to provide the ability to tolerate node failures, while the second allows for efficient and low complexity repair. The proposed codes achieve better repair bandwidth compared to maximum distance separable (MDS) codes, codes constructed using piggybacks, and local reconstruction/Pyramid codes, while they have a better repair complexity compared to MDS, Zigzag, Pyramid codes, and codes constructed using piggybacks. Next, we consider the notion of information-theoretic security in DSSs. To this end, we assume an eavesdropper model that has access to the data stored on a subset of nodes and a subset of the data exchanged during the repair of nodes. We present a secure coding scheme that is secure against this adversary. Furthermore, the proposed scheme ensures an efficient node repair. The security is achieved by appending the data with random symbols, and encoding the resulting sequence using a concatenated coding scheme consisting of a Gabidulin code and a repairable fountain code. We then move toward the problem of private information retrieval (PIR) in DSSs. PIR is the problem of retrieving data from the nodes without revealing the identity of the requested data (or files) to the nodes. A scheme that achieves PIR is referred to as a PIR protocol, and its efficiency is determined by its PIR rate, i.e., the ratio of downloaded data and the requested file size. For the DSS scenario, we present PIR protocols for two cases: first, where some nodes act as spies, which do not collaborate, with a goal of determining what the user is requesting. We refer to this scenario as the noncolluding case, and, second, where these nodes collaborate together in order to determine the request of a user, which we refer to as the colluding case. For the noncolluding case, we show that it is possible to achieve MDS-PIR capacity, i.e., the maximum PIR rate achieved by any protocol when the DSS stores the data using an MDS code, even when the codes used by the DSS are non-MDS. We go on to give a necessary and a sufficient condition based on the structure of the storage code for our PIR protocol to achieve this capacity. We refer to such codes as MDS-PIR capacity-achieving codes. Furthermore, we show that the rates achieved by these codes using the proposed protocol are indeed equal to the maximum PIR rate achieved by these codes under any protocol. Subsequently, we show that cyclic codes, Reed-Muller (RM) codes, and a class of distance-optimal local reconstruction codes are MDS-PIR capacity-achieving codes. For the colluding case, we present a PIR protocol for DSSs that store data using arbitrary linear codes. Following this, as in the noncolluding case, we give a necessary and a sufficient condition based on the structure of the underlying codes that allow the proposed protocol to achieve an upper bound on the PIR rate and present codes that allow this. In particular, we show that if the underlying codes are RM codes, then the protocol achieves the maximum PIR rate. The concepts learned while finding solutions to the privacy issues for DSSs are then applied to distributed caching in wireless networks, which is one of the fundamental techniques that will be implemented in future 5G wireless networks. In the considered distributed caching scenario, the files are encoded using MDS codes and then cached on the small-cell base stations (SBSs), with the goal to reduce the backhaul usage. We introduce the notion of PIR in this setting and present a protocol that achieves PIR. One essential difference between DSSs and distributed caching is that in distributed caching the files to be cached are typically cached using codes with different rates that depend on the file popularity distribution. We assume that the SBSs can collude and as such the protocol is a generalization of the protocol considered for the colluding case in DSSs. As a surprising result, we show that to minimize backhaul usage, uniform content placement, i.e., the files are cached using a single code, is optimal.

Heteregenous distributed storage systems consists of nodes which have different CPU powers and communication bandwiths while erasure codes that are commonly used on them significantly vary in their use of these resources. Clustering the nodes of a distributed storage system based on the resource costs and assigning appropriate erasure codes to each cluster can improve the performance of file repair operations. A simulaton study shows that this approach can reduce file repair costs somewhere between 6 percent to 44 percent by dividing nodes in three clusters and using HSRC, EMBR and HMBR erasure codes on these clusters. © 2017 IEEE.

In wireless distributed storage systems, storage nodes are connected by wireless channels, which are broadcast in nature. This paper exploits this unique feature to design an efficient repair mechanism, called broadcast repair, for wireless distributed storage systems in the presence of multiple-node failures. Due to the broadcast nature of wireless transmission, we advocate a new measure on repair performance called repair-transmission bandwidth. In contrast to repair bandwidth, which measures the average number of packets downloaded by a newcomer to replace a failed node, repair-transmission bandwidth measures the average number of packets transmitted by helper nodes per failed node. A fundamental study on the storage capacity of wireless distributed storage systems with broadcast repair is conducted by modeling the storage system as a multicast network and analyzing the minimum cut of the corresponding information flow graph. The fundamental tradeoff between storage efficiency and repair-transmission bandwidth is also obtained for functional repair. The performance of broadcast repair is compared both analytically and numerically with that of cooperative repair, the basic repair method for wired distributed storage systems with multiple-node failures. While cooperative repair is based on the idea of allowing newcomers to exchange packets, broadcast repair is based on the idea of allowing a helper to broadcast packets to all newcomers simultaneously. We show that broadcast repair outperforms cooperative repair, offering a better tradeoff between storage efficiency and repair-transmission bandwidth., Comment: 28 pages, 7 figures

Distributed storage systems store some redundant data to keep the degree of availability of the stored data constant and also to increase the system's resistance against failures. This type of systems usually use pure replication or methods based on RAID systems as redundancy schemes. In this paper, we study the communication cost of a distributed data storage system using Maximum Distance Separable (MDS) erasure codes. Our focus is reduction of the cost of one-tomany communication used in data reconstruction/repair initialization and update operations. We propose the use of two different communication approaches on the area of distributed storage systems for the above operations; Steiner tree approach and multi-shortest path approach. We also analyse these two communication approaches empirically and theoretically. Our theoretical results indicate that Steiner tree approach has lower message usage, whereas, multi-shortest path approach has lower time usage for data reconstruction/repair initialization operations. On the other hand, Steiner tree approach has better message and time metrics for the data update process. Furthermore, our experimental results support these theoretical results. Thus, users can choose between the two approaches depending on their needs and priorities. © 2017 Slovak Academy of Sciences. All rights reserved.

Distributed storage systems store some redundant data to keep the degree of availability of the stored data constant and also to increase the system's resistance against failures. This type of systems usually use pure replication or methods based on RAID systems as redundancy schemes. In this paper, we study the communication cost of a distributed data storage system using Maximum Distance Separable (MDS) erasure codes. Our focus is reduction of the cost of one-to-many communication used in data reconstruction/repair initialization and update operations. We propose the use of two different communication approaches on the area of distributed storage systems for the above operations; Steiner tree approach and multi-shortest path approach. We also analyse these two communication approaches empirically and theoretically. Our theoretical results indicate that Steiner tree approach has lower message usage, whereas, multi-shortest path approach has lower time usage for data reconstruction/repair initialization operations. On the other hand, Steiner tree approach has better message and time metrics for the data update process. Furthermore, our experimental results support these theoretical results. Thus, users can choose between the two approaches depending on their needs and priorities.

The advent of the information age has bestowed upon us three challenges related to the way we deal with data. Firstly, there is an unprecedented demand for transmitting data at high rates. Secondly, the massive amounts of data being collected from various sources needs to be stored across time. Thirdly, there is a need to process the data collected and perform computations on it in order to extract meaningful information out of it. The interconnected nature of modern systems designed to perform these tasks has unraveled new difficulties when it comes to ensuring their resilience against sources of performance degradation. In the context of network communication and distributed data storage, system-level noise and adversarial errors have to be combated with efficient error correction schemes. In the case of distributed computation, the heterogeneous nature of computing clusters can potentially diminish the speedups promised by parallel algorithms, calling for schemes that mitigate the effect of slow machines and communication delay. This thesis addresses the problem of designing efficient fault tolerance schemes for the three scenarios just described. In the network communication setting, a family of multiple-source multicast networks that employ linear network coding is considered for which capacity-achieving distributed error-correcting codes, based on classical algebraic constructions, are designed. The codes require no coordination between the source nodes and are end to end: except for the source nodes and the destination node, the operation of the network remains unchanged. In the context of data storage, balanced error-correcting codes are constructed so that the encoding effort required is balanced out across the storage nodes. In particular, it is shown that for a fixed row weight, any cyclic Reed-Solomon code possesses a generator matrix in which the number of nonzeros is the same across the columns. In the balanced and sparsest case, where each row of the generator matrix is a minimum distance codeword, the maximal encoding time over the storage nodes is minimized, a property that is appealing in write-intensive settings. Analogous constructions are presented for a locally recoverable code construction due to Tamo and Barg. Lastly, the problem of mitigating stragglers in a distributed computation setup is addressed, where a function of some dataset is computed in parallel. Using Reed-Solomon coding techniques, a scheme is proposed that allows for the recovery of the function under consideration from the minimum number of machines possible. The only assumption made on the function is that it is additively separable, which renders the scheme useful in distributed gradient descent implementations. Furthermore, a theoretical model for the run time of the scheme is presented. When the return time of the machines is modeled probabilistically, the model can be used to optimally pick the scheme's parameters so that the expected computation time is minimized. The recovery is performed using an algorithm that runs in quadratic time and linear space, a notable improvement compared to state-of-the-art schemes. The unifying theme of the three scenarios is the construction of error-correcting codes whose encoding functions adhere to certain constraints. It is shown that in many cases, these constraints can be satisfied by classical constructions. As a result, the schemes presented are deterministic, operate over small finite fields and can be decoded using efficient algorithms.

We present a general framework for private information retrieval (PIR) from arbitrary coded databases that allows one to adjust the rate of the scheme to the suspected number of colluding servers. If the storage code is a generalized Reed--Solomon code of length $n$ and dimension $k$, we design PIR schemes that achieve a PIR rate of $\frac{n-(k+t-1)}{n}$ while protecting against any $t$ colluding servers, for any $1\leq t\leq n-k$. This interpolates between the previously studied cases of $t=1$ and $k=1$ and achieves PIR capacity in both of these cases asymptotically as the number of files in the database grows.

This paper presents a flexible irregular model for heterogeneous cloud storage systems and investigates how the cost of repairing failed nodes can be minimized. The fractional repetition code, originally designed for minimizing repair bandwidth for homogeneous storage systems, is generalized to the irregular fractional repetition code, which is adaptable to heterogeneous environments. The code structure and the associated storage allocation can be obtained by solving an integer linear programming problem. For moderate sized networks, a heuristic algorithm is proposed and shown to be near-optimal by computer simulations., Comment: 12 pages, 10 figures. to appear in IEEE Journal on Selected Areas in Communications 2014

An increasing number of web applications run totally or partially in the client machines - from collaborative editing tools to multi-user games. Avoiding to continuously contact the server allows to reduce latency between clients and to minimize the load on the centralized component. Novel implementation techniques, such as WebRTC, allows to address network problems that previously prevented these systems to be deployed in practice. Legion is a newly developed framework that exploits these mechanisms, allowing client web applications to replicate data from servers, and synchronize these replicas directly among them. This work aims to extend the current Legion framework with the integration of an additional legacy storage system that can be used to support web applications, Antidote. We study the best way to support Legion’s data model into Antidote, we design a synchronization mechanism, and finally we measure the performance cost of such an integration.

Three transmission schemes over a new type of multiple-access channel (MAC) model with inter-source communication links are proposed and investigated in this paper. This new channel model is well motivated by, e.g., wireless distributed storage networks, where communication to repair a lost node takes place from helper nodes to a repairing node over a wireless channel. Since in many wireless networks nodes can come and go in an arbitrary manner, there must be an inherent capability of inter-node communication between every pair of nodes. Assuming that communication is possible between every pair of helper nodes, the newly proposed schemes are based on various smart time-sharing and relaying strategies. In other words, certain helper nodes will be regarded as relays, thereby converting the conventional uncooperative multiple-access channel to a multiple-access relay channel (MARC). The diversity-multiplexing gain tradeoff (DMT) of the system together with efficient sphere-decodability and low structural complexity in terms of the number of antennas required at each end is used as the main design objectives. While the optimal DMT for the new channel model is fully open, it is shown that the proposed schemes outperform the DMT of the simple time-sharing protocol and, in some cases, even the optimal uncooperative MAC DMT. While using a wireless distributed storage network as a motivating example throughout the paper, the MAC transmission techniques proposed here are completely general and as such applicable to any MAC communication with inter-source communication links.

Regenerating codes for distributed storage have attracted much research interest in the past decade. Such codes trade the bandwidth needed to repair a failed node with the overall amount of data stored in the network. Minimum storage regenerating (MSR) codes are an important class of optimal regenerating codes that minimize (first) the amount of data stored per node and (then) the repair bandwidth. Specifically, an $[n,k,d]$-$(\alpha)$ MSR code $\mathbb{C}$ over $\mathbb{F}_q$ is defined as follows. Using such a code $\mathbb{C}$, a file $\cal{F}$ consisting of $\alpha k$ symbols over $\mathbb{F}_q$ can be distributed among $n$ nodes, each storing $\alpha$ symbols, in such a way that: The file $\cal{F}$ can be recovered by downloading the content of any $k$ of the $n$ nodes; and the content of any failed node can be reconstructed by accessing any $d$ of the remaining $n-1$ nodes and downloading $\alpha/(d-k+1)$ symbols from each of these nodes. Unfortunately, explicit constructions of $[n,k,d]$ MSR codes are known only for certain special cases: either low rate, namely $k/n0.5$ and low repair connectivity $k< d, Comment: 9 pages, 3 figures, a short version of this paper has been submitted to the 2016 IEEE International Symposium on Information Theory

Key-homomorphic pseudo random functions (KH-PRF) have many practical applications including proxy re-encryption, distributed credential protection systems and updatable encryption.We present a key-homomorphic pseudo random function that is homomorphic with respect to a significant part of the secret key and analyse its security. Previous constructions rely on the learning with errors problem which adds some small error to the homomorphic operations due to the noisy outputs. Our construction, based on elliptic curves, removes the need of adding this noise at the cost of adding a few bits to the secret key for which homomorphism does not follow. The main advantage of our construction is that homomorphism can be applied several times without incurring into errors. In particular, we show how our KH-PRF can be used to provide key updatable encryption to distributed storage networks. Also, by relaxing the security assumptions, our PRF can be modified to be homomorphic with respect to the entire key. Refereed/Peer-reviewed

In this paper, we propose a new class of regular fractional repetition (FR) codes constructed from perfect difference families and quasi-perfect difference families to store big data in distributed storage systems. The main advantage of the proposed construction method is that it supports a wide range of code parameter values compared to existing ones, which is an important feature to be adopted in practical systems. When using one instance of the proposed codes for a given parameter set, we show that the amount of stored data is very close to that of an existing state-of-the-art optimal FR code.

Active distributed storages need to assure both consistency and dynamic data support, in addition to availability, confidentiality and resiliency. Further, since storage durability suffers in untrusted and unreliable environments, it becomes crucial to (a) select the most reliable set of servers to assure data retrievability and (b) dynamically identify errant servers and restore the data to ensure data recoverability. We address the issues of concurrency, consistency, dynamic data support, data share repair, and trust management in providing persistent storage and access. The paper focuses primarily on erasure coded distributed storages (storages employing erasure coding for data dispersal). Integration of Quorum based approach using Notification propagation, with a reliability model based on server trust-reputation forms the comprehensive design proposed. Treating servers and their data shares equally at data reconstructions during data retrievals is rather inadequate in untrusted environments. The design provides a suitable platform for use of Soft Decision Decoding to overcome this inadequacy. The design has been validated by the simulation, study, and analysis carried out for Reed Solomon coded storage with varying levels of resiliency and concurrency. The proposed design can be suitably adapted in typical distributed information storages catering to global networked audience in public, untrusted, and unreliable operating environments.

Bu tezde, verinin hata toleranslı bir şekilde depolanmasını sağlamak için kullanılan kaynaklardan başta bant genişliği ve toplam işlem süresi olmak üzere çeşitli kaynakların maliyetinin düşürülmesini sağlayacak çözümler önerilmiştir. Bu tez kapsamındaki çalışmalar beş bölüme ayrılmıştır. Bunlardan ilki MDS silinti kodlama kullanan dağıtık depolama sistemlerinde düğüm tamirinin ve verinin geri çatılmasının başlatımı ve veri güncellemesi işlemlerinde kullanılan süre ve bant genişliği maliyetini düşürmeyi hedefleyen topoloji farkındalıklı bir çalışmadır. İkinci çalışmada ise yeni bir melez kodlama şeması olan Homomorfik Minimum Bant Genişliği Tamir (HMBR) kodları geliştirilmiştir. HMBR kodlama şeması düğüm tamiri ve verinin geri çatılması işlemlerinde sırasıyla bant genişliği ve işlem süresi maliyetlerini iyileştiren iki farklı yöntem sağlamaktadır. Ayrıca önerilen kodlama şeması düğüm tamiri için bütünlük kontrolü mekanizmasına sahiptir. Üçüncü çalışmada da yeni bir melez kodlama şeması olan Homomorfik Minimum Depolama Tamir (HMSR) kodları geliştirilmiştir. HMSR kodları düğüm başına minimum depolama maliyeti gerektirirken bir yandan da sırasıyla bant genişliği ve işlem süresi maliyetlerini azaltan iki farklı düğüm tamiri yöntemi sağlamaktadır. Dördüncü çalışmada ise farklı düğümlerin farklı kodları kullanabildiği kümeleme tabanlı dağıtık bir depolama sistemi tasarlanmıştır. Son çalışmada ise silinti kodlama kullanan dağıtık depolama sistemlerinde veri geri çatma işlemi gerçekleştirilirken ağda bir tıkanma oluştuğunda, TCP soketlerinin yönetilmesi yoluyla bu işlemin toplam süresini düşüren bir algoritma önerilmiştir., In this thesis, solutions for reducing the cost of the sources -mainly bandwidth and total processing time- used to store data in fault tolerant way are proposed. The work in this thesis is divided into five parts. The first of these is a topology aware solution proposed for reducing time and bandwidth used for data update and the initiation of node repair and data reconstruction processes in distributed storage systems using MDS erasure codes. The second one is a new hybrid coding scheme: Homomorphic Minimum Bandwidth Repairing (HMBR) codes. HMBR codes provide two different node repair methods as well as two different data reconstruction methods for reducing bandwidth usage and the processing time. In addition, this new coding scheme presents an integrity checking mechanism for repaired node's content. In the third part of our work, another hybrid coding scheme called Homomorphic Minimum Storage Repairing Codes (HMSR) which minimizes storage cost on the nodes is designed. Moreover, these codes present two different node repair methods reducing either bandwith usage or elapsed time in node repair. In the fourth part of the thesis, the cluster based storage system is proposed in which different nodes can use different coding schemes. As the last part of our work, an algorithm is proposed for reducing the time elapsed in data reconstruction process of distributed storage systems using erasure codes in case of network congestion.

Distributed storage systems and associated storage codes can efficiently store a large amount of data while ensuring that data is retrievable in case of node failure. The study of such systems, particularly the design of storage codes over finite fields, assumes that the physical channel through which the nodes communicate is error-free. This is not always the case, for example, in a wireless storage system. We study the probability that a subpacket is repaired incorrectly during node repair in a distributed storage system, in which the nodes communicate over an AWGN or Rayleigh fading channels. The asymptotic probability (as SNR increases) that a node is repaired incorrectly is shown to be completely determined by the repair locality of the DSS and the symbol error rate of the wireless channel. Lastly, we propose some design criteria for physical layer coding in this scenario, and use it to compute optimally rotated QAM constellations for use in wireless distributed storage systems., To appear in ISITA 2014

Cloud service providers, social networks and data-management companies are witnessing a tremendous increase in the amount of data they receive every day. All this data creates new opportunities to expand human knowledge in fields like healthcare and human behavior and improve offered services like search, recommendation, and many others. It is not by accident that many academics but also public media refer to our era as the “Big Data” era. But these huge opportunities come with the requirement for better data management systems that, on one hand, can safely accommodate this huge and constantly increasing volume of data and, on the other, serve them in a timely and useful manner so that applications can benefit from processing them. This document focuses on the above two challenges that come with “Big Data”. In more detail, we study (i) backup storage systems as a means to safeguard data against a number of factors that may render them unavailable and (ii) data placement strategies on geographically distributed storage systems, with the goal to reduce the user perceived latencies and the network and storage resources are efficiently utilized. Throughout our study, data are placed in the centre of our design choices as we try to leverage content properties for both placement and efficient storage., Les fournisseurs de services de cloud computing, les réseaux sociaux et les entreprises de gestion des données ont assisté à une augmentation considérable du volume de données qu'ils reçoivent chaque jour. Toutes ces données créent des nouvelles opportunités pour étendre la connaissance humaine dans des domaines comme la santé, l'urbanisme et le comportement humain et permettent d'améliorer les services offerts comme la recherche, la recommandation, et bien d'autres. Ce n'est pas par accident que plusieurs universitaires mais aussi les médias publics se référent à notre époque comme l'époque “Big Data”. Mais ces énormes opportunités ne peuvent être exploitées que grâce à de meilleurs systèmes de gestion de données. D'une part, ces derniers doivent accueillir en toute sécurité ce volume énorme de données et, d'autre part, être capable de les restituer rapidement afin que les applications puissent bénéficier de leur traite- ment. Ce document se concentre sur ces deux défis relatifs aux “Big Data”. Dans notre étude, nous nous concentrons sur le stockage de sauvegarde (i) comme un moyen de protéger les données contre un certain nombre de facteurs qui peuvent les rendre indisponibles et (ii) sur le placement des données sur des systèmes de stockage répartis géographiquement, afin que les temps de latence perçue par l'utilisateur soient minimisés tout en utilisant les ressources de stockage et du réseau efficacement. Tout au long de notre étude, les données sont placées au centre de nos choix de conception dont nous essayons de tirer parti des propriétés de contenu à la fois pour le placement et le stockage efficace.

International audience; Distributed storage systems rely heavily on redundancy to ensure data availability as well as durability. In networked systems subject to intermittent node unavailability, the level of redundancy introduced in the system should be minimized and maintained upon failures. Repairs are well- known to be extremely bandwidth-consuming and it has been shown that, without care, they may significantly congest the system. In this paper, we propose an approach to redundancy management accounting for nodes heterogeneity with respect to availability. We show that by using the availability history of nodes, the performance of two important faces of distributed storage (replica placement and repair) can be significantly improved. Replica placement is achieved based on complementary nodes with respect to nodes availability, improving the overall data availability. Repairs can be scheduled thanks to an adaptive per-node timeout according to node availability, so as to decrease the number of repairs while reaching comparable availability. We propose practical heuristics for those two issues. We evaluate our approach through extensive simulations based on real and well-known availability traces. Results clearly show the benefits of our approach with regards to the critical trade-off between data availability, load-balancing and bandwidth consumption.

P2P network architectures have gained popularity as applications for sharing files between users. A P2P network provides a scalable, robust, and economical storage architecture. These features have led to the extended use of P2P network applications, ranging from file sharing to data sharing for video and telecommunication domains. The shift in storage system being used from high cost, reliable servers to usercentered storage devices led to reliability and availability problems for the P2P network. Peers are machines of users that can go offline at any time. The data stored on the machines are not available during the offline time. Data replication is a common approach for handling data unavailability, which is where multiple copies of files are placed on different peers in the network. In data replication, peers transfer complete/partial data to other nodes. Therefore, data replication provides higher data availability in case of churn. I present data replication algorithms in this thesis to improve the availability of data in the network. With an increase in availability and overhead, the basic challenges faced during the development of data replication algorithm are: (i) How many replicas for a data object should be created? (ii) On which peer(s) should the replicated data objects be stored? (iii) Which files should be replicated? Initial work in data replication considered the static replication of data based on the overall availability of nodes in the network. These approaches overestimated the number of replicas, which lead to high maintenance costs. Dynamic approaches for estimating replica numbers were developed to handle this issue. From the analysis of the current approaches, I found that the proposed mechanisms for dynamic approaches to replication did not provide a balanced replication of data. Data were only replicated to highly available nodes, which were overloaded with data. The second issue was the inability to adapt to the changing behaviour of peers. In this thesis, I present an approach that selects a node set comprised of both highly available and lowly available nodes, in order to provide load balancing in the network. I provide a feedback-based approach where previous behaviours are incorporated in the next behavioural analysis. Compared to the existing approaches to replica calculation, this approach is able to determine the appropriate number of replicas and placement locations with the changing dynamics of the system. The replication system relies on node behaviour prediction algorithms using Monte Carlo simulation and Time series analysis. Each node performs an analysis on the historical traces of its online and offline times in the network. Each node shares the availability log with the replication initiator node, and the prediction of future behaviour is made based on the logs received. The data-owning peer uses this information to run the replica placement algorithm to select nodes that are present for a particular duration, supporting the presence of each others in the network. Partial data replication is supported by the system by applying Zipf distribution to calculate the most popular files. I performed the evaluation using my replication approach and dynamic replica placement algorithms, based on the following parameters: replica count, reliability of data, average availability of nodes in the replica set, and failure analysis for querying data. The replica count analysis shows that the number of replicas required were almost half compared to the previous dynamic approaches. The reliability analysis shows that overall reliability of the data was better in this approach compared to the other dynamic replica placement algorithms. My replication algorithm produced replica sets with a lower average availability compared to the replica set of the other approaches, but the reliability analysis suggests that my approach distributes data more evenly between nodes, resulting in better overall data availability. The availability of data in the network was higher than other approaches. The failure analysis for request failures for data shows that my replication algorithm has a better node selection mechanism compared to other approaches, with better data availability.

In a heterogenous networking environment, a new practical distributed storage model is defined by introducing the concepts of repair topology and retrieval sets. How to repair a failed storage node so as to minimize the system repair cost is investigated. It is shown that the repair cost minimization problem can be decomposed into a combinatorial problem and an integer linear programming problem. Moreover, a heuristic algorithm to find suboptimal repair topologies is given. Refereed/Peer-reviewed

Version vectors (VV) are used pervasively to track dependencies between replica versions in multi-version distributed storage systems. In these systems, VV tend to have a dual functionality: identify a version and encode causal dependencies. In this paper, we show that by maintaining the identifier of the version separate from the causal past, it is possible to verify causality in constant time (instead of O(n) for VV) and to precisely track causality with information with size bounded by the degree of replication, and not by the number of concurrent writers., (undefined)

Nowadays increasing multimedia applications e.g., video and voice over IP, social networks and emails poses higher demands for sever storages and bandwidth in the networks. There is a concern that existing resource may not able to support higher demands and reliability. Network coding was introduced to improve distributed storage system. This thesis proposes the way to improve distributed storage system such as increase a chance to recover data in case there is a fail storage node or link fail in a network. In this thesis, we study the concept of network coding in distributed storage systems. We start our description from easy code which is replication coding then follow with higher complex code such as erasure coding. After that we implement these concepts in our test bed and measure performance by the probability of success in download and repair criteria. Moreover we compare success probability for reconstruction of original data between minimum storage regenerating (MSR) and minimum bandwidth regenerating (MBR) method. We also increase field size to increase probability of success. Finally, link failure was added in the test bed for measure reliability in a network. The results are analyzed and it shows that using maximum distance separable and increasing field size can improve the performance of a network. Moreover it also improves reliability of network in case there is a link failure in the repair process.

Increasing the shares of Renewable Energy Sources (RES) and Distributed Energy Resources (DER) is one of the most important levers in many countries to cope with the environmental, political, and economic challenges of future energy supply. The underlying research question of this thesis is whether Distributed Storage Systems (DSS) at the end consumer level can economically foster the integration of intermittent and non-dispatchable resources by providing demand-side flexibility.

Remote Data Checking (RDC) is a technique by which clients can establish that data outsourced at untrusted servers remains intact over time. RDC is useful as a prevention tool, allowing clients to periodically check if data has been damaged, and as a repair tool whenever damage has been detected. Initially proposed in the context of a single server, RDC was later extended to verify data integrity in distributed storage systems that rely on replication and on erasure coding to store data redundantly at multiple servers. Recently, a technique was proposed to add redundancy based on network coding, which offers interesting tradeoffs because of its remarkably low communication overhead to repair corrupt servers.Unlike previous work on RDC which focused on minimizing the costs of the prevention phase, we take a holistic look and initiate the investigation of RDC schemes for distributed systems that rely on network coding to minimize the combined costs of both the prevention and repair phases. We propose RDC-NC, a novel secure and efficient RDC scheme for network coding-based distributed storage systems. RDC-NC mitigates new attacks that stem from the underlying principle of network coding. The scheme is able to preserve in an adversarial setting the minimal communication overhead of the repair component achieved by network coding in a benign setting. We implement our scheme and experimentally show that it is computationally inexpensive for both clients and servers.