Search

We consider a fully-decentralized scenario in which no central trusted entity exists and all clients are honest-but-curious. The state-of-the-art approaches to this problem often rely on cryptographic protocols, such as multiparty computation (MPC), that require mapping real-valued data to a discrete alphabet, specifically a finite field. These approaches, however, can result in substantial accuracy losses due to computation overflows. To address this issue, we propose A-MPC, a private analog MPC protocol that performs all computations in the analog domain. We characterize the privacy of individual datasets in terms of $(\epsilon, \delta)$-local differential privacy, where the privacy of a single record in each client's dataset is guaranteed against other participants. In particular, we characterize the required noise variance in the Gaussian mechanism in terms of the required $(\epsilon,\delta)$-local differential privacy parameters by solving an optimization problem. Furthermore, compared with existing decentralized protocols, A-MPC keeps the privacy of individual datasets against the collusion of all other participants, thereby, in a notably significant improvement, increasing the maximum number of colluding clients tolerated in the protocol by a factor of three compared with the state-of-the-art collaborative learning protocols. Our experiments illustrate that the accuracy of the proposed $(\epsilon,\delta)$-locally differential private logistic regression and linear regression models trained in a fully-decentralized fashion using A-MPC closely follows that of a centralized one performed by a single trusted entity.

We consider the low rank matrix completion problem over finite fields. This problem has been extensively studied in the domain of real/complex numbers, however, to the best of authors' knowledge, there exists merely one efficient algorithm to tackle the problem in the binary field, due to Saunderson et al. [1]. In this paper, we improve upon the theoretical guarantees for the algorithm provided in [1]. Furthermore, we formulate a new graphical model for the matrix completion problem over the finite field of size $q$, $\Bbb{F}_q$, and present a message passing (MP) based approach to solve this problem. The proposed algorithm is the first one for the considered matrix completion problem over finite fields of arbitrary size. Our proposed method has a significantly lower computational complexity, reducing it from $O(n^{2r+3})$ in [1] down to $O(n^2)$ (where, the underlying matrix has dimension $n \times n$ and $r$ denotes its rank), while also improving the performance.

Due to the surge of cloud-assisted AI services, the problem of designing resilient prediction serving systems that can effectively cope with stragglers/failures and minimize response delays has attracted much interest. The common approach for tackling this problem is replication which assigns the same prediction task to multiple workers. This approach, however, is very inefficient and incurs significant resource overheads. Hence, a learning-based approach known as parity model (ParM) has been recently proposed which learns models that can generate parities for a group of predictions in order to reconstruct the predictions of the slow/failed workers. While this learning-based approach is more resource-efficient than replication, it is tailored to the specific model hosted by the cloud and is particularly suitable for a small number of queries (typically less than four) and tolerating very few (mostly one) number of stragglers. Moreover, ParM does not handle Byzantine adversarial workers. We propose a different approach, named Approximate Coded Inference (ApproxIFER), that does not require training of any parity models, hence it is agnostic to the model hosted by the cloud and can be readily applied to different data domains and model architectures. Compared with earlier works, ApproxIFER can handle a general number of stragglers and scales significantly better with the number of queries. Furthermore, ApproxIFER is robust against Byzantine workers. Our extensive experiments on a large number of datasets and model architectures also show significant accuracy improvement by up to 58% over the parity model approaches.

We consider the problem of coded distributed computing where a large linear computational job, such as a matrix multiplication, is divided into $k$ smaller tasks, encoded using an $(n,k)$ linear code, and performed over $n$ distributed nodes. The goal is to reduce the average execution time of the computational job. We provide a connection between the problem of characterizing the average execution time of a coded distributed computing system and the problem of analyzing the error probability of codes of length $n$ used over erasure channels. Accordingly, we present closed-form expressions for the execution time using binary random linear codes and the best execution time any linear-coded distributed computing system can achieve. It is also shown that there exist \textit{good} binary linear codes that not only attain (asymptotically) the best performance that any linear code (not necessarily binary) can achieve but also are numerically stable against the inevitable rounding errors in practice. We then develop a low-complexity algorithm for decoding Reed-Muller (RM) codes over erasure channels. Our decoder only involves additions, subtractions, {and inversion of relatively small matrices of dimensions at most $\log n+1$}, and enables coded computation over real-valued data. Extensive numerical analysis of the fundamental results as well as RM- and polar-coded computing schemes demonstrate the excellence of the RM-coded computation in achieving close-to-optimal performance while having a low-complexity decoding and explicit construction. The proposed framework in this paper enables efficient designs of distributed computing systems given the rich literature in the channel coding theory., Comment: Accepted for publication in IEEE Journal on Selected Areas in Information Theory. arXiv admin note: substantial text overlap with arXiv:1906.10105

We consider the problem of coded computing, where a computational task is performed in a distributed fashion in the presence of adversarial workers. We propose techniques to break the adversarial toleration threshold barrier previously known in coded computing. More specifically, we leverage list-decoding techniques for folded Reed-Solomon codes and propose novel algorithms to recover the correct codeword using side information. In the coded computing setting, we show how the master node can perform certain carefully designed extra computations to obtain the side information. The workload of computing this side information is negligible compared to the computations done by each worker. This side information is then utilized to prune the output of the list decoder and uniquely recover the true outcome. We further propose folded Lagrange coded computing (FLCC) to incorporate the developed techniques into a specific coded computing setting. Our results show that FLCC outperforms LCC by breaking the barrier on the number of adversaries that can be tolerated. In particular, the corresponding threshold in FLCC is improved by a factor of two asymptotically compared to that of LCC.

A distributed computing scenario is considered, where the computational power of a set of worker nodes is used to perform a certain computation task over a dataset that is dispersed among the workers. Lagrange coded computing (LCC), proposed by Yu et al., leverages the well-known Lagrange polynomial to perform polynomial evaluation of the dataset in such a scenario in an efficient parallel fashion while keeping the privacy of data amidst possible collusion of workers. This solution relies on quantizing the data into a finite field, so that Shamir's secret sharing, as one of its main building blocks, can be employed. Such a solution, however, is not properly scalable with the size of dataset, mainly due to computation overflows. To address such a critical issue, we propose a novel extension of LCC to the analog domain, referred to as analog LCC (ALCC). All the operations in the proposed ALCC protocol are done over the infinite fields of R/C but for practical implementations floating-point numbers are used. We characterize the privacy of data in ALCC, against any subset of colluding workers up to a certain size, in terms of the distinguishing security (DS) and the mutual information security (MIS) metrics. Also, the accuracy of outcome is characterized in a practical setting assuming operations are performed using floating-point numbers. Consequently, a fundamental trade-off between the accuracy of the outcome of ALCC and its privacy level is observed and is numerically evaluated. Moreover, we implement the proposed scheme to perform matrix-matrix multiplication over a batch of matrices. It is observed that ALCC is superior compared to the state-of-the-art LCC, implemented using fixed-point numbers, assuming both schemes use an equal number of bits to represent data symbols.

We consider the critical problem of distributed learning over data while keeping it private from the computational servers. The state-of-the-art approaches to this problem rely on quantizing the data into a finite field, so that the cryptographic approaches for secure multiparty computing can then be employed. These approaches, however, can result in substantial accuracy losses due to fixed-point representation of the data and computation overflows. To address these critical issues, we propose a novel algorithm to solve the problem when data is in the analog domain, e.g., the field of real/complex numbers. We characterize the privacy of the data from both information-theoretic and cryptographic perspectives, while establishing a connection between the two notions in the analog domain. More specifically, the well-known connection between the distinguishing security (DS) and the mutual information security (MIS) metrics is extended from the discrete domain to the continues domain. This is then utilized to bound the amount of information about the data leaked to the servers in our protocol, in terms of the DS metric, using well-known results on the capacity of single-input multiple-output (SIMO) channel with correlated noise. It is shown how the proposed framework can be adopted to do computation tasks when data is represented using floating-point numbers. We then show that this leads to a fundamental trade-off between the privacy level of data and accuracy of the result. As an application, we also show how to train a machine learning model while keeping the data as well as the trained model private. Then numerical results are shown for experiments on the MNIST dataset. Furthermore, experimental advantages are shown comparing to fixed-point implementations over finite fields.

We provide a novel framework to study subspace codes for non-coherent communications in wireless networks. To this end, an analog operator channel is defined with inputs and outputs being subspaces of $\mathbb{C}^n$. Then a certain distance is defined to capture the performance of subspace codes in terms of their capability to recover from interference and rank-deficiency of the network. We also study the robustness of the proposed model with respect to an additive noise. Furthermore, we propose a new approach to construct subspace codes in the analog domain, also regarded as Grassmann codes, by leveraging polynomial evaluations over finite fields together with characters associated to finite fields that map their elements to the unit circle in the complex plane. The constructed codes, referred to as character-polynomial (CP) codes, are shown to perform better comparing to other existing constructions of Grassmann codes in terms of the trade-off between the rate and the normalized minimum distance, for a wide range of values for $n$.

We consider the problem of coded distributed computing where a large linear computational job, such as a matrix multiplication, is divided into $k$ smaller tasks, encoded using an $(n,k)$ linear code, and performed over $n$ distributed nodes. The goal is to reduce the average execution time of the computational job. We provide a connection between the problem of characterizing the average execution time of a coded distributed computing system and the problem of analyzing the error probability of codes of length $n$ used over erasure channels. Accordingly, we present closed-form expressions for the execution time using binary random linear codes and the best execution time any linear-coded distributed computing system can achieve. It is also shown that there exist good binary linear codes that attain, asymptotically, the best performance any linear code, not necessarily binary, can achieve. We also investigate the performance of coded distributed computing systems using polar and Reed-Muller (RM) codes that can benefit from low-complexity decoding, and superior performance, respectively, as well as explicit constructions. The proposed framework in this paper can enable efficient designs of distributed computing systems given the rich literature in the channel coding theory.

We consider a distributed secret sharing system that consists of a dealer, $n$ storage nodes, and $m$ users. Each user is given access to a certain subset of storage nodes, where it can download the stored data. The dealer wants to securely convey a specific secret $s_j$ to user $j$ via storage nodes, for $j=1,2,...,m$. More specifically, two secrecy conditions are considered in this multi-user context. The weak secrecy condition is that each user does not get any information about the individual secrets of other users, while the perfect secrecy condition implies that a user does not get any information about the collection of all other users' secrets. In this system, the dealer encodes secrets into several secret shares and loads them into the storage nodes. Given a certain number of storage nodes we find the maximum number of users that can be served in such a system and construct schemes that achieve this with perfect secrecy. Lower bounds on the minimum communication complexity and the storage overhead are characterized given any $n$ and $m$. We construct distributed secret sharing protocols, under certain conditions on the system parameters, that attain the lower bound on the communication complexity while providing perfect secrecy. Furthermore, we construct protocols, again under certain conditions, that simultaneously attain the lower bounds on the communication complexity and the storage overhead while providing weak secrecy, thereby demonstrating schemes that are optimal in terms of both the parameters. It is shown how to modify the proposed protocols in order to construct schemes with balanced storage load and communication complexity.