Search

In distributed machine learning, a central node outsources computationally expensive calculations to external worker nodes. The properties of optimization procedures like stochastic gradient descent (SGD) can be leveraged to mitigate the effect of unresponsive or slow workers called stragglers, that otherwise degrade the benefit of outsourcing the computation. This can be done by only waiting for a subset of the workers to finish their computation at each iteration of the algorithm. Previous works proposed to adapt the number of workers to wait for as the algorithm evolves to optimize the speed of convergence. In contrast, we model the communication and computation times using independent random variables. Considering this model, we construct a novel scheme that adapts both the number of workers and the computation load throughout the run-time of the algorithm. Consequently, we improve the convergence speed of distributed SGD while significantly reducing the computation load, at the expense of a slight increase in communication load.

We consider the setting where a master wants to run a distributed stochastic gradient descent (SGD) algorithm on $n$ workers, each having a subset of the data. Distributed SGD may suffer from the effect of stragglers, i.e., slow or unresponsive workers who cause delays. One solution studied in the literature is to wait at each iteration for the responses of the fastest $k, Comment: arXiv admin note: substantial text overlap with arXiv:2002.11005

Guess & Check (GC) codes are systematic binary codes that can correct multiple deletions, with high probability. GC codes have logarithmic redundancy in the length of the message $k$, and the encoding and decoding algorithms of these codes are deterministic and run in polynomial time for a constant number of deletions $\delta$. The unique decoding properties of GC codes were examined in a previous work by the authors. In this paper, we investigate the list decoding performance of these codes. Namely, we study the average size and the maximum size of the list obtained by a GC decoder for a constant number of deletions $\delta$. The theoretical results show that: (i) the average size of the list approaches $1$ as $k$ grows; and (ii) there exists an infinite sequence of GC codes indexed by $k$, whose maximum list size in upper bounded by a constant that is independent of $k$. We also provide numerical simulations on the list decoding performance of GC codes for multiple values of $k$ and $\delta$., Comment: This is a full version of the short paper accepted at ISIT 2019