Search

We analyze the convergence rates of $k$ nearest neighbor density estimation method, under $\ell _{\alpha} $ norm with $\alpha \in [1,\infty]$. Our analysis includes two different cases depending on whether the support set is bounded or not. In the first case, the probability density function has a bounded support. We show that if the support set is known, then the kNN density estimator is minimax optimal under $\ell _{\alpha} $ with both $\alpha \in \big[1,\infty\big)$ and $\alpha =\infty $. If the support is unknown, the kNN density estimator is still minimax optimal under $\ell _{1}$ , but is suboptimal under $\ell _{\alpha} $ for $\alpha >1$ , and not consistent under $\ell _\infty $. In the second case, the support is unbounded and the probability density function is smooth everywhere. Moreover, the Hessian is assumed to decay with the density values. For this case, our result shows that the $\ell _\infty $ error of kNN density estimation is nearly minimax optimal. The $\ell _{\alpha} $ error for the original kNN density estimator is not consistent. To address this issue, we design a new adaptive kNN estimator, which can select different $k$ for different samples. Using this adaptive estimator, the $\ell _{\alpha} $ bound is minimax optimal. For comparison, we show that the popular kernel density estimator is not minimax optimal for this case. [ABSTRACT FROM AUTHOR]

Due to the big size of data and limited data storage volume of a single computer or a single server, data are often stored in a distributed manner. Thus, performing large-scale machine learning operations with the distributed datasets through communication networks is often required. In this paper, we study the convergence rate of the distributed dual coordinate ascent for distributed machine learning problems in a general tree-structured network. Since a tree network model can be understood as the generalization of a star network, our algorithm can be thought of as the generalization of the distributed dual coordinate ascent in a star network. We provide the convergence rate of the distributed dual coordinate ascent over a general tree network in a recursive manner and analyze the network effect on the convergence rate. Secondly, by considering network communication delays, we optimize the distributed dual coordinate ascent algorithm to maximize its convergence speed. From our analytical result, we can choose the optimal number of local iterations depending on the communication delay severity to achieve the fastest convergence speed. In numerical experiments, we consider machine learning scenarios over communication networks, where local workers cannot directly reach to a central node due to constraints in communication, and demonstrate that the usability of our distributed dual coordinate ascent algorithm in tree networks. [ABSTRACT FROM AUTHOR]

k Nearest Neighbor (kNN) method is a simple and popular statistical method for classification and regression. For both classification and regression problems, existing works have shown that, if the distribution of the feature vector has bounded support and the probability density function is bounded away from zero in its support, the convergence rate of the standard kNN method, in which k is the same for all test samples, is minimax optimal. On the contrary, if the distribution has unbounded support, we show that there is a gap between the convergence rate achieved by the standard kNN method and the minimax bound. To close this gap, we propose an adaptive kNN method, in which different k is selected for different samples. Our selection rule does not require precise knowledge of the underlying distribution of features. The proposed adaptive method significantly outperforms the standard one. We characterize the convergence rate of the proposed adaptive method, and show that it matches the minimax lower bound. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the adversarial robustness of hypothesis testing rules. In the considered model, after a sample is generated, it will be modified by an adversary before being observed by the decision maker. The decision maker needs to decide the underlying hypothesis that generates the sample from the adversarially-modified data. We formulate this problem as a minimax hypothesis testing problem, in which the goal of the adversary is to design attack strategy to maximize the error probability while the decision maker aims to design decision rules so as to minimize the error probability. We consider both hypothesis-aware case, in which the attacker knows the true underlying hypothesis, and hypothesis-unaware case, in which the attacker does not know the true underlying hypothesis. We solve this minimax problem and characterize the corresponding optimal strategies for both cases. [ABSTRACT FROM AUTHOR]

Estimating Kullback-Leibler divergence from identical and independently distributed samples is an important problem in various domains. One simple and effective estimator is based on the $k$ nearest neighbor distances between these samples. In this paper, we analyze the convergence rates of the bias and variance of this estimator. Furthermore, we derive a lower bound of the minimax mean square error and show that kNN method is asymptotically rate optimal. [ABSTRACT FROM AUTHOR]

Motivated by recent data analytics applications, we study the adversarial robustness of robust estimators. Instead of assuming that only a fraction of the data points are outliers as considered in the classic robust estimation setup, in this paper, we consider an adversarial setup in which an attacker can observe the whole dataset and can modify all data samples in an adversarial manner so as to maximize the estimation error caused by his attack. We characterize the attacker’s optimal attack strategy, and further introduce adversarial influence function (AIF) to quantify an estimator’s sensitivity to such adversarial attacks. We provide an approach to characterize AIF for any given robust estimator, and then design optimal estimator that minimizes AIF, which implies it is least sensitive to adversarial attacks and hence is most robust against adversarial attacks. From this characterization, we identify a tradeoff between AIF (i.e., robustness against adversarial attack) and influence function, a quantity used in classic robust estimators to measure robustness against outliers, and design estimators that strike a desirable tradeoff between these two quantities. [ABSTRACT FROM AUTHOR]

KSG mutual information estimator, which is based on the distances of each sample to its $k$ -th nearest neighbor, is widely used to estimate mutual information between two continuous random variables. Existing work has analyzed the convergence rate of this estimator for random variables whose densities are bounded away from zero in its support. In practice, however, KSG estimator also performs well for a much broader class of distributions, including not only those with bounded support and densities bounded away from zero, but also those with bounded support but densities approaching zero, and those with unbounded support. In this paper, we analyze the convergence rate of the error of KSG estimator for smooth distributions, whose support of density can be both bounded and unbounded. As KSG mutual information estimator can be viewed as an adaptive recombination of KL entropy estimators, in our analysis, we also provide convergence analysis of KL entropy estimator for a broad class of distributions. [ABSTRACT FROM AUTHOR]

Due to the growth of modern dataset size and the desire to harness computing power of multiple machines, there is a recent surge of interest in the design of distributed machine learning algorithms. However, distributed algorithms are sensitive to Byzantine attackers who can send falsified data to prevent the convergence of algorithms or lead the algorithms to converge to value of the attackers’ choice. Some recent work proposed interesting algorithms that can deal with the scenario when up to half of the workers are compromised. In this paper, we propose a novel algorithm that can deal with an arbitrary number of Byzantine attackers. The main idea is to ask the parameter server to randomly select a small clean dataset and compute noisy gradient using this small dataset. This noisy gradient will then be used as a ground truth to filter out information sent by compromised workers. We show that the proposed algorithm converges to the neighborhood of the population minimizer regardless the number of Byzantine attackers. We further provide numerical examples to show that the proposed algorithm can benefit from the presence of good workers and achieve better performance than existing algorithms. [ABSTRACT FROM AUTHOR]

In this paper, the problem of function computation with privacy and secrecy constraints is considered. The considered model consists of three legitimate nodes (i.e., two transmitters, Alice and Bob, and a fusion center that acts as the receiver) that observe correlated sources and are connected by noiseless public channels, and an eavesdropper Eve who has full access to the public channels and also has its own source observations. The fusion center would like to compute a function of the distributed sources within a prefixed distortion level under a certain distortion metric. To facilitate the function computation, Alice and Bob will send messages to the fusion center. Different from the existing setups in function computation, we assume that there is a privacy constraint on the sources at Alice and Bob. In particular, Alice and Bob would like to enable the fusion center to compute the function, but at same time, they do not want the fusion center to learn too much information about the source observations. We introduce a quantity to precisely measure the privacy leakage to the fusion center. In addition to this privacy constraint, we also have a secrecy constraint to Eve and use equivocation of sources to measure this quantity. Under this model, we study the tradeoffs among message rates, private information leakage, equivocation, and distortion. We first consider a scenario that has only one transmitter, i.e., the source at Bob is empty, and fully single-letter characterize the corresponding regions. Then, we consider the more general case and provide both outer and inner bounds on the corresponding regions. [ABSTRACT FROM AUTHOR]

In this paper, we consider the problem of quickly detecting an abrupt change in linear regression models. Specifically, an observer sequentially obtains a sequence of observations, whose underlying linear model changes at an unknown time. Moreover, the pre-change linear model is perfectly known by the observer but the post-change linear model is unknown. The observer aims to design an efficient online algorithm to detect the presence of the change via his sequential observations. Based on different assumptions on the change-point, both non-Bayesian and Bayesian problem formulations are considered. In the non-Bayesian setting, the change-point is modeled as a fixed but unknown constant. Two performance metrics, namely the worst case detection delay and the average run length to false alarm, are adopted to evaluate the performance of detection algorithms. In the Bayesian setting, the change-point is modeled as a geometrically distributed random variable. For this case, the average detection delay and the probability of false alarm are used as performance metrics. We propose a novel algorithm, namely the parallel-sum algorithm, for the purpose of change detection. For both setups, we show that the proposed algorithm has a low computational complexity while still offering a good performance in terms of the performance metrics of the respective setting. [ABSTRACT FROM AUTHOR]

Ultrareliable low-latency communications (URLLC) is one of three emerging application scenarios in 5G new radio (NR) for which physical layer design aspects have been specified. With 5G NR, we can guarantee reliability and latency in radio access networks. However, for communication scenarios where the transmission involves both radio access and wide-area core networks, the delay in radio access networks contributes to only a portion of the end-toend (E2E) delay. In this article, we outline the delay components and packet loss probabilities in typical URLLC scenarios and formulate the constraints on E2E delay and overall packet loss probability. Then, we summarize possible solutions in the physical, link, and network layers as well as the cross-layer design. Finally, we discuss open issues in prediction and communication codesign for URLLC in wide-area, largescale networks. [ABSTRACT FROM AUTHOR]