Search

We show an improved lower bound for the Fp estimation problem in a data stream setting for p>2. A data stream is a sequence of items from the domain [n] with possible repetitions. The frequency vector x is an n-dimensional non-negative integer vector x such that x(i) is the number of occurrences of i in the sequence. Given an accuracy parameter Omega(n^{-1/p}) < \epsilon < 1, the problem of estimating Fp is to estimate \norm{x}_p^p = \sum_{i \in [n]} \abs{x(i)}^p correctly to within a relative accuracy of 1\pm \epsilon with high constant probability in an online fashion and using as little space as possible. The current space lower bound for this problem is Omega(n^{1-2/p} \epsilon^{-2/p}+ n^{1-2/p}\epsilon^{-4/p}/ \log^{O(1)}(n)+ (\epsilon^{-2} + \log (n))). The first term in the lower bound expression was proved in \cite{B-YJKS:stoc02,cks:ccc03}, the second in \cite{wz:arxiv11} and the third in \cite{wood:soda04}. In this note, we show an Omega(p^2 n^{1-2/p} \epsilon^{-2}/\log (n)) bits space bound, for Omega(pn^{-1/p}) \le \epsilon \le 1/10., Comment: 5 pages

We present an algorithm for computing $F_p$, the $p$th moment of an $n$-dimensional frequency vector of a data stream, for $2 < p < \log (n) $, to within $1\pm \epsilon$ factors, $\epsilon \in [n^{-1/p},1]$ with high constant probability. Let $m$ be the number of stream records and $M$ be the largest magnitude of a stream update. The algorithm uses space in bits $$ O(p^2\epsilon^{-2}n^{1-2/p}E(p,n) \log (n) \log (nmM)/\min(\log (n),\epsilon^{4/p-2}))$$ where, $E(p,n) = (1-2/p)^{-1}(1-n^{-4(1-2/p})$. Here $E(p,n)$ is $ O(1)$ for $p = 2+\Omega(1)$ and $ O(\log n)$ for $p = 2 + O(1/\log (n)$. This improves upon the space required by current algorithms \cite{iw:stoc05,bgks:soda06,ako:arxiv10,bo:arxiv10} by a factor of at least $\Omega(\epsilon^{-4/p} \min(\log (n), \epsilon^{4/p-2}))$. The update time is $O(\log (n))$. We use a new technique for designing estimators for functions of the form $\psi(\expect{X})$, where, $X$ is a random variable and $\psi$ is a smooth function, based on a low-degree Taylor polynomial expansion of $\psi(\expect{X})$ around an estimate of $\expect{X}$.

A data stream is viewed as a sequence of $M$ updates of the form $(\text{index},i,v)$ to an $n$-dimensional integer frequency vector $f$, where the update changes $f_i$ to $f_i + v$, and $v$ is an integer and assumed to be in $\{-m, ..., m\}$. The $p$th frequency moment $F_p$ is defined as $\sum_{i=1}^n \abs{f_i}^p$. We consider the problem of estimating $F_p$ to within a multiplicative approximation factor of $1\pm \epsilon$, for $p \in [0,2]$. Several estimators have been proposed for this problem, including Indyk's median estimator \cite{indy:focs00}, Li's geometric means estimator \cite{pinglib:2006}, an \Hss-based estimator \cite{gc:random07}. The first two estimators require space $\tilde{O}(\epsilon^{-2})$, where the $\tilde{O}$ notation hides polylogarithmic factors in $\epsilon^{-1}, m, n$ and $M$. Recently, Kane, Nelson and Woodruff in \cite{knw:soda10} present a space-optimal and novel estimator, called the log-cosine estimator. In this paper, we present an elementary analysis of the log-cosine estimator in a stand-alone setting. The analysis in \cite{knw:soda10} is more complicated., Comment: Withdrawn due to an error in proof of Lemma 2.2 (acknowledgement: Jelani Nelson, David Woodruff)

For each $p \in (0,2]$, we present a randomized algorithm that returns an $\epsilon$-approximation of the $p$th frequency moment of a data stream $F_p = \sum_{i = 1}^n \abs{f_i}^p$. The algorithm requires space $O(\epsilon^{-2} \log (mM)(\log n))$ and processes each stream update using time $O((\log n) (\log \epsilon^{-1}))$. It is nearly optimal in terms of space (lower bound $O(\epsilon^{-2} \log (mM))$ as well as time and is the first algorithm with these properties. The technique separates heavy hitters from the remaining items in the stream using an appropriate threshold and estimates the contribution of the heavy hitters and the light elements to $F_p$ separately. A key component is the design of an unbiased estimator for $\abs{f_i}^p$ whose data structure has low update time and low variance., Comment: Withdrawn due to error in analysis

Estimating the first moment of a data stream defined as $F_1 = \sum_{i \in \{1, 2, \ldots, n\}} \abs{f_i}$ to within $1 \pm \epsilon$-relative error with high probability is a basic and influential problem in data stream processing. A tight space bound of $O(\epsilon^{-2} \log (mM))$ is known from the work of [Kane-Nelson-Woodruff-SODA10]. However, all known algorithms for this problem require per-update stream processing time of $\Omega(\epsilon^{-2})$, with the only exception being the algorithm of [Ganguly-Cormode-RANDOM07] that requires per-update processing time of $O(\log^2(mM)(\log n))$ albeit with sub-optimal space $O(\epsilon^{-3}\log^2(mM))$. In this paper, we present an algorithm for estimating $F_1$ that achieves near-optimality in both space and update processing time. The space requirement is $O(\epsilon^{-2}(\log n + (\log \epsilon^{-1})\log(mM)))$ and the per-update processing time is $O( (\log n)\log (\epsilon^{-1}))$., Comment: 12 pages

The paper has been withdrawn due to an error in Lemma 1., Comment: Withdrawn

We consider a basic problem in the general data streaming model, namely, to estimate a vector $f \in \Z^n$ that is arbitrarily updated (i.e., incremented or decremented) coordinate-wise. The estimate $\hat{f} \in \Z^n$ must satisfy $\norm{\hat{f}-f}_{\infty}\le \epsilon\norm{f}_1 $, that is, $\forall i ~(\abs{\hat{f}_i - f_i} \le \epsilon \norm{f}_1)$. It is known to have $\tilde{O}(\epsilon^{-1})$ randomized space upper bound \cite{cm:jalgo}, $\Omega(\epsilon^{-1} \log (\epsilon n))$ space lower bound \cite{bkmt:sirocco03} and deterministic space upper bound of $\tilde{\Omega}(\epsilon^{-2})$ bits.\footnote{The $\tilde{O}$ and $\tilde{\Omega}$ notations suppress poly-logarithmic factors in $n, \log \epsilon^{-1}, \norm{f}_{\infty}$ and $\log \delta^{-1}$, where, $\delta$ is the error probability (for randomized algorithm).} We show that any deterministic algorithm for this problem requires space $\Omega(\epsilon^{-2} (\log \norm{f}_1))$ bits., Comment: Revised version

We present the \crprecis structure, that is a general-purpose, deterministic and sub-linear data structure for summarizing \emph{update} data streams. The \crprecis structure yields the \emph{first deterministic sub-linear space/time algorithms for update streams} for answering a variety of fundamental stream queries, such as, (a) point queries, (b) range queries, (c) finding approximate frequent items, (d) finding approximate quantiles, (e) finding approximate hierarchical heavy hitters, (f) estimating inner-products, (g) near-optimal $B$-bucket histograms, etc.., Comment: 11 pages