Search

This paper proposes Mutual Information Regularized Assignment (MIRA), a pseudo-labeling algorithm for unsupervised representation learning inspired by information maximization. We formulate online pseudo-labeling as an optimization problem to find pseudo-labels that maximize the mutual information between the label and data while being close to a given model probability. We derive a fixed-point iteration method and prove its convergence to the optimal solution. In contrast to baselines, MIRA combined with pseudo-label prediction enables a simple yet effective clustering-based representation learning without incorporating extra training techniques or artificial constraints such as sampling strategy, equipartition constraints, etc. With relatively small training epochs, representation learned by MIRA achieves state-of-the-art performance on various downstream tasks, including the linear/k-NN evaluation and transfer learning. Especially, with only 400 epochs, our method applied to ImageNet dataset with ResNet-50 architecture achieves 75.6% linear evaluation accuracy., Comment: NeurIPS 2022

Test-time adaptation (TTA) aims to adapt a trained classifier using online unlabeled test data only, without any information related to the training procedure. Most existing TTA methods adapt the trained classifier using the classifier's prediction on the test data as pseudo-label. However, under test-time domain shift, accuracy of the pseudo labels cannot be guaranteed, and thus the TTA methods often encounter performance degradation at the adapted classifier. To overcome this limitation, we propose a novel test-time adaptation method, called Test-time Adaptation via Self-Training with nearest neighbor information (TAST), which is composed of the following procedures: (1) adds trainable adaptation modules on top of the trained feature extractor; (2) newly defines a pseudo-label distribution for the test data by using the nearest neighbor information; (3) trains these modules only a few times during test time to match the nearest neighbor-based pseudo label distribution and a prototype-based class distribution for the test data; and (4) predicts the label of test data using the average predicted class distribution from these modules. The pseudo-label generation is based on the basic intuition that a test data and its nearest neighbor in the embedding space are likely to share the same label under the domain shift. By utilizing multiple randomly initialized adaptation modules, TAST extracts useful information for the classification of the test data under the domain shift, using the nearest neighbor information. TAST showed better performance than the state-of-the-art TTA methods on two standard benchmark tasks, domain generalization, namely VLCS, PACS, OfficeHome, and TerraIncognita, and image corruption, particularly CIFAR-10/100C.

We propose Few-Example Clustering (FEC), a novel algorithm that performs contrastive learning to cluster few examples. Our method is composed of the following three steps: (1) generation of candidate cluster assignments, (2) contrastive learning for each cluster assignment, and (3) selection of the best candidate. Based on the hypothesis that the contrastive learner with the ground-truth cluster assignment is trained faster than the others, we choose the candidate with the smallest training loss in the early stage of learning in step (3). Extensive experiments on the \textit{mini}-ImageNet and CUB-200-2011 datasets show that FEC outperforms other baselines by about 3.2% on average under various scenarios. FEC also exhibits an interesting learning curve where clustering performance gradually increases and then sharply drops.

We propose unsupervised embedding adaptation for the downstream few-shot classification task. Based on findings that deep neural networks learn to generalize before memorizing, we develop Early-Stage Feature Reconstruction (ESFR) -- a novel adaptation scheme with feature reconstruction and dimensionality-driven early stopping that finds generalizable features. Incorporating ESFR consistently improves the performance of baseline methods on all standard settings, including the recently proposed transductive method. ESFR used in conjunction with the transductive method further achieves state-of-the-art performance on mini-ImageNet, tiered-ImageNet, and CUB; especially with 1.2%~2.0% improvements in accuracy over the previous best performing method on 1-shot setting., Comment: Accepted to ICML 2021

The generalization ability of most meta-reinforcement learning (meta-RL) methods is largely limited to test tasks that are sampled from the same distribution used to sample training tasks. To overcome the limitation, we propose Latent Dynamics Mixture (LDM) that trains a reinforcement learning agent with imaginary tasks generated from mixtures of learned latent dynamics. By training a policy on mixture tasks along with original training tasks, LDM allows the agent to prepare for unseen test tasks during training and prevents the agent from overfitting the training tasks. LDM significantly outperforms standard meta-RL methods in test returns on the gridworld navigation and MuJoCo tasks where we strictly separate the training task distribution and the test task distribution., Comment: NeurIPS 2021

Conventional out-of-distribution (OOD) detection schemes based on variational autoencoder or Random Network Distillation (RND) have been observed to assign lower uncertainty to the OOD than the target distribution. In this work, we discover that such conventional novelty detection schemes are also vulnerable to the blurred images. Based on the observation, we construct a novel RND-based OOD detector, SVD-RND, that utilizes blurred images during training. Our detector is simple, efficient at test time, and outperforms baseline OOD detectors in various domains. Further results show that SVD-RND learns better target distribution representation than the baseline RND algorithm. Finally, SVD-RND combined with geometric transform achieves near-perfect detection accuracy on the CelebA dataset., Comment: ICLR 2020

Since deep neural networks are over-parameterized, they can memorize noisy examples. We address such a memorization issue in the presence of label noise. From the fact that deep neural networks cannot generalize to neighborhoods of memorized features, we hypothesize that noisy examples do not consistently incur small losses on the network under a certain perturbation. Based on this, we propose a novel training method called Learning with Ensemble Consensus (LEC) that prevents overfitting to noisy examples by removing them based on the consensus of an ensemble of perturbed networks. One of the proposed LECs, LTEC outperforms the current state-of-the-art methods on noisy MNIST, CIFAR-10, and CIFAR-100 in an efficient manner., Comment: ICLR 2020

We consider the problem of sparse phase retrieval from Fourier transform magnitudes to recover the $k$-sparse signal vector and its support $\mathcal{T}$. We exploit extended support estimate $\mathcal{E}$ with size larger than $k$ satisfying $\mathcal{E} \supseteq \mathcal{T}$ and obtained by a trained deep neural network (DNN). To make the DNN learnable, it provides $\mathcal{E}$ as the union of equivalent solutions of $\mathcal{T}$ by utilizing modulo Fourier invariances. Set $\mathcal{E}$ can be estimated with short running time via the DNN, and support $\mathcal{T}$ can be determined from the DNN output rather than from the full index set by applying hard thresholding to $\mathcal{E}$. Thus, the DNN-based extended support estimation improves the reconstruction performance of the signal with a low complexity burden dependent on $k$. Numerical results verify that the proposed scheme has a superior performance with lower complexity compared to local search-based greedy sparse phase retrieval and a state-of-the-art variant of the Fienup method.

We consider the classical sparse regression problem of recovering a sparse signal $x_0$ given a measurement vector $y = \Phi x_0+w$. We propose a tree search algorithm driven by the deep neural network for sparse regression (TSN). TSN improves the signal reconstruction performance of the deep neural network designed for sparse regression by performing a tree search with pruning. It is observed in both noiseless and noisy cases, TSN recovers synthetic and real signals with lower complexity than a conventional tree search and is superior to existing algorithms by a large margin for various types of the sensing matrix $\Phi$, widely used in sparse regression.

A greedy algorithm called Bayesian multiple matching pursuit (BMMP) is proposed to estimate a sparse signal vector and its support given $m$ linear measurements. Unlike the maximum a posteriori (MAP) support detection, which was proposed by Lee to estimate the support by selecting an index with the maximum likelihood ratio of the correlation given by a normalized version of the orthogonal matching pursuit (OMP), the proposed method uses the correlation given by the matching pursuit proposed by Davies and Eldar. BMMP exploits the diversity gain to estimate the support by considering multiple support candidates, each of which is obtained by iteratively selecting an index set with a size different for each candidate. In particular, BMMP considers an extended support estimate whose maximal size is $m$ in the process to obtain each of the support candidates. It is observed that BMMP outperforms other state-of-the-art methods and approaches the ideal limit of the signal sparsity in our simulation setting.

We propose Episodic Backward Update (EBU) - a novel deep reinforcement learning algorithm with a direct value propagation. In contrast to the conventional use of the experience replay with uniform random sampling, our agent samples a whole episode and successively propagates the value of a state to its previous states. Our computationally efficient recursive algorithm allows sparse and delayed rewards to propagate directly through all transitions of the sampled episode. We theoretically prove the convergence of the EBU method and experimentally demonstrate its performance in both deterministic and stochastic environments. Especially in 49 games of Atari 2600 domain, EBU achieves the same mean and median human normalized performance of DQN by using only 5% and 10% of samples, respectively., Comment: NeurIPS 2019

In this paper, we prove a sharp limit on the community detection problem with colored edges. We assume two equal-sized communities and there are $m$ different types of edges. If two vertices are in the same community, the distribution of edges follows $p_i=\alpha_i\log{n}/n$ for $1\leq i \leq m$, otherwise the distribution of edges is $q_i=\beta_i\log{n}/n$ for $1\leq i \leq m$, where $\alpha_i$ and $\beta_i$ are positive constants and $n$ is the total number of vertices. Under these assumptions, a fundamental limit on community detection is characterized using the Hellinger distance between the two distributions. If $\sum_{i=1}^{m} {(\sqrt{\alpha_i} - \sqrt{\beta_i})}^2 >2$, then the community detection via maximum likelihood (ML) estimator is possible with high probability. If $\sum_{i=1}^m {(\sqrt{\alpha_i} - \sqrt{\beta_i})}^2 < 2$, the probability that the ML estimator fails to detect the communities does not go to zero., Comment: The material in this paper was presented in part at the IEEE International Symposium on Information Theory (ISIT) 2016

In this paper, the capacity of continuous-space electromagnetic channels, where transceivers are confined in given lossy regions, is analyzed. First of all, the regions confining the transceivers are assumed to be filled with dielectric, which is either lossy or lossless. Then, for capacity analysis, we use the exact power consumption that takes into account the electromagnetic interaction between the field and the source. In addition, the exact noise model followed from the fluctuation-dissipation theorem in thermodynamics is used at the receive side. The contribution of our work is summarized as follows. First, we characterize the channel capacity as a function of the size and the physical property of the regions confining the transceivers and analytically show how the radiation efficiency affects the capacity. We also show that the outgoing channel at the transmit side and the incoming channel at the receive side are information-theoretically equivalent, and thus, the capacities of both channels are the same. Additionally, the quality factor, which is inversely proportional to the bandwidth, is theoretically derived, and the relationship between the spatial degrees of freedom of the channel and the quality factor is analyzed. Besides, we consider how the power consumption is affected by the backscattered waves and compare the recent experimental demonstration with our work by solving the gain-optimization problem with the constraint on the quality factor., Comment: 37 pages, 10 figures, submitted to IEEE Transactions on Information Theory

In this paper, we address the sparse multiple measurement vector (MMV) problem where the objective is to recover a set of sparse nonzero row vectors or indices of a signal matrix from incomplete measurements. Ideally, regardless of the number of columns in the signal matrix, the sparsity (k) plus one measurements is sufficient for the uniform recovery of signal vectors for almost all signals, i.e., excluding a set of Lebesgue measure zero. To approach the "k+1" lower bound with computational efficiency even when the rank of signal matrix is smaller than k, we propose a greedy algorithm called Two-stage orthogonal Subspace Matching Pursuit (TSMP) whose theoretical results approach the lower bound with less restriction than the Orthogonal Subspace Matching Pursuit (OSMP) and Subspace-Augmented MUltiple SIgnal Classification (SA-MUSIC) algorithms. We provide non-asymptotical performance guarantees of OSMP and TSMP by covering both noiseless and noisy cases. Variants of restricted isometry property and mutual coherence are used to improve the performance guarantees. Numerical simulations demonstrate that the proposed scheme has low complexity and outperforms most existing greedy methods. This shows that the minimum number of measurements for the success of TSMP converges more rapidly to the lower bound than the existing methods as the number of columns of the signal matrix increases., Comment: 55 pages, 8 figures, to be submitted to IEEE Transactions on Information theory, a shorter version was submitted to Proc. IEEE ISIT 2016

This paper studies the Degrees of Freedom (DoF) of the two-user multi-antenna Gaussian interference channel with an {\em instantaneous relay}, or relay without delay, where the relay transmitted signal in channel use $t$ can depend on all received signals up to and including that at channel use $t$. It is assumed that the two transmitters and the two receivers have $M$ antennas, while the relay receives through $N$ antennas and transmits through $L$ antennas. An achievable DoF region is derived, for all possible values of $(M,N,L)$, based on a memoryless linear transmission strategy at the relay that aims to {\it neutralize} as much interference as possible at the receivers. The proposed scheme is shown to attain the largest sum DoF among all memoryless linear transmission strategies at the relay and to actually be optimal for certain values of $(M,N,L)$., Comment: Presented in ISIT 2015

In this paper, we propose a noisy network coding integrated with partial decode-and-forward relaying for single-source multicast discrete memoryless networks (DMN's). Our coding scheme generalizes the partial-decode-compress-and-forward scheme (Theorem 7) by Cover and El Gamal. This is the first time the theorem is generalized for DMN's such that each relay performs both partial decode-and-forward and compress-and-forward simultaneously. Our coding scheme simultaneously generalizes both noisy network coding by Lim, Kim, El Gamal, and Chung and distributed decode-and-forward by Lim, Kim, and Kim. It is not trivial to combine the two schemes because of inherent incompatibility in their encoding and decoding strategies. We solve this problem by sending the same long message over multiple blocks at the source and at the same time by letting the source find the auxiliary covering indices that carry information about the message simultaneously over all blocks., Comment: 5 pages, 1 figure, to appear in Proc. IEEE ISIT 2015

In this paper, we take a unified approach for network information theory and prove a coding theorem, which can recover most of the achievability results in network information theory that are based on random coding. The final single-letter expression has a very simple form, which was made possible by many novel elements such as a unified framework that represents various network problems in a simple and unified way, a unified coding strategy that consists of a few basic ingredients but can emulate many known coding techniques if needed, and new proof techniques beyond the use of standard covering and packing lemmas. For example, in our framework, sources, channels, states and side information are treated in a unified way and various constraints such as cost and distortion constraints are unified as a single joint-typicality constraint. Our theorem can be useful in proving many new achievability results easily and in some cases gives simpler rate expressions than those obtained using conventional approaches. Furthermore, our unified coding can strictly outperform existing schemes. For example, we obtain a generalized decode-compress-amplify-and-forward bound as a simple corollary of our main theorem and show it strictly outperforms previously known coding schemes. Using our unified framework, we formally define and characterize three types of network duality based on channel input-output reversal and network flow reversal combined with packing-covering duality., Comment: 52 pages, 7 figures, submitted to IEEE Transactions on Information theory, a shorter version will appear in Proc. IEEE ISIT 2015

We investigate the total degrees of freedom (DoF) of the K-user rank-deficient interference channel with feedback. For the two-user case, we characterize the total DoF by developing an achievable scheme and deriving a matching upper bound. For the three-user case, we develop a new achievable scheme which employs interference alignment to efficiently utilize the dimension of the received signal space. In addition, we derive an upper bound for the general K-user case and show the tightness of the bound when the number of antennas at each node is sufficiently large. As a consequence of these results, we show that feedback can increase the DoF when the number of antennas at each node is large enough as compared to the ranks of channel matrices. This finding is in contrast to the full-rank interference channel where feedback provides no DoF gain. The gain comes from using feedback to provide alternative signal paths, thereby effectively increasing the ranks of desired channel matrices., Comment: The material in this paper will be presented in part at the IEEE International Symposium on Information Theory (ISIT) 2013 and was in part submitted to the Allerton Conference on Communication, Control, and Computing 2013

We establish an achievable rate region for discrete memoryless interference relay channels that consist of two source-destination pairs and one or more relays. We develop an achievable scheme combining Han-Kobayashi and noisy network coding schemes. We apply our achievability to two cases. First, we characterize the capacity region of a class of discrete memoryless interference relay channels. This class naturally generalizes the injective deterministic discrete memoryless interference channel by El Gamal and Costa and the deterministic discrete memoryless relay channel with orthogonal receiver components by Kim. Moreover, for the Gaussian interference relay channel with orthogonal receiver components, we show that our scheme achieves a better sum rate than that of noisy network coding., Comment: 18 pages, 4 figures

In this paper, the capacity of wireless channels is characterized based on electromagnetic and antenna theories with only minimal assumptions. We assume the transmitter can generate an arbitrary current distribution inside a spherical region and the receive antennas are uniformly distributed on a bigger sphere surrounding the transmitter. The capacity is shown to be $(\alpha P/N_0) \log e$ [bits/sec] in the limit of large number of receive antennas, where $P$ is the transmit power constraint, $\alpha$ is the normalized density of the receive antennas and $N_0$ is the noise power spectral density. Although this result may look trivial, it is surprising in two ways. First, this result holds regardless of the bandwidth (bandwidth can even be negligibly small). Second, this result shows that the capacity is irrespective of the size of the region containing the transmitter. This is against some previous results that claimed the maximum degrees of freedom is proportional to the surface area containing the transmitter normalized by the square of the wavelength. Our result has important practical implications since it shows that even a compact antenna array with negligible bandwidth and antenna spacing well below the wavelength can provide a huge throughput as if the array was big enough so that the antenna spacing is on the order of the wavelength., Comment: 5 pages, to appear in proceedings of 2013 IEEE ISIT

In this paper, we study causal discrete-memoryless relay networks (DMRNs). The network consists of multiple nodes, each of which can be a source, relay, and/or destination. In the network, there are two types of relays, i.e., relays with one sample delay (strictly causal) and relays without delay (causal) whose transmit signal depends not only on the past received symbols but also on the current received symbol. For this network, we derive two new cut-set bounds, one when the causal relays have their own messages and the other when not. Using examples of a causal vector Gaussian two-way relay channel and a causal vector Gaussian relay channel, we show that the new cut-set bounds can be achieved by a simple amplify-and-forward type relaying. Our result for the causal relay channel strengthens the previously known capacity result for the same channel by El Gamal, Hassanpour, and Mammen., Comment: 27 pages, 2 figures, submitted to IEEE Transactions on Information Theory

In this paper, we study the interference relay-without-delay channel which is an interference channel with a relay helping the communication. We assume the relay's transmit symbol depends not only on its past received symbols but also on its current received symbol, which is an appropriate model for studying amplify-and-forward type relaying when the overall delay spread is much smaller than the inverse of the bandwidth. For the discrete memoryless interference relay-without-delay channel, we show an outer bound using genie-aided outer bounding. For the Gaussian interference relay-without-delay channel, we define strong and very strong interference relay-without-delay channels and propose an achievable scheme based on instantaneous amplify-and-forward (AF) relaying. We also propose two outer bounds for the strong and very strong cases. Using the proposed achievable scheme and outer bounds, we show that our scheme can achieve the capacity exactly when the relay's transmit power is greater than a certain threshold. This is surprising since the conventional AF relaying is usually only asymptotically optimal, not exactly optimal. The proposed scheme can be useful in many practical scenarios due to its optimality as well as its simplicity., Comment: 19 pages, 2 figures

We show that the 2x2x2 interference channel, i.e., the multihop interference channel formed by concatenation of two 2-user interference channels achieves the min-cut outer bound value of 2 DoF, for almost all values of channel coefficients, for both time-varying or fixed channel coefficients. The key to this result is a new idea, called aligned interference neutralization, that provides a way to align interference terms over each hop in a manner that allows them to be cancelled over the air at the last hop.

In this paper, we show that the capacity of the three-node Gaussian relay channel can be achieved to within 1 and 2 bit/sec/Hz using compress-and-forward and amplify-and-forward relaying, respectively., Comment: 6 pages, 7 figures

In this paper, we characterize the capacity of a new class of single-source multicast discrete memoryless relay networks having a tree topology in which the root node is the source and each parent node in the graph has at most one noisy child node and any number of noiseless child nodes. This class of multicast tree networks includes the class of diamond networks studied by Kang and Ulukus as a special case, where they showed that the capacity can be strictly lower than the cut-set bound. For achievablity, a novel coding scheme is constructed where each noisy relay employs a combination of decode-and-forward (DF) and compress-and-forward (CF) and each noiseless relay performs a random binning such that codebook constructions and relay operations are independent for each node and do not depend on the network topology. For converse, a new technique of iteratively manipulating inequalities exploiting the tree topology is used., Comment: 27 pages, 2 figures, 1 table. Revised version in response to referee comments (in revision for IEEE Transactions on Information Theory)

We study a layered $K$-user $M$-hop Gaussian relay network consisting of $K_m$ nodes in the $m^{\operatorname{th}}$ layer, where $M\geq2$ and $K=K_1=K_{M+1}$. We observe that the time-varying nature of wireless channels or fading can be exploited to mitigate the inter-user interference. The proposed amplify-and-forward relaying scheme exploits such channel variations and works for a wide class of channel distributions including Rayleigh fading. We show a general achievable degrees of freedom (DoF) region for this class of Gaussian relay networks. Specifically, the set of all $(d_1,..., d_K)$ such that $d_i\leq 1$ for all $i$ and $\sum_{i=1}^K d_i\leq K_{\Sigma}$ is achievable, where $d_i$ is the DoF of the $i^{\operatorname{th}}$ source--destination pair and $K_{\Sigma}$ is the maximum integer such that $K_{\Sigma}\leq \min_m\{K_m\}$ and $M/K_{\Sigma}$ is an integer. We show that surprisingly the achievable DoF region coincides with the cut-set outer bound if $M/\min_m\{K_m\}$ is an integer, thus interference-free communication is possible in terms of DoF. We further characterize an achievable DoF region assuming multi-antenna nodes and general message set, which again coincides with the cut-set outer bound for a certain class of networks., Comment: 19 pages, 5 figures, published in IEEE Transactions on Information Theory

A noisy network coding scheme for sending multiple sources over a general noisy network is presented. For multi-source multicast networks, the scheme naturally extends both network coding over noiseless networks by Ahlswede, Cai, Li, and Yeung, and compress-forward coding for the relay channel by Cover and El Gamal to general discrete memoryless and Gaussian networks. The scheme also recovers as special cases the results on coding for wireless relay networks and deterministic networks by Avestimehr, Diggavi, and Tse, and coding for wireless erasure networks by Dana, Gowaikar, Palanki, Hassibi, and Effros. The scheme involves message repetition coding, relay signal compression, and simultaneous decoding. Unlike previous compress--forward schemes, where independent messages are sent over multiple blocks, the same message is sent multiple times using independent codebooks as in the network coding scheme for cyclic networks. Furthermore, the relays do not use Wyner--Ziv binning as in previous compress-forward schemes, and each decoder performs simultaneous joint typicality decoding on the received signals from all the blocks without explicitly decoding the compression indices. A consequence of this new scheme is that achievability is proved simply and more generally without resorting to time expansion to extend results for acyclic networks to networks with cycles. The noisy network coding scheme is then extended to general multi-source networks by combining it with decoding techniques for interference channels. For the Gaussian multicast network, noisy network coding improves the previously established gap to the cutset bound. We also demonstrate through two popular AWGN network examples that noisy network coding can outperform conventional compress-forward, amplify-forward, and hash-forward schemes., Comment: 33 pages, 4 figures, submitted to IEEE Transactions on Information Theory

In this paper, we characterize the information-theoretic capacity scaling of wireless ad hoc networks with $n$ randomly distributed nodes. By using an exact channel model from Maxwell's equations, we successfully resolve the conflict in the literature between the linear capacity scaling by \"{O}zg\"{u}r et al. and the degrees of freedom limit given as the ratio of the network diameter and the wavelength $\lambda$ by Franceschetti et al. In dense networks where the network area is fixed, the capacity scaling is given as the minimum of $n$ and the degrees of freedom limit $\lambda^{-1}$ to within an arbitrarily small exponent. In extended networks where the network area is linear in $n$, the capacity scaling is given as the minimum of $n$ and the degrees of freedom limit $\sqrt{n}\lambda^{-1}$ to within an arbitrarily small exponent. Hence, we recover the linear capacity scaling by \"{O}zg\"{u}r et al. if $\lambda=O(n^{-1})$ in dense networks and if $\lambda=O(n^{-1/2})$ in extended networks. Otherwise, the capacity scaling is given as the degrees of freedom limit characterized by Franceschetti et al. For achievability, a modified hierarchical cooperation is proposed based on a lower bound on the capacity of multiple-input multiple-output channel between two node clusters using our channel model., Comment: 14 pages, 4 figures. Accepted for publication in IEEE Transactions on Information Theory

A Gaussian multiple-input multiple-output broadcast channel (MIMO GBC) is considered. Throughout the paper it is assumed that 1) input signals are Gaussian and 2) perfect channel state information is available at the transmitter and at the receivers. By considering each data stream as a single user, the uplink-downlink signal-to-interference-plus-noise (SINR) duality is generalized to the MIMO case with general cross-talk matrix. The duality is subsequently applied to finding the solution for the SINR-balancing problem. The result serves as a tool for characterizing achievable rate regions of different coding strategies. Next, we investigate a superposition coding scheme proposed by Cover-van der Meulen-Hajek and Pursley (nicknamed CMHP [1]), where there is a common message to both users. We consider a MIMO broadcast channel with two users, each user has two antennas and the transmitter has four antennas. Assuming one common stream is sent by CMHP coding and successive decoding, a lower bound to the CMHP rate region is found. Behaviors of the CMHP rate region and sumrate are analyzed. We find the sumrate gaps between DPC, CMHP, and MMSE at high SNR for general 2-user multiple-input single-output (MISO) Gaussian broadcast channel. The result suggests when CMHP is beneficial for sumrate., QC 20110411

An efficient interference alignment (IA) scheme is developed for $K$-user single-input single-output frequency selective fading interference channels. The main idea is to steer the transmit beamforming matrices such that at each receiver the subspace dimensions occupied by interference-free desired streams are asymptotically the same as those occupied by all interferences. Our proposed scheme achieves a higher multiplexing gain at any given number of channel realizations in comparison with the original IA scheme, which is known to achieve the optimal multiplexing gain asymptotically., Comment: 3 pages, 1 figure, submitted to IEEE Comm. Letters, Mar. 2009

We consider the multiplexing gain (MUXG) of the fully connected K-user line-of-sight (LOS) interference channels (ICs). A polarimetric antenna composed of 3 orthogonal electric dipoles and 3 orthogonal magnetic dipoles is considered where all 6 dipoles are co-located. In case of K-user IC with single polarization, the maximum achievable MUXG is K regardless of the number of transmit and receive antennas because of the key-hole effect. With polarization, a trivial upper bound on the MUXG is 2K. We propose a zero forcing (ZF) scheme for the K-user LOS IC, where each user uses one or more polarimetric antennas. By using the proposed ZF scheme, we find minimal antenna configurations that achieve this bound for K <= 5. For K > 5, we show that the optimal MUXG of 2K is achieved with M = (K+1)/6 polarimetric antennas at each user., Comment: 6 pages, 5 figures, to appear in Proc. Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA, Sept. 2009

We study a multi-source Gaussian relay network consisting of $K$ source--destination pairs having $K$ unicast sessions. We assume $M$ layers of relays between the sources and the destinations. We find achievable degrees of freedom of the network. Our schemes are based on interference alignment at the transmitters and symbol extension and opportunistic interference cancellation at the relays. For $K$-$L$-$K$ networks, i.e., 2-hop network with $L$ relays, we show $\min\{K,K/2+L/(2(K-1))\}$ degrees of freedom are achievable. For $K$-hop networks with $K$ relays in each layer, we show the full $K$ degrees of freedom are achievable provided that $K$ is even and the channel distribution satisfies a certain symmetry., Comment: 6 pages, 2 figures, presented at the 47-th Allerton Conference on Communication, Control, and Computing, 2009

Motivated by fading channels and erasure channels, the problem of reliable communication over deterministic relay networks is studied, in which relay nodes receive a function of the incoming signals and a random network state. An achievable rate is characterized for the case in which destination nodes have full knowledge of the state information. If the relay nodes receive a linear function of the incoming signals and the state in a finite field, then the achievable rate is shown to be optimal, meeting the cut-set upper bound on the capacity. This result generalizes on a unified framework the work of Avestimehr, Diggavi, and Tse on the deterministic networks with state dependency, the work of Dana, Gowaikar, Palanki, Hassibi, and Effros on linear erasure networks with interference, and the work of Smith and Vishwanath on linear erasure networks with broadcast., Comment: 5 pages, to appear in proc. IEEE ISIT, June 2009

In this paper, we consider a parallel relay network where two relays cooperatively help a source transmit to a destination. We assume the source and the destination nodes are equipped with multiple antennas. Three basic schemes and their achievable rates are studied: Decode-and-Forward (DF), Amplify-and-Forward (AF), and Compress-and-Forward (CF). For the DF scheme, the source transmits two private signals, one for each relay, where dirty paper coding (DPC) is used between the two private streams, and a common signal for both relays. The relays make efficient use of the common information to introduce a proper amount of correlation in the transmission to the destination. We show that the DF scheme achieves the capacity under certain conditions. We also show that the CF scheme is asymptotically optimal in the high relay power limit, regardless of channel ranks. It turns out that the AF scheme also achieves the asymptotic optimality but only when the relays-to-destination channel is full rank. The relative advantages of the three schemes are discussed with numerical results., Comment: 35 pages, 10 figures, submitted to IEEE Transactions on Information Theory

Characterizing the capacity region of multi-source wireless relay networks is one of the fundamental issues in network information theory. The problem is, however, quite challenging due to inter-user interference when there exist multiple source--destination (S--D) pairs in the network. By focusing on a special class of networks, we show that the capacity can be found. Namely, we study a layered linear binary field network with time-varying channels, which is a simplified model reflecting broadcast, interference, and fading natures of wireless communications. We observe that fading can play an important role in mitigating inter-user interference effectively for both single-hop and multi-hop networks. We propose new encoding and relaying schemes with randomized channel pairing, which exploit such channel variations, and derive their achievable rates. By comparing them with the cut-set upper bound, the capacity region of single-hop networks and the sum capacity of multi-hop networks can be characterized for some classes of channel distributions and network topologies. For these classes, we show that the capacity region or sum capacity can be interpreted as the max-flow min-cut theorem., Comment: 19 pages, 7 figures, submitted to IEEE trans. on Information Theory

We study benefits of opportunistic routing in a large wireless ad hoc network by examining how the power, delay, and total throughput scale as the number of source- destination pairs increases up to the operating maximum. Our opportunistic routing is novel in a sense that it is massively parallel, i.e., it is performed by many nodes simultaneously to maximize the opportunistic gain while controlling the inter-user interference. The scaling behavior of conventional multi-hop transmission that does not employ opportunistic routing is also examined for comparison. Our results indicate that our opportunistic routing can exhibit a net improvement in overall power--delay trade-off over the conventional routing by providing up to a logarithmic boost in the scaling law. Such a gain is possible since the receivers can tolerate more interference due to the increased received signal power provided by the multi-user diversity gain, which means that having more simultaneous transmissions is possible., Comment: 18 pages, 7 figures, Under Review for Possible Publication in IEEE Transactions on Information Theory

The separability in parallel Gaussian interference channels (PGICs) is studied in this paper. We generalize the separability results in one-sided PGICs (OPGICs) by Sung \emph{et al.} to two-sided PGICs (TPGICs). Specifically, for strong and mixed TPGICs, we show necessary and sufficient conditions for the separability. For this, we show diagonal covariance matrices are sum-rate optimal for strong and mixed TPGICs., Comment: 5 pages, 1 figure, to appear in proc. IEEE ISIT, June 2009

We study a fading linear finite-field relay network having multiple source-destination pairs. Because of the interference created by different unicast sessions, the problem of finding its capacity region is in general difficult. We observe that, since channels are time-varying, relays can deliver their received signals by waiting for appropriate channel realizations such that the destinations can decode their messages without interference. We propose a block Markov encoding and relaying scheme that exploits such channel variations. By deriving a general cut-set upper bound and an achievable rate region, we characterize the sum capacity for some classes of channel distributions and network topologies. For example, when the channels are uniformly distributed, the sum capacity is given by the minimum average rank of the channel matrices constructed by all cuts that separate the entire sources and destinations. We also describe other cases where the capacity is characterized., Comment: 5 pages, 2 figures, to appear in proc. IEEE ISIT, June 2009

In this paper, a Gaussian two-way relay channel, where two source nodes exchange messages with each other through a relay, is considered. We assume that all nodes operate in full-duplex mode and there is no direct channel between the source nodes. We propose an achievable scheme composed of nested lattice codes for the uplink and structured binning for the downlink. We show that the scheme achieves within 1/2 bit from the cut-set bound for all channel parameters and becomes asymptotically optimal as the signal to noise ratios increase., Comment: 9 pages, 2 figures, submitted to IEEE Transactions on Information Theory

In this paper, a class of relay networks is considered. We assume that, at a node, outgoing channels to its neighbors are orthogonal, while incoming signals from neighbors can interfere with each other. We are interested in the multicast capacity of these networks. As a subclass, we first focus on Gaussian relay networks with interference and find an achievable rate using a lattice coding scheme. It is shown that there is a constant gap between our achievable rate and the information theoretic cut-set bound. This is similar to the recent result by Avestimehr, Diggavi, and Tse, who showed such an approximate characterization of the capacity of general Gaussian relay networks. However, our achievability uses a structured code instead of a random one. Using the same idea used in the Gaussian case, we also consider linear finite-field symmetric networks with interference and characterize the capacity using a linear coding scheme., Comment: 23 pages, 5 figures, submitted to IEEE Transactions on Information Theory

We consider a wireless network in which a single source node located at the center of a unit area having $m$ antennas transmits messages to $n$ randomly located destination nodes in the same area having a single antenna each. To achieve the sum-rate proportional to $m$ by transmit beamforming, channel state information (CSI) is essentially required at the transmitter (CSIT), which is hard to obtain in practice because of the time-varying nature of the channels and feedback overhead. We show that, even without CSIT, the achievable sum-rate scales as $\Theta(m\log m)$ if a cooperation between receivers is allowed. By deriving the cut-set upper bound, we also show that $\Theta(m\log m)$ scaling is optimal. Specifically, for $n=\omega(m^2)$, the simple TDMA-based quantize-and-forward is enough to achieve the capacity scaling. For $n=\omega(m)$ and $n=\operatorname{O}(m^2)$, on the other hand, we apply the hierarchical cooperation to achieve the capacity scaling., Comment: Submitted to the IEEE Transactions on Information Theory

This paper analyzes the impact and benefits of infrastructure support in improving the throughput scaling in networks of $n$ randomly located wireless nodes. The infrastructure uses multi-antenna base stations (BSs), in which the number of BSs and the number of antennas at each BS can scale at arbitrary rates relative to $n$. Under the model, capacity scaling laws are analyzed for both dense and extended networks. Two BS-based routing schemes are first introduced in this study: an infrastructure-supported single-hop (ISH) routing protocol with multiple-access uplink and broadcast downlink and an infrastructure-supported multi-hop (IMH) routing protocol. Then, their achievable throughput scalings are analyzed. These schemes are compared against two conventional schemes without BSs: the multi-hop (MH) transmission and hierarchical cooperation (HC) schemes. It is shown that a linear throughput scaling is achieved in dense networks, as in the case without help of BSs. In contrast, the proposed BS-based routing schemes can, under realistic network conditions, improve the throughput scaling significantly in extended networks. The gain comes from the following advantages of these BS-based protocols. First, more nodes can transmit simultaneously in the proposed scheme than in the MH scheme if the number of BSs and the number of antennas are large enough. Second, by improving the long-distance signal-to-noise ratio (SNR), the received signal power can be larger than that of the HC, enabling a better throughput scaling under extended networks. Furthermore, by deriving the corresponding information-theoretic cut-set upper bounds, it is shown under extended networks that a combination of four schemes IMH, ISH, MH, and HC is order-optimal in all operating regimes., Comment: 26 pages, 10 figures, 1 table, Under revision for IEEE Transactions on Information Theory

The multiplexing gain (MUXG) of $K$-user interference channel (IC) with partially connected interfering links is analyzed. The motivation for the partially connected IC comes from the fact that not all interferences are equally strong in practice. The MUXG is characterized as a function of the number ($K$) of users and the number ($N \geq 1$) of interfering links. Our analysis is mainly based on the interference alignment (IA) technique to mitigate interference. Our main results are as follows: One may expect that higher MUXG can be attained when some of interfering links do not exist. However, when $N$ is odd and $K=N+2$, the MUXG is not increased beyond the optimal MUXG of fully connected IC, which is $\frac{KM}{2}$. The number of interfering links has no influence on the achievable MUXG using IA, but affects the efficiency in terms of the number of required channel realizations: When N=1 or 2, the optimal MUXG of the fully connected IC is achievable with a finite number of channel realizations. In case of $N \geq 3$, however, the MUXG of $\frac{KM}{2}$ can be achieved asymptotically as the number of channel realizations tends to infinity., Comment: Submitted to IEEE Transactions on Information Theory (Correspondence), June 29, 2008

We study two distinct, but overlapping, networks that operate at the same time, space, and frequency. The first network consists of $n$ randomly distributed \emph{primary users}, which form either an ad hoc network, or an infrastructure-supported ad hoc network with $l$ additional base stations. The second network consists of $m$ randomly distributed, ad hoc secondary users or cognitive users. The primary users have priority access to the spectrum and do not need to change their communication protocol in the presence of secondary users. The secondary users, however, need to adjust their protocol based on knowledge about the locations of the primary nodes to bring little loss to the primary network's throughput. By introducing preservation regions around primary receivers and avoidance regions around primary base stations, we propose two modified multihop routing protocols for the cognitive users. Base on percolation theory, we show that when the secondary network is denser than the primary network, both networks can simultaneously achieve the same throughput scaling law as a stand-alone network. Furthermore, the primary network throughput is subject to only a vanishingly fractional loss. Specifically, for the ad hoc and the infrastructure-supported primary models, the primary network achieves sum throughputs of order $n^{1/2}$ and $\max\{n^{1/2},l\}$, respectively. For both primary network models, for any $\delta>0$, the secondary network can achieve sum throughput of order $m^{1/2-\delta}$ with an arbitrarily small fraction of outage. Thus, almost all secondary source-destination pairs can communicate at a rate of order $m^{-1/2-\delta}$., Comment: 28 pages, 12 figures, submitted to IEEE Trans. on Information Theory

Thesis (Ph.D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2000., Includes bibliographical references (p. 235-242)., This thesis proposes two constructive methods of approaching the Shannon limit very closely. Interestingly, these two methods operate in opposite regions, one has a block length of one and the other has a block length approaching infinity. The first approach is based on novel memoryless joint source-channel coding schemes. We first show some examples of sources and channels where no coding is optimal for all values of the signal~to-noise ratio (SNR). When the source bandwidth is greater than the channel bandwidth, joint coding schemes based on space-filling curves and other families of curves are proposed. For uniform sources and modulo channels, our coding scheme based on space-filling curves operates within 1.1 dB of Shannon's rate-distortion bound. For Gaussian sources and additive white Gaussian noise (AWGN) channels, we can achieve within 0.9 dB of the rate-distortion bound. The second scheme is based on low-density parity-check (LDPC) codes. We first demonstrate that we can translate threshold values of an LDPC code between channels accurately using a simple mapping. We develop some models for density evolution from this observation, namely erasure-channel, Gaussian-capacity, and reciprocal channel approximations. The reciprocal-channel approximation, based on dualizing LDPC codes, provides a very accurate model of density evolution for the AWGN channel. We also develop another approximation method, Gaussian approximation, which enables us to visualize infinite-dimensional density evolution and optimization of LDPC codes. We also develop other tools to better understand density evolution. Using these tools, we design some LDPC codes that approach the Shannon limit extremely closely. For multilevel AWGN channels, we design a rate 1/2 code that has a threshold within 0.0063 dB of the Shannon limit of the noisiest level. For binary input AWGN channels, our best rate 1/2 LDPC code has a .threshold within 0.0045 dB of the Shannon limit. Simulation results show that, by Sae-Young Chung., Ph.D.

Thesis (Ph.D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2000., Includes bibliographical references (p. 235-242)., This thesis proposes two constructive methods of approaching the Shannon limit very closely. Interestingly, these two methods operate in opposite regions, one has a block length of one and the other has a block length approaching infinity. The first approach is based on novel memoryless joint source-channel coding schemes. We first show some examples of sources and channels where no coding is optimal for all values of the signal~to-noise ratio (SNR). When the source bandwidth is greater than the channel bandwidth, joint coding schemes based on space-filling curves and other families of curves are proposed. For uniform sources and modulo channels, our coding scheme based on space-filling curves operates within 1.1 dB of Shannon's rate-distortion bound. For Gaussian sources and additive white Gaussian noise (AWGN) channels, we can achieve within 0.9 dB of the rate-distortion bound. The second scheme is based on low-density parity-check (LDPC) codes. We first demonstrate that we can translate threshold values of an LDPC code between channels accurately using a simple mapping. We develop some models for density evolution from this observation, namely erasure-channel, Gaussian-capacity, and reciprocal channel approximations. The reciprocal-channel approximation, based on dualizing LDPC codes, provides a very accurate model of density evolution for the AWGN channel. We also develop another approximation method, Gaussian approximation, which enables us to visualize infinite-dimensional density evolution and optimization of LDPC codes. We also develop other tools to better understand density evolution. Using these tools, we design some LDPC codes that approach the Shannon limit extremely closely. For multilevel AWGN channels, we design a rate 1/2 code that has a threshold within 0.0063 dB of the Shannon limit of the noisiest level. For binary input AWGN channels, our best rate 1/2 LDPC code has a .threshold within 0.0045 dB of the Shannon limit. Simulation results show that, by Sae-Young Chung., Ph.D.

Thesis (Ph.D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2000., Includes bibliographical references (p. 235-242)., This thesis proposes two constructive methods of approaching the Shannon limit very closely. Interestingly, these two methods operate in opposite regions, one has a block length of one and the other has a block length approaching infinity. The first approach is based on novel memoryless joint source-channel coding schemes. We first show some examples of sources and channels where no coding is optimal for all values of the signal~to-noise ratio (SNR). When the source bandwidth is greater than the channel bandwidth, joint coding schemes based on space-filling curves and other families of curves are proposed. For uniform sources and modulo channels, our coding scheme based on space-filling curves operates within 1.1 dB of Shannon's rate-distortion bound. For Gaussian sources and additive white Gaussian noise (AWGN) channels, we can achieve within 0.9 dB of the rate-distortion bound. The second scheme is based on low-density parity-check (LDPC) codes. We first demonstrate that we can translate threshold values of an LDPC code between channels accurately using a simple mapping. We develop some models for density evolution from this observation, namely erasure-channel, Gaussian-capacity, and reciprocal channel approximations. The reciprocal-channel approximation, based on dualizing LDPC codes, provides a very accurate model of density evolution for the AWGN channel. We also develop another approximation method, Gaussian approximation, which enables us to visualize infinite-dimensional density evolution and optimization of LDPC codes. We also develop other tools to better understand density evolution. Using these tools, we design some LDPC codes that approach the Shannon limit extremely closely. For multilevel AWGN channels, we design a rate 1/2 code that has a threshold within 0.0063 dB of the Shannon limit of the noisiest level. For binary input AWGN channels, our best rate 1/2 LDPC code has a .threshold within 0.0045 dB of the Shannon limit. Simulation results show that, by Sae-Young Chung., Ph.D.

Thesis (Ph.D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2000., Includes bibliographical references (p. 235-242)., This thesis proposes two constructive methods of approaching the Shannon limit very closely. Interestingly, these two methods operate in opposite regions, one has a block length of one and the other has a block length approaching infinity. The first approach is based on novel memoryless joint source-channel coding schemes. We first show some examples of sources and channels where no coding is optimal for all values of the signal~to-noise ratio (SNR). When the source bandwidth is greater than the channel bandwidth, joint coding schemes based on space-filling curves and other families of curves are proposed. For uniform sources and modulo channels, our coding scheme based on space-filling curves operates within 1.1 dB of Shannon's rate-distortion bound. For Gaussian sources and additive white Gaussian noise (AWGN) channels, we can achieve within 0.9 dB of the rate-distortion bound. The second scheme is based on low-density parity-check (LDPC) codes. We first demonstrate that we can translate threshold values of an LDPC code between channels accurately using a simple mapping. We develop some models for density evolution from this observation, namely erasure-channel, Gaussian-capacity, and reciprocal channel approximations. The reciprocal-channel approximation, based on dualizing LDPC codes, provides a very accurate model of density evolution for the AWGN channel. We also develop another approximation method, Gaussian approximation, which enables us to visualize infinite-dimensional density evolution and optimization of LDPC codes. We also develop other tools to better understand density evolution. Using these tools, we design some LDPC codes that approach the Shannon limit extremely closely. For multilevel AWGN channels, we design a rate 1/2 code that has a threshold within 0.0063 dB of the Shannon limit of the noisiest level. For binary input AWGN channels, our best rate 1/2 LDPC code has a .threshold within 0.0045 dB of the Shannon limit. Simulation results show that, by Sae-Young Chung., Ph.D.