Search

Continual learning (CL) has garnered significant attention because of its ability to adapt to new tasks that arrive over time. Catastrophic forgetting (of old tasks) has been identified as a major issue in CL, as the model adapts to new tasks. The Mixture-of-Experts (MoE) model has recently been shown to effectively mitigate catastrophic forgetting in CL, by employing a gating network to sparsify and distribute diverse tasks among multiple experts. However, there is a lack of theoretical analysis of MoE and its impact on the learning performance in CL. This paper provides the first theoretical results to characterize the impact of MoE in CL via the lens of overparameterized linear regression tasks. We establish the benefit of MoE over a single expert by proving that the MoE model can diversify its experts to specialize in different tasks, while its router learns to select the right expert for each task and balance the loads across all experts. Our study further suggests an intriguing fact that the MoE in CL needs to terminate the update of the gating network after sufficient training rounds to attain system convergence, which is not needed in the existing MoE studies that do not consider the continual task arrival. Furthermore, we provide explicit expressions for the expected forgetting and overall generalization error to characterize the benefit of MoE in the learning performance in CL. Interestingly, adding more experts requires additional rounds before convergence, which may not enhance the learning performance. Finally, we conduct experiments on both synthetic and real datasets to extend these insights from linear models to deep neural networks (DNNs), which also shed light on the practical algorithm design for MoE in CL.

In this paper, we study a sampling problem where a source takes samples from a Wiener process and transmits them through a wireless channel to a remote estimator. Due to channel fading, interference, and potential collisions, the packet transmissions are unreliable and could take random time durations. Our objective is to devise an optimal causal sampling policy that minimizes the long-term average mean square estimation error. This optimal sampling problem is a recursive optimal stopping problem, which is generally quite difficult to solve. However, we prove that the optimal sampling strategy is, in fact, a simple threshold policy where a new sample is taken whenever the instantaneous estimation error exceeds a threshold. This threshold remains a constant value that does not vary over time. By exploring the structure properties of the recursive optimal stopping problem, a low-complexity iterative algorithm is developed to compute the optimal threshold. This work generalizes previous research by incorporating both transmission errors and random transmission times into remote estimation. Numerical simulations are provided to compare our optimal policy with the zero-wait and age-optimal policies., Comment: Accepted by ACM Sigmetrics, will appear in ACM POMACS journal

We study a class of scheduling problems, where each job is divided into a batch of unit-size tasks and these tasks can be executed in parallel on multiple servers with New-Better-than-Used (NBU) service time distributions. While many delay optimality results are available for single-server queueing systems, generalizing these results to the multi-server case has been challenging. This motivated us to investigate near delay-optimal scheduling of batch jobs in multi-server queueing systems. We consider three lowcomplexity scheduling policies: the Fewest Unassigned Tasks first (FUT) policy, the Earliest Due Date first (EDD) policy, and the First-Come, First-Served (FCFS) policy. We prove that for arbitrary number, batch sizes, arrival times, and due times of the jobs, these scheduling policies are near delay-optimal in stochastic ordering for minimizing three classes of delay metrics among all causal and non-preemptive policies. In particular, the FUT policy is within a constant additive delay gap from the optimum for minimizing the mean average delay, and the FCFS policy within twice of the optimum for minimizing the mean maximum delay and the mean p-norm of delay. The key proof tools are several novel samplepath orderings, which can be used to compare the sample-path delay of different policies in a near-optimal sense., Comment: This is a preprint prepared in 2017. arXiv admin note: substantial text overlap with arXiv:1603.07322

In this paper, we consider a task offloading problem in a multi-access edge computing (MEC) network, in which edge users can either use their local processing unit to compute their tasks or offload their tasks to a nearby edge server through multiple communication channels each with different characteristics. The main objective is to maximize the energy efficiency of the edge users while meeting computing tasks deadlines. In the multi-user multi-channel offloading scenario, users are distributed with partial observations of the system states. We formulate this problem as a stochastic optimization problem and leverage \emph{contextual neural multi-armed bandit} models to develop an energy-efficient deadline-aware solution, dubbed E2DA. The proposed E2DA framework only relies on partial state information (i.e., computation task features) to make offloading decisions. Through extensive numerical analysis, we demonstrate that the E2DA algorithm can efficiently learn an offloading policy and achieve close-to-optimal performance in comparison with several baseline policies that optimize energy consumption and/or response time. Furthermore, we provide a comprehensive set of results on the MEC system performance for various applications such as augmented reality (AR) and virtual reality (VR)., Comment: 2023 IEEE Global Communications Conference

Transfer learning is a useful technique for achieving improved performance and reducing training costs by leveraging the knowledge gained from source tasks and applying it to target tasks. Assessing the effectiveness of transfer learning relies on understanding the similarity between the ground truth of the source and target tasks. In real-world applications, tasks often exhibit partial similarity, where certain aspects are similar while others are different or irrelevant. To investigate the impact of partial similarity on transfer learning performance, we focus on a linear regression model with two distinct sets of features: a common part shared across tasks and a task-specific part. Our study explores various types of transfer learning, encompassing two options for parameter transfer. By establishing a theoretical characterization on the error of the learned model, we compare these transfer learning options, particularly examining how generalization performance changes with the number of features/parameters in both underparameterized and overparameterized regimes. Furthermore, we provide practical guidelines for determining the number of features in the common and task-specific parts for improved generalization performance. For example, when the total number of features in the source task's learning model is fixed, we show that it is more advantageous to allocate a greater number of redundant features to the task-specific part rather than the common part. Moreover, in specific scenarios, particularly those characterized by high noise levels and small true parameters, sacrificing certain true features in the common part in favor of employing more redundant features in the task-specific part can yield notable benefits.

Fairness plays a crucial role in various multi-agent systems (e.g., communication networks, financial markets, etc.). Many multi-agent dynamical interactions can be cast as Markov Decision Processes (MDPs). While existing research has focused on studying fairness in known environments, the exploration of fairness in such systems for unknown environments remains open. In this paper, we propose a Reinforcement Learning (RL) approach to achieve fairness in multi-agent finite-horizon episodic MDPs. Instead of maximizing the sum of individual agents' value functions, we introduce a fairness function that ensures equitable rewards across agents. Since the classical Bellman's equation does not hold when the sum of individual value functions is not maximized, we cannot use traditional approaches. Instead, in order to explore, we maintain a confidence bound of the unknown environment and then propose an online convex optimization based approach to obtain a policy constrained to this confidence region. We show that such an approach achieves sub-linear regret in terms of the number of episodes. Additionally, we provide a probably approximately correct (PAC) guarantee based on the obtained regret bound. We also propose an offline RL algorithm and bound the optimality gap with respect to the optimal fair solution. To mitigate computational complexity, we introduce a policy-gradient type method for the fair objective. Simulation experiments also demonstrate the efficacy of our approach.

Meta-learning has arisen as a successful method for improving training performance by training over many similar tasks, especially with deep neural networks (DNNs). However, the theoretical understanding of when and why overparameterized models such as DNNs can generalize well in meta-learning is still limited. As an initial step towards addressing this challenge, this paper studies the generalization performance of overfitted meta-learning under a linear regression model with Gaussian features. In contrast to a few recent studies along the same line, our framework allows the number of model parameters to be arbitrarily larger than the number of features in the ground truth signal, and hence naturally captures the overparameterized regime in practical deep meta-learning. We show that the overfitted min $\ell_2$-norm solution of model-agnostic meta-learning (MAML) can be beneficial, which is similar to the recent remarkable findings on ``benign overfitting'' and ``double descent'' phenomenon in the classical (single-task) linear regression. However, due to the uniqueness of meta-learning such as task-specific gradient descent inner training and the diversity/fluctuation of the ground-truth signals among training tasks, we find new and interesting properties that do not exist in single-task linear regression. We first provide a high-probability upper bound (under reasonable tightness) on the generalization error, where certain terms decrease when the number of features increases. Our analysis suggests that benign overfitting is more significant and easier to observe when the noise and the diversity/fluctuation of the ground truth of each training task are large. Under this circumstance, we show that the overfitted min $\ell_2$-norm solution can achieve an even lower generalization error than the underparameterized solution.

In this paper, we consider a status update system, where an access point collects measurements from multiple sensors that monitor a common physical process, fuses them, and transmits the aggregated sample to the destination over an erasure channel. Under a typical information fusion scheme, the distortion of the fused sample is inversely proportional to the number of measurements received. Our goal is to minimize the long-term average age while satisfying the average energy and general age-based distortion requirements. Specifically, we focus on the setting in which the distortion requirement is stricter when the age of the update is older. We show that the optimal policy is a mixture of two stationary, deterministic, threshold-based policies, each of which is optimal for a parameterized problem that aims to minimize the weighted sum of the age and energy under the distortion constraint. We then derive analytically the associated optimal average age-cost function and characterize its performance in the large threshold regime, the results of which shed critical insights on the tradeoff among age, energy, and the distortion of the samples. We have also developed a closed-form solution for the special case when the distortion requirement is independent of the age, arguably the most important setting for practical applications.

Decentralized bilevel optimization has received increasing attention recently due to its foundational role in many emerging multi-agent learning paradigms (e.g., multi-agent meta-learning and multi-agent reinforcement learning) over peer-to-peer edge networks. However, to work with the limited computation and communication capabilities of edge networks, a major challenge in developing decentralized bilevel optimization techniques is to lower sample and communication complexities. This motivates us to develop a new decentralized bilevel optimization called DIAMOND (decentralized single-timescale stochastic approximation with momentum and gradient-tracking). The contributions of this paper are as follows: i) our DIAMOND algorithm adopts a single-loop structure rather than following the natural double-loop structure of bilevel optimization, which offers low computation and implementation complexity; ii) compared to existing approaches, the DIAMOND algorithm does not require any full gradient evaluations, which further reduces both sample and computational complexities; iii) through a careful integration of momentum information and gradient tracking techniques, we show that the DIAMOND algorithm enjoys $\mathcal{O}(\epsilon^{-3/2})$ in sample and communication complexities for achieving an $\epsilon$-stationary solution, both of which are independent of the dataset sizes and significantly outperform existing works. Extensive experiments also verify our theoretical findings.

To fully utilize the abundant spectrum resources in millimeter wave (mmWave), Beam Alignment (BA) is necessary for large antenna arrays to achieve large array gains. In practical dynamic wireless environments, channel modeling is challenging due to time-varying and multipath effects. In this paper, we formulate the beam alignment problem as a non-stationary online learning problem with the objective to maximize the received signal strength under interference constraint. In particular, we employ the non-stationary kernelized bandit to leverage the correlation among beams and model the complex beamforming and multipath channel functions. Furthermore, to mitigate interference to other user equipment, we leverage the primal-dual method to design a constrained UCB-type kernelized bandit algorithm. Our theoretical analysis indicates that the proposed algorithm can adaptively adjust the beam in time-varying environments, such that both the cumulative regret of the received signal and constraint violations have sublinear bounds with respect to time. This result is of independent interest for applications such as adaptive pricing and news ranking. In addition, the algorithm assumes the channel is a black-box function and does not require any prior knowledge for dynamic channel modeling, and thus is applicable in a variety of scenarios. We further show that if the information about the channel variation is known, the algorithm will have better theoretical guarantees and performance. Finally, we conduct simulations to highlight the effectiveness of the proposed algorithm.

In this paper, we study the generalization performance of overparameterized 3-layer NTK models. We show that, for a specific set of ground-truth functions (which we refer to as the "learnable set"), the test error of the overfitted 3-layer NTK is upper bounded by an expression that decreases with the number of neurons of the two hidden layers. Different from 2-layer NTK where there exists only one hidden-layer, the 3-layer NTK involves interactions between two hidden-layers. Our upper bound reveals that, between the two hidden-layers, the test error descends faster with respect to the number of neurons in the second hidden-layer (the one closer to the output) than with respect to that in the first hidden-layer (the one closer to the input). We also show that the learnable set of 3-layer NTK without bias is no smaller than that of 2-layer NTK models with various choices of bias in the neurons. However, in terms of the actual generalization performance, our results suggest that 3-layer NTK is much less sensitive to the choices of bias than 2-layer NTK, especially when the input dimension is large.

In this paper, we aim to design an optimal sampler for a system in which fresh samples of a signal (source) are sent through an unreliable channel to a remote estimator, and acknowledgments are sent back over a feedback channel. Both the forward and feedback channels could have random transmission times due to time varying channel conditions. Motivated by distributed sensing, the estimator can estimate the real-time value of the source signal by combining the signal samples received through the channel and the noisy signal observations collected from a local sensor. We prove that the estimation error is a non-decreasing function of the Age of Information (AoI) for the received signal samples and design an optimal sampling strategy that minimizes the long-term average estimation error subject to a sampling rate constraint. The sampling strategy is also optimal for minimizing the long-term average of general non-decreasing functions of the AoI. The optimal sampler design follows a randomized threshold strategy: If the last transmission was successful, the source waits until the expected estimation error upon delivery exceeds a threshold and then sends out a new sample. If the last transmission fails, the source immediately sends out a new sample without waiting. The threshold is the root of a fixed-point equation and can be solved with low complexity (e.g., by bisection search). The optimal sampling strategy holds for general transmission time distributions of the forward and feedback channels. Numerical simulations are provided to compare different sampling policies.

In this paper, we study the generalization performance of min $\ell_2$-norm overfitting solutions for the neural tangent kernel (NTK) model of a two-layer neural network with ReLU activation that has no bias term. We show that, depending on the ground-truth function, the test error of overfitted NTK models exhibits characteristics that are different from the "double-descent" of other overparameterized linear models with simple Fourier or Gaussian features. Specifically, for a class of learnable functions, we provide a new upper bound of the generalization error that approaches a small limiting value, even when the number of neurons $p$ approaches infinity. This limiting value further decreases with the number of training samples $n$. For functions outside of this class, we provide a lower bound on the generalization error that does not diminish to zero even when $n$ and $p$ are both large., Comment: Published in ICML21. This version fixes an error of Lemma 31 and other parts affected by this error. The main results remain the same except some small changes on certain coefficients of Eq.(9)

Identifying "superspreaders" of disease is a pressing concern for society during pandemics such as COVID-19. Superspreaders represent a group of people who have much more social contacts than others. The widespread deployment of WLAN infrastructure enables non-invasive contact tracing via people's ubiquitous mobile devices. This technology offers promise for detecting superspreaders. In this paper, we propose a general framework for WLAN-log-based superspreader detection. In our framework, we first use WLAN logs to construct contact graphs by jointly considering human symmetric and asymmetric interactions. Next, we adopt three vertex centrality measurements over the contact graphs to generate three groups of superspreader candidates. Finally, we leverage SEIR simulation to determine groups of superspreaders among these candidates, who are the most critical individuals for the spread of disease based on the simulation results. We have implemented our framework and evaluate it over a WLAN dataset with 41 million log entries from a large-scale university. Our evaluation shows superspreaders exist on university campuses. They change over the first few weeks of a semester, but stabilize throughout the rest of the term. The data also demonstrate that both symmetric and asymmetric contact tracing can discover superspreaders, but the latter performs better with daily contact graphs. Further, the evaluation shows no consistent differences among three vertex centrality measures for long-term (i.e., weekly) contact graphs, which necessitates the inclusion of SEIR simulation in our framework. We believe our proposed framework and these results may provide timely guidance for public health administrators regarding effective testing, intervention, and vaccination policies., Comment: Accepted to Elsevier High-Confidence Computing Journal

We consider the problem of optimizing the freshness of status updates that are sent from a large number of low-power sources to a common access point. The source nodes utilize carrier sensing to reduce collisions and adopt an asynchronized sleep-wake scheduling strategy to achieve a target network lifetime (e.g., 10 years). We use age of information (AoI) to measure the freshness of status updates, and design sleep-wake parameters for minimizing the weighted-sum peak AoI of the sources, subject to per-source battery lifetime constraints. When the sensing time (i.e., the time duration of carrier sensing) is zero, this sleep-wake design problem can be solved by resorting to a two-layer nested convex optimization procedure; however, for positive sensing times, the problem is non-convex. We devise a low-complexity solution to solve this problem and prove that, for practical sensing times that are short, the solution is within a small gap from the optimum AoI performance. When the mean transmission time of status-update packets is unknown, we devise a reinforcement learning algorithm that adaptively performs the following two tasks in an ``efficient way'': a) it learns the unknown parameter, b) it also generates efficient controls that make channel access decisions. We analyze its performance by quantifying its ``regret'', i.e., the sub-optimality gap between its average performance and the average performance of a controller that knows the mean transmission time. Our numerical and NS-3 simulation results show that our solution can indeed elongate the batteries lifetime of information sources, while providing a competitive AoI performance., Comment: arXiv admin note: substantial text overlap with arXiv:1910.00205

We consider the situation where multiple transportation service providers cooperate to offer an integrated multi-modal platform to enhance the convenience to the passengers through ease in multi-modal journey planning, payment, and first and last mile connectivity. This market structure allows the multi-modal platform to coordinate profits across modes and also provide incentives to the passengers. Accordingly, in this paper, we use cooperative game theory coupled with the hyperpath-based stochastic user equilibrium framework to study such a market. We assume that the platform sets incentives (price discount or excess charge on passengers) along every edge in the transportation network. We derive the continuity and monotonicity properties of the equilibrium flow with respect to the incentives along every edge. The optimal incentives that maximize the profit of the platform are obtained through a two time-scale stochastic approximation algorithm. We use the asymmetric Nash bargaining solution to design a fair profit sharing scheme among the service providers. We show that the profit for each service provider increases after cooperation on such a platform. Finally, we complement the theoretical results through two numerical simulations.

We consider the problem of minimizing the age of information when a source can transmit status updates over two heterogeneous channels. Our work is motivated by recent developments in 5G mmWave technology, where transmissions may occur over an unreliable but fast (e.g., mmWave) channel or a slow reliable (e.g., sub-6GHz) channel. The unreliable channel is modeled as a time-correlated Gilbert-Elliot channel at a high rate when the channel is in the 'ON' state. The reliable channel provides a deterministic but lower data rate. The scheduling strategy determines the channel to be used for transmission in each time slot, aiming to minimize the time-average age of information (AoI). The optimal scheduling problem is formulated as a Markov Decision Process (MDP), which is challenging to solve because super-modularity does not hold in a part of the state space. We address this challenge and show that a multi-dimensional threshold-type scheduling policy is optimal for minimizing the age. By exploiting the structure of the MDP and analyzing the discrete-time Markov chains (DTMCs) of the threshold-type policy, we devise a low-complexity bisection algorithm to compute the optimal thresholds. We compare different scheduling policies using numerical simulations.

In this paper, we consider a status update system, in which update packets are sent to the destination via a wireless medium that allows for multiple rates, where a higher rate also naturally corresponds to a higher error probability. The data freshness is measured using age of information, which is defined as the age of the recent update at the destination. A packet that is transmitted with a higher rate, will encounter a shorter delay and a higher error probability. Thus, the choice of the transmission rate affects the age at the destination. In this paper, we design a low-complexity scheduler that selects between two different transmission rate and error probability pairs to be used at each transmission epoch. This problem can be cast as a Markov Decision Process. We show that there exists a threshold-type policy that is age-optimal. More importantly, we show that the objective function is quasi-convex or non-decreasing in the threshold, based on to the system parameters values. This enables us to devise a \emph{low-complexity algorithm} to minimize the age. These results reveal an interesting phenomenon: While choosing the rate with minimum mean delay is delay-optimal, this does not necessarily minimize the age.

In this paper, we consider transmission scheduling in a status update system, where updates are generated periodically and transmitted over a Gilbert-Elliott fading channel. The goal is to minimize the long-run average age of information (AoI) at the destination under an average energy constraint. We consider two practical cases to obtain channel state information (CSI): (i) \emph{without channel sensing} and (ii) \emph{with delayed channel sensing}. For case (i), the channel state is revealed when an ACK/NACK is received at the transmitter following a transmission, but when no transmission occurs, the channel state is not revealed. Thus, we have to design schemes that balance tradeoffs across energy, AoI, channel exploration, and channel exploitation. The problem is formulated as a constrained partially observable Markov decision process problem (POMDP). To reduce algorithm complexity, we show that the optimal policy is a randomized mixture of no more than two stationary deterministic policies each of which is of a threshold-type in the belief on the channel. For case (ii), (delayed) CSI is available at the transmitter via channel sensing. In this case, the tradeoff is only between the AoI and energy consumption and the problem is formulated as a constrained MDP. The optimal policy is shown to have a similar structure as in case (i) but with an AoI associated threshold. Finally, the performance of the proposed structure-aware algorithms is evaluated numerically and compared with a Greedy policy.

The outbreak of the novel coronavirus (COVID-19) is unfolding as a major international crisis whose influence extends to every aspect of our daily lives. Effective testing allows infected individuals to be quarantined, thus reducing the spread of COVID-19, saving countless lives, and helping to restart the economy safely and securely. Developing a good testing strategy can be greatly aided by contact tracing that provides health care providers information about the whereabouts of infected patients in order to determine whom to test. Countries that have been more successful in corralling the virus typically use a ``test, treat, trace, test'' strategy that begins with testing individuals with symptoms, traces contacts of positively tested individuals via a combinations of patient memory, apps, WiFi, GPS, etc., followed by testing their contacts, and repeating this procedure. The problem is that such strategies are myopic and do not efficiently use the testing resources. This is especially the case with COVID-19, where symptoms may show up several days after the infection (or not at all, there is evidence to suggest that many COVID-19 carriers are asymptotic, but may spread the virus). Such greedy strategies, miss out population areas where the virus may be dormant and flare up in the future. In this paper, we show that the testing problem can be cast as a sequential learning-based resource allocation problem with constraints, where the input to the problem is provided by a time-varying social contact graph obtained through various contact tracing tools. We then develop efficient learning strategies that minimize the number of infected individuals. These strategies are based on policy iteration and look-ahead rules. We investigate fundamental performance bounds, and ensure that our solution is robust to errors in the input graph as well as in the tests themselves.

In this paper, we propose a closed queueing network model for performance analysis of electric vehicle sharing systems with a certain number of chargers in each neighborhood. Depending on the demand distribution, we devise algorithms to compute the optimal fleet size and number of chargers required to maximize profit while maintaining a certain quality of service. We show that the profit is concave with respect to the fleet size and the number of chargers at each charging point. If more chargers are installed within the city, we show that it can not only reduce the fleet size, but it also improves the availability of vehicles at all the points within a city. We further show through simulation that two slow chargers may outperform one fast charger when the variance of charging time becomes relatively large in comparison to the mean charging time.

This paper investigates the problem of regret minimization for multi-armed bandit (MAB) problems with local differential privacy (LDP) guarantee. In stochastic bandit systems, the rewards may refer to the users' activities, which may involve private information and the users may not want the agent to know. However, in many cases, the agent needs to know these activities to provide better services such as recommendations and news feeds. To handle this dilemma, we adopt differential privacy and study the regret upper and lower bounds for MAB algorithms with a given LDP guarantee. In this paper, we prove a lower bound and propose algorithms whose regret upper bounds match the lower bound up to constant factors. Numerical experiments also confirm our conclusions.

This paper studies the sample complexity (aka number of comparisons) bounds for the active best-$k$ items selection from pairwise comparisons. From a given set of items, the learner can make pairwise comparisons on every pair of items, and each comparison returns an independent noisy result about the preferred item. At any time, the learner can adaptively choose a pair of items to compare according to past observations (i.e., active learning). The learner's goal is to find the (approximately) best-$k$ items with a given confidence, while trying to use as few comparisons as possible. In this paper, we study two problems: (i) finding the probably approximately correct (PAC) best-$k$ items and (ii) finding the exact best-$k$ items, both under strong stochastic transitivity and stochastic triangle inequality. For PAC best-$k$ items selection, we first show a lower bound and then propose an algorithm whose sample complexity upper bound matches the lower bound up to a constant factor. For the exact best-$k$ items selection, we first prove a worst-instance lower bound. We then propose two algorithms based on our PAC best items selection algorithms: one works for $k=1$ and is sample complexity optimal up to a loglog factor, and the other works for all values of $k$ and is sample complexity optimal up to a log factor.

We consider reinforcement learning (RL) in Markov Decision Processes in which an agent repeatedly interacts with an environment that is modeled by a controlled Markov process. At each time step $t$, it earns a reward, and also incurs a cost-vector consisting of $M$ costs. We design model-based RL algorithms that maximize the cumulative reward earned over a time horizon of $T$ time-steps, while simultaneously ensuring that the average values of the $M$ cost expenditures are bounded by agent-specified thresholds $c^{ub}_i,i=1,2,\ldots,M$. In order to measure the performance of a reinforcement learning algorithm that satisfies the average cost constraints, we define an $M+1$ dimensional regret vector that is composed of its reward regret, and $M$ cost regrets. The reward regret measures the sub-optimality in the cumulative reward, while the $i$-th component of the cost regret vector is the difference between its $i$-th cumulative cost expense and the expected cost expenditures $Tc^{ub}_i$. We prove that the expected value of the regret vector of UCRL-CMDP, is upper-bounded as $\tilde{O}\left(T^{2\slash 3}\right)$, where $T$ is the time horizon. We further show how to reduce the regret of a desired subset of the $M$ costs, at the expense of increasing the regrets of rewards and the remaining costs. To the best of our knowledge, ours is the only work that considers non-episodic RL under average cost constraints, and derive algorithms that can~\emph{tune the regret vector} according to the agent's requirements on its cost regrets.

We consider a joint sampling and scheduling problem for optimizing data freshness in multi-source systems. Data freshness is measured by a non-decreasing penalty function of \emph{age of information}, where all sources have the same age-penalty function. Sources take turns to generate update packets, and forward them to their destinations one-by-one through a shared channel with random delay. There is a scheduler, that chooses the update order of the sources, and a sampler, that determines when a source should generate a new packet in its turn. We aim to find the optimal scheduler-sampler pairs that minimize the total-average age-penalty at delivery times (Ta-APD) and the total-average age-penalty (Ta-AP). We prove that the Maximum Age First (MAF) scheduler and the zero-wait sampler are jointly optimal for minimizing the Ta-APD. Meanwhile, the MAF scheduler and a relative value iteration with reduced complexity (RVI-RC) sampler are jointly optimal for minimizing the Ta-AP. The RVI-RC sampler is based on a relative value iteration algorithm whose complexity is reduced by exploiting a threshold property in the optimal sampler. Finally, a low-complexity threshold-type sampler is devised via an approximate analysis of Bellman's equation. This threshold-type sampler reduces to a simple water-filling sampler for a linear age-penalty function., Comment: arXiv admin note: text overlap with arXiv:1812.09463

In this paper, we consider the problem of optimizing the freshness of status updates that are sent from a large number of low-power source nodes to a common access point. The source nodes utilize carrier sensing to reduce collisions and adopt an asychronized sleep-wake strategy to achieve an extended battery lifetime (e.g., 10-25 years). We use age of information (AoI) to measure the freshness of status updates, and design the sleep-wake parameters for minimizing the weighted-sum peak AoI of the sources, subject to per-source battery lifetime constraints. When the sensing time is zero, this sleep-wake design problem can be solved by resorting to nested convex optimization; however, for positive sensing times, the problem is non-convex. We devise a low-complexity solution to solve this problem and prove that, for practical sensing times, the solution is within a small gap from the optimum AoI performance. Our numerical and NS-3 simulation results show that our solution can indeed elongate the batteries lifetime of information sources, while providing a competitive AoI performance.

This paper studies the problem of finding the exact ranking from noisy comparisons. A comparison over a set of $m$ items produces a noisy outcome about the most preferred item, and reveals some information about the ranking. By repeatedly and adaptively choosing items to compare, we want to fully rank the items with a certain confidence, and use as few comparisons as possible. Different from most previous works, in this paper, we have three main novelties: (i) compared to prior works, our upper bounds (algorithms) and lower bounds on the sample complexity (aka number of comparisons) require the minimal assumptions on the instances, and are not restricted to specific models; (ii) we give lower bounds and upper bounds on instances with unequal noise levels; and (iii) this paper aims at the exact ranking without knowledge on the instances, while most of the previous works either focus on approximate rankings or study exact ranking but require prior knowledge. We first derive lower bounds for pairwise ranking (i.e., compare two items each time), and then propose (nearly) optimal pairwise ranking algorithms. We further make extensions to listwise ranking (i.e., comparing multiple items each time). Numerical results also show our improvements against the state of the art.

In this paper, delay-optimal and energy-efficient communication is studied for a single link under Markov random arrivals. We present the optimal tradeoff between delay and power over Additive White Gaussian Noise (AWGN) channels and extend the optimal tradeoff for block fading channels. Under time-correlated traffic arrivals, we develop a cross-layer solution that jointly considers the arrival rate, the queue length, and the channel state in order to minimize the average delay subject to a power constraint. For this purpose, we formulate the average delay and power problem as a Constrained Markov Decision Process (CMDP). Based on steady-state analysis for the CMDP, a Linear Programming (LP) problem is formulated to obtain the optimal delay-power tradeoff. We further show the optimal transmission strategy using a Lagrangian relaxation technique. Specifically, the optimal adaptive transmission is shown to have a threshold type of structure, where the thresholds on the queue length are presented for different transmission rates under the given arrival rates and channel states. By exploiting the result, we develop a threshold-based algorithm to efficiently obtain the optimal delay-power tradeoff. We show how a trajectory-sampling version of the proposed algorithm can be developed without the prior need of arrival statistics.

One fundamental challenge in 5G URLLC is how to optimize massive MIMO systems for achieving low latency and high reliability. A natural design choice to maximize reliability and minimize retransmission is to select the lowest allowed target error rate. However, the overall latency is the sum of queueing latency and retransmission latency, hence choosing the lowest target error rate does not always minimize the overall latency. In this paper, we minimize the overall latency by jointly designing the target error rate and transmission rate adaptation, which leads to a fundamental tradeoff point between queueing and retransmission latency. This design problem can be formulated as a Markov decision process, which is theoretically optimal, but its complexity is prohibitively high for real-system deployments. We managed to develop a low-complexity closed-form policy named Large-arraY Reliability and Rate Control (LYRRC), which is proven to be asymptotically latency-optimal as the number of antennas increases. In LYRRC, the transmission rate is twice of the arrival rate, and the target error rate is a function of the antenna number, arrival rate, and channel estimation error. With simulated and measured channels, our evaluations find LYRRC satisfies the latency and reliability requirements of URLLC in all the tested scenarios., Comment: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. Accepted by IEEE Transactions on Wireless Communication

Millimeter wave (mmWave) technologies have the potential to achieve very high data rates, but suffer from intermittent connectivity. In this paper, we provision an architecture to integrate sub-6 GHz and mmWave technologies, where we incorporate the sub-6 GHz interface as a fallback data transfer mechanism to combat blockage and intermittent connectivity of the mmWave communications. To this end, we investigate the problem of scheduling data packets across the mmWave and sub-6 GHz interfaces such that the average delay of system is minimized. This problem can be formulated as Markov Decision Process. We first investigate the problem of discounted delay minimization, and prove that the optimal policy is of the threshold-type, i.e., data packets should always be routed to the mmWave interface as long as the number of packets in the system is smaller than a threshold. Then, we show that the results of the discounted delay problem hold for the average delay problem as well. Through numerical results, we demonstrate that under heavy traffic, integrating sub-6 GHz with mmWave can reduce the average delay by up to 70%. Further, our scheduling policy substantially reduces the delay over the celebrated MaxWeight policy.

In this paper, we consider the problem of minimizing the age of information in a multi-source system, where samples are taken from multiple sources and sent to a destination via a channel with random delay. Due to interference, only one source can be scheduled at a time. We consider the problem of finding a decision policy that determines the sampling times and transmission order of the sources for minimizing the total average peak age (TaPA) and the total average age (TaA) of the sources. Our investigation of this problem results in an important separation principle: The optimal scheduling strategy and the optimal sampling strategy are independent of each other. In particular, we prove that, for any given sampling strategy, the Maximum Age First (MAF) scheduling strategy provides the best age performance among all scheduling strategies. This transforms our overall optimization problem into an optimal sampling problem, given that the decision policy follows the MAF scheduling strategy. While the zero-wait sampling strategy (in which a sample is generated once the channel becomes idle) is shown to be optimal for minimizing the TaPA, it does not always minimize the TaA. We use Dynamic Programming (DP) to investigate the optimal sampling problem for minimizing the TaA. Finally, we provide an approximate analysis of Bellman's equation to approximate the TaA-optimal sampling strategy by a water-filling solution which is shown to be very close to optimal through numerical evaluations.

Simulated annealing (SA) method has had significant recent success in designing distributed control algorithms for wireless networks. These SA based techniques formed the basis of new CSMA algorithms and gave rise to the development of numerous variants to achieve the best system performance accommodating different communication technologies and more realistic system conditions. However, these algorithms do not readily extend to networks with noisy environments, as unreliable communication prevents them from gathering the necessary system state information needed to execute the algorithm. In recognition of this challenge, we propose a new SA algorithm that is designed to work more robustly in networks with communications that experience frequent message drops. The main idea of the proposed algorithm is a novel coupling technique that takes into account the external randomness of message passing failure events as a part of probabilistic uncertainty inherent in stochastic acceptance criterion of SA. As a result, the algorithm can be executed even with partial observation of system states, which was not possible under the traditional SA approach. We show that the newly proposed algorithm finds the optimal solution almost surely under the standard annealing framework while offering significant performance benefits in terms of its computational speed in the presence of frequent message drops., Comment: 12 pages. A short version (8-page) of this paper has been published in Annual Allerton Conference on Communication, Control, and Computing (Allerton) in 2018

This paper explores the adaptive (active) PAC (probably approximately correct) top-$k$ ranking (i.e., top-$k$ item selection) and total ranking problems from $l$-wise ($l\geq 2$) comparisons under the multinomial logit (MNL) model. By adaptively choosing sets to query and observing the noisy output of the most favored item of each query, we want to design ranking algorithms that recover the top-$k$ or total ranking using as few queries as possible. For the PAC top-$k$ ranking problem, we derive a lower bound on the sample complexity (aka number of queries), and propose an algorithm that is sample-complexity-optimal up to an $O(\log(k+l)/\log{k})$ factor. When $l=2$ (i.e., pairwise comparisons) or $l=O(poly(k))$, this algorithm matches the lower bound. For the PAC total ranking problem, we derive a tight lower bound, and propose an algorithm that matches the lower bound. When $l=2$, the MNL model reduces to the popular Plackett-Luce (PL) model. In this setting, our results still outperform the state-of-the-art both theoretically and numerically. We also compare our algorithms with the state-of-the-art using synthetic data as well as real-world data to verify the efficiency of our algorithms.

A fundamental challenge in wireless multicast has been how to simultaneously achieve high-throughput and low-delay for reliably serving a large number of users. In this paper, we show how to harness substantial throughput and delay gains by exploiting multi-channel resources. We develop a new scheme called Multi-Channel Moving Window Codes (MC-MWC) for multi-channel multi-session wireless multicast. The salient features of MC-MWC are three-fold. (i) High throughput: we show that MC-MWC achieves order-optimal throughput in the many-user many-channel asymptotic regime. Moreover, the number of channels required by a conventional channel-allocation based scheme is shown to be doubly-exponentially larger than that required by MC-MWC. (ii) Low delay: using large deviations theory, we show that the delay of MC-MWC decreases linearly with the number of channels, while the delay reduction of conventional schemes is no more than a finite constant. (iii) Low feedback overhead: the feedback overhead of MC-MWC is a constant that is independent of both the number of receivers in each session and the number of sessions in the network. Finally, our trace-driven simulation and numerical results validate the analytical results and show that the implementation complexity of MC-MWC is low., Comment: To Appear in IEEE INFOCOM 2018

Information updates in multihop networks such as Internet of Things (IoT) and intelligent transportation systems have received significant recent attention. In this paper, we minimize the age of a single information flow in interference-free multihop networks. When preemption is allowed and the packet transmission times are exponentially distributed, we prove that a preemptive Last-Generated, First-Served (LGFS) policy results in smaller age processes across all nodes in the network than any other causal policy (in a stochastic ordering sense). In addition, for the class of New-Better-than-Used (NBU) distributions, we show that the non-preemptive LGFS policy is within a constant age gap from the optimum average age. In contrast, our numerical result shows that the preemptive LGFS policy can be very far from the optimum for some NBU transmission time distributions. Finally, when preemption is prohibited and the packet transmission times are arbitrarily distributed, the non-preemptive LGFS policy is shown to minimize the age processes across all nodes in the network among all work-conserving policies (again in a stochastic ordering sense). Interestingly, these results hold under quite general conditions, including (i) arbitrary packet generation and arrival times, and (ii) for minimizing both the age processes in stochastic ordering and any non-decreasing functional of the age processes., Comment: arXiv admin note: text overlap with arXiv:1709.04956, arXiv:1701.05711

In this paper, we investigate the problem of beam alignment in millimeter wave (mmWave) systems, and design an optimal algorithm to reduce the overhead. Specifically, due to directional communications, the transmitter and receiver beams need to be aligned, which incurs high delay overhead since without a priori knowledge of the transmitter/receiver location, the search space spans the entire angular domain. This is further exacerbated under dynamic conditions (e.g., moving vehicles) where the access to the base station (access point) is highly dynamic with intermittent on-off periods, requiring more frequent beam alignment and signal training. To mitigate this issue, we consider an online stochastic optimization formulation where the goal is to maximize the directivity gain (i.e., received energy) of the beam alignment policy within a time period. We exploit the inherent correlation and unimodality properties of the model, and demonstrate that contextual information improves the performance. To this end, we propose an equivalent structured Multi-Armed Bandit model to optimally exploit the exploration-exploitation tradeoff. In contrast to the classical MAB models, the contextual information makes the lower bound on regret (i.e., performance loss compared with an oracle policy) independent of the number of beams. This is a crucial property since the number of all combinations of beam patterns can be large in transceiver antenna arrays, especially in massive MIMO systems. We further provide an asymptotically optimal beam alignment algorithm, and investigate its performance via simulations., Comment: To Appear in IEEE INFOCOM 2018. arXiv admin note: text overlap with arXiv:1611.05724 by other authors

In mobile millimeter wave (mmWave) systems, energy is a scarce resource due to the large losses in the channel and high energy usage by analog-to-digital converters (ADC), which scales with bandwidth. In this paper, we consider a communication architecture that integrates the sub-6 GHz and mmWave technologies in 5G cellular systems. In order to mitigate the energy scarcity in mmWave systems, we investigate the rate-optimal and energy-efficient physical layer resource allocation jointly across the sub-6 GHz and mmWave interfaces. First, we formulate an optimization problem in which the objective is to maximize the achievable sum rate under power constraints at the transmitter and receiver. Our formulation explicitly takes into account the energy consumption in integrated-circuit components, and assigns the optimal power and bandwidth across the interfaces. We consider the settings with no channel state information and partial channel state information at the transmitter and under high and low SNR scenarios. Second, we investigate the energy efficiency (EE) defined as the ratio between the amount of data transmitted and the corresponding incurred cost in terms of power. We use fractional programming and Dinkelbach's algorithm to solve the EE optimization problem. Our results prove that despite the availability of huge bandwidths at the mmWave interface, it may be optimal (in terms of achievable sum rate and energy efficiency) to utilize it partially. Moreover, depending on the sub-6 GHz and mmWave channel conditions and total power budget, it may be optimal to activate only one of the interfaces., Comment: A shorter version to appear in Asilomar Conference on Signals, Systems, and Computers

In this paper, we investigate scheduling policies that minimize the age of information in single-hop queueing systems. We propose a Last-Generated, First-Serve (LGFS) scheduling policy, in which the packet with the earliest generation time is processed with the highest priority. If the service times are i.i.d. exponentially distributed, the preemptive LGFS policy is proven to be age-optimal in a stochastic ordering sense. If the service times are i.i.d. and satisfy a New-Better-than-Used (NBU) distributional property, the non-preemptive LGFS policy is shown to be within a constant gap from the optimum age performance. These age-optimality results are quite general: (i) They hold for arbitrary packet generation times and arrival times (including out-of-order packet arrivals), (ii) They hold for multi-server packet scheduling with the possibility of replicating a packet over multiple servers, (iii) They hold for minimizing not only the time-average age and mean peak age, but also for minimizing the age stochastic process and any non-decreasing functional of the age stochastic process. If the packet generation time is equal to packet arrival time, the LGFS policies reduce to the Last-Come, First-Serve (LCFS) policies. Hence, the age optimality results of LCFS-type policies are also established.

An essential step for achieving multiplexing gain in MIMO downlink systems is to collect accurate channel state information (CSI) from the users. Traditionally, CSIs have to be collected before any data can be transmitted. Such a sequential scheme incurs a large feedback overhead, which substantially limits the multiplexing gain especially in a network with a large number of users. In this paper, we propose a novel approach to mitigate the feedback overhead by leveraging the recently developed Full-duplex radios. Our approach is based on the key observation that using Full-duplex radios, when the base-station (BS) is collecting CSI of one user through the uplink channel, it can use the downlink channel to simultaneously transmit data to other (non-interfering) users for which CSIs are already known. By allowing concurrent channel probing and data transmission, our scheme can potentially achieve a higher throughput compared to traditional schemes using Half-duplex radios. The new flexibility introduced by our scheme, however, also leads to fundamental challenges in achieving throughout optimal scheduling. In this paper, we make an initial effort to this important problem by considering a simplified group interference model. We develop a throughput optimal scheduling policy with complexity $O((N/I)^I)$, where $N$ is the number of users and $I$ is the number of user groups. To further reduce the complexity, we propose a greedy policy with complexity $O(N\log N)$ that not only achieves at least 2/3 of the optimal throughput region, but also outperforms any feasible Half-duplex solutions. We derive the throughput gain offered by Full-duplex under different system parameters and show the advantage of our algorithms through numerical studies., Comment: Technical report

We study the stochastic multi-armed bandit (MAB) problem in the presence of side-observations across actions that occur as a result of an underlying network structure. In our model, a bipartite graph captures the relationship between actions and a common set of unknowns such that choosing an action reveals observations for the unknowns that it is connected to. This models a common scenario in online social networks where users respond to their friends' activity, thus providing side information about each other's preferences. Our contributions are as follows: 1) We derive an asymptotic lower bound (with respect to time) as a function of the bi-partite network structure on the regret of any uniformly good policy that achieves the maximum long-term average reward. 2) We propose two policies - a randomized policy; and a policy based on the well-known upper confidence bound (UCB) policies - both of which explore each action at a rate that is a function of its network position. We show, under mild assumptions, that these policies achieve the asymptotic lower bound on the regret up to a multiplicative factor, independent of the network structure. Finally, we use numerical examples on a real-world social network and a routing example network to demonstrate the benefits obtained by our policies over other existing policies., Comment: minor revision

In this paper, we aim to obtain the optimal delay-power tradeoff and the corresponding optimal scheduling policy for an arbitrary i.i.d. arrival process and adaptive transmissions. The number of backlogged packets at the transmitter is known to a scheduler, who has to determine how many backlogged packets to transmit during each time slot. The power consumption is assumed to be convex in transmission rates. Hence, if the scheduler transmits faster, the delay will be reduced but with higher power consumption. To obtain the optimal delay-power tradeoff and the corresponding optimal policy, we model the problem as a Constrained Markov Decision Process (CMDP), where we minimize the average delay given an average power constraint. By steady-state analysis and Lagrangian relaxation, we can show that the optimal tradeoff curve is decreasing, convex, and piecewise linear, and the optimal policy is threshold-based. Based on the revealed properties of the optimal policy, we develop an algorithm to efficiently obtain the optimal tradeoff curve and the optimal policy with full information of the system. The complexity of our proposed algorithm is much lower than a general algorithm based on Linear Programming. However, usually the distribution of the arrival process is unknown to the scheduler, therefore we proposed a reinforcement learning algorithm to efficiently obtain the optimal policy under this circumstance. We also analyse in details about how the system parameters affect the optimal policy and the system performance. In the final, we use simulations to validate the derived results and the proposed algorithms., Comment: An extended version of our ICC'17 paper. arXiv admin note: substantial text overlap with arXiv:1609.03260

We propose a hybrid architecture that integrates RF (i.e., sub-6 GHz) and millimeter wave (mmWave) technologies for 5G cellular systems. In particular, communications in the mmWave band faces significant challenges due to variable channels, intermittent connectivity, and high energy usage. On the other hand, speeds for electronic processing of data is of the same order as typical rates for mmWave interfaces which makes the use of complex algorithms for tracking channel variations and adjusting resources accordingly impractical. Our proposed architecture integrates the RF and mmWave interfaces for beamforming and data transfer, and exploits the spatio-temporal correlations between the interfaces. Based on extensive experimentation in indoor and outdoor settings, we demonstrate that an integrated RF/mmWave signaling and channel estimation scheme can remedy the problem of high energy usage and delay associated with mmWave beamforming. In addition, cooperation between two interfaces at the higher layers effectively addresses the high delays caused by highly intermittent mmWave connectivity. We design a scheduler that fully exploits the mmWave bandwidth, while the RF link acts as a fallback mechanism to prevent high delay. To this end, we formulate an optimal scheduling problem over the RF and mmWave interfaces where the goal is to maximize the delay-constrained throughput of the mmWave interface. We prove using subadditivity analysis that the optimal scheduling policy is based on a single threshold that can be easily adopted despite high link variations., Comment: 29 pages - A shorter version appeared in WiOpt 2017

The problem of reducing the age-of-information has been extensively studied in the single-hop networks. In this paper, we minimize the age-of-information in general multihop networks. If the packet transmission times over the network links are exponentially distributed, we prove that a preemptive Last Generated First Served (LGFS) policy results in smaller age processes at all nodes of the network (in a stochastic ordering sense) than any other causal policy. In addition, for arbitrary general distributions of packet transmission times, the non-preemptive LGFS policy is shown to minimize the age processes at all nodes of the network among all non-preemptive work-conserving policies (again in a stochastic ordering sense). It is surprising that such simple policies can achieve optimality of the joint distribution of the age processes at all nodes even under arbitrary network topologies, as well as arbitrary packet generation and arrival times. These optimality results not only hold for the age processes, but also for any non-decreasing functional of the age processes., Comment: arXiv admin note: text overlap with arXiv:1603.06185

Cybersecurity is increasingly threatened by advanced and persistent attacks. As these attacks are often designed to disable a system (or a critical resource, e.g., a user account) repeatedly, it is crucial for the defender to keep updating its security measures to strike a balance between the risk of being compromised and the cost of security updates. Moreover, these decisions often need to be made with limited and delayed feedback due to the stealthy nature of advanced attacks. In addition to targeted attacks, such an optimal timing policy under incomplete information has broad applications in cybersecurity. Examples include key rotation, password change, application of patches, and virtual machine refreshing. However, rigorous studies of optimal timing are rare. Further, existing solutions typically rely on a pre-defined attack model that is known to the defender, which is often not the case in practice. In this work, we make an initial effort towards achieving optimal timing of security updates in the face of unknown stealthy attacks. We consider a variant of the influential FlipIt game model with asymmetric feedback and unknown attack time distribution, which provides a general model to consecutive security updates. The defender's problem is then modeled as a time associative bandit problem with dependent arms. We derive upper confidence bound based learning policies that achieve low regret compared with optimal periodic defense strategies that can only be derived when attack time distributions are known., Comment: 9 pages, 2 figures; accepted by the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17), San Francisco, CA, USA, Feb. 2017

In this work, we aim to obtain the optimal tradeoff between the average delay and the average power consumption in a communication system. In our system, the arrivals occur at each timeslot according to a Bernoulli arrival process and are buffered at the transmitter. The transmitter determines the scheduling policy of how many packets to transmit under an average power constraint. The power is assumed to be an increasing and convex function of the number of packets transmitted in each timeslot to capture the realism in communication systems. We also consider a finite buffer and allow the scheduling decision to depend on the buffer occupancy. This problem is modelled as a Constrained Markov Decision Process (CMDP). We first prove that the optimal policy of the (Lagrangian) relaxation of the CMDP is deterministic and threshold-based. We then show that the optimal delay-power tradeoff curve is convex and piecewise linear, where each of the vertices are obtained by the optimal solution to the relaxed problem. This allows us to show the optimal policies of the CMDP are threshold-based, and hence can be implemented by a proposed efficient algorithm. The theoretical results and the algorithm are validated by Linear Programming and simulations., Comment: 30 pages, 8 figures